Design system, design / execution system, design method and design / execution method

Abstract

To provide a design system and the like for enhancing a utilization rate of hardware accelerators during execution of a video pipeline.SOLUTION: A design system according to the present disclosure is for designing a video pipeline, and that comprises: an optimization unit that, based on a pipeline graph, job information related to multiple jobs included in the pipeline graph, hardware accelerator information related to multiple hardware accelerators included in the pipeline graph, and optimization conditions, creates a list of candidate combinations of an assignment of multiple jobs to multiple hardware accelerators, an execution order of the multiple jobs, and an execution timing of each of the multiple jobs; and a result output unit that outputs the list of candidate combinations created by the optimization unit to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a design system, a design / execution system, a design method, and a design / execution method. [Background technology]

[0002] Image processing functions using multiple hardware accelerators (HWA) are now in practical use. To realize such image processing functions, a mechanism is required to design and execute the combination and execution order of the hardware accelerators. This mechanism is called a video pipeline.

[0003] As an example of this mechanism, Non-Patent Document 1 discloses a technology and method for solving an optimization problem that determines the optimal execution order and execution timing of processes based on an objective function such as time when many tasks are repeatedly executed under various constraints. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Cyclic Scheduling Problems” by Thomas Kampmeyer, 2006 Summary of the Invention [Problem to be solved by the invention]

[0005] The technique in Non-Patent Document 1 uses as input data a graph structure in which multiple nodes, each representing a single job, are connected by edges (branches) in the order of execution. The execution order of each node depends on their precedence relationships (for example, a relationship in which one job cannot be executed until another job is completed). In addition to the precedence relationships, optimization parameters include the type and number of resources (hardware accelerators) that execute each job, and the processing time of each job (also called the job execution time). The technique in Non-Patent Document 1 determines the execution order and execution timing of each job based on these parameters, and can obtain a job execution schedule.

[0006] When the technique of Non-Patent Document 1 is applied to a video pipeline, jobs are assigned to each hardware accelerator, and the start timing of each job is also determined. In this case, although it is possible to optimize (minimize) the processing time using multiple hardware accelerators, each hardware accelerator has to wait a time from the end of a job execution until the start of the next job execution. In other words, there is a problem in that the overall utilization rate (operation rate) of the multiple hardware accelerators decreases.

[0007] In this way, when utilization is low, i.e., when each hardware accelerator has a long waiting time, the hardware accelerator is not being used even though there is room to run other applications.

[0008] Other objects and novel features will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0009] According to one embodiment, a design system according to the present disclosure is a design system for designing a video pipeline. an optimization unit that creates a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution order of the plurality of jobs, and execution timing of each of the plurality of jobs, based on a pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions; a result output unit that outputs a list of the combination candidates created by the optimization unit to a user; Equipped with.

[0010] According to one embodiment, a design / execution system according to the present disclosure is a design / execution system for designing and executing a video pipeline. an optimization unit that creates a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution orders of the plurality of jobs, and execution timings of the plurality of jobs based on a pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions, and selects one combination from the list of candidate combinations based on a priority table; a result output unit that outputs the one combination selected by the optimization unit to an external device; a pipeline execution unit that executes the video pipeline based on the one combination output from the result output unit; Equipped with.

[0011] According to one embodiment, a design method according to the present disclosure is a design method for designing a video pipeline. It accepts a Pipeline Graph, creating a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution orders of the plurality of jobs, and execution timings of the plurality of jobs based on the pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions; The created list of combination candidates is output to the user.

[0012] According to one embodiment, a design / execution method according to the present disclosure is a design / execution method for designing and executing a video pipeline. It accepts a Pipeline Graph, creating a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution orders of the plurality of jobs, and execution timings of the plurality of jobs based on the pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions; and selecting one combination from the list of candidate combinations based on a priority table; outputting the selected one combination to the outside; The video pipeline is executed based on the output combination. [Effects of the Invention]

[0013] According to the present disclosure, it is possible to provide a design system, a design / execution system, a design method, a design / execution method, and a program that can increase the utilization rate of a hardware accelerator when executing a video pipeline. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of a design system according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram showing an example of a pipeline graph generated by the graph editor unit shown in FIG. [Figure 3] FIG. 3 is a diagram illustrating an example of allocation of hardware accelerators that causes frame loss. [Figure 4] FIG. 4 is a diagram illustrating an example of how delay changes depending on the hardware accelerator allocation method. [Figure 5] FIG. 5 is a flowchart showing an example of the optimization process executed by the design system shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of parameters input to the optimization unit shown in FIG. [Figure 7] FIG. 7 is a diagram showing an example of a list of candidates that satisfy the constraint C1. [Figure 8] FIG. 8 is a diagram showing whether the list shown in FIG. 7 satisfies the constraint C2. [Figure 9] FIG. 9 is a diagram illustrating information output from the optimization unit illustrated in FIG. [Figure 10] FIG. 10 is a block diagram of a design / execution system according to the second embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating an example of the priority table. [Figure 12] FIG. 12 is a table showing an example of vehicle situations and priority table settings in an in-vehicle application. [Figure 13] FIG. 13 is a flowchart showing an example of optimization processing executed by the design system shown in FIG. [Figure 14] FIG. 14 is a diagram showing state transitions in the pipeline execution unit shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments will be described with reference to the drawings. Note that the drawings are simplified, and the technical scope of the embodiments should not be narrowly interpreted based on the description in the drawings. Furthermore, identical or similar elements are given the same reference numerals, and redundant explanations will be omitted.

[0016] In the following embodiments, when necessary for convenience, the description will be divided into multiple sections or embodiments. However, unless otherwise specified, they are not unrelated to each other, and one is a partial or complete modification, application example, detailed explanation, supplementary explanation, etc. of the other. Furthermore, in the following embodiments, when the number of elements (including the number, numerical value, amount, range, etc.) is mentioned, it is not limited to that specific number, and may be more or less than the specific number, unless otherwise specified or when it is clearly limited to a specific number in principle.

[0017] Furthermore, in the following embodiments, the components (including operational steps, etc.) are not necessarily essential unless otherwise specified or considered to be clearly essential in principle. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., it is intended to include those that are substantially similar or approximate to the shape, etc., unless otherwise specified or considered to be clearly not essential in principle. The same applies to the above numbers, etc. (including numbers, numerical values, amounts, ranges, etc.).

[0018] <Consideration of issues in this disclosure> Before describing embodiments of the design system and design / execution system of the present disclosure, a brief description will be given of the problems anticipated by the present disclosure. Here, reference will also be made to FIG. 2 described in the following embodiment 1 as necessary.

[0019] The design system and design / execution system disclosed herein relate to the design and execution of a pipeline graph for a video pipeline such as that shown in FIG. 2, and are systems for processing image data captured by an imaging device that requires some kind of image processing, such as an on-board camera of a vehicle such as an automobile.

[0020] Here, a pipeline graph is a graph that shows what constraints (for example, precedence relationships) each of multiple jobs is subject to and in what order they are executed. A precedence relationship indicates that one job must be completed before another job can be executed. For example, in the example of Figure 2, the precedence relationship indicates that job 3 cannot be executed unless job 2 is completed.

[0021] In conventional optimization of such video pipelines, it has been assumed that the total execution time of multiple jobs will be reduced (execution time will be minimized) using the technology described in Non-Patent Document 1. In this case, each job represents a process executed on one of the hardware accelerators, and the precedence relationship can be considered as the relationship between the output side and the input side of image data being processed in the video pipeline.

[0022] In such conventional optimization, depending on the job allocation method, each hardware accelerator has to wait for a certain period of time, and it is not possible to optimize the system to reduce this waiting time. In other words, there is a problem that the utilization rate of the hardware accelerator is low during video pipeline execution.

[0023] In particular, when the utilization rate of the hardware accelerator of an in-vehicle SoC (System on a Chip) running many applications is low, i.e., when the waiting time of the hardware accelerator is long, the problem arises that the hardware accelerator cannot be used effectively even though there is room to run other applications.

[0024] The inventors of the present disclosure have studied an optimization method that allows for changing the hardware accelerator to be used for each job in order to maximize the performance of the hardware accelerator, and have arrived at the present disclosure, which can increase the overall utilization rate of the hardware accelerator by using an algorithm that determines the optimal channel allocation and the execution order and execution timing of multiple jobs, without the user having to specify a hardware accelerator channel for each job, and can also allocate unused hardware accelerators to other applications.

[0025] (Embodiment 1) A design system according to a first embodiment of the present disclosure will be described below with reference to Figures 1 to 9. The design system according to this embodiment creates a pipeline graph based on conditions set by a user, and in particular, optimizes the allocation of hardware accelerators. In this disclosure, a detailed description will be given of the design system creating a pipeline graph used for image processing of an in-vehicle camera (hereinafter sometimes abbreviated to camera).

[0026] This design system also outputs to the user a list of hardware accelerator allocation candidates, for example, in ascending order of the number of hardware accelerators to which one or more jobs can be allocated. Here, this list of allocation candidates is a list of candidate combinations of allocation of multiple jobs to multiple hardware accelerators, the execution order of the multiple jobs, and the execution timing of each of the multiple jobs. The user can select and set one combination from this list.

[0027] <Design system configuration> First, the configuration of a design system according to the present embodiment will be described. Fig. 1 is a block diagram of a design system 1 according to the first embodiment of the present disclosure. As shown in Fig. 1, the design system 1 includes a graph editor unit 10, an optimization unit 20, and a result output unit 30.

[0028] The graph editor unit 10 is configured to create the above-described pipeline graph based on information input by the user. The user defines, for example, the processing content of each job in the pipeline graph, and sets the processing time of each job, the type of hardware accelerator to be used for each job, and the precedence relationship between multiple jobs, for the graph editor unit 10. The graph editor unit 10 creates a pipeline graph based on these definitions and settings, and outputs the created pipeline graph to the optimization unit 20.

[0029] If the user does not input the processing time for each job, the graph editor unit 10 may estimate the processing time from the processing content, or may use the predicted results from an execution simulation.

[0030] Furthermore, the design system 1 does not need to include the graph editor unit 10. In this case, the user may perform the above-described operations on an external device (not shown), and the design system 1 may receive the pipeline graph from this external device.

[0031] Here, we will explain the pipeline graph that is input to the optimization unit 20. Fig. 2 is a diagram showing an example of a pipeline graph generated by the graph editor unit 10 shown in Fig. 1. In Fig. 2, each diamond is a node in the graph structure, and represents each job. Also, the precedence relationships between jobs are indicated by arrows.

[0032] In the example shown in FIG. 2, processing is performed as follows. Job 1 is a camera job and has a processing time of 30 ms. When job 1 is completed, jobs 2 and 4 become executable. Jobs 2 and 4 are assigned to two channels ch0 and ch1 of hardware accelerator HWA1, respectively, and jobs 3 and 5 are assigned to two channels ch0 and ch1 of hardware accelerator HWA2. When job 2 is completed, job 3 becomes executable, and when job 4 is completed, job 5 becomes executable. The processing time for jobs 2 and 3 is 10 + 30 = 40 ms, and the processing time for jobs 4 and 5 is 20 + 20 = 40 ms, so the two paths have the same processing time. Finally, when both jobs 3 and 5 are completed, job 6 becomes executable, and when job 6 is completed, the series of processes is completed. The processing time for the entire process is 30 + 40 + 30 = 100 ms.

[0033] Returning to the explanation of FIG. 1, the optimization unit 20 includes an HWA (hardware accelerator) allocation optimization unit 21 and a schedule optimization unit 22. The schedule optimization unit 22 performs optimization processing to reduce the total execution time of multiple jobs and is not specific to the present disclosure. Therefore, detailed explanation thereof will be omitted. In other words, the design system 1 does not need to include the schedule optimization unit 22. Hereinafter, in the explanation of the HWA allocation optimization unit 21, it may also be referred to as the optimization unit 20.

[0034] The HWA allocation optimization unit 21 of the optimization unit 20 receives as input the pipeline graph received from the graph editor unit 10, job information regarding the multiple jobs included in the pipeline graph, hardware accelerator information regarding the multiple hardware accelerators, and optimization conditions. Based on these, the optimization unit 20 is configured to create (determine) a list of candidate combinations of allocation of the multiple jobs to the multiple hardware accelerators, the execution order of the multiple jobs, and the execution timing of each of the multiple jobs.

[0035] Here, the optimization conditions include at least an objective function that indicates how to optimize the combination and constraints on the combination. In this example, the objective function is a function for minimizing the number of hardware accelerators to which one or more jobs among multiple jobs are assigned in the pipeline graph.

[0036] The constraints also include three constraints C1, C2, and C3. Constraint C1 is a condition that "every job is assigned to only one hardware accelerator that can execute that job." This is because a job executed by a hardware accelerator can only be processed by a specific hardware accelerator, and a job must be executed only once within one cycle of the cyclically executed video pipeline.

[0037] Constraint C2 is the condition that "the total execution time of jobs executed by a hardware accelerator is shorter than the frame rate of the camera that serves as input." Here, each hardware accelerator can only execute one job at a time, and if multiple jobs are assigned to it, it executes the next job after completing the currently executed job. Therefore, if the total execution time of the multiple assigned jobs is longer than the input interval (frame rate) of the camera, there is a possibility that images input from the camera will be missed.

[0038] Here, an example of frame loss is shown in Figure 3. Figure 3 is a diagram showing an example of hardware accelerator allocation in which frame loss occurs. Here, the example shows a case where a video pipeline is executed using channel ch0 of two hardware accelerators HWA1 and HWA2.

[0039] In Figure 3(a), the total execution time of the two jobs executed on channel ch0 of the hardware accelerator HWA1 is longer than the input interval of the camera. As a result, the hardware accelerator HWA1 misses the data input from the camera in the fourth frame (timing indicated by a circle). On the other hand, in Figure 3(b), the total execution time of the two jobs executed on channel ch0 of the hardware accelerator HWA1 is shorter than the input interval of the camera. Therefore, the hardware accelerator HWA1 does not miss the data input from the camera.

[0040] Constraint C3 states that "when a hardware accelerator is assigned to each job and the job execution order and execution timing are determined, the time interval between the start timing of one or more jobs that are executed first and the end time of one or more jobs that are finished last is smaller than the upper delay limit." The upper delay limit is a value that can be set in advance by the user. This time interval (hereinafter also referred to as "delay") is directly related to real-time performance, so it is necessary to set an upper limit for this delay from the perspective of safety and functionality of camera processing.

[0041] Here, Fig. 4 shows an example of how delay changes depending on the hardware accelerator allocation method. Fig. 4 is a diagram showing an example of how delay changes depending on the hardware accelerator allocation method. Here, the example shows a case where a video pipeline is executed using two channels ch0 and ch1 of hardware accelerator HWA1 and channel ch0 of hardware accelerator HWA2.

[0042] In Figure 4(a), multiple jobs are assigned so as not to use channel ch1 of hardware accelerator HWA1. There is a precedence constraint between the job assigned to channel ch0 of hardware accelerator HWA1 and the job assigned to channel ch0 of hardware accelerator HWA2. This causes large delays, and the total execution time of all jobs exceeds the upper delay limit.

[0043] On the other hand, in the example of Figure 4(b), two jobs are allocated to channels ch0 and ch1 of the hardware accelerator HWA1, assuming that there are no precedence constraints between the two jobs allocated to channel ch0 of the hardware accelerator HWA1 in Figure 4(a). As a result, the execution timing of the two jobs allocated to channel ch0 of the hardware accelerator HWA1 is advanced. This makes it possible to reduce delays (in this example, there are no delays), and it can be seen that all jobs can be executed within the upper delay limit.

[0044] The result output unit 30 is configured to receive the list of candidate combinations created by the optimization unit 20 from the optimization unit 20, and output this list of candidate combinations to the user as a candidate list. The result output unit 30 is also configured to output to the user information about delays and utilization rates when each candidate combination in the candidate list is executed. The result output unit 30 may be, for example, a display device such as a liquid crystal display.

[0045] The result output unit 30 is also configured to accept a user's selection to select one candidate from the list of combination candidates output to the user. The user checks one or more candidates that satisfy the above three constraints C1, C2, and C3, and selects one candidate from among them. The result output unit 30 may output information about the combination selected by the user to a video pipeline execution device (not shown) or the like.

[0046] <Design system operation> Next, the operation of the design system 1 according to this embodiment will be described. Here, an optimization process will be described in which the graph editor unit 10 creates the pipeline graph shown in FIG. 2, and then the optimization unit 20 optimizes the job execution schedule. After this optimization process, the result output unit 30 outputs (presents) a candidate list to the user. FIG. 5 is a flowchart showing an example of the optimization process executed by the design system 1 shown in FIG. 1.

[0047] When the design system 1 starts the optimization process, the optimization unit 20 receives the pipeline graph created by the graph editor unit 10 and the optimization conditions (step S1). As described above, the optimization conditions include conditions related to hardware accelerators (hereinafter also referred to as HWA conditions) and upper delay limits. The HWA conditions include the maximum number of hardware accelerators present in the execution environment of the user based on the device environment in which this video pipeline is executed. Input parameters for step S1 are shown in FIG. 6. FIG. 6 is a diagram showing an example of parameters input to the optimization unit 20 shown in FIG. 1. In this example, the maximum number of hardware accelerators under the HWA conditions is two, with four channels.

[0048] Based on the information received in step S1, the optimization unit 20 first creates a list of all candidates for hardware accelerator allocation that satisfy condition 1 (step S2). The optimization unit 20 creates a list of all candidates when each job is assigned to a hardware accelerator that can be assigned to that job up to the maximum number of HMA conditions. Note that hardware accelerators of the same type are not distinguished, and all jobs are assigned to one of the hardware accelerators, but there may be a hardware accelerator to which no jobs are assigned.

[0049] The list of candidates created in step S2 is shown in FIG. 7. FIG. 7 is a diagram showing an example of a list of candidates that satisfy the constraint C1. Here, four candidates, candidate A, candidate B, candidate C, and candidate D, are listed. Note that the number of candidates increases as the number of hardware accelerators and jobs increases. Therefore, there is no need to output the list of candidates at this stage.

[0050] Next, the optimization unit 20 excludes candidates that do not satisfy the constraint C2 from the list of candidates created in step S2 (step S3). An example of checking the constraint C2 is shown in FIG. 8. FIG. 8 is a diagram showing whether the list shown in FIG. 7 satisfies the constraint C2. As can be seen from FIG. 8, candidates A and B do not satisfy the constraint C2 because the total job execution time on channel ch0 of the hardware accelerator HWA2 is longer than the camera input interval T. These candidates are excluded at this timing.

[0051] Next, for each candidate remaining in step S3, the optimization unit 20 determines the job execution order and execution timing that minimizes the length of the cycle using the method of Non-Patent Document 1 (step S4).

[0052] Next, the optimization unit 20 calculates the delay for each candidate based on the job execution order and execution timing determined in step S4. Then, the optimization unit 20 determines whether the delay for each candidate is greater than the upper delay limit. If there is a candidate whose delay is greater than the upper delay limit, the optimization unit 20 excludes that candidate at this timing because that candidate does not satisfy constraint condition C3 (step S5). Note that in this example, no candidates are excluded in step S5.

[0053] Next, for each candidate in the list of candidates remaining in step S5, the optimization unit 20 calculates the number of hardware accelerators to which one or more jobs can be assigned, and estimates and predicts the delay and power consumption (step S6). These data may be output to the user by the result output unit 30.

[0054] Next, the optimization unit 20 rearranges the candidates in the candidate list in ascending order of the number of hardware accelerators based on the number of hardware accelerators calculated in step S6, that is, sorts the combination candidates by the number of hardware accelerators (step S7).

[0055] Finally, the optimization unit 20 outputs the sorted list of candidates to the result output unit 30 (step S8), and ends this optimization process.

[0056] As a result of the above optimization process by the optimization unit 20, a list of candidates that satisfy each constraint condition is obtained, as shown in Fig. 9. Fig. 9 is a diagram showing information output from the optimization unit 20 shown in Fig. 1. The result output unit 30 displays the information shown in Fig. 9 on a display device (not shown), thereby outputting (presenting) a list of combination candidates to the user.

[0057] As shown in Fig. 9, various parameters are also displayed to the user to make it easier for them to select one combination from the list of candidate combinations. The various parameters may include, for example, parameters such as delay (the allowable time until execution of all jobs is completed), the utilization rate of the hardware accelerator, and power consumption, as well as graphs showing the execution order and timing of each process.

[0058] As described above, in the design system 1 of this embodiment, the optimization unit 20 determines the allocation of multiple jobs to multiple hardware accelerators on behalf of the user. This allows the user to easily obtain candidates for hardware accelerator allocation that will result in the highest utilization rate simply by creating a pipeline graph. Furthermore, the design system 1 of this embodiment may present a list of candidates weighted not only by utilization rate but also by other parameters such as delay. Furthermore, the design system 1 of this embodiment may be configured to allow the user to select the parameters to be weighted. This allows the user to more easily select the optimal candidate that matches the system requirements.

[0059] As described above, the design system 1 of this embodiment is a design system for designing a video pipeline and includes at least an optimization unit 20 and a result output unit 30. The optimization unit 20 is configured to create a list of candidate combinations of allocation of multiple jobs to multiple hardware accelerators, execution orders of the multiple jobs, and execution timings of each job, based on a pipeline graph, job information, hardware accelerator information, and optimization conditions. The result output unit 30 is configured to output the list of candidate combinations created by the optimization unit 20 to a user. By configuring the design system 1 in this way, the user can check multiple candidate schedules that satisfy the optimization conditions and select the optimal candidate as desired. This can increase the utilization rate of the hardware accelerator during execution of the video pipeline.

[0060] Here, the optimization conditions may include an objective function that indicates how to optimize the combination and constraints on the combination. The objective function may be a function for minimizing the number of hardware accelerators to which one or more jobs among the multiple jobs are assigned. By configuring the design system 1 in this way, there are hardware accelerators that are unused during execution of the video pipeline, and these unused hardware accelerators can be assigned to other applications.

[0061] The design method of this embodiment is a design method for designing a video pipeline. The design method is configured to receive a pipeline graph, and create a list of candidate combinations of allocation of multiple jobs to multiple hardware accelerators, execution orders of the multiple jobs, and execution timings of each job based on the pipeline graph, job information, hardware accelerator information, and optimization conditions, and output the list of candidate combinations to a user. By configuring the design method in this way, it is possible to achieve the same effects as the design system 1 described above.

[0062] (Embodiment 2) Next, a design / execution system according to a second embodiment of the present disclosure will be described with reference to Figs. 10 to 14. The design / execution system according to this embodiment includes a design system similar to the design system 1 according to the first embodiment, and a pipeline execution unit that serves as an execution system that executes the designed video pipeline. The design / execution system according to this embodiment is mounted on a vehicle such as an automobile, for example.

[0063] <Design / Execution System Configuration> First, the configuration of a design / execution system according to the present embodiment will be described. Fig. 10 is a block diagram of a design / execution system 100 according to a second embodiment of the present disclosure. As shown in Fig. 10, the design / execution system 100 includes a design system 2 and a pipeline execution unit 50.

[0064] The design system 2 according to the present embodiment differs from the design system 1 according to the first embodiment in that the optimization unit 20 further includes a candidate selection unit 23 and is provided with a priority table 40. That is, the optimization unit 20 includes an HWA allocation optimization unit 21, a schedule optimization unit 22, and a candidate selection unit 23.

[0065] As in the first embodiment, the HWA allocation optimization unit 21 is configured to create (determine) a list of candidate combinations of allocation of multiple jobs to multiple hardware accelerators, execution order of the multiple jobs, and execution timing of each of the multiple jobs, based on the pipeline graph created by the graph editor unit 10 and user input information.

[0066] The HWA allocation optimization unit 21 outputs the list of candidate combinations created to the candidate selection unit 23. The candidate selection unit 23 is configured to select one combination from the list of candidate combinations based on the priority table 40. That is, the candidate selection unit 23 uses the list of candidate combinations created by the HWA allocation optimization unit 21 and the priority table 40 to automatically rearrange the candidate combinations based on the priority of each setting value or in response to an external instruction, and selects the optimal candidate combination. The candidate selection unit 23 outputs the automatically rearranged list of candidate combinations and the selected one combination to the result output unit 30.

[0067] The result output unit 30 is configured to output one combination selected by the candidate selection unit 23 to the pipeline execution unit 50. The result output unit 30 is also configured to output a list of candidate combinations automatically sorted according to priority as a candidate list to the user and the pipeline execution unit 50. Note that when the pipeline execution unit 50 selects and sets one combination from this candidate list, the result output unit 30 does not need to output this candidate list to the user.

[0068] The priority table 40 is created by a user and stored in a storage unit (not shown) or the like. The priority table 40 holds priority setting values ​​for a plurality of elements that can be set by the user. The priority table 40 created by the user may be output to the pipeline execution unit 50. In this case, the pipeline execution unit 50 may store the priority table 40 in a storage unit (not shown).

[0069] Fig. 11 is a diagram showing an example of the priority table 40. As shown in Fig. 11, the priority table 40 includes, as multiple elements, at least one of the delay, power consumption, number of hardware accelerators, etc., described in the first embodiment. The priority table 40 may also include, as multiple elements, the total execution time when multiple jobs are executed.

[0070] For latency priority, a higher priority means lower latency. For power consumption priority, a higher priority means reduced power consumption. For hardware accelerator number priority, a higher priority means fewer hardware accelerators are used.

[0071] Here, we will explain some examples of conditions inside and outside the vehicle that affect priority. The first condition is when the remaining battery charge of the vehicle falls below a certain level. This condition means that the battery will soon run out and the vehicle will no longer be able to move. For example, because an unintentional stop on a highway due to a dead battery is dangerous, it is necessary to keep the vehicle moving until it can be evacuated to a safe location.

[0072] The second situation is when an approaching object is detected while the vehicle is moving. In this situation, there is a possibility that the vehicle may come into contact with the object if it continues to move. Furthermore, although this depends on the accuracy of the sensor, it is expected that the time until contact with the detected object will be short. Therefore, applications that detect and avoid contact with objects require high real-time performance.

[0073] The parameters for setting the priority table 40 relating to such vehicle conditions are stored in advance as setting condition parameters in a storage unit (not shown) of the pipeline execution unit 50. The setting condition parameters are not limited to parameters relating to the battery and parameters relating to the approach of an object, but may also include many parameters relating to the vehicle state and driving conditions.

[0074] Next, an example of setting the priority table 40 in an in-vehicle application will be described. Fig. 12 is a table showing the vehicle situation in an in-vehicle application and an example of setting the priority table 40. In the example shown in Fig. 12, forward collision warning and traffic sign recognition are shown as examples of in-vehicle applications. Note that when the vehicle is normally traveling, setting 1 of the priority table 40 shown in Fig. 11 is set.

[0075] For example, when the remaining battery charge falls below a reference level, both the forward collision warning and the traffic sign recognition may be set to setting 3 in the priority table 40 shown in FIG. 11. Furthermore, when an approaching object is detected while the vehicle is traveling, the forward collision warning may be set to setting 2 in the priority table 40, and the traffic sign recognition may be set to setting 4 in the priority table 40. Note that such a change in setting is made in response to an instruction from the pipeline execution unit 50. Alternatively, the pipeline execution unit 50 may change the setting in this manner based on the priority table 40 or the candidate list stored in the storage unit, without going through the determination system 2.

[0076] 10, the pipeline execution unit 50 includes an execution unit 51, an SoC control unit 52, and a situation determination unit 53. The execution unit 51 is configured to execute a video pipeline based on one optimal combination output from the design system 2. The SoC control unit 52 is configured to control a semiconductor device (not shown) mounted on the vehicle based on the video pipeline executed by the execution unit 51. The situation determination unit 53 is configured to acquire information related to the state and driving situation of the vehicle and determine the situation of the vehicle.

[0077] Furthermore, the situation determination unit 53 is configured to determine whether the current situation of the vehicle is one of the preset setting situations (stored in the priority table 40). If the situation determination unit 53 determines that the current situation is one of the setting situations, the execution unit 51 is configured to acquire a list of candidate combinations corresponding to the determined setting situation from the optimization unit 20. The execution unit 51 is configured to select an optimal combination from the acquired list of candidate combinations in accordance with setting criteria corresponding to the determined setting situation in the priority table 40. The execution unit 51 is configured to compare the video pipeline corresponding to the optimal combination with the video pipeline currently being executed, and if the video pipelines do not match, change to the video pipeline corresponding to the optimal combination and execute it.

[0078] Furthermore, when changing to a video pipeline corresponding to the optimal combination and executing the video pipeline, the execution unit 51 is configured to instruct the SoC control unit 52 to change the setting values ​​of the semiconductor device. The SoC control unit 52 changes the setting values ​​of the semiconductor device in accordance with this instruction.

[0079] <Operation of the design / execution system> Next, the operation of the design / execution system 100 according to this embodiment will be described. Here, the optimization process executed by the design system 2 and the state transition of the pipeline execution unit 50 in response to changes in the vehicle situation will be described in detail.

[0080] First, we will explain the optimization processing executed by the design system 2. Fig. 13 is a flowchart showing an example of the optimization processing executed by the design system 2 shown in Fig. 10. In the flowchart shown in Fig. 13, the same steps as those in the flowchart shown in Fig. 5 of the first embodiment are assigned the same step numbers, and their explanations will be omitted.

[0081] After excluding candidates that do not satisfy the constraint conditions from the combination candidates in the processes up to step S5 and acquiring data such as delay (step S6), the optimization unit 20 creates a list of the remaining combination candidates (step S11).

[0082] Then, the optimization unit 20 refers to the priority table 40, selects the optimal combination candidate for the current state of the vehicle from the list of combination candidates (step S12), outputs the selected optimal combination candidate to the pipeline execution unit 50 (step S13), and ends this optimization process.

[0083] In addition, the optimization unit 20 may rearrange the candidate combinations in the list according to the priority table 40, create a list of the rearranged candidate combinations, output it to the pipeline execution unit 50, and terminate this optimization process.

[0084] Next, a description will be given of state transitions of the pipeline execution unit 50 in response to changes in the vehicle situation. Fig. 14 is a diagram showing state transitions in the pipeline execution unit 50 shown in Fig. 10. When the pipeline execution unit 50 starts pipeline execution, during the execution, the situation determination unit 53 continues to monitor the detection results of various sensors provided in the vehicle.

[0085] When a change in the situation inside or outside the vehicle is detected (step S21), the situation determination unit 53 determines the situation after the situation change (step S22) and outputs the result to the execution unit 51. The execution unit 51 acquires combination candidates (i.e., multiple video pipeline candidates) from the design system 2 according to the situation inside and outside the vehicle obtained as a result of the situation determination (step S23).

[0086] Then, the execution unit 51 selects candidate video pipelines that match the conditions inside and outside the vehicle, calculates their applicability (e.g., a value indicating the degree to which they should be applied) (step S24), and selects the optimal video pipeline based on the calculation results and the priority table 40 (step S25).

[0087] Next, the execution unit 51 compares the currently set video pipeline with the selected video pipeline (step S26). If the video pipelines match, there is no need to change the settings, and the execution unit 51 ends this series of processes. On the other hand, if the video pipelines do not match, there is a need to change the settings, and the execution unit 51 prepares to change the video pipeline (step S27).

[0088] When preparing to change the video pipeline, the execution unit 51 instructs the SoC control unit 52 to change the setting values ​​of the SoC (not shown) (step S28). Upon receiving this instruction, the SoC control unit 52 changes the setting values ​​of each SoC, and when the setting of each SoC is complete, notifies the execution unit 51 (step S29). Then, the execution unit 51 executes the newly set video pipeline.

[0089] As described above, the design / execution system 100 of this embodiment is a design / execution system for designing and executing a video pipeline, and includes at least an optimization unit 20, a result output unit 30, and a pipeline execution unit 50. The optimization unit 20 is configured to create candidate combinations of allocation of multiple jobs to multiple hardware accelerators, execution orders of the multiple jobs, and execution timings of each job, based on a pipeline graph, job information, hardware accelerator information, and optimization conditions. The optimization unit 20 is also configured to select one combination from a list of candidate combinations based on a priority table 40. The result output unit 30 is configured to output the one combination selected by the optimization unit 20 to the outside. The pipeline execution unit 50 is also configured to execute the video pipeline based on the one combination output from the result output unit 30. This configuration of the design / execution system 100 makes it possible to select and execute a video pipeline corresponding to a more suitable combination without requiring user selection.

[0090] In the design / execution system 100 of this embodiment, one or more setting situations may be set in the priority table 40, and the situation determination unit 53 may determine whether the current situation of the vehicle is one of the one or more setting situations set in advance. Then, based on the determination result of the situation determination unit 53, the execution unit 51 may execute a video pipeline based on a setting appropriate to the determination result. In this way, by preparing multiple setting situations in the priority table 40, it is possible to automatically output an appropriate combination according to the current situation without user intervention. As a result, the user only needs to create a pipeline graph and the priority table 40, and there is no need to determine whether the video pipeline to be executed is optimal.

[0091] The design / execution method of this embodiment is a design / execution method for designing and executing a video pipeline. The design / execution method is configured to receive a pipeline graph, and create candidate combinations of allocation of multiple jobs to multiple hardware accelerators, execution orders of the multiple jobs, and execution timings of each job based on the pipeline graph, job information, hardware accelerator information, and optimization conditions, select one combination from the candidate combinations based on a priority table 40, output the selected combination to the outside, and execute the video pipeline based on the output combination. By configuring the design / execution method in this way, it is possible to achieve effects similar to those of the design / execution system 100 described above.

[0092] The invention made by the inventor has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the embodiments already described, and various modifications are possible within the scope of the gist of the invention.

[0093] Furthermore, in the present disclosure, part or all of the processing performed by the design system 1 or 2 can be realized by causing a CPU to execute a computer program.

[0094] The above-described program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSD) or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray discs or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals. [Explanation of symbols]

[0095] 1, 2 Design System 10 Graph Editor 20 Optimization Section 21 HWA Allocation Optimization Unit 22 Schedule Optimization Department 23 Candidate Selection Section 30 Result output section 40 Priority Table 50 Pipeline Execution Unit 51 Executive Department 52 SoC control unit (semiconductor device control unit) 53 Situation Assessment Department

Claims

1. A design system for designing a video pipeline, comprising: an optimization unit that creates a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution order of the plurality of jobs, and execution timing of each of the plurality of jobs, based on a pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions; a result output unit that outputs a list of the combination candidates created by the optimization unit to a user; Equipped with Design system.

2. The optimization conditions include an objective function indicating how to optimize the combination and constraints on the combination. The design system of claim 1 .

3. the objective function is a function for minimizing the number of the hardware accelerators to which one or more jobs among the plurality of jobs are assigned; The design system according to claim 2 .

4. The video pipeline is used for image processing of a camera, The constraint is: each of the plurality of jobs is assigned to only one of the plurality of hardware accelerators capable of executing the corresponding job; The total execution time of one or more jobs executed by each of the plurality of hardware accelerators is shorter than the frame rate of the camera; and When the plurality of jobs are assigned to the plurality of hardware accelerators and the execution order and execution timing of the plurality of jobs are determined, a time interval from the start time of one or more jobs to be executed first to the end time of one or more jobs to be finished last is smaller than an upper delay limit value; Including, The design system according to claim 2 .

5. The delay upper limit value can be set in advance by the user. The design system according to claim 4 .

6. a graph editor unit that creates the pipeline graph based on the input information of the user; the graph editor unit outputs the created pipeline graph to the optimization unit. The design system of claim 1 .

7. the result output unit accepts the user's selection from the output list of combination candidates. The design system of claim 1 .

8. The optimization unit an HWA allocation optimization unit that optimizes the combination so as to minimize the number of the hardware accelerators to which one or more jobs among the plurality of jobs are assigned; a schedule optimization unit that optimizes the combination so as to minimize the time required to complete execution of all of the plurality of jobs; Including, The design system of claim 1 .

9. the hardware accelerator information includes the types of the hardware accelerators and the number of the hardware accelerators for each type; The design system of claim 1 .

10. 1. A design / execution system for designing and executing a video pipeline, comprising: an optimization unit that creates a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution orders of the plurality of jobs, and execution timings of the plurality of jobs based on a pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions, and selects one combination from the list of candidate combinations based on a priority table; a result output unit that outputs the one combination selected by the optimization unit to an external device; a pipeline execution unit that executes the video pipeline based on the one combination output from the result output unit; Equipped with Design / execution system.

11. the optimization unit sorts the combination candidates in the list of combination candidates based on the priority table; the result output unit outputs a list of the combination candidates sorted by the optimization unit to a user and the pipeline execution unit. The design / execution system of claim 10.

12. The design / execution system is mounted on a vehicle, The pipeline execution unit an execution unit that executes the video pipeline based on the one combination; a semiconductor device control unit that controls a semiconductor device mounted on the vehicle based on the video pipeline executed by the execution unit; a situation determination unit that acquires information about the state and driving situation of the vehicle and determines the situation of the vehicle; Including, The design / execution system of claim 10.

13. the situation determination unit determines whether the current situation of the vehicle is one of one or more preset situations; When the situation determination unit determines that the current situation of the vehicle is one of the one or more set situations, the execution unit: obtaining a list of the combination candidates corresponding to the determined setting situation from the optimization unit; selecting an optimal combination from the acquired list of candidate combinations in accordance with a setting criterion corresponding to the determined setting situation in the priority table; comparing the video pipeline corresponding to the optimal combination with the running video pipeline; If the video pipelines do not match, the video pipeline is changed to a video pipeline corresponding to the optimal combination and executed. The design / execution system of claim 12.

14. When the execution unit changes the video pipeline to one corresponding to the optimal combination and executes the video pipeline, the execution unit instructs the semiconductor device control unit to change the setting values ​​of the semiconductor device. The design / execution system of claim 13.

15. The priority table holds priority setting values ​​for a plurality of elements that can be set by the user, the plurality of elements include at least one of a total execution time when the plurality of jobs in the pipeline graph are executed, the number of the hardware accelerators that execute any of the plurality of jobs, or power consumption; 15. The design / execution system of claim 14.

16. The optimization conditions include an objective function indicating how to optimize the combination and constraints on the combination. The design / execution system of claim 10.

17. the objective function is a function for minimizing the number of the hardware accelerators to which one or more jobs among the plurality of jobs are assigned; 17. The design / execution system of claim 16.

18. A design method for designing a video pipeline, comprising: It accepts a pipeline graph, creating a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution orders of the plurality of jobs, and execution timings of the plurality of jobs based on the pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions; outputting the created list of candidate combinations to a user; Design method.

19. A design / execution method for designing and executing a video pipeline, comprising: It accepts a pipeline graph, creating a list of candidate combinations of allocation of the plurality of jobs to the plurality of hardware accelerators, execution orders of the plurality of jobs, and execution timings of the plurality of jobs based on the pipeline graph, job information related to the plurality of jobs included in the pipeline graph, hardware accelerator information related to the plurality of hardware accelerators included in the pipeline graph, and optimization conditions; and selecting one combination from the list of candidate combinations based on a priority table; outputting the selected one combination to the outside; Executing the video pipeline based on the output combination. Design / Execution Methodology.

Images