Information processing device, information processing method, and program

Abstract

To provide an information processing device, etc. capable of performing inference processing using multiple inference models even under limited computational environments.SOLUTION: The information processing device includes an acquisition unit 150 for acquiring sensing data, a task processing unit 124 including multiple neural networks 125, 126, and a feature classification unit 122 for outputting feature values of the sensing data to each of the multiple neural networks included in the task processing unit. The multiple neural networks 125, 126 included in the task processing unit 124 execute subsequent processing in parallel and produce different outputs on the basis of the feature quantities.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, various devices have been proposed that incorporate an inference model (hereinafter referred to as an NN inference model or simply an NN) that employs a neural network (hereinafter referred to as an NN inference model). For example, it has been proposed to incorporate an NN inference model into Internet of Things (IoT) devices, which generally require low cost and low power consumption and therefore have limited computing power. For example, Patent Document 1 discloses an NN device that can reduce the power consumption of the entire device while maintaining high accuracy by having the NN inference model execute inference at a drive frequency according to the accuracy, amount of calculation, etc. required for each layer of the NN inference model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-005297 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 only discloses a technique for inference processing of a single inference model.

[0005] Therefore, the present disclosure provides an information processing device, an information processing method, and a program that are capable of performing inference processing using multiple inference models even in a limited computing environment. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires sensing data, a task processing unit that includes multiple neural networks, and a feature classification unit that outputs features of the sensing data to each of the multiple neural networks included in the task processing unit, and each of the multiple neural networks included in the task processing unit executes subsequent processing in parallel and provides different outputs based on the features.

[0007] An information processing method according to one aspect of the present disclosure is a method executed by a computer, which acquires sensing data and outputs features of the sensing data to each of a plurality of neural networks, and each of the plurality of neural networks executes subsequent processing in parallel and provides different outputs based on the features.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above-described information processing method. [Effects of the Invention]

[0009] According to one aspect of the present disclosure, it is possible to realize an information processing device or the like that is capable of performing inference processing using multiple inference models even in a limited computing environment. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of the inference processing unit according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of a table including delay flag information and forward propagation calculation methods of specialized neural networks linked to the delay flag information according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating the configuration of an SoC according to the embodiment. [Figure 5] FIG. 5 is a flowchart showing the operation of the device according to the embodiment. [Figure 6A] FIG. 6A is a schematic diagram illustrating an example of processing times and operation units of a common NN and a group of specialized NNs according to an embodiment. [Figure 6B] FIG. 6B is a schematic diagram showing another example of the processing times and operation units of the common NN and the specialized NN group according to the embodiment. [Figure 6C] FIG. 6C is a schematic diagram showing an example of processing times and units responsible for calculations after rearrangement of the calculation order of the common NN and the specialized NN group according to the embodiment. [Figure 7] FIG. 7 is a flowchart showing the operation of the multitasking training unit according to the embodiment. [Figure 8] FIG. 8 is a diagram schematically illustrating the operation of the multitasking training unit according to the embodiment. [Figure 9] FIG. 9 is a flowchart showing the operations of the delay flag information measurement unit and the delay flag correct label generation unit according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating the operation of the delay flag information measurement unit and the delay flag correct label generation unit according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing the operation of the delay flag training unit according to the embodiment. [Figure 12] FIG. 12 is a diagram illustrating the operation of the delay flag training unit according to the embodiment. [Figure 13] FIG. 13 is a table showing the occurrence rate of delay flag information according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Findings that formed the basis of this disclosure) Before describing the embodiments of the present disclosure, the background to the foundation of the present disclosure will be described.

[0012] This disclosure relates to a device (hereinafter referred to as an inference device) equipped with an inference model generated using machine learning. More specifically, it relates to a neural network-type inference device that enables maintenance of service quality by changing the order of operations, and a training process therefor.

[0013] In recent years, the adoption of deep learning technology has dramatically improved the performance of inference devices such as object detection devices and classification devices, leading to numerous research and commercialization efforts. Furthermore, in self-driving cars (robot cars) driven by machines instead of smartphones or drivers, an inference device for images transferred from a camera device, infrared array sensor, etc., is one of the elemental functions. In the case of self-driving cars, the objects refer to, for example, pedestrians, passenger cars, road signs, buildings, road areas, etc.

[0014] In addition, the field of edge AI (Artificial Intelligence) has recently begun to spread. This involves installing an NN inference model in IoT (Internet of Things) devices and allowing the devices to operate based on their own judgments, rather than constantly relying on the cloud for judgments. For example, in the field of various industrial products, including IoT devices, products have been introduced to the market that are equipped with NN-type inference devices and provide services based on the inference results. Incidentally, IoT devices are generally required to be inexpensive and operate with low power consumption, which limits their computing power.

[0015] IoT devices are required to be power-efficient. When such IoT devices provide a service using multiple NN inference models, it is difficult to complete the service within the required processing time unless the multiple NN inference models are calculated in parallel. Thus, for example, in IoT devices, it is expected that calculations for inference will be performed using multiple NN inference models arranged in parallel. The required processing time will also be referred to as the allowable time.

[0016] In addition, IoT devices are beginning to incorporate dedicated neural network processing circuits called NPUs (Neural Network Processing Units), implemented in DSPs (Digital Signal Processors) and ASICs (Application Specific Integrated Circuits), in addition to CPUs (Central Processing Units). However, as the services required for IoT devices increasingly use the inference processing of multiple neural networks as described above, the amount of processing required for these services is on the rise.

[0017] In addition to completing services within the required processing time, IoT devices are also required to be energy-efficient. As a result, IoT devices are prone to issues such as not being able to complete inference processing within the required processing time depending on the input image.

[0018] An example of a service that uses multiple NN inference models is a process that determines, through the inference processing of an NN inference model for skeleton detection, that a target person in a scene image is holding "something" in their right hand, and then determines, through the inference processing of an NN inference model for object detection, that this "something" is a smartphone.

[0019] The above-mentioned Patent Document 1 is a method assuming a service that uses the inference processing of a single NN inference model, and does not disclose a method for ensuring processing time in a service that uses the inference processing of multiple NN inference models. Furthermore, conventionally, no consideration has been given to design or technical solutions for processing that estimates the processing time required for the inference processing of multiple inference models at an early stage and performs inference processing in an optimal calculation order within the required processing time, and Patent Document 1 does not disclose anything like that.

[0020] Therefore, in a service that uses inference processing of multiple NN inference models, the inventors have conducted extensive research into information processing devices and the like that can perform inference processing within the required processing time, and have devised the information processing method and the like described below.

[0021] An information processing device according to one aspect of the present disclosure comprises an acquisition unit that acquires sensing data, an inference processing unit that inputs the sensing data into an inference model and acquires information regarding the inference result and the processing times of multiple subsequent tasks of the inference model processing, a determination unit that determines a task schedule for processing the multiple subsequent tasks of a task processing unit that processes the multiple subsequent tasks based on the information regarding the processing times of the multiple subsequent tasks, and a control unit that inputs the inference result to the task processing unit and causes the multiple subsequent tasks to be processed according to the determined task schedule.

[0022] This allows the information processing device to determine a task schedule for task processing units (e.g., multiple NN inference models) that process multiple subsequent tasks based on information about the processing times of the multiple subsequent tasks for the sensing data.The information processing device determines the task schedule based on information about the processing times of the multiple subsequent tasks so that the processing times of the task processing units are within the required processing time, thereby enabling inference processing using multiple inference models even in a limited computing environment.

[0023] Also, for example, the inference model may include a first inference model and a second inference model, the sensing data being input into the first inference model to obtain the inference result, and the obtained inference result or intermediate data of the inference being input into the second inference model to obtain information regarding the processing time of the multiple subsequent tasks.

[0024] This reduces the amount of calculation required by the second inference model compared to when sensing data is input to the second inference model. Therefore, it is possible to shorten the processing time in the second inference model. Furthermore, when the inference result is input, the input to the second inference model is the same as the input to the subsequent task processing unit, which is expected to improve the accuracy or precision of information regarding the processing time of the subsequent task processing unit. When intermediate data of the inference is input, it is possible to start processing the second inference model before the inference of the first inference model is completed, thereby further shortening the processing time of the second inference model.

[0025] Also, for example, the inference model may include a first inference model and a second inference model, and the sensing data may be input into the first inference model to obtain the inference result, and the sensing data may be input into the second inference model to obtain information regarding the processing times of the multiple subsequent tasks.

[0026] This allows the second inference model to start processing using the sensing data without waiting for the inference result to be output from the first inference model, which means that the timing at which the second inference model starts processing can be accelerated.

[0027] Furthermore, for example, the information relating to the processing times of the plurality of subsequent tasks may include information relating to delays relative to a predetermined time that is set for the processing times of the plurality of subsequent tasks.

[0028] This allows the task schedule of the task processing unit to be determined based on information about delays. By determining the task schedule based on information about delays of multiple subsequent tasks so that the processing time of the task processing unit falls within the required processing time, the information processing device increases the certainty that inference processing can be performed within the required processing time in a service that uses inference processing of multiple inference models.

[0029] Also, for example, if the information regarding the delay includes information indicating that the delay is less than a predetermined value, a first schedule may be determined, and if the information regarding the delay includes information indicating that the delay is greater than or equal to a predetermined value, a second schedule may be determined in which the processing times of the multiple subsequent tasks are shorter than those of the first schedule.

[0030] As a result, if the delay is equal to or greater than a predetermined value, a second schedule that shortens the processing time is determined, thereby increasing the certainty that the inference processing can be performed within the required processing time.

[0031] Also, for example, a first rule may be used to determine the first schedule, and a second rule may be used to determine the second schedule, such that the second schedule has a shorter processing time for the multiple subsequent tasks than the first schedule.

[0032] In this way, by determining a schedule based on the first and second rules, the amount of calculation and the calculation time can be reduced compared to when a schedule is dynamically searched for.

[0033] Also, for example, the inference model may perform an inference process that is a pre-processing common to the multiple subsequent tasks.

[0034] This allows the overall computational complexity and memory usage of the inference processing required for each task to be reduced by using an inference model for preprocessing that is common to each task.

[0035] Also, for example, the inference model and the task processing unit may be neural network models, the inference results may be features of the sensing data, and the task schedule may include the order of memory loading and processing of the task processing unit.

[0036] This makes it possible to apply a high-performance NN inference model to limited computing environments such as IoT devices. In addition, because the processing time of the subsequent task is inferred based on the features of the sensing data, it becomes easier to infer the processing time according to the difficulty of task processing for the sensing data.

[0037] Also, a method executed by a computer includes acquiring sensing data, inputting the sensing data into an inference model, acquiring an inference result and information regarding processing times for multiple subsequent tasks of processing of the inference model, inputting the inference result to a task processing unit that processes the multiple subsequent tasks to process the multiple subsequent tasks, measuring an inference time including the time from inputting the inference result to the task processing unit to completion of processing of the multiple subsequent tasks, and training the inference model by machine learning using the sensing data as input data, information regarding the processing times for the multiple subsequent tasks as output data, and the measured inference time as reference data. Also, a program is a program for causing a computer to execute the above information processing method.

[0038] This allows an inference model to be generated that realizes the processing executed by the information processing device. By performing inference processing using the generated inference model, it becomes possible to determine a task schedule based on information about the processing times of multiple subsequent tasks so that the processing time of the task processing unit is within the required processing time.

[0039] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0040] Specific examples of an information processing device according to one aspect of the present disclosure will be described below with reference to the drawings. Each of the embodiments shown here illustrates a specific example of the present disclosure. Therefore, the numerical values, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components. Furthermore, the contents of each of the embodiments can be combined.

[0041] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales of the figures do not necessarily match. Furthermore, in each figure, substantially the same components are given the same reference numerals, and redundant explanations are omitted or simplified.

[0042] Furthermore, in this specification, terms indicating relationships between elements, such as "same" and "match," as well as numerical values ​​and numerical ranges, are not expressions that express only the strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about a few percent.

[0043] (Embodiment) Hereinafter, an information processing system according to this embodiment will be described with reference to FIGS.

[0044] [1. Information Processing System Configuration] First, the configuration of an information processing system 10 according to this embodiment will be described with reference to Figures 1 to 4. Figure 1 is a block diagram showing the functional configuration of the information processing system 10 according to this embodiment. Figure 2 is a block diagram showing the functional configuration of an inference processing unit 110 according to this embodiment.

[0045] As shown in Fig. 1, the information processing system 10 includes a device 100, a camera 200, and a training device 300. The device 100 performs inference processing on an input image from the camera 200 and provides a service according to the inference result. The training device 300 generates NN data 400 for the device 100 to perform the inference processing. Each component will be described below.

[0046] The device 100 is, for example, an IoT device, and has a function of providing multiple services based on an input image acquired from a camera 200. The device 100 has an inference processing unit 110 and a task execution unit 140. The input image acquired from the camera 200 is an example of sensing data.

[0047] The inference processing unit 110 performs inference processing on the input image input from the camera 200 using the NN data 400 generated by the training device 300. The inference processing unit 110 performs inference processing using multiple NNs arranged in parallel. In other words, the NN data 400 is configured to include multiple NNs arranged in parallel. The NN data 400 is data that stores, for example, the layer structure, weights, and biases of the NN.

[0048] 2, the NN data 400 includes a common NN 121 and a group of specialized NNs 124. The common NN 121 is an NN for pre-processing in the NN data 400, and the group of specialized NNs 124 is an NN for subsequent processing in the NN data 400.

[0049] The time required for inference processing in device 100 may vary depending on the input image. Therefore, depending on the input image, inference processing may take a long time, and it may not be possible to provide the service within the processing time required by device 100. Therefore, the inference processing unit 110 uses a pre-processing NN to obtain an inference result regarding the time for subsequent inference processing for the input image, and changes the order of subsequent inference processing (an example of a task schedule) based on the obtained inference result. The configuration of the inference processing unit 110, etc. will be described below.

[0050] The inference processing unit 110 has an NN inference unit 120, an NN inference calculation management unit 130, and an acquisition unit 150. The inference processing unit 110 is an inference processing device that can change the calculation order. The inference processing unit 110 can also be said to be an calculation order variable inference device. The inference processing unit 110 is also an example of an information processing device.

[0051] The NN inference unit 120 performs inference processing on the input image input from the camera 200 using a plurality of NNs based on the NN data 400 generated by the training device 300 .

[0052] The NN inference unit 120 performs inference processing using a common NN 121 and a group of specialized NNs 124 including a plurality of NNs arranged in parallel. The NN inference unit 120 first performs forward propagation calculation processing using the common NN 121. The common NN 121 is, for example, a forward propagation type NN.

[0053] The common NN 121 is an NN that outputs an inference processing result and delay flag information that are common to the multiple NNs in the specialized NN group 124. The common NN 121 executes an inference process that is a pre-processing common to each task. The common NN 121 can also be said to be a model executed as a pre-processing common to each task. The common NN 121 includes a feature classification unit 122 and a delay flag classification unit 123. The feature classification unit 122 and the delay flag classification unit 123 are, for example, trained NNs. The common NN 121 is an example of an inference processing unit. Furthermore, the inference process that is a pre-processing is not limited to processing using an NN.

[0054] The NN inference unit 120 receives an input image as input to the feature classification unit 122 of the common NN 121, and acquires an inference processing result that is the output of the feature classification unit 122. The inference processing result (inference result) is, for example, a feature amount (image feature amount), but is not limited to this. Below, an example in which the inference processing result is a feature amount will be described.

[0055] The feature classification unit 122 is a trained NN (NN for feature classification) that is trained to receive an input image as input and output feature quantities corresponding to the input image. The feature classification unit 122 performs inference processing common to each of the multiple NNs in the group of specialized NNs 124 that are subjected to forward propagation calculation processing. The feature quantities are information common to each of the multiple NNs in the group of specialized NNs 124.

[0056] The NN inference unit 120 outputs the feature values ​​output by the feature classification unit 122 to each of the multiple NNs in the specialized NN group 124. In this embodiment, the NN inference unit 120 also outputs the feature values ​​to the delay flag classification unit 123. In this manner, the feature values ​​output by the feature classification unit 122 may be used in the delay flag classification unit 123. In this embodiment, the same feature values ​​are input to the delay flag classification unit 123 and each of the multiple NNs in the specialized NN group 124.

[0057] The NN inference unit 120 inputs the feature amount from the feature classification unit 122 to the delay flag classification unit 123, thereby acquiring delay flag information, which is the output of the delay flag classification unit 123. The delay flag classification unit 123 is a trained NN (an NN for delay flag classification) that is trained to receive the feature amount as input and output delay flag information corresponding to the feature amount. The delay flag information indicates a time (inference time) including the time from when the feature amount (inference result) is input to the specialized NN group 124 (a task processing unit or a part thereof) to when the processing of multiple subsequent tasks is completed. For example, the delay flag information indicates an inference result indicating whether the inference processing time of the entire NN inference unit 120 exceeds a threshold. The delay flag information can also be considered information regarding a delay relative to a predetermined time (e.g., a threshold). The predetermined time is a time determined relative to the processing time of multiple subsequent tasks. The threshold may be set based on, for example, the processing time required for the device 100 or the service content of the device 100. Alternatively, the threshold may be set in advance. The delay flag information may be, for example, information indicating an inference result as to whether or not the processing time of the entire device 100 exceeds a threshold. The delay flag information may be, for example, information indicating an inference result as to whether or not the total processing time of the NN inference unit 120 and the task executing unit 140 exceeds a threshold.

[0058] The delay flag information may be, for example, information indicating "OFF" when the inference processing time is equal to or less than a threshold, and information indicating "ON" when the inference processing time exceeds the threshold. Furthermore, multiple "ON" settings may be used. The delay flag information may be, for example, information indicating "ON1" when the inference processing time exceeds a first threshold and is equal to or less than a second threshold that is greater than the first threshold, and information indicating "ON2" when the inference processing time exceeds the second threshold. The first threshold may be, for example, 10 msec, and the second threshold may be, for example, 15 msec, but are not limited thereto. The delay flag information is an example of information regarding the processing times of multiple subsequent tasks. The processing times of the multiple subsequent tasks may be, for example, the processing times of the specialized neural network 124, or the total processing time of the specialized neural network 124 and the task executing unit 140. In this case, the specialized neural network 124 and the task executing unit 140 are examples of a task processing unit.

[0059] In this embodiment, an example will be described in which the delay flag information is set to three levels: "OFF," "ON1," and "ON2." Note that the delay flag information may be set to two or four or more levels.

[0060] The delay flag information is not limited to information indicating the level of delay, such as "OFF," "ON1," and "ON2," and may be, for example, the processing time (computation time) itself.

[0061] It is desirable that the delay flag classification unit 123 quickly outputs the delay flag information. Therefore, it is preferable that the delay flag classification unit 123 is an NN with a smaller amount of calculation than the feature classification unit 122 and the specialized NN group .

[0062] The common NN 121 configured as described above is an example of a logical model that outputs feature quantities and delay flag information when an input image is received from the camera 200. The feature classification unit 122 is an example of a first inference model, and the delay flag classification unit 123 is an example of a second inference model. In other words, the inference model is configured to include the first inference model and the second inference model.

[0063] The NN inference calculation management unit 130 determines a method of forward propagation calculation (e.g., calculation order) for processing multiple tasks of the group of specialized NNs 124 subsequent to the common NN 121 based on the delay flag information from the delay flag classification unit 123, and notifies the group of specialized NNs 124 of the method of forward propagation calculation. Specifically, the NN inference calculation management unit 130 notifies the NN inference unit 120, which controls the processing of the group of specialized NNs 124, of the determined method of forward propagation calculation. This allows the NN inference unit 120 to input the feature quantities from the feature classification unit 122 to the group of specialized NNs 124 and cause the group of specialized NNs 124 to process multiple subsequent NNs using the determined method of forward propagation calculation. The NN inference calculation management unit 130 is an example of a determination unit that determines the method of forward propagation calculation and a control unit that causes subsequent tasks to be processed.

[0064] In this embodiment, the NN inference computation management unit 130 determines the method of the subsequent forward propagation computation of the common NN 121 based on the delay flag information and a table 500 shown in Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the table 500 that includes delay flag information according to this embodiment and the forward propagation computation methods of the specialized NNs 124 linked to the delay flag information. The parallel computation items in Fig. 3 indicate the methods of the subsequent forward propagation computation of the common NN 121, for example, the methods of the forward propagation computation of the specialized NNs 124.

[0065] As shown in FIG. 3, the delay flag information "OFF" is associated with the parallel calculation item "optimal parallel calculation processing" (see the first line 501 in FIG. 3), the delay flag information "ON1" is associated with the parallel calculation item "delay-handling parallel calculation processing" (see the second line 502 in FIG. 3), and the delay flag information "ON2" is associated with the parallel calculation item "delay-handling parallel calculation processing + CPU overclocking processing" (see the third line 503 in FIG. 3). Note that "ON" here means that the calculation time of the NN inference unit 120 exceeds the allowable time and that the calculation order needs to be changed from the calculation order indicated by the "optimal parallel calculation processing." Note that the forward transmission order is an example of a task schedule.

[0066] "Optimal parallel computation processing" refers to performing computation processing in accordance with a parallel computation method (for example, a parallel computation order) determined based on the number of memory accesses, processing time (computation time), power consumption, etc. For example, when the delay is less than a predetermined value (for example, the delay is less than a threshold value), the order of forward propagation computations (an example of a first schedule) is determined using the "optimal parallel computation processing." Note that the "optimal parallel computation processing" is an example of a first rule. A delay less than a predetermined value includes no delay, or a delay length or degree less than a predetermined value.

[0067] "Delay-tolerating parallel computing" refers to performing computing processing according to a parallel computing method determined to shorten the processing time compared to "optimal parallel computing." For example, when the delay is equal to or greater than a predetermined value (e.g., the delay is equal to or greater than a threshold value), the order of forward propagation computations (an example of a second schedule) is determined using "delay-tolerating parallel computing." The parallel computing method is, for example, but not limited to, a method in which the computation order is rearranged so that a process whose processing time varies depending on the number of detection candidate regions in the input image (e.g., NMS (Non-Maximum Suppression) process) is started with priority. Note that "delay-tolerating parallel computing" is an example of a second rule in which the processing time of multiple subsequent tasks is shorter than that of the first rule. Furthermore, "a delay equal to or greater than a predetermined value" includes the presence of a delay, or the length or degree of the delay being equal to or greater than a predetermined value.

[0068] NMS is an algorithm that deletes a detection region candidate if the degree of overlap (IoU value) between the candidate and a higher-scoring detection region candidate exceeds a threshold set by the training process.

[0069] "Delay countermeasure parallel computation processing + CPU overclocking processing" indicates that, in addition to "delay countermeasure parallel computation processing," the setting of the drive frequency of a CPU (for example, CPU 602 shown in FIG. 4, which will be described later) is increased. In this way, the delay flag information may include information related to the setting of the CPU drive frequency in addition to changing the operation order of the parallel computation. In other words, taking measures against delay is not limited to changing the parallel computation, but may also include changing the settings of each processing unit (for example, the drive frequency setting) provided in the device 100 in addition to or instead of changing the parallel computation.

[0070] The calculation order indicated by the "optimum parallel calculation process" and the calculation order indicated by the "delay handling parallel calculation process" are set in advance and stored in the memory 605 or the like.

[0071] In this way, there may be multiple types of delay flag information, such as the delay flags "ON1" and "ON2." For example, there may be multiple types of delay flag information depending on the measurement result of the processing time by the delay flag information measurement unit 302, which will be described later.

[0072] The specialized NNs 124 include a plurality of NNs that each produce a different output based on the feature values ​​from the feature classification unit 122. In this embodiment, the specialized NNs 124 include a first task NN 125 and a second task NN 126. Below, an example will be described in which the first task NN 125 is an NN for object detection and the second task NN 126 is an NN for skeleton detection, but this is not limiting. Also, an example will be described in which the first task NN 125 includes NMS processing and the second task NN 126 does not include NMS processing, but this is not limiting. The specialized NNs 124 (the NN inference unit 120 that performs processing using the specialized NNs 124) is an example of a task processing unit. When the task processing unit is configured to include the specialized NNs 124, each of the inference model and the task processing unit is an NN model.

[0073] The first task NN 125 is a trained NN that is trained to receive the feature values ​​from the feature classifier 122 and output an inference result for object detection according to the feature values. The inference result, which is the output of the first task NN 125, is output to the task execution unit 140 corresponding to the first task NN 125.

[0074] The second task NN 126 is a trained NN that is trained to receive the feature values ​​from the feature classification unit 122 as input and output an inference result for skeleton detection according to the feature values. The inference result, which is the output of the second task NN 126, is output to the task execution unit 140 corresponding to the second task NN 126.

[0075] Note that object detection and skeleton detection are examples of multiple subsequent tasks.

[0076] When the delay flag classification unit 123 outputs delay flag information including "ON1" or "ON2", the calculation order of the first task NN 125 and the second task NN 126 is changed from when the delay flag information was "OFF". In this embodiment, the calculation order in the first task NN 125, which includes at least the NMS processing, is changed from when the delay flag information was "OFF".

[0077] The NN inference calculation management unit 130 instructs the NN inference unit 120 to perform the forward propagation calculation of the specialized NN group 124 in accordance with the parallel calculation items (parallel calculation method) determined based on, for example, the delay flag information and the table 500. This allows the NN inference calculation management unit 130 to change the calculation order according to the inference result (delay flag information) regarding the time required for the inference processing on the input image, thereby preventing the processing time from becoming long.

[0078] The acquisition unit 150 acquires an input image from the camera 200. The acquisition unit 150 outputs the acquired input image to the NN inference unit 120.

[0079] Subsequently, the task execution unit 140 executes a preset task based on the inference result of the NN inference unit 120. The task execution unit 140 may be, for example, a display unit that displays the inference result, a light emitting unit that emits light according to the inference result, or a transmission unit that transmits the inference result to an external device.

[0080] 4 is a schematic diagram showing the configuration of a system-on-a-chip (SoC) 601 of the device 100 according to this embodiment. In this embodiment, an NPU is mounted.

[0081] 4, the SoC 601 includes a CPU 602, a first NPU 603, a second NPU 604, and a memory 605. Each function of the NN inference unit 120 is realized by implementing a trained NN (NN data 400) in the SoC 601.

[0082] The CPU 602 is a processing unit that executes various processes in the device 100. The CPU 602 executes NMS processing, processing of the task execution unit 140, and the like.

[0083] The first NPU 603 and the second NPU 604 are dedicated circuits for performing NN processing. The first NPU 603 and the second NPU 604 execute the NN processing in the device 100. The first NPU 603 and the second NPU 604 execute, for example, processing using the common NN 121 and processing using the specialized NN group 124. In this embodiment, the first NPU 603 and the second NPU 604 execute skeleton detection processing, object detection processing, and delay label determination processing.

[0084] The memory 605 stores programs executed by the CPU 602, the first NPU 603, and the second NPU 604. The memory 605 stores the NN data 400.

[0085] As described above, the inference processing unit 110 is configured to acquire an input image, and when the input image is input to the common NN 121, output feature values ​​and delay label information, determine a forward propagation operation method (optimal parallel operation processing, delayed target parallel operation processing, etc.) for processing the group of specialized NNs 124 based on the delay label information, input the feature values ​​to the group of specialized NNs 124, and have the group of specialized NNs 124 process the group of specialized NNs 124 using the determined forward propagation operation method. For example, the inference processing unit 110 includes an acquisition unit 150 that acquires an input image, the common NN 121 that inputs the input image to the common NN 121 and outputs feature values ​​and delay label information, an NN inference operation management unit 130 (an example of a determination unit) that determines a forward propagation operation method (optimal parallel operation processing, delayed target parallel operation processing, etc.) for processing the group of specialized NNs 124 based on the delay label information, and an NN inference operation management unit 130 (an example of a control unit) that inputs the feature values ​​to the group of specialized NNs 124 and has the group of specialized NNs 124 perform the determined forward propagation operation.

[0086] Next, we will explain the training device 300. The training device 300 generates NN data 400 used for inference processing by the NN inference unit 120 of the device 100. The training device 300 has a multitask training unit 301, a delay flag information measurement unit 302, a delay flag correct label generation unit 303, a delay flag training unit 304, a training label database 305 (training label DB305), and a scene image database 306 (scene image DB306).

[0087] The multitasking training unit 301 trains the feature classifying unit 122 and the specialized NN group 124 from the NN data 400. In this embodiment, the multitasking training unit 301 performs training for object detection and skeleton detection.

[0088] The multi-task training unit 301 performs training processing in which multi-task learning is applied to training of a neural network. As will be described in detail later, the multi-task training unit 301 receives as input a scene image acquired from a scene image DB 306, and uses as reference data (teacher data) correct labels in skeleton detection corresponding to the scene image and correct labels in object detection corresponding to the scene image acquired from a training label DB 305, and generates neural networks (NNs) (common NN 121 and specialized NN group 124) based on backpropagation (BP) or the like.

[0089] Multitask learning is a machine learning technique that solves multiple tasks with a single model. It is a technique that aims to improve the prediction accuracy of tasks by simultaneously training multiple related tasks to acquire "common factors" between the tasks. In the field of image recognition, multiple tasks such as object classification, object detection, and object region (segmentation) recognition are sometimes simultaneously trained. In this embodiment, the multiple tasks are skeleton detection and object detection.

[0090] Research has been conducted into the application of multi-task learning to training processing of NN data, and the multi-task training unit 301, for example, trains a NN used commonly for each task (e.g., a common NN 121) and a NN specialized for each task (e.g., a group of specialized NNs 124). In addition, using a commonly used NN has the advantage of reducing the overall computational complexity and memory usage of the inference processing required for processing each task.

[0091] The delay flag information measurement unit 302 and the delay flag correct label generation unit 303 perform processing for generating training data when the delay flag training unit 304 trains the delay flag classification unit 123 .

[0092] The delay flag information measurement unit 302 measures the processing time of the forward propagation operation of the common NN 121 and the specialized NN group 124 generated by the multitask training unit 301. In this embodiment, the delay flag information measurement unit 302 measures the processing time of at least the NN 125 for the first task. This is because the NN 125 for the first task includes NMS processing, and the processing time varies depending on the number of detection candidate regions in the input image. Furthermore, the NN 126 for the second task does not include processing such as NMS processing whose processing time varies depending on the number of detection candidate regions. In other words, the processing time does not vary significantly depending on the input image, so in this embodiment, the processing time is not measured.

[0093] The delay flag correct label generating unit 303 generates a delay flag correct label to be used for training the common NN 121 in the delay flag training unit 304, using information based on the measurement result of the processing time.

[0094] The delay flag training unit 304 performs a process of training the common NN 121 of the common NN 121 and the specialized NN group 124 generated by the multitask training unit 301 using the delay flag correct label generated by the delay flag correct label generation unit 303. The delay flag training unit 304 updates weight data and bias data in the delay flag classification unit 123 of the common NN 121 based on the backpropagation algorithm, using, for example, scene images as input data and delay flag correct labels as reference data (teaching data). In other words, the delay flag training unit 304 trains the delay flag classification unit 123 of the common NN 121 using the scene images and the delay flag correct labels. Note that the delay flag training unit 304 may train the feature classification unit 122 together with the delay flag classification unit 123, for example, using the delay flag correct labels.

[0095] The training label DB 305 stores training labels for generating the NN data 400. For example, the training label DB 305 stores, for one scene image, a training label for when skeleton detection is performed and a training label for when object detection is performed.

[0096] The scene image DB 306 stores scene images for generating the NN data 400 .

[0097] The training label DB 305 and the scene image DB 306 are realized by, for example, a semiconductor memory, but are not limited to this.

[0098] As described above, the training device 300 is configured to acquire a scene image, input the scene image to the common NN 121 to acquire features, input the features to the group of specialized NNs 124, cause the group of specialized NNs 124 to process, measure the time from when the scene image is input to the common NN 121 to when the processing by the group of specialized NNs 124 is completed, and train the common NN 121 by machine learning using the scene image as input data, information related to the processing time of the group of specialized NNs 124 as output data, and the measured time as reference data. For example, the training device 300 includes a delay flag information measurement unit 302 that acquires a scene image, inputs the scene image to a common NN 121 to acquire features, inputs the features to a group of specialized NNs 124, causes the group of specialized NNs 124 to process the features, and measures the time from when the scene image is input to the common NN 121 to when the processing by the group of specialized NNs 124 is completed, and a delay flag training unit 304 that trains the common NN 121 by machine learning using the scene image as input data, information related to the processing time of the group of specialized NNs 124 as output data, and the measured time as reference data.

[0099] [2. Operation of information processing system] Next, the operation of the information processing system 10 will be described with reference to FIGS.

[0100] [2-1. Equipment Operation] First, the processing in the device 100 will be described with reference to Figs. 5 to 6C. Fig. 5 is a flowchart showing the operation of the device 100 according to this embodiment. For convenience, Fig. 5 describes an example in which the delay flag information included in the table 500 is of two types, "ON (ON1)" and "OFF". In other words, it is assumed that the delay flag classification unit 123 is trained to output either "ON" or "OFF".

[0101] 5, the acquisition unit 150 of the device 100 acquires an input image from the camera 200 (S101). The acquisition unit 150 outputs the input image acquired in step S101 to the NN inference unit 120.

[0102] Next, when an input image is input from the acquisition unit 150, the NN inference unit 120 first executes a forward propagation calculation process using the common NN 121 (S102). Specifically, the NN inference unit 120 inputs the input image to the feature classification unit 122 of the common NN 121, thereby acquiring feature amounts that are output from the feature classification unit 122. The NN inference unit 120 then inputs the acquired feature amounts to the delay flag classification unit 123, thereby acquiring delay flag information that is output from the delay flag classification unit 123. The delay flag information is a result of inference made by the delay flag classification unit 123 from the feature amounts, and in this embodiment indicates the result of inference as to whether or not the inference processing time of the entire NN inference unit 120 has exceeded a threshold.

[0103] The NN inference unit 120 outputs the acquired delay flag information to the NN inference calculation management unit 130. The NN inference unit 120 may further output the delay flag information to a functional unit that is included in the device 100 and is external to the inference processing unit 110. The functional unit may be, for example, a functional unit that is responsible for a graphics function.

[0104] Next, the NN inference computation management unit 130 acquires delay flag information (S103). The NN inference computation management unit 130 determines whether the acquired delay flag information is “ON” (S104). If the delay flag information is “OFF,” that is, if the inference result indicates that the service can be completed within the processing time required by the device 100 even if the inference processing is performed according to the “optimal parallel computation processing” (No in S104), the NN inference computation management unit 130 determines the forward propagation computation method of the specialized NN group 124 to be “optimal parallel computation processing” based on the table 500 shown in FIG. 3 (S105). In other words, if the delay flag information is “OFF,” the NN inference computation management unit 130 determines to execute the processing of the specialized NN group 124 in an order based on the “optimal parallel computation processing.” The order based on the “optimal parallel computation processing” is an example of a first schedule. The NN inference computation management unit 130 outputs the determined forward propagation computation method of the specialized NN group 124 to the NN inference unit 120.

[0105] When the NN inference unit 120 acquires from the NN inference computation management unit 130 that the forward propagation computation method of the specialized NN group 124 is "optimal parallel computation processing," it executes the parallel forward propagation computation processing in the order indicated by the "optimal parallel computation processing." For example, the NN inference unit 120 executes the parallel forward propagation computation processing of the specialized NN group 124 in the most efficient order (S106). The NN inference unit 120 inputs the feature amounts output by the feature classification unit 122 to each NN in the specialized NN group 124, performs the inference processing according to a schedule based on the "optimal parallel computation processing," and acquires the inference results (skeleton detection results, object detection results, etc.) that are the output of each NN.

[0106] 6A is a schematic diagram showing an example of the processing time and operation units of the common NN 121 and the specialized NN group 124 according to this embodiment. Specifically, the first information 700a shown in FIG. 6A shows an example of the processing time and operation units when the processing is completed within the allowable time even when the processing is performed in the operation order based on the "optimal parallel operation processing." Note that FIGS. 6A to 6C will explain an example in which the NMS processing is outsourced to the CPU 602.

[0107] Area 701 shows the occupation rate of each unit (CPU 602, first NPU 603, and second NPU 604) in the common NN 121. In area 701, processing is performed by the first NPU 603 (NPU1 in FIGS. 6A to 6C) and the second NPU 604 (NPU2 in FIGS. 6A to 6C) using the common NN 121. Area 702 shows the occupation rate of each unit in the first task NN 125. In area 702, object detection processing is performed by at least one of the first NPU 603 and the second NPU 604 using the first task NN 125.

[0108] Area 703 indicates the occupation rate of each unit in the second task NN 126. In area 703, skeleton detection processing is performed by at least one of the first NPU 603 and the second NPU 604 using the second task NN 126. Areas 704a and 704b indicate the occupation rate of each unit in the NMS processing of the first task NN 125. In areas 704a and 704b, NMS processing is performed by the CPU 602.

[0109] 6A, if the result in step S104 is No, the NN inference computation management unit 130 instructs the NN inference unit 120 to perform the forward propagation computation of the specialized NN group 124 using "optimal parallel computation processing," which indicates that computation processing is performed using a parallel computation method that balances the number of memory accesses, computation time, etc. In this case, even if the processing is performed using the computation order based on the "optimal parallel computation processing," it is possible to complete the processing within the allowable time.

[0110] Referring again to FIG. 5, if the delay flag information is "ON," that is, if the inference result indicates that performing the inference processing according to the "optimal parallel computation processing" would not allow the service to be completed within the processing time required by the device 100 (Yes in S104), the NN inference computation management unit 130 determines the forward propagation computation method of the specialized NN group 124 to be "delay countermeasure parallel computation processing" based on the table 500 shown in FIG. 3 (S107). In other words, if the delay flag information is "ON," the NN inference computation management unit 130 determines to execute the processing of the specialized NN group 124 in an order based on the "delay countermeasure parallel computation processing." The order based on the "delay countermeasure parallel computation processing" is an example of a second schedule. The NN inference computation management unit 130 outputs the determined forward propagation computation method of the specialized NN group 124 to the NN inference unit 120.

[0111] When the NN inference unit 120 acquires from the NN inference computation management unit 130 that the forward propagation computation method of the specialized NN group 124 is "delay countermeasure parallel computation processing," it executes parallel forward propagation computation processing in the order indicated by the "delay countermeasure parallel computation processing." For example, the NN inference unit 120 executes parallel forward propagation computation for the specialized NN group 124 in a set order (S108). The NN inference unit 120 inputs the feature amounts output by the feature classification unit 122 to each NN in the specialized NN group 124, performs inference processing according to a schedule based on the "delay countermeasure parallel computation processing," and acquires inference results (skeleton detection results, object detection results, etc.) that are the output of each NN.

[0112] 6B is a schematic diagram showing an example of the processing time and operation units of the common NN 121 and the specialized NN group 124 according to this embodiment. Specifically, the second information 700b shown in FIG. 6B shows an example of the processing time and operation units when processing does not finish within the allowable time when processing is performed in the operation order based on the "optimal parallel operation processing." FIG. 6B shows a case where the allowable time is exceeded when performing the optimal parallel operation processing, i.e., a Yes determination is made in step S104.

[0113] As shown in Fig. 6B, for example, the time required for NMS processing (region 704b) is longer than the time required for NMS processing (region 704a) in Fig. 6A, causing the inference time of the inference processing unit 110 to exceed the allowable time. Depending on the input image, the time required for NMS processing may become longer as shown in Fig. 6B.

[0114] Therefore, as shown in step S107 of Fig. 5, the NN inference computation management unit 130 determines to perform the inference processing according to "delay countermeasure parallel computation processing" instead of "optimum parallel computation processing" because there is a possibility that the allowable time will be exceeded if the inference processing is performed according to "optimum parallel computation processing." In this embodiment, "delay countermeasure parallel computation processing" refers to rearranging the computation order so that the NMS processing (processing performed in area 704b) can be started preferentially. Note that, hereinafter, exceeding the allowable time will also be referred to as "a delay occurring."

[0115] 6C is a schematic diagram showing an example of the processing time and operation units after rearrangement of the operation order of the common NN 121 and the specialized NN group 124 according to this embodiment. The third information 700c shown in FIG. 6C shows an example of the processing time and operation units when processing is performed in an operation order based on "delay countermeasure parallel operation processing" when the processing does not finish within the allowable time in an operation order based on "optimal parallel operation processing." The third information 700c is obtained by changing the processing order from the second information 700b shown in FIG. 6B. FIG. 6C shows the processing order etc. after the change when step S104 is determined to be Yes.

[0116] As shown in FIG. 6C, the calculation order according to the "delay countermeasure parallel calculation processing" gives priority to starting NMS processing. This calculation order has the timing of starting NMS processing earlier than the calculation order according to the "optimal parallel calculation processing" so as to advance the timing of starting NMS processing. This makes it possible to prevent the inference time of the inference processing unit 110 from exceeding the allowable time even if the time required for NMS processing is long. This calculation order (processing order) is an example of a task schedule. The task schedule may also include the order of memory loading of the NN models of the specialized NN group 124.

[0117] 6B and 6C, in the "optimum parallel computation processing" and the "delay countermeasure parallel computation processing", the processing timing of the NMS processing is changed, but the processing content is not changed, and no processing is deleted. Therefore, the accuracy of the inference results when inference processing is performed by the "optimum parallel computation processing" and the "delay countermeasure parallel computation processing" is substantially the same.

[0118] In this embodiment, as shown in Figures 6B and 6C, an example is shown in which the layer whose processing time varies greatly depending on the input image is a layer that performs NMS processing, but it may also be a layer that uses another algorithm whose processing time varies greatly depending on the input image.

[0119] The NN inference unit 120 outputs the inference result output by the calculation in step S106 or S108 to the task execution unit 140.

[0120] 5 again, the task execution unit 140 operates based on the inference result acquired from the NN inference unit 120, and provides a service to the user. That is, various task processes are executed by the task execution unit 140 (S109).

[0121] As described above, the device 100 has the inference processing unit 110 that can determine the load of the inference processing early and change the calculation order to provide the inference service within the allowable time, and thus can operate while ensuring a certain level of service capacity. Furthermore, when the delay flag information is "OFF," the inference processing unit 110 can provide the inference service with optimal parallel calculations, and can also contribute to power saving.

[0122] Furthermore, the inference processing unit 110 estimates the processing time required for the subsequent NN inference processing in multiple NNs in the initial stage (preprocessing) from the feature amounts of the input image. Then, the inference processing unit 110 can perform the NN inference processing in parallel computation in an operation order that allows the processing to be completed within the required processing time.

[0123] In the above, an example has been described in which the calculation processes are executed in the order shown in FIG. 6C when the delay flag information is "ON." However, the present invention is not limited to this, and the order of the calculation processes may be changed depending on the delay flag information. For example, if the delay flag information includes information indicating the degree of delay (e.g., delay level: "large," "medium," or "small," or an estimated value of delay time), parallel calculation items may be set for each degree of delay. In other words, the order of the calculation processes may be set for each degree of delay. Then, in step S107, the NN inference calculation management unit 130 may determine the parallel calculation item according to the degree of delay included in the delay flag information as the method of the forward propagation calculation of the specialized NN group 124.

[0124] [2-2. Operation of the training device] Next, the operation of the training device 300 will be described with reference to Figs. 7 to 13. First, the operation of the multitasking training unit 301 will be described with reference to Figs. 7 and 8. Fig. 7 is a flowchart showing the operation of the multitasking training unit 301 according to this embodiment. Fig. 8 is a diagram schematically showing the operation of the multitasking training unit 301 according to this embodiment.

[0125] 7, the multitasking training unit 301 acquires a scene image from the scene image DB 306 and acquires the correct label of each NN from the training label DB 305 (S201). The multitasking training unit 301 acquires a scene image, a correct label for the first task NN 125 for the scene image (first task NN correct label), and a correct label for the second task NN 126 for the scene image (second task NN correct label). In this embodiment, the multitasking training unit 301 acquires the correct label for the object detection NN and the correct label for the skeleton detection NN.

[0126] Next, the multitasking training unit 301 performs a forward propagation operation using the common NN 121 and the group of specialized NNs 124. For example, the multitasking training unit 301 performs a forward propagation operation using the feature classification unit 122 of the common NN 121 and the group of specialized NNs 124 (S202).

[0127] As shown in FIG. 8, the multitasking training unit 301 acquires features that are the output of the common NN 121 by inputting a scene image into the common NN 121, and further acquires a first inference result that is the output of the NN 125 for the first task by inputting the acquired features into the NN 125 for the first task, and a second inference result that is the output of the NN 126 for the second task by inputting the acquired features into the NN 126 for the second task.

[0128] 7 again, next, the multitasking training unit 301 uses the correct labels of each NN as reference data (teaching data) to perform backpropagation calculations on the feature classifying unit 122 and the specialized NN group 124 to update the weights and biases (S203). Based on the backpropagation method, the multitasking training unit 301 regards the output value of a loss function as an error and performs a backpropagation calculation to update the weights (weight data) and biases (bias data) in the feature classifying unit 122 of the common NN 121 and the specialized NN group 124.

[0129] 8, the loss function for the first task NN 125 represents the degree of error between the first inference result of the first task NN 125 and the correct label of the NN for the first task. The loss function for the second task NN 126 represents the degree of error between the second inference result of the second task NN 126 and the correct label of the NN for the second task.

[0130] The multitasking training unit 301 acquires a first inference result, which is the output of the first task NN 125, based on the scene image, and acquires an output value of a loss function based on the first inference result and the correct label of the first task NN. The multitasking training unit 301 then regards the output value as an error and performs a backpropagation operation to update the weights and biases in the feature classifier 122 of the common NN 121 and the first task NN 125 of the specialized NN group 124.

[0131] Next, the multitasking training unit 301 acquires a second inference result, which is the output of the second task NN 126, based on the scene image, and acquires an output value of a loss function based on the second inference result and the correct label of the second task NN. Then, the multitasking training unit 301 regards the output value as an error and performs a backpropagation operation to update the weights and biases of the feature classifier 122 of the common NN 121 and the second task NN 126 of the specialized NN group 124. In this way, the multitasking training unit 301 alternately performs the update process, for example.

[0132] 7 again, the multitasking training unit 301 determines whether the accuracy rate of the NN inference result (or the error rate compared with the correct label of the scene image) satisfies a requirement (S204). The requirement may be, for example, the accuracy rate required for the device 100. In other words, the multitasking training unit 301 may make the determination in step S204 based on whether the accuracy rate of the NN inference result is equal to or greater than the accuracy rate required for the device 100.

[0133] If the accuracy rate of the NN inference result satisfies the requirement (Yes in S204), the multitasking training unit 301 ends the process. On the other hand, if the accuracy rate of the NN inference result does not satisfy the requirement (No in S204), the multitasking training unit 301 returns to step S201 and continues the process. In other words, if the accuracy rate of the NN inference result does not satisfy the requirement, the multitasking training unit 301 continues training using various scene images and the correct labels associated with them, and continues the training process until the accuracy rate of the NN inference result satisfies the requirement.

[0134] Next, the operations of the delay flag information measurement unit 302 and the delay flag correct label generation unit 303 will be described with reference to Fig. 9 and Fig. 10. Fig. 7 is a flowchart showing the operations of the delay flag information measurement unit 302 and the delay flag correct label generation unit 303 according to this embodiment. Fig. 10 is a diagram schematically showing the operations of the delay flag information measurement unit 302 and the delay flag correct label generation unit 303 according to this embodiment.

[0135] 9, the delay flag information measurement unit 302 acquires an arbitrary scene image (S301). The delay flag information measurement unit 302 may acquire, for example, a scene image used in multitask learning by the multitask training unit 301.

[0136] Next, the delay flag information measurement unit 302 measures the calculation time of the forward propagation calculation of the common NN 121 and the group of specialized NNs 124 trained by the multitask training unit 301 (S302). In this embodiment, the delay flag information measurement unit 302 measures the calculation time of the first task NN 125 including the NMS process.

[0137] In step S302, the delay flag information measurement unit 302 inputs a scene image to the common NN 121 to acquire feature results and delay flag information, and further inputs the feature to the first task NN 125 to process the first task NN 125, measuring the time from when the scene image is input to the common NN 121 to when the processing of the specialized NNs 124 is completed. Note that in step S302, the delay flag information measurement unit 302 only needs to measure at least the time from when the feature is input to the first task NN 125 to when the processing of the specialized NNs 124 is completed. FIG. 10 illustrates an example in which the delay flag information measurement unit 302 measures the time (an example of inference time) from when the feature is input to the first task NN 125 to when the processing of the specialized NNs 124 is completed.

[0138] The processing time may be measured using the same hardware device as the device 100. The processing time may also be calculated by simulation.

[0139] 10, the delay flag information measurement unit 302 acquires information about the feature obtained by inputting a scene image to the common NN 121 and the processing times of multiple subsequent tasks of the processing of the common NN 121 (delay flag information that is the output of the delay flag classification unit 123), and measures the processing time of the first task NN 125 when the feature is input. Measuring the processing time is an example of measuring delay flag information.

[0140] 9 again, the delay flag information measurement unit 302 determines whether the measured processing time exceeds a threshold value (S303). The threshold value is, for example, set in advance. If the processing time exceeds the threshold value (Yes in S303), the delay flag information measurement unit 302 sets the delay flag to ON and stores the scene image in the delay flag information DB 302a (see FIG. 10) (S304). If the processing time does not exceed the threshold value (No in S303), the delay flag information measurement unit 302 sets the delay flag to OFF and stores the scene image in the delay flag information DB 302a (S305).

[0141] When the processing time is measured in each of the first task NN 125 and the second task NN 126, it is possible that (i) a delay occurs only in the first task NN 125 (exceeding the allowable time), or (ii) a delay occurs only in the second task NN 126. In this case, the contents of the "parallel computation item" (for example, the computation order after rearrangement) may differ between (i) and (ii). In this way, when the contents of the "parallel computation item" differ, there may be multiple types of delay flags "ON" for each content of the "parallel computation item."

[0142] Next, the delay flag correct label generation unit 303 generates a delay flag correct label based on the delay flag information in the delay flag information DB 302a. The delay flag correct label generation unit 303 generates a delay flag correct label by labeling the scene image with a delay flag "ON" or "OFF" (S306). It can also be said that the delay flag correct label generation unit 303 generates a delay flag correct label by associating the delay flag information with the scene image. The delay flag correct label generation unit 303 stores the generated delay flag correct label in the delay flag correct label DB 303a (see FIG. 10).

[0143] Next, the operation of the delay flag training unit 304 will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a flowchart showing the operation of the delay flag training unit 304 according to this embodiment. Fig. 12 is a diagram schematically showing the operation of the delay flag training unit 304 according to this embodiment.

[0144] As shown in FIG. 11, the delay flag training unit 304 acquires a scene image from the scene image DB 306, and acquires a delay flag correct label corresponding to the scene image from the delay flag correct label DB 303a (S401).

[0145] Next, the delay flag training unit 304 performs a forward propagation operation using the common NN 121 (S402). The delay flag training unit 304 performs a forward propagation operation using, for example, the delay flag classification unit 123 of the common NN 121.

[0146] As shown in Figure 12, the delay flag training unit 304 inputs a scene image into the feature classification unit 122 of the common NN 121, acquires feature values ​​that are the output of the feature classification unit 122, and further inputs the acquired feature values ​​into the delay flag classification unit 123 to acquire delay flag information that is the output of the delay flag classification unit 123.

[0147] 11 again, the delay flag training unit 304 then performs backpropagation calculations on the common NN 121 using the delay flag correct labels as reference data (teaching data) to update the weights and biases (S403). Based on the error backpropagation method, the delay flag training unit 304 regards the output value of the loss function as an error and performs backpropagation calculations to update the weights (weight data) and biases (bias data) in the delay flag classification unit 123 of the common NN 121.

[0148] As shown in FIG. 12, the loss function in the common NN 121 indicates how much error there is between the inference result (delay flag information) of the delay flag classification unit 123 of the common NN 121 and the delay flag correct label corresponding to the scene image.

[0149] The delay flag training unit 304 acquires delay flag information, which is the output of the delay flag classification unit 123, based on the scene image, and acquires the output value of the loss function based on the delay flag information and the delay flag correct label. Then, the delay flag training unit 304 regards the output value as an error and performs a backpropagation operation to update the weights and biases in the delay flag classification unit 123 of the common NN 121.

[0150] In this way, in steps S402 and S403, the common NN 121 (e.g., the delay flag classification unit 123) is trained by machine learning using the scene image as input data, the delay flag information output from the delay flag classification unit 123 of the common NN 121 as output data, and the time measured by the delay flag information measurement unit 302 as reference data.

[0151] The delay flag training unit 304 may regard the output value as an error and perform a backpropagation operation to update the weights and biases in the feature classification unit 122 and the delay flag classification unit 123. That is, the delay flag training unit 304 may target the feature classification unit 122 in addition to the delay flag classification unit 123 for training. This updates the weights and biases in the feature classification unit 122 as well as the delay flag classification unit 123, and is expected to improve the accuracy of the delay flag information output by the delay flag classification unit 123.

[0152] 11 again, the delay flag training unit 304 determines whether the accuracy rate of the NN inference result (or the error rate comparing the inference result with the delay flag correct label) satisfies a requirement (S404). The requirement may be, for example, the accuracy rate required for the device 100. In other words, the delay flag training unit 304 may make the determination in step S404 based on whether the accuracy rate of the NN inference result is equal to or greater than the accuracy rate required for the device 100.

[0153] If the accuracy rate of the NN inference result satisfies the requirement (Yes in S404), the delay flag training unit 304 ends the process. On the other hand, if the accuracy rate of the NN inference result does not satisfy the requirement (No in S404), the delay flag training unit 304 returns to step S401 and continues the process. In other words, if the accuracy rate of the NN inference result does not satisfy the requirement, the delay flag training unit 304 continues training using various scene images and the correct labels associated with them, and continues the training process until the accuracy rate of the NN inference result satisfies the requirement.

[0154] After the delay flag training unit 304 completes the training process, the NN data 400 becomes capable of executing the above-described operation of the device 100. The generated NN data 400 is transmitted to the device 100.

[0155] Although the training device 300 according to this embodiment performs training processing sequentially, it may also perform multi-task learning on the feature classifier 122 and specialized NNs 124 of the common NN 121 and the delay flag classifier 123 of the common NN 121, and at that time, measure the forward propagation operation of the specialized NNs 124 to acquire delay flag information. Furthermore, during the backpropagation operation, the training device 300 may update the weights and biases of the delay flag classifier 123 of the common NN 121 using error information that sets the delay flag information (delay flag correct label) as the correct answer, and update the weights and biases of the feature classifier 122 of the common NN 121 and the specialized NNs 124 using the error information from the multi-task learning. In this way, the training device 300 may be configured to perform multi-task learning on the feature classifier 122 and specialized NNs 124 of the common NN 121 and the delay flag classifier 123 of the common NN 121.

[0156] Here, the output of the evaluation of the training result of the training device 300 according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a table showing the occurrence rate of delay flag information according to this embodiment.

[0157] As an example of an output of the training result evaluation of the training device 300 according to this embodiment, in addition to inference quality information such as the mAP (Mean Average Precision) of the specialized NN group 124, a configuration is also possible in which the "occurrence rate of delay flag information" shown in FIG. 13 is output. The results shown in FIG. 13 can also motivate developers to change the NN configuration or hyperparameters to increase the incidence rate of "OFF" delay flag information. Furthermore, by displaying the incidence rate of inference delays that do not satisfy the required specifications (the incidence rate corresponding to the "NG" delay flag information in FIG. 13) or information on the corresponding scene images, the training device 300 can encourage developers to consider measures to prevent inference delays from occurring during the NN training process.

[0158] (Other embodiments) Although the present disclosure has been described above based on the embodiments, the present disclosure is not limited to the above-described embodiments.

[0159] For example, in the above embodiment, an example has been described in which feature quantities output by the feature classification unit 122 are input to the delay flag classification unit 123 as input data, but the input data is not limited to this. The input data may be, for example, an input image from the camera 200. For example, the NN inference unit 120 may input the input image from the camera 200 to the delay flag classification unit 123 to acquire delay flag information output by the delay flag classification unit 123. In this case, in step S402 of FIG. 11, the delay flag training unit 304 acquires delay flags output by the delay flag classification unit 123 by inputting a scene image to the delay flag classification unit 123 of the common NN 121. Then, in step S403 of FIG. 11, the delay flag training unit 304 performs backpropagation calculations on the delay flag classification unit 123 of the common NN 121 using the delay flag correct label as reference data (teaching data) to update the weights and biases of the delay flag classification unit 123. Furthermore, for example, the feature classification unit 122 and the delay flag classification unit 123 may receive the same input data (input image).

[0160] Furthermore, the input data of the delay flag classification unit 123 may be, for example, intermediate data of the inference by the feature classification unit 122. For example, the NN inference unit 120 may acquire delay flag information that is the output of the delay flag classification unit 123 by causing the feature classification unit 122 to output intermediate data of the inference and inputting the intermediate data to the delay flag classification unit 123.

[0161] In the above embodiment, the processing of the subsequent task is a process using an NN, and the NN inference calculation management unit 130 determines the calculation order of the process using an NN. However, the present invention is not limited to this. The processing of the subsequent task may be a process that does not use an NN, and the NN inference calculation management unit 130 may determine the calculation order of the process that does not use an NN.

[0162] In the above embodiment, an example has been described in which the SoC 601 of the device 100 is equipped with an NPU, but the device 100 does not necessarily have to be equipped with an NPU. The SoC 601 may be equipped with only the CPU 602 of the CPU 602 and the NPU, and the processes described above may be executed by the CPU 602, for example.

[0163] In the above embodiment, the NN inference processing unit 110 is used in the product field of IoT devices, but the product field is not limited to this. The NN inference processing unit 110 can also be applied to product fields other than IoT devices, such as self-driving cars, robots, and unmanned aerial vehicles such as drones.

[0164] The order of the processes described in the above embodiment is merely an example. The order of the processes may be changed, the processes may be executed in parallel, or some of the processes may not be executed.

[0165] Furthermore, each component described in the embodiments may be implemented as software or, typically, as an LSI, an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. While LSI is used here, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI depending on the level of integration. Furthermore, the integration method is not limited to LSI; dedicated circuits or general-purpose processors may also be used. Field-programmable gate arrays (FPGAs), which can be programmed after LSI fabrication, or reconfigurable processors, which allow the reconfiguration of the connections and settings of circuit cells within an LSI, may also be used. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, that technology may naturally be used to integrate the components.

[0166] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

[0167] Furthermore, the training device included in the information processing system may be realized as a single device or may be realized by multiple devices. For example, each processing unit of the training device may be realized by two or more server devices. When the information processing system is realized by multiple server devices, the components included in the information processing system may be distributed among the multiple server devices in any manner. Furthermore, the communication method between the multiple server devices is not particularly limited.

[0168] Furthermore, the technology of the present disclosure may be a program for causing a computer to execute the characteristic processes of the above-described information processing method, or may be a non-transitory computer-readable recording medium on which the program is recorded. It goes without saying that the program can be distributed via a transmission medium such as the Internet. For example, the program and a digital signal comprising the program may be transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc. The program and a digital signal comprising the program may be recorded on a recording medium and transferred, or transferred via a network, etc., and executed by another independent computer system.

[0169] In each embodiment, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for that component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. [Industrial Applicability]

[0170] The present disclosure is applicable to information processing devices and the like that use inference processing of multiple NNs. [Explanation of symbols]

[0171] 10 Information Processing Systems 100 equipment 110 Inference processing unit 120 NN inference section 121 Common NN (inference processing unit) 122 Feature Classification Unit 123 Delay Flag Classification Unit 124 Specialized NN group 125 NN for the first task 126 NN for the second task 130 NN inference calculation management unit (decision unit, control unit) 140 Task Execution Unit 150 Acquisition Department 200 cameras 300 training equipment 301 Multitasking Training Department 302 Delay Flag Information Measurement Unit 302a Delay flag information DB 303 Delay Flag Correct Label Generation Unit 303a Delay Flag Correct Label DB 304 Delay Flag Training Department 305 Training Label DB 306 Scene Image DB 400 NN data 500 tables 501 Line 1 502 2nd line 503 Line 3 601 SoC 602 CPU 603 1st NPU 604 2nd NPU 605 memory 700a First Information 700b 2nd information 700c 3rd information 701, 702, 703, 704a, 704b area

Claims

1. an acquisition unit that acquires sensing data; a task processing unit including a plurality of neural networks; a feature classification unit that outputs a feature amount of the sensing data to each of the plurality of neural networks included in the task processing unit, Each of the plurality of neural networks included in the task processing unit executes subsequent processing in parallel and produces a different output based on the feature amount. Information processing device.

2. Each of the plurality of neural networks executes the subsequent process in parallel based on a first schedule determined in accordance with a parallel operation method determined based on at least one of the number of memory accesses, the processing time, and the power consumption. The information processing device according to claim 1 .

3. When a delay with respect to a predetermined time determined for the processing times of the plurality of subsequent processes is equal to or greater than a predetermined time, the subsequent processes are executed in parallel based on a second schedule determined so that the processing times of the plurality of subsequent processes are shorter than those of the first schedule. The information processing device according to claim 2 .

4. When a delay with respect to a predetermined time set for the processing times of a plurality of subsequent processes is equal to or greater than a predetermined time, the subsequent processes are executed in parallel so as to start preferentially a process whose processing time varies depending on the number of detection candidate areas of the sensing data. The information processing device according to claim 3 .

5. The information processing device includes a CPU (Central Processing Unit) that executes a plurality of subsequent processes; If the delay with respect to a predetermined time set for the processing times of the plurality of subsequent processes is equal to or greater than a predetermined time, the setting of the drive frequency of the CPU is increased. The information processing device according to claim 4 .

6. The plurality of neural networks include a first neural network that outputs an inference result of object detection according to the feature amount, and a second neural network that outputs an inference result of skeleton detection according to the feature amount. The information processing device according to claim 1 .

7. The feature classification unit outputs the same feature to each of the plurality of neural networks. The information processing device according to claim 1 .

8. 1. A computer-implemented method comprising: Acquire sensing data, outputting the feature quantities of the sensing data to each of a plurality of neural networks; Each of the plurality of neural networks executes subsequent processing in parallel and produces a different output based on the feature amount. Information processing methods.

9. A program for causing a computer to execute the information processing method according to claim 8.

Images