Data processing apparatus, method therefor, and program

Abstract

To reduce a program size or a circuit scale in a data processing apparatus which executes various post-processing on a product sum operation result.SOLUTION: A data processing apparatus comprises: product sum operation means which performs a product sum operation on data on the basis of a given product sum operation parameter; operation result holding means which holds an operation result of the product sum operation means; command processing means which, on the basis of a given command, performs processing defined in the command on the product sum operation result held in the operation result holding means; and distribution means which receives a parameter string in which the product sum operation parameter and the command are combined, and gives the product sum operation parameter to the product sum operation means and gives the command to the command processing means on the basis of contents and order of the combination.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a data processing device, method, and program for performing processing on a plurality of types of neural networks. [Background technology]

[0002] There is a demand for hardware implementation technology that can install neural networks (hereafter referred to as NN) on devices with less resource consumption. NNs are known as a method used in deep learning, and are widely used in recognition processing. For example, Convolutional Neural Networks (hereafter referred to as CNN), which is a type of NN, demonstrate excellent performance, mainly in tasks such as image recognition. Most NNs consist of product-sum operations, and are easily able to benefit from parallel processing using dedicated hardware. In addition, various recognition processes can be handled by simply changing the weight coefficients, i.e., the input parameters. The advantage of NNs is that they can perform in a variety of recognition processes on the same hardware.

[0003] For example, one recognition process determines where a desired object is located in an input image, that is, the process receives an input image as input and returns the position coordinates of the object in the input image as output.

[0004] In the recognition process described above, an input image is first fed into a NN to extract feature data. The feature data is, for example, the same two-dimensional data as the input image, and the NN is given weighting coefficients in advance so that it outputs feature data according to the purpose of the recognition process. For example, the NN in the recognition process described above outputs feature data with large values ​​near the position of the subject and small values ​​elsewhere. Alternatively, when the NN can handle multiple types of subjects, it outputs three-dimensional data made up of multiple sheets of the above-mentioned two-dimensional data stacked together as feature data. Each piece of two-dimensional data that makes up this three-dimensional feature data is called a channel.

[0005] In the above-mentioned recognition process, the position coordinates of the subject are then calculated from the feature data. Since the feature data has large values ​​at the same positions as the subject in the input image, it is possible to estimate the position coordinates of the subject by, for example, finding the position where the value is maximum or a local maximum in the two-dimensional feature data. If the feature data is three-dimensional, the channel corresponding to the subject whose position is to be found is referenced, and the same process as for two-dimensional data is carried out.

[0006] As described above, the recognition process can be generally divided into a front-stage part where the NN calculation is performed, and a back-stage part where the final recognition process result is obtained using the NN calculation result. In the following, the front-stage part is called the product-sum calculation part, and the back-stage part is called the post-processing part.

[0007] On the other hand, the increase in program size and circuit scale due to the diversification of recognition processing using NNs is a problem. For example, in recognition processing using NNs, the optimal number of NN layers and the number of dimensions per layer change depending on the purpose and recognition target. Therefore, the more diverse the recognition processing the hardware can handle, the more complex the control sequence becomes, leading to an increase in program size and circuit scale.

[0008] In Patent Document 1, the processing sequence of the hardware that performs the NN operation is given as an external input. In the data processing device in Patent Document 1, a concatenation of multiple pieces of control data is given as an external input. Each piece of control data corresponds to one processing unit into which the NN operation is divided, and the order of the control data indicates the order of processing. This makes it possible to execute new recognition processing by changing the control data without adding a new sequence to the data processing device.

[0009] In Patent Document 2, the processing sequence of the hardware that performs the NN operation is given as an external input. In the data processing device according to Patent Document 2, various arithmetic processes that constitute the NN operation are executed based on instructions. The desired NN operation is realized by combining multiple instructions, and the order in which the instructions are given represents the order of processing. This makes it possible to execute new recognition processing by changing the instruction sequence without adding a new sequence to the data processing device.

[0010] In Patent Document 3, software is generated according to the type of recognition processing. In the NN model generation device in Patent Document 3, by providing hardware information and algorithm information, software having a sequence required to perform a desired NN calculation is generated. In other words, different software is used for each type of recognition processing. This limits the sequences to be installed in one piece of software, preventing an increase in program size. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] JP 2020-140546 A [Patent Document 2] U.S. Patent No. 9836691 [Patent Document 3] JP 2022-66974 A Summary of the Invention [Problem to be solved by the invention]

[0012] On the other hand, in the method of providing the sequence of the NN calculation part as an external input as in Patent Document 1 and Patent Document 2, it is not possible to suppress the increase in program size or circuit scale due to the diversification of the post-processing part of the recognition process. In the method of providing the sequence of only the product-sum calculation part as an external input, it is necessary to realize the post-processing part with separate software or dedicated hardware. Therefore, it is not possible to suppress the increase in program size or circuit scale due to them.

[0013] In addition, the method of generating software for each type of recognition processing as in Patent Document 3 reduces the program size for each type of recognition processing, but incurs overhead such as the processing time required to rewrite the program each time the recognition processing is switched. Therefore, multiple types of recognition processing can be supported by one piece of hardware only under limited conditions, such as when real-time performance is not required and there is ample memory and bus bandwidth. [Means for solving the problem]

[0014] In order to solve the above problems, according to one aspect of the present invention, a data processing device comprises a product-sum operation means for performing a product-sum operation on data based on given product-sum operation parameters, an operation result holding means for holding an operation result of the product-sum operation means, a command processing means for performing a process defined in a given command on the product-sum operation result held in the operation result holding means, and a distribution means for receiving a parameter string combining the product-sum operation parameters and the command, and providing the product-sum operation parameters to the product-sum operation means and providing the command to the command processing means based on the content and order of the combination. Effect of the Invention

[0015] According to the present invention, in a data processing device that processes a plurality of types of neural networks, it becomes possible to perform more types of processing while suppressing an increase in program size or circuit scale. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a recognition processing unit in the first and second embodiments; [Diagram 2] FIG. 1 is a diagram showing a configuration of a recognition process to be executed in a first embodiment; [Diagram 3] FIG. 13 is a flowchart showing an example of a process for executing different post-processing steps for each recognition process; [Figure 4] FIG. 1 is a diagram showing an example of the configuration of a data processing device according to a first and second embodiment; [Diagram 5] FIG. 2 is a flowchart showing the process of the command processing unit 102 according to the first embodiment; [Figure 6] FIG. 6 is a diagram showing the difference in program size between the flowcharts shown in FIG. 3 and FIG. 5; [Figure 7] FIG. 1 is a diagram showing the configuration of parameters that are input data in the first embodiment; [Figure 8] FIG. 1 is a diagram showing a memory map in a recognition processing unit in a first embodiment; [Figure 9] FIG. 1 is a diagram showing an operation sequence in a recognition processing unit in the first embodiment; [Figure 10] FIG. 13 is a diagram showing the configuration of a recognition process to be executed in the second embodiment; [Figure 11] FIG. 11 is a flowchart showing a command processing unit according to a second embodiment; [Figure 12] FIG. 11 is a diagram showing the configuration of parameters that are input data in the second embodiment; [Figure 13] FIG. 11 is a diagram showing a memory map in a recognition processing unit in a second embodiment; [Figure 14] FIG. 11 is a diagram showing an operation sequence in a recognition processing unit in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. Note that the configurations described in the following embodiments are representative examples, and the scope of the present invention is not necessarily limited to those specific configurations.

[0018] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. In each embodiment, the entire sequence of the recognition process, including the product-sum operation part and the post-processing part, is received as a parameter string (hereinafter, simply referred to as parameters) in which the parameters of each process are arranged, and processing is performed according to the contents of the parameter string. This allows for diversification of the recognition process sequence while suppressing an increase in program size or circuit scale.

[0019] In the data processing device described below, a combination of parameters for controlling the multiply-and-accumulate part and parameters for controlling the post-processing part is given as an input, and the order of the parameters is the processing order in the sequence of the multiply-and-accumulate part and the post-processing part. Each embodiment shows what parameters are given to realize the target recognition processing, and what processing the data processing device performs on those parameters.

[0020] In the following embodiment, for simplicity, the part corresponding to the flowchart in Fig. 2 is implemented as software, and the purpose of the data processing device according to this embodiment is to prevent an increase in program size. If the part is implemented as hardware, it is also possible to prevent an increase in circuit size.

[0021] <Embodiment 1> In the first embodiment, the data processing device realizes the post-processing part of the target recognition process by giving appropriate parameters. In addition, the definition of a sequence by parameters and the configuration of a data processing device and method required to realize it are clarified. As a result, the data processing device according to this embodiment realizes preventing an increase in program size due to diversification of recognition processes.

[0022] <Post-processing and sequence diversification> First, as a premise for explaining the data processing device in this embodiment, an example of the recognition process and the product-sum operation part and post-processing part included therein will be given. Based on that, how to implement the sequence of the post-processing part and what problems will arise in terms of program size will be described.

[0023] Fig. 2 shows the configuration of the recognition process to be executed in the first embodiment. The recognition process shown in Fig. 2 consists of one CNN and two post-processing processes that use its output. This recognition process receives an input image 200 and post-processing settings 203, and returns a first processing result 206 and a second processing result 207.

[0024] The first CNN 201 receives an input image 200 and performs a product-sum operation. As a result of the product-sum operation, it returns first feature data 202. The first feature data 202 is a score map that indicates the score of a position corresponding to the input image 200, and is made up of multiple score maps. Hereinafter, each score map is referred to as a channel.

[0025] The first CNN 201 is trained in advance to output a high score in an area where a specific object appears in the input image 200. For example, one channel may output a high score in an area where a person's head appears, and another channel may output a high score in an area where a person's torso appears.

[0026] The first position estimation process 204 refers to a specific channel of the first feature data 202, and determines the coordinate values ​​in the input image 200 for a position with a higher score than the surroundings. The determined coordinate values ​​are returned as the first process result 206.

[0027] The second position estimation process 205 refers to another channel of the first feature data 202, performs the same process as the first position estimation process 204, and returns a second processing result 207. The first position estimation process 204 and the second position estimation process 205 refer to different channels according to the post-processing settings 203. For example, the first position estimation process 204 refers to a channel corresponding to a person's head, while the second position estimation process 205 refers to a channel corresponding to a person's torso.

[0028] As described above, in the recognition process shown in Fig. 2, a specific function is realized by combining CNN and post-processing. For example, the positions of the head and torso of a person appearing in an input image 200 are obtained and output as a first processing result 206 and a second processing result 207, respectively.

[0029] In recognition processing including NN, the sequence of the post-processing portion changes significantly depending on the purpose and target of the recognition processing. In the recognition processing shown in Fig. 2, the first CNN 201 is the product-sum operation portion, and each processing subsequent to that corresponds to the post-processing portion. For example, if it is sufficient to obtain the position of only the person's head, the second position estimation processing 205 is excluded from the recognition processing shown in Fig. 2, and the output is also only the first processing result 206. Alternatively, for example, if it is desired to obtain not only the position but also the size of the person's head, the second position estimation processing 205 is replaced by another processing, and the size of the person's head is output from the second processing result 207.

[0030] Fig. 3 shows an example of a conventional flowchart for executing different post-processing for each recognition process. When post-processing is started, in step S301, the process branches depending on the type of recognition process. Here, it is assumed that the process proceeds to branch 0, taking as an example the case of executing the post-processing portion of the recognition process shown in Fig. 2. In step S302, input data required for post-processing is input from outside. Here, post-processing settings 203 required for executing the first position estimation process 204 and the second position estimation process 205 are input.

[0031] In step S303, a position estimation process is executed. This corresponds to the first position estimation process 204 in FIG. 2. In step S304, a position estimation process is executed. This corresponds to the second position estimation process 205 in FIG. 2. In step S305, the result of the post-processing is output to the outside. Here, the processing results of steps S303 and S304, i.e., the first processing result 206 and the second processing result 207 in FIG. 2, are output.

[0032] When the process of the flowchart shown in Fig. 3 is implemented as software, the total number of steps increases as the number of types of recognition processes increases. For example, when adding a recognition process that has only one position estimation process, steps S307-309 leading to branch 1 are added. In this case, even if each step is the same as that included in branch 0, if the number of times they are executed or the order of execution is different, each step at the branch destination must be added.

[0033] Therefore, in the conventional technology, as the number of types of recognition processing increases, the program size of the software describing the sequence of the post-processing portion increases. Therefore, in the data processing device according to the present embodiment, the sequence of the entire recognition processing, including the multiply-and-accumulate portion and the post-processing portion, is input as a parameter from outside, thereby suppressing the increase in program size due to the diversification of recognition processing.

[0034] <Example of data processing device configuration> 4 is a block diagram showing an example of the configuration of a data processing device for processing various types of recognition processing while suppressing an increase in program size. The input unit 401 is a device for inputting instructions and data from a user, and includes a keyboard, a pointing device, buttons, etc. Alternatively, the display unit 404 described below and the input unit 401 may be the same device, as in a known touch screen device. In that case, an input via the touch screen is treated as an input to the input unit 401.

[0035] The data storage unit 402 is a section for storing image data. It is usually configured as a hard disk, a flexible disk, a CD-ROM, a CD-R or DVD, a memory card, a CF card, a smart media, an SD card, a memory stick, an xD picture card, a USB memory, or the like. In addition to image data, the data storage unit 402 can also store programs and other data. Alternatively, a part of the RAM 409 may be used as the data storage unit 402. Alternatively, it may be configured virtually by using a storage device of a device connected via a communication unit 403 (described later).

[0036] The communication unit 403 is an I / F for communication between devices. Note that FIG. 5 shows a diagram in which the input unit 401, data storage unit 402, and display unit 404 (described later) are all included in one device. However, these parts may be connected by a communication path using a known communication method, and may be configured as such as above as a whole. The display unit 404 is a device that displays images before and after image processing, or images such as a GUI. Generally, a CRT or liquid crystal display is used. Alternatively, it may be an external display device connected by a cable or the like.

[0037] The image processing unit 405 receives a command from the CPU 406, which will be described later, reads out image data written in the data storage unit 402, and adjusts the range of pixel values. The processed result is written in the RAM 409, which will be described later. The CPU 406 controls the operation of the entire device. The ROM 408 and the RAM 409 provide the CPU 406 with programs, data, working areas, and the like required for the processing. When a program required for the processing, which will be described later, is stored in the data storage unit 402 or the ROM 408, the program is once loaded into the RAM 409 and then executed. Alternatively, when the device receives a program via the communication unit 403, the program is once recorded in the data storage unit 402 and then loaded into the RAM 409. Alternatively, the program may be directly loaded from the communication unit 403 directly into the RAM 409 and then executed. Although the configuration in FIG. 4 has only one CPU (CPU 406), a configuration in which multiple CPUs are provided may also be used.

[0038] The recognition processing unit 407 receives commands from the CPU 406 and performs recognition processing, for example, as shown in Fig. 2. The recognition processing unit 407 has internal storage necessary for executing arithmetic processing. Data received by the recognition processing unit 407 as input may be received directly from the communication unit 403, or data that has been placed once in the RAM 409 may be read. Data returned by the recognition processing unit 407 as output may be sent directly to the communication unit 403, or written to the RAM 409. There are various other components in addition to those described above in the system configuration of the data processing device, but as they are not the focus of the present invention, description of these will be omitted.

[0039] <Example of the recognition processing unit configuration> 1 shows the configuration of the recognition processing unit 407. The recognition processing unit 407 has three calculation units, namely, a parameter distribution unit 100, a product-sum calculation unit 101, and a command processing unit 102, and three storage units, namely, a feature data holding unit 103, an input data holding unit 104, and an output data holding unit 105.

[0040] The parameter distribution unit 100 receives parameters from an external source and distributes the parameters to the product-sum calculation unit 101 or the command processing unit 102 depending on the contents of the parameters. When the product-sum calculation unit 101 and the command processing unit 102 each complete the processing corresponding to the received parameter, they send an end notification to the parameter distribution unit 100. When the parameter distribution unit 100 receives the end notification, it starts distributing the next parameter.

[0041] The parameters that the parameter distribution unit 100 provides to the product-sum calculation unit 101 are called product-sum calculation parameters. The product-sum calculation parameters include information necessary for the product-sum calculation unit 101 to perform the product-sum calculation part of the recognition process. For example, the product-sum calculation parameters include parameters such as the weighting coefficients of the NN and information such as the number of channels of feature data to be calculated.

[0042] The parameters that the parameter distribution unit 100 gives to the command processing unit 102 are called post-processing parameters. The post-processing parameters are, in other words, a sequence of commands. A series of processes executed by the command processing unit 102, that is, the post-processing portion of the recognition process, is realized by a combination of predefined processes. The parameter distribution unit 100 gives one command to the command processing unit 102, causing it to execute one of the predefined processes.

[0043] The parameter distribution unit 100 is responsible for managing the sequence of a series of recognition processes executed by the recognition processing unit 407. The parameter distribution unit 100 operates the product-sum operation unit 101 and the command processing unit 102 in an appropriate order according to the recognition process sequence. Meanwhile, the order is determined by the arrangement order of the product-sum operation parameters and post-processing parameters in the parameters. Therefore, it is the role of the side that provides the parameters as external input to the parameter distribution unit 100 to provide a processing order according to the recognition process sequence.

[0044] Similarly, the parameter distribution unit 100 is responsible for controlling the sequence of the post-processing portion of the recognition process executed by the command processing unit 102, and the processing order is determined by the arrangement order of the commands in the post-processing parameters. Therefore, it is the role of the side that provides the parameters as external input to the parameter distribution unit 100 to provide a processing order according to the sequence of the post-processing portion.

[0045] The product-sum calculation unit 101 performs the product-sum calculation part of the recognition processing according to the product-sum calculation parameters received from the parameter distribution unit 100. In the product-sum calculation part, the product-sum calculation unit 101 receives an input image from the outside, calculates feature data, and writes the calculated feature data to the feature data holding unit 103. In the first embodiment, the product-sum calculation unit 101 is configured with dedicated hardware consisting of a large number of multipliers and adders, and is capable of processing the product-sum calculation part of the recognition processing more efficiently than, for example, performing the product-sum calculation by the CPU 406. The feature data holding unit 103 is a product-sum calculation result holding unit that holds the product-sum calculation result in this product-sum calculation part as feature data.

[0046] The calculation process performed by the product-sum calculation unit 101 is not necessarily limited to the product-sum calculation, so long as it has characteristics similar to the product-sum calculation, such as playing a role in feature extraction in NN calculation and being easily parallelized by dedicated hardware. For example, in a NN using 1-bit feature data and weighting coefficients that take two values ​​of ±1, an XNOR calculation and a popcount process are performed instead of multiplication and addition. In addition, the calculation process performed by the product-sum calculation unit 101 may include processes other than the product-sum calculation, such as activation function processing and pooling processing, before and after the calculation.

[0047] The command processing unit 102 performs processing of the post-processing portion of the recognition processing in accordance with the post-processing parameters received from the parameter distribution unit 100. In the post-processing portion, the command processing unit 102 reads out the feature data calculated by the product-sum calculation unit 101 from the feature data storage unit 103, and performs processing in accordance with the post-processing settings stored in the input data storage unit 104. The processing results are written to the output data storage unit 105. The command processing unit 102 also performs processing such as receiving post-processing settings from outside and storing them in the input data storage unit 104, and passing the processing results written in the output data storage unit 105 to the outside.

[0048] The post-processing parameters provided by the parameter distributor 100 to the command processor 102 are configured as a string of small parameters indicating predefined processes. Hereinafter, this string of small parameters will be referred to as a command. The command processor 102 executes a process for each command according to the type of command. The above-mentioned processes are carried out during the course of a series of sequences provided by the string of commands.

[0049] Note that the components listed as separate elements in the recognition processing unit 407 may each be physically realized by one device. For example, the parameter distribution unit 100 and the command processing unit 102 can be realized as a CPU 406 and a program installed therein. In this case, the CPU 406 receives parameters as the parameter distribution unit 100, and transfers the parameters to the product-sum calculation unit 101 if the parameters are intended for the product-sum calculation unit 101. On the other hand, if the parameters are intended for the command processing unit 102, the CPU 406 performs processing equivalent to the command processing unit 102 as is.

[0050] 5 shows a flowchart of the processing performed by the command processing unit 102. When the command processing unit 102 starts processing, in step S501 it receives commands one by one from the parameter distribution unit 100, and starts a command loop each time it receives a command.

[0051] In step S502, the process branches depending on the value of the TYPE field of the command. TYPE is a value that indicates the type of command, i.e., which of the predefined processes it is. If TYPE is 0, it indicates "input process" and branches to step S503. If TYPE is 1, it indicates "execute" and branches to step S505. If TYPE is 2, it indicates "output process" and branches to step S504.

[0052] In step S503, a process corresponding to step S302 in Fig. 3 is performed. That is, input data for post-processing is received from the outside and stored in input data holding unit 104. In step S504, a process corresponding to step S305 in Fig. 3 is performed. That is, the output result of post-processing stored in output data holding unit 105 is output to the outside.

[0053] In step S505, one of the predefined post-processing processes is selected to perform the process equivalent to steps S303-304 in Fig. 3. A command with TYPE set to 1 has a SUB_TYPE field, and in step S505, the process branches depending on the value of SUB_TYPE. For example, to select the position estimation process performed in steps S303-304, SUB_TYPE is set to 0. To perform other processes, a value other than 0 is set in SUB_TYPE, but this is not handled in the first embodiment.

[0054] In step S506, a position estimation process is executed. The feature data stored in the feature data storage unit 103 is referenced, and the coordinates of a position with a high score in a specific channel are obtained according to the post-processing settings stored in the input data storage unit 104. The obtained result is stored in the output data storage unit 105 as the processing result.

[0055] In step S507, the command processing unit 102 sends an end notification to the parameter distribution unit 100 to end the current command loop and wait for the next command.

[0056] <Program size> Figure 6 shows the relationship between the flowcharts shown in Figures 3 and 5 and their program sizes. The left side shows the difference in program size when the flowcharts in Figure 3 and Figure 5 are implemented in software, and the right side shows the difference in program size when the program sizes of each process are stacked up.

[0057] The program size 600 for input processing, the program size 601 for output processing, and the program size for position estimation processing are included in both the cases of Fig. 3 and Fig. 5. There is no difference in program size between Fig. 3 and Fig. 5 due to these.

[0058] The program size 603 for other post-processing is the increase in program size when a type of post-processing is added in addition to the position estimation process, such as the linking process in step S313 in Fig. 3. If it is assumed that the types of recognition processes supported are exactly the same in Fig. 3 and Fig. 5, the types of post-processing added are also exactly the same, so there is no difference in program size due to these.

[0059] A program size 604 of a common portion in the flowchart of FIG. 3 is an increase in program size due to processing that does not depend on the type of recognition processing, such as steps S300-301 and step S307.

[0060] 3, the program size 605 of the non-common part is the increase in program size due to the part describing the sequence for each type of recognition processing. The program size 604 of the non-common part increases as the number of branches after step S301 increases due to the increase in the number of types of recognition processing.

[0061] 5, the program size 606 of the common part is the increase in program size due to the parts describing sequences such as repetition and branching other than each command process such as steps S503 to 506. The program size 606 of the common part will not increase even if the number of types of recognition processes increases, unless the number of types of command processes that make up the recognition processes increases.

[0062] As described above, assuming that the types of recognition processing supported in Figures 3 and 5 are the same, the cumulative program size in the flowchart in Figure 3 increases as the number of types of recognition processing increases. On the other hand, the cumulative program size in the flowchart in Figure 5 is constant regardless of the number of types of recognition processing. By providing different sequences as parameters from outside for each type of recognition processing, the data processing device according to this embodiment can support a variety of recognition processing while suppressing an increase in program size.

[0063] <Parameters and Sequences> The following describes an example of how parameters are actually given to realize the recognition process shown in Fig. 2. Fig. 7 shows an example of the configuration of parameters given to the recognition processing unit 407 to execute the recognition process shown in Fig. 2. The parameters are a combination of product-sum operation parameters 700 and post-processing parameters 701, in this order. Therefore, in the recognition processing unit 407, after the product-sum operation portion is processed by the product-sum operation unit 101, the command processing unit 102 processes the post-processing portion.

[0064] The first CNN 201 is executed with the product-sum operation parameters 700. The product-sum operation parameters 700 include control information required for the processing of the product-sum operation unit 101. In addition, the DST field specifies which address in the feature data storage unit 103 the first feature data 202 is to be stored in. Details of the control information are omitted since they are not relevant to the gist of the present invention.

[0065] The post-processing parameters 701 are made up of a post-processing parameter header 702 and commands 703 to 707. The number of commands is given in the LENGTH field of the post-processing parameter header 702. The parameter distributor 100 refers to the LENGTH field and sends the necessary number of commands to the command processor 102.

[0066] In the first command 703, 0 is set in the TYPE field and input processing is performed. The post-processing settings 203 are received from the outside and stored in the input data holding unit 104. The DST field sets the address in the input data holding unit 104 at which the post-processing settings 203 is to be stored. The SIZE field gives the data size received from the outside, i.e., the data size of the post-processing settings 203.

[0067] In the second command 704, 1 is set in the TYPE field and 0 is set in the SUB_TYPE field, and the first position estimation process 204 is executed. In order to provide the storage locations of the first feature data 202 in the SRC0 field and the post-processing setting 203 in the SRC1 field, the same addresses as the DST field of the product-sum operation parameter 700 and the DST field of the first command 703 are set, respectively. The address in the output data holding unit 105 at which the first processing result 206 is to be stored is set in the DST field.

[0068] In the third command 705, 1 is set in the TYPE field and 0 is set in the SUB_TYPE field, and the second position estimation process 205 is executed. In order to provide the storage locations of the first feature data 202 in the SRC0 field and the post-processing setting 203 in the SRC1 field, the same addresses as the DST field of the product-sum operation parameter 700 and the DST field of the first command 703 are set, respectively. The address in the output data holding unit 105 at which the second processing result 207 is to be stored is set in the DST field.

[0069] In the fourth command 706, 2 is set in the TYPE field and output processing is performed. In the output processing, the first processing result 206 stored in the output data holding unit 105 is output to the outside. In the SRC field, the same address as the DST field of the second command 704 is set to provide a place to store the first processing result 206 to be output.

[0070] In the fifth command 707, 2 is set in the TYPE field and output processing is performed. In the output processing, the second processing result 207 stored in the output data holding unit 105 is output to the outside. In the SRC field, the same address as the DST field of the third command 705 is set to provide a place to store the second processing result 207 to be output.

[0071] Fig. 8 shows a memory map when the recognition processing unit 407 performs processing using the parameters shown in Fig. 7. Note that in Fig. 7, the feature data storage unit 103, the input data storage unit 104, and the output data storage unit 105 are separate memories, but a single memory such as the RAM 409 may serve as all three.

[0072] The feature data storage unit 103 is provided with a first feature data area 800, which stores the first feature data 202. The start address of the first feature data area 800 is AAAA according to the DST field of the multiply-add operation parameter 700. In the second command 704 and the third command 705 which access the first feature data area 800, AAAA is set in the corresponding field, i.e., the SRC0 field.

[0073] A post-processing setting area 801 is provided in the input data holding unit 104, and stores the post-processing settings 203. The start address of the post-processing setting area 801 is BBBB according to the DST field of the first command 703. In the second command 704 and the third command 705 which access the post-processing setting area 801, BBBB is set in the corresponding field, i.e., the SRC1 field.

[0074] The output data holding unit 105 is provided with a first processing result area 802 and a second processing result area 803, which respectively store the first processing result 206 and the second processing result 207. In accordance with the DST field of the second command 704, the start address of the first processing result area 802 becomes DDDD. In accordance with the DST field of the third command 705, the start address of the second processing result area 803 becomes EEEE. In the fourth command 706 and the fifth command 707 which access the first processing result area 802 and the second processing result area 803, DDDD and EEEE are set in the corresponding field, i.e., the SRC field, respectively.

[0075] For example, the first processing result area 802 and the second processing result area 803 may be linked and arranged, and the first processing result 206 and the second processing result 207 may be output to the outside in one output process. In that case, the fourth command 706 and the fifth command 707 are integrated into one command, and the address of the start position of the first processing result area 802 may be set in the SRC field. Also, 2×FFFF, which is the sum of the sizes of the first processing result area 802 and the second processing result area 803, may be set in the SIZE field.

[0076] Fig. 9 shows a sequence in which the recognition processing unit 407 performs processing using the parameters shown in Fig. 7. In step S900, the parameter distribution unit 100 receives parameters from an external device and starts the series of recognition processing shown in Fig. 2. In step S901, the parameter distribution unit 100 transmits product-sum operation parameters 700 to the product-sum operation unit 101.

[0077] In step S902, the product-sum calculation unit 101 starts processing of the first CNN 201 according to the received product-sum calculation parameters 700. First, the product-sum calculation unit 101 receives an input image 200 from outside. Next, the product-sum calculation unit 101 calculates the first feature data 202 and stores it in the feature data storage unit 103. Finally, the parameter distribution unit 100 is notified of the completion of processing. In step S903, the parameter distribution unit 100 starts transferring the post-processing parameters 701. In accordance with the setting of the LENGTH field of the post-processing parameter header 702, five commands are transmitted to the command processing unit 102. First, the first command 703 is transmitted.

[0078] In step S904, the command processing unit 102 starts input processing in accordance with the first command 703. First, the command processing unit 102 receives the post-processing settings 203 from the outside. Next, the command processing unit 102 stores the received post-processing settings 203 in the input data holding unit 104. Finally, the command processing unit 102 notifies the parameter distribution unit 100 of the completion of processing.

[0079] In step S905, the parameter distribution unit 100 transmits a second command 704 to the command processing unit 102. In step S906, the command processing unit 102 starts a first position estimation process 204 in accordance with the second command 704. First, the command processing unit 102 reads out the first feature data 202 from the feature data storage unit 103 and the post-processing settings 203 from the input data storage unit 104. Next, a specific channel of the first feature data 202 is referenced to determine the coordinates of a position with a high score, and these are stored in the output data storage unit 105 as a first processing result 206. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0080] In step S907, the parameter distribution unit 100 transmits a third command 705 to the command processing unit 102. In step S908, the command processing unit 102 starts the second position estimation process 205 in accordance with the third command 705. First, the command processing unit 102 receives the first feature data 202 from the feature data storage unit 103 and the post-processing settings 203 from the input data storage unit 104. Next, a specific channel of the first feature data 202 is referenced to determine the coordinates of a position with a higher score than the surroundings, and the coordinates are stored in the output data storage unit 105 as the second processing result 207. Finally, the parameter distribution unit 100 is notified of the completion of the process.

[0081] In step S909, the parameter distribution unit 100 transmits a fourth command 706 to the command processing unit 102. In step S910, the command processing unit 102 starts output processing in accordance with the fourth command 706. First, the first processing result 206 is read from the output data holding unit 105. Next, the first processing result 206 that has been read is output to the outside. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0082] In step S911, the parameter distribution unit 100 transmits a fifth command 707 to the command processing unit 102. In step S912, the command processing unit 102 starts output processing in accordance with the fifth command 707. First, the second processing result 207 is read from the output data holding unit 105. Next, the second processing result 207 that has been read is output to the outside. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0083] As described above, by providing the parameters shown in Fig. 7, the recognition processing unit 407 can perform the series of recognition processes shown in Fig. 2, from receiving the input image 200 and the post-processing settings 203 to returning the first processing result 206 and the second processing result 207. Furthermore, when performing recognition processing different from that shown in Fig. 2, a sequence can be defined with parameters different from those shown in Fig. 7. This allows the recognition processing unit 407 to handle a greater number of types of recognition processing without increasing the program size.

[0084] In the first embodiment, it has been shown that fluctuations in the sequence of the post-processing portion of the recognition process do not affect the program size of the command processing unit 102, and an increase in the overall program size of the recognition processing unit 407 is suppressed. This is realized by a configuration in which a different post-processing sequence for each recognition process is given from the outside as a command string, and the parameter distribution unit 100 commands the command processing unit 102 to execute processing in the order of the commands in the post-processing parameters.

[0085] <Embodiment 2> In the second embodiment, a recognition processing sequence is given in an order that includes not only parameters for controlling the post-processing portion of the recognition processing, i.e., a command sequence, but also parameters for the product-sum operation portion of the recognition processing. By giving parameters with an appropriate configuration, recognition processing that cannot be defined by simply rearranging the commands as in the first embodiment is realized. Also, an example of command extension and sequence giving for more efficient recognition processing is shown.

[0086] In the following, the differences from the first embodiment and newly added parts will be mainly described, and the rest will be omitted. The configuration and features of the data processing device in the first embodiment are the same in the second embodiment unless otherwise specified.

[0087] <Post-processing and sequence diversification> Fig. 10 shows the configuration of the recognition process to be recognized in the second embodiment. The recognition process shown in Fig. 10 consists of three CNNs and three post-processing processes that use their outputs. It receives an input image 1000 and post-processing settings 1009, and returns a first processing result 1006, a second processing result 1011, and a third processing result 1013.

[0088] A first CNN 1001 receives an input image 1000 and performs a product-sum operation. As a result of the product-sum operation, it returns first feature data 1002. The first feature data 1002 is a score map that indicates the score at the position corresponding to the input image 1000, and is made up of multiple score maps. Hereinafter, each score map is referred to as a channel.

[0089] The second CNN 1003 receives the first feature data 1002 and performs a further product-sum operation. The result of the product-sum operation is returned as second feature data 1004. The third CNN 1007 receives the first feature data 1002, similar to the second CNN 1003, and performs a further product-sum operation. The result of the product-sum operation is returned as third feature data 1008.

[0090] The second CNN 1003 and the third CNN 1007 are trained in advance to output a high score in an area where a specific object is captured in the input image 1000. For example, the second feature data 1004 output by the second CNN 1003 has a high score in an area where a person's head is captured. Also, the third feature data 1008 output by the third CNN 1007 has a high score in an area where a person's torso is captured. Therefore, in the second embodiment, the function realized by the first CNN 201 in the first embodiment is realized by three linked CNN groups, the first CNN 1001, the second CNN 1003, and the third CNN 1007.

[0091] The first position estimation process 1005 refers to the second feature data 1004, finds the coordinate values ​​in the input image 1000 for a position where the score is higher than the surroundings, and returns the found coordinate values ​​as a first processing result 1006. Note that the first position estimation process 1005 finds coordinate values ​​for each of a plurality of subjects, and the first processing result 1006 includes a list of a plurality of coordinate values.

[0092] A second position estimation process 1010 refers to the third feature data 1008, finds coordinate values ​​in the input image 1000 for positions with higher scores than the surrounding areas, and returns the found coordinate values ​​as a second processing result 1011. Similar to the first processing result 1006, the second processing result 1011 also includes coordinate values ​​for multiple subjects as a list.

[0093] As described above, in the second embodiment, a series of processes from the first CNN 1001 to the second position estimation process 1010 realizes a function almost equivalent to the recognition process shown in Fig. 2. In addition to this, in the second embodiment, a linking process 1012 is performed.

[0094] In the linking process 1012, the first processing result 1006 and the second processing result 1011 are received, and the subjects whose position coordinates have been determined are linked together. For example, a head and a torso that are presumed to belong to the same person are linked together using information such as the distance between the subjects and their positional relationship. The linking result is given as a pair of a head index and a torso index, and this is returned as a third processing result 1013.

[0095] As described above, in the recognition process shown in Fig. 10, a specific function is realized by combining CNN and post-processing. For example, the positions of the heads of multiple people appearing in the input image 1000 are obtained and output as a first processing result 1006. Also, the positions of the torsos of multiple people appearing in the input image 1000 are obtained and output as a second processing result 1011. Furthermore, if there is a person whose head and torso are both appearing in the input image 1000, the position of the subject in each of the first processing result 1006 and the second processing result 1011 is output as a third processing result 1013.

[0096] In the recognition process shown in Fig. 10, the first CNN 1001, the second CNN 1003, and the third CNN 1007 correspond to the product-sum calculation part. The first position estimation process 1005, the second position estimation process 1010, and the linking process 1012 correspond to the post-processing part. To execute only the post-processing part of the recognition process shown in Fig. 10, for example, the procedure shown in branch 2 of the flowchart shown in Fig. 3 is followed.

[0097] In step S310, input data required for post-processing is input from outside. Here, post-processing settings 1009 required for executing the first position estimation process 1005, the second position estimation process 1010, and the first linking process 1012 are input. In step S311, a position estimation process is executed. This corresponds to the first position estimation process 1005 in FIG. 10. In step S312, a position estimation process is executed. This corresponds to the second position estimation process 1010 in FIG. 10.

[0098] In step S313, a linking process is executed. This corresponds to the first linking process 1012 in Fig. 10. In step S314, the result of the post-processing is output to the outside. Here, the results of the processes in steps S311-313, that is, the first process result 1006, the second process result 1011, and the third process result 1013 in Fig. 10, are output.

[0099] A feature of the recognition process shown in Fig. 10 is that there is a branch in the data dependency. That is, the process branches at the first feature data 1002 into a process continuing from the second CNN 1003 and a process continuing from the third CNN 1007. These processes merge again at the first linking process.

[0100] In the second embodiment, in a branched recognition process, an appropriate sequence is defined by parameters to speed up parallel processing. The following describes the necessary extension of the command processing unit 102 and how to provide parameters to speed up the recognition process shown in FIG. 10.

[0101] <Configuration example of data processing device according to this embodiment> In the second embodiment, similarly to the first embodiment, a data processing device and a recognition processing unit 407 having the configurations shown in Fig. 4 and Fig. 1 are used. The following describes parts that are different from the first embodiment and newly added parts.

[0102] FIG. 11 shows a flowchart of the processing performed by the command processing unit 102 in the second embodiment. Steps S1100-1106 are equivalent to steps S500-506 in the first embodiment. In step S1107, the command processing unit 102 performs a linking process. By referring to the two results of the position estimation process stored in the output data holding unit 105, the subjects included in the results of the position estimation process are linked together in accordance with the post-processing settings stored in the input data holding unit 104. The linking result is stored in the output data holding unit 105 as the processing result.

[0103] In the second embodiment, the recognition processing unit 407 communicates with an external computing device such as the CPU 406. To achieve this, the command processing unit 102 defines a process for communicating with the external computing device. To execute this, 3 is set in the TYPE field of the command.

[0104] In step S1108, for commands with TYPE field set to 3, the type of communication with the external arithmetic device is selected. A command with TYPE set to 3 has a SUB_TYPE field. If the value of the SUB_TYPE field is 0, the process proceeds to step S1109 to instruct the start of arithmetic processing. If the value is 1, the process proceeds to step S1110 to wait for the end of arithmetic processing.

[0105] In step S1109, the command processing unit 102 transmits a message to the external arithmetic device to instruct it to start processing. The content of the message to be transmitted is specified in the MSG field in the command. In step S1110, the command processing unit 102 transmits a message to the external arithmetic device to wait for completion. After transmitting the message, the command processing unit 102 waits until the external arithmetic device receives a notification of completion of the processing specified in the MSG field.

[0106] For simplicity, the external computing device hereinafter refers to the CPU 406. The CPU 406 receives a start instruction message from the command processing unit 102 and starts the specified processing. It also receives a completion waiting message from the command processing unit 102 and returns a completion notification immediately if the specified processing is completed. If the processing is not completed, it continues the processing and returns a completion notification after completion.

[0107] As described above, the command processing unit 102 in the second embodiment supports commands necessary for communication with an external computing device in addition to the three types of commands supported in the first embodiment. Note that the recognition processing unit 407 including the command processing unit 102 according to the flowchart shown in Fig. 11 can execute the recognition process shown in Fig. 2 as in the first embodiment by providing the parameters shown in Fig. 7. Similarly, any of the recognition processes that were executable in the first embodiment can also be processed in the second embodiment by providing the same parameters.

[0108] <Parameters and Sequences> The following describes an example of how to actually provide parameters to realize the recognition process shown in Fig. 10. In the second embodiment, the recognition processing unit 407 and the CPU 406 perform parallel processing to speed up the recognition process shown in Fig. 10. Within the parameters provided to the recognition processing unit 407, a sequence required for cooperation between the recognition processing unit 407 and the CPU 406 is defined.

[0109] Fig. 12 shows an example of the configuration of parameters given to the recognition processing unit 407 to execute the recognition processing shown in Fig. 10. These parameters are a combination of a first product-sum operation parameter 1200, a second product-sum operation parameter 1201, a first post-processing parameter 1202, a third product-sum operation parameter 1206, and a second post-processing parameter 1207 in this order. Therefore, in the second embodiment, the post-processing parts are not executed all at once as in the flowchart shown in Fig. 3, but the product-sum operation part and the post-processing part are executed alternately.

[0110] The first product-sum operation parameters 1200 execute the first CNN 1001. The first product-sum operation parameters 1200 include control information required for processing by the product-sum operation unit. In addition, the DST field specifies which address in the feature data storage unit 103 the first feature data 1004 is to be stored in. The second product-sum operation parameters 1201 execute the second CNN 1003. The second product-sum operation parameters 1201 include control information required for processing by the product-sum operation unit. In addition, the DST field specifies which address in the feature data storage unit 103 the second feature data 1008 is to be stored in.

[0111] The first post-processing parameter 1202 consists of a first post-processing parameter header 1203 and commands 1204-1205. The number of commands is given in the LENGTH field of the first post-processing parameter header 1203. The parameter distributor 100 refers to the LENGTH field and sends the necessary number of commands to the command processor 102.

[0112] In the first command 1204, 0 is set in the TYPE field and input processing is performed. Post-processing settings 1009 are received from the outside and stored in the input data holding unit 104. The DST field sets the address in the input data holding unit 104 at which the post-processing settings 1009 is to be stored. The SIZE field gives the data size received from the outside, i.e., the data size of the post-processing settings 1009.

[0113] In the second command 1205, 3 is set in the TYPE field and 0 is set in the SUB_TYPE field to perform start instruction processing. A message is transmitted from the command processing unit 102 to the CPU 406. The CPU 406 starts the first position estimation processing 1005 according to the message content. The contents of the SRC0, SRC1, and DST fields required for the first position estimation processing 1005 are provided in the MSG field. To provide the storage locations of the second feature data 1004 and the post-processing settings 1009 in the SRC0 field and the SRC1 field, respectively, the same addresses as the DST field of the second multiply-add operation parameter 1201 and the DST field of the first command 1204 are set. In the DST field, an address in the output data holding unit 105 at which the first processing result 1006 is to be stored is set.

[0114] The third CNN 1003 is executed by the third product-sum operation parameters 1206. The third product-sum operation parameters 1201 include control information required for processing by the product-sum operation unit. In addition, the DST field specifies an address in the feature data storage unit 103 at which the third feature data 1008 is to be stored.

[0115] The second post-processing parameter 1207 is made up of a first post-processing parameter header 1208 and commands 1209-12014. The number of commands is given in the LENGTH field of the first post-processing parameter header 1208. The parameter distributor 100 refers to the LENGTH field and sends the necessary number of commands to the command processor 102.

[0116] In the third command 1209, 1 is set in the TYPE field and 0 is set in the SUB_TYPE field, and the second position estimation process 1010 is executed. To provide the storage locations of the third feature data 1008 in the SRC0 field and the post-processing setting 1009 in the SRC1 field, the same addresses as the DST field of the second multiply-add operation parameter 1206 and the DST field of the first command 1204 are set, respectively. In the DST field, the address in the output data holding unit 105 at which the second processing result 1011 is to be stored is set.

[0117] In the fourth command 1210, 3 is set in the TYPE field and 1 is set in the SUB_TYPE field, and end waiting processing is performed. The command processing unit 102 inquires of the CPU 406 about the end status of the first position estimation processing 1005 whose start was instructed by the second command 1205. If the first position estimation processing 1005 is completed, the CPU 406 returns an end notification to the command processing unit 102. The command processing unit 102 waits until it receives an end notification from the CPU 406.

[0118] As described above, the first position estimation process 1005, whose start was instructed by the second command 1205, is awaited to finish by the fourth command 1210. Therefore, the command processing unit 102 and the CPU 406 perform processing in parallel during the period from when the command processing unit 102 processes the second command 1205 to when it processes the fourth command 1210. This makes it possible to shorten the overall processing time compared to when the first position estimation process 1005 is processed by the command processing unit 102.

[0119] In the fifth command 1211, 1 is set in the TYPE field and 1 is set in the SUB_TYPE field, and the first linking process 1012 is executed. The SRC0 field is given the storage location of the first processing result 1006, the SRC1 field is given the storage location of the second processing result 1011, and the SRC2 field is given the storage location of the post-processing setting 1009. Therefore, the value of the DST field set in the MSG field of the second command 1205 and the same address as the DST field of the first command 1204 are set, respectively. The address in the output data holding unit 105 at which the third processing result 1013 is to be stored is set in the DST field.

[0120] In the sixth command 1212, 2 is set in the TYPE field and output processing is performed. The first processing result 1006 stored in the output data holding unit 105 is output to the outside. In the SRC field, the same address as the value of the DST field of the MSG field of the second command 1205 is set to provide a place to store the first processing result 1006 to be output.

[0121] In the seventh command 1213, 2 is set in the TYPE field and output processing is performed. The second processing result 1011 stored in the output data holding unit 105 is output to the outside. The same address as the DST field of the third command 1209 is set in the SRC field to provide a place to store the first processing result 1011 to be output.

[0122] In the eighth command 1214, 2 is set in the TYPE field and output processing is performed. The third processing result 1013 stored in the output data holding unit 105 is output to the outside. The same address as the DST field of the fifth command 1211 is set in the SRC field to provide a place to store the third processing result 1013 to be output.

[0123] Fig. 13 shows a memory map when the recognition processing unit 407 performs processing using the parameters shown in Fig. 12. As in the first embodiment, areas are allocated to the feature data storage unit 103, the input data storage unit 104, and the output data storage unit 105 for each piece of data to be stored.

[0124] The feature data storage unit 103 is provided with a first feature data area 1300, a second feature data area 1301, and a third feature data area 1302, which store the first feature data 1002, the second feature data 1004, and the third feature data 1008, respectively. According to the DST field of the first multiply-accumulate parameter 1200, the start address of the first feature data area 800 is AAAA. According to the DST field of the second multiply-accumulate parameter 1201, the start address of the second feature data area 1301 is BBBB. According to the DST field of the third multiply-accumulate parameter 1206, the start address of the third feature data area 1302 is CCCC. In the second command 1205 that accesses the second feature data area 1301, BBBB is set in the corresponding field, i.e., the SRC0 field of the MSG field. In the third command 1209 that accesses the third feature data area 1302, CCCC is set in the corresponding field, i.e., the SRC0 field.

[0125] The input data holding unit 104 is provided with a post-processing setting area 1303, which stores the post-processing settings 1009. The start address of the post-processing setting area 1303 is DDDD in accordance with the DST field of the first command 1204. In the second command 1205, third command 1209, and fifth command 1211 which access the post-processing setting area 801, DDDD is set in the corresponding field.

[0126] The output data holding unit 105 is provided with a first processing result area 1304, a second processing result area 1305, and a third processing result area 1306, which respectively store the first processing result 1006, the second processing result 1011, and the third processing result 1013. According to the MSG field of the second command 1205, the start address of the first processing result area 1304 is FFFF. According to the DST field of the third command 1209, the start address of the second processing result area 1305 is GGGG. In the fifth command 1211 which refers to the first processing result area 1304 and the second processing result area 1305, FFFF and GGGG are set in the corresponding SRC0 field and SRC1 field, respectively. Similarly, FFFF and GGGG are set in the SRC fields of the sixth command 1212 and the seventh command 1213, respectively. According to the DST field of the fifth command 1211, the start address of the third processing result area 1306 is HHHH. The corresponding field of the eighth command 1214 referencing the third processing result area 1306, ie, the SRC field, is set to HHHH.

[0127] Fig. 14 shows a sequence in which the recognition processing unit 407 performs processing using the parameters shown in Fig. 12. In the second embodiment, the command processing unit 102 and the CPU 406 perform parallel processing from the second command 1205 to the fifth command 1211 shown in Fig. 12. Fig. 14 shows a sequence that includes the operation of the CPU 406 in addition to Fig. 9 in the first embodiment.

[0128] In step S1400, the parameter distribution unit 100 receives parameters from an external device and starts a series of recognition processes shown in Fig. 10. In step S1401, the parameter distribution unit 100 transmits the first product-sum operation parameters 1200 to the product-sum operation unit 101. In step S1402, the product-sum operation unit 101 starts processing of the first CNN 1001 according to the received first product-sum operation parameters 1200. First, the product-sum operation unit 101 receives an input image 1000 from an external device. Next, the product-sum operation unit 101 calculates first feature data 1002 and stores it in the feature data storage unit 103. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0129] In step S1403, the parameter distribution unit 100 transmits the second product-sum operation parameters 1201 to the product-sum operation unit 101. In step S1404, the product-sum operation unit 101 starts processing of the second CNN 1003 according to the received second product-sum operation parameters 1201. First, the product-sum operation unit 101 receives the first feature data 1002 from the feature data storage unit 103. Next, the product-sum operation unit 101 calculates the second feature data 1004 and stores it in the feature data storage unit 103. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0130] In step S1405, the parameter distributor 100 starts transferring the first post-processing parameters 1202. In accordance with the settings in the LENGTH field of the post-processing parameter header 1203, a command is sent twice to the command processor 102. First, the first command 1204 is sent. In step S1406, the command processor 102 starts input processing in accordance with the first command 1204. First, the command processor 102 receives post-processing settings 1009 from the outside. Next, the received post-processing settings 1009 is stored in the input data storage unit 104. Finally, the parameter distributor 100 is notified of the completion of processing.

[0131] In step S1407, the parameter distribution unit 100 transmits the second command 1205 to the command processing unit 102. In step S1408, the command processing unit 102 instructs the CPU 406 to start the first position estimation process 1005 in accordance with the second command 1205. The command processing unit 102 transmits the contents of the MSG field of the second command 1205 to the CPU 406. The CPU 406 starts the first position estimation process 1005 in accordance with the contents of the received MSG field.

[0132] In step S1409, the parameter distributor 100 transmits the third product-sum operation parameters 1206 to the command processor 102. Meanwhile, the CPU 406 reads out the second feature data 1004 from the feature data storage unit 103 and the post-processing settings 1009 from the input data storage unit 104. The CPU 406 refers to a specific channel of the first feature data 1004, and obtains the coordinates of a position with a high score.

[0133] In step S1410, the product-sum calculation unit 101 starts processing of the third CNN 1007 according to the received third product-sum calculation parameters 1206. First, the product-sum calculation unit 101 receives the first feature data 1002 from the feature data storage unit 103. Next, the product-sum calculation unit 101 calculates the third feature data 1008 and stores it in the feature data storage unit 103. Finally, the product-sum calculation unit 101 notifies the parameter distribution unit 100 of the completion of processing. Meanwhile, the CPU 406 stores the coordinates calculated in step S1409 in the output data storage unit 105 as the first processing result 1006. As a result, the CPU 406 has completed the first position estimation process 1005 instructed to start by the command processing unit 102 in step S1408.

[0134] In step S1411, the parameter distribution unit 100 starts transferring the second post-processing parameters 1207. In accordance with the setting of the LENGTH field of the post-processing parameter header 1208, the command is transmitted six times to the command processing unit 102. First, the third command 1209 is transmitted. In step S1412, the command processing unit 102 starts the second position estimation process 1010 in accordance with the third command 1209. First, the third feature data 1008 is read from the feature data storage unit 103, and the post-processing setting 1009 is read from the input data storage unit 104. Next, a specific channel of the third feature data 1008 is referenced, and the coordinates of the position with the highest score are obtained and stored in the output data storage unit 105 as the second processing result 1011. Finally, the parameter distribution unit 100 is notified of the completion of the process.

[0135] In step S1413, the parameter distribution unit 100 transmits a fourth command 1210 to the command processing unit 102. In step S1414, the command processing unit 102 waits for the CPU 406 to finish the first position estimation process 1005 in accordance with the fourth command 1210. First, the command processing unit 102 inquires of the CPU 406 about the completion status of the first position estimation process 1005. The command processing unit 102 continues the inquiry until receiving a completion notification from the CPU 406.

[0136] In the second embodiment, the processing of the CPU 406 is completed in step S1410, but it should be noted that this is not defined in advance by parameters. In other words, the timing at which the processing of the CPU 406 is completed varies within the range of steps S1408-1414 depending on factors that vary at the time of execution of the input image 1000, etc.

[0137] If the first position estimation process 1005 is completed, the CPU 406 returns a completion notification to the command processing unit 102 as a response to the inquiry. Note that, although the second embodiment is configured such that the command processing unit 102 actively waits for the completion, the CPU 406 may actively notify the command processing unit 102 of the completion of the first position estimation process 1005 by an interrupt, for example. When the command processing unit 102 completes waiting for the completion of the first position estimation process 1005, it returns a completion notification to the parameter distribution unit 100.

[0138] In step S1415, the parameter distribution unit 100 transmits a fifth command 1211 to the command processing unit 102. In step S1416, the command processing unit 102 starts the linking process 1012 in accordance with the fifth command 1211. First, the first processing result 1006 and the second processing result 1011 are read from the output data holding unit 105, and the post-processing setting 1009 is read from the input data holding unit 104. Next, for a plurality of subjects included in the first processing result 1006 and the second processing result 1011, i.e., the heads and torsos of people, subjects that are estimated to belong to the same person are linked together. The result of the linking is stored in the output data holding unit 105 as a third processing result 1013. Finally, the parameter distribution unit 100 is notified of the completion of the processing.

[0139] In step S1417, the parameter distribution unit 100 transmits the sixth command 1212 to the command processing unit 102. In step S1418, the command processing unit 102 starts output processing in accordance with the sixth command 1212. First, the first processing result 1006 is read from the output data holding unit 105. Next, the first processing result 1006 that has been read is output to the outside. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0140] In step S1419, the parameter distribution unit 100 transmits the seventh command 1213 to the command processing unit 102. In step S1420, the command processing unit 102 starts output processing in accordance with the seventh command 1213. First, the second processing result 1011 is read from the output data holding unit 105. Next, the second processing result 1011 that has been read is output to the outside. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0141] In step S1421, the parameter distribution unit 100 transmits the eighth command 1214 to the command processing unit 102. In step S1422, the command processing unit 102 starts output processing in accordance with the eighth command 1214. First, the third processing result 1013 is read from the output data holding unit 105. Next, the read third processing result 1013 is output to the outside. Finally, the parameter distribution unit 100 is notified of the completion of processing.

[0142] As described above, by providing the parameters shown in Fig. 12, the recognition processing unit 407 can perform the series of recognition processes shown in Fig. 10. That is, the recognition processing can be performed from receiving the input image 1000 and the post-processing setting 1009 to returning the first processing result 1006, the second processing result 1011, and the third processing result 1013. In the second embodiment, as in the first embodiment, the recognition processing unit 407 can handle different recognition processes without increasing the program size.

[0143] In the second embodiment, over the period from step S1408 to step S1414, the recognition processing unit 407 processes the third CNN 1008 and the second position estimation process 1010 in parallel, and the CPU 406 processes the first processing result 1006. This makes it possible to speed up the recognition process by parallel processing in the sequence definition by the parameters shown in FIG.

[0144] In the second embodiment, the recognition processing unit 407 and the CPU 406 perform parallel processing, and the parameter distribution unit 100 is responsible for controlling these. Meanwhile, the command processing unit 102 issues a processing start instruction to the CPU 406 and waits for the end, and these sequences are defined by parameters. Therefore, the only new configuration required to add parallel processing is to add steps S1108-1110 of the flowchart shown in Fig. 11 to the command processing unit 102. Also, for example, when changing the timing or number of times to start processing and wait for the end, it is only necessary to change the parameters, so that an increase in the program size of the recognition processing unit 407 as a whole can be prevented.

[0145] On the other hand, if the second position estimation process 1010 tends to take longer to process than the first position estimation process 1005, for example, the order of the processes can be changed to increase the speed. First, the third CNN 1007 is processed before the second CNN 1003. The second CNN 1003 and the first position estimation process 1005 are processed by the recognition processing unit 407, and the second position estimation process 1010 is processed by the CPU 406. To achieve this, the second product-sum operation parameters 1201 and the third product-sum operation parameters 1206 are switched, and the settings of the MSG field of the second command 1205 and each field of the third command are switched.

[0146] In this way, in the data processing device according to this embodiment, it is possible to redefine the recognition processing sequence according to the purpose simply by changing the parameter order. Even in this case, it is not necessary to add a new component to the recognition processing unit 407, especially the command processing unit 102, so that it is possible to handle different recognition processing sequences while suppressing an increase in program size.

[0147] In the second embodiment, a sequence including not only the post-processing part of the recognition process but also the product-sum operation part is defined as a parameter, thereby achieving an effect different from that of the first embodiment, that is, speeding up the recognition process. This is achieved by a configuration in which the parameter distribution unit 100 receives parameters that combine product-sum operation parameters and post-processing parameters, and determines the processing order of the product-sum operation unit 101 and the command processing unit 102 according to the arrangement of the parameters.

[0148] <Other embodiments> <Definition of multiply-and-accumulate part> In the first and second embodiments, image recognition processing is taken as an example, and an example of handling recognition processing including CNN is shown. However, this data processing device is not limited to CNN, and can be applied to recognition processing in general including NN in general. For example, the first CNN 201 in the first embodiment may be replaced with a general multilayer perceptron that does not perform convolutional operations.

[0149] <Definition of post-processing part> In the first and second embodiments, the process of obtaining a recognition result from the output of the neural network is post-processed by the command processing unit 102, and the sequence is defined by post-processing parameters.

[0150] However, the processing executed by the command processing unit 102 does not have to be processing located at a later stage of the NN. For example, part of the NN calculation may be executed as processing by the command processing unit 102, or a part corresponding to pre-processing of the NN to be processed next may be processed in advance by the command processing unit 102. In either case, a new SUB_TYPE is defined for the command with TYPE of 1 in the flowchart shown in Fig. 5. The necessary function is added to the command processing unit 102, and is used by being called by a command with the corresponding SUB_TYPE set.

[0151] <Configuration of command processing unit 102> In the first and second embodiments, the post-processing is divided into three or four types of processing, and a sequence is defined as a string of commands corresponding to the processing. Then, the command processing unit 102 determines the type of processing from the TYPE field of the command, as shown in the flowcharts of Figs. 5 and 11.

[0152] However, the definition of the TYPE field in the flowcharts shown in Figures 5 and 11 is merely an example, and the commands used in the first and second embodiments may be integrated and divided, and the TYPE and SUB_TYPE may be differentiated by different definitions. For example, the input process and the output process may be aggregated into one TYPE, and the configuration may be changed to differentiate by SUB_TYPE. Alternatively, the position estimation process may be divided into smaller process groups, and these may be added as new SUB_TYPE branches below the branch with TYPE being 1.

[0153] <Parallel processing of the product-sum calculation unit 101 and the command processing unit 102> In the first and second embodiments, the parameter distributor 100 distributes parameters to the product-sum calculator 101 and the command processor 102, and causes them to operate alternately.

[0154] However, the product-sum calculation unit 101 and the command processing unit 102 may be configured to operate in parallel. For example, when processing the parameters shown in Fig. 12, the parameter distribution unit 100 may start transferring the third product-sum calculation parameters 1206 to the product-sum calculation unit 101 without waiting for the command processing unit 102 to complete processing of the second command 1205.

[0155] In this case, for example, a field indicating whether the parameter distribution unit 100 will wait for a completion notification can be added to each command of the parameters shown in Fig. 12. This field will be 0 for the second command 1205, and all others will be 1. The parameter distribution unit 100 will check this field for each command, and will wait for a completion notification only if necessary.

[0156] <Skipping by the parameter distribution unit 100> In the first and second embodiments, the parameter distributor 100 processes all of the multiply-add operation parameters and post-processing parameters in the parameters.

[0157] However, the parameter distribution unit 100 may be configured to selectively process only a part of the data depending on the status of the recognition process. For example, if a position with a high score is not found in the first feature data 202 in the first position estimation process 204 shown in Fig. 2, that is, if the position of the person's head is not output, it is not necessary to execute the first processing result 206.

[0158] In this case, for example, a configuration may be added in which the parameter distribution unit 100 skips the transfer of the corresponding command when it receives an abnormal end notification from the command processing unit 102. When the first processing result 206 is empty, the command processing unit 102 sends an abnormal end notification to the parameter distribution unit 100. Upon receiving the abnormal end notification, the parameter distribution unit 100 does not transfer the fourth command 706 to the command processing unit 102, and proceeds to the next fifth command 707.

[0159] <Implementation form of the recognition processing unit 407> In the first and second embodiments, the recognition processing unit 407 is described as hardware constituting a part of the data processing device according to the present invention. However, for example, the entire recognition processing unit 407 may be realized as a program that runs on a CPU. In that case, too, the recognition processing unit 407 has functions corresponding to the parameter distribution unit 100, the product-sum calculation unit 101, and the command processing unit 102, and provides a recognition processing sequence by parameters such as those shown in Figs. 7 and 12.

[0160] In the above embodiment, the convolution calculation is performed by hardware, but it may be performed by a processor such as a CPU, a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processing Unit).

[0161] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0162] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0163] 100 Parameter distribution section 101 Multiply-and-accumulate section 102 Command processing section 103 Feature data storage unit 104 Input data storage unit 105 Output data storage unit

Claims

1. a multiply-and-accumulate operation means for performing a multiply-and-accumulate operation on data based on given multiply-and-accumulate operation parameters; an operation result holding means for holding the operation result of the sum-of-products operation means; command processing means for performing processing defined in a given command on the result of the sum-of-products operation held in the operation result holding means, based on the given command; a distribution means for receiving a parameter string that combines the sum-of-products operation parameters and the commands, and for providing the sum-of-products operation parameters to the sum-of-products operation means and the commands to the command processing means based on the content and order of the combination.

2. the distribution means provides the sum-of-products operation parameters and the command to the sum-of-products operation means and the command processing means; 2. The data processing apparatus according to claim 1, wherein said sum-of-products operation means and said command processing means perform processing in the order in which said distribution means provides said sum-of-products operation parameters and said commands.

3. 2. The data processing apparatus according to claim 1, wherein said product-sum operation means performs product-sum operations in a neural network based on the contents of said product-sum operation parameters.

4. further comprising input data holding means for holding input data used in the processing of said command processing means; 2. The data processing device according to claim 1, wherein said command processing means performs processing to store input data received from an external source in said input data holding means based on the content of said command.

5. further comprising output data holding means for holding output data obtained by processing of said command processing means; 2. The data processing apparatus according to claim 1, wherein said command processing means performs processing to output the output data held in said output data holding means to an external device based on the content of said command.

6. input data holding means for holding input data used in processing by said command processing means; further comprising output data holding means for holding output data obtained by processing of said command processing means; 2. The data processing device according to claim 1, wherein the command processing means performs processing based on the content of the command by referring to the data held by the operation result holding means and the input data holding means, and stores the data obtained by the processing in the output data holding means.

7. The computer further includes a calculation processing means for performing any of the processes performed by the command processing means based on the content of the command, 2. The data processing device according to claim 1, wherein said command processing means commands said arithmetic processing means to start processing based on the content of said command, and waits for completion of processing by said arithmetic processing means.

8. 2. The data processing apparatus according to claim 1, wherein said distribution means determines whether to wait for completion of processing by said command processing means based on the content of said command.

9. 2. The data processing device according to claim 1, wherein said sum-of-products operation means performs said sum-of-products operation in parallel with processing defined in said command by said command processing means.

10. the command processing means notifies the distribution means of the results of the processing performed based on the command; 2. The data processing apparatus according to claim 1, wherein said distribution means switches the command to be sent to said command processing means based on a notification from said command processing means.

11. 2. The data processing apparatus according to claim 1, wherein said product-sum calculation means calculates feature data by performing a product-sum calculation in recognition processing based on said product-sum calculation parameters.

12. 12. The data processing apparatus according to claim 11, wherein said command processing means calculates the result of said recognition processing from said feature data based on said command.

13. A data processing method for a data processing device including a product-sum calculation means, a calculation result holding means, and a command processing means, comprising: a distribution step of receiving a parameter string combining product-sum operation parameters and commands, and providing the product-sum operation parameters to the product-sum operation means and the commands to the command processing means based on the content and order of the combination; a product-sum operation step of performing a product-sum operation on data by the product-sum operation means based on given product-sum operation parameters; a storing step of storing the calculation result of the product-sum calculation step in the calculation result holding means; a command processing step of performing a process defined in a given command on the product-sum operation result held in the operation result holding means, based on the given command, by the command processing means.

14. 2. A program that causes a computer to function as each of the means of the data processing device according to claim 1.

Images