Information processing apparatus, and information processing method

Abstract

To provide an object detection technique with high calculation efficiency.SOLUTION: A method according to the present invention has the steps of, by using characteristics of an image including an object, detecting the object, and detecting a part of the object from the image based on selected parameters selected from a plurality of parameters based on a direction of the detected object and the characteristics.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object detection technique. [Background technology]

[0002] Hierarchical computation methods (pattern recognition methods based on deep learning technology) such as convolutional neural networks (CNNs) have attracted attention as object detection methods that are robust against object movements.

[0003] Patent document 1 discloses a technology that provides a learning device that outputs first information regarding the orientation of a face included in a target image and a learning device that outputs second information regarding the positions of facial features included in the target image, and acquires the second information from the target image using the second learning device that corresponds to the first information.

[0004] Patent Document 2 discloses a technology for selecting parameters of an (m+1)th stage machine learning model corresponding to the mth stage recognition result from parameters of multiple machine learning models expanded in memory by lightweight analysis processing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-151938 A [Patent Document 2] Patent Publication No. 2021-196893 Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology disclosed in Patent Document 1, by using different learning devices for extracting the first information and acquiring the second information, it is possible to use learning devices optimized for each. Therefore, it is possible to acquire the first information and the second information more efficiently than by providing a learning device capable of simultaneously outputting the first information and the second information. On the other hand, when extracting information closely related to the first information and the second information, the first learning device and the second learning device may each calculate similar features in a redundant manner, which reduces the computational efficiency.

[0007] In addition, in the technology disclosed in Patent Document 2, the time required to load the parameters of the machine learning model into memory can be hidden during the calculation of the m-th stage machine learning model. On the other hand, since the amount of calculation of the machine learning model itself cannot be reduced, the calculation time of the (m+1)-th stage machine learning model cannot be reduced. The present invention provides an object detection technology with high calculation efficiency. [Means for solving the problem]

[0008] One aspect of the present invention is characterized by comprising a first detection means for detecting an object from an image including the object using features of the image, and a second detection means for detecting a part of the object from the image based on a selection parameter selected from a plurality of parameters based on the orientation of the object detected by the first detection means and the features. Effect of the Invention

[0009] According to the present invention, a computationally efficient object detection technique can be provided. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram showing an example of the arrangement of a CNN processing unit 201. [Diagram 2] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device. [Diagram 3] FIG. 2 is a block diagram showing an example of the functional configuration of a calculation processing unit 103. [Figure 4] A diagram showing an example of a CNN network configuration. [Diagram 5] 5A to 5C are diagrams showing examples of a score map 501, a region width map 502, and a region height map 503. [Figure 6] FIG. 2 is a block diagram showing an example of a functional configuration for performing face detection and facial organ detection. [Figure 7] 1 is a flowchart of face detection and facial organ detection. [Figure 8] 8A to 8C are diagrams showing examples of a score map 801, a face region width map 802, a face region height map 803, and a face direction map 804. [Figure 9] FIG. 2 is a diagram showing an example of the data configuration in a RAM 205. [Figure 10] FIG. 10 shows an example of a right eye likelihood map 1001, a left eye likelihood map 1002, a nose likelihood map 1003, and a mouth likelihood map 1004. [Figure 11] FIG. 6 is a block diagram showing an example of the configuration of a learning process of a common CNN process 601, a face detection CNN process 602, and a face organ detection CNN process 605. [Figure 12] 11 is a flowchart of a learning method according to a learning sequence 1101. [Figure 13] FIG. 11 is a diagram for explaining an outline of functions according to the second embodiment. [Figure 14] 10 is a flowchart of a detection process according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [First embodiment] In this embodiment, an information processing device that operates as an object detector and its operation will be described. The information processing device according to this embodiment detects a person's face area from an image (face detection) and detects each organ (eyes, nose, mouth) that is a part of the face (face organ detection). First, an example of the hardware configuration of the information processing device according to this embodiment will be described with reference to the block diagram of FIG. 2.

[0013] The image input unit 202 is a device for acquiring an input image. For example, the image input unit 202 may be a device that acquires an input image via a network such as a LAN or the Internet. Alternatively, the image input unit 202 may be an imaging device that captures a moving image and acquires an image of each frame in the moving image as an input image. Alternatively, the image input unit 202 may be an imaging device that captures a still image and acquires the still image as an input image. When the image input unit 202 is an imaging device, the imaging device includes an optical system such as a lens, a photoelectric conversion device such as a CCD (Charge-Coupled Devices) or a CMOS (Complimentary Metal Oxide Semiconductor) sensor, a driver circuit that controls the photoelectric conversion device, an A / D converter, and the like.

[0014] The CNN processing unit 201 performs the face detection and face organ detection by inputting the input image acquired by the image input unit 202 to a CNN, which is an example of a hierarchical neural network, and performing the CNN calculation (CNN calculation). The CNN processing unit 201 will be described in detail later.

[0015] A CPU (Central Processing Unit) 203 executes various processes using computer programs and data stored in a RAM 205. As a result, the CPU 203 controls the operation of the entire information processing device, and executes or controls various processes described as processes performed by the information processing device.

[0016] For example, the CPU 203 performs various detection tasks based on various features obtained by the detection process in the CNN processing unit 201. Furthermore, the CPU 203 can execute various applications using the results of various detection processes. For example, the CPU 203 uses the results of face detection and face organ detection to compare with a face database and acquire information corresponding to the face and face organs in the input image from the face database. Also, for example, the CPU 203 can perform control such as focusing on the face and face organs in the input image by feeding back the results of face detection and face organ detection to the image input unit 202 and using them for focus control of the optical system.

[0017] A ROM (Read Only Memory) 204 stores setting data for the information processing device, computer programs and data related to the startup of the information processing device, computer programs and data related to the basic operation of the information processing device, and the like.

[0018] The RAM (Random Access Memory) 205 has an area for storing computer programs and data loaded from the ROM 204, and an area for storing input images acquired by the image input unit 202. The RAM 205 further has a work area used when the CPU 203 and the CNN processing unit 201 execute various processes. In this way, the RAM 205 can provide various areas as appropriate. The RAM 205 can be configured, for example, with a large-capacity DRAM (Dynamic Random Access Memory) or the like.

[0019] The user interface unit 206 is a user interface such as a keyboard, a mouse, and a touch panel screen, and allows the user to input various instructions to the CPU 203 by operating it.

[0020] The image input unit 202, the CNN processing unit 201, the CPU 203, the ROM 204, the RAM 205, and the user interface unit 206 are all connected to a system bus 207.

[0021] Next, a configuration example of the above-mentioned CNN processing unit 201 will be described with reference to the block diagram of Fig. 1. In this embodiment, the functional units shown in Fig. 1 will be described as being all implemented in hardware, but this is not limiting, and some may be implemented in software (computer program). In the latter case, the software is executed by the CPU 203 or the control unit 107, thereby realizing the functions of the functional units corresponding to the software.

[0022] The I / F unit 101 functions as an interface for the CNN processing unit 201 to perform data communication with the outside, and is an interface that can be accessed by, for example, the CPU 203, the DMAC 102, and the control unit 107.

[0023] A DMAC (Direct Memory Access Controller) 102 controls data transfer between the CNN processing unit 201 and the RAM 205 in accordance with settings made by the control unit 107 .

[0024] The calculation processing unit 103 performs a convolution calculation using the weighting coefficients stored in the buffer 104 and the features (the calculation results of the layer (previous layer) before the layer to be processed in the CNN) stored in the buffer 105.

[0025] The buffer 104 is a memory that holds weighting coefficients used in the calculations of multiple layers in the CNN and can supply the held weighting coefficients with low latency to the calculation processing unit 103. The buffer 104 can be implemented using, for example, a high-speed static RAM (SRAM) or a register.

[0026] The buffer 105 is a memory for storing the results of calculations by the calculation processing unit 103 and the results of calculations by the conversion processing unit 106, and can be implemented using, for example, a high-speed SRAM or a register, similar to the buffer 104.

[0027] The conversion processing unit 106 performs nonlinear conversion on the calculation result (predetermined convolution calculation result) by the calculation processing unit 103. For the nonlinear conversion, functions such as ReLU (Rectified Linear Unit) and sigmoid function are used. In the case of ReLU, this can be realized by threshold processing, and in the case of the sigmoid function, values ​​are converted using a lookup table or the like. The control unit 107 controls the operation of the entire CNN processing unit 201. The control unit 107 can be implemented using a sequencer that controls the calculation processing unit 103, a simple CPU, or the like.

[0028] Here, a more detailed description will be given of the calculation processing unit 103. In the calculation processing unit 103, when the size of the kernel (filter coefficient matrix) of the convolution calculation is columnSize×rowSize and the number of feature maps in the previous layer is L, one feature is calculated by performing a convolution calculation as shown in the following formula (1).

[0029]

number

[0030] …(1) input(x,y) : Reference pixel value at 2D coordinates (x,y) output(x,y): Calculation result in two-dimensional coordinates (x,y) weight(column,row): Weighting coefficient at coordinate (column, row) L: Number of feature maps in the previous layer columnSize: The horizontal size of the 2D convolution kernel rowSize : The vertical size of the 2D convolution kernel Generally, in CNN operations, multiple convolution kernels are scanned in units of pixels of an image according to the above formula (1), and multiply-and-accumulate operations are repeated, and the final product-and-accumulate result is subjected to nonlinear transformation (activation processing) to calculate a feature map. In other words, the pixel data of one feature map is generated by multiple spatial filter operations (hierarchical spatial filter operations) and a nonlinear operation on their sum. The weighting coefficients correspond to the spatial filter coefficients. In reality, multiple feature maps are generated for each layer.

[0031] An example of the functional configuration of the arithmetic processing unit 103 will be described with reference to the block diagram of Fig. 3. As shown in Fig. 3, the arithmetic processing unit 103 has a multiplier 301 and an accumulator 302, and executes a convolution operation between a weighting coefficient 303 and a feature (CNN feature plane) 304 according to the above formula (1) using the multiplier 301 and the accumulator 302. The conversion processing unit 106 executes a nonlinear conversion on the result of the convolution operation.

[0032] Hereinafter, features are referred to as feature maps because they can be expressed in multiple two-dimensional map formats. In a typical CNN, the above process is repeated the number of times equal to the number of feature maps to be generated. The calculated feature maps are stored in buffer 105.

[0033] An example of the network configuration of a CNN for implementing object detection using the CNN will be explained with reference to Figure 4. For the sake of explanation, a small CNN (three layers) will be used here. In practice, many more feature planes and multiple layers are often used.

[0034] When performing CNN arithmetic processing on an input image, the input layer 401 corresponds to a raster-scanned image of a predetermined size. Feature planes 403a to 403c are feature planes of the first hierarchical layer 408, feature planes 405a and 405b are feature planes of the second hierarchical layer 409, and feature plane 407 is a feature plane of the third hierarchical layer 410.

[0035] A feature surface is a data surface that corresponds to the processing result of a predetermined feature extraction operation (convolution operation and nonlinear processing). Since the feature surface is the processing result for a raster-scanned image, the processing result is also expressed as a surface. The feature surfaces 403a to 403c are calculated by the convolution operation and nonlinear processing corresponding to the input layer 401.

[0036] For example, feature plane 403a is calculated by performing a convolution operation using a two-dimensional convolution kernel 4021a shown in schematic form and a nonlinear transformation on the result of the convolution operation. Feature plane 405a is the result of performing a convolution operation on feature planes 403a to 403c using convolution kernels 4041a, 4042a, and 4043a, respectively, and performing a nonlinear transformation on the sum of the results of the convolution operations.

[0037] Also, for example, feature surface 403b is calculated by a convolution operation using a two-dimensional convolution kernel 4021b shown in schematic form and a nonlinear transformation of the result of the convolution operation, while feature surface 403c is calculated by a convolution operation using a two-dimensional convolution kernel 4021c shown in schematic form and a nonlinear transformation of the result of the convolution operation.

[0038] Feature plane 405b is the result of performing convolution operations on feature planes 403a to 403c using convolution kernels 4041b, 4042b, and 4043b, respectively, and performing nonlinear transformation on the sum of the results of the convolution operations.

[0039] In CNN, feature planes are calculated sequentially in this way using hierarchical processing that uses convolution kernels. When a certain amount of learning has been performed, the feature plane 407 of the final layer has a higher data value corresponding to the position of the detection target object than the data values ​​at other positions. Also, when determining the region of the detection target object, the data value of the region of the detection target object is higher than the data values ​​of other regions.

[0040] Next, the CPU 203 or the like searches for a location where the calculated feature surface 407 of the detection target object shows a value higher than a predetermined threshold, thereby performing object detection. It is also possible to generate multiple feature surfaces in the final layer and detect the area of ​​the detection target object. The detection operation for these feature surfaces is called post-detection processing.

[0041] FIG. 5 shows an example of a score map 501, a region width map 502, and a region height map 503 of the object to be detected, which are feature surfaces of the final layer, and the CNN is trained to output these.

[0042] The value at each position of the score map 501 indicates the "possibility of being a detection target object" at that position, and indicates a value indicating likelihood or reliability that becomes larger as the possibility of being a detection target object increases. Therefore, a position in the score map 501 with a value larger than a predetermined threshold value can be determined as the position of the detection target object. In the example of Fig. 5, the position with a value of 255 can be determined as the center position of the detection target object.

[0043] The value at each position in the region width map 502 indicates the "estimated width of the region of the detection target object" at that position. Therefore, the value of the position in the region width map 502 corresponding to the center position of the detection target object identified from the score map 501 can be obtained as the "width of the region of the detection target object."

[0044] The value at each position in the region height map 503 indicates the "estimated height of the region of the detection target object" at that position. Therefore, the value of the position in the region height map 503 that corresponds to the center position of the detection target object identified from the score map 501 can be obtained as the "height of the region of the detection target object."

[0045] In FIG. 5, the position in score map 501 where the value is “255” (the position on the left) is specified as the position of object 1, and the position in score map 501 where the value is “255” (the position on the right) is specified as the position of object 2.

[0046] Then, since the value corresponding to the position of object 1 in area width map 502 is 10, and the value corresponding to the position of object 1 in area height map 503 is 5, the width of the area of ​​object 1 can be specified as 10, and the height as 5.

[0047] Similarly, the value corresponding to the position of object 2 in region width map 502 is 4, and the value corresponding to the position of object 2 in region height map 503 is 8, so the width and height of the region of object 2 can be specified as 4 and 8, respectively.

[0048] In addition, when the detection target object is captured large in the image or when the threshold for the score map is low, multiple positions where the detection target object is successfully detected may appear close to each other even though there is only one detection target object. In such cases, in order to narrow down the successful detection of the detection target object to one, the detection targets may be combined into one using NMS (Non-Maximum Suppression) technology.

[0049] Next, an example of a functional configuration for performing face detection and face organ detection, which is realized by the CPU 203 and the CNN processing unit 201, will be described with reference to the block diagram of FIG. 6. The functional configuration realized by the CPU 203 and the CNN processing unit 201 includes a common feature generator, a first detector, and a second detector. The common feature generator executes a common CNN process 601. The first detector executes a face detection CNN process 602, a face detection post-processing 603, and a direction information calculation 604. The second detector executes a face organ detection CNN process 605 and a face organ detection post-processing 606. The face detection and face organ detection according to this embodiment will be described with reference to the flowchart of FIG. 7.

[0050] In step S701, the CPU 203 executes various initializations of the CNN processing unit 201. In step S702, the control unit 107 performs DMAC setting for transferring weighting coefficients (common CNN weights) required for performing the common CNN processing 601 from the RAM 205 to the buffer 104. The weighting coefficients may be generated in advance by learning and stored in the RAM 205, and then copied and used.

[0051] In step S703, the control unit 107 performs DMAC setting for transferring an input image used as input data in the common CNN processing 601 and features (common features) output as output data by the common CNN processing 601 between the RAM 205 and the buffer 105.

[0052] The input image may be directly acquired from the image input unit 202, or may be one stored in the RAM 205. The number of feature planes of the common feature is determined in advance depending on the complexity and accuracy of the task required for object detection. It is preferable that the number of feature planes is greater than the total number of feature planes generated based on the common feature (feature planes indicating the score, region width, and region height by the face detection CNN, and feature planes indicating the likelihood of each organ (eyes, nose, mouth) generated by the face organ detection CNN).

[0053] In step S704, the common feature generator executes the common CNN process 601 and stores the result of the common CNN process 601 in the RAM 205. More specifically, the DMAC 102 transfers the weighting coefficients stored in the RAM 205 to the buffer 104 according to the DMAC settings set in step S702. Furthermore, the DMAC 102 transfers the input image stored in the RAM 205 to the buffer 105 according to the DMAC settings set in step S703. Then, the common CNN is configured by setting the "weighting coefficients stored in the buffer 104" to the CNN, and the "input image stored in the buffer 105" is input to the common CNN to perform an operation of the common CNN (CNN operation) to obtain a common feature. More specifically, the operation processing unit 103 performs a convolution operation using the weighting coefficients stored in the buffer 104 and the input image stored in the buffer 105. The conversion processing unit 106 performs a nonlinear conversion on the result of the convolution operation, and stores the common feature, which is the result of the nonlinear conversion, in the buffer 105. The DMAC 102 transfers the common features stored in the buffer 105 to the RAM 205 in accordance with the DMAC settings in step S703.

[0054] In step S705, the control unit 107 performs DMAC setting for transferring weighting coefficients for face detection CNN (face detection CNN weights) from the RAM 205 to the buffer 104. In step S706, the control unit 107 performs DMAC setting for transferring common features used as input data in the face detection CNN and face detection features output as output data by the face detection CNN between the RAM 205 and the buffer 105.

[0055] In step S707, the first detector executes the face detection CNN process 602 to obtain face detection features, and stores the obtained face detection features in the RAM 205. More specifically, the DMAC 102 transfers weighting coefficients for face detection CNN from the RAM 205 to the buffer 104 according to the DMAC settings set in step S705. Furthermore, the DMAC 102 transfers the common features stored in the RAM 205 to the buffer 105 according to the DMAC settings set in step S706. Then, a face detection CNN is configured by setting "weighting coefficients for face detection CNN stored in the buffer 104" in a CNN (having the same structure as the CNN used in step S704), and the face detection features are obtained by inputting "common features stored in the buffer 105" to the face detection CNN and performing an operation (CNN operation) of the face detection CNN. More specifically, the operation processing unit 103 performs a convolution operation using the weighting coefficients stored in the buffer 104 and the common features stored in the buffer 105. The conversion processing unit 106 performs nonlinear conversion on the result of the convolution operation, and stores the face detection features resulting from the nonlinear conversion in the buffer 105. The DMAC 102 transfers the face detection features stored in the buffer 105 to the RAM 205 in accordance with the DMAC setting in step S706.

[0056] Then, the CPU 203 executes face detection post-processing 603 and direction information calculation 604 using the face detection features stored in the RAM 205. The face detection post-processing 603 and direction information calculation 604 will be described with reference to FIG.

[0057] As shown in Fig. 8, the face detection features include a score map 801, a face region width map 802, a face region height map 803, and a face direction map 804. The value at each position on the score map 801 indicates the "possibility of being a face" at that position, and the value indicates the likelihood or reliability, which becomes larger as the possibility of being a face increases. Therefore, a position on the score map 801 with a value larger than a predetermined threshold can be determined as the position of a face. In the example of Fig. 8, the position with a value of 255 can be determined as the center position of the face.

[0058] The value at each position of the face region width map 802 indicates the "estimated width of the face region" at that position. Therefore, the value of the position in the face region width map 802 that corresponds to the center position of the face identified from the score map 801 can be obtained as the "width of the face region."

[0059] The value at each position on the face region height map 803 indicates the "estimated height of the face region" at that position. Therefore, the value of the position in the face region height map 803 that corresponds to the center position of the face identified from the score map 801 can be obtained as the "height of the face region."

[0060] The value at each position of the face direction map 804 indicates the "estimated face direction" at that position. Therefore, the value of the position in the face direction map 804 that corresponds to the center position of the face identified from the score map 801 can be acquired as the "face direction." The "face direction" is learned to have a range of possible values ​​from -128 to 127, and indicates an angle multiplied by π / 128 clockwise with respect to the height direction of the image.

[0061] That is, in the face detection post-processing 603, the "face position" is obtained from the score map 801 as described above, the "face area width" is obtained from the face area width map 802 as described above, the "face area height" is obtained from the face area height map 803 as described above, and the "face orientation" is obtained from the face orientation map 804 as described above.

[0062] In the example of FIG. 8, the position in score map 801 where the value is “255” (the position on the left) is identified as the position of face 1, and the position in score map 801 where the value is “255” (the position on the right) is identified as the position of face 2.

[0063] Then, the value "10" at the position corresponding to the position of face 1 in the face area width map 802 is identified as the width of the area of ​​face 1, and the value "4" at the position corresponding to the position of face 2 in the face area width map 802 is identified as the width of the area of ​​face 2.

[0064] Then, the position value "5" corresponding to the position of face 1 in the face area height map 803 is identified as the height of the area of ​​face 1, and the position value "8" corresponding to the position of face 2 in the face area height map 803 is identified as the height of the area of ​​face 2.

[0065] In the face direction map 804, the position value "-51" corresponding to the position of face 1 is identified as the direction of face 1 (angle is -51xπ / 128). On the other hand, the position value "13" corresponding to the position of face 2 in the face direction map 804 is identified as the direction of face 2 (angle is 13xπ / 128).

[0066] Then, in the direction information calculation 604, a composite angle is generated from the orientation of each face acquired in the face detection post-processing 603. The method of generating the composite angle is not limited to a specific method. For example, the average value of the angles of each face acquired in the face detection post-processing 603 may be generated as the composite angle, or the angle of the face closest to the center of the image may be used as the composite angle. Also, a weighted average value obtained by weighting the angles of each face acquired in the face detection post-processing 603 (a face angle closer to the center of the image is weighted with a larger weight value) may be generated as the composite angle. Also, the magnitude of the score of each face may be used as the weight value. Note that when there is only one orientation of the face acquired in the face detection post-processing 603, the angle according to the one orientation is used as the composite angle.

[0067] Then, "directional information for determining a weighting factor to be used by the second detector" is specified from the composite angle. In this embodiment, four types of weighting factors are generated in advance by learning as candidates for the weighting factor to be used by the second detector, and stored in the RAM 205. Then, the weighting factor to be used by the second detector is selected from the four types of weighting factors based on the directional information.

[0068] In this embodiment, if the composite angle is within the angle range of -32π / 128 to 31π / 128, directional information 0 is selected as the directional information, and if the composite angle is within the angle range of 32π / 128 to 95π / 128, directional information 1 is selected as the directional information. If the composite angle is within the angle range of 96π / 128 to 127π / 128 or -128π / 128 to -97π / 128, directional information 2 is selected as the directional information. If the composite angle is within the angle range of -96π / 128 to -33π / 128, directional information 3 is selected as the directional information.

[0069] In the example of FIG. 8, when the average value of -19π / 128, which is the angle of face 1 -51π / 128 and the angle of face 2 13π / 128, is obtained as the composite angle, directional information 0 is selected as the corresponding directional information.

[0070] In step S708, the control unit 107 performs DMAC setting to transfer the weighting coefficient (face organ CNN weighting) corresponding to the selected directional information (directional information 0 in the example of Figure 8) selected in the directional information calculation 604 in step S707 from the "weighting coefficients corresponding to each of directional information 0 to directional information 3" stored in the RAM 205 to the buffer 104.

[0071] 9, the RAM 205 stores CNN weight data 901 for common CNN, which is a weighting coefficient for the common CNN, and CNN weight data 902 for face detection CNN, which is a weighting coefficient for the face detection CNN. The RAM 205 also stores CNN weight data corresponding to direction information 0 903, which is a weighting coefficient corresponding to direction information 0, and CNN weight data corresponding to direction information 1 904, which is a weighting coefficient corresponding to direction information 1. The RAM 205 also stores CNN weight data corresponding to direction information 2 905, which is a weighting coefficient corresponding to direction information 2, and CNN weight data corresponding to direction information 3 906, which is a weighting coefficient corresponding to direction information 3.

[0072] In step S709, the control unit 107 performs DMAC setting for transferring common features used as input data in the face organ detection CNN and face organ features output as output data by the face organ detection CNN between the RAM 205 and the buffer 105.

[0073] In step S710, the second detector executes face organ detection CNN processing 605 to obtain face organ features, and then executes face organ detection post-processing 606 to detect face organs from the face organ features. More specifically, DMAC 102 transfers weighting coefficients (weighting coefficients for face organ detection CNN) corresponding to the selected direction information selected in direction information calculation 604 from RAM 205 to buffer 104 according to the DMAC settings set in step S708. Furthermore, DMAC 102 transfers common features stored in RAM 205 to buffer 105 according to the DMAC settings set in step S709. Then, a face organ detection CNN is constructed by setting "weighting coefficients stored in buffer 104" to a CNN (with the same structure as the CNN used in step S704), and "common features stored in buffer 105" are input to the face organ detection CNN to perform calculations (CNN calculations) of the face organ detection CNN to obtain face organ features. More specifically, the calculation processing unit 103 performs a convolution operation using the weighting coefficients stored in the buffer 104 and the common features stored in the buffer 105. The transformation processing unit 106 performs a nonlinear transformation on the result of the convolution operation, and stores the facial organ features resulting from the nonlinear transformation in the buffer 105. The DMAC 102 transfers the facial organ features stored in the buffer 105 to the RAM 205 in accordance with the DMAC setting in step S709.

[0074] Then, the CPU 203 executes face organ detection post-processing 606 using the face organ features stored in the RAM 205. The face organ detection post-processing 606 will be described with reference to FIG.

[0075] As shown in FIG. 10, the facial organ features include a right eye likelihood map 1001, a left eye likelihood map 1002, a nose likelihood map 1003, and a mouth likelihood map 1004.

[0076] The value at each position of the right eye likelihood map 1001 indicates the "likelihood of being the right eye" at that position, and the value indicates the likelihood or reliability, which becomes larger as the possibility of being the right eye increases. Therefore, a position in the right eye likelihood map 1001 with a value larger than a predetermined threshold can be determined as the position of the right eye. In the example of Fig. 10, the position with a value of 255 in the right eye likelihood map 1001 can be determined as the center position of the right eye.

[0077] The value at each position of the left eye likelihood map 1002 indicates the "likelihood of being the left eye" at that position, and the value indicates the likelihood or reliability, which becomes larger as the possibility of being the left eye increases. Therefore, a position in the left eye likelihood map 1002 having a value greater than a predetermined threshold can be determined as the position of the left eye. In the example of Fig. 10, the position in the left eye likelihood map 1002 having a value of 255 can be determined as the center position of the left eye.

[0078] The value at each position of the nose likelihood map 1003 indicates the "likelihood of being a nose" at that position, and the value indicates the likelihood or reliability, which becomes larger as the possibility of being a nose increases. Therefore, a position in the nose likelihood map 1003 having a value larger than a predetermined threshold can be determined as the position of the nose. In the example of Fig. 10, the position of the value 255 in the nose likelihood map 1003 can be determined as the center position of the nose.

[0079] The value at each position on the mouth likelihood map 1004 indicates the "likelihood of being a mouth" at that position, and the value indicates the likelihood or reliability, which becomes larger as the possibility of being a mouth increases. Therefore, a position on the mouth likelihood map 1004 with a value larger than a predetermined threshold value can be determined as the position of the mouth. In the example of Fig. 10, the position on the mouth likelihood map 1004 with a value of 255 can be determined as the center position of the mouth.

[0080] That is, in the face organ detection post-processing 606, the position of the right eye is obtained from the right eye likelihood map 1001 as described above, the position of the left eye is obtained from the left eye likelihood map 1002 as described above, the position of the nose is obtained from the nose likelihood map 1003 as described above, and the position of the mouth is obtained from the mouth likelihood map 1004 as described above.

[0081] In the face organ detection of this embodiment, one position is detected for each of the right eye, left eye, nose, and mouth, but multiple positions for each organ (for example, inner corner, outer corner, lower eyelid, upper eyelid, pupil, etc.) may be detected. The detection method is to train the face organ detection CNN so as to output a feature map corresponding to each part, and similarly to the above explanation, the position of each feature map where the value exceeds a predetermined threshold can be determined as the detection position of each organ.

[0082] In addition, it is preferable to output each detection result in association with each other. For example, in the case of association of face organ detection results, the position of the nose, which is approximately near the center of the face, and the positions of the right eye, left eye, and mouth, which are near the position of the nose, are output in association with each other, so that the positions of these organs can be output as the positions of organs constituting the same face. In addition, in the case of association of face organ detection results and face detection results, the center position of an object near the position of the nose can be output as a common face.

[0083] In this manner, in this embodiment, face detection and face organ detection can be performed from the common features generated by the common feature generator. Also, in this embodiment, an example has been described in which, when performing face organ detection, directional information is obtained from the face detection result, and a weighting factor according to this directional information can be used.

[0084] Next, the learning methods of the common CNN process 601, the face detection CNN process 602, and the face organ detection CNN process 605 (that is, the learning methods of the CNNs that execute each CNN process) will be described. In this embodiment, learning is performed by a method called multitask learning. Multitask learning is a method of simultaneously learning multiple tasks that share a feature surface. In this embodiment, the tasks of face detection and face organ detection are learned simultaneously. The learning of the common CNN process 601, the face detection CNN process 602, and the face organ detection CNN process 605 may be performed by an information processing device, or may be performed by another device different from the information processing device. In the latter case, the weight coefficients that are the results of learning are stored in the RAM 205 of the information processing device.

[0085] An example of the configuration of the learning process of the common CNN process 601, the face detection CNN process 602, and the face organ detection CNN process 605 will be described with reference to the block diagram of Fig. 11. In this embodiment, the learning targets are the common CNN process 601, the face detection CNN process 602, and the face organ detection CNN process 605. By learning the common CNN process 601, it is possible to obtain the weighting coefficient of the CNN that executes the common CNN process 601, and by learning the face detection CNN process 602, it is possible to obtain the weighting coefficient of the CNN that executes the face detection CNN process 602. Also, by learning the face organ detection CNN process 605, it is possible to obtain the weighting coefficient of the CNN that executes the face organ detection CNN process 605.

[0086] In learning, a learning data set is input to the CNN to perform calculations on the CNN, the results of the calculations are compared with a teacher data set corresponding to the learning data set that has been prepared in advance, and the weighting coefficients of the CNN are updated based on the results of the comparison (CNN weight update 1102).

[0087] The learning datasets include a learning dataset 1191 corresponding to directional information 0, a learning dataset 1192 corresponding to directional information 1, a learning dataset 1193 corresponding to directional information 2, and a learning dataset 1194 corresponding to directional information 3. The pre-prepared teacher datasets corresponding to the learning datasets include a teacher dataset 1181 corresponding to directional information 0, a teacher dataset 1182 corresponding to directional information 1, a teacher dataset 1183 corresponding to directional information 2, and a teacher dataset 1184 corresponding to directional information 3.

[0088] As described above, the direction information is calculated based on the direction of the face to be detected. For example, the learning data set 1191 for direction information 0 is a group of images including many faces with face directions of -32π / 128 to 31π / 128. The teacher data set 1181 for direction information 0 is an ideal feature map group output when face detection and face organ detection are performed using the learning data set 1191 for direction information 0. Note that even the data set (learning data set, teacher data set) for direction information 0 does not need to include only face directions of -32π / 128 to 31π / 128, and may include other face directions. This makes it possible to obtain a highly robust detection result even if there is some variation in the input image. The same explanation applies to the learning data set and teacher data set for each of direction information 1, 2, and 3.

[0089] In the CNN weight update 1102, the error backpropagation method is generally used, and various methods may be used for efficient learning. The learning sequence 1101 controls the entire learning.

[0090] The learning method according to the learning sequence 1101 will be described with reference to the flowchart in Fig. 12. In step S1201, various data sets are divided into batches consisting of multiple data. The division is performed in units of batches so that the learning data set and the teacher data set correspond to each other. In this embodiment, this batch is the unit for switching between learning for each directional information.

[0091] In step S1202, the value of a variable D indicating directional information is initialized to 0. In step S1203, a batch to be used for learning is selected. In step S1204, branching is performed depending on the directional information.

[0092] The processes in steps S1205 to S1208 prepare for each learning depending on the value of variable D. Specifically, the batch selected in step S1203 is set to be used in learning, and the weighting coefficient for the face organ detection CNN process 605 is switched depending on the value of variable D. At this time, regardless of the value of variable D, the weighting coefficients of the common CNN process 601 and the face detection CNN process 602 are not switched, and learning of the common CNN process 601 and the face detection CNN process 602 proceeds without depending on the value of variable D.

[0093] That is, if the value of the variable D is 0, the process proceeds to step S1205 via step S1204. In step S1205, the batch corresponding to the direction information 0 is set to be used in learning, and the weighting coefficient corresponding to the direction information 0 is set as the weighting coefficient for the face part detection CNN process 605.

[0094] If the value of the variable D is 1, the process proceeds to step S1206 via step S1204. In step S1206, the batch corresponding to the direction information 1 is set to be used in learning, and the weighting coefficient corresponding to the direction information 1 is set as the weighting coefficient for the face organ detection CNN process 605.

[0095] If the value of the variable D is 2, the process proceeds to step S1207 via step S1204. In step S1207, the batch corresponding to the direction information 2 is set to be used in learning, and the weighting coefficient corresponding to the direction information 2 is set as the weighting coefficient for the face organ detection CNN process 605.

[0096] If the value of the variable D is 3, the process proceeds to step S1208 via step S1204. In step S1208, the batch corresponding to the direction information 3 is set to be used in learning, and the weighting coefficient corresponding to the direction information 3 is set as the weighting coefficient for the face organ detection CNN process 605.

[0097] In step S1209, learning is performed using the set batch by using the CNN with the set weighting coefficients. Learning is performed by performing face detection and face organ detection using images included in the batch, and feeding back the difference between the resulting feature map and the correct feature map to the weighting coefficients.

[0098] In step S1210, the value of variable D is incremented by 1, and the incremented value is divided by 4, and the resulting value is set as the value of variable D. In step S1211, it is determined whether the value of variable D is 0. A value of variable D of 0 indicates that learning has been performed for all of the direction information 0 to 3. If the value of variable D is 0, the process proceeds to step S1212, and if the value of variable D is not 0, the process proceeds to step S1204.

[0099] In step S1212, it is determined whether learning has been completed for all batches. If learning has been completed for all batches, the process proceeds to step S1213, and if there are any batches for which learning has not been completed, the process proceeds to step S1203.

[0100] In step S1213, it is determined whether or not a learning end condition has been satisfied. The learning end condition is not limited to a specific condition, and may be, for example, that the number of learning times is equal to or greater than a threshold, that the elapsed time since the start of learning is equal to or greater than a threshold, that the difference is less than a threshold, or that the amount of change in the difference is less than a threshold. If the learning end condition has been satisfied, the process according to the flowchart in FIG. 12 ends, and if the learning end condition has not been satisfied, the process proceeds to step S1202.

[0101] By using such a learning method, weighting coefficients capable of detecting faces and facial organs can be acquired for the common CNN processing 601, the face detection CNN processing 602, and the face organ detection CNN processing 605.

[0102] In the CNN process, it is considered that the local features of the object in the image are automatically extracted and hierarchically organized to detect the object. Therefore, the CNN process for detecting the face generates parts such as facial organs that constitute the face as intermediate features by learning. This embodiment utilizes this phenomenon, and proceeds with learning so that the features of the parts of the detected object generated during the detection task become common features output by the common CNN. As a result, when performing the face detection task and the face organ detection task, the calculations for calculating the common features can be combined at once, and the amount of calculations can be reduced. In addition, it has been found that the same effect can be obtained even if the learning of the face organ detection CNN and the learning of the common CNN and face detection CNN are individually learned using different data sets, instead of the multitask learning as in this embodiment.

[0103] In addition, the face organ detection CNN can use weighting coefficients that have been trained in advance using a database that matches the directional information obtained from the face detection results. The directional information obtained from the face direction roughly matches the direction of the face organs contained in the face, and the position, size, and shape of each face organ within the face are almost fixed. This makes it possible to limit the variations in the input that should be detected by the face organ detection CNN. In this way, the common CNN and face detection CNN are trained with a robust dictionary that does not depend on the object direction, etc., while the face organ detection CNN, which performs more detailed detection, has a lower robustness and is trained with a dictionary for each variation, making it possible to achieve the minimum necessary network configuration.

[0104] In this embodiment, the face direction is described as a two-dimensional direction in the image plane, but in three-dimensional space, the face direction has a roll angle (in-plane rotation), pitch angle, and yaw angle, which can be used for detection and weighting coefficient selection. In this case, it is necessary to learn weighting coefficients according to the roll angle, pitch angle, and yaw angle.

[0105] In addition, in this embodiment, an example is shown in which the face direction is divided into four directions every 90°, but it may be divided into eight directions every 45°, for example, and the number of direction divisions is not limited to a specific number.

[0106] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified, it is assumed that the present embodiment is the same as the first embodiment. In this embodiment, an information processing device that operates as an object detector for detecting the types of members piled up on a belt conveyor and the portions of the members that can be grasped by a robot arm (parts area) will be described.

[0107] FIG. 13 is a diagram for explaining the outline of this function. In this embodiment, as shown in FIG. 13(a), it is assumed that the parts that flow in a pile on the belt conveyor are grasped and sorted by a robot arm. When a part is included in the photographing area, an image is taken and the type of the part and the part area that can be grasped are detected. An image after detection is shown in FIG. 13(b), in which the type (part A, B, C) and the orientation of each part are identified by the first detector, and the part area that can be grasped (shown by the shaded area in the figure) is detected by the second detector.

[0108] In this embodiment, the CNN used in the first detector is trained to be able to detect the type and orientation of the component, and the post-detection process also detects them. The CNN used in the second detector is trained to be able to detect grippable part regions in the component, and the post-detection process also detects grippable part regions. Furthermore, the common feature generator outputs feature surfaces that indicate the possibility of the presence of parts that make up the component, and is trained to enable the first detector to identify the type and orientation of the component, and the second detector to detect the part regions.

[0109] The detection process according to this embodiment will be described with reference to the flowchart in Fig. 14. In Fig. 14, the same process steps as those shown in Fig. 7 are assigned the same step numbers, and the description of those process steps will be omitted.

[0110] In step S1401, the control unit 107 performs DMAC setting for transferring weighting coefficients for the component detection CNN from the RAM 205 to the buffer 104. In step S1402, the control unit 107 performs DMAC setting for transferring common features used as input data in the component detection CNN and component detection features output as output data by the component detection CNN between the RAM 205 and the buffer 105.

[0111] In step S707, the first detector executes the component detection CNN process to detect components as shown in FIG. 13(b) and detect the number and orientation of components in the image. More specifically, the DMAC 102 transfers weighting coefficients for component detection CNN from the RAM 205 to the buffer 104 according to the DMAC settings set in step S1401. Furthermore, the DMAC 102 transfers the common features stored in the RAM 205 to the buffer 105 according to the DMAC settings set in step S1402. Then, a component detection CNN is configured by setting the “weighting coefficients for component detection CNN stored in the buffer 104” to a CNN (with the same structure as the CNN used in step S704), and the “common features stored in the buffer 105” are input to the component detection CNN to perform a calculation (CNN calculation) of the component detection CNN to obtain the component detection features. More specifically, the calculation processing unit 103 performs a convolution calculation using the weighting coefficients stored in the buffer 104 and the common features stored in the buffer 105. The conversion processing unit 106 performs nonlinear conversion on the result of the convolution operation, and stores the component detection features resulting from the nonlinear conversion in the buffer 105. In accordance with the DMAC setting in step S1402, the DMAC 102 transfers the component detection features stored in the buffer 105 to the RAM 205. The component detection features include maps (score map, component area width map, component area height map, component orientation map) in which the target in the face detection features is a component rather than a face.

[0112] Then, the CPU 203 uses the component detection features stored in the RAM 205 to obtain the "position of the component," "width of the component area," "height of the component area," and "orientation of the component" in the same manner as in the first embodiment.

[0113] In step S1403, control unit 107 determines whether component region detection has been performed for all components detected by the first detector. If the result of this determination is that component region detection has been performed for all components detected by the first detector, the process according to the flowchart in Fig. 14 ends. On the other hand, if there are still components detected by the first detector that have not yet been subjected to component region detection, the process proceeds to step S1404.

[0114] In step S1404, the control unit 107 sets a component that has not yet undergone component area detection among the components detected by the first detector as a target component. The control unit 107 then performs DMAC setting to transfer weight coefficients for component area detection CNN for identifying a component area for the target component from the RAM 205 to the buffer 104. The orientation of each component is detected by the first detector. The weight coefficients for component area detection CNN are learned from a plurality of learning data sets that roughly classify the orientations of components, and once the orientation of a component is determined, an appropriate weight coefficient can be selected. In other words, the control unit 107 performs DMAC setting to transfer weight coefficients corresponding to the orientation of the target component from the RAM 205 to the buffer 104 as weight coefficients for component area detection CNN for identifying the component area of ​​the target component.

[0115] In step S1405, DMAC settings are performed to transfer the common features used as input data in the component area detection CNN and the component area features output as output data by the component area detection CNN between the RAM 205 and the buffer 105. Since the positions of the components have been identified, it is also possible to reduce the amount of calculation by limiting the size for calculating the common features and component area features from the image to the periphery of the positions of the components.

[0116] In step S1406, the second detector detects the part area of ​​each component. More specifically, the DMAC 102 transfers the weighting coefficient corresponding to the direction of the target component from the RAM 205 to the buffer 104 according to the DMAC setting set in step S1404. Furthermore, the DMAC 102 transfers the common feature stored in the RAM 205 to the buffer 105 according to the DMAC setting set in step S1405. Then, a part area detection CNN is configured by setting the "weighting coefficient stored in the buffer 104" to a CNN (having the same structure as the CNN used in step S704), and the "common feature stored in the buffer 105" is input to the part area detection CNN to perform an operation (CNN operation) of the part area detection CNN to obtain the part area feature. More specifically, the operation processing unit 103 performs a convolution operation using the weighting coefficient stored in the buffer 104 and the common feature stored in the buffer 105. The conversion processing unit 106 performs a nonlinear conversion on the result of the convolution operation, and stores the part area feature resulting from the nonlinear conversion in the buffer 105. DMAC 102 transfers the component area features stored in buffer 105 to RAM 205 in accordance with the DMAC settings in step S1405. The component area features include maps (score map, component area width map, component area height map, and component orientation map) in which the target of face detection features is a component rather than a face. CPU 203 then uses the component area features stored in RAM 205 to detect the component area in the same manner as in the first embodiment.

[0117] Thus, according to this embodiment, the first detector detects the type and orientation of the component from the common features output from the common feature generator, and an appropriate weighting coefficient is selected from the orientation of the component to detect the graspable component area of ​​the component. When many components are photographed in one image, efficient detection is possible by determining the area in which the components exist and detecting the areas of the parts that make up the component with the area at the center. Furthermore, since the common features are features that are commonly required when detecting a component and when detecting the parts that make up the component, sharing the common features can reduce the processing load of the entire task.

[0118] In the first and second embodiments, an example of recognition processing using CNN has been described, but this is not limiting, and various detection algorithms can be applied. In that case, the algorithm for acquiring common features and the algorithm for processing the detection task may be different methods. In other words, the weighting coefficient is merely one example of a parameter in the detection algorithm. Furthermore, the weighting coefficient selected according to the direction information is also merely one example of a selection parameter selected from a plurality of parameters based on the direction of the detected object. Furthermore, a face or a member is merely one example of an object to be detected, and a facial organ or part area is merely one example of a part of an object.

[0119] In the first and second embodiments, the convolution calculation and coefficient rotation processing are performed by hardware. However, the convolution calculation and coefficient rotation processing may be performed by a processor such as a CPU / GPU (Graphics Processing Unit) / DSP (Digital Signal Processing Unit).

[0120] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0121] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0122] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0123] The invention of this specification includes the following information processing device, information processing method, and computer program. (Item 1) a first detection means for detecting an object from an image including the object using a feature of the image; a second detection means for detecting a part of the object from the image based on a selection parameter selected from a plurality of parameters based on a direction of the object detected by the first detection means and the feature; An information processing device comprising: (Item 2) The information processing device according to item 1, characterized in that the second detection means generates a composite angle by combining angles of the directions of the respective objects detected by the first detection means, and selects, as the selection parameter, a parameter corresponding to an angle range to which the composite angle belongs from a plurality of parameters corresponding to a plurality of angle ranges. (Item 3) 3. The information processing device according to item 2, wherein the second detection means determines, as the composite angle, an average value of angles of the directions of the objects detected by the first detection means. (Item 4) 3. The information processing device according to item 2, wherein the second detection means sets, as the composite angle, an angle of a direction of an object that is closest to a center of the image among the objects detected by the first detection means. (Item 5) 3. The information processing device according to item 2, wherein the second detection means determines, as the composite angle, a weighted average value obtained by weighting the angles of the directions of the respective objects detected by the first detection means. (Item 6) The information processing device according to item 1, characterized in that, for each object detected by the first detection means, the second detection means detects a part of the object from the image based on a selection parameter selected from a plurality of parameters based on a direction of the object and the feature. (Item 7) The first detection means configures a first hierarchical neural network in which a weighting factor for object detection is set in the hierarchical neural network, and detects the object using the first hierarchical neural network; The second detection means configures a second hierarchical neural network by setting a weighting coefficient, which is the selection parameter, in the hierarchical neural network, and detects a part of the object using the second hierarchical neural network. 7. The information processing device according to any one of items 1 to 6, (Item 8) moreover, 8. The information processing device according to item 7, further comprising a means for configuring a third hierarchical neural network in which a weighting coefficient for detecting the feature is set in the hierarchical neural network, and detecting the feature using the third hierarchical neural network. (Item 9) moreover, 9. The information processing device according to any one of items 1 to 8, further comprising a means for outputting results of detection by the second detection means in association with each other. (Item 10) moreover, 10. The information processing device according to any one of items 1 to 9, further comprising an imaging means for capturing the image. (Item 11) An information processing method performed by an information processing device, a first detection step in which a first detection means of the information processing device detects an object from an image including the object using a feature of the image; a second detection step in which a second detection means of the information processing device detects a part of the object from the image based on a selection parameter selected from a plurality of parameters based on a direction of the object detected in the first detection step and the feature; An information processing method comprising: (Item 12) A computer program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 9.

[0124] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0125] 101: I / F section 102: DMAC 103: Arithmetic processing section 104: Buffer 105: Buffer 106: Conversion processing section 107: Control section

Claims

1. a first detection means for detecting an object from an image including the object using a feature of the image; a second detection means for detecting a part of the object from the image based on a selection parameter selected from a plurality of parameters based on a direction of the object detected by the first detection means and the feature; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the second detection means generates a composite angle by combining angles of directions of the objects detected by the first detection means, and selects, as the selection parameter, a parameter corresponding to an angle range to which the composite angle belongs from a plurality of parameters corresponding to a plurality of angle ranges.

3. 3. The information processing apparatus according to claim 2, wherein the second detection means determines, as the composite angle, an average value of angles of the directions of the objects detected by the first detection means.

4. 3. The information processing apparatus according to claim 2, wherein the second detection means determines, as the composite angle, an angle of a direction of an object that is closest to a center of the image among the objects detected by the first detection means.

5. 3. The information processing apparatus according to claim 2, wherein the second detection means determines, as the composite angle, a weighted average value obtained by weighting angles of the directions of the respective objects detected by the first detection means.

6. 2. The information processing device according to claim 1, wherein the second detection means detects, for each object detected by the first detection means, a portion of the object from the image based on a selection parameter selected from a plurality of parameters based on a direction of the object and the feature.

7. The first detection means configures a first hierarchical neural network in which a weighting factor for object detection is set in the hierarchical neural network, and detects the object using the first hierarchical neural network; The second detection means configures a second hierarchical neural network by setting a weighting factor, which is the selection parameter, in the hierarchical neural network, and detects a part of the object using the second hierarchical neural network.

2. The information processing apparatus according to claim 1,

8. moreover, 8. The information processing apparatus according to claim 7, further comprising: means for constructing a third hierarchical neural network in which weighting coefficients for detecting the feature are set in the hierarchical neural network, and for detecting the feature using the third hierarchical neural network.

9. moreover, 2. The information processing apparatus according to claim 1, further comprising: a means for outputting the results of detection by said second detection means in association with each other.

10. moreover, The information processing apparatus according to claim 1 , further comprising an image capturing unit for capturing the image.

11. An information processing method performed by an information processing device, a first detection step in which a first detection means of the information processing device detects an object from an image including the object using a feature of the image; a second detection step in which a second detection means of the information processing device detects a part of the object from the image based on a selection parameter selected from a plurality of parameters based on a direction of the object detected in the first detection step and the feature; An information processing method comprising:

12. A computer program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 9.

Images