Image processing device and image processing method

Abstract

To provide an image processing device and image processing method, which enable accurate estimation of a vector representing a relative position relationship between parts of an object using machine learning.SOLUTION: An image processing device disclosed herein comprises detection means configured to detect a first part and a second part of a specific object from an image and a relative position vector from the first part to the second part using a convolutional neural network. The convolutional neural network is configured to generate a feature map showing an approximate direction of the relative position vector. The image processing device trains the detection means using a loss function associated with the relative position vector and a loss function associated with the feature map.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a technique for detecting a subject. [Background technology]

[0002] To detect body postures with joints, it is known to use neural networks to estimate the positions of the joints from images and two-dimensional vector fields associated with the connections between the joints (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-510417 Summary of the Invention [Problem to be solved by the invention]

[0004] However, since estimating a two-dimensional vector field is a regression problem, it is not easy to properly train a neural network, and as a result, there is room for improvement in the accuracy of estimating a two-dimensional vector field.

[0005] In one aspect, the present invention provides an image processing device and an image processing method capable of accurately estimating a vector representing a relative positional relationship between parts of a subject using machine learning. [Means for solving the problem]

[0006] The above-mentioned object is achieved by an image processing device having a detection means for detecting a first part and a second part of a specific subject from an image and a relative position vector from the first part to the second part using a convolutional neural network, and a learning means for training the detection means, wherein the convolutional neural network is configured to generate a feature map indicating an approximate direction of the relative position vector, and the learning means trains the detection means using a loss function for the relative position vector and a loss function for the feature map. Effect of the Invention

[0007] According to the present invention, it is possible to provide an image processing device and an image processing method capable of accurately estimating a vector representing the relative positional relationship between parts of a subject using machine learning. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an image processing apparatus according to an embodiment; [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an image processing apparatus when performing a detection process according to an embodiment; [Diagram 3] Flowchart of detection process according to the embodiment [Figure 4] FIG. 1 is a diagram showing an example of a CNN constituting an estimation unit in an embodiment; [Diagram 5] FIG. 1 is a diagram for explaining a detection process according to an embodiment; [Figure 6] FIG. 1 is a block diagram showing an example of a functional configuration of an image processing device when executing a learning process according to an embodiment; [Figure 7] Flowchart of learning process according to the embodiment [Figure 8] FIG. 1 is a diagram for explaining a learning process according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing device 100 according to this embodiment. The CPU 101 uses the RAM 103 as a work memory and executes various processes described below by executing control programs stored in the ROM 102 and the storage unit 104. The storage unit 104 includes a storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores the programs (firmware, OS, applications, etc.) executed by the CPU 101 and various data.

[0011] The input unit 105 is a general term for input devices that the image processing device 100 has and that can be operated by a user, such as a keyboard, a mouse, buttons, switches, and a touch panel. The display unit 106 is a display that displays a screen (GUI) provided by the OS and applications being executed by the CPU 101. The display unit 106 may be a touch display.

[0012] The communication unit 107 is a communication interface that complies with one or more wired and / or wireless communication standards. The image processing device 100 can communicate with external devices such as an image capture device through the communication unit 107.

[0013] In FIG. 1, for ease of explanation and understanding, it is assumed that the CPU 101 executes a program to control all the processing of the image processing device 100. However, for at least a part of the image processing described later, the CPU 101 may execute the processing using other hardware. For example, the CPU 101 may use hardware capable of executing the processing faster than the CPU 101 depending on the content of the processing. Such hardware includes ASIC, DSP (Digital Signal Processor), GPU (Graphic Processing Unit), NPU (Neural Processing Unit), etc. For example, for the processing using a neural network described later, the NPU may execute the processing instead of the CPU 101, and the processing result may be notified to the CPU 101.

[0014] The image processing device 100 can be implemented by a computer device such as a personal computer, a tablet, a smartphone, etc. The image processing device 100 may also have an imaging function.

[0015] <Explanation of subject detection process> First, the subject detection process using learned motion parameters will be described, and then the learning process of the motion parameters used in the subject detection process will be described.

[0016] In the detection process, multiple parts of a specific subject are detected from an input image using learned motion parameters, and parts of the same subject are associated with each other. In this example, a human is detected as the specific subject, and the head and torso are detected as the multiple parts. However, the specific subject is not limited to a human, and may be other types of subjects such as animals. Furthermore, the specific subject is not limited to a living thing, and may be any type of subject whose parts can be detected individually.

[0017] 2 is a block diagram showing the image processing device 100 that executes the subject detection process, expressed by functional blocks. In the following, for ease of explanation and understanding, it is assumed that each functional block is realized by the CPU 101 executing a program. However, one or more functional blocks may be realized by hardware other than the CPU 101. Note that one functional block may be realized by multiple pieces of hardware, or one piece of hardware may realize multiple functional blocks.

[0018] The image acquisition unit 201 acquires an image in which a specific subject is captured. The image acquisition unit 201 may acquire an image captured by an imaging device connected to the image processing device, or may acquire an image stored in the storage unit 104.

[0019] The estimation unit 202 applies processing using the motion parameters acquired by the parameter acquisition unit 205 to the image acquired by the image acquisition unit 201, and generates various feature maps. The estimation unit 202 generates, as feature maps, a head likelihood map indicating the head position, a body likelihood map indicating the body position, a part-to-part vector map, and a part-to-part direction likelihood map for a specific subject in the image. Each feature map will be described later.

[0020] In this embodiment, the estimation unit 202 applies processing to an image using machine learning, specifically, a convolutional neural network (CNN), to generate various feature maps. The operation parameters acquired by the parameter acquisition unit 205 are the connection weight coefficients between layers of the CNN used by the estimation unit 202. Therefore, the estimation unit 202 executes various processes using machine learning by setting the operation parameters in the CNN and using them.

[0021] The detection unit 203 uses a feature map generated by a hidden layer (convolutional layer, pooling layer, etc.) of the CNN included in the estimation unit 202 to detect the body position, head position, and relative position vector from the body to the head of the specific subject.

[0022] A connection unit 204 uses the detection result of the detection unit 203 to associate the torso and head of the same specific subject.

[0023] The parameter acquisition unit 205 reads out the learned motion parameters from the storage unit 104 and provides them to the estimation unit 202 .

[0024] Next, the subject detection operation of the image processing device 100 will be described with reference to the flowchart shown in Fig. 3 and the block diagram shown in Fig. 2. The steps executed by each functional block shown in Fig. 2 are executed by hardware (CPU 101 or other hardware) that realizes the functional block. This operation may be executed in response to a user instruction via the input unit 105, for example, or in response to any other event. In addition, it may be executed as part of a function provided by an image processing application running on the image processing device 100, for example.

[0025] The subject detection process in this embodiment is a process of detecting two or more parts of a predetermined subject to be detected (specific subject) and then associating the detected parts with each subject.

[0026] In S301, the image acquisition unit 201 acquires one frame of an input image (which may be a still image or a video). Here, it is assumed that an image in which horses 501 and 502, which are multiple specific subjects, overlap each other, as shown in Fig. 5(a) has been acquired.

[0027] In S302, the estimation unit 202 inputs an input image to the CNN and generates various feature maps (head likelihood map, torso likelihood map, inter-part vector map, inter-part direction likelihood map) in the hidden layer. Learned motion parameters (inter-layer connection weight coefficients) are set in the CNN from the parameter acquisition unit 205.

[0028] FIG. 4 is a diagram schematically showing the configuration of the CNN used by the inference unit 202 and the operation of generating each feature map. In FIG. 4, "Conv" indicates the processing by the convolutional layer. The feature map of the i-th layer (2 ≤ i ≤ N) is generated by a convolutional operation that takes the feature map of the (i - 1)-th layer as input. The convolutional operation on the input image may be, for example, an operation of applying a spatial filter (also called a kernel) while shifting the position with respect to the input image. Note that the processing by the convolutional layer may include, in addition to the convolutional operation, applications of pooling and activation functions.

[0029] Among the predetermined four channels of the feature map of the N-th layer, which is the final layer, they are output as the head likelihood map, the body likelihood map, and the inter-part vector maps (two channels in the X direction and the Y direction), respectively. Also, among the predetermined four channels of the feature map of the K-th layer (1 < K < N), which is an intermediate layer, they are output as the inter-part direction-specific likelihood maps. Since the generation of the inter-part direction-specific likelihood maps corresponds to a classification process for roughly estimating the direction (angle) of the relative vector from the body to the head (here, in units of π / 2), it is less difficult than the regression process for estimating the relative vector itself. However, by obtaining the rough direction of the relative position vector, the accuracy of associating the body and the head of the same subject can be improved.

[0030] When the operation parameters (the coupling weight coefficients between the layers of the CNN) are appropriately learned by the learning process described later, the head likelihood map has a peak of likelihood at the coordinates corresponding to the head position of the specific subject shown in the input image. Similarly, the body likelihood map has a peak of likelihood at the coordinates corresponding to the body position of the specific subject shown in the input image.

[0031] The inter-part vector map is a two-channel map composed of the maps of the X-direction component and the Y-direction component of the relative position vector from the body to the head. The map of the X-direction component has the value of the X direction of the relative position vector from the body to the head of the same subject at the coordinates corresponding to the body position of the specific subject. The map of the Y-direction component has the value of the Y direction of the relative position vector from the body to the head of the same subject at the coordinates corresponding to the body position of the specific subject.

[0032] The likelihood map for each direction between parts is a four-channel map. The map for the kth channel is the angle θ i (k-1)π / 2≦θ ii The inter-part direction likelihood map is a feature map that indicates, for each torso, the approximate direction in which the corresponding head exists (the approximate direction of the relative position vector) in an angle range having a predetermined size.

[0033] An example of the inter-part direction likelihood map generated by the estimation unit 202 for the input image of Fig. 5(a) is shown in Fig. 5(c). A likelihood peak exists for horse 501 in the map of the first channel, and a likelihood peak exists for horse 502 in the map of the third channel. Therefore, the inter-part direction likelihood map shown in Fig. 5(c) indicates that there is a high probability that the angle θ between the relative position vector from the torso to the head is 0° to 90° for horse 501, and 180° to 270° for horse 502. The estimation unit 202 outputs each of the generated maps to the detection unit 203.

[0034] In S303, the detection unit 203 detects the head position of the specific subject from the head likelihood map among the maps generated by the estimation unit 202. The detection unit 203 detects, for example, coordinates corresponding to a likelihood peak in the head likelihood map as the head position.

[0035] Specifically, the detection unit 203 detects a likelihood peak in the head likelihood map. For example, the detection unit 203 detects, as a candidate pixel, a pixel whose value (likelihood) is greater than all surrounding pixels among the pixels constituting the map. The surrounding pixels may be, for example, pixels adjacent vertically, horizontally, and diagonally. The detection unit 203 then detects, as a peak pixel, a pixel having a value (likelihood) exceeding a predetermined threshold among the detected candidate pixels. The detection unit 203 sets the image coordinates of the peak pixel as the head position of the specific subject. If the movement parameters have been properly learned, the head likelihood map generated from the input image shown in FIG. 5(a) has pixels at the head positions of the horses 501 and 502 that have values ​​(likelihood) exceeding the threshold.

[0036] Furthermore, the detection unit 203 detects the body position of the specific subject from the body likelihood map in the same manner as in detecting the head position.

[0037] Furthermore, detection unit 203 acquires a relative position vector to the head of the same subject for each detected torso from the inter-part vector map (X direction and Y direction). If the motion parameters have been properly learned, in the inter-part vector map, the pixel values ​​of the coordinates corresponding to the torso position of the subject become the X direction component and the Y direction component of the relative position vector from the torso to the head of the same subject.

[0038] Therefore, the detection unit 203 obtains pixel values ​​corresponding to the coordinates of each of the torso positions detected using the torso likelihood map for the inter-part vector map of each channel. Then, the detection unit 203 determines a vector having the two pixel values ​​as an X-direction component and a Y-direction component as a relative position vector for the head of the same subject corresponding to the detected torso.

[0039] The detected i-th body position is (x i B , y i B ), and the inter-site vector map is M u (x, y), M v (x, y), the relative position vector (u i , v i )teeth u i =M u (x i B , y i B ) v i =M v (x i B , y i B ) It becomes.

[0040] An example of the detection results of the head position, body position, and relative position vector for the input shown in Fig. 5(a) is shown in Fig. 5(d). 511 and 512 are the detected head positions. 521 and 522 are the detected body positions. 531 is the relative position vector at the body position 521, and 532 is the relative position vector at the body position 522. The detection unit 203 outputs the detected head position, body position, and relative position vector to the connection unit 204.

[0041] In S304, the connection unit 204 associates the body and head of the same subject using the body position, head position, and relative position vector detected by the detection unit 203 in S303. First, the connection unit 204 obtains the fitness of the association for all combinations of the detected body and head. The connection unit 204 obtains the fitness MatchingScore(i, j) between the i-th body and the j-th head by the following formula. Here, i and j are integers greater than or equal to 1 and less than or equal to the total number.

Equation

[0042] The fitness obtained in this way is such that the smaller the value, the higher the fitness. Therefore, by selecting the combinations in ascending order of the fitness obtained for all combinations, the body and head of the same subject can be associated. An example of the association result for the image shown in Fig. 5(a) is shown in Fig. 5(e). The body 521 and head 512 of the horse 501, and the body 522 and head 511 of the horse 502 are correctly associated respectively.

[0043] By the processing of S301 to S304, the image processing unit 100 can detect the head position and body position of the specific subject shown in the input image and associate the head positions of the same subject.

[0044] <Modified Example of S304> When the relative position vector is correctly detected, the torso and head of the same subject can be associated with each other by the above-mentioned process. However, there are cases where the relative position vector is not correctly detected. In order to prevent incorrect association when the relative position vector is incorrect, the likelihood map by direction between parts can be referenced.

[0045] The process of S304 when referring to the inter-part direction likelihood map will be described. First, the connection unit 204 calculates the relative position vector (u i , v i The reliability Coef(i) for the relative position vector is calculated based on the consistency between the inter-part direction likelihood map and the relative position vector.

[0046] The connection part 204 is a relative position vector (u i , v i ) angle θ i From the i-th body position, a channel where a peak should exist in the inter-part direction likelihood map is determined. For example, the angle θ i In this case, the first channel of the inter-body direction likelihood map is the i-th torso position (x i B , y i B ) should have a peak at

[0047] The connection unit 204 connects the i-th body position (x i B , y i B) value. The connection unit 204 then considers it to be a peak if it has a value greater than a predetermined threshold. If the only channel of the inter-part direction likelihood map in which the pixel of the coordinates of the trunk position has a value considered to be a peak is the channel corresponding to the angle θ formed by the relative position vector, the connection unit 204 determines that the reliability of the relative position vector is high (reliable). On the other hand, if a channel of the inter-part direction likelihood map other than the channel corresponding to the angle θ formed by the relative position vector has a value in which the pixel of the coordinates of the trunk position has a value considered to be a peak, the connection unit 204 determines that the reliability of the relative position vector is low (unreliable).

[0048] The connection unit 204 sets the reliability Coef(i) to a first value (e.g., 0) when the reliability of the relative position vector is determined to be high, and sets the reliability Coef(i) to a second value (e.g., ∞) when the reliability is determined to be low. Note that the second value does not actually need to be infinity, and may be a flag (e.g., 1) that invalidates the matching score MatchingScore.

[0049] Next, the connection unit 204 calculates the matching score MatchingScore(i, j) for all combinations of the detected torso and head by the following formula: This is the same as the previous formula except that Coef(i) is added.

number

[0050] The degree of compatibility reflecting the reliability Coef(i) in this way becomes smaller as the degree of compatibility becomes higher. Furthermore, for combinations with low reliability of the relative position vector, the degree of compatibility becomes infinite. Therefore, by selecting combinations in ascending order of degree of compatibility from among the degrees of compatibility calculated for all combinations, it is possible to associate the torso and head of the same subject based on the relative position vector with high reliability. Note that if Coef(i) is a value indicating low reliability, that combination may be excluded.

[0051] <Explanation of learning process> Next, a description will be given of the learning process of the motion parameters used in the estimation unit 202. Fig. 6 is a block diagram expressing the image processing device 100 that executes the learning process of the motion parameters by functional blocks. The same reference numerals are used for the same functional blocks as in Fig. 2, and the description will be omitted.

[0052] In the following, for ease of explanation and understanding, it is assumed that each functional block is realized by the CPU 101 executing a program. However, one or more functional blocks may be realized by hardware other than the CPU 101. Note that one functional block may be realized by multiple pieces of hardware, or one piece of hardware may realize multiple functional blocks.

[0053] The operation of the estimation unit 202 generating various feature maps for the input image supplied from the image acquisition unit 201 using the operation parameters acquired from the parameter acquisition unit 205 is as described with reference to Fig. 4. When executing the learning process, the estimation unit 202 outputs the generated various feature maps to the loss calculation unit 602.

[0054] The teacher data acquisition unit 601 acquires teacher data corresponding to the input image. Details of the teacher data will be described later. The teacher data may be read out from the storage unit 104 in advance, or may be input by the user via the input unit 105.

[0055] The loss calculation unit 602 calculates a loss value based on various feature maps generated by the estimation unit 202 and the teacher data acquired by the teacher data acquisition unit 601. The method of calculating the loss value will be described later.

[0056] The parameter update unit 603 updates the operation parameters so as to reduce the loss value based on the loss value calculated by the loss calculation unit 602. The parameter update unit 603 overwrites the updated operation parameters in the acquisition destination of the operation parameters (for example, the storage unit 104).

[0057] The operation of the learning process of the operational parameters in the image processing device 100 will be described with reference to the flowchart shown in FIG. 7 and the block diagram shown in FIG. 6. The steps executed by each functional block shown in FIG. 7 are executed by hardware (CPU 101 or other hardware) that realizes the functional block. The hardware that realizes the functional block shown in FIG. 7 functions as a learning means. This operation may be executed in response to a user instruction via the input unit 105, for example, or in response to any other event. In addition, it may be executed as part of a function provided by an image processing application running on the image processing device 100, for example.

[0058] The learning process of the motion parameters is a process in which an image containing a specific subject is used as an input image, and the motion parameters are optimized so that the above-mentioned various feature maps can be appropriately generated by the estimation unit 202 (CNN in this embodiment). The processes (learning loop) of S701 to S705 described later are repeatedly executed until a termination condition (for example, the loss becomes less than a threshold) is satisfied.

[0059] Here, it is assumed that the learning dataset is stored in advance in a location (e.g., storage unit 104) accessible from image acquisition unit 201 and teacher data acquisition unit 601. The learning dataset is made up of multiple combinations of learning input images containing multiple specific subjects and teacher data (correct answer data) corresponding to the input images.

[0060] In S701, the image acquisition unit 201 acquires one frame of an input image for learning (which may be a still image or a video). Here, it is assumed that the input image for learning is an image in which horses 801 to 804, which are specific subjects, are overlapped, as shown in Fig. 8(a). Note that the specific subjects do not necessarily have to be overlapped, and may be adjacent or closely adjacent images.

[0061] In S702, the teacher data acquisition unit 601 acquires teacher data corresponding to the input image. In this embodiment, the teacher data is a list of head positions and body positions for each specific subject captured in the input image, as shown in FIG. 8(b). In FIG. 8(b), the i-th body position (x i B , y i B ) and the i-th head position (x i H , y i H ) is the body position and head position of the i-th specific subject. The teacher data acquisition unit 601 outputs the acquired teacher data to the loss calculation unit 602.

[0062] In S703, the estimation unit 202 generates feature maps (body likelihood map, head likelihood map, inter-part vector map, inter-part direction likelihood map) from the input image using CNN, using the motion parameters acquired by the parameter acquisition unit 205. The estimation unit 202 outputs the generated feature maps to the loss calculation unit 602.

[0063] In S704, the loss calculation unit 602 calculates a loss value to be used for updating the motion parameters using the feature maps and the teacher data. The loss calculation unit 602 linearly combines the loss values ​​for each feature map to calculate a loss value to be used for updating the motion parameters. An example of calculating the loss for each feature map will be described below.

[0064] <Loss value calculation for head likelihood map> First, the loss calculation unit 602 calculates the correct value ^(M c H ) is generated by the following formula:

number

[0065] Also, the function G(x, y, c x , c y ,σ), is the point (c x , c y ) is a two-dimensional Gaussian function with kernel size σ centered on the correct value ^(M c H ) has a peak value at the head position and decays as it moves away from the head position. The correct value of the head likelihood map corresponding to the input image in Figure 8(a) is shown in Figure 8(c).

[0066] Next, the loss calculation unit 602 calculates the head likelihood map M c H and the correct value of the head likelihood map generated from the training data^(M c H ) loss L c H is calculated by the following formula:

number

[0067] <Loss value calculation for body likelihood map> The loss calculation unit 602 calculates the loss of the body likelihood map in the same way as the loss of the head likelihood map. That is, the loss calculation unit 602 calculates the correct answer value ^(M c B ) is generated by the following formula:

number

[0068] Correct answer^(M c B ) has a peak value at the torso position and decays as it moves away from the head position. The correct value of the torso likelihood map corresponding to the input image in Figure 8(a) is shown in Figure 8(d).

[0069] Next, the loss calculation unit 602 receives the torso likelihood map M c B and the correct value of the torso likelihood map calculated from the training data^(M c B ) loss L c B is calculated by the following formula:

number

[0070] <Calculation of loss value of vector map between parts> The loss calculation unit 602 calculates the inter-part vector map M u , M v Loss on L UV The ideal inter-part vector map M is calculated as follows: u , M v In the case of the loss calculation unit 602, the pixel value of the body position is the directional component of the relative position vector to the head corresponding to the body. UV The body position (x i B , y i B ) pixel value and the directional component of the relative position vector M u , M v The loss is calculated as the square error between the value of UV can be calculated by the following formula:

number

[0071] <Calculation of loss value for likelihood map by direction between parts> The loss calculation unit 602 calculates the loss value of the inter-part direction likelihood map as follows. First, the loss calculation unit 602 calculates the correct value ^(M D )

[0072] Correct value of likelihood map for each direction between parts^(M D ) is a four-channel map, as explained using Fig. 5(b) and Fig. 5(c). Each channel is a map of the relative position vector (u i , v i ) angle θ i The correct value of the likelihood map for each direction between parts ^(M D ) has a peak at a fuselage position with a relative position vector in the angular range corresponding to that channel.

[0073] Ω k = kπ / 2, the angle θ between the relative position vectors i Omega (k-1) ≦θ i <Ω k A target object that satisfies this has a peak in the k-th channel. Therefore, the loss calculation unit 602 calculates the correct value ^(MD ) is calculated.

number

[0074] Next, the loss calculation unit 602 calculates the torso likelihood map M D and the correct answer^(M D ) loss L D is calculated as follows:

number

[0075] The coordinates of the relative position vector calculated from the training data corresponding to FIG. 8(a) and the angle θ between the relative position vector i The correct value of the generated inter-part direction likelihood map ^(M D ) is shown in Figure 8(f).

[0076] In this embodiment, the angle θ i The possible angular range of 360° (2π) was divided into 90° (π / 2) intervals, and the number of channels in the inter-part direction likelihood map was set to 4. However, the number of divisions of the angular range is not limited to 4, and may be any integer equal to or greater than 2. Also, the angular range does not have to be uniform.

[0077] The loss calculation unit 602 linearly combines all the calculated loss values ​​as shown in the following formula to obtain the final loss value L Total Calculate.

number

[0078] The coefficient λ applied to each loss value in the linear combination difference H , λ B , λ UV , λ Dmay be a predetermined fixed value. Alternatively, it may be a value corresponding to the number of learning times at that time. For example, when the number of learning times is less than a predetermined value, λ H = λ B = λ UV = 0 may be used. In this case, the learning of the operation parameters is performed based only on the loss L D regarding the inter-site direction-specific likelihood map. This corresponds to preferentially learning the inter-site direction-specific likelihood map over other maps. As a result, processing using the information of the inter-site direction-specific likelihood map, such as the determination of the reliability of the relative vector, can be accurately performed at an early stage.

[0079] In S704, the parameter update unit 603 updates the operation parameters based on the loss value calculated by the loss calculation unit 602. The update of the operation parameters can be performed by a known method based on the stochastic gradient descent method (SGD), such as the Momentum method or the Adam method, based on the error backpropagation method. The parameter update unit 603 records the updated operation parameters in the storage unit 104.

[0080] When the update of the operation parameters is completed, if a predetermined end condition is satisfied, the learning process ends, and if not, the process returns to S701 to continue learning. The end condition may be, for example, the number of updates of the operation parameters, or the loss value becoming less than a predetermined value. By performing the processes of S701 to S705, the operation parameters (inter-layer coupling weight coefficients of the CNN) used in the detection process can be learned.

[0081] <Modification of S704> In S704 of the above-described learning process, the loss for the inter-site direction-specific likelihood map was calculated for the entire region of each channel. However, a non-target region where the loss is not calculated for each channel may be set.

[0082] The non-target region can be set, for example, around the torso position of the non-target subject determined for each channel. The condition for the i-th subject to be a non-target subject for the k-th channel is the angle θ i is not included in the angle range of the kth channel. Specifically, if δ is a margin (0 or more), the angle θ i But [Ω (k-1) -δ,Ω (k-1) ] range, and [Ω k ,Ω k +δ]. The margin δ can be set arbitrarily, but in this embodiment, δ=π / 8.

[0083] Specifically, the loss calculation unit 602 calculates a target determination map ^(F D ) is generated for each channel using the following formula:

number

[0084] Here, the function Circle(x, y, c x , c y ,σ), is the point (c x ,v y The function is 0 inside the circle of radius σ centered on F and 1 elsewhere. D ) means that areas with a value of 0 are non-target areas and do not require loss calculation.

[0085] The target determination map ^(F D An example of the relative position vector of a horse 804 is shown in FIG. i is excluded from the loss calculation for the angle range of channel 2. Therefore, in the second channel of the object determination map, the value of the circular region centered on the torso position of the subject 804 is 0 (shown in black in the figure).

[0086] Target Judgment Map^(F D) based on the loss L for the likelihood map by direction between parts, excluding non-target regions D The calculation of can be expressed by the following formula.

number

[0087] By using the object determination map, robust learning can be performed for training data in which the angle θ between the relative position vectors is close to the boundary of the angle range.

[0088] As described above, according to the present embodiment, a likelihood map for each part direction is generated as a feature map, which roughly classifies the angles of the relative position vectors between different parts of a specific subject. By using the likelihood map for each part direction for matching between parts, it is possible to accurately match between parts belonging to the same subject even when the estimation accuracy of the relative position vector is low. Furthermore, learning the likelihood map for each part direction is easier than learning the relative position vector map (part vector map). Therefore, by learning the likelihood map for each part direction in preference to the relative part vector map, it is possible to improve the learning efficiency of the final relative part vector map.

[0089] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0090] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) a detection means for detecting a first region and a second region of a specific subject from an image and a relative position vector from the first region to the second region using a convolutional neural network; A learning means for learning the detection means, the convolutional neural network is configured to generate a feature map indicative of an approximate direction of the relative position vector; The image processing device according to claim 1, wherein the learning means trains the detection means using a loss function related to the relative position vector and a loss function related to the feature map. (Item 2) 2. The image processing device according to item 1, wherein the learning means further uses a loss function related to the position of the first part and a loss function related to the position of the second part to train the detection means. (Item 3) 3. The image processing device according to item 1 or 2, wherein the learning means trains the detection means by prioritizing a loss function related to the feature map over other loss functions when the number of times of learning is less than a predetermined value. (Item 4) 4. The image processing device according to any one of items 1 to 3, wherein the feature map indicates an approximate direction of the relative position vector in an angle range having a predetermined magnitude. (Item 5) 5. The image processing device according to item 4, wherein the feature map has a channel for each of the angle ranges, and each channel indicates the likelihood that a second part corresponding to a first part exists. (Item 6) 6. The image processing device according to item 5, wherein the learning means sets an area in the feature map where a calculation of a loss function is not performed. (Item 7) 7. The image processing device according to item 6, wherein the learning means sets an area in which the loss function is not calculated for a feature map of a channel corresponding to an angle range that does not correspond to the direction of the relative position vector. (Item 8) The image processing device according to item 7, characterized in that the learning means sets a predetermined range centered on the first part different from the first part corresponding to the relative position vector as a region in which the loss function is not calculated, in a feature map of a channel corresponding to an angle range not corresponding to the direction of the relative position vector. (Item 9) 9. The image processing device according to any one of items 1 to 8, further comprising a matching unit that matches the first and second parts of a same subject based on the relative position vector detected by the detection unit. (Item 10) 10. The image processing device according to item 9, wherein the association means determines the reliability of the relative position vector based on consistency with an approximate direction indicated by the feature map. (Item 11) An image processing method executed by an image processing device having a detection means for detecting a first part and a second part of a specific subject from an image and a relative position vector from the first part to the second part using a convolutional neural network, comprising: generating, by the convolutional neural network, a feature map indicative of an approximate direction of the relative position vector; training the detection means using a loss function related to the relative position vector and a loss function related to the feature map. (Item 12) A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 10.

[0091] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0092] 100: image processing device, 101: CPU, 102: ROM, 103: RAM, 104: storage unit, 202: estimation unit, 203: detection unit, 204: connection unit

Claims

1. a detection means for detecting a first portion and a second portion of a specific subject from an image and a relative position vector from the first portion to the second portion using a convolutional neural network; A learning means for learning the detection means, the convolutional neural network is configured to generate a feature map indicative of an approximate direction of the relative position vector; The image processing device according to claim 1, wherein the learning means trains the detection means using a loss function related to the relative position vector and a loss function related to the feature map.

2. The image processing apparatus according to claim 1 , wherein the learning means further uses a loss function related to the position of the first part and a loss function related to the position of the second part to train the detection means.

3. 2. The image processing apparatus according to claim 1, wherein said learning means trains said detection means by prioritizing a loss function related to said feature map over other loss functions when the number of times of learning is less than a predetermined value.

4. The image processing apparatus according to claim 1 , wherein the feature map indicates an approximate direction of the relative position vector in an angle range having a predetermined size.

5. The image processing apparatus according to claim 4 , wherein the feature map has a channel for each of the angle ranges, and each channel indicates a likelihood that a second part corresponding to a first part exists.

6. The image processing apparatus according to claim 5 , wherein the learning means sets an area in the feature map where the calculation of the loss function is not performed.

7. The image processing device according to claim 6 , wherein the learning means sets an area in which the calculation of the loss function is not performed for a feature map of a channel corresponding to an angle range not corresponding to the direction of the relative position vector.

8. The image processing device according to claim 7, characterized in that the learning means sets a predetermined range centered on the first part different from the first part corresponding to the relative position vector as an area in which the loss function is not calculated, in a feature map of a channel corresponding to an angle range not corresponding to the direction of the relative position vector.

9. 2. The image processing device according to claim 1, further comprising a matching means for matching the first and second parts detected by the detection means with parts of the same subject based on the relative position vector detected by the detection means.

10. 10. The image processing apparatus according to claim 9, wherein the associating means determines the reliability of the relative position vector based on consistency with the approximate direction indicated by the feature map.

11. 1. An image processing method executed by an image processing device having a detection means for detecting a first portion and a second portion of a specific subject from an image and a relative position vector from the first portion to the second portion using a convolutional neural network, comprising: generating, by the convolutional neural network, a feature map indicative of an approximate direction of the relative position vector; training the detection means using a loss function related to the relative position vector and a loss function related to the feature map.

12. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 10.

Images