Image processing device, image processing method, and program

Abstract

The image processing device 10 includes a feature map generating unit 11 that generates a first feature map for identifying the horizontal position of a person in an image and a second feature map for identifying the vertical position of a person in an image, and a grouping unit 12 that groups each joint detected from an image into a corresponding person using the horizontal and vertical positions of each joint and the first and second feature maps. It is characterized by:

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method for processing image data of an image including a person, and further to a program for implementing the same. To The present invention also relates to an image processing apparatus and a feature map generating device for generating a feature map used in the image processing method, and further to a learning model generating device for generating a learning model used in generating the feature map. [Background technology]

[0002] In recent years, research into estimating human posture from images has been attracting attention. This type of research is expected to be used in fields such as image surveillance systems and sports. In addition, estimating human posture from images can be used to analyze the movements of store clerks in a store, for example, and is thought to contribute to efficient product placement.

[0003] In estimating the posture of a person from such an image, it is important to correctly associate the joints detected from the image with the person in the image. This is because, when there are multiple people in an image, if the detected joints are associated with the wrong person, the posture estimation accuracy will be significantly reduced.

[0004] For example, Non-Patent Document 1 discloses a system that associates joints in an image with a person. Specifically, when image data of an image including a person is input, the system disclosed in Non-Patent Document 1 detects all of the person's joints from the image data using a convolutional neural network that has learned images of the person's joints.

[0005] Furthermore, the system disclosed in Non-Patent Document 1 uses a convolutional neural network that has been trained on the entire person image for each person to generate a feature map from the image data that indicates an instance segmentation of the person for each person in the image. Then, for each feature map, the system disclosed in Non-Patent Document 1 compares the instance segmentation in the feature map with the detected joints and associates the detected joints with the corresponding person. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Kaiming He, et al., “Mask R-CNN.”, 2017 IEEE International Conference on Computer Vision (ICCV 2017) Summary of the Invention [Problem to be solved by the invention]

[0007] However, the system disclosed in the above-mentioned Non-Patent Document 1 requires the generation of a feature map for each person in the image, which causes a problem that the processing load on the system increases as the number of people in the image increases. Therefore, the system disclosed in the above-mentioned Non-Patent Document 1 is limited in the fields in which it can be applied.

[0008] An example of the object of the present invention is to provide an image processing device, an image processing method, and program Another example of an object of the present invention is to provide a feature map generating device and a learning model generating device that are applicable to an image processing device. [Means for solving the problem]

[0009] In order to achieve the above object, an image processing device according to one aspect of the present invention comprises: a feature map generating means for generating a first feature map identifying a horizontal position of a person in an image and a second feature map identifying a vertical position of the person in the image; a grouping means for grouping each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; The present invention is characterized in that it is equipped with:

[0010] In order to achieve the above object, a feature map generating device according to one aspect of the present invention comprises: a feature map generating means for generating a first feature map that identifies a horizontal position of a person in an image and a second feature map that identifies a vertical position of the person in the image; It is characterized by:

[0011] In order to achieve the above object, a learning model generation device according to one aspect of the present invention comprises: a learning model generating means for generating a learning model by machine learning a relationship between the image and the first feature map and the second feature map, using image data of an image including a person, a first feature map that specifies a horizontal position of the person in the image, and a second feature map that specifies a vertical position of the person in the image as training data; It is characterized by:

[0012] In order to achieve the above object, an image processing method according to one aspect of the present invention comprises: generating a first feature map identifying a horizontal position of a person in an image and a second feature map identifying a vertical position of the person in the image; a grouping step of grouping each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; having It is characterized by:

[0013] Further, in order to achieve the above object, in one aspect of the present invention, program to the computer, generating a first feature map identifying a horizontal position of a person in an image and a second feature map identifying a vertical position of the person in the image; a grouping step of grouping each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; Run 、 It is characterized by: Effect of the Invention

[0014] As described above, according to the present invention, association between joints and people is performed regardless of the number of people present in an image. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an image processing device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram specifically illustrating an example of the configuration of the image processing device according to the first embodiment. [Diagram 3] FIG. 3(a) is a diagram showing an example of a first feature map generated in the first embodiment, and FIG. 3(b) is a diagram showing an example of a second feature map generated in the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining the process in the grouping unit in the first embodiment. [Diagram 5]FIG. 5 is a flow diagram showing the operation of the image processing device in the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of a feature map generating device according to the second embodiment. [Figure 7] FIG. 7 is a configuration diagram showing an example of the configuration of a learning model generating device in the third embodiment. [Figure 8] FIG. 8 is a flow diagram showing the operation of the learning model generating device in the third embodiment. [Figure 9] FIG. 9 is a block diagram showing an example of a computer that realizes the image processing device, the feature map generating device, and the learning model generating device 30 according to the first to third embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (Embodiment 1) In the first embodiment, an image processing device, an image processing method, and a program for image processing will be described with reference to FIGS. 1 to 5. FIG.

[0017] [Device configuration] First, a schematic configuration of an image processing device in the first embodiment will be described with reference to Fig. 1. Fig. 1 is a configuration diagram showing a schematic configuration of the image processing device in the first embodiment.

[0018] An image processing device 10 according to the first embodiment shown in Fig. 1 is a device for processing image data of an image including a person. As shown in Fig. 1, the image processing device 10 includes a feature map generating unit 11 and a grouping unit 12.

[0019] The feature map generating unit 11 generates a first feature map and a second feature map. The first feature map is a map for identifying the horizontal position of a person in an image. The second feature map is a map for identifying the vertical position of a person in an image.

[0020] The grouping unit 12 uses the horizontal and vertical positions of each joint detected from the image, and the first and second feature maps to group each joint into a corresponding person.

[0021] In this way, in the image processing device 10, the first feature map and the second feature map are generated, so that association between joints and people can be performed regardless of the number of people present in the image.

[0022] Next, the configuration and functions of the image processing device 10 in the first embodiment will be specifically described with reference to Figs. 2 to 5. Fig. 2 is a block diagram specifically showing an example of the configuration of the image processing device in the first embodiment. Fig. 3(a) is a diagram showing an example of a first feature map generated in the first embodiment, and Fig. 3(b) is a diagram showing an example of a second feature map generated in the first embodiment. Fig. 4 is a diagram for explaining the processing in the grouping unit in the first embodiment.

[0023] As shown in FIG. 2, in the first embodiment, the image processing device 10 includes an image data acquisition unit 13, a storage unit 14, and a joint detection unit 15, in addition to the feature map generation unit 11 and grouping unit 12 described above.

[0024] The image data acquisition unit 13 acquires image data 17 of an image including a person captured by an imaging device, and stores the acquired image data 17 in the storage unit 14. The image data may be acquired from the imaging device or from an external storage device that stores image data. The storage unit 14 further stores a learning model 16, which will be described later.

[0025] In the first embodiment, the feature map generating unit 11 generates, as a first feature map, a map that is composed of the same number of pixels as the pixels constituting the image and that assigns a numerical value indicating the position of the person in the horizontal direction to each pixel in an area corresponding to a person, as shown in Fig. 3(a). Also, the feature map generating unit 11 generates, as a second feature map, a map that is composed of the same number of pixels as the pixels constituting the image and that assigns a numerical value indicating the position of the person in the vertical direction to each pixel in an area corresponding to a person, as shown in Fig. 3(b).

[0026] 3(a), the feature map generating unit 11 assigns the following values ​​to pixels in an area corresponding to a person in the first feature map as "numerical values ​​indicating the horizontal position of the person." The assigned value indicates the ratio (e.g., 0.1W) of the horizontal distance from the origin of the first feature map to the reference point of the person to the horizontal length W of the first feature map.

[0027] 3(b), the feature map generating unit 11 assigns the following values ​​to pixels in the area corresponding to the person in the second feature map as "numerical values ​​indicating the position of the person in the vertical direction." The assigned value indicates the ratio (e.g., 0.25H) of the vertical distance from the origin of the second feature map to the reference point of the person to the vertical length H of the second feature map.

[0028] In Figures 3(a) and (b), the origin of the map is set to the top left corner point, but this is not limited thereto. Also, in Figures 3(a) and (b), the reference point of the person is set to the base of the person's neck, but this is not limited thereto.

[0029] In addition, in the first embodiment, the feature map generating unit 11 can also generate the first feature map and the second feature map by using the learning model 16 stored in the storage unit 14. The learning model 16 is constructed in advance by machine learning the relationship between an image including a person and the first feature map and the second feature map. An example of the machine learning method is deep learning. The constructed learning model 16 is stored in the storage unit 14. The construction of the learning model 16 is performed by a learning model generating device described later.

[0030] The joint detection unit 15 detects the joints of a person from the image of the image data acquired by the image data acquisition unit 13. Specifically, the joint detection unit 15 can detect the joints of a person in the image data by applying the image data to a learning model for joint detection. An example of the machine learning model is a model that learns the relationship between the image of a person and each joint of the person in the image by machine learning. When image data is input, the machine learning model outputs, for example, a heat map indicating the probability that each joint exists for each joint in the image. In this case, the joint detection unit 15 detects each joint based on the output heat map.

[0031] In the embodiment 1, the method of detecting the joints from the image is not limited. For example, the joint detection unit 15 can detect each joint from the image data by using a feature amount for each joint that is prepared in advance.

[0032] In the first embodiment, the grouping unit 12 identifies, for each joint detected from the image, the numerical value of the pixel corresponding to that joint in the first feature map and the numerical value of the pixel corresponding to that joint in the second feature map.

[0033] Next, for each joint detected from the image, the grouping unit 12 calculates the distance between the joint and each person in the image using the two specified values ​​and a value indicating the horizontal position of the person and a value indicating the vertical position of the person. After that, the grouping unit 12 determines the person corresponding to each joint based on the distance calculated for each person, and performs grouping based on the determination result.

[0034] Specifically, as shown in Fig. 4, the grouping unit 12 projects each joint detected by the joint detection unit 15 onto a first feature map and a second feature map. In the example of Fig. 4, only the joint J is projected onto the first feature map and the second feature map.

[0035] Then, the grouping unit 12 identifies, for the joint J, the numerical value LX(J) of the corresponding pixel in the first feature map and the numerical value LY(J) of the corresponding pixel in the second feature map. 1 and P 2 For each, a number LX(N i ) and a number indicating the vertical position LY(N i ) to identify N i shows the reference points for each person used to create the first and second feature maps.

[0036] Thereafter, the grouping unit 12 groups the identified numerical values ​​LX(J) and LY(J) and the numerical value LX(N i ) and LY(N i ) into the following equation 1 to find the joint J and the figure P i Distance Ad(J,P i ) is calculated.

[0037] (Number 1) Ad (J, P i ) = [LX (J) - LX (N i )] 2 + [LY (J) - LY (Ni )] 2

[0038] In the example in Figure 4, person P 1 For LX(J)=0.3W, LX(N 1 )=0.3W, LY(J)=0.2H, LY(N 1 )=0.2H, so Ad(J,P 1 )=0. On the other hand, person P 2 For LX(N 2 )=0.5W, LY(N 2 )=0.4H, so Ad(J,P 2 )=0.04(W+H) 2 Therefore, in the example of FIG. 5, the grouping unit 12 classifies the person to which the joint J corresponds as person P 1 is decided.

[0039] Furthermore, in order to avoid obviously unnatural grouping, the grouping unit 12 can set conditions to determine the person corresponding to the joints, such as that the calculated distance does not exceed a set value, that multiple joints of the same type do not correspond to the same person, etc.

[0040] [Device operation] Next, the operation of the image processing device 10 in the first embodiment will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing the operation of the image processing device in the first embodiment. In the following description, Figs. 1 to 4 will be referred to as appropriate. Also, in the first embodiment, an image processing method is implemented by operating the image processing device 10. Therefore, the description of the image processing method in the first embodiment will be replaced by the following description of the operation of the image processing device 10.

[0041] As shown in FIG. 5, first, the image data acquisition unit 13 acquires image data 17 of an image including a person, and stores the acquired image data 17 in the storage unit 15 (step A1).

[0042] Next, the feature map generating unit 11 applies the image data acquired in step A1 to the learning model 16 stored in the storage unit 15 to generate a first feature map and a second feature map (step A2).

[0043] Next, the joint detection unit 15 detects the joints of the person in the image from the image data acquired in step A1 (step A3). Also, in step A3, the joint detection unit 15 specifies the coordinates of each of the detected joints.

[0044] Next, the grouping unit 12 projects each of the joints detected in step A3 onto a first feature map and a second feature map (step A4).

[0045] Next, the grouping unit 12 identifies, for each joint, the numerical value of the pixel corresponding to that joint in the first feature map and the numerical value of the pixel corresponding to that joint in the second feature map. Then, the grouping unit 12 calculates, for each joint, the distance between that joint and each person in the image using the identified value and the numerical value indicating the position of each person (step A5).

[0046] Next, the grouping unit 12 determines, for each joint, a person corresponding to that joint based on the distance calculated for each person in step A5, and performs grouping based on the determination result (step A6).

[0047] Thereafter, the grouping unit 12 outputs the grouping result obtained in step A6 (step A7). The grouping result is used to estimate the posture of a person, for example, in a system for estimating the posture of a person.

[0048] As described above, according to the first embodiment, when the detected joints are projected onto the first feature map and the second feature map, the distance between each joint and each person is calculated. Then, when the distance is calculated, the person to which the joint corresponds is easily identified. In other words, according to the first embodiment, it is possible to associate the joints with the people regardless of the number of people present in the image.

[0049] [program] The program for image processing in the first embodiment may be any program that causes a computer to execute steps A1 to A7 shown in Fig. 5. By installing and executing this program in a computer, the image processing device 10 and the image processing method in the first embodiment can be realized. In this case, the processor of the computer functions as the feature map generating unit 11, the grouping unit 12, the image data acquiring unit 13, and the joint detecting unit 15, and performs the processing.

[0050] In the first embodiment, the storage unit 14 may be realized by storing a data file in a storage device such as a hard disk provided in the computer, or may be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0051] The program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the feature map generating unit 11, the grouping unit 12, the image data acquiring unit 13, and the joint detecting unit 15.

[0052] (Embodiment 2) In the second embodiment, a feature map generating device, a feature map generating method, and a program for generating a feature map will be described with reference to Fig. 6. Fig. 6 is a configuration diagram showing an example of the configuration of the feature map generating device in the second embodiment.

[0053] As shown in Fig. 6, in the second embodiment, a feature map generating device 20 is a device for generating the first feature map and the second feature map shown in Fig. 3(a) and (b). As shown in Fig. 6, the feature map generating device 20 includes an image data acquiring unit 21, a storage unit 22, and a feature map generating unit 23.

[0054] The image data acquiring unit 21, the storage unit 22, and the feature map generating unit 23 are configured similarly to the image data acquiring unit 13, the storage unit 14, and the feature map generating unit 11 shown in Fig. 2 in embodiment 1. In addition, the image data acquiring unit 21, the storage unit 22, and the feature map generating unit 23 have similar functions to the image data acquiring unit 13, the storage unit 14, and the feature map generating unit 11.

[0055] Specifically, the image data acquisition unit 21, like the image data acquisition unit 13, acquires image data 25 of an image including a person captured by an imaging device, and stores the acquired image data 25 in the storage unit 22. Like the storage unit 15, the storage unit 22 also stores a learning model 24.

[0056] Similar to the feature map generation unit 11, the feature map generation unit 23 generates a first feature map shown in Fig. 3(a) and a second feature map shown in Fig. 3(b). Also in the second embodiment, the feature map generation unit 23 generates the first feature map and the second feature map by using the learning model 24 stored in the storage unit 22.

[0057] The learning model 24, like the learning model 16, is constructed in advance by machine learning the relationship between an image including a person and the first and second feature maps by deep learning or the like. The learning model 24 is stored in the storage unit 22. The construction of the learning model 24 is also performed by a learning model generation device described later.

[0058] In this way, the first feature map and the second feature map can be generated by the feature map generating device 20. Note that in the second embodiment, the feature map generating device 20 may be configured to include only the feature map generating unit 23.

[0059] In the second embodiment, the feature map generation method is realized by executing steps similar to steps A1 and A2 shown in Fig. 5 in the feature map generation device 20. Furthermore, the feature map generation device 20 and the feature map generation method in the second embodiment can be realized by using a program that causes a computer to execute steps A1 and A2 shown in Fig. 5.

[0060] (Embodiment 3) In the third embodiment, a learning model generating device, a learning model generating method, and a program for generating a learning model will be described with reference to FIGS.

[0061] [Device configuration] First, the configuration of the learning model generating device in the third embodiment will be described with reference to Fig. 7. Fig. 7 is a configuration diagram showing an example of the configuration of the learning model generating device in the third embodiment.

[0062] The learning model generation device 30 shown in Fig. 7 in the third embodiment is a device for generating the learning model used in the first and second embodiments. As shown in Fig. 7, the learning model generation device 30 in the third embodiment includes a training data acquisition unit 31, a storage unit 32, and a learning model generation unit 33.

[0063] The training data acquisition unit 31 acquires training data 35. The training data 35 is composed of image data of an image including a person, a first feature map that specifies the horizontal position of the person in the image, and a second feature map that specifies the vertical position of the person in the image. The acquired training data 35 is stored in the storage unit 32.

[0064] The learning model generation unit 33 performs machine learning of the relationship between the image and the first feature map and the relationship between the image and the second feature map using the training data 35 stored in the storage unit 32. This generates a learning model 34. Examples of machine learning techniques include deep learning.

[0065] Specifically, the learning model generation unit 33 first inputs image data of an image into the learning model, and causes the learning model to output a first feature map and a second feature map. Then, the learning model generation unit 33 calculates the difference between the output first feature map and second feature map and the first feature map and second feature map used as training data. Furthermore, the learning model generation unit 33 updates the parameters of the learning model so as to reduce the calculated difference. In this way, the parameters of the learning model are updated by the training data, and the learning model 34 is generated.

[0066] [Device operation] Next, the operation of the learning model generation device 30 in the third embodiment will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing the operation of the learning model generation device in the third embodiment. In the following description, Fig. 7 will be referred to as appropriate. Also, in the third embodiment, the learning model generation method is implemented by operating the learning model generation device 30. Therefore, the description of the learning model generation method in the third embodiment will be replaced with the following description of the operation of the learning model generation device 30.

[0067] 8, first, the training data acquisition unit 31 acquires image data of an image including a person, a first feature map corresponding to the image data, and a second feature map corresponding to the image data as training data 35 (step B1). The training data acquisition unit 31 also stores the acquired training data 35 in the storage unit 32.

[0068] Next, the learning model generation unit 33 learns the relationship between the image and the first feature map and the relationship between the image and the second feature map by machine learning using the training data 35 stored in the storage unit 32 (step B2). As a result, a learning model 34 is generated.

[0069] In this way, according to the third embodiment, the learning model 34 is generated. The learning model 34 learns the relationship between the image data and the first feature map, and the relationship between the image data and the second feature map. The generated learning model 34 can be used in the first and second embodiments.

[0070] [program] The program in the third embodiment may be any program that causes a computer to execute steps B1 to B2 shown in Fig. 8. By installing and executing this program in a computer, the learning model generation device 30 and the learning model generation method in the third embodiment can be realized. In this case, the processor of the computer functions as the training data acquisition unit 31 and the learning model generation unit 33 and performs the processing.

[0071] In the third embodiment, the storage unit 32 may be realized by storing a data file in a storage device such as a hard disk provided in the computer, or may be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0072] The program in the third embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as either the training data acquisition unit 31 or the learning model generation unit 33.

[0073] (physical configuration) Here, a computer that executes a program to realize the image processing device 10, the feature map generation device 20, and the learning model generation device 30 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of a computer that realizes the image processing device, the feature map generation device, and the learning model generation device 30 in the first to third embodiments.

[0074] 9, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0075] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this embodiment, the GPU or FPGA can execute the programs in the embodiments.

[0076] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and is stored in the storage device 113, into the main memory 112, and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0077] Moreover, the program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed on the Internet connected via the communication interface 117.

[0078] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0079] Data reader / writer 116 mediates data transmission between CPU 111 and recording medium 120, reads programs from recording medium 120, and writes processing results in computer 110 to recording medium 120. Communication interface 117 mediates data transmission between CPU 111 and other computers.

[0080] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

[0081] The image processing device 10, the feature map generating device 20, and the learning model generating device 30 in the embodiments can be realized by using hardware corresponding to each part, instead of a computer with a program installed. Furthermore, the image processing device 10, the feature map generating device 20, and the learning model generating device 30 may each be partially realized by a program and the remaining part by hardware. An example of the hardware is an electronic circuit.

[0082] A part or all of the above-described embodiment can be expressed by (Additional Notes 1) to (Additional Notes 20) described below, but is not limited to the following descriptions.

[0083] (Appendix 1) a feature map generator configured to generate a first feature map that identifies a horizontal position of a person in an image and a second feature map that identifies a vertical position of the person in the image; a grouping unit that groups each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; Equipped with 13. An image processing device comprising:

[0084] (Appendix 2) The feature map generation unit, generating, as the first feature map, a map that is composed of the same number of pixels as the pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a horizontal direction to each of the pixels in an area corresponding to the person; generating, as the second feature map, a map that is composed of the same number of pixels as the pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a vertical direction to each of the pixels in the area that corresponds to the person; 2. An image processing device as described in claim 1.

[0085] (Appendix 3) the numerical value indicating the horizontal position of the person is a value indicating a ratio of a horizontal distance from an origin of the first feature map to a reference point of the person to a horizontal length of the first feature map, the numerical value indicating the vertical position of the person is a value indicating a ratio of a vertical distance from an origin of the second feature map to a reference point of the person to a vertical length of the second feature map; 3. An image processing device as described in claim 2.

[0086] (Appendix 4) The feature map generation unit, generating the first feature map and the second feature map using a learning model that performs machine learning on the relationship between the image and the first feature map and the second feature map; 4. The image processing device according to claim 1,

[0087] (Appendix 5) The grouping unit: for each joint detected in the image, determining, for each person in the image, a value of a pixel corresponding to that joint in the first feature map and a value of a pixel corresponding to that joint in the second feature map; calculating a distance between the joint and the person using the two specified numerical values, a numerical value indicating the position of the person in a horizontal direction, and a numerical value indicating the position of the person in a vertical direction; determining a person corresponding to the joint based on the calculated distance, and performing grouping based on the determination result; 4. The image processing device according to claim 2 or 3.

[0088] (Appendix 6) the grouping unit determines the person corresponding to the joint under conditions that the calculated distance is not equal to or greater than a set value and that a plurality of joints of the same type do not correspond to the same person; 6. The image processing device according to claim 5.

[0089] (Appendix 7) a feature map generator that generates a first feature map that identifies a horizontal position of a person in an image and a second feature map that identifies a vertical position of the person in the image; A feature map generating device.

[0090] (Appendix 8) a learning model generation unit that uses an image including a person, a first feature map that specifies a horizontal position of the person in the image, and a second feature map that specifies a vertical position of the person in the image as training data to generate a learning model that performs machine learning on a relationship between the image and the first feature map and the second feature map; A learning model generation device comprising:

[0091] (Appendix 9) generating a first feature map identifying a horizontal position of a person in an image and a second feature map identifying a vertical position of the person in the image; a grouping step of grouping each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; having 13. An image processing method comprising:

[0092] (Appendix 10) In the feature mapping step, generating, as the first feature map, a map that is composed of the same number of pixels as the pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a horizontal direction to each of the pixels in an area corresponding to the person; generating, as the second feature map, a map that is composed of the same number of pixels as the pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a vertical direction to each of the pixels in the area that corresponds to the person; 10. The image processing method according to claim 9.

[0093] (Appendix 11) the numerical value indicating the horizontal position of the person is a value indicating a ratio of a horizontal distance from an origin of the first feature map to a reference point of the person to a horizontal length of the first feature map, the numerical value indicating the vertical position of the person is a value indicating a ratio of a vertical distance from an origin of the second feature map to a reference point of the person to a vertical length of the second feature map; 11. An image processing method according to claim 10.

[0094] (Appendix 12) In the feature map generating step, generating the first feature map and the second feature map using a learning model that performs machine learning on the relationship between the image and the first feature map and the second feature map; 12. An image processing method according to any one of claims 9 to 11.

[0095] (Appendix 13) In the grouping step, For each joint detected from the image, for each person in the image, a numerical value of a pixel corresponding to the joint in the first feature map and a numerical value of a pixel corresponding to the joint in the second feature map are identified, and a distance between the joint and the person is calculated using the two identified numerical values ​​and a numerical value indicating the horizontal position of the person and a numerical value indicating the vertical position of the person, and a person corresponding to the joint is determined based on the calculated distance, and grouping is performed based on the determination result. 12. The image processing method according to claim 10 or 11.

[0096] (Appendix 14) determining a person corresponding to each joint under the conditions that the calculated distance is not equal to or greater than a set value and that a plurality of joints of the same type do not correspond to the same person in the grouping step; 14. An image processing method according to claim 13.

[0097] (Appendix 15) On the computer, generating a first feature map identifying a horizontal position of a person in an image and a second feature map identifying a vertical position of the person in the image; a grouping step of grouping each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; Run R, Program Hmm.

[0098] (Appendix 16) In the feature mapping step, generating, as the first feature map, a map that is composed of the same number of pixels as the pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a horizontal direction to each of the pixels in an area corresponding to the person; generating, as the second feature map, a map that is composed of the same number of pixels as the pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a vertical direction to each of the pixels in the area that corresponds to the person; As described in Appendix 15 program .

[0099] (Appendix 17) the numerical value indicating the horizontal position of the person is a value indicating a ratio of a horizontal distance from an origin of the first feature map to a reference point of the person to a horizontal length of the first feature map, the numerical value indicating the vertical position of the person is a value indicating a ratio of a vertical distance from an origin of the second feature map to a reference point of the person to a vertical length of the second feature map; As described in Appendix 16 program .

[0100] (Appendix 18) In the feature map generating step, generating the first feature map and the second feature map using a learning model that performs machine learning on the relationship between the image and the first feature map and the second feature map; Any of Supplementary Notes 15 to 17 program .

[0101] (Appendix 19) In the grouping step, For each joint detected from the image, for each person in the image, a numerical value of a pixel corresponding to the joint in the first feature map and a numerical value of a pixel corresponding to the joint in the second feature map are identified, and a distance between the joint and the person is calculated using the two identified numerical values ​​and a numerical value indicating the horizontal position of the person and a numerical value indicating the vertical position of the person, and a person corresponding to the joint is determined based on the calculated distance, and grouping is performed based on the determination result. The method described in Appendix 16 or 17 program .

[0102] (Appendix 20) determining a person corresponding to each joint under the conditions that the calculated distance is not equal to or greater than a set value and that a plurality of joints of the same type do not correspond to the same person in the grouping step; As described in Appendix 19 program .

[0103] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Industrial Applicability]

[0104] As described above, according to the present invention, the association between joints and people is performed regardless of the number of people present in an image. The present invention is useful for a system that estimates the posture of a person from an image. [Explanation of symbols]

[0105] 10 Image processing device 11 Feature map generation unit 12 Grouping Section 13 Image data acquisition section 14 Storage section 15 Joint detection unit 16 Learning Model 17 Image data 20 Feature map generator 21 Image data acquisition unit 22 Memory section 23 Feature map generation unit 24 Learning Model 25 Image data 30 Learning model generation device 31 Training data acquisition section 32 Storage section 33 Learning model generation unit 34 Learning Model 35 Training Data 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display device 120 Recording media 121 Bus

Claims

1. a feature map generator configured to generate a first feature map that identifies a horizontal position of a person in an image and a second feature map that identifies a vertical position of the person in the image; a grouping unit that groups each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; Equipped with 13. An image processing device comprising:

2. The feature map generation unit, generating, as the first feature map, a map that is composed of the same number of pixels as the number of pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a horizontal direction to each of the pixels in an area that corresponds to the person; generating, as the second feature map, a map that is composed of the same number of pixels as the number of pixels that constitute the image, and that assigns a numerical value that indicates a position of the person in a vertical direction to each of the pixels in the area that corresponds to the person; The image processing device according to claim 1 .

3. the numerical value indicating the position of the person in a horizontal direction is a value indicating a ratio of a distance in a horizontal direction from an origin of the first feature map to a reference point of the person to a horizontal length of the first feature map, the numerical value indicating the vertical position of the person is a value indicating a ratio of a vertical distance from an origin of the second feature map to a reference point of the person to a vertical length of the second feature map; The image processing device according to claim 2 .

4. The feature map generation unit, generating the first feature map and the second feature map using a learning model that performs machine learning on the relationship between the image and the first feature map and the second feature map; The image processing device according to claim 1 .

5. The grouping unit: for each joint detected in the image, determining for each person in the image a value of a pixel corresponding to that joint in the first feature map and a value of a pixel corresponding to that joint in the second feature map; calculating a distance between the joint and the person using the two specified numerical values, a numerical value indicating a horizontal position of the person, and a numerical value indicating a vertical position of the person; determining a person corresponding to the joint based on the calculated distance, and performing grouping based on the determination result; The image processing device according to claim 2 .

6. the grouping unit determines the person corresponding to the joint under conditions that the calculated distance is not equal to or greater than a set value and that a plurality of joints of the same type do not correspond to the same person; The image processing device according to claim 5 .

7. a feature map generation step of generating a first feature map identifying a horizontal position of a person in an image and a second feature map identifying a vertical position of the person in the image; a grouping step of grouping each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; having 13. An image processing method comprising:

8. On the computer, a feature map generation step of generating a first feature map identifying a horizontal position of a person in an image and a second feature map identifying a vertical position of the person in the image; a grouping step of grouping each of the joints detected from the image into a corresponding person using horizontal and vertical positions of each of the joints, the first feature map, and the second feature map; A program to execute.

Images