Person estimation program, person estimation method, and person estimation device

Abstract

To provide a person estimation program, a person estimation method, and a person estimation device that improve estimation accuracy of a person.SOLUTION: A person estimation program causes a computer to execute steps: detecting, when receiving a captured image including a person, on the basis of a predetermined object detection method, an area of the person from the captured image; determining, on the basis of a predetermined skeleton estimation method, whether or not orientation of a face included in the area of the person is the front side; identifying, when the orientation of the face is not the front side, on the basis of the predetermined skeleton estimation method, a clothing area for each clothing type of the person from the area of the person; extracting a first color of the clothing area for each clothing type; referring to a storage unit that stores the person and a second color for each clothing type of the clothing area in association with each person; calculating a similarity between the first color and the second color; and estimating, on the basis of the similarity, the person.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to a person estimation program, a person estimation method, and a person estimation device. [Background technology]

[0002] In the field of personal authentication using surveillance cameras, research and development is underway on personal authentication using facial information (see, for example, Patent Document 1). Also known is a technology that compares the facial and clothing features stored in a database with the facial and clothing features of an image provided as a query to determine whether the person is the same person (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-186546 [Patent Document 2] International Publication No. 2011 / 046128 [Patent Document 3] Japanese Patent Application Publication No. 2015-032881 [Patent Document 4] Japanese Patent Publication No. 2020-086823 [Patent Document 5] Japanese Patent Application Laid-Open No. 2010-020602 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where a face is not captured due to the installation location of a surveillance camera, the height of the person being photographed, etc. For example, if a person is facing away from the surveillance camera, it may not be possible to recognize the person using facial information. Also, if a person is facing sideways relative to the surveillance camera, the accuracy of person recognition may decrease even if facial information of the profile is used, and there is a possibility that the person may be misrecognized.

[0005] Therefore, in one aspect, an object is to provide a person estimation program, a person estimation method, and a person estimation device that improve the accuracy of person estimation. [Means for solving the problem]

[0006] In one embodiment, the person estimation program, upon receiving a captured image including a person, detects an area of ​​the person from the captured image based on a predetermined object detection method, determines based on a predetermined skeleton estimation method whether a face included in the area of ​​the person is facing forward, and if the face is not facing forward, Identifying positions of a plurality of joints belonging to the legs of the person based on the predetermined skeleton estimation method, determining whether or not the person is crouching based on the positional relationship of the plurality of joints that have been identified, and if the orientation of the face is not forward and the person is not crouching, The computer executes the following process: based on the predetermined skeleton estimation method, identifying clothing areas for each clothing type of the person from the area of ​​the person; extracting a first color of the clothing area for each clothing type; referring to a memory unit that stores the person and a second color for each clothing type of the clothing area in association with each person; calculating a similarity between the first color and the second color; and estimating the person based on the similarity. [Effects of the Invention]

[0007] This can improve the accuracy of person estimation. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a person estimation device. [Figure 2] FIG. 2 is a block diagram illustrating an example of the functional configuration of the person estimation device. [Figure 3] Fig. 3(a) is an example of the captured image storage unit, and Fig. 3(b) is an example of the correct answer information storage unit. [Figure 4] Fig. 4(a) is an example of face information, and Fig. 4(b) is an example of clothing information. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the person estimation device. [Figure 6]Fig. 6(a) is an example of a captured image including a person to be estimated. Fig. 6(b) is a diagram illustrating an example of a person region. Fig. 6(c) is a diagram illustrating an example of skeleton estimation. Fig. 6(d) is an example of skeleton information. [Figure 7] FIG. 7 is a flowchart showing an example of the clothing region identification process. [Figure 8] Fig. 8(a) is an example of a captured image including a person to be estimated. Fig. 8(b) is an example of an identification image. Fig. 8(c) is an example of a person partial image. Fig. 8(d) is an example of a hat partial image, a top partial image, a bottom partial image, and a shoe partial image. [Figure 9] FIG. 9 is a flowchart showing an example of the clothing color extraction process. [Figure 10] FIG. 10 is a flowchart showing an example of the similarity calculation process. [Figure 11] FIG. 11 is a flowchart showing an example of the similarity calculation process. [Figure 12] FIG. 12 is a diagram illustrating a comparative example. [Figure 13] FIG. 13 is a diagram for explaining an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] First, the hardware configuration of a person estimation device 100 that executes a person estimation method will be described with reference to Fig. 1. The person estimation device 100 includes a CPU (Central Processing Unit) 100A as a processor, and a RAM (Random Access Memory) 100B and a ROM (Read Only Memory) 100C as memories. The person estimation device 100 also includes a network I / F (Interface) 100D and an HDD (Hard Disk Drive) 100E. An SSD (Solid State Drive) may be used instead of the HDD (Hard Disk Drive) 100E.

[0011] The person estimation device 100 may include at least one of an input I / F 100F, an output I / F 100G, an input / output I / F 100H, and a drive device 100I, as necessary. The CPU 100A to the drive device 100I are connected to each other by an internal bus 100J. In other words, the person estimation device 100 can be realized by a computer such as a PC (Personal Computer) or a server.

[0012] An input device 710 is connected to the input I / F 100F. Examples of the input device 710 include a keyboard, a mouse, and a touch panel. A display device 720 is connected to the output I / F 100G. Examples of the display device 720 include a liquid crystal display. A semiconductor memory 730 is connected to the input / output I / F 100H. Examples of the semiconductor memory 730 include a USB (Universal Serial Bus) memory and a flash memory. The input / output I / F 100H reads the person estimation program stored in the semiconductor memory 730. The input I / F 100F and the input / output I / F 100H each include, for example, a USB port. The output I / F 100G includes, for example, a display port.

[0013] A portable recording medium 740 is inserted into the drive device 100I. Examples of the portable recording medium 740 include removable disks such as CD (Compact Disc)-ROMs and DVDs (Digital Versatile Discs). The drive device 100I reads the person estimation program recorded on the portable recording medium 740. The network I / F 100D includes, for example, a LAN (Local Area Network) port and a communication circuit. The communication circuit includes either or both of a wired communication circuit and a wireless communication circuit. The wireless communication circuit is a circuit that can use wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0014] A person estimation program stored in at least one of ROM 100C, HDD 100E, and semiconductor memory 730 is temporarily stored in RAM 100B by CPU 100A. A person estimation program recorded on portable recording medium 740 is temporarily stored in RAM 100B by CPU 100A. By executing the stored person estimation program, CPU 100A realizes various functions described below and executes various processes described below. The person estimation program may be one that corresponds to a flowchart described below.

[0015] The functional configuration of the person estimation device 100 will be described with reference to Fig. 2 to Fig. 4. Note that Fig. 2 shows the main functions of the person estimation device 100.

[0016] As shown in FIG. 2, the person estimation device 100 includes a storage unit 110, a processing unit 120, an input unit 130, an output unit 140, and a communication unit 150. The storage unit 110 can be realized by either or both of the RAM 100B and the HDD 100E described above. The processing unit 120 can be realized by the CPU 100A described above. The input unit 130 can be realized by the input I / F 100F described above. The output unit 140 can be realized by the output I / F 100G described above. The communication unit 150 can be realized by the network I / F 100D described above.

[0017] The memory unit 110, processing unit 120, input unit 130, output unit 140, and communication unit 150 are connected to one another. The memory unit 110 includes a captured image memory unit 111 and a correct answer information memory unit 112. The processing unit 120 includes a person detection unit 121, a skeleton estimation unit 122, a frontal direction determination unit 123, and a clothing identification unit 124. The person detection unit 121 is an example of a detection means. The frontal direction determination unit 123 is an example of a determination means. The clothing identification unit 124 is an example of an identification means. The processing unit 120 also includes a face recognition unit 125, a clothing color extraction unit 126, a similarity calculation unit 127, and a person estimation unit 128. The clothing color extraction unit 126 is an example of an extraction means. The similarity calculation unit 127 is an example of a calculation means. The person estimation unit 128 is an example of an estimation means.

[0018] The captured image storage unit 111 stores a plurality of captured images. In this embodiment, as shown in FIG. 3(a), the captured image storage unit 111 stores captured images IM1 and IM2. Captured image IM1 is associated with the person's name "Fujitsu Taro." Captured image IM2 is associated with the person's name "Fujitsu Hanao." Both captured images IM1 and IM2 include the entire body of a person facing forward and maintaining an upright posture. As will be described in detail later, in the case of sports, the person in this embodiment is preferably an athlete of an individual sport wearing multiple-colored clothing. For example, a golfer wearing golf wear or a tennis player wearing tennis wear is desirable as a person in this embodiment. For this reason, a player of a team sport (such as a baseball player) wearing single-colored clothing (e.g., a uniform) is less desirable as a person in this embodiment than an athlete of an individual sport. Note that clothing includes an upper garment (hereinafter referred to as a top), a lower garment (hereinafter referred to as a bottom), a hat, shoes, etc. Bottoms may be pants or a skirt.

[0019] The captured image can be captured by an imaging device such as a smartphone or a camera. The captured image may be a still image or a video. A video includes a plurality of frame images that are still images. The captured image is stored in the captured image storage unit 111 via the communication unit 150. The captured image may also be stored in the captured image storage unit 111 via an input / output unit (not shown) that can be realized by an input / output I / F.

[0020] The correct answer information storage unit 112 stores multiple pieces of correct answer information. The correct answer information is information in which face information that characterizes the face of a person included in a captured image and clothing information that characterizes the clothing worn by the person are associated with the person's name. In this embodiment, as shown in FIG. 3(b), the correct answer information storage unit 112 stores one piece of correct answer information in which the person's name "Fujitsu Taro", face information "Face1", and clothing information "Cloth1" are associated with each other. The correct answer information storage unit 112 also stores another piece of correct answer information in which the person's name "Fujitsu Hanao", face information "Face2", and clothing information "Cloth2" are associated with each other. The correct answer information is generated and saved in the correct answer information storage unit 112 before executing the person estimation process described below.

[0021] As shown in Figure 4(a), the facial information "Face1" includes Width, which represents the width of the face, Height, which represents the height (or length) of the face, and Chin, which represents the two-dimensional coordinate of the chin. The facial information "Face2" is basically the same as the facial information "Face1," so it is not shown in the figure. The facial information is extracted from the captured image. For example, by using a trained model that has previously learned facial features through machine learning, facial information can be extracted from the captured image.

[0022] As shown in FIG. 4(b), the clothing information "Cloth1" includes body parts such as "foot" and "head." The clothing information "Cloth1" also includes clothing types such as "shoes" and "hat" that correspond to the body parts. Furthermore, the clothing information "Cloth1" includes numerical values ​​that represent the area ratio of the clothing color included in the clothing type and numerical values ​​that represent the clothing color. The numerical values ​​for the clothing color may be in RGB format or Lab format. For example, the clothing type "shoes" indicates that nearly 25% of the shoes are a specific color (e.g., green), nearly 19% are a different color (e.g., red), and nearly 53% are a color different from the specific color and other colors (e.g., purple). Clothing information is extracted from captured images. For example, clothing information can be extracted from captured images by using a trained model that has previously learned clothing features through machine learning.

[0023] Note that the clothing information "Cloth2" is omitted from the illustration because it is basically the same as the clothing information "Cloth1." Also, while the clothing information is shown in JSON format in FIG. 4(b), it may be in a format other than JSON. Furthermore, as will be described in detail later, even if the captured image includes a person's entire body, if the person is crouching or bending over based on the person's skeletal structure estimation, the accuracy of the person's estimation may be reduced, and therefore the extraction of clothing information is excluded.

[0024] When the person detection unit 121 receives a captured image including a person to be estimated, it detects the area of ​​the person from the captured image based on a predetermined object detection method. The predetermined object detection method is a well-known method using deep learning, such as YOLO (You Only Look Once), R-CNN (Region-Convolutional Neural Network), and DETR (Detection Transformer).

[0025] The skeleton estimation unit 122 generates skeleton information including two-dimensional coordinates of each part of a person's joints and face (e.g., left and right eyes, ears, nose (or chin), etc.) and scores indicating the reliability of the two-dimensional coordinates, based on a predetermined skeleton estimation method. Examples of the predetermined skeleton estimation method include well-known methods such as posenet, tf-pose-estimation, and tf-openpose.

[0026] The front-facing determination unit 123 determines whether the facial features included in the person's area are facing forward based on the skeletal information. For example, if the reliability of the two-dimensional coordinates representing the features of the person's face is higher than a predetermined value and twice the distance between the left and right eyes is smaller than the distance from one eye to one ear, the front-facing determination unit 123 determines that the facial features are facing sideways. If the reliability of the two-dimensional coordinates representing the features of the person's face is higher than a predetermined value and both ears are above the eyes, the front-facing determination unit 123 determines that the facial features are facing downward. In this way, the front-facing determination unit 123 identifies the positions of the person's eyes and ears, and determines whether the facial features are facing forward based on the positional relationship between the identified eye and ear positions. On the other hand, if the reliability of the two-dimensional coordinates representing the features of the person's face is lower than a predetermined value, the front-facing determination unit 123 determines that the facial features are facing backward. If the orientation of the face is not sideways, downward, or backward, the front determining unit 123 determines that the orientation of the face is frontal.

[0027] Furthermore, the front face determination unit 123 determines whether the person is in a squatting state based on the skeletal information. More specifically, the front face determination unit 123 determines whether the person is in a squatting state based on the two-dimensional coordinates of the groin joint, knee joint, and ankle joint included in the skeletal information. For example, if the angle between a thigh-equivalent line segment connecting the two-dimensional coordinates of the groin joint and the knee joint and a calf-equivalent line segment connecting the two-dimensional coordinates of the ankle joint and the knee joint is less than a predetermined angle (e.g., 60 degrees), the front face determination unit 123 determines that the person is in a squatting state. If the angle is greater than the predetermined angle, the front face determination unit 123 determines that the person is not in a squatting state. In this way, the front face determination unit 123 identifies the positions of multiple joints belonging to the person's legs and determines whether the person is in a squatting state based on the positional relationship of the identified multiple joints. Note that a squatting state may include a crouching state.

[0028] If the person's face is not facing forward and the person is not crouching, the clothing identification unit 124 identifies the clothing area for each clothing type from the person's area based on the skeletal information. Details will be described later. For example, if the reliability of the two-dimensional coordinates representing the shoulder joints is higher than a predetermined value, the hat area can be identified from the person's area by using a trained model that has previously learned hat features through machine learning. Using a similar method, the clothing identification unit 124 can also identify the areas for tops, bottoms, and shoes.

[0029] The face recognition unit 125 executes face recognition processing to recognize a person's face when the face is facing forward. For example, when a captured image including a person to be estimated facing forward is received, the face recognition unit 125 extracts face information from the captured image using a trained model that has previously learned facial features through machine learning. The face recognition unit 125 recognizes the person's face by comparing the extracted face information with multiple pieces of face information stored in the correct answer information storage unit 112. The face recognition unit 125 calculates similarities between the extracted face information and the multiple pieces of face information, and outputs the names of people associated with the face information from which each similarity was calculated, along with each similarity.

[0030] The clothing color extraction unit 126 extracts clothing colors for each clothing type in the clothing area identified by the clothing identification unit 124. This allows the clothing colors of hats, tops, bottoms, and shoes to be extracted. For example, the clothing color extraction unit 126 extracts clothing colors for each pixel in the clothing area and extracts the area ratio, which is the proportion of the top several colors (e.g., three colors), and the clothing colors. The clothing color extraction unit 126 manages the extracted area ratios and clothing colors as clothing information.

[0031] The similarity calculation unit 127 calculates the similarity between two clothing colors for each clothing type based on the clothing colors of the clothing information managed by the clothing color extraction unit 126 and the clothing colors of multiple pieces of clothing information stored in the correct answer information storage unit 112. As will be described in detail later, after calculating the similarity for each clothing type, the similarity calculation unit 127 assigns a predetermined weight to each similarity to calculate an overall similarity of the clothing information.

[0032] The person estimation unit 128 estimates a person based on the overall similarity calculated by the similarity calculation unit 127. For example, the person estimation unit 128 estimates a person by specifying the person name associated with the clothing information from which the maximum overall similarity was calculated as the person name that identifies the person to be estimated. Note that when the person to be estimated included in the captured image faces forward, the person estimation unit 128 estimates a person based on the person name output by the face recognition unit 125 and each similarity. In this case, the person estimation unit 128 estimates a person by specifying the person name associated with the face information from which the maximum similarity was calculated as the person name that identifies the person to be estimated.

[0033] The operation of the person estimation device 100 will be described with reference to FIGS.

[0034] First, when the communication unit 150 receives a captured image including a person to be estimated, the person detection unit 121 stores the captured image received by the communication unit 150 in the captured image storage unit 111 (step S1). As a result, the captured image storage unit 111 stores the captured image including the person to be estimated. The captured image may include multiple people. In this case, in subsequent processing, each of the multiple people is designated as a person to be estimated.

[0035] When the captured image is saved, the person detection unit 121 detects a person area, which is an area of ​​a person, from the captured image based on the predetermined object detection method described above (step S2). For example, as shown in FIG. 6(a), when a captured image IM3 including a person to be estimated together with a background is saved, the person detection unit 121 acquires the captured image IM3 from the captured image storage unit 111 and detects a rectangular person area AR3 from the captured image IM3, as shown in FIG. 6(b). In this embodiment, the person area AR3 includes a width and a height as two-dimensional position coordinates of the upper left corner of the person area AR3 with an origin O. Furthermore, the person area AR3 includes the horizontal direction, which is the short side direction, as the X axis, and the vertical direction, which is the long side direction, as the Y axis.

[0036] When the person region is detected, the processing unit 120 executes a loop process in which the subsequent process is repeated for each detected person (step S3). Therefore, if multiple people are included in the captured image, the person detection unit 121 detects multiple person regions, and the processing unit 120 executes the loop process for the number of people.

[0037] When the loop process starts, the skeleton estimation unit 122 first estimates the skeleton of the person included in the person area based on the person area and the predetermined skeleton estimation method described above (step S4). After estimating the person's skeleton, the skeleton estimation unit 122 generates skeleton information related to the estimated skeleton. For example, when a person area AR3 is detected as shown in FIG. 6(b), the skeleton estimation unit 122 estimates the skeleton of the person included in the person area AR3 as shown in FIG. 6(c). By estimating the person's skeleton, the skeleton estimation unit 122 can detect the two-dimensional coordinates of each part of the person's joints and face, as well as scores representing the reliability of the two-dimensional coordinates, along with part numbers representing each part. After detecting the two-dimensional coordinates of each part, scores representing the reliability of the two-dimensional coordinates, and part numbers, the skeleton estimation unit 122 generates skeleton information as shown in FIG. 6(d). The numbers "0", "...", "10", etc. included in the skeleton information represent part numbers. The two-dimensional coordinates "x0, y0", ..., and numerical values ​​"x10, y10" included in the skeletal information represent the position of each body part. The scores "cnf0", ..., and numerical values ​​"cnf10" included in the skeletal information represent the reliability of the two-dimensional coordinates. For example, as shown in Figure 6(c), if the face of the person to be estimated is facing backwards, the body part number "0" corresponding to the nose does not appear, and therefore a low score will be calculated.

[0038] After estimating the skeleton, the front face determination unit 123 determines whether the facial face is facing forward (step S5). As described above, if the facial face is not facing sideways, downwards, or backwards, the front face determination unit 123 determines that the facial face is facing forward (step S5: YES). Conversely, if the facial face is facing sideways, downwards, or backwards, the front face determination unit 123 determines that the facial face is not facing forward (step S5: NO).

[0039] If the face is facing forward, the face recognition unit 125 executes face recognition processing (step S6). On the other hand, if the face is not facing forward, the front-face determination unit 123 determines whether the person is crouching (step S7). If the person is crouching (step S7: YES), the process returns to step S3 without executing subsequent processing, since there is a possibility that the accuracy of identifying the clothing area will decrease. In other words, if the face is not facing forward but the person is crouching, the front-face determination unit 123 stops identifying the clothing area for each clothing type of the person.

[0040] As a result, another person detected by the person detection unit 121 is designated as the next processing target for bone structure estimation, etc. On the other hand, if the person is not in a crouching state (step S7: NO), the clothing identification unit 124 executes clothing region identification processing (step S8). The clothing region identification processing is processing for identifying clothing regions for each clothing type from the person region. As will be described in detail later, by performing the clothing region identification processing by the clothing identification unit 124, the clothing identification unit 124 can identify each clothing region of a hat, top, bottoms, and shoes from the person region.

[0041] After the clothing identification unit 124 has completed the clothing area identification process, the clothing color extraction unit 126 then executes a clothing color extraction process (step S9). The clothing color extraction process extracts the clothing colors of clothing such as hats, tops, bottoms, and shoes to generate clothing information. After generating the clothing information, the clothing identification unit 124 manages the generated clothing information. The clothing color extraction process will be described in detail later.

[0042] After the clothing color extraction unit 126 has completed the clothing color extraction process, the similarity calculation unit 127 then executes a similarity calculation process (step S10). The similarity calculation process calculates the similarity between the clothing colors included in the clothing information generated based on the captured image including the person to be estimated and the clothing colors included in the clothing information of the correct answer information, as well as the overall similarity described above. The similarity calculation process will be described in detail later.

[0043] When the similarity calculation unit 127 finishes executing the similarity calculation process, the person estimation unit 128 then estimates the person to be estimated (step S11). As described above, the person estimation unit 128 estimates the person by identifying the person name associated with the clothing information from which the maximum overall similarity was calculated as the person name that identifies the person to be estimated. When the person estimation unit 128 estimates the person, the processing unit 120 determines whether to end the loop processing (step S12). If another person for which estimation has not been completed has been detected by the person detection unit 121, the processing unit 120 designates the other person as the next processing target, and returns to the processing of step S3. On the other hand, if all people detected by the person detection unit 121 have been estimated, the processing unit 120 ends the loop processing and terminates the person estimation processing.

[0044] The clothing region identification process will be described with reference to FIGS.

[0045] First, as shown in FIG. 7, the clothing identification unit 124 removes background portions from a captured image (step S21). For example, as shown in FIG. 8(a), when the clothing identification unit 124 receives a captured image IM3, the clothing identification unit 124 removes background portions from the captured image using a trained model that has previously learned human features through machine learning. More specifically, as shown in FIG. 8(b), the clothing identification unit 124 distinguishes a human portion P1 and a background portion P2 from the captured image IM3 based on the trained model related to humans. After distinguishing the human portion P1 and the background portion P2, the clothing identification unit 124 generates an identification image IM4 that distinguishes the human portion P1 from the background portion P2 using different monochrome colors.

[0046] After generating the identification image IM4, the clothing identification unit 124 generates a person partial image IM5 by excluding the background portion P2 from the captured image IM3, as shown in Fig. 8(c), based on the captured image IM3 and the identification image IM4. For example, the clothing identification unit 124 can generate the person partial image IM5 by overlaying the identification image IM4 on the captured image IM3 and performing image processing to leave the person portion P1 and delete the background portion P2. Generating the person partial image IM5 can prevent the background portion P2 from being erroneously extracted as a clothing area in subsequent processing.

[0047] After removing the background portion, the clothing identification unit 124 determines whether the two-dimensional coordinates representing the shoulder joint have a high score (step S22). For example, if the score is expressed as a percentage, whether the score is a predetermined value X% or more can be used to determine whether the score is a high score. If the score is a high score (step S22: YES), the clothing identification unit 124 uses a trained model that has previously learned hat features through machine learning to identify a hat region from the person partial image and generate a hat partial image (step S23). As a result, as shown in FIG. 8(d), a hat partial image IM6 can be generated from the person partial image IM5. If the score is not high (step S22: NO), i.e., if the score is low, the clothing identification unit 124 skips the processing of step S23. Thus, if the score of the two-dimensional coordinates representing the shoulder joint is low, the accuracy of identifying the hat region may be reduced, so the clothing identification unit 124 skips the processing of step S23.

[0048] Next, the clothing identification unit 124 determines whether both the two-dimensional coordinates representing the shoulder joints and the waist joints have high scores (step S24). If they have high scores (step S24: YES), the clothing identification unit 124 uses a trained model that has previously learned the features of tops through machine learning to identify a top area from the partial person image and generate a partial top image (step S25). This allows a partial top image IM7 to be generated from the partial person image IM5, as shown in FIG. 8(d). If they do not have high scores (step S24: NO), the clothing identification unit 124 skips the processing of step S25, as in the case of a hat.

[0049] Next, the clothing identification unit 124 determines whether both the two-dimensional coordinates representing the waist joints and the ankle joints have high scores (step S26). If they have high scores (step S26: YES), the clothing identification unit 124 uses a trained model that has previously learned about pants characteristics through machine learning to identify a pants region from the person partial image and generate a pants partial image (step S27). After generating the pants partial image, the clothing identification unit 124 uses a trained model that has previously learned about skirt characteristics through machine learning to identify a skirt region from the person partial image and generate a skirt partial image (step S28). After generating the pants partial image and the skirt partial image, the clothing identification unit 124 identifies either the pants region or the skirt region as the bottoms region and generates a bottoms partial image (step S29).

[0050] Although the shapes of pants and skirts differ significantly in bottoms, they cannot be determined from the image alone. Therefore, the clothing identification unit 124 generates a pants partial image and a skirt partial image for each of the pants and skirt. The clothing identification unit 124 then applies an argmax function to the pants partial image and the skirt partial image as arguments, determining the partial image from which a larger number of pixels can be extracted as the bottoms partial image. In this embodiment, since the person is wearing pants, a bottoms partial image IM8 can be generated from the person partial image IM5, as shown in FIG. 8(d). Note that if the score is not high (step S26: NO), the clothing identification unit 124 skips steps S27 to S29, as in the case of a hat.

[0051] Next, the clothing identification unit 124 determines whether the two-dimensional coordinates representing the ankle joint have a high score (step S30). If the two-dimensional coordinates represent a high score (step S30: YES), the clothing identification unit 124 uses a trained model that has previously learned shoe features through machine learning to identify a shoe region from the person partial image and generate a shoe partial image (step S31). This allows a shoe partial image IM9 to be generated from the person partial image IM5, as shown in FIG. 8(d). If the two-dimensional coordinates represent a high score (step S30: NO), the clothing identification unit 124 skips the processing of step S31, as in the case of a hat. When the processing of step S30 or step S31 is completed, the clothing identification unit 124 ends the clothing region identification processing.

[0052] The above-mentioned clothing color extraction process will be described with reference to FIG.

[0053] When the clothing area identification process is completed, the clothing color extraction unit 126 extracts several representative colors from the hat portion image (step S41). Specifically, the clothing color extraction unit 126 extracts the color of each pixel in the hat portion image as the hat color, associates the area ratio, which is the proportion of the top several colors, with the hat color, and manages this as clothing information for the hat. The top several colors may be one color, or two or more colors. For example, if the hat color is a single color, white, the clothing color extraction unit 126 associates the area ratio "100%" with the hat color "white," and manages this as clothing information for the hat.

[0054] Next, the clothing color extraction unit 126 extracts several representative colors from the top partial image (step S42). Specifically, the clothing color extraction unit 126 extracts the color of each pixel in the top partial image as the top color, associates the area ratio, which is the percentage of the top several colors, with the hat color, and manages this as clothing information for the top. For example, if the top colors are two colors, white and navy blue, the clothing color extraction unit 126 associates, for example, an area ratio of "85%" with the top color "white," and an area ratio of "15%" with the top color "navy blue," and manages this as clothing information for the top.

[0055] Next, the clothing color extraction unit 126 extracts several representative colors from the bottom clothing partial image (step S43). Specifically, the clothing color extraction unit 126 extracts the color of each pixel in the bottom clothing partial image as a bottom clothing color, associates the bottom clothing color with an area ratio, which is the percentage occupied by the top several colors, and manages this as bottom clothing information. For example, if the bottom clothing color is a single color, black, the clothing color extraction unit 126 associates an area ratio of "100%" with the bottom color "black" and manages this as bottom clothing information.

[0056] Next, the clothing color extraction unit 126 extracts several representative colors from the shoe portion image (step S44). Specifically, the clothing color extraction unit 126 extracts the color of each pixel in the shoe portion image as a shoe color, associates the shoe color with an area ratio, which is the proportion of the top several colors, and manages this as clothing information for the shoe. For example, if the shoe color is a single color, white, the clothing color extraction unit 126 associates an area ratio of "100%" with the shoe color "white" and manages this as clothing information for the shoe. When the processing of step S24 is completed, the clothing color extraction unit 126 ends the clothing color extraction process.

[0057] The above-mentioned similarity calculation process will be described with reference to FIG.

[0058] 10, the similarity calculation unit 127 starts a loop process of executing subsequent processes for the number of pieces of correct answer information (step S61). More specifically, the similarity calculation unit 127 acquires each piece of clothing information of the hat, top, bottom, and shoes managed by the clothing color extraction unit 126, and also acquires clothing information of multiple pieces of correct answer information from the correct answer information storage unit 112, and starts the loop process.

[0059] When the loop process starts, the similarity calculation unit 127 specifies one piece of correct answer information and executes a similarity calculation process for the hat (step S62). The similarity calculation process is a process for calculating the similarity between the clothing color included in the clothing information to be processed and the clothing color included in the clothing information of the correct answer information. As will be described in detail later, the similarity calculation unit 127 executes the similarity calculation process for the hat, thereby calculating the similarity for the hat.

[0060] Next, the similarity calculation unit 127 executes a similarity calculation process for the tops (step S63). As the similarity calculation unit 127 executes the similarity calculation process for the tops, the similarity calculation unit 127 calculates the similarity for the tops. Next, the similarity calculation unit 127 executes a similarity calculation process for the bottoms (step S64). As the similarity calculation unit 127 executes the similarity calculation process for the bottoms, the similarity calculation unit 127 calculates the similarity for the bottoms. Next, the similarity calculation unit 127 executes a similarity calculation process for the shoes (step S65). As the similarity calculation unit 127 executes the similarity calculation process for the shoes, the similarity calculation unit 127 calculates the similarity for the shoes.

[0061] After calculating the similarity for the shoes, the similarity calculation unit 127 calculates the overall similarity (step S66). More specifically, the similarity calculation unit 127 calculates the overall similarity by assigning a predetermined weight to each of the similarities for the hat, top, bottoms, and shoes. The predetermined weight can be set arbitrarily for each body part. For example, a weight of "0.2" can be set for each of the head and feet, and a weight of "0.3" can be set for each of the upper body and lower body. Thus, if the similarity for the hat is calculated to be "0.8," the similarity for the top to be "0.9," the similarity for the bottoms to be "0.7," and the similarity for the shoes to be "0.6," the similarity calculation unit 127 calculates the overall similarity as follows:

[0062] Overall similarity = hat similarity "0.8" x head weight "0.2" + Top similarity "0.9" x upper body weight "0.3" + Bottom similarity "0.7" x lower body weight "0.3" + Shoe similarity "0.6" x lower body weight "0.2" =0.76

[0063] After calculating the overall similarity, the similarity calculation unit 127 determines whether or not there is any unspecified correct answer information (step S67). If there is any unspecified correct answer information, the next correct answer information is newly specified, and the processes of steps S62 to S66 are repeated. On the other hand, if all correct answer information has been specified, the similarity calculation unit 127 ends the similarity calculation process.

[0064] The above-mentioned similarity calculation process will be described with reference to FIG.

[0065] First, the similarity calculation unit 127 selects the representative color A with the largest area ratio (step S81). For example, when calculating the similarity for a top, the area ratio of the top is "85%" for the clothing color "white" and the area ratio of the top is "15%" for the clothing color "navy blue", so the similarity calculation unit 127 selects the clothing color "white" with the largest area ratio as the representative color A. In this embodiment, the hat, bottoms, and shoes are all monochromatic, but if the hat, bottoms, and shoes are not monochromatic but have multiple colors, the similarity calculation unit 127 similarly selects the representative color A.

[0066] Next, the similarity calculation unit 127 selects a representative color B of the correct information that is similar to the representative color A (step S82). For example, when selecting a representative color B for a top, if the representative color B has three colors, an off-white color that is close to white, a navy blue, and black, the similarity calculation unit 127 selects the off-white color of the correct information as the representative color B that is similar to the representative color A, regardless of the area ratio.

[0067] Next, the similarity calculation unit 127 calculates the similarity between the representative color A and the representative color B (step S83). The similarity between the representative color A and the representative color B is calculated by the following calculation formula (1). <Calculation formula (1)> Similarity + = color difference (representative color A, representative color B) ×min (area ratio of representative color A, area ratio of representative color B) That is, the similarity calculation unit 127 multiplies the color difference between representative color A and representative color B by the smaller of the area ratio of representative color A and the area ratio of representative color B, and adds the result as the similarity. This makes it possible to suppress variations in similarity due to the size of clothing appearing in the captured image. After calculating the similarity, the similarity calculation unit 127 manages the area ratio used in the similarity calculation as the used area ratio.

[0068] Next, the similarity calculation unit 127 calculates the total used area ratio (step S84). The total used area ratio is calculated by the following calculation formula (2). <Calculation formula (2)> Total area ratio used += min (area ratio of representative color A, area ratio of representative color B)

[0069] Next, the similarity calculation unit 127 determines whether the total used area ratio is 100% or more (step S85). If the total used area ratio is less than 100% (step S85: NO), the similarity calculation unit 127 repeats the processes of steps S81 to S84. That is, if calculation of all area ratios has not been completed, the similarity calculation unit 127 repeats the processes of steps S81 to S84. On the other hand, if the total used area ratio is 100% or more (step S85: YES), the similarity calculation unit 127 ends the similarity calculation process. In this way, by executing the similarity calculation process, the similarity of each processing target, such as a hat or a top, is calculated.

[0070] The present embodiment will be described in comparison with a comparative example with reference to Figures 12 and 13. Figure 12 shows the comparative example, and Figure 13 shows the embodiment.

[0071] First, referring to FIG. 12 , a case will be described in which a person facing away from the camera included in captured image C is estimated based on correct image A of a person named "Fujitsu Taro" and correct image B of a person named "Fujitsu Hanao." In this case, the face does not appear in captured image C, making it difficult to estimate the person based on face recognition processing. Even if the facial similarity between correct image A and captured image C and the facial similarity between correct image B and captured image C are calculated using face recognition processing, it is expected that there will be little discrepancy between the two similarities. Therefore, there is a possibility that the name of the person included in captured image C will be estimated to be the name of the person named in correct image B, "Fujitsu Hanao."

[0072] On the other hand, as shown in FIG. 13 , when estimating a person facing away from the camera in captured image C, which is the same as the comparative example, based on the same correct image A and correct image B as the comparative example, person estimation based on face recognition processing is difficult, as in the comparative example. However, in this embodiment, it is possible to calculate the clothing color similarity for each clothing type between correct image A and captured image C, and the clothing color similarity for each clothing type between correct image B and captured image C, without performing face recognition processing. Then, by calculating the overall similarity based on the clothing color similarity for each clothing type, it is possible to determine the similarity of the clothing as a whole. In this way, even if the person in captured image C is facing away from the camera, by utilizing the clothing color characteristics such as the hat and top, the two overall similarities significantly differ, allowing the person's name to be uniquely estimated. In this embodiment, the likelihood that the person's name in captured image C will be correctly estimated as the person's name in correct image A, "Fujitsu Taro," is improved.

[0073] As described above, according to this embodiment, correct answer information in which a person's name, face information, and clothing information are associated with each other is provided in advance (specifically, before the person estimation process is executed). Therefore, even when the face of the person to be estimated cannot be recognized, the person can be estimated using the clothing information. Furthermore, the similarity of clothing colors is calculated using multiple representative colors obtained from each piece of clothing, such as a hat or top, and the area ratio of the representative colors. This allows for accurate person estimation not only for people wearing clothing of a single color, but also for people wearing clothing containing multiple colors.

[0074] Furthermore, if the posture of the person to be estimated is significantly different from that of a person standing upright in the target image, such as when the person is crouching, it may be difficult to accurately generate a hat partial image or a top partial image. However, according to this embodiment, if the person to be estimated is crouching, person estimation based on the similarity in clothing color is excluded, so that the person can be accurately estimated. In particular, the person estimation device according to this embodiment works well in sports where participants wear different colors of clothing, such as golf wear or tennis wear, compared to sports where participants wear the same color of clothing, such as baseball uniforms.

[0075] Although the preferred embodiment of the present invention has been described above in detail, it is not limited to the specific embodiment of the present invention, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims. For example, although the clothing area identification process described above is performed serially for each clothing type, it may also be performed in parallel for each clothing type.

[0076] In addition, the following supplementary notes are provided in relation to the above description. (Supplementary Note 1) A person estimation program that causes a computer to execute a process that, upon receiving a captured image including a person, detects an area of ​​the person from the captured image based on a predetermined object detection method, determines based on a predetermined skeleton estimation method whether a facial area included in the area of ​​the person is facing forward, and if the facial area is not facing forward, identifies clothing areas for each clothing type of the person from the area of ​​the person based on the predetermined skeleton estimation method, extracts first colors of the clothing areas for each clothing type, refers to a memory that stores the person and second colors for each clothing type of the clothing areas in association with each person, calculates a similarity between the first color and the second color, and estimates the person based on the similarity. (Appendix 2) The person estimation program described in Appendix 1, characterized in that the memory unit further associates and stores the face of the person, and includes a process of recognizing the facial part, the recognition process recognizing the facial part when the facial part is facing forward, and the estimation process estimating the person based on the similarity between the recognized facial part and the face of the person stored in the memory unit. (Appendix 3) The person estimation program according to appendix 1 or 2, characterized in that the process of determining identifies the positions of the person's eyes and ears based on the predetermined skeletal estimation method, and determines whether the facial area is facing forward based on the positional relationship between the identified eye positions and ear positions. (Appendix 4) The person estimation program according to any one of Appendices 1 to 3, wherein the determining process identifies positions of a plurality of joints belonging to the person's legs based on the predetermined skeletal estimation method, and determines whether the person is crouching or not based on the positional relationship of the identified plurality of joints, and the identifying process identifies the clothing area of ​​the person for each clothing type when the facial direction is not facing forward and the person is not crouching. (Appendix 5) The person estimation program according to Appendix 4, characterized in that the identification process identifies the clothing area of ​​the person for each clothing type when the facial area is not facing forward, the person is not crouching, and the skeleton estimation accuracy by the predetermined skeleton estimation method is equal to or greater than a predetermined value. (Appendix 6) The person estimation program according to any one of Appendices 1 to 3, wherein the determining process identifies positions of a plurality of joints belonging to the person's legs based on the predetermined skeletal estimation method, and determines whether the person is crouching based on the positional relationship of the identified plurality of joints, and the identifying process stops identifying the clothing area of ​​the person for each clothing type if the facial direction is not facing forward and the person is crouching. (Supplementary Note 7) The person estimation program according to any one of Supplementary Notes 1 to 6, wherein at least one of the first color and the second color includes a plurality of colors. (Supplementary Note 8) The person estimation program according to any one of Supplementary Notes 1 to 7, wherein the people are participants in a competition wearing clothing of different colors. (Appendix 9) A person estimation method in which a computer executes the following processes: upon receiving a captured image including a person, detects an area of ​​the person from the captured image based on a predetermined object detection method; determines based on a predetermined skeleton estimation method whether a facial part included in the area of ​​the person is facing forward; if the facial part is not facing forward, identifies clothing areas for each clothing type of the person from the area of ​​the person based on the predetermined skeleton estimation method; extracts first colors of the clothing areas for each clothing type; refers to a memory unit that stores the person and second colors for each clothing type of the clothing areas in association with each person, calculates a similarity between the first color and the second color; and estimates the person based on the similarity. (Supplementary Note 10) A person estimation device comprising: a detection means for detecting a region of the person from the captured image based on a predetermined object detection method when receiving a captured image including a person; a determination means for determining whether a face included in the region of the person is facing forward based on a predetermined skeleton estimation method; an identification means for identifying a clothing region for each clothing type of the person from the region of the person based on the predetermined skeleton estimation method if the face is not facing forward; an extraction means for extracting a first color of the clothing region for each clothing type; a calculation means for calculating a similarity between the first color and the second color by referring to a memory unit that stores the person and a second color for each clothing type of the clothing region in association with each person; and an estimation means for inferring the person based on the similarity. [Explanation of symbols]

[0077] 100 Person estimation device 110 Storage section 111 Captured image storage unit 112 Correct answer information storage unit 120 Processing section 121 Person detection unit 122 Skeleton Estimation Section 123 Front judgment part 124 Clothing Specification Department 125 Face Recognition Unit 126 Clothing Color Extraction Department 127 Similarity Calculation Part 128 Character Prediction Department

Claims

1. When a captured image including a person is received, an area of ​​the person is detected from the captured image based on a predetermined object detection method; determining whether a face included in the person's region is facing forward based on a predetermined skeleton estimation method; if the face is not facing forward, identifying positions of a plurality of joints belonging to the legs of the person based on the predetermined skeletal structure estimation method, and determining whether the person is crouching based on the positional relationship of the identified plurality of joints; if the face is not facing forward and the person is not crouching, identifying a clothing area for each clothing type of the person from the area of ​​the person based on the predetermined skeleton estimation method; extracting a first color of the clothing area for each clothing type; calculating a similarity between the first color and the second color by referring to a storage unit that stores the person and the second color for each clothing type of the clothing area in association with each person; estimating the person based on the similarity; A person estimation program that causes a computer to execute the process.

2. the storage unit further associates and stores the face of the person; a process of recognizing the facial region; the recognition process recognizes the face when the face is facing forward; the estimating process estimates the person based on a similarity between the recognized facial part and a face of the person stored in the storage unit; 2. The person estimation program according to claim 1, wherein:

3. the determining process includes identifying positions of the person's eyes and ears based on the predetermined skeleton estimation method, and determining whether the facial part is facing forward based on the positional relationship between the identified positions of the eyes and ears.

3. The person estimation program according to claim 1, wherein the person estimation program is a program for estimating a person from among a plurality of people.

4. the identifying process identifies a clothing area of ​​the person for each clothing type when the orientation of the face is not to the front, the person is not crouching, and the skeleton estimation accuracy by the predetermined skeleton estimation method is equal to or greater than a predetermined value; 2. The person estimation program according to claim 1, wherein:

5. The process of identifying includes halting identification of the clothing area of ​​the person for each clothing type when the facial area is not facing forward and the person is crouching.

3. The person estimation program according to claim 1, wherein the person estimation program is a program for estimating a person from among a plurality of people.

6. When a captured image including a person is received, an area of ​​the person is detected from the captured image based on a predetermined object detection method; determining whether a face included in the person's region is facing forward based on a predetermined skeleton estimation method; if the face is not facing forward, identifying positions of a plurality of joints belonging to the legs of the person based on the predetermined skeletal structure estimation method, and determining whether the person is crouching based on the positional relationship of the identified plurality of joints; if the face is not facing forward and the person is not crouching, identifying a clothing area for each clothing type of the person from the area of ​​the person based on the predetermined skeleton estimation method; extracting a first color of the clothing area for each clothing type; calculating a similarity between the first color and the second color by referring to a storage unit that stores the person and the second color for each clothing type of the clothing area in association with each person; estimating the person based on the similarity; A person estimation method in which processing is performed by a computer.

7. a detection means for detecting an area of ​​a person from a captured image based on a predetermined object detection method when the captured image includes the person; a determining means for determining whether a face included in the area of ​​the person faces forward based on a predetermined skeletal structure estimation method, and if the face does not face forward, specifying positions of a plurality of joints belonging to the person's legs based on the predetermined skeletal structure estimation method, and determining whether the person is crouching based on the positional relationship of the specified joints; an identification means for identifying a clothing area for each clothing type of the person from the area of ​​the person based on the predetermined skeleton estimation method when the orientation of the face is not to the front and the person is not crouching; an extraction means for extracting a first color of the clothing area for each clothing type; a calculation means for calculating a similarity between the first color and the second color by referring to a storage unit that stores the person and the second color for each clothing type in the clothing area in association with each person; an estimation means for estimating the person based on the similarity; A person estimation device having the above.

Images