Image processing device, image processing method, and program

Abstract

The present invention involves: detecting a face region of a person captured in an image; detecting a body region of the person captured in the image; performing a face comparison process using image information about the face region; identifying, when the image information about the face region and image information about the body region satisfies a predetermined correspondence relationship, a correspondence relationship between the image information about the face region and the image information about the body region; and recording the image information about the body region when the image information about the body region of the person identified as a result of the face comparison process satisfies a recording condition.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In many cases, person authentication is performed using facial feature information. Patent Document 1 discloses an authentication processing technique. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 136795 Summary of the Invention [Problem to be solved by the invention]

[0004] When facial feature information cannot be obtained from an image, authentication using other body features, clothing, etc. is being considered. Here, there is a demand for highly accurate authentication of a person multiple times over a long period of time, even when facial features cannot be recognized.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an image processing device, an image processing method, and a program that solve the above-mentioned problems. [Means for solving the problem]

[0006] According to a first aspect of the present disclosure, an image processing device includes a face detection means for detecting a face region of a person appearing in an image, a body detection means for detecting a body region of the person appearing in the image, a face matching means for performing a face matching process using image information of the face region, a correspondence determination means for determining a correspondence relationship between the image information of the face region and the image information of the body region when the image information of the face region and the image information of the body region satisfy a predetermined correspondence relationship, and an image recording means for recording the image information of the body region of the person determined as a result of the face matching process, and the image recording means records the image information of the body region when the image information of the body region satisfies a recording condition.

[0007] According to a second aspect of the present disclosure, an image processing method detects a facial region of a person appearing in an image, detects a body region of the person appearing in the image, performs a face matching process using image information of the facial region, identifies a correspondence between the image information of the facial region and the image information of the body region if the image information of the facial region and the image information of the body region satisfy a predetermined correspondence relationship, and records the image information of the body region if the image information of the person's body region identified as a result of the face matching process satisfies a recording condition.

[0008] According to a third aspect of the present disclosure, the program causes a computer of an image processing device to function as a face detection means for detecting a face region of a person appearing in an image, a body detection means for detecting a body region of the person appearing in the image, a face matching means for performing face matching processing using image information of the face region, a correspondence relationship identification means for identifying a correspondence relationship between the image information of the face region and the image information of the body region when the image information of the face region and the image information of the body region satisfy a predetermined correspondence relationship, and an image recording means for recording the image information of the body region when the image information of the person's body region identified as a result of the face matching processing satisfies a recording condition. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of a verification system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a diagram illustrating a hardware configuration of an image processing device according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a functional block diagram of an image processing device according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a first diagram illustrating a relationship between a face image and a body image according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a second diagram illustrating a relationship between a face image and a body image according to an embodiment of the present disclosure. [Figure 6] FIG. 13 is a third diagram illustrating a relationship between a face image and a body image according to an embodiment of the present disclosure. [Figure 7] FIG. 4 is a fourth diagram illustrating a relationship between a face image and a body image according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a diagram showing a first processing flow of the image processing device according to the first embodiment of the present disclosure. [Figure 9] FIG. 11 is a diagram showing a second processing flow of the image processing device according to the first embodiment of the present disclosure. [Figure 10] FIG. 11 is a functional block diagram of an image processing device according to a second embodiment of the present disclosure. [Figure 11] FIG. 11 is a functional block diagram of an image processing device according to a third embodiment of the present disclosure. [Figure 12] FIG. 13 is a diagram showing a process flow according to a fourth embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram showing a process flow according to a fifth embodiment of the present disclosure. [Figure 14] FIG. 1 is a diagram illustrating a minimum configuration of an image processing device. [Figure 15] FIG. 11 is a diagram showing a processing flow by an image processing device with a minimum configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An image processing device according to an embodiment of the present disclosure will now be described with reference to the drawings. FIG. 1 is a schematic diagram of a collation system according to this embodiment. The verification system 100 includes, as examples, an image processing device 1, a camera 2, and a display device 3. It is sufficient that the verification system 100 includes at least the image processing device 1. In this embodiment, the image processing device 1 is connected to a plurality of cameras 2 and the display device 3 via a communication network. For ease of explanation, only one camera 2 is illustrated in FIG. 1. The image processing device 1 acquires a photographed image of a person to be processed from the camera 2. As an example, the image processing device 1 performs a process of verifying a person, a process of tracking a person, and the like, using the photographed image of a person acquired from the camera 2.

[0011] The matching process performed by the image processing device 1 refers to, for example, a process of using face images including face regions or body images including body regions of a plurality of persons stored in the image processing device 1 and a photographed image including face regions or body regions acquired from the camera 2 to identify a face image or body image of a person appearing in a photographed image acquired from the camera 2 from among the plurality of face images or body images stored in the image processing device 1. Details of the face images and body images will be described below with reference to Figs. 4 to 7.

[0012] FIG. 2 is a diagram showing the hardware configuration of the image processing device. 2, the image processing device 1 is a computer equipped with various hardware components such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a database 104, and a communication module 105. The display device 3 is also a computer equipped with a similar hardware configuration.

[0013] FIG. 3 is a functional block diagram of the image processing device. In the image processing device 1, a CPU 101 executes an image processing program stored in a ROM 102 or the like. In this way, the image processing device 1 exerts the functions of an input unit 11, a record determination unit 12, and a matching unit 13. The input unit 11 acquires a face image from the camera 2 . The recording decision unit 12 decides whether to record a face image or a record image. The collation unit 13 performs a collation process.

[0014] The record determination unit 12 exerts the functions of a face detection unit 21 , a body detection unit 22 , a correspondence relationship specification unit 23 , a face matching unit 24 , and an image recording unit 25 . The face detection unit 21 detects a face area appearing in the photographed image acquired from the camera 2 . The body detection unit 22 detects a body region appearing in the captured image acquired by the camera 2 . The correspondence specifying section 23 specifies the correspondence between a face image indicating a face region detected by the face detection section 21 and a body image indicating a body region detected by the body detection section 22 . The face matching unit 24 performs face matching processing using image information of the face area. When the face matching process is successful and a person is identified, the image recording unit 25 records a body image as information about the person. When a person is identified, the image recording unit 25 may further record a face image as information about the person.

[0015] The matching unit 13 performs face matching processing or body matching processing using the face image or body image recorded by the record determination unit 12. The matching unit 13 exerts the functions of a face detection unit 31, a face matching unit 32, a body detection unit 33, a body matching unit 34, and an output unit 35. The face detection unit 31 detects a face area appearing in the photographed image acquired by the camera 2 . The face matching unit 32 performs face matching processing using image information of the face area. The face matching processing uses a face matching program. The body detection unit 33 detects a body region appearing in the captured image acquired by the camera 2 . The body matching unit 34 performs a body matching process using the image information of the body region. The body matching process uses a body matching program. The output unit 35 outputs the processing result of the body matching unit 34 or the face matching unit 32.

[0016] The face matching program is a program that learns a plurality of face images and teacher data corresponding to the face images using machine learning processing such as a neural network, and at least calculates the degree of matching between the input face image and the face image to be compared. More specifically, the image processing device 1, for example, takes a face image including the entire face as input information, and uses a degree of matching indicating the likelihood of a correct answer (i.e., that the face image of the same person as the face image of the input information) regarding the plurality of face images to be compared recorded in the database as output information, and learns the input / output relationship using machine learning processing such as a neural network to generate a face matching model. The image processing device 1 generates a face matching program including a program constituting a face matching model and a neural network. The image processing device 1 may use a known technology to generate a face matching model that takes a face image including the entire face as input information and calculates the degree of matching regarding the plurality of face images to be compared recorded in the database.

[0017] The body matching program is a program that learns a plurality of body images and teacher data corresponding to the body images using machine learning processing such as a neural network, and at least calculates the degree of matching between the input body image and the body image to be compared. More specifically, the image processing device 1, for example, uses a body image as input information, and uses a degree of matching indicating the likelihood of a correct answer (i.e., that the body image of the input information is a body image of the same person) regarding the plurality of body images to be compared recorded in the database as output information, and learns the input / output relationship using machine learning processing such as a neural network to generate a body matching model. The image processing device 1 generates a body matching program including a program constituting the body matching model and a neural network. The image processing device 1 may use a known technology to generate a body matching model that uses a body image as input information and calculates the degree of matching regarding the plurality of body images to be compared recorded in the database.

[0018] The verification system 100 of the present disclosure is, as an example, an information processing system used to verify a person entering a predetermined area multiple times within the predetermined area. For example, if the predetermined area is a theme park, the verification process is performed multiple times when a person enters the theme park or at a predetermined location within the theme park (for example, an entrance to an attraction or an entrance to a store). Alternatively, the predetermined area may be a predetermined region (a country, a prefecture, a region), a public facility, a building, an office, etc. In this case, the verification system 100 is an information processing system used to verify a person multiple times within a predetermined area such as a predetermined region (a country, a prefecture, a region), a public facility, a building, an office, etc.

[0019] FIG. 4 is a first diagram showing the relationship between a face image and a body image. As shown in Fig. 4, the face image m1 may be an image region that includes a face region but not a body region, and as shown in Fig. 4, the body image m2 may be an image region that includes the entire body from head to toe, including the face, arms, legs, and torso.

[0020] FIG. 5 is a second diagram showing the relationship between the face image and the body image. As shown in Fig. 5, the face image m1 may be an image region that includes a face region but not a body region, and the body image m2 may be an image region that includes the entire body from neck to toes, such as arms, legs, and torso, but does not include a face region.

[0021] FIG. 6 is a third diagram showing the relationship between a face image and a body image. As shown in Fig. 6, the face image m1 may be an image area that includes the face area but not the body area, and the body image m2 may be an image area that includes the arms, torso, and the like from the neck to the waist and crotch area but does not include the face area.

[0022] FIG. 7 is a fourth diagram showing the relationship between a face image and a body image. As shown in Fig. 7, face image m1 may be an image area that includes a face area but does not include a body area, and body image m2 may be an image area that does not include a face area and does not include the torso from the neck to the waist or legs near the crotch.

[0023] As shown in Figures 4 to 7, the body area included in the body image may be appropriately determined. The area included in the body image may be only the information of the upper body clothes. The area included in the face image or body image may be an image including only the face area or the body area with the background cut out.

[0024] First Embodiment FIG. 8 is a diagram showing a first processing flow of the image processing device according to the first embodiment. The first processing flow shows an example of a case where a person enters a predetermined area. When a person M enters a predetermined area or passes a predetermined position, a camera 2 installed at a person shooting position at the entry position or the passing position captures the person M. The camera 2 transmits shooting information including a captured image of the person M and the ID of the camera 2 to the image processing device 1. The input unit 11 of the image processing device 1 acquires the shooting information from the camera 2 (step S101). The input unit 11 of the image processing device 1 acquires the ID of the camera 2 included in the shooting information. Based on the ID of the camera 2, the input unit 11 judges whether the camera 2 is a camera installed at a position where a recording judgment of a person appearing in a captured image is performed, such as an entry position or a predetermined person shooting position (step S102). The input unit 11 reads the camera type corresponding to the ID of the camera 2 based on a record in a camera type table of a database 104 that stores a correspondence relationship between the ID of the camera 2 and information indicating the camera type. When the camera type indicates that the camera type is a type where a recording judgment is performed, the input unit 11 outputs the shooting information to the recording judgment unit 12. If the camera type does not indicate that the camera type is a type for which recording determination is to be performed, the input unit 11 outputs the shooting information to the collation unit 13 .

[0025] The recording determination unit 12 acquires the photographing information from the input unit 11. In the recording determination unit 12, the face detection unit 21 reads the photographed image from the photographing information. The face detection unit 21 determines whether a face can be detected in the photographed image (step S103). A known technique may be used to detect a face in the photographed image. For example, the face may be detected using the reliability of the facial feature points included in the photographed image, calculated using a known technique. The face may be detected based on information obtained as a result of inputting the photographed image into a face detection model generated by machine learning. For example, the face detection model may be a model generated by performing machine learning processing on a large number of photographed images to generate an input / output relationship in which a photographed image including a face in an area is used as input information, and the area, feature points, and the reliability value of the face are used as output information. When the face detection unit 21 detects a face in the photographed image, it outputs a photographed image ID indicating the photographed image to the body detection unit 22. The face detection unit 21 also records in memory the coordinate information (face image information) of the four corners of the rectangular face image m1 including the detected face area, linked to the photographed image ID.

[0026] The body detection unit 22 determines whether a body can be detected in the photographed image indicated by the acquired photographed image ID (step S104). The detection of the body in the photographed image may utilize a known technique. For example, the detection of the body may be performed by extracting a feature amount such as a skeleton of the body shown in the image and detecting the body based on the feature amount. The detection of the body may be performed based on information obtained as a result of inputting the photographed image into a body detection model generated by machine learning. For example, the body detection model may be a model generated by performing machine learning processing on a large number of photographed images to generate an input / output relationship in which a photographed image including a body in an area is used as input information, and the feature points of the body area and skeleton and the reliability value are used as output information. When the body detection unit 22 detects a body in the photographed image, it outputs the photographed image ID indicating the photographed image to the correspondence relationship specification unit 23. As another example, the body detection unit 22 records in memory the coordinate information (body image information) of the four corners of a rectangular body image m2 including the detected body area in association with the photographed image ID.

[0027] When the correspondence specification unit 23 acquires the photographed image ID from the body detection unit 22, it assigns a tentative person ID to the face image information and the body image information recorded in the memory in association with the photographed image ID, records the tentative person ID in the memory, and specifies the correspondence (step S105). As a result, the photographed image ID, the tentative person ID, the face image information (coordinate information), and the body image information (coordinate information) are recorded in association with each other in the memory, and the face area and the body area in the photographed image of the person M are recorded in association with each other. The correspondence specification unit 23 further links the face image m1 specified from the face image information in the photographed image to the photographed image ID and the tentative person ID, and records the face image m2 specified from the body image information in the photographed image in the memory in association with each other.

[0028] When identifying the correspondence between the face image information and the body image information, the correspondence identifying unit 23 may determine the correspondence based on the coordinate information of the face image information and the body image information. For example, based on the distance between the lower left and lower right coordinates of the face image information and the upper left and upper right coordinates of the body image information, it may determine whether the left and right coordinates are within a predetermined distance, and if they are within the predetermined distance, it may determine that there is a correspondence between the face image information and the body image information (they are image information of the same person).

[0029] Alternatively, the correspondence specifying unit 23 may input a photographed image in which the face detection unit 21 detects a face and the body detection unit 22 detects a body to the correspondence specifying model, and obtain a result that the face area and the body area belong to the same person based on the result output by the correspondence specifying model, and specify the relationship between the face area and the body area based on the result. In this case, the correspondence specifying unit 23 may obtain face image information (coordinates) indicating the face area and body image information (coordinates) indicating the body area output by the correspondence specifying model, and replace them with image information recorded in the memory in association with the photographed image ID or the provisional person ID. For example, the correspondence specifying model may be a model generated by machine learning processing of a large number of photographed images, with a photographed image including a face and a body as input information and a face area and a body area of ​​a person appearing in the photographed image as output information.

[0030] The correspondence specifying unit 23 can specify the correspondence between the face region and the body region of each person even when multiple people are captured in the captured image. For example, the correspondence specifying unit 23 inputs a captured image in which the face detection unit 21 detects the faces of multiple people and the body detection unit 22 detects the bodies of multiple people to the correspondence specifying model. Then, the body detection unit 22 may obtain a result that the face region and the body region of each person are the same person's region based on the result output by the correspondence specifying model, and specify the relationship between the face region and the body region of each person based on the result. The correspondence specifying model may be a model generated by performing machine learning processing on a large number of captured images to generate an input / output relationship in which a captured image including the faces and bodies of multiple people as input information and information indicating the correspondence between the face region and the body region of each person captured in the captured image are output information.

[0031] When the correspondence specification unit 23 records information such as the face image information (coordinates), body image information (coordinates), face image m1, and body image m2 of the person appearing in the photographed image in memory in association with the photographed image ID and the tentative person ID, it determines that the correspondence has been specified, and outputs the photographed image ID and the tentative person ID to the face matching unit 24. The face matching unit 24 acquires the photographed image ID of the photographed image including the person with whom the correspondence has been specified, and the tentative person ID detected for that photographed image.

[0032] The face matching unit 24 reads a face image recorded in memory in association with the photographed image ID and the temporary person ID. The face matching unit 24 performs face matching processing of the face image using a face matching program (step S106). The face matching unit 24 inputs face images to be compared that are sequentially identified from among a plurality of face images included in the database 104. The face images to be compared may be face images registered in the database 104 in advance.

[0033] The face matching unit 24 calculates the degree of match between the face image detected by the face detection unit 21 and a face image specified from a plurality of face images (comparison target) included in the database 104 for each face image specified in order from a plurality of face images included in the database 104. As described above, the face matching program is a program using a model generated by machine learning processing. This allows the face matching unit 24 to calculate the degree of match between the face image detected by the face detection unit 21 and each face image specified from the database 104. The face matching unit 24 determines whether the highest degree of match between the face image detected by the face detection unit 21 and each face image specified from the database 104 is equal to or higher than a predetermined threshold, and thereby determines whether face matching has been successful (step S107). If the highest degree of match between the face image detected by the face detection unit 21 and each face image specified from the database 104 is equal to or higher than a predetermined threshold, the face matching unit 24 determines that face matching has been successful, and determines that the comparison target face image matches the face image detected by the face detection unit 21.

[0034] The face matching unit 24 specifies, from the database 104, personal information of the face image to be compared that is determined to match in the database 104. The personal information includes a personal ID for identifying the person in the face image. This allows the face matching unit 24 to link the captured image ID, the tentative person ID, and the person ID. In other words, it is possible to link the tentative person ID given to the person appearing in the captured image indicated by the captured image ID with the person ID of the person indicated by the face image to be compared and matched with that person. The face matching unit 24 outputs a matching result including the captured image ID, the tentative person ID, the person ID, and flag information indicating success of face matching to the image recording unit 25.

[0035] The image recording unit 25 reads the body image recorded in the memory in association with the photographed image ID, the tentative person ID, and the person ID. The image recording unit 25 judges whether the read body image and the image information of the body region included in the body image satisfy the recording condition (step S108). When the body image and the image information of the body region satisfy the recording condition, the image recording unit 25 judges to record the body image. The recording condition is, for example, information indicating a condition that the state of the image is a predetermined state. For example, the recording condition may be a predetermined condition such that at least one of the brightness and saturation indicated by the body image is equal to or greater than a predetermined threshold, or that the body image is in a state in which it can be determined that there is no blur, or the like. The recording condition may be information indicating that the posture of the person whose body region is detected is in a predetermined state. For example, the recording condition is information indicating a condition such as that the body region includes arms, that the body region includes legs, that the body region can be assumed to be in the front, or the like. A known technique may be used to determine whether these recording conditions are met. Alternatively, a recording condition determination model generated using a machine learning technique may be used to determine whether the recording conditions are met. The recording condition determination model is a learning model obtained by machine learning an input / output relationship in which a body image is used as input information and a result indicating whether a predetermined recording condition is satisfied is used as output information. By recording a body image that satisfies the predetermined condition, it is possible to record only information that is appropriate as a body image to be used for later matching.

[0036] The image recording unit 25 reads the brightness or saturation of each pixel shown in the body image, judges whether they are equal to or greater than a threshold, and judges whether the brightness or saturation shown in the body image is equal to or greater than a predetermined threshold. The image recording unit 25 may judge the edge of the body outline based on the pixels shown in the body image, and judge whether there is blurring based on the presence or absence of the edge, the area, etc. The judgment of whether the brightness or saturation of these images is equal to or greater than a threshold and the judgment of whether there is blurring may be made using known techniques.

[0037] Furthermore, the image recording unit 25 may compare the shape of the person whose body region has been detected with the shape of the person who satisfies the recording conditions stored in advance by pattern matching, and if the pattern matching results in a match, determine that the posture of the person whose body region has been detected is in a predetermined state. Alternatively, the image recording unit 25 may calculate the front direction of the person based on the shape of the person whose body region has been detected, and determine that the posture of the person whose body region has been detected is in a predetermined state if the angle between the vector of the direction and the directional vector of the shooting direction of the camera 2 is an angle at which it can be determined that the person is facing toward the camera 2.

[0038] The image recording unit 25 may also determine whether both arms and both legs are captured based on the shape of the person whose body region is detected, and if so, determine that the posture of the person whose body region is detected is in a specified state.

[0039] When the body image or image information of the body region included in the body image satisfies the recording condition, the image recording unit 25 links the body image with the person ID and flag information indicating successful face matching, and records them in the person table of the database 104 (step S109). The image recording unit 25 may read a face image recorded in the memory linked to the photographed image ID, the provisional person ID, and the person ID, and record the face image, the person ID, and flag information indicating successful face matching in the person table of the database 104. The image recording unit 25 may read a body image and a face image recorded in the memory linked to the photographed image ID, the provisional person ID, and the person ID, and record the body image, the face image, the person ID, and flag information indicating successful face matching in the person table of the database 104, and record them in the person table of the database 104. The image recording unit 25 may read the body image and face image recorded in the memory in association with the photographed image ID, the tentative person ID, and the person ID, and the photographed image in which the body image and face image appear, and record the body image, face image, photographed image, person ID, and flag information indicating successful face matching in the person table of the database 104 in association with the body image, face image, photographed image, person ID, and flag information indicating successful face matching. As with the body image, the face image and photographed image may be recorded when they satisfy a predetermined recording condition. The image processing device 1 uses the body image and face image recorded in the person table for a later person matching process. Since the body image, face image, and photographed image that satisfy the predetermined recording condition are recorded in this way, a more accurate matching process is performed.

[0040] Recording determination unit 12 repeats the above-mentioned processes of steps S101 to S109 every time a captured image is input. As a result, if camera 2 that generated the captured image is a camera installed at a position where recording determination of a person appearing in the captured image is performed, such as an entry position or a predetermined person shooting position, the body image and face image of the person appearing in the captured image are recorded in the person table.

[0041] Originally, the face image and body image of the person to be registered are registered in the person table. However, by the processing of the recording judgment unit 12, the face image and body image of the person to be registered in advance in the person table are further recorded. Alternatively, the recording judgment unit 12 may repeatedly update the face image and body image of the person to be registered in advance registered in the person table by replacing them with face images and body images generated from newly acquired photographed images. If a plurality of cameras 2 are installed in a predetermined area such as a theme park, a predetermined region (country, prefecture, region), a public facility, a building, or an office, which are provided at positions for performing recording judgment of a person appearing in a photographed image, such as an entry position or a predetermined person photographing position, the face image and body image of person M are automatically recorded and accumulated or updated in the person table when photographed by the camera 2. Therefore, for example, even if person M changes clothes in a predetermined area, the body image of person M wearing the changed clothes can be recorded. Furthermore, even if person M wears glasses or sunglasses or a mask in a predetermined area, the face image may be accumulated. The face matching process described above may be a process in which matching is performed using information on a portion of a face.

[0042] Then, when person M is photographed by camera 2 of a camera type indicating that the camera type is a type that performs matching processing, image processing device 1 performs matching processing by comparing the facial image and body image of person M contained in the captured image acquired from camera 2 of the type that performs matching processing with the facial image m1 and body image m2 of person M contained in the captured image acquired from camera 2 of the camera type indicating that the type is a type that performs recording judgment through the above-mentioned processing.

[0043] The camera 2 installed in a predetermined area in the present disclosure may be a camera to which a type ID indicating both a type for performing a recording judgment and a type for performing a matching process is assigned. In this case, the image processing device 1 may perform both the above-mentioned recording judgment process and the following matching process for the captured image acquired from the camera 2.

[0044] The above-mentioned processing described with reference to FIG. 8 is executed simultaneously in parallel for each frame of a plurality of captured images generated by controlling the capture of a plurality of cameras 2.

[0045] FIG. 9 is a diagram showing a second processing flow of the image processing device. Next, a second processing flow of the image processing device 1 will be described. The second processing flow is a processing flow of a matching process. It is assumed that the camera 2 is a camera installed at a shooting position for performing the matching process. The camera 2 shoots a person M. The camera 2 transmits shooting information including a shot image of the person M and the ID of the camera 2 to the image processing device 1. The input unit 11 of the image processing device 1 acquires the shooting information from the camera 2 (step S101 in FIG. 8). The input unit 11 of the image processing device 1 acquires the ID of the camera 2 included in the shooting information. The input unit 11 judges whether the camera 2 is a camera installed at a position for performing a recording judgment such as an entry position (step S102 in FIG. 8) based on the ID of the camera 2. If the judgment is No, the camera 2 is a camera installed at a shooting position for performing the matching process. The input unit 11 reads the camera type corresponding to the ID of the camera 2 based on the record of a camera type table of the database 104 that stores the correspondence between the ID of the camera 2 and information indicating the camera type. When the camera type indicates that the camera type is not a type for which recording judgment is performed, the camera is provided at a shooting position for performing a matching process, so the input unit 11 outputs the shooting information to the matching unit 13. The above processes are the same as those in the first processing flow described above.

[0046] The face detection unit 31 of the matching unit 13 acquires the photographing information from the input unit 11. The face detection unit 31 determines whether a face can be detected in the photographed image (step S201). A known technique may be used to detect a face in the photographed image. For example, the face may be detected using the reliability of the facial feature points included in the photographed image, calculated using a known technique. The face may be detected based on information obtained as a result of inputting the photographed image into a face detection model generated by machine learning. For example, the face detection model may be a model generated by performing machine learning processing on a large number of photographed images to generate an input / output relationship in which a photographed image including a face in an area is used as input information, and the area, feature points, and the reliability value of the face are used as output information. If the face detection unit 21 can detect a face in the photographed image, it instructs the face matching unit 32 to perform face matching. If the face detection unit 21 cannot detect a face in the photographed image, it instructs the body detection unit 33 to perform body detection.

[0047] The face matching unit 32 performs face matching processing based on the face area detected in the captured image (step S202). The face matching unit 32 inputs face images to be compared that are sequentially identified from the plurality of face images included in the database 104.

[0048] The face matching unit 32 calculates the degree of match between the face image detected by the face detection unit 31 and a face image specified from a plurality of face images (comparison target) included in the database 104 for each face image specified in order from a plurality of face images included in the database 104. As described above, the face matching program is a program using a model generated by machine learning processing. This allows the face matching unit 32 to calculate the degree of match between the face image detected by the face detection unit 31 and each face image specified from the database 104. The face matching unit 32 determines whether the highest degree of match between the face image detected by the face detection unit 31 and each face image specified from the database 104 is equal to or higher than a predetermined threshold, and thereby determines whether face matching has been successful (step S203). If the face matching unit 32 determines that the highest degree of match between the face image detected by the face detection unit 31 and each face image specified from the database 104 is equal to or higher than a predetermined threshold, it determines that the comparison target face image matches the face image detected by the face detection unit 31, and determines that face matching has been successful. If the highest degree of match between the face image detected by the face detection unit 31 and each face image identified from the database 104 is not equal to or greater than a predetermined threshold, the face matching unit 32 determines that the face matching process is unsuccessful and instructs the body detection unit 33 to perform body detection.

[0049] The face matching unit 32 identifies from the database 104 personal information of the face image to be compared that is determined to match in the database 104 (step S204). The personal information includes a personal ID for identifying the person in the face image. The face matching unit 32 outputs the personal information to the output unit 35. The output unit 35 outputs the personal information identified by the face matching unit 32 based on the captured image to a predetermined output destination device (step S205).

[0050] As a result, the image processing device 1 can perform a predetermined process using the result of the matching process of the person M appearing in the captured image. For example, when the matching system 100 of the present disclosure is used in a theme park, which is a predetermined area, the matching system 100 may be a device that uses person information to determine whether the person can enter an attraction in the theme park. For example, when the person information includes a type indicating the attraction that the person is about to enter, the output destination device may determine that the person can enter the attraction. Alternatively, when the matching system 100 of the present disclosure is used in an office, which is a predetermined area, the output destination device may use the person information to control a computer installed in the office so that the computer can be operated. For example, when the person information includes an identifier of an operable computer, the output destination device may control the computer corresponding to the identifier so that the computer can be operated.

[0051] In step S201, if a face cannot be detected, the body detection unit 33 acquires an instruction for body detection from the face detection unit 31. Alternatively, in step S203, if face matching cannot be performed, the body detection unit 33 acquires an instruction for body detection from the face matching unit 32. The body detection unit 33 determines whether a body can be detected in the photographed image (step S206). A known technique may be used to detect a body in the photographed image. For example, the body may be detected using the reliability of the feature points of the body skeleton included in the photographed image, calculated using a known technique. The body may be detected based on information obtained as a result of inputting the photographed image into a body detection model generated by machine learning. For example, the body detection model may be a model generated by performing machine learning processing on a large number of photographed images to generate an input / output relationship in which a photographed image including a body in an area is used as input information, and the area, feature points, and the value of the reliability of the body are used as output information. If the body detection unit 33 detects a body in the photographed image, it instructs the body matching unit 34 to perform body matching. If the body detection unit 33 cannot detect a body in the photographed image, it determines that the process is to be terminated.

[0052] When the body matching unit 34 receives the instruction for body matching, the body matching unit 34 performs a body matching process based on the body region detected in the captured image (step S207). The body matching unit 34 inputs face images m2 to be compared, which are sequentially identified from among the multiple body images m2 included in the database 104.

[0053] The body matching unit 34 calculates the degree of match between the body image detected by the body detection unit 33 and the body image identified from the plurality of body images (comparison target) included in the database 104 for each of the body images identified in order from the plurality of body images included in the database 104. As described above, the body matching program is a program using a model generated by machine learning processing. This allows the body matching unit 34 to calculate the degree of match between the body image detected by the body detection unit 33 and each body image identified from the database 104. The body matching unit 34 determines whether the highest degree of match between the body image detected by the body detection unit 33 and each body image identified from the database 104 is equal to or higher than a predetermined threshold and whether the identified body image in the database 104 is recorded in association with flag information indicating successful face matching, thereby determining whether body matching has been successful (step S208). When the body matching unit 34 determines that the highest degree of matching between the body image detected by the body detection unit 33 and each body image specified from the database 104 is equal to or higher than a predetermined threshold value, and when the body image of the comparison target specified in the database 104 is recorded in association with flag information indicating successful face matching, the body matching unit 34 determines that the comparison target body image matches the body image detected by the body detection unit 33 and determines that the body matching is successful. When the highest degree of matching between the body image detected by the body detection unit 33 and each body image specified from the database 104 is not equal to or higher than a predetermined threshold value, or when the body image of the comparison target specified in the database 104 is not recorded in association with flag information indicating successful face matching, the body matching unit 34 determines that the body matching process is unsuccessful and determines to end the process. By not recording body images that are not linked to flag information indicating successful face matching, it is possible to prevent the recording of body images of people who have not been able to be face matched but have only been successful in body matching.

[0054] The body matching unit 34 specifies the person information of the body image to be compared that is determined to match in the database 104 from the database 104 (step S209). The person information includes a person ID for identifying the person of the body image. The body matching unit 34 outputs the person information to the output unit 35 (step S210). The output unit 35 outputs the person information specified by the body matching unit 34 based on the captured image to a predetermined output destination device (step S211). This allows the image processing device 1 to perform a predetermined process using the result of the matching process of the person M appearing in the captured image. For example, when the matching system 100 of the present disclosure is used in a theme park that is a predetermined area, it may be a device that determines whether or not the person can enter an attraction in the theme park using the person information. For example, when the person information includes the type of attraction that the person can use, the output destination device may determine that the person can use the attraction.

[0055] The image processing device 1 can perform control so that a predetermined process is performed in the output destination device when the matching process is successful as a result of the body matching process by the body matching unit 34 even when the face detection unit 31 cannot detect a face or the face matching is unsuccessful in the face matching unit 32. Alternatively, the image processing device 1 itself may perform some process using the result of the body matching process by the body matching unit 34 even when the face detection unit 31 cannot detect a face or the face matching is unsuccessful in the face matching unit 32.

[0056] The process described above with reference to FIG. 9 is also executed simultaneously in parallel for each frame of a plurality of captured images generated by controlling the capture of a plurality of cameras 2.

[0057] In the above embodiment, the camera 2 installed at the position for recording and judging the person appearing in the photographed image, such as the entry position or the predetermined person photographing position, may be installed at a position for photographing a person at a predetermined fixed position from multiple directions. In this way, face images and body images of a person photographed from multiple directions are recorded, and the recorded images are used as comparison targets, allowing for more accurate matching of the person.

[0058] Second Embodiment FIG. 10 is a functional block diagram of an image processing apparatus according to the second embodiment. As shown in FIG. 10, the image processing device 1 further includes a tracking unit 14. The image processing device 1 may be a device that performs tracking processing of the person M based on the output result of the output unit 35. For example, even if the face detection unit 31 cannot detect a face or the face matching unit 32 fails to match a face, if the body matching unit 34 succeeds in matching as a result of the body matching processing, the output unit 35 outputs to the tracking unit 14 the person information identified by the body matching processing, the photographed image, the identification information of the camera 2 that acquired the photographed image, the installation coordinates of the camera 2, and the detection time. The tracking unit 14 records these pieces of information in a tracking table by linking them together. The matching unit 13 and the tracking unit 14 repeat the same processing. As a result, the tracking table sequentially accumulates the person information about the person M, the photographed image, the identification information of the camera 2 that acquired the photographed image, the installation coordinates of the camera 2, and the detection time. As a result, the image processing device 1 can later track the movement of the person M using the history recorded in the tracking table. The tracking unit 14 may perform tracking processing using a face image of the person M.

[0059] <Third embodiment> FIG. 11 is a functional block diagram of an image processing apparatus according to the third embodiment. In the process of the record determination unit 12 in the first embodiment, only when the face matching process is successful, the person ID indicating the person identified by the face matching process is linked to the body image, face image, and photographed image and recorded in the person table. However, in addition to the face matching process, a body detection process and a body matching process may be further performed, and when a match with the same person is determined based on the results of the face matching process and the body matching process, the person ID may be linked to the body image, face image, and photographed image and recorded in the person table. In this case, the record determination unit 12 further includes a body matching unit 26.

[0060] In this case, after the body detection unit 22 performs body detection, the body matching unit 26 performs body matching using image information of the body region previously recorded for the person identified as a result of the face matching process and image information of the body region corresponding to the image information of the face region used in the face matching process. When the image recording unit 25 determines that the image information of the body region corresponding to the image information of the face region used in the face matching process is image information of the body region of the person identified as a result of the face matching process in the body matching process, the image recording unit 25 records image information (body image) including the body region of the person identified as a result of the face matching process. The process of the body detection unit 22 and the process of the body matching unit 26 are the same as the process of the body detection unit 33 and the process of the body matching unit 34 described in the first embodiment. By such a process, the body image is recorded when the matching processes of both the face matching process and the body matching process are successful, so that the body image information of a specific person can be recorded with higher accuracy.

[0061] <Fourth embodiment> The recording condition described in the first embodiment above may be information indicating that the attributes (e.g., the color of clothes worn, the shape of clothes worn, etc.) or accessories (e.g., objects such as glasses or hats worn) of the person whose body region has been detected are different from the image information of the body region recorded for the person identified as a result of the face matching process. As a result, for example, when the clothes shown in the body image recorded in advance in the person table are different from the clothes shown in the body region of the photographed image newly processed by the recording determination unit 12 in the recording determination process, the body image can be newly recorded as if person M has changed clothes within the specified region.

[0062] According to the processing of the image processing device of each of the above-mentioned embodiments, it is possible to record body images for authenticating a person multiple times over a long period of time with high accuracy even when facial features cannot be recognized. By recording the body images, it is possible to authenticate a person multiple times over a long period of time with high accuracy even when facial features cannot be recognized.

[0063] According to the above-mentioned verification system, if a plurality of cameras 2 are installed in a predetermined area such as a theme park, a predetermined region (country, prefecture, or area), a public facility, a building, or an office, which are provided at positions for recording and judging people appearing in the captured image, such as an entry position or a predetermined person photographing position, the body image of each person is recorded. Even if the person changes clothes within the predetermined area, the person can be verified and tracked using only the body image of the person.

[0064] If the predetermined area is a theme park, a camera 2 for recording and judging is installed at the entrance gate of the theme park and at a predetermined position for each area. Based on the image taken by the camera 2 for recording and judging, the best shot body image that satisfies the recording condition of each person is recorded in the matching system 100. Even if matching cannot be performed using a face image of a person when using an attraction installed in each area, the image processing device can match the person using only the body image by the processing of the matching unit 13 described above. In the theme park, a user may perform actions such as putting on a hat, changing clothes, or putting on a mask. Even in such cases, the user can be matched with higher accuracy. Similarly, tracking of a person in a predetermined area such as a theme park can be performed using only the body image.

[0065] In the above process, the image recording unit 25 may classify the body images determined to be recorded in the recording determination process by category and register each body image. For example, the image recording unit 25 acquires the position coordinates of the camera 2 that generated the captured image. The image recording unit 25 compares the position coordinates of small areas divided in a predetermined area with the position coordinates specified for the captured image including the body image to be recorded, and identifies the small area corresponding to the body image. The image recording unit 25 may then link the identification information of the small area with the body image determined to be recorded and record it in the person table. This allows the body images to be recorded as body images to be used in the matching process for different areas in a theme park, for example. In the matching process, the matching unit 13 identifies the position where the person was photographed based on the installation position of the camera 2, and identifies the body image recorded in association with the identification information of the small area corresponding to the position coordinates of the installation position. The matching unit 13 then performs the matching process using the identified body image as an image to be compared. As an example, each area in a theme park may have a different theme, and visitors may change clothes or decorations depending on the theme. Also, visitors may be wearing normal clothes when entering and leaving an area. Even in such a case, a body image may be registered in association with location information for each area, and matching within the area may be performed using the body image registered within the area.

[0066] FIG. 12 is a diagram showing a processing flow according to the fourth embodiment. When a person M enters a predetermined area in a theme park, a camera 2 installed at the entrance position captures the person M. The camera 2 transmits the captured image of the person M and the capture information including the ID and location information of the camera 2 to the image processing device 1. The input unit 11 of the image processing device 1 acquires the capture information from the camera 2 (step S301). The subsequent processes of steps S302 to S308 are the same as those of the first embodiment. When the body image and the image information of the body area included in the body image satisfy the recording condition, the image recording unit 25 records the body image, the person ID, flag information indicating the success of face matching, and location information indicating the location where the captured image was captured in the person table of the database 104 in association with each other (step S309). The image recording unit 25 may read the captured image ID, the temporary person ID, and the face image recorded in the memory in association with the person ID, and record the face image, the person ID, flag information indicating the success of face matching, and location information in the person table of the database 104 in association with each other.

[0067] In addition, when actually performing a matching process of a person appearing in a captured image based on shooting information including the captured image captured by the camera 2 installed at each position in the theme park, the image processing device 1 identifies the theme park based on the location information included in the shooting information by the camera 2. In addition, when performing the process described with reference to Fig. 9, the image processing device 1 identifies a face image or body image to be compared from the database 104 in association with the location information indicating the area of ​​the theme park, and compares it with the face image or body image appearing in the captured image to perform a matching process similar to that of the first embodiment described with reference to Fig. 9.

[0068] The image processing device 1 may re-register recorded body images and face images at a predetermined timing. For example, the image processing device 1 may delete the body images of each person from the person table at a predetermined time such as 00:00. Then, the image processing device 1 may newly perform a recording determination process for each person and record new body images.

[0069] The image processing device 1 may create a list of person images including face images and body images for a predetermined period based on the correspondence between the recorded body images and face images, and record the list for each person. Then, based on a request from each person, data of the list of person images of the person may be transmitted to a terminal carried by the person. By transmitting the list of person images in an album format, the image processing device 1 allows each person to check images taken within a predetermined area.

[0070] The image processing device 1 may delete face images and body images recorded in the person table at a predetermined timing. For example, the image processing device 1 performs a matching process based on an image captured by a camera 2 installed near an exit of a predetermined area. The image processing device 1 may delete all image information, such as face images and body images, recorded in the person table for a person who is matched in the matching process.

[0071] <Fifth embodiment> FIG. 13 is a diagram showing a processing flow according to the fifth embodiment. In the first embodiment, the process for recording a body image is described using Fig. 8, in which a body image is recorded when face matching is successful. However, in other embodiments, when a face image with a resolution for face matching cannot be obtained from an image obtained by photographing a person at a long shooting distance, the following process may be performed.

[0072] Specifically, the input unit 11 of the image processing device 1 acquires the shooting information from the camera 2 (step S101). The input unit 11 of the image processing device 1 acquires the ID of the camera 2 included in the shooting information. Based on the ID of the camera 2, the input unit 11 judges whether the camera 2 is a camera installed at a position where a recording judgment of a person appearing in a captured image is performed, such as an entry position or a predetermined person shooting position (step S102). The input unit 11 reads the camera type corresponding to the ID of the camera 2 based on a record of a camera type table of a database 104 that stores a correspondence relationship between the ID of the camera 2 and information indicating the camera type. If the camera type indicates that the camera type is a type for which a recording judgment is performed, the input unit 11 outputs the shooting information to the recording judgment unit 12. If the camera type does not indicate that the camera type is a type for which a recording judgment is performed, the input unit 11 outputs the shooting information to the collation unit 13.

[0073] The recording determination unit 12 acquires the shooting information from the input unit 11. In the recording determination unit 12, the face detection unit 21 reads the shot image from the shooting information. The face detection unit 21 determines whether a face can be detected in the shot image (step S103). The process up to this point is the same as in the first embodiment.

[0074] If it is determined in step S103 that the face cannot be detected (No), the body detection unit 33 determines whether the body can be detected in the photographed image (step S401). The detection of the body in the photographed image may utilize a known technique. For example, the detection of the body may be performed using the reliability of the feature points of the body skeleton included in the photographed image, calculated using a known technique. The detection of the body may be performed based on information obtained as a result of inputting the photographed image into a body detection model generated by machine learning. For example, the body detection model may be a model generated by performing machine learning processing on a large number of photographed images to generate an input / output relationship in which the photographed image including the body in the region is used as input information, and the region, feature points, and the reliability value of the body are used as output information. If the body detection unit 33 detects the body in the photographed image, it records the coordinate information (body image information) of the four corners of the rectangular body image m2 including the detected body region in the memory in association with the photographed image ID (step S402). The face detection unit 21 determines whether the face can be detected in the photographed image (step S403). The image processing device 1 repeats the processes of steps S401 and S403 until the face detection unit 21 detects a face in the captured image. By this process, in a situation where a person in the captured image approaches the camera 2 from a distance, one or more body images are recorded in the memory until a face is detected.

[0075] If the face detection unit 21 determines that a face can be detected in the photographed image, it outputs a photographed image ID indicating the photographed image to the body detection unit 22. The face detection unit 21 also links coordinate information (face image information) of the four corners of a rectangular face image m1 including the detected face area to the photographed image ID and records it in memory. If the face detection unit 21 detects a body in the photographed image, it outputs the photographed image ID indicating the photographed image to the correspondence specification unit 23.

[0076] When the correspondence specification unit 23 acquires the photographed image ID from the face detection unit 21, the correspondence specification unit 23 assigns a tentative person ID to the face image information and the body image information recorded in the memory in association with the photographed image ID, records the tentative person ID in the memory, and specifies the correspondence (step S404). As a result, the photographed image ID, the tentative person ID, the face image information (coordinate information), and the body image information (coordinate information) are recorded in association with each other in the memory, and the face area and the body area in the photographed image for the person M are recorded in association with each other. The correspondence specification unit 23 further links the face image m1 specified from the face image information in the photographed image to the photographed image ID and the tentative person ID, and records the face image m2 specified from the body image information in the photographed image in the memory in association with each other.

[0077] When identifying the correspondence between the face image information and the body image information, the correspondence identifying unit 23 may determine the correspondence based on the coordinate information of the face image information and the body image information. For example, based on the distance between the lower left and lower right coordinates of the face image information and the upper left and upper right coordinates of the body image information, it may determine whether the left and right coordinates are within a predetermined distance, and if they are within the predetermined distance, it may determine that there is a correspondence between the face image information and the body image information (they are image information of the same person).

[0078] Alternatively, the correspondence specifying unit 23 may input a photographed image in which the face detection unit 21 detects a face and the body detection unit 22 detects a body to the correspondence specifying model, and obtain a result that the face area and the body area belong to the same person based on the result output by the correspondence specifying model, and specify the relationship between the face area and the body area based on the result. In this case, the correspondence specifying unit 23 may obtain face image information (coordinates) indicating the face area and body image information (coordinates) indicating the body area output by the correspondence specifying model, and replace them with image information recorded in the memory in association with the photographed image ID or the provisional person ID. For example, the correspondence specifying model may be a model generated by machine learning processing of a large number of photographed images, with a photographed image including a face and a body as input information and a face area and a body area of ​​a person appearing in the photographed image as output information.

[0079] The correspondence specifying unit 23 can specify the correspondence between the face region and the body region of each person even when multiple people are captured in the captured image. For example, the correspondence specifying unit 23 inputs a captured image in which the face detection unit 21 detects the faces of multiple people and the body detection unit 22 detects the bodies of multiple people to the correspondence specifying model. Then, the body detection unit 22 may obtain a result that the face region and the body region of each person are the same person's region based on the result output by the correspondence specifying model, and specify the relationship between the face region and the body region of each person based on the result. The correspondence specifying model may be a model generated by performing machine learning processing on a large number of captured images to generate an input / output relationship in which a captured image including the faces and bodies of multiple people as input information and information indicating the correspondence between the face region and the body region of each person captured in the captured image are output information.

[0080] When the correspondence specification unit 23 records information such as the face image information (coordinates), body image information (coordinates), face image m1, and body image m2 of the person appearing in the photographed image in memory in association with the photographed image ID and the tentative person ID, it determines that the correspondence has been specified, and outputs the photographed image ID and the tentative person ID to the face matching unit 24. The face matching unit 24 acquires the photographed image ID of the photographed image including the person with whom the correspondence has been specified, and the tentative person ID detected for that photographed image.

[0081] The face matching unit 24 reads a face image recorded in memory in association with the photographed image ID and the temporary person ID. The face matching unit 24 performs face matching processing of the face image using a face matching program (step S405). The face matching unit 24 inputs face images to be compared that are sequentially identified from among a plurality of face images included in the database 104. The face images to be compared may be face images registered in the database 104 in advance.

[0082] The face matching unit 24 calculates the degree of match between the face image detected by the face detection unit 21 and a face image specified from a plurality of face images (comparison target) included in the database 104 for each face image specified in order from a plurality of face images included in the database 104. As described above, the face matching program is a program using a model generated by machine learning processing. This allows the face matching unit 24 to calculate the degree of match between the face image detected by the face detection unit 21 and each face image specified from the database 104. The face matching unit 24 determines whether the highest degree of match between the face image detected by the face detection unit 21 and each face image specified from the database 104 is equal to or higher than a predetermined threshold, and thereby determines whether face matching has been successful (step S406). If the highest degree of match between the face image detected by the face detection unit 21 and each face image specified from the database 104 is equal to or higher than a predetermined threshold, the face matching unit 24 determines that face matching has been successful, and determines that the comparison target face image matches the face image detected by the face detection unit 21.

[0083] The face matching unit 24 specifies, from the database 104, personal information of the face image to be compared that is determined to match in the database 104. The personal information includes a personal ID for identifying the person in the face image. This allows the face matching unit 24 to link the captured image ID, the tentative person ID, and the person ID. In other words, it is possible to link the tentative person ID given to the person appearing in the captured image indicated by the captured image ID with the person ID of the person indicated by the face image to be compared and matched with that person. The face matching unit 24 outputs the matching result including the captured image ID, the tentative person ID, and the person ID to the image recording unit 25.

[0084] The image recording unit 25 reads the body image recorded in the memory in association with the photographed image ID, the tentative person ID, and the person ID. The image recording unit 25 determines whether the read body image and the image information of the body region included in the body image satisfy the recording condition (step S407). This process is the same as in the first embodiment.

[0085] When the image information of the body image or the body region included in the body image satisfies the recording condition, the image recording unit 25 records the body image in the person table of the database 104 in association with the person ID (step S408). The image recording unit 25 may read a face image recorded in the memory in association with the photographed image ID, the provisional person ID, and the person ID, and record the face image in the person table of the database 104 in association with the person ID. The image recording unit 25 may read a body image and a face image recorded in the memory in association with the photographed image ID, the provisional person ID, and the person ID, and record the body image, face image, and person ID in association with the person ID in the person table of the database 104. The image recording unit 25 may read a body image and a face image recorded in the memory in association with the photographed image ID, the provisional person ID, and the person ID, and a photographed image in which the body image and face image are shown, and record the body image, face image, photographed image, and person ID in association with the person ID in the person table of the database 104. Like the body image, the face image and the photographed image may be recorded if they satisfy a predetermined recording condition. The image processing device 1 uses the body image and face image recorded in the person table for a person matching process to be performed later. In this way, the body image, face image, and photographed image that satisfy the predetermined recording condition are recorded, so that a more accurate matching process can be performed.

[0086] According to the fifth embodiment, even if a face image cannot be detected in a captured image, a body image for recording is first stored in a memory, etc. Then, when a face image is detected, the image processing device 1 can specify the correspondence between the face image and the body image and record the body image as information of a person specified based on the face image.

[0087] FIG. 14 is a diagram showing the minimum configuration of an image processing device. FIG. 15 is a diagram showing a processing flow by an image processing device with a minimum configuration. The image processing device 1 includes at least a face detection unit 41, a body detection unit 42, a face matching unit 43, and an image recording unit 44. Then, the face detection means 41 detects the face area of ​​a person appearing in the image (step S131). The body detection means 42 detects the body region of a person appearing in the image (step S132). The face matching means 43 performs face matching processing using the image information of the face area (step S133). The image recording means 44 records the image information of the body region of the person identified as a result of the face matching process. At this time, the image recording means 44 records the image information of the body region if the image information of the body region satisfies a recording condition (step S134).

[0088] Each of the above-mentioned devices has a computer system inside. The steps of each of the above-mentioned processes are stored in a computer-readable recording medium in the form of a program, and the above processes are performed by the computer reading and executing this program. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Also, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0089] The program may be a program for implementing some of the above-mentioned functions, or may be a so-called differential file (differential program) that can implement the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0090] 1. Image processing device 2. Camera 11 Input section 12 Recording judgment section 13. Matching section 14. Tracking section 21 Face detection unit 22 Body detection unit 23 Correspondence identification unit 24. Face matching unit 25...Image recording unit 26 Field matching unit 31 Face detection unit 32 Face matching unit 33 Body detection unit 34 Field matching unit 35 Output section 100···Matching System

Claims

1. A determination means for communicating with a plurality of photographing devices in a facility and determining whether one of the plurality of photographing devices is a photographing device of a type that performs recording determination of a person appearing in a photographed image or a photographing device of a type that performs a matching process of the person based on an identifier of the photographing device; a first face detection means for detecting a face area of ​​a person appearing in an image when the photographing device is determined to be a photographing device of a type that performs a recording determination; a first body detection means for detecting a body region of the person appearing in the image when the photographing device is determined to be a photographing device of a type that performs record determination; a first face matching means for performing face matching processing using image information of the face area when the photographing device is determined to be a photographing device of a type that performs record determination; a correspondence relationship specification means for specifying a correspondence relationship between identification information of person information matched in face matching processing, the image information of the face region, and the image information of the body region when the photographing device is determined to be a photographing device of a type that performs record determination and the image information of the face region and the image information of the body region satisfy a predetermined correspondence relationship; an image recording means for recording, when it is determined that the photographing device is a photographing device of a type that performs recording determination, the identification information of the person identified as a result of the face matching process and image information of a body region in association with each other; a second face detection means for detecting a face area of ​​a person appearing in an image when the photographing device is determined to be a photographing device of a type that performs a matching process; a second body detection means for detecting a body region of the person appearing in the image when the photographing device is determined to be a photographing device of a type that performs a matching process; a second face matching means for performing face matching processing using image information of the face area when the photographing device is determined to be a photographing device of a type that performs matching processing; a second body matching means for performing a body matching process using image information of the body region when the photographing device is determined to be a photographing device of a type that performs a matching process; Equipped with The image recording means records the image information of the body region when the image information of the body region satisfies a recording condition. Image processing device.

2. The image processing apparatus according to claim 1 , wherein the recording condition is information indicating that the image is in a predetermined state.

3. the recording condition is that the posture of the person from whom the body region is detected is in a predetermined state; The image recording means does not record the image information of the body region when the image information of the body region does not satisfy the recording condition.

3. The image processing device according to claim 1 or 2.

4. 4. The image processing device according to claim 1, wherein the recording condition is information indicating that attributes or appendages of the person from whom the body region is detected are different from image information of the body region recorded for the person identified as a result of the face matching process.

5. a first body matching means for performing a body matching process when it is determined that the photographing device is a photographing device of a type that performs a record determination, using image information of a body area previously recorded for a person identified as a result of the face matching process and image information of the body area that has a corresponding relationship with the image information of the face area used in the face matching process; The image recording means records the image information of the body region of the person identified as a result of the face matching process when it is determined in the body matching process that the image information of the body region corresponding to the image information of the face region used in the face matching process is the image information of the body region of the person identified as a result of the face matching process. The image processing device according to any one of claims 1 to 4.

6. a tracking processing means for performing a tracking process using at least one of the image information of the face region and the image information of the body region; The image processing device according to claim 1 , further comprising:

7. The image processing device described in claim 1, wherein the second body matching means performs body matching processing using image information of the body area previously recorded for a person identified as a result of the face matching processing and image information of the body area of ​​the person detected by the second body detection means, and identifies the person information by the body matching processing.

8. The image processing device according to claim 7 , wherein the second body matching means identifies the person information by the body matching process when the second face matching means fails to perform the face matching process.

9. 2 . The image processing apparatus according to claim 1 , wherein the correspondence specifying means determines that the image information of the face region and the image information of the body region are in correspondence when the coordinates of the face region and the body region are within a predetermined distance.

10. A system that communicates with a plurality of imaging devices within a facility, a determination means for determining whether the plurality of photographing devices is a photographing device of a type that performs a recording determination of a person appearing in a photographed image or a photographing device of a type that performs a matching process of the person based on an identifier of the photographing device; a first face detection means for detecting a face area of ​​a person appearing in an image when the photographing device is determined to be a photographing device of a type that performs a record determination; an integrated detection means for detecting a body region of the person in the image when the photographing device is determined to be a photographing device of a type that performs a record determination; a first face matching means, when it is determined that the photographing device is a photographing device of a type that performs a record determination, performing a face matching process using image information of the face area; a correspondence specifying means for specifying a correspondence between identification information of person information matched in face matching processing, the image information of the face region, and the image information of the body region when the photographing device is determined to be a photographing device of a type that performs record determination and the image information of the face region and the image information of the body region satisfy a predetermined correspondence relationship; an image recording means, when it is determined that the photographing device is a photographing device of a type that performs recording determination, records the identification information of the person specified as a result of the face matching process in association with the image information of the body region; a second face detection means for detecting a face area of ​​a person appearing in an image when the photographing device is determined to be a photographing device of a type that performs a matching process; a second body detection means for detecting a body region of the person in the image when the photographing device is determined to be a photographing device of a type that performs a matching process; a second face matching means, when it is determined that the photographing device is a photographing device of a type that performs a matching process, performing a face matching process using image information of the face area; a second body matching means, when it is determined that the photographing device is a photographing device of a type that performs matching processing, performing a body matching process using image information of the body region; The image recording means records the image information of the body region when the image information of the body region satisfies a recording condition. Image processing methods.

11. A computer of an image processing device connected to a plurality of imaging devices in a facility, a determination means for determining whether one of the plurality of photographing devices is a photographing device of a type that performs a recording determination of a person appearing in a photographed image or a photographing device of a type that performs a matching process of the person based on an identifier of the photographing device; a first face detection means for detecting a face area of ​​a person appearing in an image when the photographing device is determined to be a photographing device of a type that performs a recording determination; a first body detection means for detecting a body region of the person appearing in the image when the photographing device is determined to be a photographing device of a type that performs record determination; a first face matching means for performing face matching processing using image information of the face area when the photographing device is determined to be a photographing device of a type that performs record determination; a correspondence relationship specification means for specifying a correspondence relationship between identification information of person information matched in face matching processing, the image information of the face region, and the image information of the body region when the photographing device is determined to be a photographing device of a type that performs record determination and the image information of the face region and the image information of the body region satisfy a predetermined correspondence relationship; an image recording means for recording, when it is determined that the photographing device is a photographing device of a type that performs recording determination, the identification information of the person identified as a result of the face matching process and image information of a body region in association with each other; a second face detection means for detecting a face area of ​​a person appearing in an image when the photographing device is determined to be a photographing device of a type that performs a matching process; a second body detection means for detecting a body region of the person appearing in the image when the photographing device is determined to be a photographing device of a type that performs a matching process; a second face matching means for performing face matching processing using image information of the face area when the photographing device is determined to be a photographing device of a type that performs matching processing; a second body matching means for performing a body matching process using image information of the body region when the photographing device is determined to be a photographing device of a type that performs a matching process; Function as a The image recording means records the image information of the body region when the image information of the body region satisfies a recording condition. program.

Images