A facial image registration system, a facial image registration method, a server, a facial image registration method for the server, and a program.

Abstract

To register a face image of a person with a suspicious behavior at a facility in the past with a database with high accuracy by a simple configuration.SOLUTION: A face image registration system 1 includes one or more first cameras 3 capable of generating a first captured image, one or more second cameras 4 capable of generating a second captured image, servers 11-13, and a terminal used by a user. The servers 11-13 each acquire the first and second captured images, and each transmit the first and second captured images to the terminal when a predetermined suspicious behavior is detected in the appearance of a person included in the second captured image. The first captured image includes a face image of a person with a suspicious behavior, who is a person whose suspicious behavior is detected, and the second captured image includes an image of a person with a suspicious behavior during the suspicious behavior. When receiving a registration instruction sent from the terminal based on the user's operation, the servers register the first captured image including the face image of the person with the suspicious behavior with a database.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a face image registration system, a face image registration method, a server, a face image registration method for the server, and a program for registering an image of the face of a person who performs suspicious behavior in a facility such as a store.

Background Art

[0002] Conventionally, in order to monitor suspicious actions of customers in a store, input image information obtained by photographing a person in the store is acquired, based on the input image information, the movement of the person's hand is tracked, and based on the tracked hand movement, a security system for detecting a person's suspicious action is known (see Patent Document 1).

[0003] The security system includes a suspect information database that stores suspect identification information for detecting that a customer who has come to the store is a suspect, and by acquiring a face image of the customer generated by a face recognition camera, the person included in the face image is recognized, and by referring to the suspect information database and comparing it with the face image, it is determined whether or not the person is a suspect.

[0004] In addition, in order to reduce the processing load of a computer for detecting illegal removal of goods in a store, using a photographed image including the display location of the goods, it is detected that a customer has taken a product from the display location, and customer information is generated that associates the number of times the customer has taken a product and information indicating whether or not the product has been settled with the identifier of the customer (see Patent Document 2).

[0005] When a predetermined condition regarding a customer is satisfied, the information processing device performs a warning process on the customer and adds the customer who is the warning target to a blacklist (information showing a list of persons requiring attention).

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Patent No. 6249201 [Patent Document 2] Patent No. 6805915 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] In facilities such as stores, even if suspicious behavior by facility users is detected by designated devices or systems, it is difficult for facility managers and staff to quickly determine whether the detection result is appropriate or not, and they may miss the opportunity to take immediate action.

[0008] In contrast, as disclosed in Patent Documents 1 and 2 above, by using a database (or blacklist) that registers information on individuals who have engaged in suspicious behavior in the past (hereinafter referred to as "suspicious individuals"), facility managers and staff can immediately recognize when a suspicious individual visits the facility again based on the registered information on suspicious individuals and begin monitoring the suspicious individual.

[0009] However, creating a database of facial images of such suspicious individuals with a simple configuration and high accuracy is not easy. Furthermore, the aforementioned Patent Documents 1 and 2 do not give any particular consideration to improving the technology for creating such a database of facial images.

[0010] Therefore, the main purpose of this disclosure is to provide a facial image registration system, a facial image registration method, a server, a facial image registration method for the server, and a program that can accurately register facial images of individuals who have performed suspicious actions in the past at a facility into a database using a simple configuration. [Means for solving the problem]

[0011] The facial image registration system of this disclosure is a facial image registration system for registering images of a person's face, comprising: at least one first camera installed in a facility and capable of generating a first captured image by photographing a person's face; at least one second camera installed in the facility and capable of generating a second captured image by photographing the person in action; and a server that processes the registration of the person's facial image included in the first captured image in a database, wherein the server acquires the first and second captured images respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images to a terminal used by a user, respectively, the first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of the suspicious behavior, and when the server receives a registration command transmitted from the terminal based on the user's operation, it registers the first captured image including the facial image of the suspicious person in the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be as follows.

[0012] The server in this disclosure is a face image registration system for registering images of a person's face, comprising: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user, wherein the server acquires the first and second captured images, respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images to the terminal, respectively, the first captured image transmitted to the terminal includes a face image of the person who exhibited the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of exhibiting the suspicious behavior, and when a registration command is received from the terminal based on the user's operation, the first captured image including the face image of the suspicious person is registered in the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be as follows.

[0013] The present disclosure relates to a server-based face image registration method for registering images of a person's face, comprising: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user. The server acquires the first and second captured images, respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images to the terminal, respectively. The terminal is used by a user who has the authority to register face images in a database. The first captured image transmitted to the terminal includes a face image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the suspicious person during the suspicious behavior. When the server receives a registration command transmitted from the terminal based on the user's operation, it registers the first captured image, including the face image of the suspicious person, in the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be as follows.

[0014] The server program of this disclosure is a face image registration system for registering images of people's faces, comprising: at least one first camera installed in a facility that generates a first captured image by capturing a person's face; at least one second camera installed in the facility that generates a second captured image by capturing the person's figure in action; a server; and a terminal used by a user. The server acquires the first and second captured images, respectively, and if a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, it transmits the first and second captured images to the terminal, respectively. The terminal is used by a user who has the authority to register face images in a database. The first captured image transmitted to the terminal includes a face image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the suspicious person during the suspicious behavior. When the terminal receives a registration command transmitted based on the user's operation, it registers the first captured image, including the face image of the suspicious person, in the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be set up to execute the process. [Effects of the Invention]

[0015] According to this disclosure, facial images of individuals who have previously engaged in suspicious behavior at the facility can be registered in a database with high accuracy using a simple configuration. [Brief explanation of the drawing]

[0016] [Figure 1] Overall configuration diagram of the facial image registration system according to the first embodiment [Figure 2] Functional block diagram of the face registration server [Figure 3] Functional block diagram of the behavior detection server [Figure 4] Functional block diagram of the movement mapping server [Figure 5] Diagram showing the store layout [Figure 6] Sequence diagram showing the operation procedure of the facial image registration system according to the first embodiment. [Figure 7] Sequence diagram showing the operation procedure of the face image registration system according to the first embodiment [Figure 8] Explanatory diagram regarding the process for identifying the face ID of a suspicious actor [Figure 9] Explanatory diagram showing an example of the selection screen for registration candidate images [Figure 10] Explanatory diagram showing an example of the confirmation screen for suspicious behavior [Figure 11] Overall configuration diagram of the face image registration system according to the second embodiment [Figure 12] Functional block diagram of the person tracking server [Figure 13] Sequence diagram showing the operation procedure of the face image registration system according to the second embodiment [Figure 14] Sequence diagram showing the operation procedure of the face image registration system according to the second embodiment

Mode for Carrying Out the Invention

[0017] The first invention made to solve the above problems is a face image registration system for registering an image of a person's face, comprising at least one first camera installed in a facility and capable of generating a first captured image by capturing the face of a person, at least one second camera installed in the facility and capable of generating a second captured image by capturing the figure of the person during movement, and a server for performing processing for registering the face image of the person included in the first captured image in a database. The server acquires the first and second captured images respectively, and when a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, transmits the first captured image and the second captured image to a terminal used by the user respectively. The first captured image transmitted to the terminal includes the face image of a suspicious actor who is the person in whom the suspicious behavior is detected, and the second captured image transmitted to the terminal includes the image of the suspicious actor during the suspicious behavior. When receiving a registration command transmitted based on the operation of the user from the terminal, registers the first captured image including the face image of the suspicious actor in the database The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be as follows.

[0018] According to this, facial images of suspicious individuals (people who have engaged in suspicious behavior at the facility in the past) can be registered in a database with high accuracy using a simple configuration. Furthermore, this method allows for the removal of unnecessary image areas from the entire image region, enabling the registration of appropriate facial images of suspicious individuals in the database.

[0021] Also, the 2 The invention is A face image registration system for registering images of a person's face, comprising: at least one first camera installed in a facility and capable of generating a first captured image by photographing a person's face; at least one second camera installed in the facility and capable of generating a second captured image by photographing the person in action; and a server that processes the registration of the person's face image included in the first captured image into a database, wherein the server acquires the first and second captured images respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images to a terminal used by a user, respectively, the first captured image transmitted to the terminal includes the face image of the person who exhibited the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of the suspicious behavior, and when the server receives a registration command transmitted from the terminal based on the user's operation, it registers the first captured image including the face image of the suspicious person into the database. The first captured image transmitted to the terminal includes a plurality of different registration candidate images, and the registration command includes information about at least one image selected from the plurality of registration candidate images based on the user's operation.

[0022] According to this system, the facial images of suspicious individuals to be registered are selected from multiple candidate images based on user input, allowing for the registration of more appropriate facial images in the database.

[0023] Also, the 3 The invention is A face image registration system for registering images of people's faces, comprising: at least one first camera installed in a facility and capable of generating a first captured image by photographing a person's face; at least one second camera installed in the facility and capable of generating a second captured image by photographing the person in action; and a server that processes for registering the face image of the person included in the first captured image in a database, wherein the server acquires the first and second captured images respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images respectively to a terminal used by a user, the first captured image transmitted to the terminal includes the face image of the person who performed the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of performing the suspicious behavior, and when it receives a registration command transmitted from the terminal based on the user's operation, it registers the first captured image including the face image of the suspicious person in the database. The system acquires a plurality of distinct first images, each containing a facial image of the suspicious person, extracts one or more images from the plurality of first images, each containing a facial image corresponding to the front of the suspicious person's face, and transmits the extracted images to the terminal.

[0024] According to this method, images containing a frontal view of the suspicious person's face are extracted from multiple captured images and registered as targets for registration, thus enabling the appropriate face images to be registered in the database.

[0025] Also, the 4 The invention is A face image registration system for registering images of people's faces, comprising: at least one first camera installed in a facility and capable of generating a first captured image by photographing a person's face; at least one second camera installed in the facility and capable of generating a second captured image by photographing the person in action; and a server that processes for registering the face image of the person included in the first captured image in a database, wherein the server acquires the first and second captured images respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images respectively to a terminal used by a user, the first captured image transmitted to the terminal includes the face image of the person who performed the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of performing the suspicious behavior, and when it receives a registration command transmitted from the terminal based on the user's operation, it registers the first captured image including the face image of the suspicious person in the database.If the suspicious behavior is detected in the figure of the person included in the second captured image, the system will acquire the time the suspicious person was photographed during the suspicious behavior included in the second captured image, and the location information of the suspicious person corresponding to that time. Based on the time the suspicious person was photographed and the location information of the suspicious person, the system will associate the person included in the first captured image with the suspicious person, thereby acquiring the first captured image including the face image of the suspicious person.

[0026] According to this method, by associating the person in the first captured image with the suspicious person in the second captured image based on the time and location information of the images taken, the face of the suspicious person in the first captured image can be identified through simple processing.

[0027] Also, the 5 The invention is configured such that the at least one first camera includes a plurality of first cameras, the server acquires a first captured image from the plurality of first cameras, acquires facial feature quantities relating to the face of the person included in the first captured image acquired from the plurality of first cameras, for persons whose facial feature quantities match, acquires multiple shooting times of the person's face included in the first captured image and multiple location information of the person corresponding to the multiple shooting times, estimates the movement path of the person based on the multiple shooting times and the multiple location information, and associates the person included in the first captured image with the suspicious person based on the movement path and the shooting time and location information of the suspicious person.

[0028] According to this method, the movement path of a person included in the first captured image can be estimated through simple processing, and the face of the suspicious person in the first captured image can be identified.

[0029] Also, the 6 The invention is A face image registration system for registering images of people's faces, comprising: at least one first camera installed in a facility and capable of generating a first captured image by photographing a person's face; at least one second camera installed in the facility and capable of generating a second captured image by photographing the person in action; and a server that processes for registering the face image of the person included in the first captured image in a database, wherein the server acquires the first and second captured images respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images respectively to a terminal used by a user, the first captured image transmitted to the terminal includes the face image of the person who performed the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of performing the suspicious behavior, and when it receives a registration command transmitted from the terminal based on the user's operation, it registers the first captured image including the face image of the suspicious person in the database.If the suspicious behavior is detected in the figure of the person included in the second captured image, the system acquires person features relating to the figure of the person performing the suspicious act, and based on these person features, associates the person included in the first captured image with the person performing the suspicious act, thereby acquiring the first captured image including the face image of the person performing the suspicious act.

[0030] According to this method, by matching the person in the first captured image with the suspicious person in the second captured image based on the person's features related to the suspicious person's appearance, the face of the suspicious person in the first captured image can be identified through a simple process.

[0031] Also, the 7 The invention further comprises the terminal used by the user.

[0032] According to this, facial images of suspicious individuals can be registered in a database with high accuracy using a simple configuration.

[0033] The following was done to solve the aforementioned problem. 8The invention relates to a face image registration method for registering images of a person's face, wherein at least one first camera installed in a facility generates a first captured image by photographing a person's face, at least one second camera installed in the facility generates a second captured image by photographing the person in action, a computer acquires the first and second captured images respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, the computer transmits the first and second captured images to a terminal, the terminal is used by a user who has the authority to register face images to a database, the first captured image transmitted to the terminal includes a face image of the person who exhibited the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of performing the suspicious behavior, and when the terminal receives a registration command transmitted based on the user's operation, the computer registers the first captured image including the face image of the suspicious person to the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be as follows.

[0034] According to this, facial images of suspicious individuals can be registered in a database with high accuracy using a simple configuration.

[0035] The following was done to solve the aforementioned problem. 9The invention relates to a face image registration system for registering images of a person's face, comprising: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user, wherein the server acquires the first and second captured images respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images to the terminal, respectively, the first captured image transmitted to the terminal includes a face image of the person who exhibited the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of performing the suspicious behavior, and when a registration command is received from the terminal based on the user's operation, the first captured image including the face image of the suspicious person is registered in the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be as follows.

[0036] According to this, facial images of suspicious individuals can be registered in a database with high accuracy using a simple configuration.

[0037] The following was done to solve the aforementioned problem. 10The invention relates to a face image registration method for a server in a face image registration system for registering images of a person's face, comprising: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user, wherein the server acquires the first and second captured images, respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, it transmits the first and second captured images to the terminal, respectively, the terminal is used by a user who has the authority to register face images in a database, the first captured image transmitted to the terminal includes a face image of the person who exhibited the suspicious behavior, the second captured image transmitted to the terminal includes an image of the suspicious person in the act of the suspicious behavior, and when the server receives a registration command transmitted from the terminal based on the user's operation, it registers the first captured image including the face image of the suspicious person in the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. The configuration will be as follows.

[0038] According to this, facial images of suspicious individuals can be registered in a database with high accuracy using a simple configuration.

[0039] The following was done to solve the aforementioned problem. 11The invention relates to a face image registration system for registering images of people's faces, comprising: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user. The server acquires the first and second captured images, respectively, and if a predetermined suspicious behavior is detected in the person's appearance included in the second captured image, the first and second captured images are transmitted to the terminal, respectively. The terminal is used by a user who has the authority to register face images in a database. The first captured image transmitted to the terminal includes a face image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the suspicious person during the suspicious behavior. When the terminal receives a registration command transmitted based on the user's operation, the first captured image including the face image of the suspicious person is registered in the database. The first captured image transmitted to the terminal is generated by extracting the face image region containing the face of the suspicious person from the entire image region of the first captured image. It is a program that executes a process.

[0040] According to this, facial images of suspicious individuals can be registered in a database with high accuracy using a simple configuration.

[0041] The embodiments of this disclosure will be described below with reference to the drawings.

[0042] (First Embodiment) Figure 1 is an overall configuration diagram of the face image registration system 1 according to the first embodiment. Figures 2, 3, and 4 are functional block diagrams of the face registration server 11, the behavior detection server 12, and the movement mapping server 13, respectively.

[0043] As shown in Figure 1, the facial image registration system 1 is used to register images of suspicious individuals in a store 2 (an example of a facility) into a database (a collection of data). The facial image registration system 1 includes multiple facial cameras 3 (an example of a first camera), multiple behavioral cameras 4 (an example of a second camera), a recorder 5, and a store terminal 6, all installed in the store 2. Each of these electronic devices 3-6 can communicate with other electronic devices via a LAN (Local Area Network) 7 installed within the store 2. The LAN 7 is connected to a wider area communication network 9, such as the Internet, via connection devices 8, such as a gateway and a router. As a result, each of the electronic devices 3-6 can communicate with electronic devices outside the store via the communication network 9.

[0044] Furthermore, the facial image registration system 1 includes a facial registration server 11, a behavior detection server 12, and a movement mapping server 13. These servers 11-13 are each connected to a communication network 9, enabling them to communicate with electronic devices 3-6 installed in the store 2.

[0045] Furthermore, the facial image registration system 1 can be applied not only to stores 2, but also to other facilities such as office buildings, public facilities like train stations and airports, and event venues. Since the multiple facial cameras 3 have similar configurations, they will be referred to as "facial camera 3" below unless otherwise specified. Similarly, since the multiple behavioral cameras 4 have similar configurations, they will be referred to as "behavioral camera 4" below unless otherwise specified.

[0046] The face-capture camera 3 consists of a video camera equipped with known hardware. The face-capture camera 3 is positioned to capture the faces of multiple people (i.e., users of the facility) staying in the store 2. In other words, the placement of the face-capture camera 3 (installation height, installation direction relative to the aisle, etc.) and shooting conditions are set to be suitable for capturing people's faces. The face-capture camera 3 generates captured images (hereinafter referred to as "face-captured images") that include the face image of each person by capturing the faces of multiple people. A face-captured image (an example of a first-image-captured image) does not need to include the entire figure of the person, but only needs to include at least the person's face or the area around the face (upper body). A face-captured image consists of moving images (multiple frames), but may also be a still image (or a single frame of a video).

[0047] The face-capture camera 3 can sequentially transmit the generated captured image data and its associated information (hereinafter referred to as "capture data") to the recorder 5 and the face registration server 11. The associated information of the captured image includes the camera ID (identifier for each camera), location information (coordinates within the image for each camera, including the installation location of each camera), and shooting conditions (including the field of view).

[0048] The behavioral camera 4, like the face-capture camera 3, consists of a video camera equipped with known hardware. The behavioral camera 4 is positioned to capture the movements of multiple people acting within the store 2. In other words, the placement and shooting conditions of the behavioral camera 4 are set to be suitable for capturing people in action (for example, a person picking up an item placed on a product display shelf). By capturing the movements of multiple people, the behavioral camera 4 generates a captured image (hereinafter referred to as "person captured image") that includes each person in action. The person captured image (an example of the second captured image) may include the entire figure of the person or the upper body of the person (especially around the hands and arms). The person captured image may consist of a moving image (multiple frames), but it may also be a still image (or a single frame of a video).

[0049] The behavioral camera 4, like the face-capture camera 3, can sequentially transmit the generated captured data to the recorder 5 and the face registration server 11.

[0050] Furthermore, a portion of the face-capture camera 3 can also function as a behavior-capture camera 4. In other words, face-capture images can be used as person-capture images if they clearly include a person's actions (i.e., the characteristics of a person's actions can be grasped from the image). Conversely, a portion of the behavior-capture camera 4 can also function as a face-capture camera 3. In other words, person-capture images can be used as face-capture images if they clearly include a person's face (i.e., the characteristics of a person's face can be grasped from the image).

[0051] Recorder 5 consists of a recording device equipped with known hardware. Recorder 5 acquires captured data from the face-capturing camera 3 and the behavior-capturing camera 4, respectively, and stores them in storage. Audio data may also be included in the captured image data. Recorder 5 may be installed outside the store 2. In that case, Recorder 5 can communicate with other electronic devices via the communication network 9.

[0052] The store terminal 6 includes computers such as PCs, tablet devices, and smartphones. The store terminal 6 is used by users such as the manager and staff of store 2. The store terminal 6 may also be installed outside of store 2 (or used outside of store 2). In that case, the store terminal 6 can communicate with other electronic devices in the facial image registration system 1 via the communication network 9.

[0053] The face registration server 11 performs processes such as registering the face images of each person (those identified as acting suspiciously) included in the face capture images into a database.

[0054] As shown in Figure 2, the face registration server 11 includes a control unit 21, a communication unit 22, and a storage unit 23.

[0055] In the control unit 21, the image data acquisition unit 31 acquires image data from the face-capture camera 3 via the communication unit 22. The image data includes face-capture image data and its associated information. The image data acquisition unit 31 can also acquire image data from the face-capture camera 3 stored in the recorder 5. The control unit 21 can function as a communication control unit that controls communication with other electronic devices by the communication unit 22.

[0056] The face image extraction unit 32 detects the faces of people contained in the acquired face images (each frame) and extracts the image region containing the detected face (hereinafter referred to as the "face image region") from the face images (the entire image region). The face image extraction unit 32 also obtains the coordinates of the position of the face within the face images (for example, a representative position such as the center or top of the face). If the face image extraction unit 32 extracts multiple face image regions for the same person, it can extract (or select) only the images in which the front of the person's face (hereinafter referred to as the "front face") is captured, based on known face recognition methods. For example, by preparing a pre-trained model using machine learning to detect a person's front face, the face image extraction unit 32 may use that pre-trained model to detect a front face. Note that images of front faces are not limited to images of faces directly in front, but may include images of faces tilted in a predetermined direction (up, down, left, or right) within a predetermined allowable angle range from directly in front.

[0057] The face feature extraction unit 33 extracts facial features (hereinafter referred to as "face features") from the extracted facial image region based on known methods. The face feature extraction unit 33 also assigns a face ID (face identifier) ​​to the extracted face features. The same face ID is assigned to facial features of the same person.

[0058] Alternatively, instead of processing in the face image extraction unit 32 and the face feature extraction unit 33, these processes may be configured to be performed by the face camera 3. In that case, the capture data acquisition unit 31 may acquire the face ID along with the capture data from the face camera 3 via the communication unit 22.

[0059] Data including face ID, camera ID, face position coordinates, and shooting time obtained for the face images to be processed (hereinafter referred to as "face data") is stored in the storage unit 23. Such face data is obtained for each face image generated by multiple face-capture cameras 3 installed at different locations (i.e., face images captured at different shooting locations). The obtained face data is then sequentially transmitted to the movement mapping server 13.

[0060] The face registration unit 34 records the face image (hereinafter referred to as "registered face image") of the person to be registered (the person judged to be acting suspiciously) in a database (a collection of data). Here, the face image is stored in the face database 35 stored in the storage unit 23. However, the face database 35 may be stored in any device other than the face registration server 11 (such as another server).

[0061] At least some of the functions of each part 31-34 in the control unit 21 can be realized by the processor executing a predetermined control program.

[0062] The communication unit 22 may consist of hardware including communication circuits for the face registration server 11 to communicate with other electronic devices via the communication network 9.

[0063] The memory unit 23 may consist of hardware such as storage for storing data and information necessary for processing by the face registration server 11. The memory unit 23 stores various data and information necessary for processing by the control unit 21.

[0064] The face registration server 11 appropriately includes one or more known computer hardware components such as a processor (CPU, MPU, etc.), memory, a display, input devices, a network interface, and storage.

[0065] Furthermore, it is also possible for the face-capture camera 3 to be equipped with at least some of the functions of the face image extraction unit 32 and the face feature extraction unit 33. In that case, the face registration server 11 can, for example, obtain information on the facial features of a person extracted by the face-capture camera 3 from the face-capture camera 3.

[0066] The behavior detection server 12 performs processing to detect specific behaviors (in this case, suspicious behaviors) from the appearance of each person included in the captured images of people.

[0067] As shown in Figure 3, the behavior detection server 12 includes a control unit 41, a communication unit 42, and a storage unit 43.

[0068] In the control unit 41, the shooting data acquisition unit 51 acquires shooting data from the action shooting camera 4 via the communication unit 42. The shooting data includes person shooting image data and its associated information. The shooting data acquisition unit 51 can also acquire shooting data from the action shooting camera 4 stored in the recorder 5.

[0069] The behavior detection unit 52 detects predetermined specific actions in the figure of a person included in the acquired person image (video). In this embodiment, the behavior detection unit 52 detects suspicious behavior of a person as a specific action. Suspicious behavior includes, for example, illegal actions by a person inside store 2 (e.g., shoplifting) and related actions (e.g., vigilance before and after shoplifting). The behavior detection unit 52 can detect specific actions of a person (in this case, suspicious behavior) based on known behavior recognition methods. For example, by preparing a pre-trained model using machine learning to detect predetermined specific actions, the behavior detection unit 52 may use that pre-trained model to detect specific actions. A person who is detected by the behavior detection unit 52 to be performing suspicious behavior is recognized as a suspicious person.

[0070] The behavioral image extraction unit 53 extracts images (some frames) in which suspicious behavior of a person is detected from the acquired person-captured images (video). The behavioral image extraction unit 53 also obtains the coordinates of the position of the person (suspicious person) within the person-captured images (for example, a representative position such as the center or top of the person). If the person-captured images consist of multiple frames (or if the position of the suspicious person changes within the video), the coordinates of the suspicious person's position are obtained from any one of the frames. Alternatively, the coordinates of the suspicious person's position may be a representative value (average value, etc.) of the coordinate values ​​obtained from multiple frames.

[0071] Data including the camera ID, the location coordinates of the suspicious person, and the time of shooting (hereinafter referred to as "suspicious behavior data") obtained with respect to the person image being processed is stored in the storage unit 43. Such suspicious behavior data only needs to be obtained for person images generated by at least one behavior-recording camera 4 in store 2. The obtained suspicious behavior data is then sequentially transmitted to the movement mapping server 13.

[0072] At least some of the functions of each part 51-33 in the control unit 41 can be realized by the processor executing a predetermined control program. In addition, the control unit 21 can function as a communication control unit that controls communication with other electronic devices by the communication unit 42.

[0073] The communication unit 42, like the communication unit 22, may be composed of hardware including communication circuits for the action detection server 12 to communicate with other electronic devices via the communication network 9.

[0074] The memory unit 43 may consist of hardware such as storage for storing data and information necessary for processing by the behavior detection server 12. The memory unit 43 stores various data and information necessary for processing by the control unit 41.

[0075] The behavior detection server 12, like the face registration server 11, is appropriately equipped with hardware for configuring a known computer. In this embodiment, the behavior detection server 12 defines a specific action of the person it detects as suspicious behavior, but this is not limited to this, and various changes are possible depending on the characteristics of the person whose face image should be registered. For example, the behavior detection server 12 may perform processing to detect specific actions by regular customers of store 2. In that case, the processing of other servers that constitute the face image registration system 1 may also be changed in accordance with the change in the target of detection by the behavior detection server 12.

[0076] The movement mapping server 13 performs processing such as associating people included in face images with suspicious individuals included in person images.

[0077] As shown in Figure 4, the movement mapping server 13 includes a control unit 61, a communication unit 62, and a storage unit 63.

[0078] In the control unit 61, the movement path generation unit 71 extracts face data relating to the same person (i.e., a person assigned the same face ID) from multiple face data received from the face registration server 11. Then, the movement path generation unit 71 estimates the movement path of the person being processed in store 2 based on the multiple face data relating to that same person (the person being processed). The multiple face data includes multiple face images captured by multiple face-capture cameras 3.

[0079] The movement path generation unit 71 can convert the position coordinates of each face (coordinates within each image) of the person to be processed, for each person included in multiple face images taken from different shooting positions, into global coordinates. In this case, the global coordinates of the face can be set as two-dimensional coordinates including coordinate values ​​on a virtual plane corresponding to the floor surface of store 2. Furthermore, the movement path generation unit 71 can estimate the movement path of the person to be processed, based on the global coordinates of the face of the person to be processed, for each person included in multiple face images taken from different shooting positions, and the time of shooting.

[0080] The behavior coordinate acquisition unit 72 can convert the position coordinates (coordinates within the image) of a suspicious person in a person image into global coordinates based on the suspicious behavior data received from the behavior detection server 12. In this case, the global coordinates of the suspicious person may be set as two-dimensional coordinates including coordinate values ​​on a virtual plane corresponding to the floor surface of the store 2.

[0081] The face ID identification unit 73 maps the movement paths of multiple subjects (more specifically, the coordinates of the points constituting the movement paths) to the position coordinates of the suspicious person and extracts the movement paths associated with the suspicious person. For example, the face ID identification unit 73 can calculate the distance between the movement paths of multiple subjects and the position coordinates of the suspicious person and extract the movement path with the smallest distance. As a result, the face ID identification unit 73 identifies the face ID of the subject corresponding to the extracted movement path as the face ID of the suspicious person.

[0082] The facial image search unit 74 causes the facial registration server 11 to search for facial images that include individuals with the same facial ID as the suspicious person.

[0083] The behavioral image search unit 75, based on the suspicious behavior data of a person whose facial ID has been identified, causes the behavior detection server 12 to search for images of people (a series of frames including the suspicious person) that include that person.

[0084] At least some of the functions of each of the parts 71-75 in the control unit 61 can be realized by the processor executing a predetermined control program. Furthermore, the control unit 61 can function as a communication control unit that controls communication with other electronic devices by the communication unit 62.

[0085] The communication unit 62, like the communication unit 22, may be composed of hardware including communication circuits for the movement mapping server 13 to communicate with other electronic devices via the communication network 9.

[0086] The memory unit 63 may consist of hardware such as storage for storing data and information necessary for processing by the movement mapping server 13. The memory unit 43 stores various data and information necessary for processing by the control unit 61.

[0087] The movement mapping server 13 is equipped with hardware suitable for configuring a known computer, similar to the face registration server 11.

[0088] In this embodiment, for the sake of explanation, a configuration consisting of three servers (face registration server 11, behavior detection server 12, and movement mapping server 13) is used, but this configuration can be changed as appropriate. For example, at least some of the functions of servers 11-13 may be performed by other servers. Alternatively, all of the functions of servers 11-13 may be implemented by a single server. Or, more servers capable of collaboratively performing the functions of servers 11-13 may be provided. At least one of these servers 11-13 may be installed within store 2.

[0089] Figure 5 is an explanatory diagram showing the layout of Store 2. Here, a convenience store is shown as an example of Store 2.

[0090] As shown in Figure 5, in store 2, multiple face-capturing cameras 3A-3L are placed in appropriate locations within store 2 (on the interior walls of store 2, in the product display area, on product display shelves, etc.). In addition, in store 2, multiple behavior-capturing cameras 4A-4D are placed in appropriate locations within store 2 (on the interior walls of store 2, on the ceiling, etc.).

[0091] Each of the face-capturing cameras 3A-3L should be positioned lower than the behavior-capturing cameras 4A-4D to facilitate capturing a person's face. Additionally, at least one of the multiple face-capturing cameras 3 (see face-capturing cameras 3A and 3L) should be positioned to capture a person's face from the front, such as when a person enters the store through the entrance 80 or when a person is paying at the cash register 81 at the counter.

[0092] Each of the behavior-recording cameras 4A-4D should have a larger field of view (or shooting range) than the face-recording cameras 3A-3L to facilitate the recording of people in action. Note that multiple behavior-recording cameras 4 are not necessarily required within the store 2; a single behavior-recording camera 4 may be installed.

[0093] Figures 6 and 7 are sequence diagrams showing the operation procedure of the face image registration system 1 according to the first embodiment. These sequence diagrams are a series, and the operation shown in Figure 6 is followed by the operation shown in Figure 7. Figure 8 is an explanatory diagram of the process for identifying the face ID of a suspicious person in step S9 in Figure 6. Figure 9 is an explanatory diagram showing an example of a screen for the user to select a face image to be registered (selection screen for candidate images to be registered) in step S13 in Figure 7. Figure 10 is an explanatory diagram showing an example of a screen for the user to confirm the suspicious behavior of a suspicious person (confirmation screen for suspicious behavior).

[0094] As shown in Figure 6, the face-capture camera 3 sequentially transmits the generated captured data to the recorder 5. The recorder 5 stores the captured data received from the face-capture camera 3 in its storage (S1).

[0095] Furthermore, the face-capture camera 3 sequentially transmits the generated captured data to the face registration server 11. The face registration server 11 detects the faces of people contained in the received captured data (face images) and extracts the face image region (S2). Subsequently, the face registration server 11 extracts face features from the extracted face image region and assigns a face ID to those face features (S3). The face registration server 11 transmits the face data, including the camera ID corresponding to the face ID, the position coordinates of the face, and the time of capture, to the movement mapping server 13 and stores it in the storage unit 23.

[0096] Meanwhile, the behavioral camera 4, like the face-capture camera 3, sequentially transmits the generated captured data to the recorder 5. The recorder 5 stores the captured data received from the behavioral camera 4 in its storage (S4).

[0097] Furthermore, the motion-capture camera 4 sequentially transmits the generated captured data to the motion detection server 12. The motion detection server 12 detects predetermined suspicious behavior in the person's appearance included in the received captured data (image of a person) (S5). The motion detection server 12 transmits the suspicious behavior data, including the coordinates of the detected suspicious person's location, to the movement mapping server 13.

[0098] The movement mapping server 13 acquires information about each face-capture camera 3 and each action-capture camera 4 (hereinafter referred to as "camera information") and stores it in the storage unit 63 (S6). The camera information includes information about the installation position and shooting conditions of each camera. When the movement mapping server 13 receives face data from the face registration server 11, it converts the position coordinates of the faces in each face data into global coordinates and estimates the movement path for each person to generate movement path information (S7). The generated movement path information is stored in the storage unit 63.

[0099] Furthermore, when the movement mapping server 13 receives suspicious behavior data from the behavior detection server 12, it obtains the global coordinates of the suspicious person in the captured image based on that suspicious behavior data (S8). In addition, the movement mapping server 13 identifies the face ID of the suspicious person by mapping the movement paths of multiple people (subjects being processed) with the position coordinates (global coordinates) of the suspicious person (S9).

[0100] Regarding the processing in step S9, consider, for example, the case in Figure 8 where a person 100 enters the store from the store entrance 80 and then moves within the store 2 as shown by the dashed line with an arrow (hereinafter referred to as the "actual movement line"). At this time, multiple face-capturing cameras 3A, 3B, 3C, 3G, 3J, and 3K generate face images of the moving person 100. The multiple locations where the face of person 100 is captured correspond to the points labeled "face image" on the actual movement line in Figure 8.

[0101] The movement mapping server 13 can estimate the movement path (trajectory of movement) of person 100 based on the global coordinates of the person's face included in the face images captured by face cameras 3A, 3B, 3C, 3G, 3J, and 3K, and the time of their capture. For example, the movement mapping server 13 can estimate a movement path that approximates the actual movement path of person 100 by connecting the points of global coordinates at each capture position of person 100's face through a simple process.

[0102] Furthermore, Figure 8 shows a case where suspicious behavior is detected from the image of person 100 captured by the behavior-recording camera 4C. The location where person 100 performing the suspicious behavior was captured corresponds to the point labeled "Suspicious Behavior Image" on the actual movement path in Figure 8. The movement path mapping server 13 can map the movement paths of multiple people (other than person 100 are not shown), including person 100's movement path, with the coordinates of the location where person 100 performed the suspicious behavior (location coordinates of the suspicious person) to extract the movement path of person 100 (the suspicious person). At this time, the movement path mapping server 13 can extract the movement path of the suspicious person from the estimated movement paths for each person, based on whether the coordinate value of the location of the person who performed the suspicious behavior (in this case, person 100) matches (or is an approximation of) the coordinate value of a point constituting each estimated movement path for one or more people (including person 100) in store 2.

[0103] Next, as shown in Figure 7, the movement mapping server 13 transmits the face ID of the identified suspicious person to the face registration server 11, and also transmits an image search command to the face registration server 11 to search for face images containing a person with a face ID that matches that face ID (S10).

[0104] The face registration server 11 searches for a corresponding face image based on the face ID of the suspicious person received from the movement mapping server 13 (S11). At this time, the face registration server 11 obtains photographic data (including one or more face images) related to the suspicious person from the photographic data stored in the recorder 5, based on the camera ID and time of shooting corresponding to the face ID of the suspicious person. Furthermore, the face registration server 11 extracts an image of the suspicious person's frontal face from the acquired face images (S12). The face registration server 11 transmits the extracted face image (image of the frontal face) and the corresponding face ID to the store terminal 6.

[0105] When the store terminal 6 receives a face capture image (a frontal view of the face) and a corresponding face ID from the face registration server 11, it displays the received face capture image on its display as a candidate image for registration (S13). This allows the user of the store terminal 6 to select one or more images to register from the displayed candidate images. For example, as shown in Figure 9, multiple candidate images (A)-(H) are displayed on the display of the store terminal 6.

[0106] Meanwhile, after sending an image search command (S10) to the face registration server 11, the movement mapping server 13 sends an image search command (S14) to the behavior detection server 12 to search for images of people, including suspicious individuals in motion.

[0107] The behavior detection server 12 searches for corresponding person images based on the face ID of the suspicious person received from the movement mapping server 13 (S15). At this time, the behavior detection server 12 obtains corresponding shooting data (including person images) from the shooting data stored in the recorder 5 based on the location coordinates and shooting time of the suspicious person received from the movement mapping server 13. Furthermore, the behavior detection server 12 transmits the person image data (suspicious behavior image data) including the person performing the suspicious behavior that it obtained thereto to the store terminal 6.

[0108] When the store terminal 6 receives a person image (including video footage of a suspicious person) from the behavior detection server 12, it displays the received person image on its display as an image for the user to confirm the suspicious behavior (hereinafter referred to as the "confirmation image") (S16). At this time, for example, as shown in Figure 10, the store terminal 6's display shows a screen for confirming suspicious behavior. The screen for confirming suspicious behavior in Figure 10 includes a display area for displaying two confirmation images taken at different times. The user can perform operations related to the display of each image (start playback, pause, fast forward, etc.) by operating the operation button 87. By checking the displayed confirmation images, the user of the store terminal 6 can determine whether it was appropriate to identify the person being processed as a suspicious person.

[0109] The user of the store terminal 6 can, after confirming on the suspicious behavior confirmation screen in Figure 10 that the person being processed has been appropriately identified as a suspicious person, select a facial image of the suspicious person to be registered and decide to register it (S17). More specifically, the user can decide on a facial image of the suspicious person to be registered by selecting one or more images that they deem suitable for registration from the candidate images (A)-(H) shown on the registration candidate image selection screen in Figure 9 above, and pressing the registration button 85.

[0110] At this time, the store terminal 6 sends a registration command to the face registration server 11 for the registration candidate image that the user has decided to register. The registration command includes identification information (for example, the registration image number) of the registration candidate image selected by the user. Based on the identification information of the registration candidate image, the face registration server 11 extracts the image to be registered (including the face image of the suspicious person) from the registration candidate image and registers it in the face database 35 (S18).

[0111] Note that the sequence (execution timing) of each operation shown in Figures 6 and 7 is not limited to those shown in the figures and can be changed as appropriate.

[0112] (Second Embodiment) Figure 11 is an overall configuration diagram of the face image registration system 1 according to the second embodiment. Figure 12 is a functional block diagram of the person tracking server 113. In Figure 11, components in the face image registration system 1 according to the second embodiment that are the same as those in the first embodiment are denoted by the same reference numerals, and detailed explanations are omitted. Furthermore, with respect to the face image registration system 1 according to the second embodiment, matters not specifically mentioned below are the same as in the first embodiment.

[0113] In the face image registration system 1 according to the second embodiment, the surveillance camera 103 has a configuration and function that is generally the same as the face-capturing camera 3 according to the first embodiment. However, the placement and shooting conditions of the surveillance camera 103 do not necessarily need to be set to prioritize the capture of people's faces. A portion of the surveillance camera 103 can also function as a behavior-capturing camera 4.

[0114] Furthermore, the face registration server 11 has a configuration and functions that are generally the same as the face registration server 11 according to the first embodiment. However, in the face registration server 11, the function of the face feature extraction unit 33 (see Figure 3) in the face registration server 11 is omitted.

[0115] Furthermore, the person tracking server 113 extracts features related to the appearance of the suspicious person (hereinafter referred to as "person features") and performs processes such as extracting images of people that include the suspicious person.

[0116] As shown in Figure 12, the person tracking server 113 includes a control unit 91, a communication unit 92, and a storage unit 93.

[0117] In the control unit 91, the image data acquisition unit 121 acquires data about suspicious persons from the behavior detection server 12 via the communication unit 92. This data about suspicious persons includes image data of a person including the suspicious person and its associated information. The control unit 91 can function as a communication control unit that controls communication with other electronic devices by the communication unit 92.

[0118] The person image extraction unit 122 detects the figure of a person contained in the acquired person image and extracts the region containing the detected person (hereinafter referred to as the "person image region"). For example, by preparing a pre-trained machine learning model for extracting the figure of a person contained in an image, the person image extraction unit 122 may use the pre-trained model to extract the figure of a person.

[0119] The person feature extraction unit 123 extracts the feature quantities (hereinafter referred to as "person features") of the person (suspicious person) in the extracted person image region based on known methods.

[0120] The image search unit 124, based on the personal features extracted for each suspicious person, causes the face registration server 11 and the behavior detection server 12 to search for face images and person images (a series of frames including the suspicious person) that correspond to the suspicious person, based on those personal features.

[0121] At least some of the functions of each of the parts 121-124 in the control unit 91 can be realized by the processor executing a predetermined control program.

[0122] The communication unit 92, like the communication unit 22, may consist of hardware including communication circuits for the person tracking server 113 to communicate with other electronic devices via the communication network 9.

[0123] The memory unit 93 may consist of hardware such as storage for storing data and information necessary for processing by the person tracking server 113. The memory unit 93 stores various data and information necessary for processing by the control unit 91.

[0124] Figures 13 and 14 are sequence diagrams showing the operation procedure of the face image registration system 1 according to the second embodiment. These sequence diagrams are a series of actions, and the actions shown in Figure 14 are performed following the actions shown in Figure 13.

[0125] As shown in Figure 13, the surveillance camera 103 sequentially transmits the generated captured data to the recorder 5. The recorder 5 stores the captured data received from the surveillance camera 103 in its storage (S21).

[0126] Meanwhile, the behavioral camera 4, like the surveillance camera 103, sequentially transmits the generated captured data to the recorder 5. The recorder 5 stores the captured data received from the behavioral camera 4 in its storage (S22).

[0127] Furthermore, the camera 4 for capturing movement sequentially transmits the generated captured data to the movement detection server 12. The movement detection server 12 detects predetermined suspicious behavior in the figure of a person included in the received captured data (image of a person) (S23). The movement detection server 12 transmits the person tracking server 113 suspicious behavior data, including the image data of the person who performed the detected suspicious action, the camera ID, and the time of capture.

[0128] When the person tracking server 113 receives suspicious behavior data from the behavior detection server 12, it extracts the person image region by detecting the person's figure (full body, upper body, or lower body, etc.) from the person image (S24). Next, the person tracking server 113 extracts person features related to the person in the extracted person image region (S25). Furthermore, the person tracking server 113 requests the recorder 5 to capture data generated by the surveillance camera 103 and the behavior recording camera 4 (S26). At this time, the person tracking server 113 can set a predetermined time period including the time the person image in which suspicious behavior was detected in step S23 was captured, and request the recorder 5 to capture data for that time period.

[0129] Subsequently, when the person tracking server 113 obtains the captured data (including images of people) from the recorder 5, it extracts images (multiple frames) from the captured images of people that contain the figure of the suspicious person (S28).

[0130] Next, as shown in Figure 14, the person tracking server 113 extracts images (one or more frames) containing the face of the suspicious person from the image containing the figure of the suspicious person extracted in step S28 (S29). The person tracking server 113 then transmits the image data containing the figure of the suspicious person extracted in step S28 and the image data containing the face of the suspicious person extracted in step S29 to the store terminal 6, respectively.

[0131] When the store terminal 6 receives image data including the image of a suspicious person from the person tracking server 113, it displays the received image including the image of the suspicious person on the display as a confirmation image, similar to the store terminal 6 in the first embodiment (see the suspicious behavior confirmation screen in Figure 10) (S30).

[0132] Furthermore, when the store terminal 6 receives image data including the face of a suspicious person from the person tracking server 113, it displays the received image including the face of the suspicious person on the display as a candidate image for registration, similar to the store terminal 6 in the first embodiment (see the candidate image selection screen in Figure 9) (S31).

[0133] On the suspicious behavior confirmation screen, the user of the store terminal 6 can confirm that it is appropriate that the person being processed has been determined to be a suspicious person, and then select the face image of the suspicious person to be registered and decide to register it (S32).

[0134] At this time, the store terminal 6 sends a registration command to the face registration server 11 for the image of the registration candidate that the user has decided to register. The registration command includes image data (image data including a face) of the registration candidate selected by the user. The face registration server 11 detects a person's face from the received image data of the registration candidate and extracts the image (face image region) containing the detected face (S33). At this time, the face registration server 11 assigns a face ID to the extracted image containing the face. Furthermore, the face registration server 11 registers the extracted image containing the face in the database as the face image of the person to be registered (the person judged to be acting suspiciously) (S34).

[0135] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0136] The facial image registration system, facial image registration method, server, server facial image registration method, and program related to this disclosure have the effect of being able to accurately register facial images of persons who have performed suspicious actions in the past at a facility into a database with a simple configuration, and are useful as a facial image registration system, facial image registration method, server, server facial image registration method, and program for registering facial images of persons who perform suspicious actions at facilities such as stores. [Explanation of Symbols]

[0137] 1: Face Image Registration System 2: Stores 3: Camera for capturing facial images (first camera) 4: Camera for recording behavior (second camera) 5: Recorder 6: Store terminal 7: LAN 8: Connected devices 9: Communication Network 11: Face registration server 12: Behavior detection server 13: Movement flow mapping server 21: Control Unit 31: Shooting data acquisition unit 32: Face image extraction unit 33: Facial Feature Extraction Unit 34: Face Registration Section 35: Face Database 41: Control Unit 51: Shooting data acquisition unit 52: Behavior detection unit 53: Behavioral Image Extraction Unit 61: Control Unit 71: Flow line generation part 72: Action coordinate acquisition unit 73: Face ID Identification Department 74: Face Image Search Department 75: Behavioral Image Search Department 80: Store entrance / exit 91: Control Unit 100: Person 103: Surveillance camera 113: Person Tracking Server 121: Shooting data acquisition unit 122: Person Image Extraction Unit 123: Human Feature Extraction Unit 124: Image Search Department

Claims

1. A facial image registration system for registering images of people's faces, A first camera installed in the facility and capable of generating a first captured image by photographing a person's face, The facility is equipped with at least one second camera capable of generating a second image by photographing the person in action, A server that processes the facial image of the person included in the first captured image to register it in a database, Equipped with, The aforementioned server, The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, the first captured image and the second captured image are transmitted to a terminal used by the user, the first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. The first captured image transmitted to the terminal is generated by extracting a face image region containing the face of the suspicious person from the entire image region of the first captured image, in a face image registration system.

2. A face image registration system for registering images of a person's face, A first camera installed in the facility and capable of generating a first captured image by photographing a person's face, The facility is equipped with at least one second camera capable of generating a second image by photographing the person in action, A server that processes the facial image of the person included in the first captured image to register it in a database, Equipped with, The aforementioned server, The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, the first captured image and the second captured image are transmitted to a terminal used by the user, the first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. The first captured image transmitted to the terminal includes a plurality of different registration candidate images, A face image registration system in which the registration command includes information about at least one image selected from among the multiple registration candidate images based on the user's operation.

3. A face image registration system for registering images of a person's face, A first camera installed in the facility and capable of generating a first captured image by photographing a person's face, The facility is equipped with at least one second camera capable of generating a second image by photographing the person in action, A server that processes the facial image of the person included in the first captured image to register it in a database, Equipped with, The aforementioned server, The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, the first captured image and the second captured image are transmitted to a terminal used by the user, the first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. A plurality of different first images are obtained, including the facial image of the person acting suspiciously. A face image registration system that extracts one or more images from the plurality of first captured images, including the face image corresponding to the front of the face of the suspicious person, and transmits the extracted images to the terminal.

4. A face image registration system for registering images of a person's face, A first camera installed in the facility and capable of generating a first captured image by photographing a person's face, The facility is equipped with at least one second camera capable of generating a second image by photographing the person in action, A server that processes the facial image of the person included in the first captured image to register it in a database, Equipped with, The aforementioned server, The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, the first captured image and the second captured image are transmitted to a terminal used by the user, the first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. If the suspicious behavior is detected in the figure of the person included in the second captured image, the time the suspicious person was photographed during the suspicious behavior included in the second captured image and the location information of the suspicious person corresponding to that time are obtained. A face image registration system that obtains a first photographed image including the face image of the suspicious person by associating the person included in the first photographed image with the suspicious person based on the time of the photograph and the location information of the suspicious person.

5. The aforementioned at least one first camera includes a plurality of first cameras, The aforementioned server, The first captured image is acquired from the plurality of first cameras. The facial feature quantities relating to the person's face included in the first captured image obtained from the plurality of first cameras are acquired. For a person whose facial features match the above, multiple shooting times of the person's face included in the first captured image and multiple location information of the person corresponding to those multiple shooting times are obtained, Based on the aforementioned multiple shooting times and the aforementioned multiple location information, the movement path of the person is estimated. The face image registration system according to claim 4, which associates the person included in the first captured image with the suspicious person based on the movement path and the time and location information of the suspicious person being photographed.

6. A face image registration system for registering images of a person's face, A first camera installed in the facility and capable of generating a first captured image by photographing a person's face, The facility is equipped with at least one second camera capable of generating a second image by photographing the person in action, A server that processes the facial image of the person included in the first captured image to register it in a database, Equipped with, The aforementioned server, The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, the first captured image and the second captured image are transmitted to a terminal used by the user, the first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. If the suspicious behavior is detected in the figure of the person included in the second captured image, the person's features related to the suspicious person's appearance are acquired. A face image registration system that, based on the aforementioned person characteristics, associates the person included in the first captured image with the suspicious person, thereby obtaining the first captured image including the face image of the suspicious person.

7. The facial image registration system according to any one of claims 1 to 6, further comprising the terminal used by the user.

8. A method for registering images of people's faces, At least one first camera installed in the facility generates a first captured image by photographing a person's face. At least one second camera installed in the facility generates a second image by capturing the figure of the person in action. Computers The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the person's appearance in the second captured image, the first captured image and the second captured image are transmitted to a terminal, which is used by a user who has the authority to register facial images in a database. The first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. A face image registration method wherein the first captured image transmitted to the terminal is generated by extracting a face image region containing the face of the suspicious person from the entire image region of the first captured image.

9. A server in a face image registration system for registering images of a person's face, comprising: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user, The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the figure of the person included in the second captured image, the first captured image and the second captured image are transmitted to the terminal, the first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. The first captured image transmitted to the terminal is generated by extracting a face image region containing the face of the suspicious person from the entire image region of the first captured image, and is transmitted to the server.

10. A face image registration system for registering images of a person's face, comprising: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user, wherein the server is a face image registration method, The server The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the person's appearance in the second captured image, the first captured image and the second captured image are transmitted to a terminal, which is used by a user who has the authority to register facial images in a database. The first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. A face image registration method wherein the first captured image transmitted to the terminal is generated by extracting a face image region containing the face of the suspicious person from the entire image region of the first captured image.

11. A face image registration system for registering images of people's faces comprises: at least one first camera installed in a facility that generates a first captured image by photographing a person's face; at least one second camera installed in the facility that generates a second captured image by photographing the person in action; a server; and a terminal used by a user. The first and second captured images are acquired, If a predetermined suspicious behavior is detected in the person's appearance in the second captured image, the first captured image and the second captured image are transmitted to a terminal, which is used by a user who has the authority to register facial images in a database. The first captured image transmitted to the terminal includes a facial image of the person who exhibited the suspicious behavior, and the second captured image transmitted to the terminal includes an image of the person exhibiting the suspicious behavior. When a registration command is received from the terminal based on the user's operation, the first captured image, including the facial image of the suspicious person, is registered in the database. The first captured image transmitted to the terminal is a program for executing a process, which is generated by extracting a face image region containing the face of the suspicious person from the entire image region of the first captured image.

Images