Lost person finder device and lost person finding system

Abstract

The present application provides a lost person finding device and a lost person finding system capable of quickly finding a lost person. The lost person finding device includes an acquisition unit, a presumption unit, and a determination unit. The acquisition unit acquires an image of a registered target person and an image of a registered guardian who is a guardian of the target person from captured images captured by each camera arranged in a prescribed range when it is detected that the registered target person and the registered guardian enter the prescribed range. The presumption unit presumes a possibility of the target person being lost based on the images of the target person and the guardian acquired by the acquisition unit. The determination unit determines a position of the target person according to a capturing position of the camera when the possibility of the target person being lost presumed by the presumption unit exceeds a prescribed threshold.

Description

Technical Field

[0001] This invention relates to a device and system for finding missing persons. Background Technology

[0002] In recent years, large shopping malls and similar facilities have seen a surge in visitors. However, many of these visitors are families with children, and as these facilities expand, the number of children going missing has also increased.

[0003] Patent document 1: Japanese Patent Application Publication No. 2016-152512.

[0004] Patent document 2: Japanese Patent Application Publication No. 2002-101444. Summary of the Invention

[0005] For example, there are many cases where children accompanied by parents or other guardians go missing while using facilities, and these children often cannot state their own names or the names of their guardians, making it time-consuming to find them. Therefore, there is a need for a system that can quickly locate missing persons, even if children or other persons accompanied by guardians go missing within facilities.

[0006] One aspect of the present invention is to provide a missing person detection device or the like that is capable of quickly locating missing persons.

[0007] A missing person detection device includes an acquisition unit, an estimation unit, and a determination unit. The acquisition unit acquires images of the registered person and their registered guardian from images captured by cameras positioned within the specified area when it detects that a registered person and their registered guardian have entered a designated area. The estimation unit estimates the likelihood of the person being missing based on the images of the person and guardian acquired by the acquisition unit. The determination unit determines the location of the person based on the camera positions when the estimated likelihood of the person being missing exceeds a predetermined threshold.

[0008] On the one hand, it enables the rapid discovery of missing persons. Attached Figure Description

[0009] Figure 1 This is an explanatory diagram illustrating an example of the missing persons discovery system of this embodiment.

[0010] Figure 2 This is a block diagram illustrating an example of the hardware structure of a mobile terminal.

[0011] Figure 3 This is a block diagram representing an example of the hardware structure of a server.

[0012] Figure 4This is a block diagram representing an example of the functional structure of a server.

[0013] Figure 5 This is an illustrative diagram showing an example of a table structure for a registration memory.

[0014] Figure 6 This is an illustrative diagram showing an example of the table structure of the memory when entering the store.

[0015] Figure 7 This is an illustrative diagram showing an example of the table structure of a real-time image storage device.

[0016] Figure 8 This is a block diagram illustrating an example of the hardware structure of a facility-side terminal.

[0017] Figure 9 This is a flowchart illustrating an example of server processing actions related to registration.

[0018] Figure 10 This is an illustrative diagram showing an example of a mobile terminal screen related to a registration request.

[0019] Figure 11 This is an illustrative diagram illustrating an example of server processing actions that extract facial images of the guardian and the subject from a provided image.

[0020] Figure 12 This is a flowchart illustrating an example of server processing actions related to monitoring.

[0021] Figure 13 This is a flowchart illustrating an example of server processing actions related to monitoring.

[0022] Figure 14 This is an illustrative diagram illustrating an example of server processing actions related to facial image recognition when a person enters a store.

[0023] Figure 15 This is an illustrative diagram showing an example of an in-store report screen from a facility-side terminal.

[0024] Figure 16 This is an illustration of an example of a missing person report screen before a protection request is received from a facility-side terminal.

[0025] Figure 17 This is an illustration of an example of a missing person notification screen for a mobile device.

[0026] Figure 18 This is an illustration of an example of a missing person report screen following a protection request from a facility-side terminal.

[0027] Explanation of reference numerals in the attached figures 1. Missing Person Discovery System 2 Mobile terminals 3 servers 4. Facilities 5 surveillance cameras 6. Facility-side terminals 31. Registered storage 32. Store entry memory 33 Real-time image storage 41 Registration Department 42 Acquisition Department 43. Speculation Department 44. Determining the Department 45. Notification Department. Detailed Implementation

[0028] The following will describe in detail, based on the accompanying drawings, embodiments of the missing persons detection device disclosed in this application. Furthermore, these embodiments are not limited to the disclosed technology. Additionally, the following embodiments can be appropriately combined within a reasonable scope.

[0029] Example Figure 1 This is an explanatory diagram illustrating an example of the missing persons detection system 1 of this embodiment. For example... Figure 1 As shown, the missing persons detection system 1 includes: a mobile terminal 2, a server 3, multiple surveillance cameras 5 within a facility 4, and a facility-side terminal 6 within the facility 4. The facility 4 includes: multiple surveillance cameras 5 located throughout the facility, and facility-side terminals 6 for staff patrolling the facility 4. The mobile terminal 2 is, for example, a smartphone or tablet with built-in camera and communication functions, or a terminal device for guardians using the facility. The server 3 is, for example, a computer or other device for detecting missing persons within the facility 4, performing various processes. Each surveillance camera 5 is located throughout the facility 4 and is a camera that captures moving images. The facility-side terminal 6 is a specified terminal device carried by staff patrolling the facility 4, such as a smartphone or tablet with built-in camera and communication functions. The facility-side terminal 6 may also include monitoring devices located in the facility 4 management room, etc.

[0030] Mobile terminal 2, server 3, surveillance camera 5, and facility-side terminal 6 are configured to communicate wirelessly via a wireless LAN (Local Area Network) router, for example. Mobile terminal 2 can send images containing the guardian and the target person to server 3 via a wireless LAN router. Alternatively, mobile terminal 2 can also communicate with server 3 via a public wireless communication network, which is self-evident. The target person is, for example, a child or elderly person under the guardianship of a guardian. The guardian is, for example, the child's parents or related personnel. The provided image is an image containing facial images representing the characteristics of the guardian and the target person. Server 3 receives the provided image from mobile terminal 2.

[0031] When server 3 detects a registration request from mobile terminal 2, it registers the facial images and features of the guardian and the person being registered. Server 3 identifies the registered guardian and person being registered from images captured by surveillance cameras 5 located at the entrance of facility 4. Server 3 uses the surveillance cameras 5 within facility 4 to capture real-time video of registered guardians and persons being registered entering facility 4. Then, based on the images captured by the surveillance cameras 5, server 3 estimates the likelihood of the person being lost and determines whether the person is indeed lost based on the estimation results. If the person is lost, server 3 notifies facility-side terminal 6 of a loss report and simultaneously notifies the guardian's mobile terminal 2. As a result, staff at facility-side terminal 6 can confirm the likelihood of the registered person being lost, and the guardian at mobile terminal 2 can also confirm the likelihood of the person being lost.

[0032] Figure 2 This is a block diagram illustrating an example of the hardware structure of mobile terminal 2. For example... Figure 2As shown, the mobile terminal 2 includes: a communication unit 11, an input unit 12, an output unit 13, a camera unit 14, a ROM (Read Only Memory) 15, a RAM (Random Access Memory) 16, a CPU (Central Processing Unit) 17, and a bus 18. The communication unit 11 is, for example, a communication interface (IF) that communicates wirelessly with a wireless LAN router and simultaneously with a public wireless communication network. The input unit 12 is an input IF such as an operating device for inputting various information. The output unit 13 is an output IF such as an audio output device or a display device for outputting various information. The camera unit 14 is, for example, used to acquire facial images representing the characteristics of a person. The ROM 15 is an area for storing various information, such as programs. The RAM 16 is an area for storing various information. The CPU 17 is used to control the entire mobile terminal 2. The bus 18 is a bus line connecting the communication unit 11, the input unit 12, the output unit 13, the camera unit 14, the ROM 15, the RAM 16, and the CPU 17.

[0033] Figure 3 This is a block diagram illustrating an example of the hardware architecture of server 3. For example... Figure 3 As shown, server 3 includes: a communication unit 21, an input unit 22, an output unit 23, an HDD (Hard Disk Drive) 24, a ROM 25, a RAM 26, a CPU 27, and a bus 28. The communication unit 21 is, for example, a communication interface that wirelessly connects to a wireless LAN router and simultaneously connects to a public wireless communication network. The input unit 22 is an input interface such as an operating device for inputting various information. The output unit 23 is an output interface such as an audio output device or a display device for outputting various information. The HDD 24 is an area for storing various information. The ROM 25 is an area for storing various information, such as programs. The RAM 26 is an area for storing various information. The CPU 27 is used to control the entire server 3. The bus 28 is a bus line connecting the communication unit 21, the input unit 22, the output unit 23, the HDD 24, the ROM 25, the RAM 26, and the CPU 27.

[0034] Figure 4 This is a block diagram illustrating an example of the functional structure of server 3. For example... Figure 4As shown, server 3 includes a storage unit 30 and a control unit 40. The storage unit 30 corresponds to, for example, semiconductor memory elements such as RAM 26 and flash memory, or storage devices such as HDD 24 and optical discs. The storage unit 30 includes a registration memory 31, an entry-time memory 32, a real-time image memory 33, and a facility-wide map memory 34. The registration memory 31 stores information including facial images and feature information of the person and their guardian. The entry-time memory 32 stores entry-time information, including images of the registered person and their guardian captured upon entering facility 4. The real-time image memory 33 stores information including images of the registered person and their guardian within facility 4. The facility-wide map memory 34 stores a map of facility 4 that allows identification of the shooting range of the surveillance cameras 5 deployed within facility 4.

[0035] The surveillance cameras 5 in this system are configured in different locations within facility 4, and the location information of each surveillance camera 5 is stored in association with the facility's map storage 34. The facility's map storage 34 maintains the ability to identify the installation coordinates and shooting range of each surveillance camera 5. The control unit 40 refers to the facility's map storage 34 to uniformly manage the camera location information within the facility.

[0036] The control unit 40 corresponds to an electronic circuit such as the CPU 27. The control unit 40 has an internal memory for storing programs and control data that define various processing steps, and executes various processes through these. The CPU 27 expands the program stored in the ROM 25 to the RAM 26, for example. The CPU 27 executes the program expanded to the RAM 26 as a process, thereby functioning as, for example, a registration unit 41, an acquisition unit 42, an estimation unit 43, a determination unit 44, and a notification unit 45. The registration unit 41, based on a registration request sent from the guardian's mobile terminal 2 before the person enters the facility 4, registers information including facial images and feature information of both the guardian and the person in the registration memory 31. Furthermore, the feature information includes, for example, name, age, gender, and features used to identify the individual's appearance.

[0037] The acquisition unit 42 associates and manages the image data captured by each surveillance camera 5 with the corresponding camera location information. The determination unit 44 refers to the camera location information of the images captured by the person whose likelihood of being lost exceeds a predetermined threshold as determined by the estimation unit 43, and determines the location of the person based on that location information. In this way, by uniformly managing the location information of multiple surveillance cameras 5 configured within the facility and determining the location of the person based on their association with the captured images, it is possible to quickly and accurately locate missing persons.

[0038] When the acquisition unit 42 detects a person's image at the entrance of facility 4, if the image contains facial images of the guardian and the person being monitored, which are already registered in the registration memory 31, then it detects that the guardian and the person being monitored have entered facility 4. Upon detecting that the guardian and the person being monitored have entered facility 4, the acquisition unit 42 acquires images and feature information, including facial images of the registered person being monitored and the guardian, from images captured by each surveillance camera 5 located within facility 4. Then, the acquisition unit 42 registers the facial images and feature information of the person being monitored and the guardian at the time of entry into the entry memory 32. Based on the facial images of the person being monitored and ...

[0039] The estimation unit 43 estimates the likelihood of the person going missing based on the expressions and behaviors of the person and their guardian in the captured images obtained by the acquisition unit 42, as well as the distance between the person and their guardian. For example, the estimation unit 43 estimates the likelihood of the person going missing using generative AI. The estimation unit 43 updates the estimated likelihood of the person going missing to the real-time image memory 33.

[0040] The inference unit 43 takes as input the behavioral characteristics, distance information, and facial expression information of the target person and their guardian obtained by the acquisition unit 42, and calculates the probability of the target person going missing through a neural network learned from training data. Unlike traditional rule-based judgments based on distance thresholds, this structure enables the rapid and highly accurate identification of missing persons through the statistical and learning-based inference processing of AI.

[0041] In addition to classification models, the inference unit 43 can also use generative models. Generative models can also include the following structure: reconstructing normal behavioral patterns when accompanied by a guardian, and detecting abnormal (wandering) states by evaluating the error of this reconstruction.

[0042] Specifically, the inference unit 43 extracts the location coordinates, behavior vectors, facial expression features, and surrounding crowd density information of the target person and their guardian from the continuous image data acquired by the acquisition unit 42. The inference unit 43 includes a classification model that uses these multiple features as input and a trained neural network to make a categorical inference about the likelihood of the target person going missing. In addition to the classification model, the inference unit 43 also utilizes a feature space of normal behavior patterns constructed by a generative model. Specifically, a generative AI model (e.g., an autoencoder or diffusion model) reconstructs the target person's temporal behavior based on normal behavior data while traveling with their guardian within the facility. When the reconstruction error exceeds a threshold, the person is deemed to be in an abnormal missing status.

[0043] Thus, the inference unit 43 has a hybrid structure that combines direct state determination based on a classification model with deviation assessment from normal behavior based on a generative model. This enables high-precision identification of situations where a person is only temporarily separated from their guardian versus abnormal states where they are actually missing.

[0044] The determination unit 44 determines whether the probability of the target person going missing, as estimated by the estimation unit 43, exceeds a predetermined threshold, such as 80%. When the probability of the target person going missing exceeds the predetermined threshold, the determination unit 44 determines the location of the target person based on the shooting position of the surveillance camera 5.

[0045] The notification unit 45 outputs a notification regarding the location of the person determined by the determination unit 44. The notification unit 45 also outputs a missing person report containing the location of the person to the facility-side terminal 6, and simultaneously outputs a missing person notification containing the possibility that the person may have gone missing to the guardian's mobile terminal 2. Furthermore, the generative AI can, for example, determine whether the possibility of the person going missing exceeds a predetermined threshold. If the possibility of the person going missing exceeds the predetermined threshold, it can also execute the function of the determination unit 44 in determining the location of the person based on the shooting location, or, for example, the function of the notification unit 45 in outputting a notification regarding the determined location of the person, and can be appropriately modified.

[0046] Figure 5 This is an explanatory diagram illustrating an example of the table structure of the registration memory 31. The registration unit 41 within the server 3 registers the contents of the registration memory 31 based on the registration request issued by the mobile terminal 2. (Example...) Figure 5As shown, the registration memory 31 stores contact information 31B, guardian's facial image 31C, guardian's feature information 31D, target person's facial image 31E, and target person's feature information 31F according to terminal ID 31A. Terminal ID 31A is the ID used to identify the guardian's mobile terminal 2. Contact information 31B is the guardian's contact information, such as the mobile terminal 2's phone number. Guardian's facial image 31C is a facial image extracted from the provided image, displaying the guardian's features. Guardian's feature information 31D includes, for example, the guardian's name, gender, age, and physical characteristics, used to confirm the guardian's identity. Furthermore, physical characteristics are information that can be obtained from the guardian's facial image. Target person's facial image 31E is a facial image extracted from the provided image, displaying the characteristics of the target person (e.g., a child under the guardian's care). Target person's feature information 31F includes, for example, the target person's name, gender, age, and physical characteristics, used to confirm the target person's identity. Furthermore, physical characteristics are information that can be obtained from the target person's facial image.

[0047] Figure 6 This is an explanatory diagram illustrating an example of the table structure of the entry-time memory 32. The acquisition unit 42 within server 3 registers the contents of the entry-time memory 32 based on the detection of the person and their guardian entering facility 4. For example... Figure 6 As shown, upon entry, the storage device 32 stores the entry time 32B, the guardian's facial image 32C, the guardian's feature information 32D, the subject's facial image 32E, and the subject's feature information 32F, respectively, according to the terminal ID 32A. Terminal ID 32A is the ID used to identify the registered guardian's mobile terminal 2. Entry time 32B is the time when the registered guardian and the subject are detected entering the facility 4, for example, the time when the registered guardian and the subject are identified based on the image captured by the surveillance camera 5 at the entrance of facility 4.

[0048] The guardian's facial image 32C upon entering the store is extracted from the image captured by the surveillance camera 5 at the entrance of facility 4. The guardian's characteristic information 32D upon entering the store is extracted from the image captured by the surveillance camera 5 at the entrance of facility 4. In addition, the characteristic information includes, for example, age and gender, appearance details such as clothing, hairstyle, and hair color. The subject's facial image 32E upon entering the store is extracted from the image captured by the surveillance camera 5 at the entrance of facility 4. The subject's characteristic information 32F upon entering the store is extracted from the image captured by the surveillance camera 5 at the entrance of facility 4. In addition, the characteristic information includes, for example, age and gender, appearance details such as clothing, hairstyle, and hair color.

[0049] Figure 7This is an explanatory diagram showing an example of the table structure of the real-time image storage 33. After detecting that the subject and guardian have entered the facility 4, the acquisition unit 42 within the server 3 registers the images of the subject and guardian within the facility 4 into the real-time image storage 33. (Example...) Figure 7 As shown, the real-time image storage device 33 stores images 33A taken by the guardian, images 33B taken by the guardian, images 33C taken by the subject, images 33D taken by the subject, and the possibility of the subject going missing 33E.

[0050] Guardian-captured image 33A is a current image of the guardian within facility 4 captured by surveillance camera 5. Guardian-captured position 33B is the shooting position of surveillance camera 5, which is currently capturing the current image of the guardian. Subject-captured image 33C is a current image of the subject captured by surveillance camera 5. Subject-captured position 33D is the shooting position of surveillance camera 5, which is currently capturing the current image of the subject. Subject-captured loss probability 33E is expressed as a percentage, based on the probability of the subject's loss predicted by prediction unit 43.

[0051] Figure 8 This is a block diagram illustrating an example of the hardware structure of facility-side terminal 6. For example... Figure 8 As shown, the facility-side terminal 6 includes a communication unit 51, an input unit 52, an output unit 53, a camera unit 54, a ROM 55, a RAM 56, a CPU 57, and a bus 58. The communication unit 51 is, for example, a communication unit that communicates wirelessly with a wireless LAN router and simultaneously with a public wireless communication network. The input unit 52 is an input unit, such as an operating device, for inputting various information. The output unit 53 is an output unit, such as an audio output device or a display device, for outputting various information. The camera unit 54, for example, acquires a facial image representing the characteristics of a missing person to confirm their identity after a staff member carrying the facility-side terminal 6 finds them. The ROM 55 is an area for storing various information, such as programs. The RAM 56 is an area for storing various information. The CPU 57 controls the entire facility-side terminal 6. The bus 58 is a bus line connecting the communication unit 51, the input unit 52, the output unit 53, the camera unit 54, the ROM 55, the RAM 56, and the CPU 57.

[0052] The operation of the missing persons discovery system 1 in this embodiment will be described next. Figure 9 This is a flowchart illustrating an example of the processing actions taken by server 3 in relation to registration processing. Figure 9 In this process, the registration unit 41 within server 3 determines whether a registration request from mobile terminal 2 has been detected (step S11). Furthermore, the registration request may include, for example, a terminal ID for identifying mobile terminal 2, contact information, characteristic information of the recipient and guardian, and a provided image containing facial images of the recipient and guardian.

[0053] When the registration unit 41 detects a registration request from the mobile terminal 2 (step S11: Yes), it obtains the terminal ID, contact information, characteristic information of the target person and guardian, and provides an image from the registration request (step S12). The registration unit 41 extracts facial images of the guardian and the target person from the provided image (step S13). The registration unit 41 analyzes the facial features of the guardian and the target person from the extracted facial images (step S14).

[0054] Registration unit 41 identifies the guardian and the target person based on the facial features obtained from the analysis results. Based on the identification results, registration unit 41 assigns the guardian's feature information to the guardian's facial image and the target person's feature information to the target person's facial image (step S15). Then, for each terminal ID, registration unit 41 registers the contact information, the facial images of the guardian and the target person, and the feature information of the guardian and the target person into the registration memory 31 (step S16), ending the process. Figure 9 The processing actions shown.

[0055] If no registration request is detected from mobile terminal 2 (step S11: No), the registration unit 41 terminates the process. Figure 9 The processing actions shown.

[0056] Figure 10 This is an explanatory diagram showing an example of a screen of mobile terminal 2 related to a registration request. For example... Figure 10 As shown, when mobile terminal 2 detects that it has sent a registration request to server 3, mobile terminal 2 receives a message from server 3: "Thank you for registering. Please send an image of the person under your supervision." Mobile terminal 2 sends a provision image 100 containing facial images of the guardian and the person under its supervision to server 3. Then, server 3 extracts a portion of the facial images and feature information of the guardian and the person under its supervision from the provision image, registers the extracted information in the registration memory 31, and sends a message confirming the image confirmation to mobile terminal 2.

[0057] Figure 11This is an explanatory diagram illustrating an example of the processing actions of server 3 in extracting facial images of a guardian and a subject from a provided image. When the registration unit 41 within server 3 receives a provided image 100 from mobile terminal 2, it extracts facial images 101 and 102 from the provided image 100. The registration unit 41 parses facial feature information from the extracted facial images 101 and 102 to identify the guardian and the subject. Based on the identification results, the registration unit 41 assigns the guardian's feature information to the guardian's facial image 102 and assigns the subject's feature information to the subject's facial image 101. The registration unit 41 registers the guardian's facial image and feature information, and the subject's facial image and feature information, to the registration memory 31. Alternatively, the registration of the guardian's facial image and feature information, and the subject's facial image and feature information, can be performed by separately acquiring the provided image of the guardian and the provided image of the subject.

[0058] Figure 12 and Figure 13 This is a flowchart illustrating an example of the processing actions taken by server 3 in relation to monitoring. Figure 12 In this process, the acquisition unit 42 within server 3 determines whether a person's image has been detected from the surveillance camera 5 at the entrance of facility 4 (step S21). When a person's image is detected (step S21: Yes), the acquisition unit 42 compares the facial image in the detected image with the facial images of the registered person and guardian already stored in the registration memory 31 (step S22).

[0059] Based on the comparison results, the acquisition unit 42 determines whether the facial image in the captured image matches the facial image of the registered subject and guardian (step S23). When the facial image in the captured image matches the facial image of the registered subject and guardian (step S23: Yes), the acquisition unit 42 determines that the registered subject and guardian have been detected entering the facility 4 (step S24).

[0060] After detecting that the registered person and guardian have entered the store, the acquisition unit 42 extracts the facial image and feature information of the person and guardian when they entered the store from the images captured by the surveillance camera 5 at the entrance of facility 4, in addition to extracting the entry time (step S25).

[0061] The acquisition unit 42 registers the entry time, facial images of the person and guardian upon entry, and entry feature information into the entry time memory 32 (step S26). Then, the notification unit 45 in the server 3 sends an entry report, including the registered entry time, facial images of the person and guardian upon entry, and entry feature information, to the facility-side terminal 6 (step S27). The facility-side terminal 6, based on the entry report, will... Figure 15The store entry report screen shown is displayed. Thus, staff at facility-side terminal 6 can confirm the entry of the person and their guardian through the store entry report screen. However, for privacy reasons, the facial images of the person and their guardian are obscured in the store entry report screen displayed on facility-side terminal 6.

[0062] Then, the control unit 40 within server 3 continuously monitors the guardian and the person being monitored through each surveillance camera 5 (step S28), registers the captured images containing facial images of the guardian and the person being monitored, along with the shooting location, to the real-time image storage 33 (step S29), and enters... Figure 13 M1 is shown. Furthermore, the acquisition unit 42 terminates the process if it does not detect a person's image (step S21: No), or if the facial image in the captured image does not match the facial images of the registered subject and guardian (step S23: No). Figure 12 The processing actions shown.

[0063] exist Figure 13 In M1 shown, the estimation unit 43 in server 3 estimates the likelihood of the subject going missing based on the behavior of the subject and guardian extracted from images taken of the subject and guardian, as well as the distance between the subject and guardian (step S31). The estimation unit 43 updates the estimated likelihood of the subject going missing to the real-time image memory 33 (step S32).

[0064] The determination unit 44 within server 3 determines whether the probability of the target person going missing reaches 80% or higher (step S33). When the probability of the target person going missing reaches 80% or higher (step S33: Yes), the determination unit 44 analyzes the situation of the target person who is likely to go missing based on the captured images (step S34). Furthermore, the situation of the target person who is likely to go missing refers to status information derived from a series of captured images of the target person and their guardian, such as, for example, a girl of about 4 years old who may have gone missing near the children's clothing store on the 1st floor.

[0065] The notification unit 45 within server 3 generates a status message for a person who is likely to go missing based on the analysis results of the status of the person (step S35). Furthermore, the status message, such as "A girl of approximately 4 years old may have gone missing near the children's clothing store on the 1st floor," is reported to the facility-side terminal 6. The notification unit 45 sends a missing person report, including the status message, the person's location, and a facial image, to the facility-side terminal 6 (step S36). Upon detecting a missing person report, the facility-side terminal 6 displays... Figure 16The missing person report screen shown is shown below. As a result, staff at facility-side terminal 6 can confirm the location and condition of a person who may be missing through the missing person report screen. However, in the missing person report screen displayed on facility-side terminal 6, the person's face is obscured for privacy protection reasons.

[0066] Notification unit 45 sends a missing person notification containing the situation of the person who may have gone missing to the guardian's mobile terminal 2, which contains their contact information (step S37). When the guardian's mobile terminal 2 detects the missing person notification, it displays an output. Figure 17 The missing person notification screen is shown. As a result, the guardian of mobile terminal 2 can confirm the possibility of the missing person through the missing person notification screen. Control unit 40 determines whether an unblocking operation corresponding to the protection request for the missing person is detected from mobile terminal 2 (step S38). In addition, the protection request for the missing person is an instruction issued to the staff of facility-side terminal 6 requesting protection for the missing person. The staff of facility-side terminal 6 can, according to the protection request for the missing person, make the unblocking button 223 on the missing person report screen become operable, and perform the unblocking operation according to the button operation. In addition, for the sake of explanation, an example of unblocking the missing person report screen according to the button operation of the unblocking button 223 is given. However, server 3 can also notify facility-side terminal 6 of the missing person report screen in the unblocked state according to the protection request from the guardian's mobile terminal 2, which can be changed appropriately.

[0067] When the notification unit 45 detects an unmasking operation (step S38: Yes), it sends a missing person report to the facility-side terminal 6, including a magnified image of the potentially missing person in the unmasked state, as well as characteristic information of the registered person and their guardian (step S39). Upon detecting the missing person report, the facility-side terminal 6 displays... Figure 18 The missing person report screen shown is used to view a magnified image of a person who may have gone missing after the obstruction has been removed, as well as facial images and features of the registered person and their guardian. Through this screen, staff at facility-side terminal 6 can not only successfully protect the person but also smoothly return them to their guardian.

[0068] The control unit 40 determines whether the protection of a person with a possibility of going missing has been completed (step S40). Furthermore, protection completion indicates that the staff has successfully protected the person. When the control unit 40 detects that the protection of a person with a possibility of going missing has been completed (step S40: Yes), it proceeds to the process of step S31, which involves estimating the likelihood of the person going missing.

[0069] If the probability of the person being lost is less than 80% (step S33: No), the control unit 40 determines whether the guardian and the person have left facility 4 (step S41). Furthermore, the departure of the guardian and the person can be determined, for example, by a report from the guardian's mobile terminal 2. When the guardian and the person leave the facility (step S41: Yes), the control unit 40 terminates the process. Figure 13 The processing actions shown.

[0070] Furthermore, if the notification unit 45 does not detect an unblocking operation (step S38: No), it proceeds to step S40 to determine whether protection of a person with a potential for getting lost has been completed. Similarly, if the control unit 40 does not detect an unblocking operation (step S40: No), it proceeds to step S38 to determine whether an unblocking operation has been detected. Additionally, if the guardian and the person in question have not left the store (step S41: No), the control unit 40 proceeds to step S28 to continuously monitor the guardian and the person in question.

[0071] Figure 14 This is an illustrative diagram illustrating an example of the processing actions of server 3 related to facial image recognition when a person enters the store. For example... Figure 14 As shown, the acquisition unit 42 within server 3 compares the images captured by the surveillance camera 5 at the entrance of facility 4 with the facial images and feature information of the registered guardians and individuals in the registration memory 31. When the captured image matches the facial image of the registered guardian or individual, the acquisition unit 42 determines that the registered guardian or individual has entered facility 4. Furthermore, the acquisition unit 42 registers the entry time, the facial image of the individual and guardian at the time of entry, and their feature information in the entry time memory 32.

[0072] Figure 15 This is an explanatory diagram illustrating an example of the store entry report screen displayed on facility-side terminal 6. Facility-side terminal 6 displays the store entry report screen based on the store entry report received from server 3. For example... Figure 15As shown, the entry report screen of facility-side terminal 6 displays: entry confirmation information 210, entry information 220, and captured image 230. Entry confirmation information 210 includes entry time 211, entry location 212, guardian's facial image at entry time 214, guardian's feature information at entry time 216, subject's facial image at entry time 213, and subject's feature information at entry time 215. Entry time 211 is the time when the guardian and subject enter facility 4. Entry location 212 is, for example, the entrance of facility 4. Guardian's facial image at entry time 214 is the guardian's facial image extracted from the captured image at entry time. Guardian's feature information at entry time 216 is the guardian's feature information extracted from the captured image at entry time. Subject's facial image at entry time 213 is the subject's facial image extracted from the captured image at entry time. Subject's feature information at entry time 215 is the subject's feature information extracted from the captured image at entry time. In addition, for privacy protection reasons, the facial images of the guardian and the person being visited are obscured when entering the store.

[0073] The entry information 220 includes: the number of guardians and children entering facility 4 and the number of guardians and children leaving facility 4 221, the age ratio of children (those currently entering facility 4) 222, and a de-obscuring button 223. The de-obscuring button 223 is an operation button used to remove the obscuring effect set on the screen.

[0074] Image 230 is a real-time image of the registered subject and a real-time image of the registered guardian. Furthermore, for privacy protection, the faces of the guardian and the subject are obscured in the captured images.

[0075] Figure 16 This is an explanatory diagram showing an example of a missing person report screen before a protection request is received by facility-side terminal 6. Facility-side terminal 6 displays the missing person report screen based on the missing person report from server 3. (Example...) Figure 16 As shown, the missing person report screen on the facility-side terminal 6 displays missing person information 240 in addition to the entry information 220 and the captured image 230. The entry information 220 includes the number of people 221, age ratio 222, and the unblock button 223, as well as the probability of getting lost 224. The probability of getting lost 224 is the likelihood of the target person getting lost, estimated by the estimation unit 43.

[0076] The missing person information 240 includes a missing person status message 241 and a missing person location information 242. The missing person status message 241 indicates the status of a registered person whose likelihood of going missing exceeds a predetermined threshold. The missing person location information 242 is information showing the location of the registered person whose likelihood of going missing exceeds the predetermined threshold as location 242A on the map of facility 4. Furthermore, the map of facility 4 is generated based on map data from the facility's map storage 34.

[0077] Figure 17 This is an illustrative diagram showing an example of a missing person notification screen on mobile terminal 2. Mobile terminal 2 displays the missing person notification screen based on a missing person notification received from server 3. Figure 17 As shown, the missing person notification screen on the guardian's mobile terminal 2 displays the missing person information 200. The missing person information 200 indicates the missing person's status.

[0078] Figure 18 This is an explanatory diagram illustrating an example of a missing person report screen displayed after a protection request is received by facility-side terminal 6. After detecting the operation of the unblocking button 223 following a protection request, facility-side terminal 6 displays a missing person report screen in an unblocked state. Figure 18 As shown, the missing person report screen of the facility-side terminal 6 displays the store visit confirmation information 210, store entry information 220, and the image of the person in question 230. In addition, it also displays the magnified image of the person in question 243 and the registered information of the person in question and their guardian 250.

[0079] The magnified image 243 of the target person is a magnified image of the target person who received the protection request in the image captured by the surveillance camera 5. The magnified image 243 is an image of the target person in the unobstructed state.

[0080] The registration information 250 includes the facial image 251 and feature information 253 of the person being registered in the registration memory 31, and the facial image 252 and feature information 254 of the guardian registered in the registration memory 31. The facial image 251 of the person being registered and the facial image 252 of the guardian are facial images in an unobstructed state. The feature information 253 of the person being registered and the feature information 254 of the guardian include, for example, name, gender, age, and physical characteristics. As a result, staff can easily identify and protect the lost person by checking the missing person report screen, and can also smoothly return the person to the guardian. Therefore, it can improve the safe and secure shopping environment for facility users, and at the same time improve facility user satisfaction.

[0081] In this embodiment, when server 3 detects that a registered person and their guardian have entered facility 4, it acquires images of the registered person and their guardian from images captured by each surveillance camera 5 located within facility 4. Based on the acquired images of the person and their guardian, server 3 infers the likelihood of the person going missing. If the inferred likelihood exceeds a predetermined threshold, server 3 determines the person's location based on the captured images. As a result, server 3 can quickly identify persons within facility 4 who are at risk of going missing. This structure, unlike traditional rule-based judgments based on distance thresholds, enables the rapid and highly accurate identification of missing persons through the statistical and learning-based inference processing of AI.

[0082] Server 3 acquires an image of the registered subject from the captured image based on the facial image of the registered subject stored in registration memory 31, and simultaneously acquires an image of the registered guardian from the captured image based on the facial image of the registered guardian. As a result, server 3 is able to identify the registered guardians and subjects entering facility 4.

[0083] When server 3 determines the location of the person, it sends a missing person report, including an image of the person with their face obscured and the person's location, to facility-side terminal 6. Upon receiving the missing person report, facility-side terminal 6 displays the image containing the person's face with their face obscured. Figure 16 The missing person report screen shown illustrates this. As a result, staff can identify the location of individuals who may be missing, and their privacy can be protected.

[0084] Once server 3 determines the location of the person in question, it sends a missing person notification, including information about the person, to the mobile terminal 2 of the registered guardian. Upon detecting the missing person notification, mobile terminal 2 displays... Figure 17 The missing person notification screen shown is used to confirm the possibility that a person may be missing.

[0085] When server 3 detects a protection request from guardian's mobile terminal 2, it sends a missing person report, including an image of the person whose face has been unobstructed and the person's location, to facility-side terminal 6. Upon receiving the missing person report, facility-side terminal 6 displays an image containing the unobstructed face of the person. Figure 18 The image shown is a screenshot of a missing person report. As a result, by reviewing the missing person report, staff are able to successfully identify and protect the missing person, and then smoothly return them to their guardian.

[0086] Server 3 associates and manages the image data captured by each surveillance camera 5 with the corresponding camera location information. Server 3 refers to the camera location information of the images captured by the person whose likelihood of being lost exceeds a predetermined threshold, as determined by the estimation unit 43, and determines the person's location based on this location information. As a result, by uniformly managing the location information of multiple surveillance cameras 5 configured within the facility and determining the location of the person based on their association with the captured images, missing persons can be found quickly and with high accuracy.

[0087] Furthermore, in this embodiment, since the registered guardian and the person being abducted are continuously monitored in real time from the time the person enters facility 4 until they leave, the guardian's mobile terminal 2 or facility-side terminal 6 can be notified when contact between the person being abducted and a third party who is not the guardian is detected. This can prevent the person being abducted within facility 4. In addition, the estimation unit 43 not only focuses on the expression and behavior of the person being abducted in the captured images, but also on the behavior of the third party approaching the person being abducted. It can not only estimate the possibility of the person being lost, but also the possibility of the person being abducted, and output a notification to the mobile terminal 2 or facility-side terminal 6 based on the abduction probability.

[0088] Furthermore, server 3 registers the images of the guardian and the subject at each shooting location to the real-time image storage 33, thus enabling the tracking of the movement trajectory of the images of the guardian and the subject in a time sequence. As a result, server 3 can not only identify the movement trajectory of the guardian and the subject from entering facility 4 to leaving the facility, but also notify the subject or guardian of the movement trajectory to facility-side terminal 6 and the guardian's mobile terminal 2.

[0089] Furthermore, for ease of explanation, the example shown illustrates the execution of the target person detection device on server 3, but it can also be executed in the cloud. The likelihood of a registered target person going missing and their location can be determined in the cloud, and the configuration can be modified accordingly. Alternatively, the target person detection device can also utilize an application within mobile terminal 2 to estimate the likelihood of a registered target person going missing and determine their location, without using server 3, and the configuration can be modified accordingly. The target person detection device can also be located in facility 4. For example, server 3 can be located in facility 4. Furthermore, some functions of server 3 can also be located in facility 4.

[0090] Registration of facial images and characteristic information of guardians and individuals can also be carried out after entering facility 4. In this case, the identification of registered guardians and individuals is based on images captured by surveillance camera 5 after registration.

[0091] The example given is a case where mobile terminal 2 connects wirelessly to server 3 via a wireless LAN, but it is not limited to wireless LAN. For example, it can also connect wirelessly via Bluetooth (registered trademark), and can be modified as appropriate.

[0092] The above embodiments illustrate the case of registering a facial image representing the characteristics of a person to the registration memory 31, but are not limited to facial images. Any image that can identify the characteristics of a person from a captured image is acceptable and can be modified appropriately.

[0093] In addition, the example given is that the server 3 is used as a target person detection device, but the surveillance camera 5 can also be used as an AI camera, and the surveillance camera 5 can perform the functions of the acquisition unit 42, the inference unit 43, the determination unit 44 and the notification unit 45, which can be modified appropriately.

[0094] Furthermore, the constituent elements of each part in the diagram need not be physically arranged as shown. That is, the specific form of the distribution and integration of each part is not limited to the diagram. Depending on various loads or usage conditions, the functional or physical distribution and integration can be carried out in any unit to constitute all or part of it.

[0095] Furthermore, all or any part of the various processing functions performed by each device can be executed on a CPU (Central Processing Unit) (or a microcomputer such as an MPU (Micro Processing Unit) or MCU (Micro Controller Unit)). Additionally, all or any part of the various processing functions can also be executed on programs parsed and executed by the CPU (or MPU, MCU, etc.), or on wired logic-based hardware; this point needs no further explanation.

[0096] Furthermore, the various processes described in this embodiment can be implemented by executing a pre-prepared program on the information processing device, and can be modified as appropriate.

[0097] Furthermore, the various processes involved in this embodiment can also be implemented by executing a program through an information processing device. For example, the functions of the above-mentioned prediction unit 43, determination unit 44, or notification unit 45 can be implemented by a processor reading and executing a program stored in the memory.

[0098] Furthermore, this embodiment can also be implemented as a method for finding missing persons to perform the above-described process. The method for finding missing persons may include: (1) acquiring images of the person and their guardian; (2) inferring the likelihood of the person being missing based on the acquired images; (3) determining the location of the person based on the inference result; and (4) notifying the person of the determination result.

[0099] Furthermore, this embodiment can also be implemented as an electronic device (or computer device) for performing the above-described method for finding missing persons. The electronic device includes at least a processor, a memory, and a communication unit, through which the processor executes the program stored in the memory to implement the above steps.

[0100] Furthermore, this embodiment can also be implemented as a storage medium storing a program for causing a computer to execute the aforementioned method for finding missing persons. The storage medium may include, for example, a hard disk, a semiconductor memory, an optical disk, etc., and the computer achieves the aforementioned functions by reading and executing the program.

[0101] Furthermore, this embodiment can also be implemented as a virtual device for unified virtual management of multiple cameras and servers configured in a real space, or the same functional structure can be implemented in a virtual environment executable on a server or in the cloud.

Claims

1. A device for finding missing persons, characterized in that, have: The acquisition unit, when it detects that a registered person and a registered guardian of the person have entered a designated area, acquires an image of the registered person and an image of the registered guardian from images captured by cameras located within the designated area; The estimation unit estimates the likelihood of the person being lost based on the images of the person being identified and the guardian obtained by the acquisition unit. as well as The determining unit determines the location of the target person based on the shooting position of the camera when the probability of the target person being lost, as estimated by the inference unit, exceeds a predetermined threshold.

2. The missing person detection device according to claim 1, characterized in that, have: The registration department is used to register the facial images of the person being identified and the guardian, as well as identification information for identifying the person being identified and the guardian, before they enter the designated area. The acquisition unit acquires an image of the registered person from the captured image based on the facial image of the registered person in the registration unit, and simultaneously acquires an image of the registered guardian from the captured image based on the facial image of the registered guardian.

3. The missing person detection device according to claim 2, characterized in that, It also has: The notification unit, when the determination unit determines the location of the target person, notifies the facility-side terminal of the location of the target person and includes an image of the target person's face being obscured.

4. The missing person detection device according to claim 3, characterized in that, When the determining unit determines the location of the person in question, the notification unit notifies the registered guardian of the person's status.

5. The missing person detection device according to claim 4, characterized in that, When the notification unit detects a protection request from the guardian's terminal, it notifies the facility-side terminal of the location of the person in question and includes an image of the person in question with their face unobstructed.

6. The missing person detection device according to claim 4, characterized in that, When the notification unit detects a protection request from the guardian's terminal, it notifies the facility-side terminal of the location of the person in question and includes an image of the person in question with their face unobstructed, as well as the identification information of the registered guardian and the registered person in question.

7. A missing persons discovery system comprising: a guardian's terminal storing an image of the person in question; a missing persons discovery device that, upon detecting a request from the guardian's terminal to register the image of the person in question, registers the image of the person in question; and a predetermined terminal within a predetermined range, characterized in that... The missing persons detection device has the following features: The acquisition unit, when it detects that a registered person and a registered guardian of the person have entered a designated area, acquires an image of the registered person and an image of the registered guardian from images captured by cameras located within the designated area; The estimation unit estimates the likelihood of the person being lost based on the images of the person being identified and the guardian obtained by the acquisition unit. The determining unit determines the location of the target person based on the shooting position of the camera when the probability of the target person being lost estimated by the inference unit exceeds a predetermined threshold. as well as The notification unit outputs the location of the target person determined by the determination unit to the designated terminal. The specified terminal has an output unit for outputting the location of the target person notified by the notification unit.

Images