Information processing system, information processing method, and program

Abstract

With the expansion of installation places of cameras, there is an increasing need to search for a specific person from a video captured by a camera.SOLUTION: According to an aspect of the present disclosure, there is provided an information processing system that causes an information terminal to display a video captured by a camera. The information processing system searches for a video including a face image from videos, displays a plurality of videos as a result of the search, selects a video from the plurality of videos according to an instruction of a user, displays the face image included in the video together with the selected video, and registers the face image according to an instruction of the user.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent document 1 discloses a program that causes one or more computers to execute the following processes: a process of displaying multiple camera icons on a display device; a process of accepting input to surround at least two or more camera icons from among the multiple camera icons displayed on the display device; and a process of searching for a person based on images captured by cameras corresponding to each of the at least two or more camera icons in response to accepting input to surround at least two or more camera icons. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2023-129429 Summary of the Invention [Problem to be solved by the invention]

[0004] As the number of locations where cameras are installed increases, there is a growing need to search for specific individuals using footage captured by the cameras. [Means for solving the problem]

[0005] An information processing system is provided that displays video captured by a camera on an information terminal. The information processing system searches for video containing facial images from the video, displays multiple videos as search results, selects a video from the multiple videos in response to a user's instruction, displays the facial image contained in the video together with the selected video, and registers the facial image in response to the user's instruction. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the server device. [Figure 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the client device. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of a surveillance camera. [Figure 5] FIG. 5 is a flowchart showing an example of information processing related to registration of a face image in the information processing system. [Figure 6] FIG. 6 is a diagram showing an example of the list display screen. [Figure 7] FIG. 7 is a diagram showing an example of a calendar screen superimposed on the display. [Figure 8] FIG. 8 is a diagram (part 1) showing an example of the video detail screen. [Figure 9] FIG. 9 is a diagram (part 2) showing an example of the video detail screen. [Figure 10] FIG. 10 is a diagram showing an example of the person registration screen. [Figure 11] FIG. 11 is a diagram showing an example of the editing screen. [Figure 12] FIG. 12 is a diagram showing an example of a movie clip creation screen. [Figure 13] FIG. 13 is a flowchart showing an example of information processing related to a face image search in the information processing system. [Figure 14] FIG. 14 is a diagram showing an example of the person search screen. [Figure 15] FIG. 15 is a diagram showing an example of the list display screen. [Figure 16] FIG. 16 is a diagram showing an example of the video detail screen. [Figure 17] FIG. 17 is a diagram showing an example of the notification setting screen. [Figure 18] FIG. 18 is a diagram showing an example of the notification setting screen. DETAILED DESCRIPTION OF THE INVENTION

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0008] <Embodiment 1> 1. System configuration diagram Fig. 1 is a diagram showing an example of the system configuration of an information processing system 1000. As shown in Fig. 1, the information processing system 1000 includes, as a system configuration, a server device 100, a client device 110, a client device 120, and a plurality of surveillance cameras 160. The server device 100, the client device 110, the client device 120, and the surveillance cameras 160 are communicatively connected via a network 150. The network 150 may include either or both of a wide area network (WAN) and the Internet. The network 150 is configured to enable communication between a plurality of devices connected to the network 150 via wired and wireless connections.

[0009] The information processing system 1000 is a system that provides a video search service.

[0010] The server device 100 is a device that provides the main functions of the information processing system 1000 and executes the main processes of the embodiment described below. The server device 100 is a device that outputs (displays) images captured by multiple surveillance cameras 160 on the client device 110 or the client device 120.

[0011] The surveillance cameras 160 are cameras installed for the purpose of monitoring and / or recording. In the specification, the surveillance cameras 160 are assumed to be installed in multiple locations in a specific store. Examples of multiple locations include multiple entrances and exits to the store (e.g., an east entrance, a west entrance, a dedicated exit, etc.). Note that these are examples and do not limit the locations where the surveillance cameras 160 are installed. For simplicity, only three surveillance cameras 160 are shown in FIG. 1, but there may be two, four, or more. It is sufficient that multiple surveillance cameras 160 are included in the information processing system 1000. In the specification, the surveillance cameras 160 are assumed to be installed in one store, but multiple cameras may be installed in each of multiple stores.

[0012] The client device 110 is a terminal device operated by the owner or manager (hereinafter simply referred to as the owner) of the surveillance camera 160. The client device 110 displays screens such as those shown in Figs. 17 and 18, which will be described later.

[0013] Client device 120 is a terminal device operated by a manager or the like of a store where surveillance camera 160 is installed. Client device 120 displays screens such as those shown in Figs. 6 to 12 and 14 to 16, which will be described later.

[0014] Here, the claimed information processing system may be composed of multiple devices or may be composed of a single device. When the claimed information processing system is composed of a single device, an example of the device is server device 100. When the claimed information processing system is composed of a plurality of devices, an example of the plurality of devices is server device 100 and at least one of multiple surveillance cameras 160 and client device 110 or 120, or a cloud server composed of multiple server devices that provide the functions of server device 100.

[0015] 2. Hardware Configuration (1) Hardware Configuration of Server Device 100 FIG. 2 is a diagram illustrating an example of a hardware configuration of the server device 100. As shown in FIG. 2, the server device 100 includes, as its hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0016] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire server device 100 .

[0017] The storage unit 220 is any one of an HDD (Hard Disk Drive), a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), etc., or any combination thereof, and stores programs, data used when the control unit 210 executes processing based on the programs, etc. The storage unit 220 is an example of a storage medium. Examples of data used when the control unit 210 executes processing based on the programs include video data sent from the surveillance camera 160, facial images of the detection target, data related to the detection target, and data related to the destination set in Figures 17 and 18, etc., which will be described later.

[0018] In the present specification, the data used by control unit 210 when executing processing based on a program is described as being stored in storage unit 220, but it may also be stored in a storage unit of another device that can communicate with server device 100. The data may be stored in a storage unit of any device as long as it can be referenced or acquired by control unit 210. When control unit 210 executes processing based on a program stored in storage unit 220, the functions of server device 100 and the processing of the flowcharts shown in Figures 5 and 13 (described below) are realized. Note that while this processing is described as being mainly executed by server device 100, it may instead be executed by client device 110, client device 120, or any of multiple surveillance cameras 160.

[0019] The communication unit 230 connects the server device 100 to the network 150 and controls communication with other devices.

[0020] The hardware configuration of each of the control unit 210, the storage unit 220, and the communication unit 230 is not limited to one. For example, multiple control units may be included in the server device 100. The same applies to the client device 110 and the client device 120 described below.

[0021] (2) Hardware configuration of the client device 110 / 120 FIG. 3 is a diagram showing an example of the hardware configuration of the client device 110 / 120. 3, client device 110 includes, as its hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, a communication unit 350, and an internal bus 360. Control unit 310, storage unit 320, input unit 330, output unit 340, and communication unit 350 are electrically connected via internal bus 360.

[0022] The control unit 310 is a CPU or the like, and controls the entire client device 110 .

[0023] The storage unit 320 is any one of an HDD, a ROM, a RAM, an SSD, etc., or any combination thereof, and stores programs, data used when the control unit 310 executes processing based on the programs, etc. The storage unit 320 is an example of a storage medium.

[0024] In the specification, the data used by control unit 310 when executing processing based on a program is described as being stored in storage unit 320, but the data may also be stored in a storage unit of another device that can communicate with client device 110. The data may be stored in a storage unit of any device as long as it can be referenced or acquired by control unit 310. The functions and the like of client device 110 are realized by control unit 310 executing processing based on the program stored in storage unit 320.

[0025] The input unit 330 is a device that inputs information to the client device 110 in response to an operator's operation. The input unit 330 accepts an operation input made by a user. The operation input is transferred as a command signal to the control unit 310 via the internal bus 360. The control unit 310 may execute predetermined control and / or calculation based on the transferred command signal as necessary. The input unit 330 may be included in the housing of the client device 110 or may be externally attached. For example, the input unit 330 may be implemented as a touch panel integrated with the output unit 340. When the input unit 330 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 330. Instead of a touch panel, a switch button, a mouse, a trackpad, a keyboard, etc. may be used as the input unit 330.

[0026] The output unit 340 is, for example, a display unit typified by a display, and is a device that outputs (displays) information as a screen of a graphical user interface (GUI) that can be operated by a user. The output unit 340 may be included in the housing of the client device 110 or may be externally attached. More specifically, the output unit 340 may be implemented as a display device such as a liquid crystal display, an organic EL (Electron-Luminescence) display, or a plasma display. It is preferable that these display devices are implemented by selectively using them depending on the type of client device 110.

[0027] The communication unit 350 connects the client device 110 to the network 150 and controls communication with other devices.

[0028] The hardware configuration of the client device 120 is similar to that of the client device 110 .

[0029] In the present specification, a PC (Personal Computer) will be used as an example of client device 110 and client device 120. However, client device 110 and client device 120 may be a smartphone, a tablet computer, or the like. The client device may be any device that can display a screen as described below, accept user operations via the screen, or the like, and transmit information to server device 100.

[0030] (3) Hardware configuration of the surveillance camera 160 FIG. 4 is a diagram showing an example of the hardware configuration of the surveillance camera 160. As shown in FIG. 4, the hardware configuration of surveillance camera 160 includes a control unit 410, a storage unit 420, an imaging unit 430, a communication unit 440, and an internal bus 450. Control unit 410, storage unit 420, imaging unit 430, and communication unit 440 are electrically connected via internal bus 450.

[0031] The control unit 410 is a CPU or the like, and controls the entire surveillance camera 160 .

[0032] The storage unit 420 is any one of an HDD, a ROM, a RAM, an SSD, etc., or any combination thereof, and stores programs, data used when the control unit 410 executes processing based on the programs, etc. The storage unit 420 is an example of a storage medium.

[0033] In the specification, the data used by control unit 410 when executing processing based on a program is described as being stored in storage unit 420, but the data may also be stored in a storage unit of another device that can communicate with surveillance camera 160. The data may be stored in a storage unit of any device as long as it can be referenced or acquired by control unit 310. The functions of surveillance camera 160 are realized by control unit 410 executing processing based on the program stored in storage unit 420.

[0034] The photographing unit 430 is a camera that photographs a subject, and includes, for example, an image sensor, a lens, and an IR cut filter.

[0035] The communication unit 440 connects the surveillance camera 160 to the network 150 and controls communication with other devices.

[0036] 3. Information Processing The information processing of the first embodiment will be described below.

[0037] (1) Overview of the process (1-1) Facial image registration The control unit 210 searches for videos containing facial images from videos captured by the surveillance camera 160 included in the information processing system 1000, and outputs (displays) multiple videos containing facial images as search results. The control unit 210 selects a video from the multiple videos included in the search results in response to a user's selection operation, and outputs (displays) the facial image of a person included in the video together with the selected video. The control unit 210 registers the facial image in response to a user's instruction.

[0038] By performing such processing, it is possible to easily find a person from the video and register the facial image of the person of interest in the information processing system 1000.

[0039] (1-2) Face image search The control unit 210 outputs (displays) the multiple registered face images. The control unit 210 selects a face image from the multiple face images in response to a selection operation by the user, searches for a video including the selected face image, and outputs (displays) the multiple videos including the selected face image as a search result. The control unit 210 selects a video from the multiple videos included in the search result in response to a selection operation by the user, and outputs (displays) the selected video.

[0040] By performing such processing, it is possible to detect and output an image of a person whose face image is selected from the registered face images.

[0041] (2) Details of the processing (2-0) Pretreatment First, as settings, the angle to be determined as the frontal angle of the face, the angle difference to be determined as being sufficiently different from the frontal angle, the number of reliable image pixels, the designated time for capturing the best shot, etc. These settings are set, for example, in response to a user's instruction via the client device 120, and are stored in each surveillance camera 160 (or the server device 100).

[0042] In this system, each surveillance camera 160 basically constantly captures video and transmits it to the server device 100 for storage. At the same time, a best shot extraction process is also performed. Here, the best shot is the optimal facial image (or full-body image) of the same person captured from different predetermined angles, and corresponds to the facial images (740, 750, 760, 770, 910, 920, 930, 940) in FIG. 8 and / or FIG. 10 (described later). This process is primarily performed on the surveillance camera 160 side, but may also be performed on the server device 100 side. The advantage of performing the process on the camera side is that it reduces the processing load and usage fees on the server side, but on the other hand, the camera side requires specifications to perform the corresponding process.

[0043] The details of the best shot extraction process are as follows. First, the surveillance camera 160 tracks a person's full-body image from the camera video. Next, a face is detected from the full-body image data and tracked. The angle of the tracked face is then detected, and an image that falls within the range of the face's frontal angle is designated as the first best shot, and the angle of the face is recorded. The first best shot may be the one with the highest score, for example, using an AI model that detects a frontal image of a face. Furthermore, the angle of the tracked face is continuously detected, and if a face whose angle is equal to or greater than the angle of the face in the first best shot is found, it is saved as the second best shot. For example, the second best shot may be the one with the highest score, for example, using an AI model that detects an image of a face facing left (or right) relative to the camera. Furthermore, an image with an angle different from both the first and second best shots is saved as the third best shot. For the third best shot, the one with the highest score may be used, for example, using an AI model that detects an image of a face facing right (or left) relative to the camera. Furthermore, for a person corresponding to at least one of the first, second, and third best shots, a full-body image (or a bust-up image, etc.) of that person may be saved as the fourth best shot. A bust-up image is an image showing the upper half of a person's body. The fourth best shot may be the one with the highest score, for example, using an AI model to detect full-body images (or bust-up images, etc.). If an image that is within the frontal angle range and exceeds the reliable image pixel count is found by continuing to detect the tracked face, the first best shot may be replaced. Similarly, the second, third, and fourth best shots may also be replaced. Furthermore, best shot extraction may be stopped if the specified time for best shot acquisition has elapsed. Furthermore, while the first, second, third, and fourth best shots are selected based on the highest score from each perspective, multiple shots, including those with the next highest scores, may also be selected.Then, once tracking is complete, the first, second, third, and fourth best shots are sent to the server device 100 along with metadata indicating the camera and date and time each image was taken. The surveillance camera 160 receives the first, second, third, and fourth best shots and the metadata indicating the camera and date and time each image was taken, and stores the images in association with each other / each image in association with the metadata. At this stage, facial images may be averaged based on the first, second, and third best shots, and facial features may be calculated and stored in association with the images and metadata. Hereinafter, this set of associated information (first, second, third, and fourth best shots / metadata indicating the camera and date and time each image was taken / facial features) is referred to as best shot information. The best shot information is stored as multiple, mutually different pieces of best shot information for each person / scene.

[0044] (2-1) Facial image registration 5 is a flowchart showing an example of information processing related to registration of a face image in the information processing system 1000. Note that, although this processing will be described as being mainly executed by the server device 100, it may instead be executed by either the client device 110 or the client device 120, or one of the multiple surveillance cameras 160.

[0045] In step S510, the control unit 210 searches for footage containing a face image from footage captured by the surveillance camera 160 included in the information processing system 1000. Specifically, first, the control unit 210 reads out a plurality of best shot information stored in the pre-processing described above. Then, for each best shot information, the control unit 210 identifies the footage based on metadata indicating the camera and date and time that captured the first (or fourth) best shot.

[0046] Note that step S510 may also be implemented as follows. Videos captured by multiple surveillance cameras 160 included in information processing system 1000 are transmitted to server device 100 via network 150 and stored in a predetermined storage area such as storage unit 220. The videos are stored in a predetermined storage area such as storage unit 220 in a manner that makes it possible to identify which surveillance camera 160 captured the video. A different folder (directory) may be created in storage unit 220 for each surveillance camera and the video may be stored in the folder, or identification information identifying the surveillance camera may be written in the file name of the video, or identification information identifying the surveillance camera may be written in the metadata of the video file. The videos are also stored in a predetermined storage area such as storage unit 220 in a manner that makes it possible to identify when the video was captured. As a mode for identifying when a video was shot, a different folder (directory) may be created for each date in the storage unit 220 or the like and the video may be stored in the folder, or the date and time information when the video was shot may be written in the file name of the video, or the date and time information when the video was shot may be written in the metadata of the video file. The control unit 210 searches for videos that include face images from the videos stored in a predetermined storage area such as the storage unit 220, and acquires the search results.

[0047] For example, the control unit 210 inputs a video into a trained model. This trained model is trained using the video as input data, whether or not the video contains a facial image, and, if the video contains a facial image, time information from the appearance to the disappearance of a person in the facial image as output data. Furthermore, if the same video contains multiple facial images, the trained model is trained using time information from the appearance to the disappearance of each person in the multiple facial images as output data. The control unit 210 obtains, as search results, information from the trained model as to whether or not the video contains a facial image, and, if the video contains a facial image, time information from when the person in the facial image appears to when the person in the facial image disappears.

[0048] In step S520, the control unit 210 generates a list display screen including reduced images (hereinafter simply referred to as thumbnails) of the timing at which the facial image appears in the video containing the facial image, based on the search result. The control unit 210 transmits the generated list display screen to the client device 120 and controls the output unit of the client device 120 to display it.

[0049] FIG. 6 is a diagram showing an example of a list display screen 600. The list display screen 600 shown in FIG. 6 includes a person video tag 610 and a person video area 670. The person video tag 610 is included in a side menu of the screen. Note that, for a screen including a side menu, the entire screen including the side menu may be referred to as the screen, or the portion excluding the side menu may be referred to as the screen. The same applies to the following screens. When the person video tag 610 is selected, the processes of steps S510 and S520 described above are executed, and thumbnails of the search results are displayed in the person video area 670. The example of FIG. 6 shows that the person video tag 610 is selected on the list display screen 600.

[0050] The device section 620 of the person video area 670 displays a list of thumbnails of people detected in the video captured by the surveillance camera 160 at the east entrance. Each thumbnail may display the person's tracking start time (not shown). Although three thumbnails are displayed in the list of thumbnails of people detected in the video captured by the surveillance camera 160 in FIG. 6, the number is not limited to this. The number of thumbnails displayed on the screen may be changed depending on the screen size. Furthermore, when the "View more button" is selected, the control unit 210 controls to display more thumbnails of people detected in the video captured by the surveillance camera 160.

[0051] The device section 630 of the person image area 670 displays a list of thumbnails of people detected in the image captured by the surveillance camera 160 at the west entrance of the store. The device section 640 of the person image area 670 displays a list of thumbnails of people detected in the image captured by the surveillance camera 160 at the dedicated exit entrance. The device sections included in the screen are arranged in order from the top (top of the screen) of the list display screen 600 in order of the devices (surveillance cameras 160) with thumbnails and the tracking start times displayed in the thumbnails. The same applies to the list display screen 1300 in Fig. 15, which will be described later.

[0052] On the list display screen 600, the multiple videos output as search results are output (displayed) in the form of reduced display images for each of the imaging devices that captured the multiple videos. The process of step S520 is an example of a process of outputting (displaying) reduced images of each of a plurality of videos containing a face image as a search result.

[0053] The list display screen 600 includes a keyword input area 650 and a period input area 660. The keyword input area 650 is an area where a keyword can be input. When a keyword is input in the keyword input area 650, the control unit 210 searches for the device name of the surveillance camera 160, the serial number of the surveillance camera 160 that can identify the surveillance camera 160, the device tag set for the surveillance camera 160, and the like, based on the keyword. The control unit 210 then searches for a facial image from the video of the corresponding surveillance camera 160, and displays the search results in the person video area 670. This processing is an example of processing for searching for a video that includes a facial image based on search conditions input via the screen.

[0054] The period input area 660 is an area where a period can be input. When the period input area 660 is selected, the control unit 210 causes the list display screen 600 to be displayed with a calendar screen superimposed thereon. FIG. 7 is a diagram showing an example of a calendar screen 1510 being superimposed and displayed. The calendar screen 1510 is a screen that displays dates and days of the week in a table format and is configured to allow the start and end times of the video to be searched for to be selected. When a period (start and end times) is input on the calendar screen 1510, the control unit 210 searches for videos shot during the input period and displays the search results in the person video area 670. This process is an example of a process for searching for videos containing facial images based on search criteria input via the screen. In the example of FIG. 7 , the period from March 29, 2024 to April 16, 2024 is selected as the period. Note that the period input area 660 may be configured to allow the period to be specified as after March 29, 2024, etc. In other words, the period may be configured to allow the period to be specified as after a certain point in time. Similarly, the period input area 660 may be configured to allow the period to be specified as before April 16, 2024, etc. In other words, the period may be configured to allow the period to be specified as up to a certain point in time.

[0055] As described above, the list display screen 600 is configured so that at least one of a period and a keyword can be input as a search condition. The list display screen 600 may also be configured so that the imaging device that captured the video can be input as a search condition.

[0056] When one of the thumbnails is selected in the person image area 670 in FIG. 6, thumbnail selection information including identification information for identifying the selected thumbnail is transmitted from the client device 120 to the server device 100.

[0057] 5, the control unit 210 determines whether or not thumbnail selection information has been received. If thumbnail selection information has been received, the control unit 210 proceeds to step S540; if thumbnail selection information has not been received, the control unit 210 returns to step S520.

[0058] In step S540, the control unit 210 selects the image of the thumbnail identified by the thumbnail selection information. In step S550, control unit 210 generates a video detail screen including a timeline of the selected video and facial images of people included in the video. More specifically, control unit 210 identifies the corresponding first, second, third, and fourth best shots for the video selected in S540 based on the best shot information, and generates a video detail screen for displaying them together with the video timeline. Control unit 210 transmits the generated video detail screen to client device 120 and controls the output unit of client device 120 to display it.

[0059] FIG. 8 is a diagram (part 1) showing an example of a video detail screen 700. The video detail screen 700 includes a display area 790. The display area 790 displays video captured by the corresponding surveillance camera 160. The video includes a person 735. The person 735 is a person included in the video selected by the thumbnail. The video also includes a background 736, which is omitted in FIG. 8 for simplification. The background 736 corresponds to a streetscape, an interior of a room, a natural landscape, or the like captured by the corresponding surveillance camera 160. By including the person 735 and the background 736 in the display area 790, the operator can understand what the person is doing and in what situation.

[0060] The video detail screen 700 includes a timeline 710. The timeline 710 indicates the time of the video displayed in the display area 790. More specifically, the timeline displays a time axis in the shape of a strip from the start point of the video to the end point of the video. In this embodiment, the timeline displays the elapsed time from the start point as a scale. However, the timeline may also display the time or date as a scale. The playback position display object 720 is an object (GUI component) that indicates the shooting date and time of the video displayed in the display area 790. When the playback position display object 720 is at the leftmost position, it indicates the start position of the video playback, and when it is at the rightmost position, it indicates the end position of the video playback. The playback position display object 720 indicates the position on the timeline 710 of the video being played in the display area 790. The operator of the client device 120 (such as the manager of a store where the surveillance camera 160 is installed) can also specify the shooting date and time of the video displayed in the display area 790 by operating the playback position display object 720. Flag 730 is a flag that indicates the period during which a person appears in the video, and is displayed on timeline 710. Flag 730 indicates that a person appeared at the time indicated at the leftmost position, and that the person disappeared at the time indicated at the rightmost position. The control unit 210 outputs a timeline relating to the selected video. The timeline is output in a manner that allows for differentiation between portions in which the person in the registered face image is shown and portions in which the person in the face image is not shown.

[0061] Facial image display area 795 displays facial images of people appearing in the video displayed in display area 790. In the example of Fig. 8, facial image display area 795 includes facial image 740, facial image 750, facial image 760, and facial image 770. Facial image 740 is a facial image facing forward relative to the surveillance camera 160. This is the first best shot corresponding to the displayed video. A facial image that shows both eyes may be said to be a facial image facing forward relative to the surveillance camera 160. A facial image whose face is facing the surveillance camera 160 may also be said to be a facial image facing forward. A facial image whose viewpoint is in the direction of the surveillance camera 160 may also be said to be a facial image facing forward. Facial image 750 is a facial image facing left relative to the surveillance camera 160. This is the second best shot corresponding to the displayed video. A facial image in which only the right eye is visible, rather than both eyes, may be said to be a facial image facing left relative to the surveillance camera 160. An image in which the right side of the face is captured but the left side of the face is not may also be said to be a facial image facing left relative to the surveillance camera 160. Facial image 760 is a facial image facing right with respect to the surveillance camera 160. This is the third best shot corresponding to the displayed video. A facial image in which only the left eye is visible, rather than both eyes, may be said to be a facial image facing right with respect to the surveillance camera 160. An image in which the left side of the face is captured but not the right side of the face may also be said to be a facial image facing right with respect to the surveillance camera 160. In addition to facing forward, facing left, and facing right, other facial orientations such as facing diagonally left, facing diagonally right, and facing backward may be used additionally or alternatively. Facial image 770 is a facial image that includes the entire body of a person. This is the fourth best shot corresponding to the displayed video. The entire body includes not only the facial image but also the person's torso, hands, feet, etc. In addition to the whole body, for example, a bust-up shot may be used in addition / alternatively.

[0062] In the face image display area 795, the multiple face images include at least one of a face image facing forward, a face image facing right, a face image facing left, and a face image including the whole body. It can also be said that the facial image display area 795 contains a plurality of facial images of the same person taken from different angles.

[0063] The control unit 210 outputs (displays) an image in which the face of a person is recognizable from among images of people included in the selected video, together with the video.

[0064] The control unit 210 may display a face image in the face image display area 795 at the time corresponding to the leftmost position of the flag 730, and may hide (or delete) the face image in the face image display area 795 at the time corresponding to the rightmost position of the flag 730. The control unit 210 may display the face image of a person in the face image display area 795 while the person appears in the video displayed in the display area 790. By performing such processing, the control unit 210 can display the face image of the person in a manner that makes it clear that the person is included in the selected video. A manner in which it is possible to identify the person as being included in the selected video is, for example, a manner in which the display area 790 (or person 735) and the facial image display area 795 are displayed on the same screen, or the facial image display area 795 is displayed superimposed on the display area 790.

[0065] The process of step S550 is an example of a process of outputting (displaying) a screen including the video selected in person video area 670 and the timeline of the video. The process of step S550 is an example of a process of outputting (displaying) a plurality of face images of the same person as the person included in the video together with the video of the thumbnail selected in person video area 670.

[0066] The facial image display area 795 further includes a "Register Person" button 775. If the person appearing in the video displayed in the display area 790 is a person who has not yet been registered in the information processing system 1000, the control unit 210 outputs the "Register Person" button 775 in the facial image display area 795. This processing can also be said to be the control unit 210 activating the "Register Person" button 775 and displaying it in the facial image display area 795 if the person appearing in the video displayed in the display area 790 is a person who has not yet been registered in the information processing system 1000. Activating a button means making the button selectable. If the face image of the person included in the video has not been registered, the control unit 210 outputs (displays) a registration button for registering the face image in a manner that allows the user to select it.

[0067] If the person appearing in the image displayed in the display area 790 is a person who has already been registered in the information processing system 1000, the control unit 210 outputs the facial image display area 795 including a "Registered" button 810. 9 is a diagram (part 2) showing an example of the video detail screen 700. The "registered" button 810 is a button indicating that the person whose face image is included in the face image display area 795 has already been registered as a person to be searched in the information processing system 1000. Note that the button is not limited to a button and may be any GUI component, image, or the like, as long as it can indicate that the person whose face image is included in the face image display area 795 has already been registered in the information processing system 1000. The facial images of people included in the video are output (displayed) in a manner that makes it possible to identify whether or not they have already been registered.

[0068] 8 is selected, the control unit 210 generates a person registration screen. The control unit 210 transmits the generated person registration screen to the client device 120 and controls the output unit of the client device 120 to display it.

[0069] 10 is a diagram showing an example of a person registration screen 900. As shown in FIG. 10, the person registration screen 900 includes face images 910, 920, 930, and 940 that are the same as the face image displayed in the face image display area 795. The control unit 210 can swap the positions of the face images (face image 910, face image 920, face image 930, face image 940, etc.) based on a drag operation by the operator of the client device 120. The image located at the position of face image 910 is used as a cover photo. The cover photo is used on screens, etc., as the face image of the saved person shown, for example, in FIGS. 14 and 15, which will be described later. Furthermore, the control unit 210 can delete a face image in response to the operator of the client device 120 clicking the "x" button in the upper right corner of each face image (face image 910, face image 920, face image 930, face image 940, etc.). This is to prevent a decrease in the accuracy of the search in step S1140, which will be described later, when a face image other than that of a desired person is mistakenly included among the face images. Note that, in order to improve the accuracy of the search, rather than simply deleting the face image, it is also possible to automatically set a face image with the same orientation as the next candidate, or to display a list of several face images with the same orientation as the next candidate and set the one selected by the user.

[0070] The person registration screen 900 also includes a GUI component 950 that indicates a plus sign. When the GUI component 950 is selected, the control unit 210 generates a screen for adding a face image. The control unit 210 transmits the generated screen to the client device 120 and controls the client device 120 to display it. The control unit 210 displays, on the screen for adding a face image, a selectable face image of the same person as the face image displayed in face image 910, face image 920, face image 930, and face image 940. The operator of the client device 120 can select a different face image of the same person via this screen and add the face image. Note that as another method for adding a face image, for example, a face image separately prepared in the client device 120 may be uploaded.

[0071] Furthermore, when one of facial images 910, 920, 930, and 940 is selected and a predetermined operation is performed, control unit 210 generates a screen for changing the facial image. Control unit 210 transmits the generated screen to client device 120 and controls it to be displayed on client device 120. Control unit 210 displays, on the screen for changing the facial image, another facial image of the same person as the selected facial image so that the selected facial image can be selected. The operator of client device 120 can change the facial image to another facial image of the same person via this screen. These processes are examples of processes for changing a registered face image or adding / deleting a face image based on a predetermined operation by the user.

[0072] 10 , the operator of the client device 120 inputs, for example, a person's name, a person tag, a memo, etc., and performs an operation to register a face image of the person as a person to be searched for. When the "Save as this content" button is selected on the person registration screen 900, the registration information, including the face image, etc., included in the person registration screen 900 is transmitted from the client device 120 to the server device 100.

[0073] 5, control unit 210 determines whether or not registration information has been received. If registration information has been received, control unit 210 proceeds to step S570, and if thumbnail selection information has not been received, control unit 210 repeats the processing of step S550.

[0074] In step S570, the control unit 210 registers the face image included in the registration information as a person to be searched (also referred to as a saved person). Specifically, among the multiple best shot information, the one corresponding to the registration information received in S560 is made identifiable by setting a flag or the like.

[0075] When the operator of the client device 120 performs a predetermined operation on the person registration screen 900 (for example, right-clicking on a face image and selecting an edit button from the displayed menu), the control unit 210 generates an edit screen on which the face image can be edited. The control unit 210 transmits the generated edit screen to the client device 120 and controls the output unit of the client device 120 to display it.

[0076] 11 is a diagram showing an example of an editing screen 1010. The editing screen 1010 includes the face image selected on the person registration screen 900. The editing screen 1010 is configured to enable trimming of the face image, rotation and tilt correction, and adjustment of brightness and contrast.

[0077] 8 includes a "Create Movie Clip" button 780. When the "Create Movie Clip" button 780 is selected, the control unit 210 creates a movie clip creation screen 1700. The control unit 210 then transmits the movie clip creation screen 1700 to the client device 120 and controls the output unit of the client device 120 to display it.

[0078] 12 is a diagram showing an example of a movie clip creation screen 1700. The movie clip creation screen 1700 is configured so that a movie clip can be created. A movie clip is a function that cuts out a video by specifying a time period that you want to keep from a video, and saves the video in the storage unit 220 or the like. Note that a movie clip may include a video to be saved, which is the specified video, and a time lapse, which is a fast-forwarded video of the specified video.

[0079] A video timeline 1710 is displayed on movie clip creation screen 1700. The operator can specify the start and end times of the video to be cut out by operating anchors 1740 and 1750 on timeline 1710. Anchor 1740 is a GUI component that specifies the start time of the video. Anchor 1750 is a GUI component that specifies the end time of the video. A still image of the video at the time specified by anchor 1740 is displayed in display area 1720. A still image of the video at the time specified by anchor 1750 is displayed in display area 1730.

[0080] The control unit 210 can create a movie clip of the video based on the operation of the operator of the client device 120 and store it in a predetermined storage area such as the storage unit 220.

[0081] The control unit 210 outputs a timeline relating to the selected video, and outputs a screen for creating a movie clip relating to the video based on a predetermined operation via the screen including the timeline.

[0082] (2-2) Face image search 13 is a flowchart showing an example of information processing related to face image search in the information processing system 1000. Note that, although this processing will be described as being mainly executed by the server device 100, it may instead be executed by either the client device 110 or 120, or one of the multiple surveillance cameras 160. In step S1110, control unit 210 generates a screen (saved person display screen) including multiple registered face images in response to a request from client device 120. Specifically, it extracts the best shot information that was flagged in step S570 from the multiple best shot information, identifies the best shot 1 for each, and generates a saved person display screen for displaying them. Control unit 210 transmits the generated saved person display screen to client device 120.

[0083] FIG. 14 is a diagram showing an example of a person search screen 1200. As shown in FIG. 14, the person search screen 1200 includes a saved person tab 1210 and a saved person display area 1250. When the saved person tab 1210 is selected, a display request for a saved object display screen is transmitted from the client device 120 to the server device 100. Then, in response to the display request, a list of face images of people whose face images were registered in step S570 is displayed in the saved person display area 1250. FIG. 14 shows that the saved person tab 1210 has been selected. As an example, the saved person display area 1250 in FIG. 14 displays face image 1220, face image 1230, and face image 1240 as a list of saved people.

[0084] The operator of client device 120 (such as the owner of surveillance camera 160) selects a face image of a person to be searched from the face images displayed in saved person display area 1250. When the operator of client device 120 selects a face image, selection information including identification information for identifying the selected face image is transmitted from client device 120 to server device 100.

[0085] In step S1120, control unit 210 determines whether or not selection information has been received from client device 120. If control unit 210 has received selection information, it proceeds to step S1130, and if control unit 210 has not received selection information, it repeats the processing of step S1120.

[0086] In step S1130, the control unit 210 selects a face image based on the identification information included in the selection information. In step S1140, control unit 210 searches for video captured by surveillance camera 160 that includes the selected face image. Specifically, first, best shot information for the face image selected in step S1130 is referenced, and the corresponding face feature amount is used to identify the video that includes that person using an AI model or the like. Note that if the face feature amount was not calculated in the preprocessing described above, the face feature amount can be calculated in a similar manner at this point. Note that step S1140 may be implemented as follows. For example, the control unit 210 inputs a facial image and multiple videos stored in the storage unit 220 or the like as input data to the trained model. This trained model is trained using the facial image and the multiple videos as input data, and whether the videos include a person identical to the person in the facial image, and, if the video includes the same person as the person in the facial image, time information from the appearance to the disappearance of the person in the facial image as output data. Furthermore, if the same video includes multiple facial images, the trained model is trained using time information from the appearance to the disappearance of each of the multiple facial images as output data.

[0087] The control unit 210 obtains, as search results, information from the trained model as to whether the video contains the same person as the person in the facial image, and, if the video contains the same person as the person in the facial image, information on the time from when this person appeared to when they disappeared.

[0088] In step S1150, the control unit 210 generates a list display screen including thumbnails of the timings when facial images of videos including the same person as the person in the selected facial image appeared, based on the search results. The control unit 210 transmits the generated list display screen to the client device 120 and controls the output unit of the client device 120 to display it.

[0089] FIG. 15 is a diagram showing an example of a list display screen 1300. The list display screen 1300 shown in FIG. 15 includes a saved person display area 1320. The saved person display area 1320 displays a face image 1310 selected on the person search screen 1200 and a list of thumbnails of videos that include the same person as the person in the face image 1310. The list of thumbnails is displayed for each device. For simplicity's sake, FIG. 15 displays only a list of thumbnails of videos captured by the surveillance camera 160 at the east entrance of the store as the device section 1340, but the present invention is not limited to this. As shown on the list display screen 600 in FIG. 6, a list of thumbnails is displayed for each device section.

[0090] When one of the thumbnails is selected on the list display screen 1300 of FIG. 15, thumbnail selection information including identification information for identifying the selected thumbnail is transmitted from the client device 120 to the server device 100.

[0091] 13, the control unit 210 determines whether or not thumbnail selection information has been received. If thumbnail selection information has been received, the control unit 210 proceeds to step S1170; if thumbnail selection information has not been received, the control unit 210 repeats the processes of steps S1150 and S1160.

[0092] In step S1170, the control unit 210 selects the image of the thumbnail identified by the thumbnail selection information. In step S1180, control unit 210 generates a video detail screen including a timeline of the selected video and facial images of people included in the video. Control unit 210 transmits the generated video detail screen to client device 120 and controls the output unit of client device 120 to display it.

[0093] Fig. 16 is a diagram showing an example of a video detail screen 1400. The video detail screen 1400 is almost the same as the video detail screen 700 shown in Fig. 8 and the video detail screen 700 shown in Fig. 9. However, the face image display area 1410 included in the video detail screen 1400 does not include the "Register Person" button 775 included in the video detail screen 700 in Fig. 8 and the "Registered" button 810 in Fig. 9.

[0094] (2-3) Destination settings In response to a request from the client device 110, the control unit 210 generates a notification setting (also called destination setting) screen and transmits it to the client device 110.

[0095] Fig. 17 is a diagram showing an example of a notification setting screen 1800. As shown in Fig. 17, the notification setting screen 1800 includes a notification setting tag 1810 and a notification setting area 1820. When the notification setting tag 1810 is selected, a request to generate a notification setting screen is transmitted from the client device 110 to the server device 100. Then, in response to the generation request, a GUI component 1830 that allows a destination to be set for each device is displayed in the notification setting area 1820. The example in Fig. 17 shows that the notification setting tag 1810 has been selected. This process is an example of a process for outputting a notification setting screen 1800 for a destination to which information indicating that a person in a registered face image has been detected is to be sent when the person in the face image is detected in a video.

[0096] The GUI component 1830 is configured so that the destination of email and Webhook linkage can be set for each device name. The device name may be a name that can uniquely identify the device, or a name that can identify a tag (device tag) attached to the device. A device tag can be attached to a group of surveillance cameras 160. The notification setting screen 1800 is configured so that the destination can be set for each of multiple cameras or for each group of cameras.

[0097] When a device name (for example, “Entrance East Exit” 1820) is selected in the GUI component 1830, the control unit 210 generates a notification setting screen 1900 and controls the output unit 340 of the client device 110 to display it.

[0098] 18 is a diagram showing an example of a notification settings screen 1900. An item 1910 allows the user to set the destination of notification emails. An item 1920 allows the user to set the destination of Webhook collaboration. Multiple contacts can be set as destinations of Webhook collaboration. In FIG. 18, the notification setting screen 1900 is configured so that users can be set as destinations.

[0099] As described above, according to the first embodiment, it is possible to easily find a person from a video and register a facial image of the person of interest in the information processing system 1000. Furthermore, according to the first embodiment, it is possible to detect and output a video of a person whose facial image is selected from the registered facial images.

[0100] <Additional Notes> It may be provided in the following manner. (Appendix 1) An information processing system that displays an image captured by a camera on an information terminal, Searching the video for videos containing facial images; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying a facial image included in the selected video together with the selected video; registering the face image in accordance with a user's instruction; Information processing system. (Appendix 2) An information processing system that displays an image captured by a camera on an information terminal, Displaying multiple registered face images, selecting a face image from the plurality of face images in response to a user instruction; Searching for a video containing a person corresponding to the selected face image; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying the selected image; Information processing system. (Appendix 3) 10. The information processing system of claim 1, The facial images to be displayed are multiple facial images of the same person. Information processing system. (Appendix 4) 4. The information processing system according to claim 3, The plurality of facial images include facial images of the same person taken from different angles. Information processing system. (Appendix 5) 5. The information processing system according to claim 4, the plurality of face images include at least one of a face image facing forward, a face image facing right, a face image facing left, and a face image including the entire body; Information processing system. (Appendix 6) 10. The information processing system of claim 1, displaying the face image together with information as to whether the face image has been registered in a manner that makes it possible to identify the face image; Information processing system. (Appendix 7) 10. The information processing system of claim 1, displaying a button for registering the face image together with the face image; Information processing system. (Appendix 8) 10. The information processing system according to claim 1, Displaying a timeline related to the video together with the video; Information processing system. (Appendix 9) 9. The information processing system according to claim 8, the timeline displays the portion of the registered face image in a distinguishable manner; Information processing system. (Appendix 10) 9. The information processing system according to claim 8, displaying a screen for creating a movie clip related to the video based on a predetermined operation via the screen including the timeline; Information processing system. (Appendix 11) 10. The information processing system according to claim 1, The search for the video containing the face image is performed based on search conditions input by the user; The search conditions include at least one of a period and a keyword. Information processing system. (Appendix 12) 10. The information processing system according to claim 1, The plurality of images are displayed in the form of thumbnails for each camera that captured the plurality of images. Information processing system. (Appendix 13) 10. The information processing system according to claim 1, If the registered face image is detected in the video, the user is notified. Information processing system. (Appendix 14) 14. The information processing system of claim 13, The notification destination when the registered face image is detected from the video can be set for each of the plurality of cameras or for each group of cameras. Information processing system. (Appendix 15) 10. The information processing system according to claim 1, The registered face image can be changed or a face image can be added based on a user's instruction. Information processing system. (Appendix 16) 10. The information processing system according to claim 1, The facial image is displayed in a manner that makes it clear that the facial image is a person included in the selected video. Information processing system. (Appendix 17) 10. The information processing system according to claim 1, displaying an image of a person whose face is recognizable among the people included in the selected video together with the video; Information processing system. (Appendix 18) An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising: Searching the video for videos containing facial images; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying a facial image included in the selected video together with the selected video; registering the face image in accordance with a user's instruction; Information processing methods. (Appendix 19) An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising: Displaying multiple registered face images, selecting a face image from the plurality of face images in response to a user instruction; Searching for a video containing a person corresponding to the selected face image; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying the selected image; Information processing methods. (Appendix 20) A computer that displays the images captured by the camera on an information terminal. Searching the video for videos containing facial images; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying a facial image included in the selected video together with the selected video; registering the face image in accordance with a user's instruction; A program for executing a process. (Appendix 21) A computer that displays the images captured by the camera on an information terminal. Displaying multiple registered face images, selecting a face image from the plurality of face images in response to a user instruction; Searching for a video containing a person corresponding to the selected face image; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying the selected image; A program for executing a process.

[0101] While various embodiments of the present invention have been described, they are presented as examples and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. The embodiments and modifications thereof are intended to be included within the scope and spirit of the invention, and are also intended to be included in the scope of the invention and its equivalents as defined in the claims.

[0102] For example, part of the processing of the server device 100 described above may be executed by the client device 110 and / or the client device 120. Also, part of the processing of the server device 100 described above may be executed by the surveillance camera 160. [Explanation of symbols]

[0103] 100: Server device 110: Client device 150: Network 160: Surveillance camera 210: Control unit 220: Storage section 230: Communications Department 1000: Information Processing Systems

Claims

1. An information processing system that displays an image captured by a camera on an information terminal, Searching the video for videos containing facial images; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying a facial image included in the selected video together with the selected video; registering the face image in accordance with a user's instruction; Information processing system.

2. An information processing system that displays an image captured by a camera on an information terminal, Displaying multiple registered face images, selecting a face image from the plurality of face images in response to a user instruction; Searching for a video containing a person corresponding to the selected face image; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; Displaying the selected image. Information processing system.

3. 2. The information processing system according to claim 1, The facial images to be displayed are multiple facial images of the same person. Information processing system.

4. 4. The information processing system according to claim 3, The plurality of facial images include facial images of the same person taken from different angles. Information processing system.

5. 5. The information processing system according to claim 4, the plurality of face images include at least one of a face image facing forward, a face image facing right, a face image facing left, and a face image including the entire body; Information processing system.

6. 2. The information processing system according to claim 1, displaying the face image together with information as to whether the face image has been registered in a manner that makes it possible to identify the face image; Information processing system.

7. 2. The information processing system according to claim 1, displaying a button for registering the face image together with the face image; Information processing system.

8. 3. The information processing system according to claim 1, Displaying a timeline related to the video together with the video; Information processing system.

9. 9. The information processing system according to claim 8, the timeline displays the portion of the registered face image in a distinguishable manner; Information processing system.

10. 9. The information processing system according to claim 8, displaying a screen for creating a movie clip related to the video based on a predetermined operation via the screen including the timeline; Information processing system.

11. 3. The information processing system according to claim 1, The search for the video containing the face image is performed based on search conditions input by the user; The search conditions include at least one of a period and a keyword. Information processing system.

12. 3. The information processing system according to claim 1, The plurality of images are displayed in the form of thumbnails for each camera that captured the plurality of images. Information processing system.

13. 3. The information processing system according to claim 1, If the registered face image is detected in the video, the user is notified. Information processing system.

14. 14. The information processing system according to claim 13, The notification destination when the registered face image is detected from the video can be set for each of the plurality of cameras or for each group of cameras. Information processing system.

15. 3. The information processing system according to claim 1, The registered face image can be changed or a face image can be added based on a user's instruction. Information processing system.

16. 3. The information processing system according to claim 1, The facial image is displayed in a manner that makes it clear that the facial image is a person included in the selected video. Information processing system.

17. 3. The information processing system according to claim 1, displaying an image of a person whose face is recognizable among the people included in the selected video together with the video; Information processing system.

18. An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising: Searching the video for videos containing facial images; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying a facial image included in the selected video together with the selected video; registering the face image in accordance with a user's instruction; Information processing methods.

19. An information processing method executed by an information processing system that displays an image captured by a camera on an information terminal, comprising: Displaying multiple registered face images, selecting a face image from the plurality of face images in response to a user instruction; Searching for a video containing a person corresponding to the selected face image; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; Displaying the selected image. Information processing methods.

20. A computer that displays the images captured by the camera on an information terminal. Searching the video for videos containing facial images; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; displaying a facial image included in the selected video together with the selected video; registering the face image in accordance with a user's instruction; A program for executing a process.

21. A computer that displays the images captured by the camera on an information terminal. Displaying multiple registered face images, selecting a face image from the plurality of face images in response to a user instruction; Searching for a video containing a person corresponding to the selected face image; displaying a plurality of videos as a result of said search; Selecting an image from the plurality of images in response to a user instruction; Displaying the selected image. A program for executing a process.

Images