Information processing device, information processing method, and program

Abstract

To provide a program that allows a user to easily search for a target person from surveillance images of a desired surveillance camera.SOLUTION: A program causes one or more computers to execute processes of: displaying a plurality of camera icons on a display device; accepting input enclosing at least two or more camera icons from among the plurality of camera icons displayed on the display device, and searching for a person on the basis of images taken by a camera corresponding to each of the at least two or more camera icons in response to the accepting the input enclosing at least two or more camera icons.SELECTED DRAWING: Figure 28

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a person search system that displays an image list of a detected person in the footage of a camera selected from multiple cameras by user operation, and searches for the target person by matching the image with the person images, searching for images from other cameras excluding the camera from which the person image of the target person selected from the image list was obtained. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6573346 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the person search system exemplified in Patent Document 1, when a user selects a portrait image of a target person from an image list, the user cannot arbitrarily specify a camera to be searched. In view of the above-mentioned problems, the present invention has an object to provide an information processing device, an information processing method, and a program that enable a user to easily search for a target person from within surveillance footage from a desired surveillance camera. [Means for solving the problem]

[0005] According to one aspect of the present invention, a method includes a process of displaying a plurality of camera icons on a display device, a process of receiving an input to surround the camera icons displayed on the display device, and a process of searching for a person based on images captured by the cameras corresponding to the camera icons in response to receiving the input, and when a plurality of camera icons are surrounded by the input in the process of searching for the person, the method causes one or more computers to execute a process of searching for a common character that appears in common in images captured by the cameras corresponding to all of the surrounded camera icons. When a plurality of camera icons are surrounded by the input, this includes a case where three or more camera icons are surrounded by the input. The program is provided.

[0006] According to another aspect of the present invention, the method includes the steps of: a computer causes a display device to display a plurality of camera icons; a computer receives an input to surround the camera icons displayed on the display device; and a computer performs a person search based on images captured by cameras corresponding to the camera icons in response to receiving the input. In the person search step, when a plurality of camera icons are surrounded by the input, a common character that appears in common in images captured by the cameras corresponding to all of the surrounded camera icons is searched for. and when a plurality of camera icons are surrounded by the input, this includes a case where three or more camera icons are surrounded. A method for processing information is provided.

[0007] According to yet another aspect of the present invention, there is provided a person search system in which a camera, an information processing terminal capable of specifying person search conditions in response to receiving an input of encircling a camera icon displayed on a display device, and a server are connected via a network, the server comprising means for receiving the person search conditions from the information processing terminal, and means for searching for the person satisfying the search conditions, and the means for searching for the person searches for common characters that appear in common in images captured by the cameras corresponding to all of the surrounded camera icons when a plurality of camera icons are surrounded by the input. death , When a plurality of camera icons are surrounded by the input, this includes a case where three or more camera icons are surrounded. A people search system is provided. Effect of the Invention

[0008] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that allow a user to easily search for a target person from within surveillance footage from a desired surveillance camera. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of an overall configuration of a person search system according to a first embodiment. [Diagram 2] FIG. 4 is a diagram showing an example of information stored in a registrant information DB in the first embodiment. [Diagram 3] 4 is a diagram showing an example of information stored in a video information DB in the first embodiment. FIG. [Figure 4] 4 is a diagram showing an example of information stored in a camera information DB in the first embodiment. FIG. [Diagram 5] 5 is a diagram illustrating an example of information stored in a detection history information DB in the first embodiment. FIG. [Figure 6] 2 is a block diagram showing an example of a hardware configuration of a user terminal in the first embodiment. FIG. [Figure 7] 2 is a block diagram showing an example of a hardware configuration of a management server according to the first embodiment. FIG. [Figure 8] 13 is a flowchart illustrating an example of a process of a management server in the first embodiment. [Figure 9] FIG. 4 is a sequence diagram showing an example of processing of the person search system in the first embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a search screen relating to a first search method. [Figure 11] FIG. 11 is a diagram showing an example of a search screen relating to a first search method. [Figure 12] FIG. 11 is a diagram showing an example of a search result according to a first search method. [Figure 13] FIG. 11 is a diagram showing an example of a search result according to a first search method. [Figure 14] FIG. 11 is a diagram showing an example of a search result according to a first search method. [Figure 15] FIG. 4 is a sequence diagram showing an example of processing of the person search system in the first embodiment. [Figure 16] FIG. 11 is a diagram showing an example of a search screen relating to a second search method. [Figure 17] FIG. 11 is a diagram showing an example of a search result according to a second search method. [Figure 18] FIG. 4 is a sequence diagram showing an example of processing of the person search system in the first embodiment. [Figure 19] FIG. 13 is a diagram showing an example of a search screen relating to a third search method. [Figure 20] FIG. 11 is a diagram showing an example of a search result according to a third search method. [Figure 21] FIG. 4 is a sequence diagram showing an example of processing of the person search system in the first embodiment. [Figure 22] FIG. 13 is a diagram showing an example of a search screen relating to a fourth search method. [Figure 23] FIG. 13 is a diagram showing an example of a search result according to a fourth search method. [Figure 24] FIG. 11 is a diagram showing an example of a search screen in the second embodiment. [Diagram 25] FIG. 11 is a diagram illustrating an example of a search result in the second embodiment. [Figure 26] FIG. 13 is a diagram showing an example of a search screen in the third embodiment. [Figure 27] FIG. 13 is a diagram showing an example of a search screen in the fourth embodiment. [Figure 28] FIG. 13 is a sequence diagram showing an example of a process of the person search system in the fifth embodiment. [Figure 29] FIG. 13 is a diagram showing an example of a search screen relating to a fifth search method. [Diagram 30] FIG. 13 is a diagram showing an example of a search result according to a fifth search method. [Diagram 31] FIG. 13 is a diagram showing an example of a search result according to a fifth search method. [Diagram 32] FIG. 13 is a sequence diagram showing an example of a process of the person search system in the fifth embodiment. [Diagram 33]FIG. 13 is a diagram showing an example of a search screen relating to a sixth search method. [Diagram 34] FIG. 13 is a diagram showing an example of a search result according to a sixth search method. [Diagram 35] 23 is a flowchart showing an example of processing of the person search system in the sixth embodiment. [Diagram 36] 23 is a flowchart showing an example of processing of the person search system in the seventh embodiment. [Figure 37] FIG. 13 is a block diagram showing a configuration of an information processing device according to an eighth embodiment. [Figure 38] FIG. 13 is a block diagram showing an example of an overall configuration of a person search system according to a modified embodiment. [Figure 39] FIG. 13 is a diagram showing an example of a search screen in a modified embodiment. [Diagram 40] FIG. 13 is a diagram showing an example of a search screen in a modified embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. In the drawings, similar or corresponding elements are designated by the same reference numerals, and descriptions thereof may be omitted or simplified.

[0011] [First embodiment] 1 is a block diagram showing an example of the overall configuration of a person search system 1 according to this embodiment. The person search system 1 is a computer system that searches for a person designated as a search target by a user on a user terminal 10 (hereinafter referred to as a "target person") from within surveillance video by comparing the person with video stored in a management server 20.

[0012] 1, the person search system 1 includes a user terminal 10, a management server 20, and a surveillance camera 30. The user terminal 10, the management server 20, and the surveillance camera 30 are connected via a network NW1 and a network NW2 such as the Internet or a LAN (Local Area Network). In this embodiment, the network NW1 and the network NW2 are different networks, but may be configured as the same network.

[0013] The user terminal 10 is a terminal used by a user of the present system when performing various input operations such as specifying a target person, specifying a surveillance camera 30 corresponding to the video to be searched, and specifying search conditions. As the user terminal 10, for example, a laptop computer, a desktop computer, a tablet terminal, a smartphone, etc. are preferably used. When the user terminal 10 accepts an input operation from a user, it requests the management server 20 to execute a predetermined process based on the input information.

[0014] The management server 20 is a device that manages images captured by the surveillance camera 30, analyzes the images, performs image matching, and manages biometric information of registered persons and detected persons. As the management server 20, for example, a server computer is suitably used.

[0015] The surveillance camera 30 is, for example, a photographing device such as a digital video camera or an infrared camera. The type of surveillance camera 30 can be appropriately selected depending on the subject to be monitored and the installation environment. The surveillance camera 30 can be installed in any location such as a residential area, a downtown area, a transportation facility, a financial institution, a factory, an office building, a theme park, or an event venue.

[0016] 1, the management server 20 includes a registrant information DB 21, a video information DB 22, a camera information DB 23, and a detection history information DB 24. However, the databases included in the management server 20 are not limited to these.

[0017] FIG. 2 is a diagram showing an example of information stored in the registrant information DB21 in this embodiment. Data items of the registrant information include a registrant ID, a registered face image, facial feature values, name, sex, age, registration date and time, and reason for registration. However, the data items are not limited to these. In this embodiment, a "registrant" refers to a person who is registered in various lists such as a VIP list, a blacklist, and a customer list. The reason for registration is the reason why a person is registered in the registrant information DB21. For example, if the registrant information is a blacklist that registers suspicious persons, the detailed reason is recorded.

[0018] 3 is a diagram showing an example of information stored in the video information DB 22 in this embodiment. Data items of the video information include a video ID unique to each video data, a file name, a file save destination, a camera ID of the surveillance camera 30 that captured the video, a shooting location, a shooting start date and time, a shooting end date and time, a DB registration date and time, and a frame rate. However, the data items are not limited to these.

[0019] 4 is a diagram showing an example of information stored in the camera information DB 23 in this embodiment. Data items of the camera information include a camera ID unique to each surveillance camera 30, location information indicating the installation location of the surveillance camera 30, the shooting direction of the surveillance camera 30, and the camera type. However, the data items are not limited to these.

[0020] 5 is a diagram showing an example of information stored in the detection history information DB24 in this embodiment. Data items of the detection history information include detection SEQ, detected face image, face feature amount, video ID for identifying the video of the detection source, camera ID, detection time, and registered user ID. However, the data items are not limited to these. Detection SEQ is a unique number that is newly issued each time a person is detected. Detection time is a timestamp of the display frame where the face image was detected.

[0021] Fig. 6 is a block diagram showing an example of a hardware configuration of the user terminal 10 in this embodiment. As shown in Fig. 6, the user terminal 10 is a computer that performs calculations, control, and storage, and includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, a storage 104, a communication I / F (Interface) 105, a display device 106, an input device 107, and a medium reading device 108. Each device is connected to each other via a bus, wiring, a drive device, etc.

[0022] The CPU 101 is a processor that performs a predetermined operation according to a program stored in the ROM 103 or the storage 104, and has a function of controlling each part of the user terminal 10. The RAM 102 is composed of a volatile storage medium, and provides a temporary memory area necessary for the operation of the CPU 101.

[0023] The ROM 103 is composed of a non-volatile storage medium, and stores necessary information such as programs used for the operation of the user terminal 10. The storage 104 is composed of a non-volatile storage medium, and stores programs and data for the operation of the user terminal 10. The storage 104 is composed of, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0024] The communication I / F 105 is a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and 4G, and is a module for communicating with other devices.

[0025] The display device 106 is a liquid crystal display, an OLED (Organic Light Emitting Diode) display, or the like that displays moving images, still images, characters, and the like, and is used to present information to the user.

[0026] The input device 107 is a keyboard, a pointing device, a button, etc., and receives operations by a user. The display device 106 and the input device 107 may be integrally formed as a touch panel.

[0027] The medium reading device 108 is a device that reads video data and still image data recorded on a recording medium. Examples of the recording medium include a DVD disk, a Blu-Ray (registered trademark) disk, a magnetic tape, a flash memory, a smartphone, etc. The medium reading device 108 is configured by, for example, a disk drive, a card reader, an image scanner, etc.

[0028] Fig. 7 is a block diagram showing an example of a hardware configuration of the management server 20 in this embodiment. As shown in Fig. 7, the management server 20 is a computer that performs calculations, control, and storage, and includes a CPU 201, a RAM 202, a ROM 203, a storage 204, a communication I / F (Interface) 205, a display device 206, an input device 207, and a medium reading device 208. Each device is connected to each other via a bus, wiring, a drive device, etc.

[0029] It should be noted that the hardware configurations shown in Figures 6 and 7 are merely examples, and other devices may be added, or some devices may not be provided. Some devices may be replaced with other devices having similar functions. Some functions of this embodiment may be provided by other devices via a network, or the functions of this embodiment may be distributed and realized among a plurality of devices. In this way, the hardware configurations shown in Figures 6 and 7 can be modified as appropriate.

[0030] Next, the operation of the person search system 1 in this embodiment will be described with reference to FIGS.

[0031] <Person detection processing> 8 is a flowchart showing an example of the processing of the management server 20 in this embodiment. This processing is executed when the management server 20 detects a person appearing in a video captured by the surveillance camera 30 and registers a face image of the detected person in the detection history information DB 24 in a searchable state.

[0032] In step S101, the management server 20 determines whether or not video data has been acquired from the surveillance camera 30 or the medium reading device 208. If the management server 20 determines that video data has been acquired (step S101: YES), the process proceeds to step S102.

[0033] On the other hand, if the management server 20 determines that the video data has not been acquired (step S101: NO), the process proceeds to step S103.

[0034] In step S102, the management server 20 stores the video data in the storage 204. At this time, the management server 20 issues a unique video ID to the video data, and registers in the video information DB 22 the video information including the video ID.

[0035] In step S103, the management server 20 determines whether or not to execute a person detection process on the video data stored in the storage 204. Here, when the management server 20 determines to execute a person detection process (step S103: YES), the process proceeds to step S104.

[0036] On the other hand, if the management server 20 determines not to execute the person detection process (step S103: NO), the process of FIG. 8 ends.

[0037] In step S104, the management server 20 loads into the RAM 202, from among the video data stored in the storage 204, video data on which human detection processing has not yet been performed.

[0038] In step S105, the management server 20 determines whether or not a human face is detected from the video data. If the management server 20 determines that a human face is detected (step S105: YES), the process proceeds to step S106.

[0039] On the other hand, if the management server 20 determines that a human face has not been detected (step S105: NO), the process proceeds to step S115.

[0040] In step S106, the management server 20 issues a detection SEQ in response to the detection of a face image. Next, when the management server 20 acquires a face image from the video data (step S107), the management server 20 calculates a face feature amount from the face image (step S108).

[0041] In step S109, the management server 20 determines whether the detected face image is an image of a person that has been detected for the first time. For example, the management server 20 can determine whether the face image is registered by searching the detection history information DB 24 based on the face feature amount.

[0042] Here, if the management server 20 determines that the face image is of a person detected for the first time (step S109: YES), the process proceeds to step S110. On the other hand, if the management server 20 determines that the face image is of a person who has already been detected (step S109: NO), the process proceeds to step S111.

[0043] In step S110, the management server 20 issues a unique detected person ID to the detected person (hereinafter referred to as a "detected person"). In this embodiment, the detected person ID is assumed to be common among a plurality of videos. Meanwhile, in step S111, the management server 20 acquires the detected person ID from the detection history information DB 24 using, for example, face feature amounts as a key.

[0044] In step S112, the management server 20 determines whether or not the detected person is a registrant in the registrant information DB 21. For example, the management server 20 can determine that the detected person is a registrant when a registrant ID is associated with a detected person ID in the detection history information DB 24. The management server 20 may also determine whether or not the detected person is a registrant by searching the registrant information DB 21 based on a facial feature amount.

[0045] Here, if the management server 20 determines that the detected person is a registrant (step S112: YES), the process proceeds to step S113. On the other hand, if the management server 20 determines that the detected person is not a registrant (step S112: NO), the process proceeds to step S114.

[0046] In step S113, the management server 20 acquires from the registrant information DB 21 a registrant ID related to the detected person.

[0047] In step S114, the management server 20 creates detection history information regarding the detected person, and registers this in the detection history information DB 24.

[0048] In step S115, the management server 20 determines whether the video data has ended. If the management server 20 determines that the video data has ended (step S115: YES), the process of FIG.

[0049] On the other hand, if the management server 20 determines that the video data has not ended (step S115: NO), the process returns to step S104.

[0050] <Search process for target person> Next, a description will be given of a search process for a target person in the person search system 1. In this embodiment, four types of methods are illustrated for a user to specify a target person for a tracking operation. However, the search method for a target person is not limited to only four types.

[0051] (First search method) 9 is a sequence diagram showing an example of the processing of the person search system 1 in this embodiment. This processing relates to a first search method that is executed when a user selects an image of a target person from the detected person list.

[0052] In step S201, the user terminal 10 displays a search screen on the display device 106. Fig. 10 is a diagram showing an example of the search screen related to the first search method. Here, the left region of the search screen is provided with, from the top, an input field for a search period, an input field for searching and specifying a search area, and a display field for a map M1 showing the search area.

[0053] In addition, the right side area of ​​the search screen is provided with three tabs T1 to T3 for selecting a search method. On the screen, the "Detected Person List" tab T1 is selected, and guidance information regarding the search method ("Select the camera icon on the map. People photographed by the specified camera within the search period will be displayed in list form.") is displayed in tab T1.

[0054] In step S202, the user terminal 10 determines whether or not a camera icon has been selected on the map. If the user terminal 10 determines that the camera icon has been selected (step S202: YES), the process proceeds to step S203.

[0055] On the other hand, if the user terminal 10 determines that the camera icon has not been selected on the map (step S202: NO), the process of step S202 is repeated.

[0056] In step S203, the user terminal 10 identifies the surveillance camera 30 corresponding to the camera icon designated by the user, and acquires the camera ID. In the example of Fig. 10, the camera icon selected by the operation of the mouse pointer P is the camera icon C1. In the following description, it is assumed that the reference numeral attached to the camera icon is the same as the camera ID of the surveillance camera 30 corresponding to the camera icon.

[0057] In step S204, the user terminal 10 makes a request to the management server 20 to transmit a list of persons detected by the identified surveillance camera 30. The transmission request includes the camera ID of the identified surveillance camera 30.

[0058] In step S205, the management server 20 refers to the detection history information DB 24 using the camera ID included in the transmission request received from the user terminal 10 as a key, and acquires the detection history information related to the identified surveillance camera 30. If a search period is specified on the search screen, only detection history information whose detection time is included in the search period can be extracted.

[0059] In step S206, the management server 20 transmits the detected person list created based on the detection history information related to the identified surveillance camera 30 to the user terminal 10. Next, the user terminal 10 displays the detected person list received from the management server 20 on the search screen (step S207).

[0060] FIG. 11 is a diagram showing an example of a search screen relating to the first search method. Here, a detected person list LS1 relating to the video captured by the surveillance camera 30 with a camera ID of "C1" is displayed in a tab T1 of the search screen. The data items of the detected person list LS1 include a face image of the detected person, a detection date and time, a registered face image, detailed information, and an alert. However, the data items of the detected person list LS1 are not limited to these. The registered face image is displayed when the detected person is registered as a registrant in the registrant information DB21. Also, for example, when the reason for registration of the registrant is "suspect person" or the like, it is preferable to display a figure or the like for calling attention in the data column of the alert.

[0061] The dashed arrow A1 in Figure 11 indicates that the user has performed an operation (hereinafter referred to as a ``drag operation'' or ``drag'') using the mouse pointer P to superimpose a facial image in the detected person list LS1 onto another surveillance camera 30 (camera ID: C2) displayed on the map M1.

[0062] In step S208, the user terminal 10 determines whether or not a face image in the detected person list has been dragged to a camera icon on the map. If the user terminal 10 determines that a face image has been dragged to a camera icon on the map (step S208: YES), the process proceeds to step S209.

[0063] On the other hand, if the user terminal 10 determines that the face image has not been dragged to the camera icon on the map (step S208: NO), the process returns to step S202.

[0064] In step S209, the user terminal 10 identifies the destination of the facial image, that is, the surveillance camera 30 corresponding to the camera icon to be searched, and acquires the camera ID. In the example of Fig. 11, the camera ID of the surveillance camera 30 corresponding to the camera icon of the destination is "C2".

[0065] In step S210, the user terminal 10 acquires a search period specified by the user on the search screen. In the example of Fig. 11, "2019 / 12 / 31 12:00" to "2019 / 12 / 31 14:00" is specified as the search period.

[0066] In step S211, the user terminal 10 transmits a search request for a target person to the management server 20. The search request includes information such as a facial image of the target person, a detected person ID, a search period, and a camera ID of the surveillance camera 30 to be searched.

[0067] In step S212, the management server 20 searches for detection history information relating to the surveillance camera 30 to be searched, based on the face image included in the search request received from the user terminal 10.

[0068] In step S213, the management server 20 transmits the search results to the user terminal 10. The search results include, for example, a video ID that identifies video data in which the target person appears, information on the shooting date and time of the target person, and the like.

[0069] In step S214, the user terminal 10 displays the search results on the display device 106. Fig. 12 is a diagram showing an example of a search result related to the first search method. In Fig. 12, in a tab T1 on the search screen, the search execution conditions (target person's face image (IMG-1) / search target camera (C2) / search target period) specified by the user and the search results (video / shooting date and time (2019 / 12 / 31 13:15:20 - 2019 / 12 / 31 13:15:30) / matching score (95.2)) R1 are displayed.

[0070] Note that various variations in the information displayed as the search results are possible. Fig. 13 and Fig. 14 are diagrams showing examples of search results related to the first search method. In the example of Fig. 13, a screen for playing an extracted video MV in which a target person P-1 appears is superimposed on the search screen. Also, in the extracted video MV, icons I-1 to I-3 are superimposed around the target person P-1 as indicators indicating the direction of the installation position and the camera ID of a surveillance camera 30 other than the surveillance camera 30 that captured the extracted video MV.

[0071] In addition, it is preferable that the user can request a process of matching the facial image of the target person P-1 with a video taken by a camera (camera ID: C1 / C4 / C7) other than the surveillance camera 30 related to the extracted video MV by specifying the facial portion of the target person P-1 on the same screen and performing a drag operation to overlap it with an icon in the extracted video MV. This allows the user to efficiently track the target person P-1.

[0072] 14, unlike the case of Fig. 13, only the icon (camera ID: C7) indicating the surveillance camera 30 installed in the movement direction of the target person P-1 obtained from the extracted video MV is displayed. Furthermore, on the map M1, the camera icon C7 indicating the surveillance camera 30 (camera ID: C7) installed in the movement direction of the target person P-1 is highlighted by being surrounded by a circular dashed line B.

[0073] The user can preferably request a process of matching the facial image of the target person P-1 with a video captured by a camera (camera ID: C7) other than the surveillance camera 30 related to the extracted video MV by specifying the facial portion of the target person P-1 on the screen and dragging it over the camera icon C7 highlighted on the map M1. This allows the user to efficiently track the target person P-1.

[0074] The method of highlighting the camera icon is not limited to the above, and for example, the display style, such as color, size, shape, etc., may be changed according to the matching score.

[0075] (Second search method) 15 is a sequence diagram showing an example of the processing of the person search system 1. This processing relates to a second search method that is executed when the user specifies a target person from within the video displayed on the search screen.

[0076] In step S301, the user terminal 10 displays a search screen on the display device 106. Fig. 16 is a diagram showing an example of the search screen relating to the second search method. A map M1 of the search target area specified by a user operation is displayed in the map display field of the search screen. Also, in the right area of ​​the search screen, the "Video List" tab T2 is selected.

[0077] The dashed arrow A2 in Figure 16 indicates that the user has used the mouse pointer P to designate the face of the target person P-2 displayed in the selected video MV, and performed a drag operation to superimpose the face of the target person P-2 over the camera icon C3 displayed on the map M1.

[0078] In step S302, the user terminal 10 identifies the surveillance cameras 30 included in the displayed map. In the example of Fig. 16, the camera IDs (C1 to C15) of the surveillance cameras 30 corresponding to the camera icons C1 to C15 displayed on the map M1 are identified.

[0079] In step S303, the user terminal 10 transmits a request to transmit a video list to the management server 20. The request includes all camera IDs identified in step S302.

[0080] In step S304, the management server 20 refers to the detection history information DB 24 using the camera ID included in the transmission request received from the user terminal 10 as a key, acquires detection history information related to the identified surveillance camera 30, and transmits a video list created based on the detection history information to the user terminal 10. If a search period is specified on the search screen, only detection history information whose detection time is included in the search period can be extracted.

[0081] In step S305, the user terminal 10 displays the video list received from the management server 20 on the display device 106. In the example of Fig. 16, a video list LS2 of multiple surveillance cameras 30 corresponding to camera icons C1 to C15 displayed on a map M1 is displayed.

[0082] In step S306, the user terminal 10 determines whether or not any video has been selected from the video list. If the user terminal 10 determines that a video has been selected (step S306: YES), the process proceeds to step S307.

[0083] On the other hand, if the user terminal 10 determines that no video has been selected (step S306: NO), the process of step S306 is repeated.

[0084] In step S307, the user terminal 10 transmits a video data delivery request to the management server 20. The delivery request includes the video ID of the video selected from the video list.

[0085] In step S308, the management server 20 acquires the video data from the storage 204 based on the distribution request received from the user terminal 10, and distributes the video data to the user terminal 10.

[0086] In step S309, the user terminal 10 plays the video received from the management server 20. In the example of Fig. 16, since a thumbnail image of the video of the surveillance camera 30 with a camera ID of "C1" has been selected, the video MV is enlarged and displayed in the display field for the selected video located below the video list LS2.

[0087] In step S310, the user terminal 10 determines whether the video being played has been dragged to a camera icon on the map. If the user terminal 10 determines that the video being played has been dragged to a camera icon on the map (step S310: YES), the process proceeds to step S311.

[0088] On the other hand, if the user terminal 10 determines that the video being played back has not been dragged to the camera icon on the map (step S310: NO), the process returns to step S306.

[0089] In step S311, the user terminal 10 acquires information on the display frame of the video during the drag operation and information on the position coordinates of the mouse pointer within the display frame during the drag operation. The position coordinates of the mouse pointer indicate the position of the user's face within the video.

[0090] In step S312, the user terminal 10 identifies the search target (drag destination) surveillance camera 30. In the example of Fig. 16, the surveillance camera 30 with the camera ID "C3" is identified.

[0091] Next, the user terminal 10 acquires a search period (step S313). In the example of Fig. 16, "2019 / 12 / 31 12:00" to "2019 / 12 / 31 14:00" is acquired as the search period.

[0092] In step S314, the user terminal 10 transmits a search request for the target person to the management server 20. The search request includes information such as the camera ID of the surveillance camera 30, the video ID, the display frame, and the position coordinates of the target person.

[0093] When a face detection frame that detects the face of each person is superimposed and displayed in a specifiable state in the video distributed from the management server 20, the ID of the face detection frame of the target person specified by the user may be included in the search request. In this case, the information on the position coordinates of the target person can be omitted.

[0094] In step S315, the management server 20 acquires a display frame at the time of the drag operation from the video data corresponding to the video ID based on the information included in the search request. Next, the management server 20 detects a face image displayed at the position coordinates of the target person from the display frame (step S316).

[0095] If no person's face exists at the position coordinates in the video specified by the user terminal 10, the management server 20 preferably returns error information to the user terminal 10. However, if only one person exists in the video, the face image of the person can be detected without considering the position coordinates.

[0096] In step S317, the management server 20 calculates the facial feature amount of the detected face image. If the person detection process has already been executed in the selected video, the facial feature amount of the target person can be acquired from the detection history information DB 24.

[0097] In step S318, the management server 20 searches the detection history information DB 24 for detection history information related to the search target (drag destination) surveillance camera 30 based on the calculated facial feature amount. Specifically, the management server 20 sequentially matches the facial feature amount calculated from the face image with the facial feature amount of the detected person included in the detection history information, and extracts people whose similarity (matching score) is equal to or greater than a predetermined threshold.

[0098] In this embodiment, when the matching score calculated by the matching process is equal to or greater than a predetermined threshold, the result of the matching process is deemed to be a match, and when the matching score is less than the predetermined threshold, the result of the matching process is deemed to be a mismatch.

[0099] In step S319, the management server 20 transmits the search results to the user terminal 10. The search results include, for example, information for reading out the surveillance video in which the target person appears, information on the shooting date and time of the target person, and the like.

[0100] In step S320, the user terminal 10 displays the search results received from the management server 20 on the display device 106. Fig. 17 is a diagram showing an example of a search result relating to the second search method. In Fig. 17, in tab T2 on the search screen, the search execution conditions specified by the user (target person's face image (IMG-2) / search target camera (C3) / search target period) and the search result (video / shooting date and time (2019 / 12 / 31 13:15:20 - 2019 / 12 / 31 13:15:29) / matching score (96.3)) R2 are displayed.

[0101] (Third search method) 18 is a sequence diagram showing an example of the processing of the person search system 1. This processing relates to a third search method that is executed by the user specifying a target person from an image file arbitrarily specified by the user.

[0102] In step S401, the user terminal 10 displays a search screen on the display device 106. Fig. 19 is a diagram showing an example of the search screen relating to the third search method. Here, the "Image File" tab T3 is selected in the right-hand region of the search screen. In the tab T3, an image IMG-3 of a file loaded by a user operation is displayed. The image files that can be loaded are preferably still image files and moving image files.

[0103] The dashed arrow A3 in Figure 19 indicates that the user uses the mouse pointer P to designate the facial portion of the target person P-4 contained in the image IMG-3 loaded onto the search screen, and performs a drag operation to superimpose the facial portion of the target person P-4 on the camera icon C2 displayed on the map M1.

[0104] In step S402, the user terminal 10 reads the image file designated by the user operation, and displays the image on the search screen.

[0105] In step S403, the user terminal 10 determines whether or not the image of the person being displayed has been dragged to the camera icon on the map. If the user terminal 10 determines that the image has been dragged to the camera icon on the map (step S403: YES), the process proceeds to step S404.

[0106] On the other hand, if the user terminal 10 determines that the image has not been dragged to the camera icon on the map (step S403: NO), the process of step S403 is repeated.

[0107] In step S404, the user terminal 10 identifies the destination of the image file, that is, the surveillance camera 30 corresponding to the camera icon to be searched, and acquires the camera ID. In the example of Fig. 19, the camera ID of the surveillance camera 30 corresponding to the camera icon of the destination is "C2".

[0108] In step S405, the user terminal 10 acquires the position coordinates of the mouse pointer during the drag operation within the image. Next, the user terminal 10 acquires the search period designated by the user on the search screen (step S406).

[0109] In step S407, the user terminal 10 transmits a search request for the target person to the management server 20. The search request includes information such as the loaded image file, the position coordinates of the face specified in the image, the search target period, and the camera ID of the surveillance camera 30 that is the drag destination.

[0110] In step S408, the management server 20 detects a facial image of a target person from among the images in the image file based on the position coordinates of the face included in the search request. In the example of Fig. 19, the management server 20 detects a facial image of a target person P-4 from among the image IMG-3.

[0111] If no person's face exists at the position coordinates in the image specified by the user terminal 10, the management server 20 preferably returns error information to the user terminal 10. However, if only one person exists in the image, the face image of the person can be detected without considering the position coordinates.

[0112] Next, the management server 20 calculates a facial feature amount from the facial image of the detected target person (step S409).

[0113] In step S410, the management server 20 searches for detection history information related to the drag destination surveillance camera 30 based on the calculated facial feature amount. Specifically, the management server 20 sequentially compares the facial feature amount calculated from the face image with the facial feature amount of the detected person included in the detection history information, and extracts people whose similarity is equal to or greater than a predetermined threshold value.

[0114] In step S411, the management server 20 transmits the search results to the user terminal 10. The search results include, for example, information for reading out the surveillance video in which the target person appears, information on the shooting date and time of the target person, and the like.

[0115] In step S412, the user terminal 10 displays the search results on the display device 106. Fig. 20 is a diagram showing an example of a search result relating to the third search method. In Fig. 20, in tab T3 on the search screen, the search execution conditions specified by the user (target person's face image (IMG-4) / search target camera (C2) / search target period) and the search results (video / shooting date and time (2019 / 12 / 31 13:15:20 - 2019 / 12 / 31 13:15:30) / matching score (93.4)) R3 are displayed.

[0116] (4th search method) 21 is a sequence diagram showing an example of the processing of the person search system 1. This processing relates to a fourth search method executed by a user specifying a target person from among the images of the search results. The fourth search method is a search method in which a user specifies an arbitrary person from among the images of the search results obtained by executing any one of the first to third search methods described above.

[0117] In step S501, the user terminal 10 reproduces the video of the search result received from the management server 20 on the display device 106.

[0118] 22 is a diagram showing an example of a search screen according to the fourth search method. Here, the previous search result, video MV-2, is superimposed on the search screen. The target person of face image IMG-6 included in the search execution conditions is the person with the symbol P-6 in video MV-2.

[0119] The dashed arrow A4 in Figure 22 indicates that the user uses the mouse pointer P to designate a person P-7, who is different from the target person P-6 in the video MV-2, as a new target person, and performs a drag operation to superimpose the video MV-2 on the camera icon C13 relating to another surveillance camera 30.

[0120] In step S502, the user terminal 10 determines whether the video being played has been dragged to a camera icon on the map. If the user terminal 10 determines that the video being played has been dragged to a camera icon on the map (step S502: YES), the process proceeds to step S503.

[0121] On the other hand, if the user terminal 10 determines that the video being played back has not been dragged to the camera icon on the map (step S502: NO), the process of step S502 is repeated.

[0122] In step S503, the user terminal 10 acquires information about the display frame of the video at the time of the drag operation and the position coordinates of the designated person within the display frame.

[0123] In step S504, the user terminal 10 specifies the search target (drag destination) surveillance camera 30. In the example of Fig. 22, the camera ID of the surveillance camera 30 corresponding to the drag destination camera icon is "C13".

[0124] Next, the user terminal 10 acquires a search period (step S505). In the example of Fig. 22, "2019 / 12 / 31 12:00" to "2019 / 12 / 31 14:00" is specified as the search period.

[0125] In step S506, the user terminal 10 transmits a search request for the specified person to the management server 20. The search request includes information such as the video ID of the drag source video, the display frame, the position coordinates of the specified person's face, and the camera ID of the surveillance camera 30 at the drag destination (search target).

[0126] In step S507, the management server 20 acquires a display frame during a drag operation based on the information included in the search request.

[0127] In step S508, the management server 20 detects a face image of the designated person present at the designated position coordinates from within the display frame. In the example of Fig. 22, the management server 20 detects a face image of the target person P-7 from within the display frame at the time of the drag operation.

[0128] If no person's face exists at the position coordinates specified on the user terminal 10 side, the management server 20 preferably returns error information to the user terminal 10. However, if only one person exists in the video, the face image of the person may be automatically detected without taking into account the specified position coordinates.

[0129] In step S509, the management server 20 calculates the facial feature amount of the detected face image. If the person detection process has already been executed in the selected video, the facial feature amount of the specified person can be acquired from the detection history information DB 24.

[0130] In step S510, the management server 20 searches the detection history information DB 24 for detection history information related to the drag destination surveillance camera 30 based on the calculated facial feature amount. Specifically, the management server 20 sequentially compares the facial feature amount calculated from the face image of the specified person with the facial feature amount of the detected person included in the detection history information, and extracts people whose similarity is equal to or greater than a predetermined threshold.

[0131] In step S511, the management server 20 transmits the search results to the user terminal 10. The search results include, for example, information for reading out the surveillance video in which the specified person appears, information on the shooting date and time of the target person, and the like.

[0132] In step S512, the user terminal 10 displays the search results received from the management server 20 on the display device 106. Fig. 23 is a diagram showing an example of a search result relating to the fourth search method. In Fig. 23, in tab T2 on the search screen, the search execution conditions specified by the user (target person's face image (IMG-7) / search target camera (C13) / search target period) and the search result (video / shooting date and time (2019 / 12 / 31 13:15:20 - 2019 / 12 / 31 13:15:30) / matching score (95.0)) R4 are displayed.

[0133] According to this embodiment, a user can search for a target person in the surveillance video captured by the surveillance camera 30 corresponding to the camera icon on the map by performing a drag operation to superimpose the face of the target person displayed on the screen (or the entire image showing the target person) onto a camera icon on the map. In other words, the user can easily search for a target person in the surveillance video captured by the desired surveillance camera 30.

[0134] [Second embodiment] This embodiment differs from the first embodiment in that the face image of a single target person designated on the screen by the user is collated collectively with images from multiple surveillance cameras 30 designated on a map.

[0135] Fig. 24 is a diagram showing an example of a search screen in this embodiment. The search target area shown on map M2 is larger than the search target area on map M1 in Fig. 10 and the like. The scale of map M2 is also different from that of map M1. Therefore, adjacent camera icons are displayed overlapping each other on a part of map M2.

[0136] The dashed arrow A5 in Figure 24 indicates that the user uses the mouse pointer P to specify the face of the target person P-8 in image IMG-8 loaded onto the search screen, and performs a drag operation to superimpose image IMG-8 on multiple camera icons superimposed on map M2.

[0137] Note that the method of designating multiple camera icons at once is not limited to this. For example, the mouse pointer P may be dragged over multiple camera icons to overlap them. In other words, multiple camera icons can be designated even if they are not displayed in a superimposed manner.

[0138] FIG. 25 is a diagram showing an example of a search result in this embodiment. In FIG. 25, the search execution conditions (target person's face image (IMG-8) / search target camera (C7 / C8 / C9) / search target period) specified by the user and the search result (video / shooting date and time / shooting camera (C7 / C8 / C9) / matching score (95.3 / 94.6 / 97.8)) R5 are displayed in the tab T3 of the search screen. By referring to the search result R5, the user can see that the target person was captured on all of the surveillance cameras 30 with the camera IDs "C7", "C8", and "C9". In addition, in the search result R5, the video is displayed in chronological order, so the user can easily understand the movement line of the target person.

[0139] According to this embodiment, the user does not need to specify the camera icons one by one related to the surveillance cameras 30 to be searched. Since the user can request the process of matching the facial image of the target person with the images from the multiple surveillance cameras 30 all at once, the user can perform the search work more efficiently.

[0140] [Third embodiment] This embodiment differs from the above-described embodiments in that the user can request a search process for a target person by dragging a camera icon designated by the user on a map onto the facial image of the target person displayed on the same screen.

[0141] Fig. 26 is a diagram showing an example of a search screen in this embodiment. A dashed arrow A6 in Fig. 26 indicates that a user drags a camera icon designated on a map to a face image displayed in the detected person list LS1. That is, the direction of the user's drag operation in Fig. 26 is opposite to that in Fig. 11.

[0142] In this embodiment, even if the user performs a drag operation in the opposite direction to that in the first embodiment, the same search process as in the first embodiment is executed. As a result, according to this embodiment, the same effects as in the first embodiment are achieved.

[0143] [Fourth embodiment] This embodiment differs from the above-described embodiments in that the user can perform a drag operation to superimpose a plurality of camera icons designated on a map onto the face image of a target person being displayed on the same screen.

[0144] Fig. 27 is a diagram showing an example of a search screen in this embodiment. The dashed arrow A7 in Fig. 27 indicates that the user drags multiple camera icons designated on the map M1 together onto one face image displayed in the detected person list LS1. That is, the direction of the user's drag operation in Fig. 27 is opposite to that in Fig. 19, Fig. 24, etc.

[0145] In this embodiment, even if the user performs a drag operation in the opposite direction to that in the first and second embodiments, a search process common to the above-mentioned embodiments is executed. As a result, according to this embodiment, the same effects as those of the other embodiments are achieved.

[0146] [Fifth embodiment] This embodiment differs from the above-described embodiments in that a fifth search method can be executed, in which a drag operation is performed to cause a camera icon selected on a map to overlap another camera icon.

[0147] Furthermore, in this embodiment, a sixth search method can also be executed in which a drag operation is performed to superimpose a part or all of the search results obtained by executing the fifth search method on a camera icon on a map.

[0148] (Fifth search method) 28 is a sequence diagram showing an example of the processing of the person search system 1 in this embodiment. This processing relates to the fifth search method.

[0149] In step S601, the user terminal 10 displays a search screen on the display device 106. Fig. 29 is a diagram showing an example of the search screen relating to the fifth search method. Here, a check box CB is included in the display field for search options. In this embodiment, the user switches the search method by turning the check box CB on and off.

[0150] Specifically, when the check box CB is OFF, the search targets common characters that appear in both surveillance cameras 30. On the other hand, when the check box is ON, the search targets combinations of multiple people who appear at the same time (together) among common characters.

[0151] In step S602, the user terminal 10 determines whether or not the selected camera icon on the map has been dragged to another camera icon. If the user terminal 10 determines that the selected camera icon has been dragged to another camera icon (step S602: YES), the process proceeds to step S603.

[0152] On the other hand, if the user terminal 10 determines that the selected camera icon has not been dragged to another camera icon (step S602: NO), the process of step S602 is repeated.

[0153] In step S603, the user terminal 10 identifies the surveillance cameras 30 corresponding to the camera icons of the drag source and the drag destination. In the example of Fig. 29, the camera ID of the surveillance camera 30 corresponding to the camera icon C1 of the drag source is "C1". And the camera ID of the surveillance camera 30 corresponding to the camera icon C2 of the drag destination is "C2".

[0154] In step S604, the user terminal 10 acquires the specification information of the person to be searched for specified on the search screen. Next, the user terminal 10 acquires the search period specified on the search screen by the user operation (step S605).

[0155] In step S606, the user terminal 10 transmits a search request for common characters to the management server 20. The search request includes information such as the camera IDs of the two surveillance cameras 30 at the drag source and drag destination, the search target period, and specified information for the search target.

[0156] In step S607, the management server 20 determines whether the search target is a "common character." If the management server 20 determines that the search target is a "common character" (step S607: YES), the process proceeds to step S608.

[0157] On the other hand, if the management server 20 determines that the search target is not a "combination of common characters" but a "common character combination" (step S607: NO), the process proceeds to step S609.

[0158] In step S608, the management server 20 searches for people (common characters) that appear in both the surveillance cameras 30 at the drag source and the drag destination based on the search conditions included in the search request.

[0159] On the other hand, in step S609, the management server 20 searches for a combination of people (common characters) who are both captured on both the surveillance cameras 30 at the drag source and drag destination at the same time, based on the search conditions included in the search request.

[0160] In step S610, the management server 20 transmits the search results to the user terminal 10. The search results include, for example, information for reading out the surveillance video in which the common characters appear, information on the shooting date and time of the common characters, and the like.

[0161] In step S611, the user terminal 10 displays the search results on the display device 106. Figures 30 and 31 are diagrams showing examples of search results according to the fifth search method.

[0162] In Fig. 30, the search execution conditions include not performing "combination detection" of common characters. That is, in the example of Fig. 29 described above, this corresponds to the case where the check box CB is set to OFF. Therefore, as the search result R6, list information of people who appeared in both of the two surveillance cameras 30 (camera ID: C1 / C2) that were the subject of the search is displayed.

[0163] On the other hand, in Fig. 31, "combination detection" of common characters is included in the search execution conditions. Therefore, the search result R7 displays list information of combinations of people who appeared together in both of the two surveillance cameras 30 (camera ID: C1 / C2) that are the search targets. The images included in the search result R7 may be extracted from only the image scenes in which the people appear at the same time.

[0164] (6th search method) 32 is a sequence diagram showing an example of the processing of the person search system 1 in this embodiment. This processing relates to the sixth search method and can be executed after the processing in FIG.

[0165] In step S612, the user terminal 10 determines whether the search result has been dragged to the camera icon displayed on the map. If the user terminal 10 determines that the search result has been dragged to the camera icon (step S612: YES), the process proceeds to step S613.

[0166] On the other hand, if the user terminal 10 determines that the search result has not been dragged to the camera icon (step S612: NO), the processing in FIG. 32 ends.

[0167] In step S613, the user terminal 10 identifies the camera ID of the surveillance camera 30 to be newly searched based on the user's operation. Fig. 33 is a diagram showing an example of a search screen related to the sixth search method. The dashed arrow A8 in Fig. 33 indicates that the user specifies a part of the search results of a previously executed search process with the mouse pointer P and performs a drag operation to superimpose the search results on a camera icon on the map.

[0168] 33, the combination of people at the top of the list information of the search result R7 is specified. The management server 20 searches for the video of the surveillance camera 30 related to the dragged camera icon C3 based on the combination of face images of the specified people.

[0169] In step S614, the user terminal 10 transmits a search request for common characters to the management server 20. The search request includes information such as identification information for identifying the search results of a previously executed search process, the search target period, the camera ID of the surveillance camera 30 to be searched, and designation information of the person to be searched.

[0170] In step S615, the management server 20 searches the detection history information DB 24 for common characters included in the search results who are also captured on the identified surveillance camera 30, based on the search conditions included in the search request. Here, the people to be searched for differ depending on the specified information on the search screen.

[0171] For example, when the search target is "common characters," people who are also captured by the third camera among the common characters captured by both the first and second cameras are searched for.

[0172] On the other hand, when the search target is a "combination of common characters," a search is performed for combinations of common characters who are also captured simultaneously by the third camera among combinations of common characters who are captured simultaneously by the first and second cameras.

[0173] Fig. 34 is a diagram showing an example of a search result related to the sixth search method. Search result R8 indicates that the combination of people specified in Fig. 33 was extracted from the video of the surveillance camera 30 (camera ID: C3) that is the drag destination (search target).

[0174] According to this embodiment, the user can perform a drag operation to overlap camera icons on the map, thereby obtaining list information of people who are commonly captured on multiple surveillance cameras 30, and list information of combinations of people who are commonly captured at the same time on multiple surveillance cameras 30. This allows the user to efficiently narrow down the target person and the people related to the target person.

[0175] Furthermore, according to this embodiment, a drag operation can be performed to superimpose the search result itself on a camera icon on the map, so that the results of a previously executed search process can be further narrowed down.

[0176] [Sixth embodiment] This embodiment differs from the above-mentioned embodiment in that the time required for the target person to travel is estimated from the distance between the surveillance camera 30 that captured the target person (hereinafter referred to as the "first camera") and another surveillance camera 30 (hereinafter referred to as the "second camera") corresponding to the dragged camera icon, and the search target period is automatically determined based on the shooting time and the required time in the first camera. Note that the "required time" is the time required for the target person to travel from the location of the first camera to the location of the second camera.

[0177] 35 is a flowchart showing an example of the processing of the person search system 1 in this embodiment. This processing can be executed between step S209 and step S210 shown in FIG.

[0178] In step S701, the user terminal 10 acquires the capture time of the display frame that was displayed on the screen when the face image of the target person was dragged.

[0179] In step S702, the user terminal 10 acquires the distance between the surveillance camera 30 that captured the target person and the surveillance camera 30 that corresponds to the dragged camera icon. The distance between the two surveillance cameras 30 can be calculated, for example, from the position information stored in the camera information DB 23. Also, if there is a table that records the distance between the cameras, the distance can be acquired by referring to the table based on the two camera IDs.

[0180] In step S703, the user terminal 10 estimates the range of the required time based on the predetermined speed and the distance between the cameras. As the "predetermined speed", it is preferable to set the range of the average human walking speed, for example, 3 to 5 km / sec.

[0181] In step S704, the user terminal 10 determines the search target period in the drag destination surveillance camera 30 based on the shooting time and the range of the required time. After that, the process proceeds to step S210 shown in FIG.

[0182] According to this embodiment, based on the distance between the surveillance camera 30 that captured the video of the specified target person and the surveillance camera 30 that corresponds to the camera icon of the drag destination of the target person's face image, the search target period of the drag destination surveillance camera 30 can be automatically narrowed down. This improves the speed of the search process in the person search system 1.

[0183] [Seventh embodiment] This embodiment differs from the sixth embodiment described above in that the video used to specify the target person is analyzed to calculate the movement speed of the target person, and the search period for the surveillance cameras 30 to be searched is automatically determined based on the movement speed and the distance between the two surveillance cameras 30.

[0184] 36 is a flowchart showing an example of the processing of the person search system 1 in this embodiment. This processing can be executed between step S312 and step S314 shown in FIG.

[0185] In step S801, the user terminal 10 acquires the capture time of the display frame that was displayed on the screen when the face image of the target person was dragged.

[0186] In step S802, the user terminal 10 calculates the moving speed of the target person in the video. For example, the moving speed of the target person is calculated based on the length of time that the target person is actually captured by the surveillance camera 30 and the moving distance of the target person obtained from the video information (e.g., magnification, shooting angle, etc.).

[0187] In step S803, the user terminal 10 acquires the distance between the surveillance camera 30 that captured the image of the target person and the surveillance camera 30 that corresponds to the dragged camera icon.

[0188] In step S804, the user terminal 10 estimates the range of the required time based on the moving speed and the distance between the cameras. For example, a case will be described where the moving speed of the target person calculated in step S802 is 4.5 km / sec and the distance between the cameras is 0.4 km. In this case, by assuming that the moving speed of the target person ranges from 4 to 5 km / sec, the range of the required time can be calculated as 4.8 minutes (=(0.4 / 5)×60) to 6 minutes (=(0.4 / 4)×60).

[0189] In step S805, the user terminal 10 determines the search target period in the drag destination surveillance camera 30 based on the shooting time and the range of the required time. After that, the process proceeds to step S314 shown in FIG.

[0190] According to this embodiment, it is possible to automatically narrow down the search target period in the drag destination surveillance camera 30 based on the moving speed of the target person obtained by video analysis and the distance between the two surveillance cameras 30. This improves the speed of the search process in the person search system 1.

[0191] [Eighth embodiment] FIG. 37 is a block diagram showing the configuration of the information processing device 100 in this embodiment. The information processing device 100 includes a display control unit 100A, an operation receiving unit 100B, and a processing request unit 100C. The display control unit 100A displays a map of a search target area and a plurality of camera icons indicating the installation positions of the surveillance cameras on the map on the same screen. The operation receiving unit 100B receives an operation of superimposing a first camera icon on a second camera icon among the plurality of camera icons. Based on the operation, the processing request unit 100C requests a search process for a common appearance that appears in both the first surveillance video of the surveillance camera related to the first camera icon and the second surveillance video taken by the surveillance camera related to the second camera icon. According to this embodiment, a user can easily search for a target person from the surveillance video of a desired surveillance camera.

[0192] [Modified embodiment] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention without departing from the gist of the present invention. For example, it should be understood that an embodiment in which a part of the configuration of any embodiment is added to another embodiment, or an embodiment in which a part of the configuration of another embodiment is replaced with a part of the configuration of another embodiment, is also an embodiment to which the present invention can be applied.

[0193] In the above embodiment, a face image is used as biometric information, but the biometric information used in the present invention is not limited to a face image. For example, the management server 20 may perform a matching process using other biometric information such as an iris image or an auricle image instead of a face image. The management server 20 may also perform a matching process based on a combination of different types of biometric information.

[0194] In addition to facial images, when searching for a person from a video, information on the person's belongings may be detected and stored in the database in association with the facial image. Specific examples of the belongings include clothing, hats, shoes, bags, glasses, and various accessories. Similarly, the age and gender of a person may be estimated by video analysis and stored in the database in association with the facial image. The user terminal 10 may be configured to request the management server 20 to perform a matching process based on a combination of the facial image (person image) of the target person and the specified information. In this case, the user can search for the target person by specifying at least one of the information on the person's belongings and the attribute information such as the target person's age and gender on the screen.

[0195] The configuration of the present invention can also be applied to searching for items. For example, attribute information such as the license plate, shape, size, and color of a vehicle may be detected from the image captured by the surveillance camera 30 using image analysis technology, and the attribute information may be stored in a database. This allows a user to easily search for a target vehicle from the image captured by the desired surveillance camera 30.

[0196] Also, the video captured by each surveillance camera 30 may be analyzed, and the characteristics of each person, such as the gait, obtained by the analysis may be stored in a database. When a target person is specified in the displayed video by a user operation, the target person's gait is analyzed, and the target person can be searched for by comparing the analysis result with the gait of the detected person in the database. In this way, the use of gait authentication technology has the advantage that the target person can be searched for even if the target person is wearing a mask or sunglasses.

[0197] In the above-described first embodiment, the case where the user selects one face image from the detected person list has been described. However, the user terminal 10 may further include a function that enables the user to perform a drag operation to superimpose multiple face images selected by the user from the detected person list onto one camera icon.

[0198] When multiple target persons specified by user operation are captured simultaneously on the drag destination surveillance camera 30, the management server 20 can extract the desired video scene and output it to the user terminal 10. For example, when person X and person Y are specified, a video in which the two persons are captured simultaneously can be searched for.

[0199] Similarly, if at least one of multiple target persons specified by user operation is captured on the drag destination surveillance camera 30, the management server 20 can extract video scenes relating to that person and output them to the user terminal 10.

[0200] In the above-described first embodiment, a configuration has been described in which the user terminal 10 presents the detected person list distributed from the management server 20 to the user. The management server 20 may further include a configuration for automatically selecting face images suitable for search (face matching) from the surveillance video and outputting them to the user terminal 10 as, for example, a detected person list. Specifically, it is preferable to select a face image in which the detected person faces the front toward the surveillance camera 30 and include it in the detected person list. This can improve the accuracy of the subsequent matching process performed on the target person.

[0201] In the above embodiment, a case has been described in which images of the surveillance cameras 30 arranged in an outdoor search target area are collected and a target person is searched for from the images. However, the search target area is not limited to outdoors. For example, indoor facilities such as transportation facilities and commercial facilities can be set as the search target area. In this case, the search target area may be displayed as a guide map for each floor. For example, when the present invention is applied to tracking a person in a building having multiple floors, the positional relationship between the target person and the surveillance cameras 30 and the guide map may be displayed not only in two dimensions but also in three dimensions on the search screen or the like.

[0202] In the above embodiment, the management server 20 is provided with a database such as the video information DB 22, and the collation process and search process are performed within the server. However, each of the surveillance cameras 30 may be configured to perform part of the process of the management server 20.

[0203] FIG. 38 is a block diagram showing an example of the overall configuration of a person search system 2 in a modified embodiment. Here, a user terminal 10, a management server 20, and a plurality of monitoring cameras 30 are connected via a network NW3 such as the Internet. Each of the plurality of monitoring cameras 30 includes a search program 31 and an in-camera DB 32. The search program 31 is a program that searches for a person matching a face image specified by the user terminal 10 from among images captured by the camera in response to a search request from the user terminal 10. The in-camera DB 32 is a database that stores data synchronized with, for example, the video information DB 22 and the detection history information DB 24 of the management server 20. The in-camera DB 32 stores and accumulates images captured by the camera.

[0204] When a user performs an operation of dragging a person image or a face image onto a camera icon on a map, the user terminal 10 transmits a search request to the surveillance camera 30 corresponding to the camera icon directly or indirectly via the management server 20. The search request is search instruction information including the person image etc. dragged by the user. When the surveillance camera 30 receives the search request from the user terminal 10, it executes a search process on the in-camera DB 32 and returns the search results to the user terminal 10. In this case, since the multiple surveillance cameras 30 execute the search process in a distributed manner, there is an advantage that the processing load on the management server 20 can be reduced.

[0205] In the above-described embodiments, the case where a camera icon or the like is designated by a drag operation using a mouse pointer has been described, but the drag operation using a mouse pointer is merely one example of a designation operation. For example, a configuration may be adopted in which a user can perform a designation operation by swiping or tapping with a finger instead of using a mouse pointer.

[0206] In the above-mentioned fifth embodiment, for example, in order to search for a person who appears in both the first and second cameras (a common character), a first camera icon corresponding to the first camera is dragged to a second camera icon corresponding to the second camera. However, the method of designating multiple camera icons is not limited to this.

[0207] Fig. 39 and Fig. 40 are diagrams showing an example of a search screen in a modified embodiment. The dashed arrow A9 in Fig. 39 indicates that the user performs an operation of surrounding the periphery with the locus of the mouse pointer P so as to include the two camera icons in order to specify the camera icon C1 (first camera icon) and the camera icon C2 (second camera icon). The dashed arrow A10 indicates that the user performs an operation of dragging the camera icon group (or the area surrounding the camera icon group) specified by the operation of the dashed arrow A9 onto a face image in the detected person list LS1.

[0208] The dashed arrow A11 in FIG. 40 indicates an operation in which the user specifies the surveillance camera 30 to be searched by tracing the periphery of multiple camera icons (C1, C2, C4, C7 to C9) with a finger F. The dashed arrow A12 indicates an operation in which the user drags a face image in the detected person list LS1 to the group of camera icons specified by the operation of the dashed arrow A11. Similarly, the user may specify the surveillance camera 30 to be searched by clicking multiple camera icons with the mouse pointer P or tapping with a finger F, and then drag the person image to the group of camera icons (or the area near the group of camera icons). In either case, as in the case of the fifth embodiment, it is possible to search for a person who is commonly captured in all of the surveillance cameras 30 to be searched. The number of camera icons specified as the search targets is not limited, and may be three or more.

[0209] The scope of each embodiment also includes a processing method in which a program that operates the configuration of the embodiment to realize the functions of the above-mentioned embodiment is recorded on a recording medium, the program recorded on the recording medium is read as code, and the program is executed on a computer. That is, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, not only the recording medium on which the above-mentioned program is recorded, but also the program itself is included in each embodiment.

[0210] Examples of the recording medium that can be used include a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a non-volatile memory card, etc. Furthermore, the scope of each embodiment is not limited to those that execute processing by a single program recorded on the recording medium, but also includes those that execute processing by operating on an OS in cooperation with other software and functions of an expansion board.

[0211] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0212] (Appendix 1) a display control unit that displays a map of a search target area and a plurality of camera icons indicating installation positions of surveillance cameras on the map on the same screen; an operation receiving unit that receives an operation of superimposing a first camera icon on a second camera icon among the plurality of camera icons; a processing request unit that requests, based on the operation, a search process for a common character that appears in both a first surveillance video of the surveillance camera associated with the first camera icon and a second surveillance video taken by the surveillance camera associated with the second camera icon; and An information processing device comprising:

[0213] (Appendix 2) the processing request unit requests a search process for a combination of the common characters appearing simultaneously in both the first surveillance video and the second surveillance video, 2. An information processing device according to claim 1.

[0214] (Appendix 3) the display control unit displays a search result of the search process on the screen; the operation reception unit receives an operation to superimpose a display area of ​​the search result on a third camera icon displayed on the map, the processing request unit requests a second search process for narrowing down the common characters included in the search result to only those people who also appear in the third surveillance video of the surveillance camera related to the third camera icon. 3. The information processing device according to claim 1 or 2.

[0215] (Appendix 4) The display control unit displays an extracted video obtained by extracting a scene in which the common character appears from each of the first surveillance video and the second surveillance video. 3. The information processing device according to claim 1 or 2.

[0216] (Appendix 5) the operation reception unit receives an operation of superimposing a designated person image related to a designated person designated in the extracted video on the camera icon on the map; the processing request unit requests execution of a matching process between the designated person image and a surveillance video captured by the surveillance camera indicated by the camera icon; 5. The information processing device according to claim 4.

[0217] (Appendix 6) the operation reception unit receives an operation of collectively superimposing a plurality of designated person images related to a designated person designated in the extracted video on one of the camera icons on the map; the processing request unit requests execution of a matching process between each of the plurality of specified person images and a surveillance video captured by the surveillance camera indicated by the camera icon; 5. The information processing device according to claim 4.

[0218] (Appendix 7) the operation reception unit receives an operation to collectively superimpose a designated person image related to a designated person designated in the extracted video onto the multiple camera icons superimposed and displayed on the map; the processing request unit requests execution of a matching process between the designated person image and surveillance video captured by the surveillance cameras associated with each of the plurality of camera icons; 5. The information processing device according to claim 4.

[0219] (Appendix 8) the display control unit changes a display format of the camera icon in accordance with a result of the matching process in each of the surveillance cameras. 8. The information processing device according to claim 7.

[0220] (Appendix 9) The display control unit causes a sign indicating a direction of an installation location of another surveillance camera different from the surveillance camera related to the extracted image to be superimposed on the map. 9. An information processing device according to any one of claims 4 to 8.

[0221] (Appendix 10) The display control unit displays identification information of the surveillance camera located in a direction in which the common character moves, superimposed on the extracted image. An information processing device according to any one of appendix 4 to 8.

[0222] (Appendix 11) The display control unit displays a list including a person image of the common character on the screen. 4. An information processing device according to any one of claims 1 to 3.

[0223] (Appendix 12) the operation reception unit receives an operation of superimposing the person image specified from the list on any one of the camera icons displayed on the map; the processing request unit requests execution of a matching process between the person image and surveillance video captured by the surveillance cameras associated with each of the plurality of camera icons; 12. The information processing device according to claim 11.

[0224] (Appendix 13) the operation receiving unit receives an operation of collectively superimposing the plurality of person images included in the list on one of the camera icons on the map; the processing request unit requests execution of a matching process between each of the plurality of person images and a surveillance video captured by the surveillance camera indicated by the camera icon; 12. The information processing device according to claim 11.

[0225] (Appendix 14) the operation reception unit receives an operation to collectively superimpose the person images included in the list on the plurality of camera icons superimposed and displayed on the map; the processing request unit requests execution of a matching process between the person image and surveillance video captured by the surveillance cameras associated with each of the plurality of camera icons; 12. The information processing device according to claim 11.

[0226] (Appendix 15) the display control unit changes a display format of the camera icon in accordance with a result of the matching process in each of the surveillance cameras. 15. The information processing device according to claim 14.

[0227] (Appendix 16) displaying, on the same screen, a map of a search target area and a plurality of camera icons indicating installation positions of surveillance cameras on the map; receiving an operation of superimposing at least one first camera icon among the plurality of camera icons on a second camera icon; A step of requesting a search process for a common character appearing in both a first surveillance video of the surveillance camera associated with the first camera icon and a second surveillance video captured by the surveillance camera associated with the second camera icon based on the operation; An information processing method comprising:

[0228] (Appendix 17) On the computer, displaying, on the same screen, a map of a search target area and a plurality of camera icons indicating installation positions of surveillance cameras on the map; receiving an operation of superimposing at least one first camera icon among the plurality of camera icons on a second camera icon; A step of requesting a search process for a common character appearing in both a first surveillance video of the surveillance camera associated with the first camera icon and a second surveillance video captured by the surveillance camera associated with the second camera icon based on the operation; A recording medium on which a program for executing the above is recorded. [Explanation of symbols]

[0229] NW1, NW2, NW3...Network 1,2... Person search system 10. User terminal 20. Management Server 21. Registered User Information DB 22. Video information database 23 Camera information DB 24. Detection history information DB 30. Surveillance Camera 31. Search Program 32. Camera internal database 100.... Information processing device 100A···Display control unit 100B Operation reception section 100C Processing request section

Claims

1. displaying a plurality of camera icons on a display device; A process of receiving an input surrounding the camera icon displayed on the display device; In response to receiving the input, a process of searching for a person based on an image captured by a camera corresponding to the camera icon is included; In the process of searching for a person, when a plurality of camera icons are surrounded by the input, one or more computers are caused to execute a process of searching for a common character that appears in images captured by the cameras corresponding to all of the surrounded camera icons; When a plurality of camera icons are surrounded by the input, this includes a case where three or more camera icons are surrounded.

2. A process of receiving an input for selecting a target for the search; a process of changing the target in the process of searching for the person in response to receiving an input for selecting the target; The program according to claim 1 , further comprising:

3. In response to receiving the input surrounding the plurality of camera icons, the plurality of camera icons are displayed on the display device in a manner different from other camera icons.

3. The program according to claim 1 or 2.

4. causing a computer to display a plurality of camera icons on a display device; receiving an input by the computer surrounding the camera icon displayed on the display device; and a step of performing a search for a person based on an image captured by a camera corresponding to the camera icon in response to receiving the input by the computer, In the step of searching for a person, when a plurality of camera icons are surrounded by the input, searching for a common character that appears in the images captured by the cameras corresponding to all the surrounded camera icons; In the case where a plurality of camera icons are surrounded by the input, this includes a case where three or more camera icons are surrounded.

5. receiving an input by the computer for selecting a target of the search; changing the target in the process of searching for the person in response to receiving an input for selecting the target by the computer; The information processing method according to claim 4 , further comprising:

6. In response to receiving the input surrounding the plurality of camera icons, the computer displays the plurality of camera icons on the display device in a manner different from other camera icons.

6. The information processing method according to claim 4 or 5.

7. A person search system in which a camera, an information processing terminal capable of specifying a person search condition in response to receiving an input surrounding a camera icon displayed on a display device, and a server are connected via a network, The server, means for receiving the search conditions for the person from the information processing terminal; means for searching for the person who satisfies the search condition; the person searching means, when a plurality of camera icons are surrounded by the input, searches for common characters appearing in the images captured by the cameras corresponding to all of the surrounded camera icons; In the case where a plurality of camera icons are surrounded by the input, this includes a case where three or more camera icons are surrounded.

Images