Search processing device, search processing method and program

Abstract

To easily grasp, in the case where a retrieval target is imaged by a plurality of cameras, by which camera and how many times the retrieval target is imaged.SOLUTION: A retrieval processing device comprises a tracking unit, a setting unit, a calculation unit and a display control unit. The tracking unit detects a mobile included in a video and makes the mobile correspondent in a time-sequence manner, thereby tracking the mobile. The setting unit receives setting of a retrieval target included in the video. The calculation unit calculates a first feature amount indicating features of the retrieval target and a second feature amount indicating features of the mobile and calculates first similarity of the first and second feature amounts. The display control unit receives a selection of a first display mode in which the mobile of the first similarity equal to or higher than a first threshold is displayed or a second display mode in which the mobiles are displayed in order from the highest first similarity for each camera capturing the video and displays a retrieval result corresponding to the retrieval target on a display unit in the selected display mode.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a search processing device, a search processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is a conventional technology that inputs a person image as a search query and sequentially displays similar people from among people captured by each camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5586736 [Patent Document 2] JP 2020-154478 A [Patent Document 3] JP 2020-154479 A [Patent Document 4] Patent No. 6659524 [Patent Document 5] Patent No. 6833617 [Patent Document 6] Patent No. 6649232 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, when a search target is captured on multiple cameras, it is not easy to know which camera captured the target and how many times it was captured. [Means for solving the problem]

[0005] A search processing device according to an embodiment includes a tracking unit, a setting unit, a calculation unit, and a display control unit. The tracking unit detects a moving object included in a video and tracks the moving object by associating the moving object in a time series. The setting unit accepts a setting of a search target included in the video. The calculation unit calculates a first feature amount indicating a feature of the search target and a second feature amount indicating a feature of the moving object, and calculates a first similarity between the first and second feature amounts. The display control unit accepts a selection of a first display mode in which the moving object having the first similarity equal to or greater than a first threshold is displayed, or a second display mode in which the moving object is displayed in descending order of the first similarity for each camera that captured the video, and displays a search result according to the search target on the display unit in the selected display mode. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is a diagram showing an example of a functional configuration of the search processing device according to the embodiment. [Diagram 2] FIG. 2 is a diagram showing Example 1 (first display mode) of display information according to the embodiment. [Diagram 3] FIG. 11 is a diagram showing a second example (second display mode) of display information according to the embodiment. [Figure 4] 1 is a flowchart showing an example of a search processing method according to an embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of the search processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a search processing device, a search processing method, and a program will be described in detail with reference to the accompanying drawings.

[0008] First, an example of the functional configuration of the search processing device according to the embodiment will be described.

[0009] [Example of functional configuration] 1 is a diagram illustrating an example of a functional configuration of a search processing device 100 according to an embodiment. The search processing device 100 according to the embodiment includes a tracking unit 1, a storage unit 2, a setting unit 3, a calculation unit 4, a display control unit 5, a display unit 6, and an optimization unit 7.

[0010] The tracking unit 1 uses information on each moving object acquired from the video captured by each camera to perform overall optimal association with the same moving object through collective processing. Specifically, the tracking unit 1 aggregates the video captured by each camera, estimates the same moving object contained in each video, and associates the moving objects collectively. Here, the input video information may be a stream or a frame image sequence.

[0011] The moving object is a person such as a pedestrian, but is not limited to this. The moving object may be any moving object such as a vehicle or an aircraft.

[0012] In this embodiment, the tracking unit 1 targets images captured by a group of surveillance cameras fixed at specific locations, detects people in each image, and performs tracking processing to match the same people between temporally consecutive frame images. The tracking unit 1 detects a moving object using, for example, a feature extraction model based on the appearance of the moving object (a person in this embodiment). For detecting and tracking people in the camera, methods described in, for example, Patent Documents 2 and 3 are used.

[0013] In the following description, the person tracking result is called a tracklet. The tracking unit 1 stores rectangular coordinate information indicating the position of the person in each frame and a cutout image of the rectangle or a range with a small margin in a storage unit 2 (e.g., a database). When a cutout image of a range with a small margin is stored, for example, accessories of the person (e.g., a walking stick and a wheelchair) are included, and the detection accuracy of the person can be further improved by using the accessories to detect the person.

[0014] The tracking unit 1 collects the coordinate information and the cut-out images for the number of frames in which the person was tracked (for which a tracklet was obtained), and stores them in the storage unit 2. Each tracklet also includes information for identifying the camera that captured the image.

[0015] The tracking unit 1 constantly processes the camera image, and sequentially registers the camera image and information such as the tacklet in the storage unit 2. The camera image and the tacklet may be registered when tracking ends and no additional information is added to the tracklet, or at a fixed timing. Examples of the timing when tracking ends and no additional information is added to the tracklet include when the target person moves out of the field of view of the camera, when person detection fails, or when matching fails.

[0016] When tracking is completed and the information is registered in the storage unit 2, the person who is currently being captured by the camera will not be included in the person search results. Also, when the tracklet is updated at regular intervals, the current position of the person can be searched for, although there is a delay for the update timing.

[0017] The setting unit 3 sets a person to be searched. For example, the setting unit 3 may set a person to be searched in response to a person designation by the user. The method of designating a person by the user may be arbitrary. For example, the setting unit 3 may accept the designation of a person to be searched for by a reception unit (not shown in FIG. 1) of a person on a video. The reception unit will be described below using a GUI (Graphical User Interface), but is not limited to the GUI and may be a CUI (Character User Interface) or a physical button such as a keyboard. Specifically, the setting unit 3 may accept the designation of a person to be searched for by a GUI that accepts a click on a person on a video. Also, for example, the setting unit 3 may accept the designation of a person to be searched for by accepting a selection of a tracklet corresponding to the person to be searched from display information including one or more tracklets stored in the storage unit 2. Specifically, when the designation of a person to be searched for is accepted from the outside by the reception unit, the reception unit is not limited to a GUI and may be a device such as a camera or a scanner.

[0018] Also, for example, the setting unit 3 may automatically set a person who matches a pre-specified condition, such as a specific action or behavior, as a person to be searched for by using an image recognition technique.

[0019] When the setting unit 3 receives a setting of a person to be searched for from a user, a search query for searching for the person is determined. A search using the search query causes a person identical to the person set by the setting unit 3 or a similar person to be displayed on the display unit 6. The setting unit 3 may receive settings such as a person size and a time period, and may mask the search results to be displayed. That is, the setting unit 3 may further receive at least one of a range setting of a size range of the search target and a date and time range in which the search target appears, and the display control unit 5 may not display search results that are not included in the range setting.

[0020] The calculation unit 4 calculates (extracts) features for determining the similarity between a person identified from the search query and a person included in a cut-out image stored in the memory unit 2. Any method may be used for extracting the person's features. For example, the methods described in Patent Documents 4 and 5 may be used to extract the person's features. In Patent Documents 4 and 5, the tracklet's person features are expressed as vectors or subspaces. The calculation unit 4 extracts features for identifying the same person from the appearance, such as the person's entire body.

[0021] The feature amount is extracted when a tracklet is registered or updated in the storage unit 2. The calculation unit 4 calculates the feature amount of the registered or updated tracklet.

[0022] Next, the calculation unit 4 calculates a similarity indicating the similarity between the person specified by the search query and the person included in the extracted image stored in the storage unit 2, using the feature amount.

[0023] The similarity is calculated, for example, by the distance between vectors, cosine similarity, etc. Furthermore, the calculation unit 4 normalizes the range of values ​​indicating the similarity, and calculates the similarity that can take a value between 0 and 1. When the feature amount is represented by a subspace, the similarity between the spaces is calculated by the mutual subspace method or the like in the same way as in the case of vectors.

[0024] Furthermore, the calculation unit 4 calculates the ranking of the similarity for each camera.

[0025] The calculation unit 4 may receive a designation of a target person from a user through a GUI that receives narrowing down of target persons, and calculate a similarity between some of the people stored in the storage unit 2 limited by the designation and a person specified by the search query. Specifically, the target person is narrowed down based on the time period in which the person was captured and the corresponding camera, etc.

[0026] The display control unit 5 controls the display of the display information displayed on the display unit 6. For example, the display information includes the similarity obtained by the calculation unit 4, a tracklet (search result) corresponding to the person with the similarity, and a ranking in the camera. The person in the search result is displayed, for example, by cropping the camera image. Also, for example, the search result is displayed as a video or animation for each person in the in-camera tracking result.

[0027] The display unit 6 displays display information. Examples of the display information will be described later with reference to FIGS.

[0028] The optimization unit 7 optimizes the feature extraction model by performing adaptive learning to adapt the feature extraction model based on the appearance of a moving object (a person in this embodiment) to the installation environment of the camera, using the tracking result obtained by the tracking unit 1. In the adaptive learning, learning is performed using, for example, images actually captured by a camera installed in the installation environment.

[0029] To calculate the feature amount, for example, a feature extractor is used that has learned feature expressions based on the appearance of a person, so that the same person appears closer and other people appear farther away. When targeting people photographed by multiple cameras with different shooting conditions (e.g., lighting and distance to the subject, etc.), the scale of the obtained similarity is different for each camera. The similarity of a person captured by a camera with a shooting environment similar to the search query tends to be slightly higher for both the person and other people, and conversely, the similarity of a person captured by a camera with a different shooting environment tends to be slightly lower.

[0030] Therefore, if the similarity of people in all cameras is treated uniformly and the results are displayed in order of similarity, many other people will be mixed in at the top, making it difficult to confirm in which camera the person being searched for has been captured and how many times. Therefore, the display control unit 5 selectively displays two display modes: a first display mode that displays people with a similarity equal to or greater than a predetermined threshold by camera, and a second display mode that displays search results in a specific camera in order of similarity.

[0031] <Example of the first display mode> 2 is a diagram showing an example 1 (first display mode) of display information according to the embodiment. The display information in the first display mode includes display areas 201-203.

[0032] The display area 201 includes a search query and a GUI for adjusting the threshold.

[0033] The display area 202 displays search results organized by camera in descending order of similarity to the search query. Information displayed in the display area 202 includes a person image (e.g., a cropped image cut out from an image), camera information indicating the camera number and position, the date and time the image was taken, etc. Since a Tracklet stores multiple time-sequential person images, the person images may be displayed as GIF animations. The time when the person was first detected is displayed around the cropped image.

[0034] The display area 203 displays location information (e.g., a drawing, a map, etc.) indicating the location of the cameras. For example, the display control unit 5 highlights, for example, on a map, cameras in which a certain threshold or more of people have been found (in the example of FIG. 2, the cameras with camera numbers 15 and 18).

[0035] The first display mode, which displays people with a similarity equal to or greater than a predetermined threshold by camera, is used, for example, to screen which camera the person in question was captured in. In the first display mode, people in each camera are displayed, for example, in chronological order. Note that information from cameras that do not capture the person in question (cameras that capture only people with a similarity less than a predetermined value) does not need to be included in the display information.

[0036] In addition, any threshold value can be selected within the range of values, and the search results displayed change depending on the threshold value. Therefore, the display control unit 5 arranges a GUI part that accepts the selection of a threshold value in the display information (display area 201 in the example of FIG. 2) so that the search results can be changed interactively.

[0037] In this embodiment, the threshold is actually a continuous value in the range of 0 to 1, but the display information may include a GUI part that accepts the selection of the threshold as a discretized value. The discretized value is, for example, a level display such as low, medium, high, etc.

[0038] When displaying the search results, the display control unit 5 may also display the similarity to the search query. The similarity may be expressed as a discretized numerical value, or may be displayed by displaying a bar indicating the level of similarity in gradational colors.

[0039] When a map is used as the location information, as long as the relative positions of the cameras are known, the displayed map may be a hand-drawn map, or the camera's field of view and optical axis direction may be represented on the map.

[0040] If each camera has been calibrated in advance and the position of a person can be mapped on a map, the position of the person may be displayed using an icon, etc. The icon may be, for example, a dot or a circle.

[0041] Here, an unintended other person (a person not identical to the search query) may be displayed as a search result, which may hinder the search for the person to be found. The setting unit 3 may further receive a setting of an excluded target to be excluded from the search target. This makes it possible to, for example, not display search results for a specific person. Specifically, the calculation unit 4 calculates a feature amount (third feature amount) indicating the feature of the excluded target and a feature amount (second feature amount) of a moving object included in the video, and calculates a feature amount (second similarity) of the second and third feature amounts. Then, in the first and second display modes, a moving object whose second similarity is equal to or greater than a second threshold value is not displayed.

[0042] For example, the display control unit 5 displays a menu in response to a right click on an image of a person in the search results, and interactively accepts a setting to not display the results (setting to be excluded) via the menu.

[0043] In this case, the display control unit 5 does not control the display of the tracklet to be excluded. However, since the calculation unit 4 has extracted features from the tracklets of all moving objects in advance, the similarity to the feature of the excluded target and the similarity order can be calculated. Therefore, when the setting unit 3 accepts the setting of a search query to search for the tracklet to be excluded, the display control unit 5 may also not display tracklets that are equal to or greater than the threshold for determining the excluded target (tracklets similar to the excluded target) based on the similarity to the feature of the excluded target indicated by the search query and the similarity order. In other words, the display control unit 5 may not display other people who are thought to be the same person as the person who has been set not to be displayed. Note that the threshold used for determining the similarity of the excluded target is set to be higher than the threshold used for determining the similarity of the display target, for example.

[0044] The calculation unit 4 may also calculate the attribute value of the tracklet in advance using a person attribute recognition technology, and the setting unit 3 may accept the designation of a specific attribute value as a search option. This makes it possible to display search results of only people wearing suits, or to exclude people in their 20s from the search results, for example. The person attribute recognition technology may be, for example, the technology disclosed in Patent Document 6.

[0045] The display control unit 5 may also display a GUI for accepting the selection (designation) of a new search query from the search results displayed on the display unit 6, and the setting unit 3 may set the search target by the search query selected by the GUI. This allows, for example, the user to select a person of interest from the search results as a new search query.

[0046] Next, a second display mode will be described, in which search results for a specific camera are displayed in order of similarity.

[0047] <Example of the second display mode> 3 is a diagram showing a second example (second display mode) of the display information according to the embodiment. The display information in the first display mode includes display areas 211-213.

[0048] The display area 211 includes a search query and identification information for identifying the target camera (in the example of FIG. 3, the camera with the camera number 15).

[0049] Display area 212 displays the search results for the target cameras in order of their similarity to the search query.

[0050] The display area 213 displays placement information (for example, a drawing, a map, etc.) indicating the placement of the cameras. For example, the display control unit 5 highlights the target camera on the map, for example.

[0051] The second display mode displays in detail a specific camera referring to the search results. The display information in the second display mode is displayed when a specific camera is selected by the user via a GUI that accepts the specification of a camera number, for example. The method of specifying the camera may be, for example, a method of specifying the camera ID when the camera ID is managed, or a method of specifying the camera position shown on a map with a mouse or the like. The number of cameras specified may be one or more.

[0052] The search results are displayed in order of similarity, but the number of search results to be displayed may be configurable. Since the number of search results that can be displayed on the screen is limited, the display control unit 5 may allow the results to be viewed using a UI such as a scroll bar if the number exceeds the limit, or the search results may be displayed in multiple lines at once.

[0053] When the display control unit 5 determines that the person in the search query is captured by the same camera multiple times, the display control unit 5 may display that fact on the display unit 6. Alternatively, the search processing device 100 may notify the user by outputting a sound and highlighting the person in question.

[0054] The search processing device 100 may also notify the user when conditions other than that the person in the search query is captured by the same camera multiple times are met. For example, the search processing device 100 may also notify the user when conditions such as a person moving from camera A to camera B, returning to camera A, and then moving to camera B are met. The display control unit 5 displays a GUI for setting such conditions on the display unit 6.

[0055] <Example of search processing method> 4 is a flowchart showing an example of a search processing method according to an embodiment. First, the tracking unit 1 detects a moving object (a person in this embodiment) included in a video, and tracks the moving object by associating the moving object in a time series (step S1). Next, the setting unit 3 accepts the setting of a search target included in the video (step S2). Next, the calculation unit 4 calculates a feature amount (first feature amount) indicating the characteristic of the search target and a feature amount (second feature amount) indicating the characteristic of the moving object, and calculates a similarity between the first and second feature amounts (first similarity) (step S3).

[0056] Next, the display control unit 5 accepts a selection of a first display mode in which moving objects whose first similarity is equal to or greater than a threshold (first threshold) are displayed, or a second display mode in which moving objects are displayed in descending order of first similarity for each camera that captured the video (step S4). Next, the display control unit 5 displays the search results corresponding to the search target on the display unit 6 in the display mode selected in the process of step S5 (step S5).

[0057] As described above, according to the search processing device 100 of the embodiment, when a search target is captured in a plurality of cameras, it is possible to easily grasp how many times the search target was captured in each camera. For example, it is possible to easily refer to which camera a person designated as a search target was captured in and what actions he or she took in each camera by using any method. For example, in the first display mode, even if the range of similarity values ​​differs for each camera, the user can roughly screen cameras in which the search target may be captured. Then, the user can check the search target in detail in a specific camera by using the second display mode that displays the search target in order of similarity in a specific camera.

[0058] Conventionally, for example, search results for multiple cameras with different ranges of similarity values ​​were displayed in order of rank, which meant that the visibility was poor when checking how many times an image was captured on each camera.

[0059] Finally, an example of the hardware configuration of the search processing device 100 of this embodiment will be described.

[0060] [Example of hardware configuration] 5 is a diagram showing an example of a hardware configuration of the search processing device 100 according to the embodiment. The search processing device 100 includes a processor 301, a main storage device 302, an auxiliary storage device 303, a display device 304, an input device 305, and a communication IF 306. The processor 301, the main storage device 302, the auxiliary storage device 303, the display device 304, the input device 305, and the communication IF 306 are connected via a bus 310.

[0061] The processor 301 executes a program read from the auxiliary storage device 303 to the main storage device 302. The main storage device 302 is a memory such as a ROM and a RAM. The auxiliary storage device 303 is a HDD, a memory card, or the like.

[0062] The display device 304 displays the above-mentioned display information (see FIGS. 2 and 3), etc. The input device 305 accepts input from the user. The communication IF 306 is an interface for connecting to the search processing device 100.

[0063] The search processing device 100 may not include the display device 304 and the input device 305. When the search processing device 100 does not include the display device 304 and the input device 305, for example, the display function and the input function of an external terminal connected via the communication IF 306 may be used.

[0064] The programs executed by the search processing device 100 are provided as computer program products stored in a computer-readable storage medium such as a CD-ROM, memory card, CD-R, or DVD (Digital Versatile Disc) in an installable or executable format file.

[0065] Furthermore, the program executed by the search processing device 100 may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0066] Furthermore, the programs executed by the search processing device 100 may be configured to be provided via a network such as the Internet without being downloaded.

[0067] Moreover, the programs executed by the search processing device 100 may be provided by being pre-installed in a ROM or the like.

[0068] The programs executed by the search processing device 100 have a modular configuration including functions that can be realized by the programs among the above-mentioned functional configuration of the search processing device 100 (see FIG. 1). The functions realized by the programs are loaded into the main storage device 302 by the processor 301 reading the programs from a storage medium such as the auxiliary storage device 303 and executing them. In other words, the functions realized by the programs are generated on the main storage device 302.

[0069] Some or all of the functions of the search processing device 100 may be realized by hardware such as an integrated circuit (IC).

[0070] Furthermore, when multiple processors are used to realize the respective functions, each processor may realize one of the respective functions, or may realize two or more of the respective functions.

[0071] Furthermore, the operation form of the computer that realizes the search processing device 100 may be arbitrary. For example, the search processing device 100 may be realized by a single computer. Also, for example, the search processing device 100 may be operated as a cloud system on a network.

[0072] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0073] 1. Tracking Department 2 Storage section 3. Settings 4 Calculation section 5. Display control section 6 Display section 7 Optimization Section 100 Search processing device 301 Processor 302 Main storage 303 Auxiliary storage device 304 Display device 305 Input Device 306 Communication Interface 310 Bus

Claims

1. a tracking unit that detects a moving object included in a video and tracks the moving object by associating the moving object in a time series; a setting unit that receives a setting of a search target included in the video; a calculation unit that calculates a first feature amount indicating a feature of the search target and a second feature amount indicating a feature of the moving object, and calculates a first similarity between the first and second feature amounts; a display control unit that accepts selection of a first display mode in which the moving objects having the first similarity equal to or greater than a first threshold value are displayed for the images captured by a plurality of cameras, or a second display mode in which the moving objects are displayed in descending order of the first similarity for the images captured by a designated one of the plurality of cameras, and displays a search result corresponding to the search target on a display unit in the selected display mode; A search processing device comprising:

2. The setting unit determines a search query from the setting of the search target, the display control unit displays, on the display unit, display information including the search query, the search result, and arrangement information indicating an arrangement of the cameras. The search processing device according to claim 1 .

3. the display control unit displays, among the cameras included in the arrangement information, a position of a camera that captured an image including the search result in an emphasized manner. The search processing device according to claim 2 .

4. The setting unit further receives a setting of an exclusion target to be excluded from the search target, the calculation unit calculates a third feature amount indicating a feature of the object to be excluded and the second feature amount, and calculates a second similarity between the second and third feature amounts; the first and second display modes do not display the moving object whose second similarity is equal to or greater than a second threshold; The search processing device according to any one of claims 1 to 3.

5. The present invention further includes an optimization unit that optimizes a feature extraction model by performing adaptive learning to adapt a feature extraction model based on the appearance of the moving object to an installation environment of the camera using a tracking result obtained by the tracking unit, The tracking unit detects the moving object using the feature extraction model. The search processing device according to claim 1 .

6. The setting unit accepts a setting of the search target via a reception unit that accepts designation of the search target from the video. The search processing device according to claim 1 .

7. The setting unit accepts a setting of the search target via a reception unit that accepts a designation of the search target from the search result. The search processing device according to any one of claims 1 to 6.

8. The setting unit further accepts at least one range setting of a size range of the search target and a date and time range in which the search target appears, The display control unit does not display the search results that are not included in the range setting. The search processing device according to any one of claims 1 to 7.

9. detecting a moving object included in the video and tracking the moving object by associating the moving object in a time series; A step of accepting a setting of a search target included in the video; calculating a first feature amount indicating a feature of the search target and a second feature amount indicating a feature of the moving object, and calculating a first similarity between the first and second feature amounts; accepting a selection of a first display mode in which the moving objects having the first similarity equal to or greater than a first threshold value are displayed for the images captured by a plurality of cameras, or a second display mode in which the moving objects are displayed in descending order of the first similarity for the images captured by a designated one of the plurality of cameras, and displaying a search result corresponding to the search target on a display unit in the selected display mode; A search processing method comprising:

10. Computer, a tracking unit that detects a moving object included in a video and tracks the moving object by associating the moving object in a time series; a setting unit that receives a setting of a search target included in the video; a calculation unit that calculates a first feature amount indicating a feature of the search target and a second feature amount indicating a feature of the moving object, and calculates a first similarity between the first and second feature amounts; a display control unit that accepts a selection of a first display mode in which the moving objects having the first similarity equal to or greater than a first threshold value are displayed for the images captured by a plurality of cameras, or a second display mode in which the moving objects are displayed in descending order of the first similarity for the images captured by a designated one of the plurality of cameras, and displays a search result corresponding to the search target on a display unit in the selected display mode; A program to function as a

Images