Search device, search method, and search program

Abstract

A threshold derivation unit (47) sets, as a target cluster, each of a plurality of clusters obtained by clustering a plurality of feature amounts stored in a feature database (48), and derives a threshold for the target cluster from the distribution of feature amounts in the target cluster. A search unit (44) uses, as a target threshold, a threshold for a cluster from among the clusters that possesses a search feature that is a feature amount of an image in a search request, and identifies a feature amount corresponding to the search feature from the plurality of feature amounts stored in the feature database (48).

Description

[Technical Field]

[0001] The present disclosure relates to a technology for searching for an object captured in image data obtained by a camera that captures a target space as a shooting area, using an image of the object as a search key. [Background technology]

[0002] In spaces where many people gather, such as large facilities such as stations, airports, and commercial facilities, or city blocks, a means for locating a specific person is required. For example, such a means is necessary when searching for a lost child, a wanderer, or a person separated from their companion at the request of a space user. Also, such a means is necessary when searching for a user who does not show up at the designated location at the scheduled time or entry time. Furthermore, such a means is necessary when searching for a user who has left something behind or who is found to have not completed the necessary procedures after leaving the store. Furthermore, from a crime prevention perspective, such a means is necessary when locating and apprehending an escaped shoplifter, molester, or assaulter, or when analyzing the behavior of a key witness in a criminal investigation.

[0003] In spaces where many people gather, many network cameras are often installed for security purposes. For this reason, a person search process is being studied that extracts features of a person from camera footage and uses these features to search for when and on which camera the person in question was captured in live or recorded footage. Live footage refers to real-time footage.

[0004] The features of a person that can be extracted from camera footage include (1) to (4) below. (1) Features that can be expressed in words, such as the color and shape of clothing or belongings, height, gender, and age. (2) Image features such as HoG. HoG stands for Histograms of Oriented Gradients. (3) Vector data in which a person's facial features are converted into a comparable form, as typified by face recognition technology. (4) Vector data in which a person's entire body features are converted into a comparable form.

[0005] When searching for a person, a person identification process is used in which, if the distance between the feature values ​​of two person images is equal to or less than a threshold, the two person images are determined to be images of the same person. Here, in person search processes using feature values, differences in the distance between feature values ​​occur due to differences in people's appearances or camera shooting conditions. As a result, there is a possibility of "mis-searches" where the wrong person is searched for, or "missed searches" where the person to be searched for is omitted from the search results.

[0006] Patent Document 1 describes a technology for solving problems caused by differences in shooting conditions. Patent Document 1 addresses the issue that, in face identification processing, the threshold value for the degree of similarity of facial features varies depending on the combination of cameras. To address this issue, Patent Document 1 identifies a person using a different logic, then calculates the error rate of facial feature matching using the identification result as the correct answer, and adjusts the threshold value so that the error rate is constant for each combination of cameras. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2020-187531 Summary of the Invention [Problem to be solved by the invention]

[0008] The technology described in Patent Document 1 is a technology that simply sets a threshold value for each combination of cameras. However, the optimal threshold value varies depending on the appearance of the target person. For example, the distribution of feature values ​​for a person wearing dark clothing from top to bottom may be small, while the distribution for a person wearing light clothing from top to bottom and dark clothing from bottom to top may be large. In this case, the threshold value for a person wearing dark clothing from top to bottom may be set to a relatively small value compared to a person wearing light clothing from top to bottom and dark clothing from bottom to bottom. Therefore, the technology described in Patent Document 1 may not be able to completely prevent false detections or missed searches, and may not be able to properly search for people. The present disclosure aims to enable an appropriate search for objects captured in image data. [Means for solving the problem]

[0009] A search device according to the present disclosure includes: a threshold derivation unit that determines, as a target cluster, each of a plurality of clusters obtained by clustering a plurality of feature quantities stored in the feature database, and derives a threshold for the target cluster from a distribution of the feature quantities in the target cluster; a search unit that uses the threshold derived by the threshold derivation unit for a cluster to which a search feature, which is a feature for an image in a search request, belongs as a target threshold, from among the plurality of feature values ​​stored in the feature database, to identify a feature value corresponding to the search feature; Equipped with. [Effects of the Invention]

[0010] In the present disclosure, a threshold is derived for each cluster obtained by clustering features, and a search is performed using the threshold for the cluster corresponding to the search feature. This allows the search to be performed using an appropriate threshold corresponding to the search feature, making it possible to properly search for the target object. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a configuration diagram of a search system 100 according to a first embodiment. [Figure 2]FIG. 2 is a hardware configuration diagram of a feature extraction device 30 and a search device 40 according to the first embodiment. [Figure 3] 10 is a flowchart of a collection process according to the first embodiment. [Figure 4] 10 is a flowchart of a search process according to the first embodiment. [Figure 5] FIG. 4 is an explanatory diagram of a threshold database 49 according to the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a cluster according to the first embodiment. [Figure 7] 4 is a flowchart of a threshold value derivation process according to the first embodiment. [Figure 8] FIG. 2 is a diagram illustrating the effects of the search system 100 according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Embodiment 1 In the first embodiment, a case where the target object is a person will be described. That is, in the first embodiment, a case where a person is searched for will be described. However, the target object is not limited to a person, and may be an animal such as a dog or a cat, or an object such as a bag.

[0013] ***Configuration Description*** The configuration of a search system 100 according to the first embodiment will be described with reference to FIG. The search system 100 includes a plurality of cameras 10, a hub 20, a feature extraction device 30, and a search device 40. In Fig. 1, the search system 100 includes N cameras 10, namely, camera 10-1 to camera 10-N, where N is an integer of 2 or greater. Each camera 10 is connected to the hub 20 via a transmission line. The hub 20 is connected to the feature extraction device 30 via a transmission line. The feature extraction device 30 is connected to the search device 40 via a transmission line.

[0014] The cameras 10 are installed in various locations within the target space for person search. The cameras 10 capture images of people moving through the target space. The cameras 10 transmit the captured images to the hub 20 via a transmission path such as an IP network. IP stands for Internet Protocol. The cameras 10 may be placed without sharing a common field of view. In other words, there may be blind spots within the target space that are not captured by the cameras 10. In the first embodiment, camera 10 is assumed to be an IP camera that compresses video and transmits it over an IP network. However, camera 10 may be a camera that transmits uncompressed video signals over a coaxial cable, or may be a camera that uses another transmission method.

[0015] The hub 20 receives the video data transmitted by the camera 10 and transmits it to the feature extraction device 30 . If the camera 10 is connected to the Internet using a public line and transmits video data to the Internet, the feature extraction device 30, which is also connected to the Internet, may receive the video data via the Internet. In this configuration, the Internet corresponds to the hub 20. If the data transmission method of the camera 10 is a protocol other than IP, the hub 20 is an aggregation device that supports that protocol.

[0016] The feature extraction device 30 is a computer that extracts feature amounts that can be used for person identification from people captured in video data obtained by the camera 10. The feature extraction device 30 includes, as functional components, a video data acquisition unit 31, an object detection unit 32, and a feature extraction unit 33.

[0017] The search device 40 is a computer that searches for people in response to a search request from a user. In the first embodiment, the search device 40 has a database function that manages feature amounts of people for searching. Note that the database function may be realized by a device external to the search device 40. The search device 40 includes, as functional components, a feature acquisition unit 41, a database registration unit 42, a request acquisition unit 43, a search unit 44, an output unit 45, a feature extraction unit 46, and a threshold derivation unit 47. The search device 40 also includes, as database functions, a feature database 48 and a threshold database 49.

[0018] The hardware configuration of the feature extraction device 30 and the search device 40 according to the first embodiment will be described with reference to FIG. The feature extraction device 30 and the search device 40 each include the following hardware components: a processor 101, a memory 102, a storage 103, and a communication interface 104. The processor 101 is connected to other hardware components via signal lines and controls the other hardware components.

[0019] The processor 101 is an IC that performs processing. IC stands for Integrated Circuit. Specific examples of the processor 101 include a CPU, a DSP, and a GPU. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. GPU stands for Graphics Processing Unit.

[0020] The memory 102 is a storage device that temporarily stores data. Specific examples of the memory 102 include SRAM and DRAM. SRAM stands for Static Random Access Memory. DRAM stands for Dynamic Random Access Memory.

[0021] The storage 103 is a storage device that stores data. A specific example of the storage 103 is an HDD. HDD is an abbreviation for Hard Disk Drive. The storage 103 may also be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. SD is an abbreviation for Secure Digital. DVD is an abbreviation for Digital Versatile Disk.

[0022] The communication interface 104 is an interface for communicating with external devices. Specific examples of the communication interface 104 include Ethernet (registered trademark), USB, and HDMI (registered trademark) ports. USB stands for Universal Serial Bus. HDMI stands for High-Definition Multimedia Interface.

[0023] The functions of the functional components of the feature extraction device 30 and the search device 40 are realized by software. The storage 103 of the feature extraction device 30 stores a program that realizes the function of each functional component of the feature extraction device 30. In the feature extraction device 30, this program is loaded into the memory 102 by the processor 101 and executed by the processor 101. In this way, the function of each functional component of the feature extraction device 30 is realized. Similarly, storage 103 of search device 40 stores a program that realizes the function of each functional component of search device 40. In search device 40, this program is read into memory 102 by processor 101 and executed by processor 101. In this way, the function of each functional component of search device 40 is realized.

[0024] The storage 103 of the search device 40 realizes a database function.

[0025] 2 shows only one processor 101. However, the feature extraction device 30 and the search device 40 may each include multiple processors 101, and the multiple processors 101 may execute programs that realize the respective functions in a coordinated manner.

[0026] ***Explanation of Operation*** The operation of the search system 100 according to the first embodiment will be described with reference to FIGS. The operation procedure of the search system 100 according to the embodiment 1 corresponds to the search method according to the embodiment 1. Furthermore, the program that realizes the operation of the search system 100 according to the embodiment 1 corresponds to the search program according to the embodiment 1.

[0027] The operation of the search system 100 according to the first embodiment includes a collection process for collecting feature amounts, a search process for performing a search, and a threshold derivation process for deriving a threshold.

[0028] The collection process according to the first embodiment will be described with reference to FIG. The collection process is always running while the search system 100 is in operation.

[0029] (Step S11: Transmission standby process) After the feature extraction device 30 is started, the video data acquisition unit 31 of the feature extraction device 30 waits for the transmission of video data from the camera 10 via the hub 20. The search device 40 may be always running, or may be started simultaneously with the feature extraction device 30.

[0030] (Step S12: Reception determination process) If the video data acquisition unit 31 of the feature extraction device 30 has not received the video data, the process returns to step S11. On the other hand, when the video data acquisition unit 31 receives video data, it decodes the received video data and outputs the decoded video data to the object detection unit 32. At this time, the decoded video is output together with a camera ID, which is the identifier of the camera 10 that captured the video data, and the time the video data was received (= the capture time). ID is an abbreviation for IDentifier.

[0031] Here, the camera ID can be identified by referring to a table that shows the correspondence between the IP address of the camera 10 and the camera ID, which is stored in advance in the feature extraction device 30. Alternatively, the IP address of the camera 10 itself may be used as the camera ID. However, the camera ID can be any information that is unique to the camera 10 and that can be used to link the actual camera 10 with the video data sent by some means.

[0032] (Step S13: Target extraction process) The object detection unit 32 of the feature extraction device 30 detects a person, who is an object appearing in the decoded video, from the decoded video output in step S13. Then, the object detection unit 32 outputs the detection result of the person, who is the detected object, and the camera ID and shooting time, which are set with the decoded video, to the feature extraction unit 33. Object detection is performed using a method that uses image analysis techniques such as HoG. Object detection may also be performed using a method that uses a machine learning approach such as CNN, Faster R-CNN, or SSD. CNN stands for Convolutional Neural Network. Faster R-CNN stands for Faster-Region-based CNN. SSD stands for Single Shot Detector.

[0033] The target to be detected must match the feature amount extracted in the process of step S14 described below. For example, if the feature amount requires a whole-body image of a person, the target detection unit 32 must detect a whole-body image of the person. If the feature amount requires a facial feature, the target detection unit 32 must detect a facial image. The detection result is an image of the detected person cut out from the decoded video. The detection result may be a set of the decoded video and position information within the video where the person was detected. If the feature extraction unit 33 has a means for accessing the recorded decoded video, the detection result may be a set of information that can identify the frame number of the recorded decoded video and position information within the video where the person was detected.

[0034] Depending on the feature extracted in the process of step S14 (described later), multiple consecutive frames may be required. For example, when a feature of a person's movement is extracted, multiple consecutive frames are required. In this case, the object detection unit 32 needs to continuously detect the same person across multiple frames and output the result as the detection result.

[0035] (Step S14: Feature extraction process) The feature extraction unit 33 of the feature extraction device 30 extracts feature amounts from the detection results output in step S13. The extracted features are those that can be used to calculate the similarity of people. For example, the features are image features such as HoG. Alternatively, the features are vector data obtained by applying deep learning to convert image features of a person's entire body into a comparable form. If the detection results are the result of continuously detecting the same person across multiple frames, the features may be gait features that are characteristics of the person's way of walking. Gait features include the period and amplitude of arm and leg swings, the period and amplitude of upper body sway, proportions, posture, etc. If the detection results are the result of continuously detecting the same person across multiple frames, the features may be a set of information obtained by extracting features that can be obtained from a single frame for each of the multiple frames.

[0036] (Step S15: Registration process) The feature extraction unit 33 of the feature extraction device 30 outputs the feature amount extracted in step S14 to the search device 40 together with the camera ID and the shooting time output in step S13. Then, the feature acquisition unit 41 of the search device 40 outputs a set of the feature amount output by the feature extraction unit 33, the camera ID, and the shooting time to the database registration unit 42. The database registration unit 42 registers the set of the feature amount output by the feature acquisition unit 41, the camera ID, and the shooting time in the feature database 48 as a new feature record.

[0037] The database registration unit 42 may appropriately delete records from the feature database 48 that have been registered for a certain period of time. When registering a new record, the database registration unit 42 may overwrite and save the old record. Alternatively, the database registration unit 42 may delete records from the feature database 48 based on other rules.

[0038] (Step S16: End determination process) The database registration unit 42 of the search device 40 determines whether or not a termination condition is satisfied. The termination condition may be, for example, a termination request from a user. The termination condition may also be a termination trigger generated by a mechanism other than the search system 100, such as a timer. If the termination condition is satisfied, the database registration unit 42 terminates the process. On the other hand, if the termination condition is not satisfied, the database registration unit 42 returns the process to step S11.

[0039] The search process according to the first embodiment will be described with reference to FIG. The search process is triggered by a request from the user.

[0040] (Step S21: Input waiting process) After the device is started, the request acquisition unit 43 of the search device 40 waits for input of a search request. The search request is input by a user. The search request includes image data of a person to be searched for, which is the object to be searched for. The search request may include at least one of the camera ID that captured the image data of the person to be searched for and the time when the image data of the person to be searched for was captured.

[0041] (Step S22: Input determination process) If a search request has not been input, the request acquisition unit 43 of the search device 40 returns the process to step S21. On the other hand, when a search request is input, the request acquisition unit 43 acquires the search request and outputs image data of the person to be searched for included in the search request to the feature extraction unit 46. When the search request includes at least one of the camera ID and the shooting time, the request acquisition unit 43 outputs the information included in the search request to the search unit 44.

[0042] The image data of the person to be searched for must be an image from which a feature amount used in person search can be extracted. For example, if the feature amount is a feature amount of a whole-body image, the image data of the person to be searched for must be a whole-body image that satisfies the conditions for extracting the whole-body image feature. The image data of the person to be searched for may also be a set of multiple image data. For example, the image data of the person to be searched for may be a set of image data taken from multiple angles or a set of images of the person in various clothing.

[0043] The camera ID and the shooting time are used to identify the starting point of the search. For example, if image data of the person to be searched for was captured by any camera 10 in the target area, the camera ID and the shooting time are the camera ID of the camera 10 that captured the image data and the time of the image capture. Alternatively, the camera ID and the shooting time may be the camera 10 that captured the image data at a location estimated based on eyewitness testimony of the person to be searched for, and the estimated shooting time. Alternatively, the camera ID and the shooting time may be identified from some electronic log information linked to the person to be searched for, such as IC card touch information, 2D code read information, or beacon reception record.

[0044] (Step S23: Feature extraction process) The feature extraction unit 46 of the search device 40 extracts feature amounts as search features from the image data of the person to be searched that was output in step S22. When the search request includes multiple pieces of image data of the person to be searched, the feature extraction unit 46 extracts feature amounts from each piece of image data. The feature amounts extracted here are the same as the feature amounts extracted in step S14 of FIG. 3. The feature extraction unit 46 outputs the extracted search features to the search unit 44.

[0045] Based on the determination process of step S26, the processes of steps S24 and S25 are executed with each camera 10 as the target camera 10. In the first embodiment, the search system 100 includes N cameras 10, camera 10-1 to camera 10-N. Therefore, for each integer i, i=1,...,N, the processes of steps S24 and S25 are executed with camera 10-i as the target camera 10.

[0046] (Step S24: Threshold extraction process) The search unit 44 of the search device 40 uses the feature quantity output by the feature extraction unit 46 to obtain a threshold value to be used for search from the threshold database 49 as the target threshold value.

[0047] This will be described in detail with reference to FIGS. Records are stored in the threshold database 49 by a threshold derivation process, which will be described later. As shown in Fig. 5, the threshold database 49 stores records for the target camera 10 and the target cluster, with each camera 10 being a target camera 10 and each cluster for the target camera 10 being a target cluster. Specifically, for the target camera 10 and the target cluster, a record including a camera ID, a cluster ID, a cluster center point, a cluster size, and a threshold is stored. 6, a cluster for each camera 10 is obtained by clustering a plurality of feature amounts for a person, which is an object, captured in image data obtained by that camera 10. Here, it is assumed that the number of clusters for camera 10-i is Di (2 in FIG. 6). The cluster center point is the average value of the features belonging to the cluster. The cluster center point may be the center of gravity of the features belonging to the cluster. Alternatively, the cluster center point may be the feature that has the smallest average distance from other features among the features belonging to the cluster. The cluster size is the average value of the distance between the cluster center point and the feature values ​​belonging to the cluster. The cluster size may also be an index such as the variance or standard deviation of the feature values ​​belonging to the cluster.

[0048] The cluster center point and cluster size are information that enable identification of the location and range of a cluster in the feature space. Therefore, each record may include not only the cluster center point and cluster size, but also area information for each area obtained by Voronoi division of the feature space based on the cluster center point.

[0049] The search unit 44 identifies the cluster to which the search feature output in step S23 belongs among the multiple clusters for the target camera 10-i. The search unit 44 obtains, from the threshold database 49, the threshold value in the record corresponding to the target camera 10-i and the identified cluster, as the target threshold value.

[0050] The search unit 44 identifies the cluster to which the search feature belongs by the following method 1 or method 2. (Method 1) The search unit 44 calculates the distance between the cluster center point of each of multiple clusters for the target camera 10-i and the search feature. The search unit 44 identifies the cluster with the shortest calculated distance as the cluster to which the search feature belongs. (Method 2) The search unit 44 sets each of the multiple clusters for the target camera 10-i as a cluster to be calculated. The search unit 44 calculates the distance between the cluster center point of the cluster to be calculated and the search feature. The search unit 44 divides the calculated distance by the cluster size of the cluster to be calculated. The search unit 44 identifies the cluster with the smallest calculated value as the cluster to which the search feature belongs. If the records in the threshold database 49 contain information about the Voronoi-divided regions, the search unit 44 may identify the cluster to which the search feature belongs based on the region information.

[0051] (Step S25: Neighborhood search process) The search unit 44 of the search device 40 performs a neighborhood search based on the search feature, and identifies a record having a feature similar to the search feature from among a plurality of records for the target camera 10-i stored in the feature database 48. Specifically, the search unit 44 sets the threshold value acquired in step S24 as the target threshold value. The search unit 44 identifies one or more feature amounts corresponding to the search feature from the feature amounts of the multiple records for the target camera 10-i stored in the feature database 48. At this time, the search unit 44 identifies one or more feature amounts from the feature amounts of the multiple records for the camera 10-i whose distance from the search feature is equal to or less than the target threshold value. The search unit 44 identifies the record corresponding to the identified feature amount as a record having a feature close to the search feature.

[0052] (Step S26: Camera determination process) The search unit 44 of the search device 40 determines whether or not the processes of steps S24 and S25 have been executed for all cameras 10 as the target camera 10. If the processes have been executed, the search unit 44 proceeds to step S27. On the other hand, if the processes have not been executed, the search unit 44 returns to step S24 and executes the processes for a new camera 10 as the target camera 10.

[0053] (Step S27: Output process) The output unit 45 of the search device 40 outputs the records identified in step S25, with each camera 10 being the target camera 10. At this time, the output unit 45 outputs the records identified in step S25 after organizing, integrating, or converting them into a format that is easy to handle as search results.

[0054] An example of sorting is sorting records. For example, the search unit 44 sorts records in descending order of similarity to the search feature. By sorting records in descending order of similarity, it becomes possible to present records to the user in descending order of reliability. The output unit 45 regards each record identified in step S25 as a target record, and calculates the distance between the feature amount of the target record and the search feature, and the threshold T ik Specifically, the output unit 45 calculates the similarity Sim by dividing the distance by a threshold value T ik The similarity Sim is calculated from the value obtained by dividing by . (Formula 1) Sim=1-Dist / T ik Dist is the distance between the feature of the target record and the search feature. T ik is the threshold used to identify the target record. The distance is calculated by the threshold T ik Since the threshold T ik is configured to be larger for larger clusters. Therefore, by calculating the similarity using Equation 1, clusters are sorted with a similarity that is close to reality, regardless of their size. Note that a large cluster is one in which the distance between features is large even for people with similar appearances.

[0055] The search unit 44 may sort the images in ascending or descending order of the shooting time instead of the similarity. The search unit 44 may also sort the images in order of a value obtained by combining the similarity, the time, and other information with a priority.

[0056] An example of integration is the extraction of representative records. For example, suppose that the records identified in step S25 include multiple records of video data captured at similar times by cameras 10 at the same or nearby locations. In this case, the search unit 44 selects only a representative portion of the multiple records and excludes the rest. The output unit 45 outputs only the remaining records.

[0057] Examples of conversion include extracting necessary information from a record and adding necessary information. For example, the search unit 44 extracts, from the information contained in the record, the shooting time, the camera ID, and an image of a person, or an image in which a rectangle surrounding the person is superimposed on a video frame containing the person. The search unit 44 then adds the person's search reliability score to the extracted information and outputs the result. The search reliability score may be, for example, the above-mentioned similarity or distance.

[0058] (Step S28: End determination process) The search unit 44 of the search device 40 determines whether a termination condition is met. The termination condition may be, for example, a termination request from a user. The termination condition may also be a termination trigger generated by a mechanism other than the search system 100, such as a timer. If the termination condition is satisfied, the search unit 44 terminates the process. On the other hand, if the termination condition is not satisfied, the search unit 44 returns the process to step S21.

[0059] The threshold value derivation process according to the first embodiment will be described with reference to FIG. The threshold value derivation process operates when a condition is met.

[0060] (Step S31: Execution waiting process) After the device is started, the threshold value derivation unit 47 of the search device 40 waits for the conditions to be met. The condition is any one of the following (A) to (D) or a combination of any two or more of them: (A) A certain amount of time has passed since the previous threshold derivation process was executed. (B) The number of records stored in the feature database 48 has exceeded a certain number. (C) The number of records in the feature database 48 that contain a specific camera ID has exceeded a certain number. (D) A user has requested that a threshold derivation process be executed.

[0061] (Step S32: Condition determination process) If the condition is not met, the threshold value derivation unit 47 of the search device 40 returns the process to step S31. On the other hand, if the condition is met, the threshold value derivation unit 47 advances the process to step S33.

[0062] Based on the determination process of step S38, the processes of steps S33 to S37 are executed with each camera 10 as the target camera 10. In the first embodiment, the search system 100 includes N cameras 10, camera 10-1 to camera 10-N. Therefore, for each integer i, i=1,...,N, the processes of steps S33 to S37 are executed with camera 10-i as the target camera 10.

[0063] (Step S33: Readout process) The threshold value derivation unit 47 of the search device 40 reads out the record for the target camera 10-i from the feature database 48. The threshold derivation unit 47 may read all records for the target camera 10-i. Alternatively, the threshold derivation unit 47 may read only some records randomly sampled from the records for the target camera 10-i. Alternatively, the threshold derivation unit 47 may read only records limited to specific conditions, such as a time period (e.g., nighttime) and a season, from among the records for the target camera 10-i.

[0064] (Step S34: Clustering process) The threshold value derivation unit 47 of the search device 40 clusters the feature quantities in the records read in step S33 in a feature quantity space. The threshold derivation unit 47 can perform clustering using existing algorithms such as the k-Means algorithm, Mean Shift, and Gaussian Mixture Model. Alternatively, the threshold derivation unit 47 may divide the feature space into subspaces of fixed sizes and treat each subspace as one cluster. Here, for the target camera 10-i, cluster D i1 From Cluster D ik Suppose the features are clustered into k clusters.

[0065] Based on the determination process in step S36, the process in step S35 is executed for each cluster as the target cluster. i1 From Cluster D ik Therefore, for each integer j, j=1,...,k, there are k clusters D ij The process of step S35 is executed with this as the target cluster.

[0066] (Step S35: Threshold value calculation process) The threshold derivation unit 47 of the search device 40 calculates the threshold value of the target cluster D ij From the distribution of ij Calculate the threshold for The purpose of deriving this threshold is to solve the problem of variations in the distance between features at each location in the feature space. Therefore, an index value corresponding to the distance between features is identified for each cluster, and a threshold is calculated according to the index value. For example, the threshold derivation unit 47 determines the distance between the features for the target cluster D ij The threshold value deriving unit 47 determines the variance or standard deviation from the cluster center point of the target cluster D. ij Alternatively, the index may be an average value of the distance between each feature belonging to the feature and its nearest neighbor. The threshold value derivation unit 47 calculates the threshold value by multiplying the index by a fixed coefficient.

[0067] (Step S36: Cluster determination process) The threshold derivation unit 47 of the search device 40 determines whether or not the process of step S35 has been executed for all clusters as target clusters. If the process has been executed, the threshold derivation unit 47 proceeds to step S37. On the other hand, if the process has not been executed, the threshold derivation unit 47 returns to step S35 and executes the process for a new cluster as the target cluster.

[0068] (Step S37: Threshold value update process) The threshold value derivation unit 47 of the search device 40 updates the threshold value for the target camera 10-i in the threshold value database 49 with the threshold value calculated in step S35. The configuration of the threshold value database 49 is as shown in FIG. Specifically, the threshold derivation unit 47 deletes the record of the target camera 10-i from the threshold database 49. Then, the threshold derivation unit 47 calculates a cluster D for each integer j, j=1,...,k, for the target camera 10-i. ij The record is registered in the threshold database 49. At this time, the threshold derivation unit 47 calculates the cluster center point and the cluster size and sets them in the respective fields. The threshold derivation unit 47 also sets the threshold calculated in step S35 in the threshold field.

[0069] (Step S38: Camera determination process) The threshold derivation unit 47 of the search device 40 determines whether or not the processes from step S33 to step S37 have been executed for all cameras 10 as the target camera 10. If the processes have been executed, the threshold derivation unit 47 proceeds to step S39. On the other hand, if the processes have not been executed, the threshold derivation unit 47 returns the process to step S33 and executes the processes for a new camera 10 as the target camera 10.

[0070] (Step S39: End determination process) The threshold value derivation unit 47 of the search device 40 determines whether or not a termination condition is satisfied. The termination condition may be, for example, a termination request from a user. The termination condition may also be a termination trigger generated by a mechanism other than the search system 100, such as a timer. If the termination condition is satisfied, the threshold value derivation unit 47 terminates the process. On the other hand, if the termination condition is not satisfied, the threshold value derivation unit 47 returns the process to step S31.

[0071] ***Effects of the First Embodiment*** As described above, the search system 100 according to the first embodiment derives a threshold value for each cluster obtained by clustering feature amounts, and performs a search using the threshold value for the cluster corresponding to the search feature. This enables a search to be performed using an appropriate threshold value corresponding to the search feature, making it possible to appropriately search for the target object.

[0072] The effects of the search system 100 according to the first embodiment will be specifically described with reference to FIG. Figure 8 shows an image in which feature amounts are plotted in feature amount space G1. Feature amount group G51 and feature amount group G52 are distributions made up of feature amounts of people wearing similar clothing. Feature amount group G51 is a distribution of people wearing dark clothing on both the top and bottom. Feature amount group G52 is a distribution of people wearing light-colored clothing on top and dark-colored clothing on the bottom. In FIG. 8, feature group G51 has a small distribution variance, while feature group G52 has a large distribution variance. In this case, a person wearing clothing as represented by feature group G51 can be identified using a relatively small threshold value. On the other hand, a person wearing clothing as represented by feature group G52 cannot be identified unless a relatively large threshold value is used. In this way, the appropriate threshold value may differ depending on the appearance of the object. In this case, if a uniform threshold value is set for camera 10, the identification accuracy will vary depending on the appearance of the object. In contrast, the search system 100 according to the first embodiment performs a search using a threshold value for the cluster corresponding to the search feature. Therefore, the threshold value used varies depending on whether the search feature belongs to feature amount group G51 or feature amount group G52. This enables an appropriate search for the target object.

[0073] ***Other Configurations*** <Variation 1> In the first embodiment, in step S27 of Fig. 4, the output unit 45 organizes, integrates, or converts the identified records into a format that is easy to handle as search results, and then outputs the results. As a first modification, the output unit 45 may estimate the movement route of the person to be searched for from the identified records, and output the estimated movement route.

[0074] A method for estimating a travel route will now be described in detail. (1) As explained as an example of integration in step S27 of FIG. 4, the output unit 45 excludes the rest of the records relating to video data captured by cameras 10 at similar or nearby locations and at similar shooting times, leaving only a representative portion of the records. (2) The output unit 45 sorts the remaining records in order of shooting time. (3) The output unit 45 plots the installation locations of the cameras 10 identified from the camera IDs of the records and connects them with arrows in the order of the records. The route indicated by the plotted points and arrows is the travel route of the person to be searched.

[0075] At this time, the output unit 45 calculates the likelihood of the movement path by statistical processing based on the reliability of the record identified in step S25 and the probability of movement between the cameras 10 on the movement path. Then, the output unit 45 outputs the likelihood together with the movement path. The reliability of a record is the distance between the feature of that record and the search feature. The reliability of a record may be the similarity described in the example of sorting in step S27 of Fig. 4. The probability of movement is calculated based on, for example, whether the movement can be performed without being captured by other cameras 10, whether the movement is possible taking into account the time of capture, etc.

[0076] The output unit 45 may estimate multiple travel routes for one person to be searched for by changing the method of selecting the representative records in (1), etc. Then, the output unit 45 may output each travel route together with the likelihood.

[0077] <Variation 2> In the first embodiment, each functional component is realized by software. However, as a second modification, each functional component may be realized by hardware. The differences between the first embodiment and the second modification will be described below.

[0078] When each functional component is realized by hardware, the feature extraction device 30 and the search device 40 include electronic circuits instead of the processor 101, memory 102, and storage 103. The electronic circuits are dedicated circuits that realize the functions of each functional component, memory 102, and storage 103.

[0079] Possible electronic circuits include single circuits, composite circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, and FPGAs. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be realized by one electronic circuit, or each functional component may be realized by distributing it among a plurality of electronic circuits.

[0080] <Variation 3> As a third modification, some of the functional components may be realized by hardware, and other functional components may be realized by software.

[0081] The processor 101, memory 102, storage 103, and electronic circuitry are collectively referred to as a processing circuit. In other words, the functions of the respective functional components are realized by the processing circuit.

[0082] In addition, the word "unit" in the above description may be read as a "circuit," "step," "procedure," "process," or "processing circuit."

[0083] The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Also, one or more of them may be implemented partially. Note that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as needed. [Explanation of symbols]

[0084] 100 Search system, 10 Camera, 20 Hub, 30 Feature extraction device, 31 Video data acquisition unit, 32 Object detection unit, 33 Feature extraction unit, 40 Search device, 41 Feature acquisition unit, 42 Database registration unit, 43 Request acquisition unit, 44 Search unit, 45 Output unit, 46 Feature extraction unit, 47 Threshold derivation unit, 48 Feature database, 49 Threshold database.

Claims

1. a threshold derivation unit that defines each of a plurality of clusters obtained by clustering a plurality of feature amounts stored in a feature database for a plurality of objects captured in image data as a target cluster, and derives a threshold for the target cluster from an index value of a distribution of the feature amounts in the target cluster; a search unit that identifies one of the plurality of clusters as a cluster to which a search feature, which is a feature amount for an object captured in image data in a search request, belongs, and that uses the threshold derived by the threshold derivation unit for the cluster to which the search feature belongs as a target threshold to identify one or more feature amounts, among the plurality of feature amounts stored in the feature database, whose distance from the search feature is equal to or less than the target threshold, as feature amounts corresponding to the search feature; A search device comprising:

2. The feature database stores a plurality of feature amounts for a plurality of objects captured in image data obtained by a plurality of cameras, the threshold derivation unit, with each of the plurality of cameras as a derivation target camera, derives the threshold for each of a plurality of clusters obtained by clustering a plurality of feature amounts for a plurality of objects captured in image data obtained by the derivation target camera; The search unit regards each of the plurality of cameras as a search target camera, identifies one of a plurality of clusters for the search target camera as a cluster to which a search feature, which is a feature amount for an object captured in image data in a search request, belongs, and uses the threshold value for the cluster to which the search feature belongs as a threshold value for the object, to identify one or more feature amounts, among the plurality of feature amounts for the plurality of objects captured in image data obtained by the search target camera, whose distance from the search feature is equal to or less than the threshold value for the object, as a feature amount corresponding to the search feature. The search device according to claim 1 .

3. The search device further an output unit that sets each of the one or more feature amounts identified by the search unit as a feature amount of a target, and calculates a similarity between the feature amount of the target and the search feature from a value obtained by dividing the distance between the feature amount of the target and the search feature by the threshold value of the target used when identifying the feature amount of the target; The search device according to claim 1 .

4. The output unit outputs information stored in the feature database in association with each of the one or more specified feature amounts, sorted in order of similarity. The search device according to claim 3 .

5. the computer determines each of a plurality of clusters obtained by clustering a plurality of feature amounts stored in a feature database for a plurality of objects captured in image data as a target cluster, and derives a threshold value for the target cluster from an index value of a distribution of the feature amounts in the target cluster; A search method in which a computer identifies one of the plurality of clusters as the cluster to which a search feature, which is a feature of an object captured in image data in a search request, belongs, and uses the threshold derived for the cluster to which the search feature belongs as a target threshold to identify one or more feature values ​​among the plurality of feature values ​​stored in the feature database whose distance from the search feature is less than or equal to the target threshold as feature values ​​corresponding to the search feature.

6. a threshold derivation process for determining, as a target cluster, each of a plurality of clusters obtained by clustering a plurality of feature amounts stored in a feature database for a plurality of objects captured in image data, and deriving a threshold for the target cluster from an index value of the distribution of the feature amounts in the target cluster; a search process of identifying one of the plurality of clusters as a cluster to which a search feature, which is a feature amount for an object captured in image data in a search request, belongs, and using the threshold value derived by the threshold value derivation process for the cluster to which the search feature belongs as a target threshold value, identifying one or more feature values, among the plurality of feature values ​​stored in the feature database, whose distance from the search feature is equal to or less than the target threshold value, as feature values ​​corresponding to the search feature; A search program that causes a computer to function as a search device.

Images