A human body real-time archive retrieval method, device and storage medium

Abstract

The present disclosure relates to the technical field of image processing, and discloses a method, device and storage medium for retrieving a human body real-time archive. The method comprises: determining a first retrieval archive based on a first edge relationship between a human body feature included in each target real-time image in a target real-time image subset and a centroid feature corresponding to a target archive, and a second edge relationship between human body features of any two target real-time images in the target real-time image subset in different time and space domains, and determining a second retrieval archive based on a third edge relationship between a remaining archive and images in the remaining real-time image subset, wherein the target archive, the remaining archive, the target real-time image subset and the remaining real-time image subset are obtained based on an image to be queried, an alternative archive image and an alternative real-time image, merging the first retrieval archive and the second retrieval archive, and using the merged archive as a human body retrieval result of the image to be queried, thereby reducing the amount of retrieval calculation and improving the efficiency of retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and provides a method, a device and a storage medium for retrieving real-time human body archives. Background Art

[0002] At present, in the field of intelligent security, especially in the field of image detection, image searching and image file searching are very important basic functions.

[0003] In the common real-time human body search process, the feature data included in the human body picture is not like the facial feature data. Only the feature value can achieve a relatively accurate clustering effect. The reason is that the facial data mainly depicts the facial features of the face, and the repetition between different people is small. In the real-time human body search, the feature data included in the human body picture will depend on the information such as the person's physique and clothing, which is prone to errors. Therefore, human body clustering will not only do a simple similarity comparison, but will generally be clustered under certain time and space constraints. That is, when searching for human bodies with pictures, users will be required to fill in some spatiotemporal information to assist in the search in addition to uploading human body pictures. The above process will obviously introduce a large amount of calculation, and the retrieval response speed is poor. Summary of the invention

[0004] The disclosed embodiments provide a method, device and storage medium for retrieving real-time human body archives, which are used to reduce the amount of calculation for retrieving a query image and improve the efficiency of the retrieval.

[0005] The specific technical solutions provided by this disclosure are as follows:

[0006] In a first aspect, an embodiment of the present disclosure provides a method for retrieving a real-time human body archive, the method being applied to a smart terminal, comprising:

[0007] Determine a first retrieval profile based on a first edge relationship between a human body feature included in each target real-time image in the target real-time image subset and a centroid feature corresponding to the target profile, and a second edge relationship between human body features of any two target real-time images in the target real-time image subset in different spatiotemporal domains, wherein the spatiotemporal domain is determined based on a sliding window and a geographic location identifier carried by the target real-time image;

[0008] Based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subsets, a second search archive is determined; wherein the target archive and the remaining archives are obtained by dividing the candidate archives according to the first similarity between the image to be queried and any one of the candidate archive pictures in the candidate archives, the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the candidate real-time picture set according to the second similarity, the second similarity is used to characterize the similarity between the image to be queried and any one of the candidate real-time pictures, and the candidate real-time picture set includes each candidate real-time picture that does not meet the archiving conditions of the candidate archive;

[0009] The first search file and the second search file are merged, and the merged file is used as the human body search result of the image to be queried.

[0010] Optionally, the target archive and the remaining archives are obtained by:

[0011] Extracting human features from the image to be queried, and determining the centroid features corresponding to each candidate file image in the candidate file;

[0012] Calculate the cosine similarity between the human body feature and each centroid feature respectively, and use the cosine similarity between the human body feature and each centroid feature as the first similarity;

[0013] The candidate archive pictures corresponding to the first similarity greater than the first preset threshold are determined as target archives, and the candidate archive pictures corresponding to the first similarity not greater than the first preset threshold are determined as remaining archives.

[0014] Optionally, the target real-time picture subset and the remaining real-time picture subsets are obtained in the following manner:

[0015] Determine the real-time human body features of any candidate real-time picture in the candidate real-time picture set;

[0016] Calculate the cosine similarity between the human body feature and each real-time human body feature respectively, and use the cosine similarity between the human body feature and each real-time human body feature as the second similarity;

[0017] The candidate real-time pictures corresponding to the second similarity greater than the second preset threshold are determined as the target real-time picture subset, and the candidate real-time pictures corresponding to the second similarity not greater than the second preset threshold are archived as the remaining real-time picture subset, wherein the second preset threshold is greater than the first preset threshold.

[0018] Optionally, the first edge relationship is determined by:

[0019] Calculate the cosine similarity between the human features included in each target real-time image in the target real-time image subset and the centroid features corresponding to the target file;

[0020] Determine that the target real-time image and the candidate archive image corresponding to the cosine similarity greater than a third preset threshold are the same person, wherein the third preset threshold is greater than the second preset threshold;

[0021] A first side relationship is established between the target real-time image and the alternative profile image based on the determination that they are the same person.

[0022] Optionally, the second edge relationship is determined by:

[0023] The target real-time picture subset is divided into multiple time domains in sequence according to the window length corresponding to the sliding window, and the target real-time picture subset corresponding to any time domain is divided into multiple space domains according to the geographic location identifier carried by the target real-time picture, and the target real-time picture subsets in different time and space domains are obtained based on the time domain and the space domain;

[0024] The cosine similarity between the human features of any two target real-time images in the target real-time image subsets in different time and space domains is calculated respectively;

[0025] For each spatiotemporal domain, two target real-time images corresponding to the cosine similarity greater than the tidal threshold of the spatiotemporal domain are judged to be the same person, where different spatiotemporal domains have different tidal thresholds, and each tidal threshold is used to represent the minimum probability value of judging two target real-time images as the same person in the corresponding spatiotemporal domain;

[0026] A second side relationship is established based on two target real-time images determined to be the same person.

[0027] Optionally, the first search profile is determined by:

[0028] The first side relationship and the second side relationship are aggregated to obtain a first search file.

[0029] Optionally, determining a second retrieval archive based on a third edge relationship between the remaining archives and the pictures in the remaining real-time picture subset includes:

[0030] Calculate the cosine similarity between the human features corresponding to any candidate archive picture in the remaining archives and the human features corresponding to any candidate real-time picture in the remaining real-time picture subset;

[0031] Determine that the candidate archive picture and the candidate real-time picture corresponding to the cosine similarity greater than a fourth preset threshold are the same person, wherein the fourth preset threshold is greater than the third preset threshold;

[0032] A third-side relationship is established based on the alternative archive image and the alternative real-time image determined to be the same person, and the third-side relationship is determined as the second retrieval archive.

[0033] In a second aspect, the embodiment of the present disclosure further provides a human body real-time archive retrieval device, comprising:

[0034] A first search archive generating unit is used to determine a first search archive based on a first edge relationship between a human body feature included in each target real-time image in the target real-time image subset and a centroid feature corresponding to the target archive, and a second edge relationship between human body features of any two target real-time images in the target real-time image subset in different spatiotemporal domains, wherein the spatiotemporal domain is determined based on a sliding window and a geographic location identifier carried by the target real-time image;

[0035] The second search archive generation unit is used to determine the second search archive based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subsets; wherein the target archive and the remaining archives are obtained by dividing the candidate archives according to the first similarity between the image to be queried and any one of the candidate archive pictures in the candidate archives, the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the candidate real-time picture set according to the second similarity, the second similarity is used to characterize the similarity between the image to be queried and any one of the candidate real-time pictures, and the candidate real-time picture set includes each candidate real-time picture that does not meet the archiving conditions of the candidate archive;

[0036] The file merging unit is used to merge the first search file and the second search file, and use the merged file as the human body search result of the image to be queried.

[0037] Optionally, the target archive and the remaining archives are obtained by:

[0038] Extracting human features from the image to be queried, and determining the centroid features corresponding to each candidate file image in the candidate file;

[0039] Calculate the cosine similarity between the human body feature and each centroid feature respectively, and use the cosine similarity between the human body feature and each centroid feature as the first similarity;

[0040] The candidate archive pictures corresponding to the first similarity greater than the first preset threshold are determined as target archives, and the candidate archive pictures corresponding to the first similarity not greater than the first preset threshold are determined as remaining archives.

[0041] Optionally, the target real-time picture subset and the remaining real-time picture subsets are obtained in the following manner:

[0042] Determine the real-time human body features of any candidate real-time picture in the candidate real-time picture set;

[0043] Calculate the cosine similarity between the human body feature and each real-time human body feature respectively, and use the cosine similarity between the human body feature and each real-time human body feature as the second similarity;

[0044] The candidate real-time pictures corresponding to the second similarity greater than the second preset threshold are determined as the target real-time picture subset, and the candidate real-time pictures corresponding to the second similarity not greater than the second preset threshold are archived as the remaining real-time picture subset, wherein the second preset threshold is greater than the first preset threshold.

[0045] Optionally, the first edge relationship is determined by:

[0046] Calculate the cosine similarity between the human features included in each target real-time image in the target real-time image subset and the centroid features corresponding to the target file;

[0047] Determine that the target real-time image and the candidate archive image corresponding to the cosine similarity greater than a third preset threshold are the same person, wherein the third preset threshold is greater than the second preset threshold;

[0048] A first side relationship is established between the target real-time image and the alternative profile image based on the determination that they are the same person.

[0049] Optionally, the second edge relationship is determined by:

[0050] The target real-time picture subset is divided into multiple time domains in sequence according to the window length corresponding to the sliding window, and the target real-time picture subset corresponding to any time domain is divided into multiple space domains according to the geographic location identifier carried by the target real-time picture, and the target real-time picture subsets in different time and space domains are obtained based on the time domain and the space domain;

[0051] The cosine similarity between the human features of any two target real-time images in the target real-time image subsets in different time and space domains is calculated respectively;

[0052] For each spatiotemporal domain, two target real-time images corresponding to the cosine similarity greater than the tidal threshold of the spatiotemporal domain are judged to be the same person, where different spatiotemporal domains have different tidal thresholds, and each tidal threshold is used to represent the minimum probability value of judging two target real-time images as the same person in the corresponding spatiotemporal domain;

[0053] A second side relationship is established based on two target real-time images determined to be the same person.

[0054] Optionally, the first search archive generating unit is used for:

[0055] The first side relationship and the second side relationship are aggregated to obtain a first search file.

[0056] Optionally, based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subset, a second search archive is determined, and the second search archive generation unit is used to:

[0057] Calculate the cosine similarity between the human features corresponding to any candidate archive picture in the remaining archives and the human features corresponding to any candidate real-time picture in the remaining real-time picture subset;

[0058] Determine that the candidate archive picture and the candidate real-time picture corresponding to the cosine similarity greater than a fourth preset threshold are the same person, wherein the fourth preset threshold is greater than the third preset threshold;

[0059] A third-side relationship is established based on the alternative archive image and the alternative real-time image determined to be the same person, and the third-side relationship is determined as the second retrieval archive.

[0060] In a third aspect, a smart terminal includes:

[0061] A memory for storing executable instructions;

[0062] A processor is used to read and execute executable instructions stored in a memory to implement any method as described in the first aspect.

[0063] In a fourth aspect, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any method described in the first aspect.

[0064] The beneficial effects of the present disclosure are as follows:

[0065] In summary, in the embodiments of the present disclosure, a method, device and storage medium for retrieving real-time human archives are provided. The method is applied to an intelligent terminal, including: determining a first retrieval archive based on a first edge relationship between a human feature included in each target real-time image in a target real-time image subset and a centroid feature corresponding to a target archive, and a second edge relationship between human features of any two target real-time images in the target real-time image subset in different time and space domains, wherein the time and space domain is determined based on a sliding window and a geographic location identifier carried by the target real-time image; after determining the first retrieval archive, determining a second retrieval archive based on a third edge relationship between the remaining archives and images in the remaining real-time image subset; it should be noted that the target archive and the remaining archive are candidate archives that are selected according to the relationship between the image to be queried and the image in the candidate archives. The first similarity between any candidate archive pictures is used for division, the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the candidate real-time picture set according to the second similarity, the second similarity here is used to characterize the similarity between the image to be queried and any candidate real-time picture, the candidate real-time picture set includes each candidate real-time picture that does not meet the archiving conditions of the candidate archive, the first retrieval archive and the second retrieval archive are merged, and the merged archive is used as the human body retrieval result of the image to be queried, the above-mentioned method of dividing the candidate archive pictures into target archives and remaining archives, dividing the target real-time picture into target real-time picture subsets and remaining real-time picture subsets, and performing separate similarity comparisons effectively reduces the amount of calculation of the image to be queried during the retrieval process and improves the efficiency of retrieval.

[0066] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0068] Figure 1 A schematic diagram of a process for obtaining a target file and remaining files in an embodiment of the present disclosure;

[0069] Figure 2 A schematic diagram of a process for obtaining a target real-time picture subset and a remaining real-time picture subset in an embodiment of the present disclosure;

[0070] Figure 3 A schematic diagram of a process for retrieving a real-time human body archive in an embodiment of the present disclosure;

[0071] Figure 4 A schematic diagram of a process for establishing a first edge relationship in an embodiment of the present disclosure;

[0072] Figure 5 A schematic diagram of a process for establishing a second edge relationship in an embodiment of the present disclosure;

[0073] Figure 6 A schematic diagram of a process for determining a second search profile in an embodiment of the present disclosure;

[0074] Figure 7 A schematic diagram of the logical architecture of a human body real-time archive retrieval device according to an embodiment of the present disclosure;

[0075] Figure 8 Schematic diagram of the physical architecture of the smart terminal in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the technical solution of the present disclosure, rather than all the embodiments. Based on the embodiments recorded in the present disclosure document, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present disclosure.

[0077] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented using sequences other than those illustrated or described herein.

[0078] The preferred embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0079] In the disclosed embodiment, the implementation of the retrieval method for the real-time human body archive is mainly executed on the smart terminal side, that is, on the smart terminal side: a first retrieval archive is determined based on the first edge relationship and the second edge relationship, a second retrieval archive is determined based on the third edge relationship, the first retrieval archive and the second retrieval archive are merged, and the merged archive is used as the human body retrieval result of the image to be queried, which is described in detail below.

[0080] In an embodiment of the present application, the target archive and the remaining archive are obtained by dividing the alternative archives according to the first similarity between the image to be queried and any one of the alternative archive images in the alternative archives, and the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the alternative real-time picture set according to the second similarity. The second similarity is used to characterize the similarity between the image to be queried and any one of the alternative real-time pictures. The alternative real-time picture set includes each alternative real-time picture that does not meet the archiving conditions of the alternative archive.

[0081] See also Figure 1 and Figure 2 As shown, the method of obtaining the target archive, the remaining archive, the target real-time image subset and the remaining real-time image subset is first introduced.

[0082] (1) The target file and the remaining files are obtained by:

[0083] Step 101: extracting human features from the image to be queried, and determining the centroid features corresponding to each candidate file image in the candidate files.

[0084] The above-mentioned image to be queried can be an image directly taken by the smart terminal or an image pre-entered into the smart terminal by the user. The above-mentioned candidate archives are usually in the portrait aggregation business, where the front-end camera captures the image, and the smart terminal (for example, a computer, etc.) performs an aggregation algorithm on the image to obtain a result in the form of one file for each person, and the result is stored as an archive in the database or memory, collectively referred to as the historical background file.

[0085] In the prior art, when performing real-time human archive retrieval, it is necessary to screen out all images of the same person as the image to be queried from the historical archives, but the above screening process is relatively cumbersome and the corresponding amount of calculation is also large.

[0086] In the embodiment of the present application, in order to reduce the amount of calculation for retrieval, each candidate file picture in the candidate file is classified, that is, divided into target files and remaining files. In addition, the pictures to be queried for real-time human file retrieval in the embodiment of the present application are all human pictures, which usually carry information such as face, human body, shooting time and location. Considering that human body pictures are different from face pictures, it is impossible to search based on obvious facial features. For this reason, during the implementation process, human body features are extracted from the pictures to be queried through a human body analysis model, for example, the human body features are converted into a 1*512-dimensional array, each dimension of which is a float64 array.

[0087] Furthermore, in order to calculate the first similarity between each alternative file picture in the alternative file and the picture to be queried, it is also necessary to determine the centroid feature corresponding to each alternative file picture in the alternative file. The centroid feature here is usually a picture feature with the highest file quality score, such as the length-to-width ratio of height, etc., as the file index.

[0088] Step 102: Calculate the cosine similarity between the human body feature and each centroid feature respectively, and use the cosine similarity between the human body feature and each centroid feature as the first similarity.

[0089] After determining the human features of the image to be queried and the centroid features of the candidate archive images, the cosine similarity between the human features of the image to be queried and each centroid feature is calculated to obtain multiple cosine similarities. In the implementation process, each of the above cosine similarities is used as the first similarity.

[0090] Step 103: Determine the candidate archive pictures corresponding to the first similarity greater than the first preset threshold as the target archive, and determine the candidate archive pictures corresponding to the first similarity not greater than the first preset threshold as the remaining archive.

[0091] Taking into account that the value of the first similarity is uncertain, that is, when the value of the first similarity is large, it indicates that the image to be queried is very similar to the candidate archive image, and the corresponding candidate archive image can be used as a retrieval object for the real-time archive of the human body; when the value of the first similarity is small, it indicates that the image to be queried is not similar to the candidate archive image, and the corresponding candidate archive image cannot be used as a retrieval object for the real-time archive of the human body.

[0092] During the implementation process, in order to screen out alternative archive pictures with higher similarity, the alternative archive pictures corresponding to the first similarity greater than the first preset threshold are determined as target archives, and the alternative archive pictures corresponding to the first similarity not greater than the first preset threshold are determined as remaining archives, that is, the alternative archives are divided according to the first preset threshold, and the first preset threshold here can be pre-configured according to the actual application scenario.

[0093] (2) The target real-time image subset and the remaining real-time image subset are obtained in the following way:

[0094] Since the generation of human body real-time archives is dynamic, historical archives are generally generated in two ways: one is to wait for the images entering the clustering process to reach a certain amount of data to start clustering, and the other is to cluster regularly, for example, once every 12 hours. Regardless of which of the above clustering methods is used, there will be some newly collected images that are not included in the historical archives. These images are called candidate real-time images, that is, candidate real-time images are images that have not yet been determined as candidate archives.

[0095] Step 201: Determine the real-time human body features of any candidate real-time picture in the candidate real-time picture set.

[0096] Considering that there may be search objects with high similarity to the query image in the candidate real-time images, in order to screen out these images, for any candidate real-time image in the candidate real-time image set, the real-time human features of each candidate real-time image can be determined through the human body analysis model.

[0097] Step 202: Calculate the cosine similarity between the human body feature and each real-time human body feature respectively, and use the cosine similarity between the human body feature and each real-time human body feature as the second similarity.

[0098] After determining the real-time human features of each candidate real-time image, the cosine similarity between the human features of the query image and each real-time human feature is calculated to obtain multiple cosine similarities. During the implementation process, the cosine similarities between the above human features and each real-time human feature are used as the second similarity.

[0099] Step 203: Determine the candidate real-time pictures corresponding to the second similarity greater than the second preset threshold as the target real-time picture subset, and archive the candidate real-time pictures corresponding to the second similarity not greater than the second preset threshold as the remaining real-time picture subset, wherein the second preset threshold is greater than the first preset threshold.

[0100] Taking into account that the value of the second similarity is uncertain, that is, when the value of the second similarity is large, it indicates that the query image is very similar to the candidate real-time image, and the corresponding candidate real-time image can be used as a retrieval object for the human body real-time archive; when the value of the second similarity is small, it indicates that the query image is not similar to the candidate real-time image, and the corresponding candidate real-time image cannot be used as a retrieval object for the human body real-time archive.

[0101] During the implementation, in order to screen out candidate real-time pictures with higher similarity, the candidate real-time pictures corresponding to the second similarity greater than the second preset threshold are determined as pictures in the target real-time picture subset, and the candidate real-time pictures corresponding to the second similarity not greater than the second preset threshold are determined as pictures in the remaining real-time picture subset, that is, the candidate real-time pictures are divided according to the second preset threshold, and the second preset threshold here can be pre-configured according to the actual application scenario. In addition, since the uncertainty of the candidate real-time pictures is large, in order to reduce the amount of calculation, the above second preset threshold is greater than the first preset threshold.

[0102] The above-mentioned processing method of dividing the candidate archive images into target archives and remaining archives, dividing the target real-time images into target real-time image subsets and remaining real-time image subsets, and performing similarity comparison respectively, for example, can be processed simultaneously by multiple threads, aiming to limit the calculation complexity and thus improve the response speed of the retrieval.

[0103] After determining the target archive, remaining archives, target real-time image subset, and remaining real-time image subset, refer to Figure 3 As shown, in the embodiment of the present disclosure, the specific execution process of retrieving the real-time human body archive is as follows:

[0104] Step 301: Determine a first retrieval archive based on a first edge relationship between a human body feature included in each target real-time image in the target real-time image subset and a centroid feature corresponding to the target archive, and a second edge relationship between human body features of any two target real-time images in the target real-time image subset in different spatiotemporal domains, wherein the spatiotemporal domain is determined based on a sliding window and a geographic location identifier carried by the target real-time image.

[0105] In order to make the retrieval of the human body real-time archive easier and faster, the first side relationship and the second side relationship can be obtained at the same time, where the first side relationship is established when the target real-time image in the target real-time image subset is highly similar to the candidate archive image in the target archive, and the second side relationship is established when any two target real-time images in the target real-time image subset are highly similar. Determining the first retrieval archive based on the first side relationship and the second side relationship is the key calculation part in the embodiment of the present application.

[0106] For details, see Figure 4 As shown, the first edge relationship is determined by:

[0107] Step 401: Calculate the cosine similarity between the human features included in each target real-time image in the target real-time image subset and the centroid features corresponding to the target file.

[0108] After determining the human features corresponding to each target real-time picture and the centroid features corresponding to the alternative file pictures included in the target file, based on the alternative file pictures in the target file with higher similarity to the picture to be queried, and based on the target real-time picture in the target real-time picture subset with higher similarity to the picture to be queried, during the implementation process, the similarity between the target file and the target real-time picture subset is further calculated. Specifically, the cosine similarity between the human features included in each target real-time picture in the target real-time picture subset and the centroid features corresponding to the target file is calculated, so as to further screen the target real-time pictures and alternative file pictures that can be clustered into the same person.

[0109] Step 402: Determine the target real-time image and the candidate archive image corresponding to the cosine similarity greater than a third preset threshold as the same person, wherein the third preset threshold is greater than the second preset threshold.

[0110] In the process of further screening, the third preset threshold for determining the same person is set to be greater than the above-mentioned second preset threshold. That is, in order to be able to screen out the target real-time image and the alternative file image that can be archived as the same person, the target real-time image and the alternative file image corresponding to the cosine similarity greater than the above-mentioned third preset threshold are determined to be the same person.

[0111] Step 403: Establish a first edge relationship between the target real-time image and the candidate profile image based on the determination that they are the same person.

[0112] Since, in the field of image-based file aggregation, the similarity between images is characterized by edge relationships, accordingly, after the target real-time image and the candidate file image of the same person are determined in step 402, a first edge relationship is established based on the target real-time image and the candidate file image determined to be the same person. In this way, in the subsequent processing process, the target real-time image and the candidate file image with a higher correlation can be directly determined based on the first edge relationship.

[0113] For details, see Figure 5 As shown, the second edge relationship is determined by:

[0114] Step 501: Divide the target real-time picture subset into multiple time domains in sequence according to the window length corresponding to the sliding window, and divide the target real-time picture subset corresponding to any time domain into multiple space domains according to the geographic location identifier carried by the target real-time picture, and obtain the target real-time picture subsets in different time and space domains based on the time domain and the space domain.

[0115] Taking into account the ambiguity of human body features and in order to further reduce the amount of calculation, in an embodiment of the present application, the target real-time image subsets are further divided by sliding windows and geographic location identifiers, so that target real-time image subsets in different time and space domains can be processed simultaneously by multiple parallel threads.

[0116] Specifically, in the time dimension, the window length corresponding to the sliding window can be set according to the actual application scenario. At the same time, after sorting the target real-time pictures in the target real-time picture subset in chronological order, the sorted target real-time pictures are divided into multiple time domains according to the above window length. The result is that there are N target real-time pictures corresponding to each time domain.

[0117] In the spatial dimension, given that each target real-time image carries a geographic location identifier, it is necessary to first extract the geographic location identifier based on each target real-time image. On this basis, the target real-time images with the same geographic location identifier, that is, within the same geographic range, are divided. Specifically, the target real-time image subset corresponding to each time domain is divided into multiple spatial domains according to the geographic location identifier. It should be noted that the above-mentioned geographic range division can be flexibly set according to the situation.

[0118] After obtaining the time domain and the space domain, the target real-time pictures in the same time domain and the same space domain are divided into the same time-space domain, so that the target real-time picture subset is divided into multiple small picture sets in different time-space domains.

[0119] Step 502: Calculate the cosine similarity between the human features of any two target real-time pictures in the target real-time picture subsets in different time and space domains.

[0120] During the implementation, after the target real-time image subsets are divided into different spatiotemporal domains, threads can be created based on different spatiotemporal domains for synchronous processing to improve the retrieval efficiency. Specifically, the cosine similarity between the human features of any two target real-time images in the target real-time image subsets is calculated, thereby obtaining the cosine similarity between the human features of the target real-time images in each spatiotemporal domain.

[0121] Step 503: For each spatiotemporal domain, two target real-time images corresponding to a cosine similarity greater than a tidal threshold of the spatiotemporal domain are determined to be the same person, wherein different spatiotemporal domains have different tidal thresholds corresponding to them, and each tidal threshold is used to represent the minimum probability value of determining two target real-time images as the same person in the corresponding spatiotemporal domain.

[0122] Considering that human features are easily affected by clothing and environment in the actual monitoring process, for example, two women wearing the same white coat in different spaces (drug store A and hospital B) are easily identified as the same person in the clustering algorithm. In addition, the nature of the spatial domain will also affect the discrimination of clustering. For example, there are many people with similar clothing appearing in the vegetable market. If the same discrimination threshold as that of a certain building is used, it may affect the accuracy of clustering. For this reason, the embodiment of the present application sets tidal thresholds for different spatiotemporal domains, that is, each spatiotemporal domain sets a different similarity threshold for judging the same person.

[0123] In the specific implementation process, the tidal threshold corresponding to the current spatiotemporal domain is first determined, that is, the minimum probability value of determining the two target real-time images as the same person in the current spatiotemporal domain. Then, it is determined whether the cosine similarity calculated in the above step 502 is greater than the corresponding tidal threshold. When it is determined that the cosine similarity is greater than the corresponding tidal threshold, the corresponding two target real-time images are determined to be the same person.

[0124] Step 504: Establish a second edge relationship based on the two target real-time pictures determined to be the same person. Similarly, in the field of image aggregation, the similarity between pictures is characterized by edge relationships. After the two target real-time pictures determined to be the same person are determined in step 503, a second edge relationship is established based on the two target real-time pictures determined to be the same person. In this way, in the subsequent processing process, the target real-time pictures with a higher correlation in the target real-time picture subset can be directly determined based on the second edge relationship.

[0125] After obtaining the above-mentioned first edge relationship and second edge relationship, the first search file is determined in the following manner: the first edge relationship and the second edge relationship are aggregated to obtain the first search file.

[0126] During the implementation process, after the first edge relationship and the second edge relationship are determined, the above edge relationships are summarized, including but not limited to removing repeated parts in the above edge relationships, and sorting out pictures related to the first edge relationship and the second edge relationship together to obtain a first search file.

[0127] During the implementation process, after similarity calculation is performed based on the target archive and the target real-time image subset to obtain the first edge relationship and the second edge relationship respectively, and the first retrieval archive is obtained by summarizing, the similarity calculation is also performed on the remaining archives and the remaining real-time image subsets accordingly, so as to determine the second retrieval archive. Different from the calculation process of the first retrieval archive, only the similarity needs to be calculated in the determination process of the second retrieval archive, and there is no need to divide the time and space domains. In other words, the images in the remaining archives and the remaining real-time image subsets are roughly screened, and only the images with higher similarity are selected as the second retrieval archive.

[0128] Step 302: Determine a second retrieval archive based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subset.

[0129] Because, in the actual archiving process, the method of obtaining the real-time archive of the human body only by relying on the target archive and the target real-time image subset may miss some images that may belong to the archive of the same person. Therefore, it is also necessary to determine the images in the archive that belong to the same person based on the remaining archives and the remaining real-time image subsets. This part belongs to the rough calculation part in this application.

[0130] Specifically, based on the third edge relationship between the remaining archives and the images in the remaining real-time image subset, the second search archive is determined, see Figure 6 As shown, including:

[0131] Step 601: Calculate the cosine similarity between the human features corresponding to any candidate archive picture in the remaining archives and the human features corresponding to any candidate real-time picture in the remaining real-time picture subset.

[0132] Similarly, in order to determine whether the images in the remaining archives and the remaining real-time image subsets are the archives of the same person, during the implementation process, it is necessary to first extract the corresponding human features from the alternative archive images, and then extract the corresponding human features from the alternative real-time images.

[0133] Furthermore, the cosine similarity between the human features of each candidate archive picture in the remaining archives and the human features corresponding to each candidate real-time picture in the remaining real-time picture subset is calculated to determine whether the picture is of the same person as the picture to be queried.

[0134] Step 602: Determine the candidate archive image and the candidate real-time image corresponding to the cosine similarity greater than a fourth preset threshold as the same person, wherein the fourth preset threshold is greater than the third preset threshold.

[0135] In order to improve the retrieval efficiency, the selection of the alternative archive pictures and the alternative real-time pictures here requires setting the fourth preset threshold in advance. The fourth preset threshold here is greater than the third preset threshold, that is, the requirement for judging the same person from the remaining archives and the remaining real-time picture subsets is higher.

[0136] During the implementation process, if the cosine similarity between the candidate archive picture and the candidate real-time picture is greater than a fourth preset threshold, the candidate archive picture and the candidate real-time picture are determined to be the same person.

[0137] Step 603: Establish a third-side relationship based on the candidate archive image and the candidate real-time image determined to be the same person, and determine the third-side relationship as the second retrieval archive.

[0138] Similarly, after determining that the alternative archive pictures and the alternative real-time pictures are the same person through step 602, a third-side relationship is established based on the alternative archive pictures and the alternative real-time pictures determined to be the same person, so as to directly determine the alternative archive pictures and the alternative real-time pictures with a higher correlation based on the third-side relationship. In this case, the third-side relationship can be determined as the second retrieval file.

[0139] Step 303: merge the first search file and the second search file, and use the merged file as the human body search result of the image to be queried.

[0140] During the implementation process, after obtaining the first retrieval archive and the second retrieval archive, the above-mentioned first retrieval archive and the second retrieval archive are merged. The merger here can be to aggregate the pictures corresponding to the clustering results of the target archive and the target real-time image subset into the pictures corresponding to the clustering results of the remaining archive and the remaining real-time image subset, or to aggregate the pictures corresponding to the clustering results of the remaining archive and the remaining real-time image subset into the pictures corresponding to the clustering results of the target archive and the target real-time image subset, so that the merged archive is used as the human body retrieval result of the image to be queried.

[0141] Based on the same inventive concept, refer to Figure 7 As shown, in an embodiment of the present disclosure, a device for retrieving a real-time human body archive is provided, comprising:

[0142] The first search archive generating unit 701 is used to determine the first search archive based on the first edge relationship between the human body features included in each target real-time image in the target real-time image subset and the centroid feature corresponding to the target archive, and the second edge relationship between the human body features of any two target real-time images in the target real-time image subset in different spatiotemporal domains, wherein the spatiotemporal domain is determined based on the sliding window and the geographic location identifier carried by the target real-time image;

[0143] The second search archive generating unit 702 is used to determine the second search archive based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subsets; wherein the target archive and the remaining archives are obtained by dividing the candidate archives according to the first similarity between the image to be queried and any one of the candidate archive pictures in the candidate archives, the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the candidate real-time picture set according to the second similarity, the second similarity is used to characterize the similarity between the image to be queried and any one of the candidate real-time pictures, and the candidate real-time picture set includes each candidate real-time picture that does not meet the archiving conditions of the candidate archive;

[0144] The file merging unit 703 is used to merge the first search file and the second search file, and use the merged file as the human body search result of the image to be queried.

[0145] Optionally, the target archive and the remaining archives are obtained by:

[0146] Extracting human features from the image to be queried, and determining the centroid features corresponding to each candidate file image in the candidate file;

[0147] Calculate the cosine similarity between the human body feature and each centroid feature respectively, and use the cosine similarity between the human body feature and each centroid feature as the first similarity;

[0148] The candidate archive pictures corresponding to the first similarity greater than the first preset threshold are determined as target archives, and the candidate archive pictures corresponding to the first similarity not greater than the first preset threshold are determined as remaining archives.

[0149] Optionally, the target real-time picture subset and the remaining real-time picture subsets are obtained in the following manner:

[0150] Determine the real-time human body features of any candidate real-time picture in the candidate real-time picture set;

[0151] Calculate the cosine similarity between the human body feature and each real-time human body feature respectively, and use the cosine similarity between the human body feature and each real-time human body feature as the second similarity;

[0152] The candidate real-time pictures corresponding to the second similarity greater than the second preset threshold are determined as the target real-time picture subset, and the candidate real-time pictures corresponding to the second similarity not greater than the second preset threshold are archived as the remaining real-time picture subset, wherein the second preset threshold is greater than the first preset threshold.

[0153] Optionally, the first edge relationship is determined by:

[0154] Calculate the cosine similarity between the human features included in each target real-time image in the target real-time image subset and the centroid features corresponding to the target file;

[0155] Determine that the target real-time image and the candidate archive image corresponding to the cosine similarity greater than a third preset threshold are the same person, wherein the third preset threshold is greater than the second preset threshold;

[0156] A first side relationship is established between the target real-time image and the alternative profile image based on the determination that they are the same person.

[0157] Optionally, the second edge relationship is determined by:

[0158] The target real-time picture subset is divided into multiple time domains in sequence according to the window length corresponding to the sliding window, and the target real-time picture subset corresponding to any time domain is divided into multiple space domains according to the geographic location identifier carried by the target real-time picture, and the target real-time picture subsets in different time and space domains are obtained based on the time domain and the space domain;

[0159] The cosine similarity between the human features of any two target real-time images in the target real-time image subsets in different time and space domains is calculated respectively;

[0160] For each spatiotemporal domain, two target real-time images corresponding to the cosine similarity greater than the tidal threshold of the spatiotemporal domain are judged to be the same person, where different spatiotemporal domains have different tidal thresholds, and each tidal threshold is used to represent the minimum probability value of judging two target real-time images as the same person in the corresponding spatiotemporal domain;

[0161] A second side relationship is established based on two target real-time images determined to be the same person.

[0162] Optionally, the first search archive generating unit 701 is used to:

[0163] The first side relationship and the second side relationship are aggregated to obtain a first search file.

[0164] Optionally, based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subset, a second search archive is determined, and the second search archive generating unit 702 is used to:

[0165] Calculate the cosine similarity between the human features corresponding to any candidate archive picture in the remaining archives and the human features corresponding to any candidate real-time picture in the remaining real-time picture subset;

[0166] Determine that the candidate archive picture and the candidate real-time picture corresponding to the cosine similarity greater than a fourth preset threshold are the same person, wherein the fourth preset threshold is greater than the third preset threshold;

[0167] A third-side relationship is established based on the alternative archive image and the alternative real-time image determined to be the same person, and the third-side relationship is determined as the second retrieval archive.

[0168] Based on the same inventive concept, refer to Figure 8 As shown, an embodiment of the present disclosure provides a server, including: a memory 801, used to store executable instructions; a processor 802, used to read and execute the executable instructions stored in the memory, and execute any one of the methods of the first aspect above.

[0169] Based on the same inventive concept, an embodiment of the present disclosure provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any method described in the first aspect above.

[0170] In summary, in the embodiments of the present disclosure, a method, device and storage medium for retrieving real-time human archives are provided. The method is applied to an intelligent terminal, including: determining a first retrieval archive based on a first edge relationship between a human feature included in each target real-time image in a target real-time image subset and a centroid feature corresponding to a target archive, and a second edge relationship between human features of any two target real-time images in the target real-time image subset in different time and space domains, wherein the time and space domain is determined based on a sliding window and a geographic location identifier carried by the target real-time image; after determining the first retrieval archive, determining a second retrieval archive based on a third edge relationship between the remaining archives and images in the remaining real-time image subset; it should be noted that the target archive and the remaining archive are candidate archives that are selected according to the relationship between the image to be queried and the image in the candidate archives. The first similarity between any candidate archive pictures is used for division, the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the candidate real-time picture set according to the second similarity, the second similarity here is used to characterize the similarity between the image to be queried and any candidate real-time picture, the candidate real-time picture set includes each candidate real-time picture that does not meet the archiving conditions of the candidate archive, the first retrieval archive and the second retrieval archive are merged, and the merged archive is used as the human body retrieval result of the image to be queried, the above-mentioned method of dividing the candidate archive pictures into target archives and remaining archives, dividing the target real-time picture into target real-time picture subsets and remaining real-time picture subsets, and performing separate similarity comparisons effectively reduces the amount of calculation of the image to be queried during the retrieval process and improves the efficiency of retrieval.

[0171] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program product systems. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product system implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0172] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program product systems according to the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0175] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for retrieving real-time human body archives, characterized in that: The method is applied to a smart terminal and includes: Determine a first retrieval profile based on a first edge relationship between a human body feature included in each target real-time picture in the target real-time picture subset and a centroid feature corresponding to the target profile, and a second edge relationship between human body features of any two target real-time pictures in the target real-time picture subset in different spatiotemporal domains, wherein the spatiotemporal domain is determined based on a sliding window and a geographic location identifier carried by the target real-time picture; Based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subsets, a second search archive is determined; wherein the target archive and the remaining archives are obtained by dividing the candidate archives according to the first similarity between the image to be queried and any one of the candidate archive pictures in the candidate archives, the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the candidate real-time picture set according to the second similarity, the second similarity is used to characterize the similarity between the image to be queried and any one of the candidate real-time pictures, and the candidate real-time picture set includes each candidate real-time picture that does not meet the archiving condition of the candidate archive; The first search file and the second search file are merged, and the merged file is used as the human body search result of the image to be queried.

2. The method according to claim 1, characterized in that The target file and the remaining files are obtained by: Extracting human features from the image to be queried, and determining the centroid features corresponding to each candidate file image in the candidate file; Calculating the cosine similarity between the human body feature and each of the centroid features respectively, and taking the cosine similarity between the human body feature and each of the centroid features as the first similarity; The candidate archive pictures corresponding to the first similarity greater than the first preset threshold are determined as target archives, and the candidate archive pictures corresponding to the first similarity not greater than the first preset threshold are determined as remaining archives.

3. The method according to claim 2, characterized in that The target real-time picture subset and the remaining real-time picture subset are obtained in the following manner: Determine the real-time human body features of any candidate real-time picture in the candidate real-time picture set; respectively calculating the cosine similarities between the human body feature and each of the real-time human body features, and taking the cosine similarities between the human body feature and each of the real-time human body features as the second similarity; The candidate real-time pictures corresponding to the second similarity greater than a second preset threshold are determined as a target real-time picture subset, and the candidate real-time pictures corresponding to the second similarity not greater than a second preset threshold are archived as a remaining real-time picture subset, wherein the second preset threshold is greater than the first preset threshold.

4. The method according to claim 3, characterized in that The first edge relationship is determined by: Calculate the cosine similarity between the human features included in each target real-time image in the target real-time image subset and the centroid features corresponding to the target file; Determine that the target real-time picture and the candidate profile picture corresponding to the cosine similarity greater than a third preset threshold are the same person, wherein the third preset threshold is greater than the second preset threshold; The first edge relationship is established based on the target real-time image and the candidate profile image being determined to be the same person.

5. The method according to claim 1, characterized in that The second side relationship is determined by: The target real-time picture subset is divided into a plurality of time domains in sequence according to the window length corresponding to the sliding window, and the target real-time picture subset corresponding to any one of the time domains is divided into a plurality of space domains according to the geographic location identifier carried by the target real-time picture, and based on the time domain and the space domain, target real-time picture subsets in different time and space domains are obtained; Respectively calculating the cosine similarity between the human features of any two target real-time images in the target real-time image subsets in different spatiotemporal domains; For each spatiotemporal domain, the two target real-time pictures corresponding to the cosine similarity greater than the tidal threshold of the spatiotemporal domain are determined to be the same person, wherein the tidal thresholds corresponding to different spatiotemporal domains are different, and each tidal threshold is used to represent the minimum probability value of determining the two target real-time pictures as the same person in the corresponding spatiotemporal domain; The second edge relationship is established based on the two target real-time images determined to be the same person.

6. The method according to claim 1, characterized in that The first search profile is determined by: The first edge relationship and the second edge relationship are aggregated to obtain the first search profile.

7. The method according to claim 4, characterized in that The determining of the second search archive based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subset includes: Respectively calculating the cosine similarity between the human features corresponding to any one of the candidate archive pictures in the remaining archives and the human features corresponding to any one of the candidate real-time pictures in the remaining real-time picture subsets; Determine that the candidate archive picture and the candidate real-time picture corresponding to the cosine similarity greater than a fourth preset threshold are the same person, wherein the fourth preset threshold is greater than the third preset threshold; A third side relationship is established between the candidate profile picture and the candidate real-time picture based on the determination that they are the same person, and the third side relationship is determined as the second retrieval profile.

8. A human body real-time archive retrieval device, characterized in that: include: A first search archive generating unit is used to determine a first search archive based on a first edge relationship between a human body feature included in each target real-time image in the target real-time image subset and a centroid feature corresponding to the target archive, and a second edge relationship between human body features of any two target real-time images in the target real-time image subset in different spatiotemporal domains, wherein the spatiotemporal domain is determined based on a sliding window and a geographic location identifier carried by the target real-time image; The second search archive generation unit is used to determine the second search archive based on the third edge relationship between the remaining archives and the pictures in the remaining real-time picture subsets; wherein the target archive and the remaining archives are obtained by dividing the candidate archives according to the first similarity between the image to be queried and any one of the candidate archive pictures in the candidate archives, and the target real-time picture subset and the remaining real-time picture subset are obtained by dividing the candidate real-time picture set according to the second similarity, and the second similarity is used to characterize the similarity between the image to be queried and any one of the candidate real-time pictures, and the candidate real-time picture set includes each candidate real-time picture that does not meet the archiving condition of the candidate archive; The file merging unit is used to merge the first search file and the second search file, and use the merged file as the human body search result of the image to be queried.

9. An intelligent terminal, characterized in that: include: A memory for storing executable instructions; A processor, configured to read and execute the executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 7.

Images