An image retrieval method, an electronic device, and a storage medium
By performing multi-level filtering of image features and target objects, the problem of wasted computational resources and low efficiency in existing image retrieval methods is solved, achieving efficient and accurate image retrieval, especially maintaining good results under dim or occluded conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2024-02-29
- Publication Date
- 2026-07-07
AI Technical Summary
Existing image retrieval methods waste computational resources and have low retrieval efficiency, especially in low-light conditions or under occlusion.
By acquiring the features of the image to be retrieved and calculating its similarity with pre-stored images, the first target image with high similarity is initially selected. Then, the target object is cropped and its features are extracted, and further filtering is performed in combination with the target category, reducing multiple sorting operations and improving retrieval efficiency and accuracy.
It effectively reduces the waste of computing resources, improves the efficiency and accuracy of image retrieval, and can maintain good retrieval results even in special scenarios.
Smart Images
Figure CN118210939B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image retrieval method, electronic device, and storage medium. Background Technology
[0002] With the explosive growth of computer applications, many important downstream application technologies have also emerged in the computer field, such as the technology of using computers to analyze and process images, and to perform similarity matching and retrieval of images in image databases.
[0003] In practice, the inventors of this application have found that current image retrieval methods typically take a query image as input, analyze the feature vector of the query image, calculate the similarity difference with images in the database, and return the image result most similar to the query result. This requires multiple sorting operations, resulting in wasted computing resources and increased time consumption. Furthermore, the retrieval effect is poor for images under dim conditions or with occlusion. Alternatively, descriptive tags or annotations are added to each image in the database, and related images are searched using provided keywords or query statements during retrieval. This easily leads to excessive repetitive operations, resulting in wasted computing resources and affecting image retrieval efficiency. Summary of the Invention
[0004] The main technical problem addressed by this application is to provide an image retrieval method, electronic device, and storage medium that can effectively improve image retrieval efficiency, avoid wasting computing resources, and effectively improve the accuracy of image retrieval.
[0005] To address the aforementioned technical problems, this application provides an image retrieval method comprising: responding to an image retrieval request for an image to be retrieved, acquiring a first feature of the image to be retrieved, and determining at least one first target image from a pre-stored image based on the first feature and a second feature of a pre-stored image; acquiring a target object and a target category of each first target image, determining at least one second target image from the first target images based on the target object, and acquiring a third feature of the target object in the second target image; acquiring a fourth feature of the target to be retrieved in the image to be retrieved, and determining a retrieval target image of the image to be retrieved from the at least one first target image based on the fourth feature, the third feature, and the target category.
[0006] In some embodiments, the step of obtaining a first feature of the image to be retrieved in response to an image retrieval request for the image to be retrieved, and determining at least one first target image from a pre-stored image based on the first feature and a second feature of a pre-stored image, includes: in response to an image retrieval request for the image to be retrieved, performing feature extraction on the image to be retrieved to obtain the first feature; obtaining a pre-stored image in a pre-trained retrieval model and the second feature of the pre-stored image, and calculating the similarity between the second feature and the first feature to determine the first similarity; if the first similarity is greater than a first threshold, then determining the pre-stored image corresponding to the first similarity as the first target image.
[0007] In some embodiments, obtaining the target object and the target category of each first target image, determining at least one second target image from the first target image based on the target object, and obtaining a third feature of the target object in the second target image includes: determining the target category of the target object in each first target image; determining at least one second target image in the first target image based on the outline of the target object; wherein the second target image is a part of the first target image; and obtaining a third feature of the target object in the second target image corresponding to the target category based on the outline of the target object.
[0008] In some embodiments, the method further includes: determining whether the target object in the first target image is complete; if the target object is incomplete, fusing the third feature of the second target image corresponding to the target object with the fourth feature of the target object in the image to be retrieved to obtain a first fused feature; and saving the first fused feature to the retrieval model to update the third feature corresponding to the target object in the second target image of the retrieval model.
[0009] In some embodiments, the method further includes: when the target category corresponding to the target to be retrieved is missing in the retrieval model, the fourth feature of the target to be retrieved in the image to be retrieved corresponding to the target category is saved to the retrieval model to update the target category of the target object in the first target image of the retrieval model.
[0010] In some embodiments, obtaining a fourth feature of the target to be retrieved in the image to be retrieved, and determining a retrieval target image of the image to be retrieved from at least one first target image based on the fourth feature, the third feature, and the target category, includes: calculating the similarity between the fourth feature and the third feature to determine a second similarity; filtering initial retrieval target images from the first target images according to the target category; sorting the initial retrieval target images according to the second similarity, and using the initial retrieval target image with the highest second similarity as the final retrieval target image.
[0011] In some embodiments, the method further includes: if the second similarity is greater than a second threshold, associating the second feature corresponding to the pre-stored image with the target category of the target object and saving it to the retrieval model to update the second feature of the pre-stored image in the retrieval model.
[0012] In some embodiments, after acquiring the target image of the image to be retrieved, the method further includes: marking the target in the image to be retrieved and tracking it in real time; determining whether the target meets a preset alarm rule; if the alarm rule is not met, it is marked as normal; if the alarm rule is met, it is marked as abnormal and an alarm is triggered; wherein the alarm rule is that the position or shape of the target to be retrieved is abnormal.
[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, the electronic device including a memory and a processor coupled to the memory, the memory storing at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the above-mentioned image retrieval method.
[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium having at least one program, which, when loaded and executed by a processor, is used to implement the above-mentioned image retrieval method.
[0015] Unlike current technologies, the image retrieval method provided in this application includes: in response to an image retrieval request for an image to be retrieved, obtaining a first feature of the image to be retrieved, and determining at least one first target image from the pre-stored images based on the first feature and a second feature of the pre-stored images; obtaining a target object and a target category of the target object in each first target image, determining at least one second target image from the first target images based on the target object, and obtaining a third feature of the target object in the second target image; obtaining a fourth feature of the target object to be retrieved in the image to be retrieved, and determining a retrieval target image of the image to be retrieved from at least one first target image based on the fourth feature, the third feature, and the target category; that is, in this application, retrieval target images that meet the image retrieval request can be filtered from the pre-stored images by using the target category and features of the target object, which can effectively improve image retrieval efficiency, avoid waste of computing resources, and effectively improve the accuracy of image retrieval. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:
[0017] Figure 1 This is a flowchart illustrating an embodiment of the image retrieval method of this application;
[0018] Figure 2 This is a flowchart illustrating an embodiment of step S10 in this application;
[0019] Figure 3 This is a flowchart illustrating an embodiment of step S20 in this application;
[0020] Figure 4 This is a flowchart illustrating an embodiment of step S21 in this application;
[0021] Figure 5 This is a flowchart illustrating an embodiment of step S30 in this application;
[0022] Figure 6 This is a schematic flowchart of an embodiment of obtaining the target image in this application;
[0023] Figure 7 This is a schematic diagram of the structure of an embodiment of the image retrieval system in this application;
[0024] Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device in this application;
[0025] Figure 9 This is a schematic diagram of an embodiment of a computer-readable storage medium in this application. Detailed Implementation
[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0028] Current image retrieval methods are content-based. They extract low-level features (such as color, texture, and shape) to represent each image as a feature vector, and then use similarity metrics (such as Euclidean distance and cosine similarity) to compare the similarity between different images. During retrieval, a query image is typically provided as input. The system analyzes the feature vector of the query image, calculates the similarity difference with images in the database, and returns the image most similar to the query image. Another common image retrieval technique is tag- or annotation-based image retrieval. Users add descriptive tags or annotations to each image in the database, and relevant images are searched using user-provided keywords or queries. However, retrieval in complex scenes requires significant computation, impacting retrieval efficiency, and complex backgrounds also affect retrieval accuracy.
[0029] Current solutions involve obtaining the hash features of the image to be retrieved; based on the hash features, determining the target cluster corresponding to the hash features of the image to be retrieved from at least two clusters; obtaining the ranking information of each image in the target cluster (similarity information between the images in the target cluster and the image to be retrieved, and image category information); and then ranking the images in the target cluster based on the ranking information to obtain an image sequence, ultimately determining the retrieval result for the image to be retrieved. However, obtaining relatively accurate features by determining the cluster corresponding to the hash features of the image to be retrieved from at least two clusters, obtaining the ranking information (similarity, category, etc.) of that cluster, and then ranking the images in the target cluster based on the ranking information requires two or more ranking operations. This results in a waste of computing resources and an increase in overall time consumption. Moreover, features extracted in this way may not have good retrieval results when retrieving images in special scenarios, such as images under dim conditions or images under occlusion.
[0030] Therefore, an image retrieval method is proposed that can effectively improve image retrieval efficiency, avoid wasting computing resources, and effectively improve the accuracy of image retrieval.
[0031] First, an initial screening process is performed. The similarity between the image to be detected and pre-stored images is calculated, and the first target image with the highest similarity is extracted. Then, a general retrieval model performs image matting on the target object in the extracted first target image, and a general target recognition model extracts the target features of the target object to be retrieved. A similarity threshold is set, and first target images below the similarity threshold are filtered out, removing those with too low similarity or that do not meet the requirements. In this way, highly similar target images can be obtained without sorting a massive number of pre-stored images, reducing time consumption and ensuring the accuracy of the retrieval results.
[0032] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the image retrieval method of this application; it should be noted that, if there is a substantial result, the method of this application does not rely on... Figure 1 The sequence of processes shown is limited.
[0033] like Figure 1 As shown, the image retrieval method includes the following steps:
[0034] S10. In response to an image retrieval request for an image to be retrieved, obtain a first feature of the image to be retrieved, and determine at least one first target image from the pre-stored images based on the first feature and a second feature of the pre-stored images.
[0035] The image to be retrieved is the image that needs to be retrieved. It can be output by the user or obtained based on the image retrieval request. The pre-stored image is a sample image that is pre-stored in the retrieval model. It can be obtained after pre-training or after training the current image to be retrieved.
[0036] Specifically, after receiving an image retrieval request for the image to be retrieved, the retrieval model responds to the image retrieval request by obtaining the first feature of the image to be retrieved and extracting the second feature of the pre-stored image from the database of the retrieval model. Then, it compares the first feature with the second feature of the pre-stored image to determine at least one first target image from the pre-stored images.
[0037] In some embodiments, the first feature of the image to be retrieved may be a full-image feature, used to represent the entire image to be retrieved; the second feature of the pre-stored image may be a full-image feature, used to represent the entire pre-stored image.
[0038] S20. Obtain the target object and target category of each first target image, determine at least one second target image from the first target image based on the target object, and obtain the third feature of the target object in the second target image.
[0039] Some of the first target images contain a target object, while others may not contain a target image. The target object may be a part of the first target image, so the second target image may be a part extracted from the first target image.
[0040] Specifically, after determining at least one first target image, the target object to be retrieved in each first target image is obtained, and the target category of the target object is determined; then, the first target image is cropped or cut out according to the area where the target object is located to obtain a second target image containing the target object, and then the third feature corresponding to the target object in the second target image is obtained.
[0041] In some embodiments, the third feature is a target feature of the target object, used to represent the target object.
[0042] S30. Obtain the fourth feature of the target to be retrieved in the image to be retrieved, and determine the retrieval target image of the image to be retrieved from at least one first target image based on the fourth feature, the third feature and the target category.
[0043] Some of the images to be searched also contain the target to be searched, and the target to be searched in the images to be searched corresponds to the target object in the first target image.
[0044] Specifically, after obtaining the third feature of the target object in the second target image, the fourth feature of the target to be retrieved and its corresponding target to be retrieved are obtained from the image to be retrieved. Then, the fourth feature of the target to be retrieved and the third feature of the target object are compared, and the retrieval target image of the image to be retrieved is determined from at least one first target image according to the target category.
[0045] In some embodiments, the fourth feature is a target feature of the target to be retrieved, used to represent the target to be retrieved.
[0046] In this embodiment, a preliminary screening is first performed using the first feature of the image to be retrieved and the second feature of the pre-stored image to select at least one first target image. Then, based on the target object in the first target image, a portion is cropped from the first target image as a second target image. Next, a second screening is performed using the third feature of the target object in the second target image, the fourth feature of the target to be retrieved in the image to be retrieved, and the target category to determine the retrieval target image from the first target image. That is, this application performs two-stage feature screening and avoids multiple sorting, which can effectively reduce the waste of computing resources, improve the efficiency of image retrieval, and improve the accuracy of image retrieval.
[0047] In some embodiments, a similarity calculation can be performed between a first feature of the image to be retrieved and a second feature of a pre-stored image to determine a first target image.
[0048] See Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of step S10 in this application.
[0049] like Figure 2 As shown, step S10 includes the following steps:
[0050] S11. In response to an image retrieval request for the image to be retrieved, feature extraction is performed on the image to be retrieved to obtain the first feature.
[0051] Feature extraction here refers to the extraction of features from the entire image to be retrieved, meaning the first feature is the full-image feature.
[0052] Specifically, after obtaining an image retrieval request for the image to be retrieved, the retrieval model responds to the image retrieval request by performing full-image feature extraction on the image to be retrieved, so as to obtain the first feature representing the image to be retrieved.
[0053] In some embodiments, the full-image features can be features such as color features, texture features, shape features, etc., or one or more combinations thereof, which can represent the image to be retrieved.
[0054] S12. Obtain the pre-stored image and the second feature of the pre-stored image from the pre-trained retrieval model, and calculate the similarity between the second feature and the first feature to determine the first similarity.
[0055] The retrieval model is obtained through pre-training. The pre-trained retrieval model stores pre-stored images, which can be obtained by pre-training on images to be retrieved acquired in the previous time period.
[0056] Specifically, after obtaining the first feature of the image to be retrieved, a pre-stored image is extracted from the pre-trained retrieval model, and the second feature of the pre-stored image is extracted; then, the first feature of the image to be retrieved and the second feature of the pre-stored image are used to calculate the similarity, and the similarity between the image to be retrieved and the pre-stored image is compared to obtain the first similarity.
[0057] In some embodiments, the similarity calculation of the first similarity can be Euclidean distance calculation, cosine similarity calculation, or other similarity calculation methods, as long as the similarity between the two images can be determined.
[0058] S13. If the first similarity is greater than the first threshold, then the pre-stored image corresponding to the first similarity is determined as the first target image.
[0059] The first threshold is set to filter pre-stored images with low similarity, and can be set according to actual needs.
[0060] Specifically, after obtaining the first similarity between each image to be retrieved and each pre-stored image, each pre-stored image with a similarity less than or equal to a first threshold is excluded, while the pre-stored image with a first similarity greater than the first threshold is taken as the first target image. Since there may be multiple pre-stored images with a first similarity greater than the first threshold, the first target image can be at least one.
[0061] In this embodiment, similarity is calculated by using the first feature of the image to be retrieved and the second feature of the pre-stored image to determine the first similarity between the image to be retrieved and the pre-stored image. Then, preliminary screening is performed based on the first threshold to select at least one first target image from the pre-stored images, providing a first target image for subsequent further screening, which can effectively improve image retrieval efficiency.
[0062] In some embodiments, a second target image can be cropped from the first target image based on the target object in the first target image.
[0063] See Figure 3 , Figure 3 This is a flowchart illustrating an embodiment of step S20 in this application.
[0064] like Figure 3As shown, step S20 includes the following steps:
[0065] S21. Obtain the target object in each first target image and determine the target category of the target object in each first target image.
[0066] The target object is the small target to be retrieved, which is a part of the first target image and corresponds to the target to be retrieved in the image to be retrieved; the target category of the target object is the classification of the target object type, and different types of target objects are classified into different target categories.
[0067] Specifically, after acquiring at least one first target image, the target object in each first target image is identified and acquired, and the acquired target objects are classified into target categories to determine the target category corresponding to the target object in the first target image.
[0068] S22. In the first target image, at least one second target image is determined by the outline of the target object; wherein the second target image is a part of the first target image.
[0069] Specifically, after acquiring the target object in the first target image, the first target image is cropped based on the outline area of the target object, and the cropped portion is used as the second target image.
[0070] S23. Based on the outline of the target object, obtain the third feature of the target object in the second target image corresponding to the target category.
[0071] The outline of the target object is the region of the target object in the first target image.
[0072] Specifically, after acquiring at least one second target image, the third feature of the target object in the second target image corresponding to the target category is acquired, and the target object in the second target image is identified by the third feature. Since the second target image is a part cropped from the first target image, the third feature can also represent the target object in the first target image.
[0073] In this embodiment, the first target image is cropped by cropping the area where the target object is located, so as to determine the second target image from the first target image, which provides a basis for subsequent image retrieval with small targets and can effectively improve the accuracy of image retrieval.
[0074] In some embodiments, under special circumstances (dim lighting, unclear target, target occlusion, etc.), the target object in the first target image may be incomplete, affecting the accuracy of subsequent image retrieval. Therefore, it is necessary to fuse the characteristic information of the target object with the features of the image to be retrieved in order to update the third feature of the target object in the retrieval model.
[0075] See Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of step S21 in this application.
[0076] like Figure 4 As shown, step S21 includes the following steps:
[0077] S211. Determine whether the target object in the first target image is complete.
[0078] In order to obtain more accurate search results, it is necessary to judge the completeness of the target object.
[0079] Specifically, after acquiring the target object in the first target image, the outline of the target object is judged to determine whether the target object in the first target image is complete.
[0080] S212. If the target object is incomplete, the third feature corresponding to the target object is fused with the fourth feature of the target object in the image to be retrieved to obtain the first fused feature.
[0081] In order to avoid the impact of incomplete target objects on the accuracy of search results, it is necessary to fuse the features of the target object and the characteristics of the target to be detected in order to improve search accuracy.
[0082] Specifically, after determining that the target object in the first target image is incomplete, the third feature corresponding to the target object and the fourth feature of the target to be retrieved in the image to be retrieved are fused to obtain the first fused feature.
[0083] S213. Save the first fusion feature to the retrieval model to update the third feature corresponding to the target object in the second target image of the retrieval model.
[0084] The first fusion feature will be saved in the retrieval model because the first fusion feature corresponds to the target object. Therefore, the third feature corresponding to the target object in the second target image of the retrieval model can be updated according to the first fusion feature.
[0085] Specifically, after obtaining the first fusion feature of the corresponding target object, the first fusion is saved to the retrieval model, and the third feature corresponding to the target object in the second target image in the retrieval model is updated according to the first fusion feature.
[0086] In this embodiment, the accuracy of image retrieval can be effectively improved in special environments such as unclear targets due to dim lighting conditions or partial occlusion of the target to be detected.
[0087] In some embodiments, for target categories that are not present in the retrieval model, the missing target categories can be saved to the retrieval model while the image retrieval process is being performed, so as to update the target category of the target object in the first target image of the retrieval model.
[0088] Specifically: when the retrieval model lacks the target category corresponding to the target to be retrieved, the fourth feature of the target to be retrieved in the image to be retrieved corresponding to the target category is saved to the retrieval model to update the target category of the target object in the first target image of the retrieval model, so that the corresponding target category in the retrieval model is complete, thereby improving the accuracy of subsequent image retrieval.
[0089] In some embodiments, the fourth feature of the target to be retrieved and the third feature of the target object can be compared, and the retrieval target image can be filtered from the first target image based on the target type.
[0090] See Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of step S30 in this application.
[0091] like Figure 5 As shown, step S30 includes the following steps:
[0092] S31. Obtain the fourth feature of the target in the image to be retrieved, and calculate the similarity between the fourth feature and the third feature to determine the second similarity.
[0093] The fourth feature is the characteristic information of the target to be retrieved in the image to be retrieved, and the third feature is the feature of the target object in the second target image. Since the target to be retrieved corresponds to the target object, the similarity between the fourth feature and the third feature can be calculated.
[0094] Specifically, after obtaining the third feature of the target object in the second target image, the fourth feature of the target to be retrieved in the image to be retrieved is obtained. Then, the similarity between the fourth feature of the target to be retrieved in the image to be retrieved and the third feature of the target object in the second target image is calculated. The similarity between the target to be retrieved in the image to be retrieved and the target object in the second target image is compared to obtain the second similarity.
[0095] In some embodiments, the similarity calculation of the second similarity can be Euclidean distance calculation, cosine similarity calculation, or other similarity calculation methods, as long as the similarity between the target object in the image to be retrieved and the target object in the second target image can be determined.
[0096] S32. Select the initial search target image from the first target image according to the target category.
[0097] Since the second target image is a part of the first target image, the target object in the second target image is also in the first target image. The target category can correspond to the target object in the first target image or the target object in the second target image.
[0098] Specifically, based on the target category of the target object, the first target image containing the target object can be filtered out to obtain the initial search target image; if the target category is missing, it is considered that the target object is missing, and the corresponding first target image is excluded.
[0099] S33. Sort the initial search target images according to the second similarity, and use the initial search target image with the highest second similarity as the final search target image.
[0100] The second similarity can represent the similarity between the target image to be retrieved in the image to be retrieved and the target object in the first target image. Since the initial target image to be retrieved is selected from the first target image, the final target image to be retrieved can be selected based on the second similarity.
[0101] Specifically, after filtering the first target image according to the target category to obtain the initial search target image, the initial search target image is further filtered according to the obtained second similarity, that is, the initial search target image is sorted according to the value of the second similarity, and then the initial search target image with the highest second similarity is taken as the final search target image.
[0102] In this embodiment, similarity is calculated by using the fourth feature of the target in the image to be retrieved and the third feature of the target object in the second target image. Then, the final target image is determined based on the obtained second similarity and target category, which can further improve the accuracy of image retrieval.
[0103] In some embodiments, the method further includes: if the second similarity is greater than the second threshold, associating the second feature corresponding to the pre-stored image with the target category of the target object and saving it to the retrieval model to update the second feature of the pre-stored image in the retrieval model.
[0104] In some embodiments, after obtaining the target image of the image to be retrieved, alarm rules can be set to trigger an alarm when an anomaly occurs in the target image.
[0105] See Figure 6 , Figure 6 This is a schematic flowchart of an embodiment of obtaining the target image in this application.
[0106] See Figure 6 After obtaining the target image of the image to be retrieved, the following steps are also included:
[0107] S34. Mark the target to be retrieved in the image to be retrieved and track it in real time.
[0108] Among them, real-time tracking allows for real-time image retrieval of the target object in the acquired image to be retrieved.
[0109] Specifically, after obtaining the target image corresponding to the image to be retrieved, the target to be retrieved in the image to be retrieved is determined and the target to be retrieved in the image to be retrieved is marked in real time.
[0110] S35. Determine whether the target to be searched meets the preset alarm rules. If the alarm rules are not met, mark it as normal; if the alarm rules are met, mark it as abnormal and trigger an alarm.
[0111] The alarm rule is that the location or shape of the target to be searched is abnormal.
[0112] Specifically, after marking and tracking the target in the image to be retrieved in real time, it is determined whether the position or shape of the target has become abnormal and meets the preset alarm rules. If the position or shape of the target has become abnormal, such as changing position or shape, it means that the preset alarm rules have been met, and the target is marked as an abnormal situation and an alarm is triggered to remind relevant personnel to handle it. If the target has not changed, it means that the alarm rules have not been met, and the target is marked as normal.
[0113] In some embodiments, if the image to be retrieved belongs to a special scene (dim lighting, unclear target, target occlusion, etc.), the user can choose to enable the temporary feature extraction option, that is, to temporarily extract features from the target object in the image to be detected, and then fuse the newly extracted features with the original image features of the image to be retrieved to obtain fused features. The fused features are then stored in the retrieval model for use in this retrieval. After the retrieval is completed, the user can choose whether to delete the image features compared in this instance. This operation can solve the problem of poor image retrieval results in special scenes and avoid the redundant operation of performing feature extraction every time a retrieval request is received.
[0114] The extracted features are stored in the database of the retrieval model. When a retrieval request is received, the feature file with the highest similarity to the image to be retrieved is searched in the database for comparison. The target tracking model is used to track the target in the video in real time and mark the detection box. When the target becomes abnormal at a certain moment and meets the alarm rules, it is marked as a target alarm box (the detection box is green and the alarm box is red) and the alarm device is triggered so that relevant personnel can deal with it in time.
[0115] For example, the system can detect whether trash cans in the community are overflowing, whether traffic cones on the road are placed correctly, or whether crash barriers on the road are damaged. Based on real-time tracking, it can determine whether the target to be searched meets the preset alarm rules and issue an alarm if the alarm rules are met.
[0116] In this embodiment, by tracking the target in the image to be retrieved in real time, it is possible to quickly determine whether the state of the target has changed and issue an alarm for abnormal situations in a timely manner.
[0117] In some embodiments, this application can be divided into two modes: feature extraction mode and real-time detection alarm mode. The feature extraction mode can extract sample features with high accuracy and store them in the retrieval model for direct use during retrieval. The real-time detection alarm mode can track and mark the retrieved target, which is divided into detection mark and alarm mark. When it is an alarm mark, the alarm device will be triggered to alert relevant personnel to quickly handle the abnormal situation.
[0118] The feature extraction modes are as follows:
[0119] First, video footage to be detected is acquired using a video capture device. This footage is then processed to obtain the video stream information for each frame, i.e., the image to be retrieved. Next, the similarity is calculated based on the feature information of the image to be retrieved and a pre-stored image, yielding the first similarity between the pre-stored template image and the image to be retrieved. A first threshold is set, and template samples with high first similarity (the first threshold is set by the user; if not set, the default value is used) are extracted from the pre-stored images as the first target images. Samples meeting the requirements are saved, thus completing the initial screening. Then, a general object detection model is used to extract features from the entire template sample image. The target object to be detected in the entire image is then cut out to obtain a smaller image, i.e., the second target image. A general object recognition model is used to extract features from the target object in the second target image, and the similarity is calculated between this second target image and the target object in the image to be recognized, yielding the second similarity. A similarity threshold is set, i.e., a second threshold is set. Based on the second threshold, the first target image is screened and filtered. First target images with similarity below the threshold or that do not meet the requirements are deleted. The remaining first target images are the high-quality template samples. They are associated with the target category of the target object and saved in the retrieval model for subsequent use. In case of special retrieval requests, such as unclear targets due to dim lighting conditions or partial occlusion of the target to be detected, the retrieval effect is not good. Users can choose to perform temporary feature extraction, that is, extract the features of the target to be retrieved from the image to be retrieved, fuse the extracted features with the features of the target object in the retrieval model, and save them to the retrieval model. The retrieval model is then selected for comparison to improve the retrieval rate. After the retrieval is completed, users can choose to delete the new features obtained by this fusion to save the memory of the comparison library, or not delete them for use in the next retrieval of similar scenarios.
[0120] The real-time detection alarm modes are as follows:
[0121] When a retrieval request is received, the target is first detected in each frame of the video using a target detection model. Based on the specific target category input by the user, the retrieval model is searched. If found, feature comparison is performed and subsequent steps are executed. If not found, a message is displayed indicating that the target category does not exist, and feature extraction is performed to improve the retrieval model's resources; this is called feature extraction mode. After successful comparison, the detected target object is marked. If the alarm rules are not met, it is marked as a green target detection box; if the alarm rules are met, it is marked as a red target alarm box. The target is tracked in real-time using a target tracking model. When a marked target meets the alarm rules at a certain moment, an alarm device is triggered to alert relevant personnel to handle the abnormal situation promptly.
[0122] In this embodiment, the first similarity between the image to be retrieved and the pre-stored image is calculated, and template samples with high first similarity are extracted, i.e., the first target image. Next, the full-image features of the first target image are extracted using a general object detection model. Then, the target object to be retrieved is extracted again by cutting out the image in the full image, i.e., cutting out the image to obtain the second target image. The second feature extraction obtains the features of the target object in the second target image. The features of the target object are then compared with the characteristics of the target object in the image to be retrieved to obtain the second similarity. A similarity threshold is then set to filter out the first target images that do not meet the second similarity requirements. After obtaining high-quality image features and associating them with the category to which the target belongs, they are saved into the retrieval model. This eliminates the need for multiple sorting operations to obtain image features, greatly reducing the time consumption. Moreover, when detecting images in special scenes, users can also choose to perform temporary feature extraction on the target object in the image to be detected to ensure the accuracy of the retrieval. During image retrieval, the system searches for and compares the features to be retrieved in the retrieval model. If the comparison is successful, the retrieved target object is marked. If the alarm rules are not met, it is marked as a green detection box; if the alarm rules are met, it is marked as a red alarm box. The target object is tracked in real time using a target tracking model. If the target object meets the alarm rules at a certain moment in the video footage, it is marked as a red alarm box, and the alarm device is triggered to issue an alarm, so that relevant personnel can handle the situation quickly, further improving the product's practicality.
[0123] See Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of the image retrieval system in this application.
[0124] like Figure 7 As shown, the image retrieval system 400 includes: a first determining module 410, a second determining module 420, and a third determining module 430; wherein, the first determining module 410, in response to an image retrieval request for an image to be retrieved, acquires a first feature of the image to be retrieved, and determines at least one first target image from the pre-stored images based on the first feature and a second feature of the pre-stored images; the second determining module 420 is used to retrieve the target object and the target category of the target object in each first target image, determine at least one second target image from the first target images based on the target object, and acquire a third feature of the target object in the second target image; the third determining module 430 is used to acquire a fourth feature of the target to be retrieved in the image to be retrieved, and determine the retrieval target image of the image to be retrieved from at least one first target image based on the fourth feature, the third feature, and the target category.
[0125] See Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the electronic device described in this application. This electronic device can perform the image retrieval step in the above-described method.
[0126] like Figure 8 As shown, the electronic device 500 includes a memory 520, a processor 510 coupled to the memory, and at least one computer program stored in the memory 520 and executable on the processor 510. When the processor 510 loads and executes the at least one computer program, it implements the image retrieval step in the above method. For related details, please refer to the detailed description in the above method, which will not be repeated here.
[0127] Please see Figure 9 , Figure 9 This is a schematic diagram of an embodiment of a computer-readable storage medium in this application.
[0128] like Figure 9 As shown, the computer-readable storage medium 600 stores at least one program 610, which, when loaded and executed by a processor, is used to implement the image retrieval step in the above method. For related details, please refer to the detailed description in the above method; it will not be repeated here.
[0129] The above scheme obtains the first target image by initially screening the full-image features of the image to be retrieved and the pre-stored image using a similarity threshold. Then, it further screens the target features of the target object in the first target image and the target object in the image to be retrieved using a similarity threshold. This can effectively improve the efficiency of image retrieval, avoid the waste of computing resources, and effectively improve the accuracy of image retrieval by using the target features of local images for screening in complex scenarios.
[0130] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of apparatuses or units, and may be electrical, mechanical, or other forms.
[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0132] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0134] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. An image retrieval method, characterized in that, include: In response to an image retrieval request for an image to be retrieved, a first feature of the image to be retrieved is obtained, and at least one first target image is determined from the pre-stored images based on the first feature and a second feature of the pre-stored images; The method involves obtaining a target object and its target category in each first target image, determining at least one second target image from the first target images based on the target object, and obtaining a third feature of the target object in the second target image. Specifically, the method involves determining the target category of the target object in each first target image; determining at least one second target image based on the outline of the target object in the first target image; the second target image being a part of the first target image; and obtaining the third feature of the target object in the second target image corresponding to the target category based on the outline of the target object. A fourth feature of the target to be retrieved in the image to be retrieved is obtained, and a retrieval target image of the image to be retrieved is determined from the at least one first target image based on the fourth feature, the third feature, and the target category. Specifically, a second similarity is determined by calculating the similarity between the fourth feature and the third feature; an initial retrieval target image is selected from the first target images based on the target category; the initial retrieval target images are sorted according to the second similarity, and the initial retrieval target image with the highest second similarity is used as the final retrieval target image.
2. The method according to claim 1, characterized in that, The step of responding to an image retrieval request for an image to be retrieved, obtaining a first feature of the image to be retrieved, and determining at least one first target image from a pre-stored image based on the first feature and a second feature of a pre-stored image, includes: In response to an image retrieval request for the image to be retrieved, feature extraction is performed on the image to be retrieved to obtain the first feature; Obtain the pre-stored image and the second feature of the pre-stored image from the pre-trained retrieval model, and calculate the similarity between the second feature and the first feature to determine the first similarity. If the first similarity is greater than the first threshold, then the pre-stored image corresponding to the first similarity is determined to be the first target image.
3. The method according to claim 2, characterized in that, Also includes: Determine whether the target object in the first target image is complete; If the target object is incomplete, the third feature corresponding to the target object and the fourth feature of the target object in the image to be retrieved are fused to obtain the first fused feature; The first fused feature is saved to the retrieval model to update the third feature corresponding to the target object in the second target image of the retrieval model.
4. The method according to claim 2, characterized in that, Also includes: If the target category corresponding to the target to be retrieved is missing in the retrieval model, the fourth feature of the target to be retrieved in the image to be retrieved corresponding to the target category is saved to the retrieval model to update the target category of the target object in the first target image of the retrieval model.
5. The method according to claim 2, characterized in that, Also includes: If the second similarity is greater than the second threshold, the second feature corresponding to the pre-stored image is associated with the target category of the target object and saved to the retrieval model to update the second feature of the pre-stored image in the retrieval model.
6. The method according to claim 1, characterized in that, After obtaining the target image for the image to be retrieved, the process further includes: The target to be retrieved in the image to be retrieved is marked and tracked in real time; Determine whether the target to be retrieved meets the preset alarm rules. If it does not meet the alarm rules, it is marked as a normal situation; if it meets the alarm rules, it is marked as an abnormal situation and an alarm is triggered. The alarm rule is that the location or shape of the target to be retrieved is abnormal.
7. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to the memory, the memory storing at least one computer program, which, when loaded and executed by the processor, is used to implement the image retrieval method as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium has at least one program that, when loaded and executed by a processor, is used to implement the image retrieval method as described in any one of claims 1-6.