Image retrieval method, device, storage medium, equipment and product

Abstract

This application provides an image retrieval method, apparatus, storage medium, device, and product, relating to the field of image processing. The method includes: acquiring a target image of a target object to be retrieved; extracting target material features corresponding to the target object; based on the target material features, acquiring at least one sample image from multiple sample images that has an object material similar to the target image, obtaining at least one candidate image; and acquiring candidate images similar to the target image from the at least one candidate image, obtaining an image retrieval result. The embodiments of this application can avoid misjudgments caused by using appearance features to filter images, thereby solving the problem of decreased retrieval accuracy when the object's projected shape changes, making the candidate images more relevant to the target object, and improving the accuracy and robustness of the image retrieval results.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to image retrieval methods, apparatus, storage media, devices, and products. Background Technology

[0002] In security inspection scenarios, to assist in the identification of objects to be retrieved, it is usually necessary to retrieve images of objects similar to the object to be retrieved from a large-scale sample image database. Since the placement and orientation of the object to be retrieved in security inspection scenarios are highly uncertain, how to stably and reliably retrieve images matching the object from a large number of sample images under different detection perspectives has become one of the urgent problems to be solved.

[0003] Therefore, traditional image retrieval methods mainly employ retrieval schemes based on visual appearance feature similarity. However, since X-ray imaging is essentially a two-dimensional projection of a three-dimensional object in different poses, the projection shape of the same object changes significantly when rotated, tilted, or stacked, resulting in vastly different image appearance features of the same object from different detection perspectives. Consequently, when the object's pose changes, traditional image retrieval methods experience a significant drop in the accuracy and recall of the retrieval results, thus reducing the accuracy and robustness of image retrieval. Summary of the Invention

[0004] In view of this, the embodiments of this application aim to provide an image retrieval method, apparatus, storage medium, device and product to solve the problem of low accuracy and robustness of image retrieval in the prior art.

[0005] In a first aspect, one embodiment of this application provides an image retrieval method, comprising: acquiring a target image collected for a target object to be retrieved, and extracting target material features corresponding to the target object to be retrieved; based on the target material features, acquiring at least one sample image from multiple sample images that is similar to the object material of the target image, thereby obtaining at least one candidate image; and acquiring candidate images similar to the target image from the at least one candidate image, thereby obtaining an image retrieval result.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, acquiring a target image of the target object to be retrieved and extracting target material features corresponding to the target object to be retrieved includes: acquiring images of the target object to be retrieved under different energy level rays to obtain a target image; based on the target image, determining the material response index corresponding to each pixel in the target region corresponding to the target object in the target image; and based on the material response index corresponding to each pixel, determining the target material features corresponding to the target object to be retrieved.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the target material features corresponding to the target object to be retrieved are determined based on the material response index corresponding to each pixel, including: for multiple preset material value ranges, according to the material response index corresponding to each pixel, the number of pixels falling into each preset material value range is counted; based on the number of pixels in each preset material value range, the target material features corresponding to the target object to be retrieved are determined.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the target material features corresponding to the target object to be retrieved are determined based on the number of pixels in each preset material value range, including: determining the proportion of each preset material value range corresponding to the number of pixels in each preset material value range; and determining the target material features corresponding to the target object to be retrieved based on the proportion of each preset material value range corresponding to the number of pixels in each preset material value range.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, based on the target material features, obtaining at least one sample image similar to the object material of the target image from multiple sample images to obtain at least one candidate image includes: determining the similarity between the target material features and the material features corresponding to the multiple sample images respectively, obtaining the material similarity between the target image and the multiple sample images respectively; based on the material similarity between the target image and the multiple sample images respectively, obtaining at least one sample image similar to the object material of the target image from the multiple sample images to obtain at least one candidate image.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, based on the material similarity between the target image and multiple sample images, at least one sample image with object material similarity to the target image is obtained from the multiple sample images to obtain at least one candidate image, including: determining the target number of sample images with the highest material similarity to the target image from the multiple sample images as candidate images to obtain at least one candidate image.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, obtaining candidate images similar to the target image from at least one candidate image to obtain image retrieval results includes: extracting visual features from the target image to obtain target visual features; determining the similarity between the target visual features and the corresponding visual features of the candidate images for at least one candidate image to obtain visual similarity between the target image and the candidate images; and obtaining candidate images similar to the target image from at least one candidate image based on the visual similarity and material similarity between the target image and at least one candidate image to obtain image retrieval results.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, based on the visual similarity and material similarity between the target image and at least one candidate image, candidate images similar to the target image are obtained from at least one candidate image to obtain image retrieval results, including: for at least one candidate image, performing weighted fusion processing on the visual similarity and material similarity between the target image and the candidate image to obtain a comprehensive similarity between the target image and the candidate image; and based on the comprehensive similarity between the target image and at least one candidate image respectively, obtaining candidate images similar to the target image from at least one candidate image to obtain image retrieval results.

[0013] Secondly, one embodiment of this application provides an image retrieval device, comprising: a feature extraction module, configured to acquire a target image collected for the target object to be retrieved, and extract target material features corresponding to the target object to be retrieved; a first acquisition module, configured to acquire at least one sample image similar to the object material of the target image from multiple sample images based on the target material features, thereby obtaining at least one candidate image; and a second acquisition module, configured to acquire candidate images similar to the target image from the at least one candidate image, thereby obtaining an image retrieval result.

[0014] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the image retrieval method described in the first aspect.

[0015] Fourthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to execute the image retrieval method described in the first aspect.

[0016] Fifthly, one embodiment of this application provides a computer program product including instructions that, when executed on an electronic device, cause the electronic device to implement the image retrieval method described in the first aspect.

[0017] In this application, the target material features of the object to be retrieved are first extracted, and candidate images with similar materials are selected based on these features, thereby filtering out interference samples with mismatched materials. Then, candidate images similar to the target image are determined to obtain the image retrieval results. Since material features are less affected by placement posture and detection angle than appearance visual features, their material features remain stable even if the placement posture changes. Therefore, image selection based on material features avoids misjudgments caused by using appearance features for image selection, thus solving the problem of decreased retrieval accuracy when the object's projection shape changes. This makes the candidate images more relevant to the target object to be retrieved, improving the accuracy and robustness of the image retrieval results. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 The diagram shown is a flowchart of an image retrieval method provided in an embodiment of this application.

[0020] Figure 2 The diagram shown is a flowchart illustrating the candidate image determination process provided in an embodiment of this application.

[0021] Figure 3 The diagram shown is a schematic diagram of material statistical fingerprint construction provided in an embodiment of this application.

[0022] Figure 4 The diagram shown is a structural schematic of an image retrieval device provided in an embodiment of this application.

[0023] Figure 5 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented even without certain specific details. In some instances, methods and means well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0027] Furthermore, the terms “first,” “second,” “third,” and “fourth” are used only for distinguishing descriptions and should not be interpreted as indicating or implying relative importance.

[0028] In related technologies, traditional image retrieval methods mainly employ retrieval schemes based on visual appearance feature similarity. However, since X-ray imaging is essentially a two-dimensional projection of a three-dimensional object in different postures, the projection shape of the same object will change significantly when rotated, tilted, or stacked, resulting in a large difference in the image appearance features of the same object under different detection perspectives.

[0029] For example, in a security check scenario, the shape of an X-ray projection of a knife-shaped object placed in a bag can vary significantly depending on whether the knife is positioned parallel, perpendicular, or at an angle to the X-ray beam. If the angle of the knife in the sample image differs from the angle of the captured object, searching based on visual appearance features may easily fail to find a match, resulting in missed detections. This significantly reduces the accuracy and recall of the search results, thereby lowering the accuracy and robustness of image retrieval.

[0030] To address the aforementioned technical issues, this application provides an image retrieval scheme. By first using stable material features to filter out candidate images with matching materials, and then performing subsequent similarity matching based on the candidate images, the scheme can effectively reduce the impact of object pose changes on the retrieval results and improve the accuracy and robustness of the retrieval results.

[0031] The following is combined Figures 1 to 2 The image retrieval method provided in this application is described in detail.

[0032] Figure 1 The diagram shown is a schematic flowchart of an image retrieval method provided in an embodiment of this application; as follows: Figure 1 As shown, the method includes the following steps.

[0033] Step S110: Obtain the target image collected for the target object to be retrieved, and extract the target material features corresponding to the target object to be retrieved.

[0034] Here, the target object to be retrieved is the object that needs to be retrieved. For example, in a security check scenario, the target object to be retrieved is the items in the luggage. The target image is an image obtained by collecting data on the target object to be retrieved. The target material features are features that can characterize the material composition of the object to be retrieved. For example, target material features include material response index, material fingerprint, or material type.

[0035] It should be noted that the target material feature reflects the distribution or composition characteristics of the internal physical material. Since the material composition of an object is relatively stable under multiple viewing angles, this target material feature will not change with the placement of the object to be retrieved or the change of the detection viewing angle.

[0036] In some embodiments, an image acquisition device can be used to acquire an image of the target object to be retrieved, thereby obtaining a target image. The image acquisition device can be an imaging sensor such as a camera, scanner, or spectral imager that can capture surface or transmission information of the target object to be retrieved.

[0037] In practice, there are various ways to extract the target material features corresponding to the target object to be retrieved. For example, the material fingerprint or reflectance distribution features of the target object to be retrieved can be extracted based on energy dispersive spectroscopy; or, other physical sensor data that can capture material information (such as spectral, polarization, ultrasonic, etc.) can be used to detect the target material features of the target object to be retrieved.

[0038] Step S120: Based on the target material features, at least one sample image with an object material similar to the target image is obtained from multiple sample images to obtain at least one candidate image.

[0039] The sample images are images of known objects collected in advance. For example, images of previously seized non-compliant items and common luggage items can be collected. For each sample image, the material characteristics of the object reflected in the sample image are determined in advance. The method for determining the material characteristics of the object in the sample image can be the same as the method for extracting the target material characteristics described above, and will not be repeated here.

[0040] In this embodiment, material similarity refers to the matching of the target object to be retrieved and the object in the sample image in terms of material composition. For example, if the target object to be retrieved is a steel knife, and the sample image shows a stainless steel knife, then the two objects can be said to have similar materials.

[0041] In practice, the material features of the object in each sample image can be compared with the target material features to obtain at least one sample image with a material similar to the target image, which is then used as a candidate image. For example, the material feature can be a material type, including metal, wood, plastic, ceramic, etc. When the material type corresponding to the target material matches the material type corresponding to the sample image, the two objects are considered similar in material, and the sample image is determined as a candidate image. For instance, if the target object to be retrieved is wooden chopsticks, then the target material feature is wood, and all sample images with the material type of wood can be extracted and used as candidate images.

[0042] Step S130: Obtain candidate images similar to the target image from at least one candidate image to obtain image retrieval results.

[0043] In some embodiments, visual features of an image can be used to obtain candidate images similar to the target image from at least one candidate image. For example, visual features of the candidate images and target visual features of the target image can be extracted from at least one candidate image that has a similar object material to the target image. By comparing the similarity between the visual features of the candidate images and the target visual features, candidate images that are visually similar to the target image are determined as image retrieval results.

[0044] In this application, the target material features of the object to be retrieved are first extracted, and candidate images with similar materials are selected based on these features, thereby filtering out interference samples with mismatched materials. Then, candidate images similar to the target image are determined to obtain the image retrieval results. Since material features are less affected by placement posture and detection angle than appearance visual features, their material features remain stable even if the placement posture changes. Therefore, image selection based on material features avoids misjudgments caused by using appearance features for image selection, thus solving the problem of decreased retrieval accuracy when the object's projection shape changes. This makes the candidate images more relevant to the target object to be retrieved, improving the accuracy and robustness of the image retrieval results.

[0045] In practical applications, the thickness of an object can interfere with the accuracy of material feature extraction, leading to deviations in the detected material features for objects of the same material but different thicknesses. To address this issue, this application also provides a preferred method. Optionally, acquiring a target image of the target object to be retrieved and extracting target material features corresponding to the target object to be retrieved includes: acquiring images of the target object to be retrieved under different energy level rays to obtain a target image; determining the material response index corresponding to each pixel in the target region corresponding to the target object in the target image based on the target image; and determining the target material features corresponding to the target object based on the material response index corresponding to each pixel.

[0046] Here, different energy levels of radiation can be, for example, X-rays of different energy levels. For instance, a dual-energy X-ray imaging device can be used to image the target object to be retrieved, obtaining a high-energy image acquired under high-energy radiation and a low-energy image acquired under low-energy radiation.

[0047] In some embodiments, a projected image of the target object can be reconstructed from images acquired under different energy level rays using image fusion or matter decomposition algorithms, and used as the target image. This target image retains visual appearance information such as the projected outline, internal structural edges, and texture patterns of the target object.

[0048] Furthermore, after obtaining the target image, a target detection algorithm can be used to identify the target object within the image, thereby segmenting a target region containing only the target object. For example, the target region is the smallest bounding rectangle of the target object. After determining the target region, the material response index corresponding to each pixel in the target region is calculated. The material response index characterizes the attenuation characteristics of the material corresponding to that pixel to different energy levels of radiation and is related to the atomic number and density of the material.

[0049] In some embodiments, the specific implementation of determining the material response index corresponding to each pixel in the target region corresponding to the target object to be retrieved based on the target image is as follows: for each pixel in the target region, the ratio of the pixel value corresponding to the pixel in the high-energy image and the high-energy reference image is determined to obtain a first ratio, wherein the high-energy reference image is an image acquired under high-energy rays for an unobstructed background; the ratio of the pixel value corresponding to the pixel in the low-energy image and the low-energy reference image is determined to obtain a second ratio, wherein the low-energy reference image is an image acquired under low-energy rays for an unobstructed background; and the material response index corresponding to the pixel is determined based on the first ratio and the second ratio.

[0050] Specifically, there are various ways to determine the material response index corresponding to a pixel based on the first ratio and the second ratio. For example, pixels in the target area... Pixel values ​​in high-energy images are represented as In low-energy images, pixel values ​​are represented as The pixel values ​​of an image acquired under high-energy rays with an unobstructed background are represented as follows: The pixel values ​​of an image acquired under low-energy rays with an unobstructed background are represented as follows: This pixel value characterizes the projection intensity received at the location of that pixel. According to the Beer-Lambert law, the pixel... Material Response Index Possible forms: This material response index can reduce the influence of the object's equivalent thickness, allowing it to primarily reflect material differences. For example, a thick steel plate and a thin steel plate, despite significant differences in grayscale on separate high-energy and low-energy images, will have similar calculated material response indices. It should be noted that the calculation method for the material response index provided in this embodiment is not limited to the above expression; any function capable of characterizing the difference in dual-energy attenuation can be used to determine the material response index.

[0051] Furthermore, after obtaining the material response index of each pixel, the statistical characteristics such as the average, median, and quantile of the material response index of each pixel in the target area can be used as the target material feature; alternatively, the material response index of each pixel in the target area can be directly used as the target material feature to retain the fluctuation information of the material response index in the full value range and to more comprehensively depict the composition of the material in the target area.

[0052] This application utilizes images under different energy level rays, which helps to reduce the interference of object thickness on the attenuation signal during the calculation of the material response index of each pixel, so that it accurately reflects the properties of the material itself, thereby avoiding deviations in material features caused by thickness differences, further improving the accuracy of material feature extraction, and ensuring the correctness of candidate image selection.

[0053] To obtain global features characterizing the overall material properties of the entire target object to be retrieved, embodiments of this application also provide an optimization scheme. Optionally, based on the material response index corresponding to each pixel, the target material features corresponding to the target object to be retrieved are determined, including: for multiple preset material value ranges, according to the material response index corresponding to each pixel, counting the number of pixels falling into each preset material value range; and based on the number of pixels in each preset material value range, determining the target material features corresponding to the target object to be retrieved.

[0054] The preset material value range refers to multiple pre-defined intervals based on the range of material response index values. For example, the range of material response index values, from minimum to maximum, can be divided into 10 preset material value ranges.

[0055] In practice, for each preset material value range, the number of pixels whose material response index falls within that range is determined. The number of pixels in each preset material value range is then used as a component of the feature vector. This feature vector is then used to form a material statistical fingerprint vector, which serves as the target material feature.

[0056] In a specific example, the material response index ranges from [0, 2]. After being divided into 10 preset material value intervals, each interval has a length of 0.2. Statistics show that within the target area, 12 pixels have a material response index falling within the interval [0, 0.2), and 25 pixels fall within the interval [0.2, 0.4). This process continues until all pixel counts for each preset material value interval are counted. Arranging the pixel counts for each preset material value interval in the order of the intervals yields a 10-dimensional material statistical fingerprint vector, which serves as the target material feature.

[0057] This application embodiment discretizes the continuous material response index into multiple intervals and counts the number of pixels in each interval, thereby transforming material information sensitive to a single pixel into a global description of the overall material composition of the object. When the object rotates or changes its posture, the target material features that reflect the overall material composition have higher stability than single-point features, which is beneficial to improving the accuracy of the retrieval results.

[0058] In practical applications, if the target object to be retrieved changes its posture, the number of pixels occupied by the target area in the target image acquired by the image acquisition device will also change. This change will cause differences in the number of pixels counted in each preset material value range, which will affect the target material characteristics. When selecting candidate images from the sample images based on the target material characteristics, the selection results will be biased.

[0059] To eliminate this deviation and avoid interference from fluctuations in the total number of pixels caused by pose changes on the statistical results of material features, embodiments of this application also provide further optimization methods. Optionally, based on the number of pixels in each preset material value interval, the target material features corresponding to the target object to be retrieved are determined, including: determining the proportion of pixels corresponding to each preset material value interval based on the number of pixels in each preset material value interval; and determining the target material features corresponding to the target object to be retrieved based on the proportion of pixels corresponding to each preset material value interval.

[0060] The quantity percentage refers to the proportion of the number of pixels that fall within the corresponding preset material value range to the total number of pixels in the target area.

[0061] In practice, the proportion of each preset material value interval is used instead of the number of pixels as a component of the feature vector. The feature vector components of each preset material value interval are combined to form a normalized material statistical fingerprint vector, which is used as the final target material feature. After normalization, the proportion of the number of pixels eliminates the influence of differences in the total number of pixels in the target area, ensuring that the target material features obtained for the same material object at different distances and of different sizes remain consistent.

[0062] For example, let the division be into If there are several preset material value ranges, then the pixel percentage of the k-th preset material value range is... for: in, This represents the number of pixels that fall within the k-th preset material value range. This represents the number of pixels falling within the j-th preset material value range. Correspondingly, this is the material statistical fingerprint vector. It can be represented as: Since the statistics are based on the overall proportions, when the target object changes its posture in three-dimensional space, the relative proportions of its overall material composition remain stable. Therefore, the determined target material features possess posture invariance.

[0063] This application embodiment uses normalized pixel ratio to characterize target material features, ensuring that the material features of the same object remain consistent under different poses. This eliminates the interference of pose changes on the material feature determination results, improves the stability of material features, and makes image retrieval results more accurate and reliable.

[0064] To ensure that the determined candidate images and the target image have similar material features, embodiments of this application also provide a method for determining candidate images based on material similarity. The following, in conjunction with... Figure 2 Describe in detail the process of determining candidate images based on material similarity.

[0065] Figure 2 The diagram shown is a schematic flowchart of a candidate image determination process provided in an embodiment of this application. Figure 1 Extending from the illustrated embodiment Figure 2 The illustrated embodiment will be described in detail below. Figure 2 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0066] like Figure 2 As shown, based on the target material features, at least one sample image with an object material similar to the target image is obtained from multiple sample images to obtain at least one candidate image (step S120 above) may include the following steps.

[0067] S210, determine the similarity between the target material features and the material features corresponding to the multiple sample images respectively, and obtain the material similarity between the target image and the multiple sample images respectively.

[0068] Material similarity refers to the degree of matching between the target material features and the corresponding material features in the sample image. It can be determined by calculating the Euclidean distance, Bach distance, and cosine similarity between the two features. A higher material similarity indicates a greater degree of similarity between the materials.

[0069] The following explanation uses Bach distance as an example to illustrate the calculation process of material similarity. Bach distance measures the similarity between two histogram distributions. When both the target material feature and the sample material feature are material statistical fingerprint vectors (i.e., histogram vectors) represented by histograms, the smaller the Bach distance, the closer the two histogram distributions are, and the higher the material similarity between them.

[0070] In practice, a first histogram vector representing the target material features of the target image and pre-stored second histogram vectors representing the sample material features of each sample image can be obtained. The first histogram vector consists of multiple first feature vector components; the second histogram vector consists of multiple second feature vector components. The similarity between the first histogram vector and each second histogram vector is measured using Bhattacharyya distance, and this is used as the material similarity. The calculation formula is as follows: in, This represents the first eigenvector component of the k-th preset material value range. This represents the second feature vector component of the k-th preset material value interval. This metric effectively quantifies the degree of overlap and similarity between two statistical distributions; the closer the value is to 1, the more similar the material distributions.

[0071] S220: Based on the material similarity between the target image and multiple sample images, at least one sample image with an object material similar to that of the target image is obtained from the multiple sample images to obtain at least one candidate image.

[0072] In practice, sample images with a material similarity greater than a similarity threshold can be identified as candidate images. For example, the similarity threshold can be set to 0.7. When the calculated material similarity is greater than 0.7, the corresponding sample image is selected as a candidate image.

[0073] This application provides a specific implementation method for determining candidate images. By measuring the degree of matching of material features between the target image and each sample image through material similarity, it is easy to quickly filter out candidate images with high material feature matching degree, effectively filter out irrelevant samples with large differences in material composition, reduce the amount of computation when determining candidate images similar to the target image, and improve the overall processing efficiency of image retrieval.

[0074] To balance retrieval efficiency and recall, this application embodiment also sets an upper limit on the number of selected candidate images, and determines candidate images based on this upper limit. Optionally, based on the material similarity between the target image and multiple sample images, at least one sample image with object material similarity to the target image is obtained from the multiple sample images to obtain at least one candidate image, including: determining the target number of sample images with the highest material similarity to the target image from the multiple sample images as candidate images to obtain at least one candidate image.

[0075] In practice, multiple sample images can be sorted from high to low according to material similarity, and the top target number of sample images after sorting can be selected as candidate images.

[0076] In some embodiments, the target number may be a pre-set fixed number, or it may be determined based on the total number of sample images. For example, the total number of sample images at a preset ratio may be used as the target number.

[0077] In practical applications, an excessively large number of targets will increase the computational load in the subsequent process of determining image retrieval results, while an excessively small number of targets may miss sample images that are truly similar in material to the target image, resulting in a decrease in retrieval recall. Therefore, the embodiments of this application can flexibly set the number of targets according to actual retrieval needs and computing power conditions, taking into account both computational efficiency and retrieval recall.

[0078] For example, to balance recall and computational efficiency, the target number can be the maximum between a first number and a second number. The first number is a fixed number, such as 50. The second number is 10% of the total number of sample images.

[0079] This application embodiment ensures that candidate images with high material similarity are selected while effectively controlling the number of candidate images. It avoids increasing the subsequent calculation load due to an excessive number of candidate images, and also avoids filtering out correct results due to an insufficient number of candidate images, thus balancing retrieval efficiency and retrieval accuracy.

[0080] To further improve the accuracy of search results, candidate images similar to the target image are identified from multiple dimensions. This application provides a preferred method for image detection, as detailed below.

[0081] In some embodiments, obtaining candidate images similar to the target image from at least one candidate image to obtain image retrieval results includes: extracting visual features from the target image to obtain target visual features; determining the similarity between the target visual features and the corresponding visual features of the candidate images for at least one candidate image to obtain visual similarity between the target image and the candidate images; and obtaining candidate images similar to the target image from at least one candidate image based on the visual similarity and material similarity between the target image and at least one candidate image to obtain image retrieval results.

[0082] Among them, visual features can be conventional image features such as shape features, contour features, texture features, and grayscale distribution features of the target object to be retrieved, which can be directly extracted from the target image through a deep learning model.

[0083] For example, a pre-trained convolutional neural network (such as ResNet-50) is used as a visual feature extractor. The feature vector of the target image is extracted by the visual feature extractor and used as the target visual feature of the target image.

[0084] In practice, after extracting the target visual features, the visual similarity between the target visual features and the corresponding sample visual features of each candidate image can be calculated. Specifically, the visual similarity can be obtained by calculating the cosine similarity between the target visual features and the corresponding sample visual features of each candidate image. The closer the visual similarity is to 1, the higher the degree of matching between the two visual features. For example, the formula for calculating visual similarity is as follows: SV(Q,Di)=FV(Q)·FV(Di) / (||FV(Q)||·||FV(Di)||) Where SV(Q, Di) represents the visual similarity between the target image Q and the candidate image Di, FV(Q) represents the target visual feature vector corresponding to the target image, FV(Di) represents the sample visual feature vector corresponding to the candidate image Di, "·" represents the inner product operation of vectors, and ||·|| represents the L2 norm of the vector.

[0085] In some embodiments, a third number of candidate images with the highest material similarity can be determined to form a first candidate set; a fourth number of candidate images with the highest visual similarity can be determined to form a second candidate set. Candidate images that exist in both the first and second candidate sets are identified as candidate images similar to the target image, and these candidate images can be used as image retrieval results.

[0086] This application embodiment performs a comprehensive search by combining information from two dimensions: material similarity and visual similarity. It not only uses material features to filter out irrelevant samples with large material differences, but also combines visual features to ensure appearance similarity. This enables multi-dimensional matching of target images and sample images, avoiding the problem of biased results caused by single-dimensional retrieval, and effectively improving the accuracy of image retrieval results.

[0087] To effectively fuse visual similarity and material similarity to form a more accurate similarity judgment, embodiments of this application can obtain a comprehensive similarity through weighted fusion. Optionally, based on the visual similarity and material similarity between the target image and at least one candidate image, candidate images similar to the target image are obtained from the at least one candidate image to obtain image retrieval results. This includes: for at least one candidate image, performing weighted fusion processing on the visual similarity and material similarity between the target image and the candidate image to obtain a comprehensive similarity between the target image and the candidate image; and based on the comprehensive similarity between the target image and at least one candidate image respectively, obtaining candidate images similar to the target image from the at least one candidate image to obtain image retrieval results.

[0088] In practice, for each candidate image, a weighted sum of visual similarity and material similarity can be calculated according to preset weights, and the sum is used as the comprehensive similarity. The preset weights can be flexibly adjusted according to the retrieval needs of the actual application scenario. For example, when the application scenario focuses more on material matching (such as detecting whether the object is a tool made of a hard material), the weight of material similarity can be increased; when the focus is more on appearance shape matching, the weight of visual similarity can be increased.

[0089] Furthermore, after obtaining the comprehensive similarity of all candidate images, the candidate images with a comprehensive similarity greater than a preset comprehensive threshold can be used as the final image retrieval result, or the candidate images ranked from high to low according to comprehensive similarity can be selected as the final image retrieval result.

[0090] This application embodiment integrates material similarity and visual similarity into a unified comprehensive similarity by weighted fusion, thereby more accurately reflecting the overall similarity between the target image and the candidate image, facilitating the rapid selection of candidate images that meet the requirements, and making the image retrieval results more reasonable.

[0091] The above text provides a detailed description of the embodiments corresponding to the image retrieval method. In order to enable those skilled in the art to further understand the technical solution of this method, specific application scenarios are given below.

[0092] In the airport's baggage security checkpoint, a piece of passenger luggage enters the existing dual-energy X-ray security screening equipment via a conveyor belt. The equipment completes the scan within seconds, simultaneously acquiring and outputting two digital images of the same piece of luggage under high-energy and low-energy X-rays. At this point, the image retrieval function deployed within the security checkpoint is triggered, and the entire retrieval process unfolds sequentially.

[0093] First, it is necessary to extract the material statistical fingerprint vector. For example... Figure 3 As shown, a target image is obtained for the target object to be retrieved (i.e., items contained in luggage). This target image includes a high-energy image and a low-energy image, and a pre-set reference image for the corresponding unoccluded background area. Using a target detection algorithm, region of interest extraction is performed on the high-energy image and the low-energy image, that is, the target area corresponding to the target object to be retrieved is extracted. For each pixel in the target area, the material response index that can effectively reduce the influence of the equivalent thickness of the object is calculated according to the Beer-Lambert law, resulting in a material response index map.

[0094] Then, the material response index of each pixel in the target area is converted into target material features with viewpoint and pose invariance. Specifically, material interval division is performed in advance, that is, the range of material response index values ​​is divided into K preset material value intervals. Every pixel in the target area is traversed, and the number of pixels falling into each preset material value interval is counted. However, simple count is easily affected by changes in the target area scale and object distance. Therefore, it is necessary to normalize the number of pixels in each interval, that is, to convert the number of pixels in each preset material value interval into the proportion of the total number in the entire target area (for example, the K preset material value intervals correspond to the proportions P1, P2, ... P...). K The material statistical fingerprint vector, composed of the proportion of each preset value interval, reflects the relative proportion of different materials within the target area and can be used as the target material feature of the target image. Since the material statistical fingerprint vector depicts the relative composition and distribution of materials within an object, when the same object rotates, tilts, or stacks in three-dimensional space, causing changes in its projected shape, the statistical characteristics of its overall material proportions remain basically stable. At this point, the target material feature reflecting the material composition of the object has been extracted.

[0095] After extracting the target material features, the first stage, coarse screening, begins. The airport security database has pre-built material fingerprint indexes for a large number of sample images. These material fingerprint indexes represent the material features corresponding to each sample image. Bhattacharyya distance is used to measure the material similarity between the target material features and the corresponding material features of each sample image. All sample images are sorted in descending order of material similarity, and the number of target images with the highest material similarity is selected as candidate images. The number of target images is adaptively determined based on the total size of the database, specifically the larger of 10% and 50 of the total number of sample images. This step retains candidate images with materials similar to the items, significantly reducing the computational load of the subsequent second-stage fine matching process.

[0096] Furthermore, based on the target visual features of the target image, a second stage of refined retrieval is performed on the candidate images. In this stage, a pre-trained convolutional neural network is used as a visual feature extractor. For both the target image and each candidate image, feature vectors are extracted as their visual representations. Then, the cosine similarity between the target image and each candidate image in the visual feature space is calculated and used as the visual similarity between the target image and the candidate images.

[0097] Finally, the two different similarity scores are weighted and fused to generate a comprehensive similarity score. The weights used in the fusion process allow the security inspection system to flexibly adjust them according to the actual scenario. For example, when faced with a large number of items that look similar but are made of different materials, the weight of the material item can be appropriately increased to enhance physical prior knowledge. Based on the comprehensive similarity score, the candidate images are ranked, and the images with the highest comprehensive similarity scores are returned as the final image retrieval results, which are then presented sequentially on the image judge's interface.

[0098] In this scenario, this image retrieval method improves security inspection efficiency and image judgment accuracy. Firstly, it utilizes existing dual-energy X-ray security inspection equipment for image retrieval, eliminating the need for additional hardware. Furthermore, because material statistical fingerprint vectors are highly insensitive to object thickness, rotation, and stacking posture, even if items are tilted or flipped in luggage, sample images with similar material composition can still be accurately identified in the first stage, avoiding the problem of poor retrieval accuracy caused by relying solely on appearance features. Actual operational data shows that even with viewing angle changes of 40° to 60°, the recall rate of this embodiment remains above 80%, far superior to the approximately 50% of traditional solutions. Secondly, through a two-layer cascaded architecture of coarse material screening followed by fine visual sorting, the retrieval time is reduced from several minutes requiring traversing the entire database for high-dimensional visual feature comparison to within seconds, meeting the business requirements of real-time processing, significantly reducing staff waiting time, and effectively improving the throughput of security checkpoints.

[0099] The above text combined Figures 1 to 3 The image retrieval method embodiments of this application are described in detail below, in conjunction with... Figure 3 This application describes in detail embodiments of the image retrieval device. It should be understood that the descriptions of the image retrieval method embodiments correspond to the descriptions of the image retrieval device embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.

[0100] Figure 4 The diagram shown is a structural schematic of an image retrieval device provided in an embodiment of this application. Figure 4 As shown, the image retrieval device 40 provided in this embodiment includes: Feature extraction module 410 is used to acquire a target image collected for the target object to be retrieved and extract the target material features corresponding to the target object to be retrieved. The first acquisition module 420 is used to acquire at least one sample image that is similar to the object material of the target image from multiple sample images based on the target material features, so as to obtain at least one candidate image; The second acquisition module 430 is used to acquire candidate images similar to the target image from at least one candidate image to obtain image retrieval results.

[0101] Based on any optional technical solution in the embodiments of this application, the feature extraction module 410 is optionally further configured to acquire a target image collected for the target object to be retrieved, and extract the target material features corresponding to the target object to be retrieved, including: acquiring images of the target object to be retrieved under different energy level rays to obtain a target image; determining the material response index corresponding to each pixel in the target region corresponding to the target object to be retrieved based on the target image; and determining the target material features corresponding to the target object to be retrieved based on the material response index corresponding to each pixel.

[0102] Based on any optional technical solution in the embodiments of this application, the feature extraction module 410 is optionally further configured to: for multiple preset material value ranges, count the number of pixels falling into each preset material value range according to the material response index corresponding to each pixel; and determine the target material features corresponding to the target object to be retrieved based on the number of pixels in each preset material value range.

[0103] Based on any optional technical solution in the embodiments of this application, the feature extraction module 410 is optionally further configured to: determine the proportion of the number of pixels in each preset material value range based on the number of pixels in each preset material value range; and determine the target material feature corresponding to the target object to be retrieved based on the proportion of the number of pixels in each preset material value range.

[0104] Based on any optional technical solution in the embodiments of this application, the first acquisition module 420 is optionally further configured to: determine the similarity between the target material feature and the material features corresponding to the multiple sample images respectively, and obtain the material similarity between the target image and the multiple sample images respectively; based on the material similarity between the target image and the multiple sample images respectively, obtain at least one sample image from the multiple sample images that is similar to the object material of the target image, and obtain at least one candidate image.

[0105] Based on any optional technical solution in the embodiments of this application, the first acquisition module 420 is optionally further configured to determine the target number of sample images with the highest material similarity to the target image among multiple sample images as candidate images, thereby obtaining at least one candidate image.

[0106] Based on any optional technical solution in the embodiments of this application, the first acquisition module 420 is optionally further configured to: extract visual features from the target image to obtain target visual features; determine the similarity between the target visual features and the visual features corresponding to the candidate images for at least one candidate image to obtain visual similarity between the target image and the candidate images; and obtain candidate images similar to the target image from at least one candidate image based on the visual similarity and material similarity between the target image and at least one candidate image to obtain image retrieval results.

[0107] Based on any optional technical solution in the embodiments of this application, the first acquisition module 420 is optionally further configured to: perform weighted fusion processing on the visual similarity and material similarity between the target image and the candidate image for at least one candidate image to obtain a comprehensive similarity between the target image and the candidate image; and based on the comprehensive similarity between the target image and at least one candidate image respectively, acquire candidate images similar to the target image from at least one candidate image to obtain image retrieval results.

[0108] The image retrieval device provided in this application embodiment can execute the image retrieval method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0109] It is worth noting that in the above-described embodiments of the target detection device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0110] Below, for reference Figure 5 This describes an electronic device according to embodiments of the present application. Figure 5 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0111] like Figure 5 As shown, the electronic device 50 includes one or more processors 501 and memory 502.

[0112] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 50 to perform desired functions.

[0113] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may execute the program instructions to implement the image retrieval methods of the various embodiments of this application described above and / or other desired functions. Various contents, such as feature extraction and similarity calculation, may also be stored in the computer-readable storage medium.

[0114] In one example, the electronic device 50 may also include an input device 503 and an output device 504, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0115] The input device 503 may include, for example, a keyboard, a mouse, etc.

[0116] The output device 504 can output various information to the outside, including feature extraction, similarity calculation, etc. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0117] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device 50 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 50 may include any other suitable components depending on the specific application.

[0118] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image retrieval methods according to various embodiments of this application as described above.

[0119] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0120] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the image retrieval methods according to various embodiments of this application described above.

[0121] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0122] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0123] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0124] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0125] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0126] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An image retrieval method characterized by, include: Acquire a target image of the target object to be retrieved, and extract the target material features corresponding to the target object to be retrieved; Based on the target material features, at least one sample image with an object material similar to the target image is obtained from multiple sample images to obtain at least one candidate image; From the at least one candidate image, candidate images similar to the target image are obtained to obtain image retrieval results.

2. The method of claim 1, wherein, The step of acquiring the target image collected for the target object to be retrieved and extracting the target material features corresponding to the target object to be retrieved includes: The target image is obtained by acquiring images of the target object under different energy level rays. Based on the target image, determine the material response index of each pixel within the target region corresponding to the target object to be retrieved in the target image; Based on the material response index corresponding to each pixel, the target material features corresponding to the target object to be retrieved are determined.

3. The method according to claim 2, characterized in that, The step of determining the target material features corresponding to the target object to be retrieved based on the material response index corresponding to each pixel includes: For multiple preset material value ranges, the number of pixels falling into each preset material value range is counted based on the material response index corresponding to each pixel. Based on the number of pixels in each of the preset material value ranges, the target material features corresponding to the target object to be retrieved are determined.

4. The method according to claim 3, characterized in that, The step of determining the target material features corresponding to the target object to be retrieved based on the number of pixels in each of the preset material value ranges includes: Based on the number of pixels in each of the preset material value ranges, determine the proportion of the number corresponding to each of the preset material value ranges; Based on the proportion of quantities corresponding to each of the preset material value ranges, the target material features corresponding to the target object to be retrieved are determined.

5. The method according to claim 1, characterized in that, The step of obtaining at least one sample image with an object material similar to the target image from multiple sample images based on the target material features, to obtain at least one candidate image, includes: Determine the similarity between the target material feature and the material features corresponding to the plurality of sample images respectively, and obtain the material similarity between the target image and the plurality of sample images respectively; Based on the material similarity between the target image and the plurality of sample images, at least one sample image with an object material similar to that of the target image is obtained from the plurality of sample images, thus obtaining the at least one candidate image.

6. The method according to claim 5, characterized in that, The step of obtaining at least one sample image with an object material similar to the target image from the plurality of sample images based on the material similarity between the target image and the plurality of sample images, to obtain the at least one candidate image, includes: The target number of sample images with the highest material similarity to the target image among the multiple sample images are determined as candidate images, thus obtaining at least one candidate image.

7. The method according to claim 5, characterized in that, The step of obtaining candidate images similar to the target image from the at least one candidate image to obtain image retrieval results includes: Visual features are extracted from the target image to obtain the target visual features; For the at least one candidate image, the similarity between the target visual feature and the visual feature corresponding to the candidate image is determined to obtain the visual similarity between the target image and the candidate image; Based on the visual similarity and material similarity between the target image and the at least one candidate image, candidate images similar to the target image are obtained from the at least one candidate image to obtain the image retrieval result.

8. The method according to claim 7, characterized in that, The step of obtaining candidate images similar to the target image from the at least one candidate image based on the visual similarity and material similarity between the target image and the at least one candidate image, and obtaining the image retrieval result, includes: For the at least one candidate image, a weighted fusion process is performed on the visual similarity and material similarity between the target image and the candidate image to obtain a comprehensive similarity between the target image and the candidate image; Based on the comprehensive similarity between the target image and the at least one candidate image, candidate images similar to the target image are obtained from the at least one candidate image to obtain the image retrieval result.

9. An image retrieval device, characterized in that, include: The feature extraction module is used to acquire a target image collected for the target object to be retrieved, and extract the target material features corresponding to the target object to be retrieved; The first acquisition module is used to acquire at least one sample image that is similar to the object material of the target image from multiple sample images based on the target material features, so as to obtain at least one candidate image; The second acquisition module is used to acquire candidate images similar to the target image from the at least one candidate image to obtain image retrieval results.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image retrieval method according to any one of claims 1 to 8.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the image retrieval method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to implement the image retrieval method according to any one of claims 1 to 8.

Images