Image search method and device, storage medium and program product

Abstract

One or more embodiments of the invention provide an image search method and device, a storage medium and a program product. The image search method comprises the following steps: acquiring a to-be-recognized image, and extracting a global semantic feature vector and a local detail feature vector from the to-be-recognized image; performing quality evaluation on the to-be-identified image to obtain a quality evaluation parameter value; determining a first weight for the global semantic feature vector and a second weight for the local detail feature vector based on the quality evaluation parameter value; wherein the value of the first weight is in negative correlation with the quality evaluation parameter value, and the value of the second weight is in positive correlation with the quality evaluation parameter value; and based on the first weight and the second weight, in combination with the global semantic feature vector and the local detail feature vector, performing similarity calculation on the to-be-recognized image and a reference image of a search object stored in a target database, so as to determine a target search object matched with the to-be-recognized image based on a similarity calculation result.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image processing technology, and more particularly, to an image search method, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous development of computer vision and image recognition technologies, image search has been widely used in application scenarios such as museum explanations, cultural relic identification, and product comparison. Users only need to take an image of the target object, such as an exhibit or product, to quickly identify and obtain relevant information, greatly improving the interactive experience and information acquisition efficiency.

[0003] In related technologies, image search methods often use a single image feature for matching. However, in practice, image acquisition is often affected by various interference factors, which can affect the accuracy and stability of feature extraction. Especially in the case of poor image quality, relying on a single type of image feature for matching is prone to recognition failures or misjudgments, making it unable to meet the needs of image search in complex environments. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide an image search method, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, an image search method is proposed, including: Acquire an image to be identified, and extract a global semantic feature vector and a local detail feature vector from the image to be identified; Performing quality assessment on the image to be identified to obtain a quality assessment parameter value; Determining a first weight for the global semantic feature vector and a second weight for the local detail feature vector based on the quality assessment parameter value; wherein the value of the first weight is negatively correlated with the quality assessment parameter value, and the value of the second weight is positively correlated with the quality assessment parameter value; Based on the first weight and the second weight, and in combination with the global semantic feature vector and the local detail feature vector, a similarity calculation is performed between the image to be identified and a reference image of the search object stored in the target database, so as to determine the target search object that matches the image to be identified based on the result of the similarity calculation.

[0006] According to a second aspect of the embodiments of this specification, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0007] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0008] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0009] The technical solutions provided by the embodiments of this specification may have the following beneficial effects: In the embodiments of the present specification, by extracting the global semantic feature vector and local detail feature vector of the image to be identified, the overall content and local details of the image can be fully understood, and then the quality of the image to be identified is evaluated to obtain the quality evaluation parameter value. The quality evaluation parameter value can reflect the availability of the image to be identified at the global semantic level and the local detail level, respectively. Therefore, the weight distribution between the global semantic feature vector and the local detail feature vector is dynamically adjusted based on the quality evaluation parameter value. This dynamic adjustment strategy can show good robustness on images of different qualities and can effectively adapt to situations with different image qualities. Finally, based on the first weight and the second weight, and in combination with the global semantic feature vector and the local detail feature vector, the similarity between the image to be identified and the reference image of the search object stored in the target database is calculated, which can comprehensively consider the overall and detail information of the image, thereby improving the accuracy of image matching.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the architecture of an image search service system provided by an exemplary embodiment.

[0012] Figure 2 The figure is a flowchart of an image search method provided by an exemplary embodiment.

[0013] Figure 3 This is a schematic diagram of similarity calculation provided by an exemplary embodiment.

[0014] Figure 4 It is a schematic diagram of another similarity calculation provided by an exemplary embodiment.

[0015] Figure 5It is a structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0016] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0017] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0018] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0019] Based on the problems in the related art, the embodiments of this specification provide an image search method, an electronic device, a computer-readable storage medium, and a computer program product.

[0020] Figure 1 FIG. 1 is a schematic diagram of an image search service system provided by an exemplary embodiment. Figure 1 As shown, the system may include a server 11 , a network 12 , and several terminals, such as a PC (Personal Computer) 13 , a mobile phone 14 , and the like.

[0021] The server 11 may be a physical server including an independent host, or a virtual server hosted by a host cluster. During operation, the server 11 may run a server-side program of an image search application to realize a corresponding image search service platform.

[0022] PC 13 and mobile phone 14 are only some types of terminals that can be used by users. In fact, users can obviously also use terminals such as the following types: tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and one or more embodiments of this specification do not limit this. During operation, the terminal can run the client-side program of the image search application to realize the client of the image search service. Among them, the client application of the above-mentioned image search service can be started and run on the terminal. The client-side program can be a native application installed on the terminal, or the client-side program can be a small program, a quick application or other similar forms. Of course, when using web page technologies such as HTML5 or similar, the relevant functions can be implemented through the page displayed by the browser. The browser here can be an independent browser application or a browser module embedded in certain applications.

[0023] Regarding the network 12 for interaction between terminals such as PC 13 and mobile phone 14 and server 11, communication can be achieved using a wired or wireless network based on the communication methods supported by the terminals, and this specification does not limit this. For example, if PC 13 supports both wired and wireless communication, then communication can be achieved using either a wired or wireless network as needed, while mobile phone 14 generally only supports wireless communication and thus can achieve communication using a wireless network.

[0024] Exemplarily, the image search method provided in this specification can be executed by the server 11, and the terminal uploads the captured image to the server 11, so that the server 11 executes the image search method provided in this specification; alternatively, the image search method provided in this specification can be executed jointly by the server 11 and the terminal, and the terminal can determine the global semantic feature vector, local detail feature vector, first weight, and second weight of the captured image, and upload these intermediate data to the server, so that the server 11 executes the subsequent image search process; alternatively, the server 11 can send relevant data in the target database to the terminal, so that the image search method is executed in the terminal. This embodiment does not impose any restrictions on this, and specific settings can be made according to actual application scenarios.

[0025] See also Figure 2 , shows a flow chart of an image search method, which is exemplified by an electronic device (such as the above-mentioned server, user terminal or other device) executing the image search method. The method includes: In S201 , an image to be recognized is obtained, and a global semantic feature vector and a local detail feature vector are extracted from the image to be recognized.

[0026] In a possible implementation, the image to be identified is obtained by photographing the object to be searched.

[0027] In another possible implementation, in order to further improve recognition accuracy, the image to be recognized is obtained by preprocessing the captured image of the search object, and the preprocessing includes at least one of the following: (1) Denoising: Remove noise from the image and restore the true details of the image. Noise usually comes from imperfections in the shooting process, transmission process, or image processing. Common noise includes Gaussian noise and salt and pepper noise. The impact of noise on the image is reduced by using smoothing filters (such as mean filtering and median filtering) or more complex algorithms (such as wavelet transform and non-local mean denoising). In the denoising process, the focus is on reducing noise while preserving the details and structure of the image as much as possible.

[0028] (2) Image enhancement: This process improves the visual quality of an image, making it clearer, more vivid, and easier to observe or understand. Its purpose is to enhance the image's details, contrast, brightness, and other visual effects, rather than removing noise. This process can increase the image's contrast (e.g., histogram equalization), adjust brightness, or sharpen the image. Alternatively, it can use edge enhancement, Laplacian operator, gamma correction, and other methods to enhance the image's details and visual effects. Alternatively, it can perform local enhancement on specific areas of the image, such as sharpening certain parts or increasing the saturation of certain colors.

[0029] (3) Geometric correction: This is done by using geometric transformations to correct the image's shooting angle. This is usually applied to images with perspective distortion or incorrect shooting angles, so that the image presents a geometric shape that is more consistent with the actual scene. In geometric correction, Canny edge detection can be used to identify important boundaries and object outlines in the image. The image is then translated, rotated, scaled, and sheared using affine transformations to adjust the image's shape and angle.

[0030] (4) Search object extraction processing: Extract meaningful targets (such as exhibits, objects, people, etc.) from complex backgrounds, reduce background interference, and thus improve the accuracy of target detection and recognition. The search object in the captured image can be detected by saliency detection or target detection. In search object extraction, saliency detection can identify the area in the image that is most likely to contain the target, suppress the influence of the background, and reduce interference during processing. Saliency detection methods include algorithms based on color, texture, brightness, and contrast. In the search object extraction process, target detection can determine the exact location of the search object in the image and conduct further analysis. Through preliminary screening, the influence of background and noise can be reduced and the focus can be placed on the target area. Saliency detection and target detection can be used in combination. First, potential target areas are obtained through saliency detection, and then the search object is accurately extracted through target detection.

[0031] Exemplarily, after acquiring the image to be identified, the electronic device further extracts global semantic feature vectors and local detail feature vectors from the image to be identified. The global semantic feature vectors can reflect the overall structure and semantic information of the image, thereby improving recognition stability when the image quality is poor or details are missing; the local detail feature vectors can reflect subtle, recognizable regional information in the image, thereby enhancing the ability to distinguish targets with similar structures but different details.

[0032] For the extraction of global semantic feature vectors, the electronic device can input the image to be recognized into a trained image encoder to obtain the global semantic feature vector output by the image encoder for representing the overall semantic information of the image.

[0033] Optionally, the image encoder can be based on the CLIP (Contrastive Language–Image Pretraining) model. The CLIP model, through large-scale image-text comparison pretraining, can extract image features with strong semantic consistency and exhibits excellent versatility and transferability. To further enhance the image encoder's recognition performance in specific search object domains, the CLIP model can be fine-tuned based on a training dataset containing images of the search object. The training dataset includes at least one image of the search object and a global semantic feature vector as a supervisory signal. This fine-tuning process improves the image encoder's ability to distinguish the search object's category, structural details, or semantic attributes, thereby enhancing retrieval accuracy.

[0034] Furthermore, after obtaining the global semantic feature vector output by the image encoder, to improve its compatibility with subsequent fusion processing, the global semantic feature vector can be post-processed according to the needs of the actual application scenario. For example, the global semantic feature vector can be normalized (such as L2 normalization) to keep it consistent in numerical scale with the local detail feature vector, thus avoiding fusion imbalance caused by different scales.

[0035] For the extraction of local detail feature vectors, electronic devices can identify the local features of the image to be identified at different scales, extract the discriminative key points and the descriptor information corresponding to the key points, aggregate the key points and the descriptors corresponding to the key points, and finally generate a local detail feature vector for characterizing the local detail information of the image.

[0036] Local features can be extracted and encoded in the following ways: In an optional solution, the electronic device can detect scale space extreme points in the image as key points based on the SIFT (Scale-Invariant Feature Transform) operator, and extract the related directional gradient histogram as a descriptor, which has good scale invariance and rotation invariance.

[0037] In another optional solution, the electronic device can grid the local area of ​​the image based on the HOG (Histogram of Oriented Gradients) operator, and calculate the distribution histogram of the gradient direction in each unit to describe the local texture features. This is suitable for image scenes with obvious edge and contour information.

[0038] After obtaining multiple key points and their corresponding descriptors, the electronic device can further transform the variable-length set of local descriptors into a fixed-dimensional local detail feature vector using a feature aggregation algorithm. For example, the BoW (Bag of Words) method can be used to construct a visual dictionary, quantize and encode all descriptors, and then count the frequency of occurrence of each visual word to form a vector representation. Alternatively, the VLAD (Vector of Locally Aggregated Descriptors) method can be used to accumulate the residual vectors between descriptors and their corresponding centers based on cluster centers, balancing descriptive power with computational efficiency.

[0039] To improve the compatibility of the local detail feature vector and the global semantic feature vector during the fusion process, the local detail feature vector can also be post-processed according to the needs of the actual application scenario. For example, the dimension of the local feature vector can be compressed to reduce redundant information and control computational overhead; or the local feature vector can be normalized to unify its scale characteristics and enhance the fusion effect with the global feature vector; noise suppression and other processing can also be performed based on specific scenarios to improve the final retrieval accuracy.

[0040] In S202 , quality assessment is performed on the image to be recognized to obtain a quality assessment parameter value.

[0041] The quality assessment process evaluates the usability of the image to be recognized at both the global semantic level and the local detail level. Different quality assessment parameter values ​​reflect different trends in the reliability of the global semantic feature vector and the local detail feature vector. A pre-trained quality assessment model can be used to assess the quality of the image to be recognized.

[0042] Exemplarily, the quality assessment parameter values ​​include at least one of the following: the occlusion area of ​​the search object in the image to be identified, the lighting quality, the resolution, and the degree of camera angle tilt. This provides data for subsequent dynamic adjustment of feature weights, improving adaptability to complex shooting conditions.

[0043] The occlusion area refers to the proportion of the area of ​​the search object that is blocked by other objects in the image. The more severe the occlusion, the more incomplete the overall image information.

[0044] Lighting quality reflects the brightness uniformity and light-dark contrast of an image. Insufficient or excessive lighting will affect the recognition of local details.

[0045] Image resolution refers to the pixel density of an image. The lower the resolution, the blurrier the details, which is not conducive to feature extraction.

[0046] The shooting angle tilt is used to measure the degree to which the image deviates from the standard observation angle when it is captured. Too large an angle may cause perspective distortion or object deformation.

[0047] In S203, a first weight for the global semantic feature vector and a second weight for the local detail feature vector are determined based on the quality assessment parameter value; wherein the value of the first weight is negatively correlated with the quality assessment parameter value, and the value of the second weight is positively correlated with the quality assessment parameter value.

[0048] In this step, the degree of reliance on global features and local features is dynamically adjusted based on the differences in quality assessment parameter values, which helps to reduce the interference of image defects on image search results and improve search accuracy.

[0049] Take the occlusion area of ​​the object to be searched in the image to be identified as an example: when the occlusion area is large, the complete outline and spatial structure of the object to be searched in the image will be destroyed, resulting in a decrease in the expressive power of the global semantic feature vector, making it difficult to effectively capture the overall appearance information. In contrast, the local detail feature vector can extract local information such as texture and edges in the unoccluded area and still has a certain recognition ability. Therefore, when the occlusion area increases, the first weight of the global semantic feature vector should be reduced, and the second weight of the local detail feature vector should be increased, that is, the occlusion area is negatively correlated with the first weight and positively correlated with the second weight.

[0050] Take lighting quality as an example: when lighting quality is poor, such as in the presence of severe shadows, backlighting, or underexposure, detailed textures, edges, and other areas are prone to blurring and information loss, affecting the stability of local descriptors. In contrast, global semantic features can still capture macroscopic information such as contours or posture at an abstract level, and have a certain degree of robustness. Therefore, when lighting quality decreases, the first weight should be increased and the second weight should be decreased to enhance search robustness. In other words, lighting quality is negatively correlated with the first weight and positively correlated with the second weight.

[0051] Taking resolution as an example, image resolution reflects the pixel density of an image. In low-resolution images, local details (such as edges and corners) are easily lost or blurred, affecting the discriminative power of local descriptors. Global semantic features, on the other hand, focus on the overall image structure and still have a certain degree of expressive power even at low resolutions. Therefore, when resolution is low, the first weight should be increased and the second weight should be decreased. In other words, resolution is negatively correlated with the first weight and positively correlated with the second weight.

[0052] Taking the degree of tilt at the shooting angle as an example, tilt (e.g., a significant deviation from the front of the search object) can distort the image's geometric structure, disrupting the spatial layout of the original scene and reducing the expressive power of global semantic features, as these rely on the consistency of the overall structure. Local features (such as local texture, edges, and corners) are typically extracted from local regions and are relatively less dependent on the overall image geometry. They may also maintain a certain degree of stability despite angle variations. Therefore, as the degree of tilt at the shooting angle increases, the first weight of the global semantic feature vector should be reduced, while the second weight of the local detail feature vector should be increased to improve overall search robustness. In other words, the degree of tilt at the shooting angle is negatively correlated with the first weight and positively correlated with the second weight. It is understood that even if the image to be identified has undergone geometric correction, the degree of tilt at the original shooting angle can still be estimated based on the image registration parameters (such as the affine or perspective transformation matrix) obtained during the correction process.

[0053] In S204, based on the first weight and the second weight, and in combination with the global semantic feature vector and the local detail feature vector, a similarity calculation is performed between the image to be identified and the reference image of the search object stored in the target database, so as to determine the target search object that matches the image to be identified based on the result of the similarity calculation.

[0054] Among them, the target database is used to store the global semantic feature vector and local detail feature vector of at least one reference image of the search object. The method of extracting the global semantic feature vector and the local detail feature vector can be found in the above description and will not be repeated here. Optionally, the target database is used to store the global semantic feature vector and local detail feature vector of at least two reference images of the search object, and the at least two reference images are obtained by shooting at different shooting angles. When the user actually shoots, he may be in non-ideal conditions such as non-frontal, low light, tilted, occluded, etc. Multi-angle reference images can increase the hit probability. After the database contains features of different angles, even if the viewing angle of the image to be identified is rare, it is possible to find a relatively close matching target in the reference set. This structure can also improve the hit rate for low-quality images or edge-shot images, thereby improving the overall search effect.

[0055] In some embodiments, see Figure 3 The electronic device can perform weighted fusion on the global semantic feature vector and the local detail feature vector of the image to be identified according to the first weight and the second weight to obtain a first fused feature vector. The weighting method can be in the form of a linear combination, for example: the first fused feature vector = the first weight * the global semantic feature vector + the second weight * the local detail feature vector; the first weight can adjust the proportion of the global semantic feature in the final feature representation, which is used to express the overall layout and structural semantics of the image; the second weight can adjust the proportion of the local detail feature in the final feature representation, which is used to capture key local textures, details, corner points and other information; the weighted fusion design allows the electronic device to selectively emphasize more discernible feature types when facing different image qualities (such as occlusion, insufficient lighting, etc.), thereby improving the accuracy and robustness of matching.

[0056] Similarly, for each reference image stored in the target database, the electronic device can also use the same first weight and second weight to perform weighted fusion of the corresponding global semantic feature vector and local detail feature vector, thereby generating a second fused feature vector. This process maintains the consistency of the feature space and ensures that the image to be identified and the reference image in the target database are compared under a unified fusion rule. Subsequently, the electronic device can determine the target search object that the image to be identified is most likely to match based on the similarity between the first fused feature vector and each second fused feature vector (for example, using cosine similarity, Euclidean distance or other measurement methods). In this embodiment, a unified fusion strategy is used between the image to be identified and the reference image in the target database, so that the similarity calculation is more semantically consistent, thereby improving the search accuracy.

[0057] In one possible implementation, the electronic device can use a full comparison method to calculate the similarity between the first fused feature vector and all second fused feature vectors in the target database. Based on the similarity calculation results, the electronic device selects the target reference image corresponding to the second fused feature vector that is most similar to the first fused feature vector to determine the target search object that is most likely to match the image to be recognized. This method is suitable for scenarios with small databases or extremely high confidence requirements for recognition results.

[0058] In another possible implementation, to improve search efficiency, after generating a second fused feature vector corresponding to each reference image in the target database, the electronic device may construct an approximate nearest neighbor (ANN) index structure based on the second fused feature vector corresponding to the reference image in the target database, such as by constructing an inverted index, a vector quantization index, a KD tree (K-Dimensional Tree), or a hash structure. The electronic device then uses the first fused feature vector to query the approximate nearest neighbor index structure, quickly obtaining multiple candidate second fused feature vectors similar to the first fused feature vector. Based on the similarity between the first fused feature vector and each candidate second fused feature vector, the target search object matching the image to be identified is determined. This embodiment, through the index structure, can effectively reduce the retrieval time complexity and meet the real-time search requirements of image databases at the million or even billion level. It achieves high-speed response with minimal sacrifice of accuracy, making it suitable for scenarios with strong real-time requirements.

[0059] In other embodiments, see Figure 4 The electronic device uses a metric such as cosine similarity, Euclidean distance, or dot product to calculate a first similarity between the global semantic feature vector of the image to be identified and the global semantic feature vector of the reference image in the target database. This similarity is used to measure the consistency of the overall structure and semantic scene, thereby enabling rapid assessment of the global visual semantic relevance between images. Similarly, the electronic device can use a metric such as cosine similarity, Euclidean distance, or dot product to calculate a second similarity between the local detail feature vector of the image to be identified and the local detail feature vector of the reference image in the target database. The second similarity focuses more on texture and structural information at the microscopic level. Next, the electronic device combines the first and second similarities associated with the same reference image of the search object in a weighted manner to obtain a comprehensive similarity between the reference image and the image to be identified. Finally, based on the comprehensive similarity between the reference images of each search object and the image to be identified, the electronic device selects the search object corresponding to the target reference image with the highest similarity as the matching result for the image to be identified.

[0060] This implementation, through a separate scoring and dynamic weighted fusion mechanism for global and local feature similarity, improves image search accuracy while enhancing interpretability and robustness, significantly enhancing its practical value and reliability in multi-source image retrieval tasks. The calculation of the first and second similarities can be performed in parallel, improving computational efficiency.

[0061] In one possible implementation, the electronic device may adopt a full comparison method. Specifically, the electronic device may calculate the first similarity between the global semantic feature vector of the image to be identified and the global semantic feature vectors of all reference images in the target database, and calculate the second similarity between the local detail feature vector of the image to be identified and the local detail feature vectors of all reference images in the target database. Furthermore, the electronic device performs a weighted summation of the first similarity and the second similarity related to the same reference image of the search object according to the first weight and the second weight, and obtains the comprehensive similarity between the reference image of the search object and the image to be identified. Finally, the target search object that matches the image to be identified can be determined by using the reference image with the highest comprehensive similarity. The advantage of this implementation method is that all similarity scores can be obtained for comprehensive analysis without discarding the relevant information of any reference image in any database. It is suitable for scenarios where the database size is small or the accuracy requirements are extremely high.

[0062] In another possible implementation, to improve search efficiency, a first approximate nearest neighbor index structure corresponding to the global semantic features of the reference images in the target database and / or a second approximate nearest neighbor index structure corresponding to the local detail feature vectors of the reference images in the target database can be pre-built. These two index structures can be constructed using algorithms such as HNSW (Hierarchical Navigable Small World), LSH (Locality-Sensitive Hashing), and IVF-PQ (Inverted File with Product Quantization), respectively, to support fast searches of high-dimensional features. For example, the first approximate nearest neighbor index structure can be constructed using the HNSW algorithm, and the second approximate nearest neighbor index structure can be constructed using the IVF-PQ algorithm.

[0063] The electronic device can use the global semantic feature vector of the image to be identified to query the first approximate nearest neighbor index structure to obtain multiple candidate global semantic feature vectors similar to the image to be identified; and then calculate the first similarity between the global semantic feature vector of the image to be identified and each candidate global semantic feature vector.

[0064] The electronic device can use the local detail feature vector of the image to be identified to query the second approximate nearest neighbor index structure to obtain multiple candidate local detail feature vectors similar to the image to be identified; and then calculate the second similarity between the local detail feature vector of the image to be identified and each candidate local detail feature vector.

[0065] The electronic device then performs a weighted summation of the first and second similarities associated with the same reference image, using the first and second weights, to obtain a comprehensive similarity between the reference image and the image to be identified. Finally, the target search object that matches the image to be identified is determined using the reference image with the highest comprehensive similarity. This structured index query method significantly improves retrieval efficiency, achieving millisecond-level responses in massive image databases. It is suitable for applications with large image volumes, frequent queries, and high real-time requirements, such as exhibit retrieval, mobile image recognition, or online product comparison systems.

[0066] In some embodiments, the search object in the target database may also carry display location information, such as the coordinates of the location where the search object is placed, installed, or displayed in the real world (e.g., latitude and longitude, Bluetooth beacon area number, etc.). The image to be identified may also carry shooting location information, such as the image shooting location obtained through satellite positioning, Bluetooth indoor positioning, Wi-Fi positioning, etc. Based on the matching results between the shooting location information carried by the image to be identified and the display location information carried by the search object in the target database, as well as the similarity calculation results, the electronic device can determine with greater confidence that the target search object matches the image to be identified. On the one hand, in large-scale image search databases, location matching can be used to narrow the search candidate set, avoiding the calculation of feature similarity for all images in the database, thereby improving efficiency. For example, if the shooting location is in Hall A, only the search object displayed in Hall A will be matched, avoiding the inclusion of irrelevant objects in Hall B or other areas. On the other hand, when the image to be identified is blurred, obscured, tilted, or other factors that reduce visual similarity, location information can serve as a reliable supplementary basis to improve the robustness of the judgment.

[0067] The image search methods described in the embodiments of this specification can be applied in a variety of scenarios. For example, they can be used in museum tour systems. Visitors use their mobile phones or camera-equipped tour guides to take photos of exhibits. After the system recognizes the exhibit image, it matches the corresponding image features in a database and quickly returns a voice explanation, text introduction, or interactive content for the exhibit.

[0068] For example, it can be applied to urban public guide systems. Users use their mobile phones to take pictures of historical buildings, sculptures, billboards or bus stops in the city. The system identifies the target through image features and geographic location information and provides introductory information or navigation services.

[0069] For example, it can be applied to smart shopping mall shopping guide systems. Customers take pictures of store entrances, promotional advertisements or displayed goods, and the system recognizes them and provides current promotion information, store navigation or product details.

[0070] For example, it can be applied to industrial equipment or parts identification. Maintenance personnel take pictures of a mechanical structure or part, and the system identifies its model, instructions or maintenance records to assist in rapid maintenance.

[0071] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0072] In some embodiments, an embodiment of this specification further provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0073] Figure 5 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510. Of course, it may also include hardware required for other functions. One or more embodiments of this specification can be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0074] In some embodiments, the image search device can be applied to Figure 5 The device shown in the figure is used to implement the technical solution of this specification. The image search device may include: The feature extraction module is used to obtain an image to be identified and extract a global semantic feature vector and a local detail feature vector from the image to be identified.

[0075] The quality assessment module is used to perform quality assessment on the image to be identified and obtain a quality assessment parameter value.

[0076] A weight determination module is used to determine a first weight for the global semantic feature vector and a second weight for the local detail feature vector based on the quality assessment parameter value; wherein the value of the first weight is negatively correlated with the quality assessment parameter value, and the value of the second weight is positively correlated with the quality assessment parameter value.

[0077] An image search module is configured to perform similarity calculation between the image to be identified and a reference image of the search object stored in a target database based on the first weight and the second weight, and in combination with the global semantic feature vector and the local detail feature vector, so as to determine a target search object that matches the image to be identified based on the result of the similarity calculation.

[0078] In one implementation, the quality assessment parameter value includes a value of at least one of the following quality assessment parameters: an occlusion area, illumination quality, resolution, and tilt degree of a shooting angle of the object to be searched in the image to be identified.

[0079] In one implementation, the target database is used to store the global semantic feature vector and the local detail feature vector of at least one reference image of the search object.

[0080] The image search module is specifically used to weightedly fuse the global semantic feature vector and the local detail feature vector of the image to be identified according to the first weight and the second weight to obtain a first fused feature vector; weightedly fuse the global semantic feature vector and the local detail feature vector of the reference image in the target database according to the first weight and the second weight to obtain a second fused feature vector; and determine the target search object that matches the image to be identified based on the similarity between the first fused feature vector and the second fused feature vector.

[0081] Exemplarily, the image search module is specifically used to construct an approximate nearest neighbor index structure based on the second fused feature vector of the reference image in the target database; use the first fused feature vector to query the approximate nearest neighbor index structure to obtain multiple candidate second fused feature vectors similar to the first fused feature vector; based on the similarity between the first fused feature vector and each candidate second fused feature vector, determine the target search object that matches the image to be identified.

[0082] In one implementation, the target database is used to store the global semantic feature vector and the local detail feature vector of at least one reference image of the search object.

[0083] The image search module is specifically used to calculate a first similarity between the global semantic feature vector of the image to be identified and the global semantic feature vector of the reference image in the target database, and to calculate a second similarity between the local detail feature vector of the image to be identified and the local detail feature vector of the reference image in the target database; according to the first weight and the second weight, the first similarity and the second similarity related to the same reference image are weightedly summed to obtain a comprehensive similarity between the reference image and the image to be identified; based on the comprehensive similarity between the reference image in the target database and the image to be identified, a target search object that matches the image to be identified is determined.

[0084] Exemplarily, the target database includes: a first approximate nearest neighbor index structure corresponding to the global semantic feature quantity of the reference image in the target database.

[0085] The image search module is specifically used to use the global semantic feature vector of the image to be identified to query the first approximate nearest neighbor index structure to obtain multiple candidate global semantic feature vectors similar to the image to be identified; and calculate the first similarity between the global semantic feature vector of the image to be identified and each candidate global semantic feature vector.

[0086] Exemplarily, the target database further includes: a second approximate nearest neighbor index structure corresponding to the local detail feature vector of the reference image in the target database.

[0087] The image search module is specifically used to use the local detail feature vector of the image to be identified to query the second approximate nearest neighbor index structure to obtain multiple candidate local detail feature vectors similar to the image to be identified; and calculate the second similarity between the local detail feature vector of the image to be identified and each candidate local detail feature vector.

[0088] In one implementation, the image search module is specifically used to determine the target search object that matches the image to be identified based on the matching result between the shooting location information carried by the image to be identified and the display location information carried by the search object in the target database, as well as the result of the similarity calculation.

[0089] In one implementation, the feature extraction module is specifically used to input the image to be identified into a trained image encoder to obtain the global semantic feature vector output by the image encoder; and to identify local features of the image to be identified at different scales, obtain key points and descriptors corresponding to the key points, and aggregate the key points and the descriptors corresponding to the key points to obtain the local detail feature vector.

[0090] In one implementation, the image to be identified is obtained by preprocessing a captured image of the search object, wherein the preprocessing includes at least one of the following: denoising processing, image enhancement processing, geometric correction processing, and search object extraction processing.

[0091] In one implementation, the target database is used to store global semantic feature vectors and local detail feature vectors of at least two reference images of the search object, where the at least two reference images are captured at different shooting angles.

[0092] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0093] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0094] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0095] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0096] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. An image search method, comprising: Acquire an image to be identified, and extract a global semantic feature vector and a local detail feature vector from the image to be identified; Performing quality assessment on the image to be identified to obtain a quality assessment parameter value; Determining a first weight for the global semantic feature vector and a second weight for the local detail feature vector based on the quality assessment parameter value; wherein the value of the first weight is negatively correlated with the quality assessment parameter value, and the value of the second weight is positively correlated with the quality assessment parameter value; Based on the first weight and the second weight, and in combination with the global semantic feature vector and the local detail feature vector, a similarity calculation is performed between the image to be identified and a reference image of the search object stored in the target database, so as to determine the target search object that matches the image to be identified based on the result of the similarity calculation.

2. The method according to claim 1, wherein the quality assessment parameter value comprises a value of at least one of the following quality assessment parameters: an occlusion area of ​​the object to be searched in the image to be identified, illumination quality, resolution, and tilt degree of the shooting angle; And / or, the image to be identified is obtained by preprocessing a captured image of the search object, wherein the preprocessing includes at least one of the following: denoising processing, image enhancement processing, geometric correction processing, and search object extraction processing; And / or, the target database is used to store the global semantic feature vectors and local detail feature vectors of at least two reference images of the search object, and the at least two reference images are obtained by shooting at different shooting angles.

3. The method according to claim 1, wherein the target database is used to store the global semantic feature vector and the local detail feature vector of at least one reference image of the search object; The performing similarity calculation between the image to be identified and a reference image of a search object stored in a target database based on the first weight and the second weight and in combination with the global semantic feature vector and the local detail feature vector, and determining a target search object that matches the image to be identified based on a result of the similarity calculation, includes: performing weighted fusion on the global semantic feature vector and the local detail feature vector of the image to be identified according to the first weight and the second weight to obtain a first fused feature vector; performing weighted fusion on the global semantic feature vector and the local detail feature vector of the reference image in the target database according to the first weight and the second weight to obtain a second fused feature vector; Based on the similarity between the first fused feature vector and the second fused feature vector, a target search object matching the image to be identified is determined.

4. The method according to claim 3, wherein determining the target search object matching the image to be identified based on the similarity between the first fused feature vector and the second fused feature vector comprises: constructing an approximate nearest neighbor index structure based on a second fused feature vector of a reference image in the target database; Using the first fused feature vector, querying the approximate nearest neighbor index structure to obtain a plurality of candidate second fused feature vectors similar to the first fused feature vector; Based on the similarities between the first fused feature vector and each candidate second fused feature vector, a target search object matching the image to be identified is determined.

5. The method according to claim 1, wherein the target database is used to store the global semantic feature vector and the local detail feature vector of at least one reference image of the search object; The performing similarity calculation between the image to be identified and a reference image of a search object stored in a target database based on the first weight and the second weight and in combination with the global semantic feature vector and the local detail feature vector, and determining a target search object that matches the image to be identified based on a result of the similarity calculation, includes: Calculating a first similarity between a global semantic feature vector of the image to be identified and a global semantic feature vector of a reference image in the target database, and calculating a second similarity between a local detail feature vector of the image to be identified and a local detail feature vector of the reference image in the target database; performing a weighted summation of the first similarity and the second similarity associated with the same reference image according to the first weight and the second weight to obtain a comprehensive similarity between the reference image and the image to be identified; Based on the comprehensive similarity between the reference image in the target database and the image to be identified, a target search object matching the image to be identified is determined.

6. The method according to claim 5, wherein the target database comprises: A first approximate nearest neighbor index structure corresponding to the global semantic feature quantity of the reference image in the target database, and a second approximate nearest neighbor index structure corresponding to the local detail feature vector of the reference image in the target database; The calculating a first similarity between the global semantic feature vector of the image to be identified and the global semantic feature vector of the reference image in the target database includes: Using the global semantic feature vector of the image to be identified, querying the first approximate nearest neighbor index structure to obtain a plurality of candidate global semantic feature vectors similar to the image to be identified; Calculating first similarities between the global semantic feature vector of the image to be identified and each candidate global semantic feature vector; The calculating a second similarity between the local detail feature vector of the image to be identified and the local detail feature vector of the reference image in the target database includes: Using the local detail feature vector of the image to be identified, querying the second approximate nearest neighbor index structure to obtain a plurality of candidate local detail feature vectors similar to the image to be identified; A second similarity between the local detail feature vector of the image to be identified and each candidate local detail feature vector is calculated.

7. The method according to claim 1, wherein determining the target search object that matches the image to be identified based on the result of similarity calculation comprises: Based on the matching result between the shooting location information carried by the image to be identified and the display location information carried by the search object in the target database, and the result of the similarity calculation, the target search object matching the image to be identified is determined.

8. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by running the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images