Image retrieval method and device and storage medium

Abstract

The invention discloses an image retrieval method and device and a storage medium, and the method comprises the steps: carrying out the image partitioning of a plurality of target images, obtaining a plurality of target partitioning images of each target image, obtaining the attention thermodynamic information of each target partitioning image for each target image, and obtaining the attention thermodynamic information of each target partitioning image; according to the attention thermodynamic information, carrying out image feature extraction on the corresponding target block images to obtain block image features of each target block image, integrating the block image features into image features of the target image, then receiving a to-be-retrieved image, carrying out feature extraction on the to-be-retrieved image to obtain features of the to-be-retrieved image, and carrying out feature extraction on the to-be-retrieved image to obtain the to-be-retrieved image; and determining a retrieval result image based on the matching of the to-be-retrieved image features and the image features of the plurality of target images. According to the embodiment of the invention, the block image features of the plurality of target block images obtained based on the attention thermodynamic information are used as the image features of the target image, so that when image retrieval is performed on the to-be-retrieved image, the accuracy and efficiency of image retrieval can be improved.

Description

Technical Field

[0001] The present application relates to the field of image retrieval technology, and in particular to an image retrieval method, device and storage medium. Background Art

[0002] Image retrieval often needs to be performed between a massive amount of input images and a huge database of images. Common image retrieval schemes include two major steps: coarse sorting and fine sorting. The coarse sorting is to find multiple database images that are similar to the input image in the entire database, while the fine sorting is to perform more accurate fine sorting on these found database images to obtain accurate results that are more similar to the input image. However, if the key database images are not retrieved during the coarse sorting, then no matter how high the accuracy of the fine sorting is, the correct retrieval results cannot be obtained, so this scheme will have the problem of unsatisfactory image retrieval effect. In addition, since the input image and multiple database images need to be matched during the coarse sorting and fine sorting, there will be repeated matching calculations for the same image, which will affect the efficiency of image retrieval. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present application provide an image retrieval method, device, and storage medium, which can improve the accuracy and efficiency of image retrieval.

[0005] In one aspect, an embodiment of the present application provides an image retrieval method, comprising the following steps:

[0006] Performing image segmentation on a plurality of target images respectively to obtain a plurality of target segmented images of each of the target images;

[0007] For each of the target images, acquiring attention thermal information of each of the target block images, and performing image feature extraction on each of the target block images based on the attention thermal information of each of the target block images to obtain block image features of each of the target block images;

[0008] For each of the target images, integrating the block image features of a plurality of the target block images into an image feature of the target image;

[0009] Receiving an image to be retrieved, performing feature extraction on the image to be retrieved, and obtaining features of the image to be retrieved;

[0010] Based on the matching of the to-be-retrieved image features with the image features of the plurality of target images, a retrieval result image is determined from the plurality of target images.

[0011] On the other hand, an embodiment of the present application further provides an image retrieval device, comprising:

[0012] An image blocking unit is used to block the multiple target images respectively to obtain multiple target block images of each target image;

[0013] a first extraction unit, configured to obtain, for each target image, attention thermal information of each target block image, and perform image feature extraction on each target block image based on the attention thermal information of each target block image to obtain a block image feature of each target block image;

[0014] a feature processing unit, configured to integrate the block image features of a plurality of the target block images into an image feature of the target image for each of the target images;

[0015] A second extraction unit is used to receive the image to be retrieved, perform feature extraction on the image to be retrieved, and obtain features of the image to be retrieved;

[0016] The image retrieval unit is configured to determine a retrieval result image from the plurality of target images based on matching the features of the image to be retrieved with the image features of the plurality of target images.

[0017] Optionally, the image blocking unit is further configured to:

[0018] Performing object detection on each of the plurality of target images to obtain a plurality of first sub-block images of each of the target images;

[0019] Performing subject object detection on each of the plurality of target images to obtain a plurality of second sub-block images of each of the target images;

[0020] Performing semantic segmentation on the plurality of target images respectively to obtain a plurality of third sub-block images of each of the target images;

[0021] For each of the target images, a plurality of the target block images are determined based on a plurality of the first sub-block images, a plurality of the second sub-block images, and a plurality of the third sub-block images.

[0022] Optionally, the image blocking unit is further configured to:

[0023] Acquire confidence scores of the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images;

[0024] The first sub-block images, the second sub-block images, and the third sub-block images whose confidences meet a predetermined condition are integrated into a plurality of target block images.

[0025] Optionally, the image blocking unit is further configured to:

[0026] determining a first center position of each of the first sub-block images, a second center position of each of the second sub-block images, and a third center position of each of the third sub-block images;

[0027] If a first center position of the first sub-block image, a second center position of the second sub-block image, and a third center position of the third sub-block image meet a first condition, an overlapping area of ​​the first sub-block image, the second sub-block image, and the third sub-block image is determined as the target block image.

[0028] Optionally, the image blocking unit is further configured to:

[0029] Calling the first block model to perform image block division on the plurality of target images respectively to obtain a plurality of fourth sub-block images of each of the target images;

[0030] calling a second block model to perform image segmentation on each of the plurality of target images to obtain a plurality of fifth sub-block images of each of the target images, wherein the first block model and the second block model are pre-trained so that a first block granularity of the first block model is greater than a second block granularity of the second block model;

[0031] For each of the target images, a plurality of the fourth sub-block images and a plurality of the fifth sub-block images are integrated into a plurality of the target block images.

[0032] Optionally, the image blocking unit is further configured to:

[0033] Performing image segmentation on the plurality of target images respectively to obtain a plurality of candidate segmented images of the respective target images;

[0034] For each of the target images, a plurality of the target block images having image areas greater than a preset area threshold are determined from the plurality of candidate block images.

[0035] Optionally, the image blocking unit is further configured to:

[0036] Using a text encoder to encode the text in the target image to obtain an image semantic label;

[0037] Segmenting the target image based on the image semantic label to obtain segmentation frame lines;

[0038] Using a visual encoder to visually encode the frame lines in the target image to obtain a frame line mask;

[0039] If the segmentation frame line matches the frame line mask, the target image is segmented using the segmentation frame line to obtain a plurality of candidate segmented images.

[0040] Optionally, the first extraction unit is further configured to:

[0041] Obtaining image key points in each target block image;

[0042] For the image key points, the attention thermal information of the target block image is obtained.

[0043] Optionally, the first extraction unit is further configured to:

[0044] Performing key point detection on the target image to obtain image key point distribution of the target image;

[0045] The image key points in each of the target block images are obtained in the image key point distribution according to the position information of each of the target block images in the target image.

[0046] Optionally, the first extraction unit is further configured to:

[0047] Performing object recognition on the target image to obtain a recognized object;

[0048] Determining key point positioning rules for the object based on the identified type of the object;

[0049] Based on the key point positioning rule, key points are positioned on the object, thereby forming an image key point distribution of the target image.

[0050] Optionally, the attention thermal information is an attention thermal matrix, and each element in the attention thermal matrix corresponds one-to-one to each pixel in the target block image; and the first extraction unit is further configured to:

[0051] If one of the pixel points in the target block image is the image key point, setting the value of the element corresponding to the pixel point to the first element value;

[0052] If one of the pixel points in the target block image is not the image key point, the value of the element corresponding to the pixel point is set to 0.

[0053] Optionally, the first extraction unit is further configured to:

[0054] Extracting basic features and enhancing local features of each target block image to obtain basic image features and local image enhancement features of each target block image;

[0055] The block image features of each target block image are generated according to the attention thermal information, the basic image features and the local image enhancement features of each target block image.

[0056] Optionally, the basic image feature is a pixel value matrix of each pixel in the target block image, the local image enhancement feature is a pixel enhancement value matrix of each pixel in the target block image, and the attention thermal information is an attention thermal matrix. Each element in the attention thermal matrix corresponds one-to-one to each pixel in the target block image, wherein, if one of the pixels in the target block image is the image key point, the value of the element corresponding to the pixel is a first element value; if one of the pixels in the target block image is not the image key point, the value of the element corresponding to the pixel is 0; the first extraction unit is further used to:

[0057] Based on the attention heat matrix, sensitively adjust the pixel value matrix to obtain a first matrix;

[0058] Based on the attention heat matrix, background noise suppression is performed on the pixel enhancement value matrix to obtain a second matrix;

[0059] Based on the first matrix and the second matrix, a total matrix is ​​determined, and the total matrix is ​​used as the block image feature of the target block image.

[0060] Optionally, the first extraction unit is further configured to:

[0061] Multiplying the elements of the attention heat matrix by a first ratio and adding 1 to obtain a sensitivity adjustment coefficient matrix; multiplying the pixel value matrix by the sensitivity adjustment coefficient matrix to obtain the first matrix;

[0062] The pixel enhancement value matrix is ​​multiplied by the attention heat matrix to obtain the second matrix.

[0063] Optionally, the first extraction unit is further configured to:

[0064] Obtaining a first weight of the first matrix and a second weight of the second matrix;

[0065] Based on the first weight and the second weight, a weighted sum is performed on the first matrix and the second matrix to obtain the total matrix.

[0066] Optionally, each element of the pixel enhancement value matrix is ​​a first value, and the image retrieval device further includes a numerical calculation unit, which is configured to:

[0067] Performing enhancement processing on each pixel point in the target block image to obtain an enhanced pixel value of the pixel point;

[0068] Determine an average value of the pixel value of the pixel point and the enhanced pixel value;

[0069] The first value is determined based on the average value of each of the pixel points in the target block image.

[0070] On the other hand, an embodiment of the present application further provides an electronic device, including:

[0071] at least one processor;

[0072] at least one memory for storing at least one program;

[0073] When at least one of the programs is executed by at least one of the processors, the image retrieval method as described above is implemented.

[0074] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the processor-executable computer program is executed by the processor, it is used to implement the image retrieval method as described above.

[0075] On the other hand, an embodiment of the present application also provides a computer program product, including a computer program or computer instructions, wherein the computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of the electronic device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the electronic device performs the image retrieval method as described above.

[0076] The embodiments of the present application include at least the following beneficial effects: first, multiple target images are segmented respectively to obtain multiple target block images of each target image; then, for each target image, the attention thermal information of each target block image is obtained, and image features of each target block image are extracted according to the attention thermal information of each target block image to obtain the block image features of each target block image. Since the block image features of the target block images are obtained by extracting image features according to their attention thermal information, the block image features can express the target block images more accurately; then, for each target image, the block image features of the multiple target block images obtained are integrated into the target image Image features, since the block image features of multiple target block images of the target image are integrated into the overall image features, these block image features can accurately express each block part of the target image. Therefore, when the image to be retrieved is received, the feature extraction of the image to be retrieved is first performed to obtain the image features to be retrieved, and then based on the matching of the image features to be retrieved with the image features of multiple target images, the retrieval result image can be accurately determined among the multiple target images. There is no need to perform coarse sorting and then fine sorting as in the prior art. Therefore, the problems of unsatisfactory image retrieval accuracy and low image retrieval efficiency existing in the related art can be improved, thereby effectively improving the accuracy and efficiency of image retrieval.

[0077] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0079] Figure 1A-1B is a system architecture diagram used in the image retrieval method of the embodiment of the present application;

[0080] Figure 2A-2E This is a scene interface diagram of the image retrieval method of an embodiment of the present application being applied in a scene of an image search engine;

[0081] Figure 3 is a flow chart of the image retrieval method according to an embodiment of the present application;

[0082] Figure 4 yes Figure 3 A flowchart of an embodiment of step 310;

[0083] Figure 5 yes Figure 4 A schematic diagram of an example of a specific implementation of

[0084] Figure 6 yes Figure 3 A flowchart of another embodiment of step 310;

[0085] Figure 7 This is a schematic diagram of an implementation of performing image segmentation on a target image to obtain multiple candidate segmented images in an embodiment of the present application;

[0086] Figure 8 This is a flow chart of performing image segmentation on a target image to obtain multiple candidate segmented images in an embodiment of the present application;

[0087] Figure 9 is a specific flow chart of obtaining the attention thermal information of the target block image in step 320;

[0088] Figure 10 This is a specific flow chart for obtaining the distribution of key points of a target image in an embodiment of the present application;

[0089] Figure 11 yes Figure 9 Corresponding implementation diagram;

[0090] Figure 12 This is a block diagram of a specific implementation of obtaining the distribution of image key points of a target image in an embodiment of the present application;

[0091] Figure 13 is a schematic diagram of the process of obtaining the block image features of the target block image in step 320;

[0092] Figure 14 is a flowchart of the detailed process of the image retrieval method provided in an embodiment of the present application;

[0093] Figure 15 is a schematic diagram of an image retrieval device provided in an embodiment of the present application;

[0094] Figure 16 This is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be considered as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0096] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0097] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0098] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0099] 1) Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI domains. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0100] 2) Computer vision (CV) is the science of enabling machines to "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision, such as the Swin Transformer, ViT, V-MOE, and MAE, can be fine-tuned to quickly and widely apply to specific downstream tasks. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0101] 3) Feature extraction model refers to an algorithm or model used to automatically extract features from raw data in machine learning and deep learning tasks. These raw data can be images, speech, text, or other forms of data. The purpose of feature extraction is to convert the raw data into a representation with higher expressiveness and interpretability to facilitate subsequent tasks such as classification, regression, and clustering. Typically, the feature extraction model performs a series of mathematical transformations and conversions on the raw data, such as convolution, pooling, normalization, and dimensionality reduction, to extract the most representative and useful feature information. Common feature extraction models include convolutional neural networks (CNN), recurrent neural networks (RNN), and autoencoders. These models have been widely used in image recognition, speech recognition, natural language processing, and other fields, and have achieved good results.

[0102] 4) Semantic segmentation is a technique that assigns a semantic label to each pixel in an image, enabling precise and accurate image segmentation and understanding. It is commonly used in fields such as computer vision and natural language processing. Unlike traditional image segmentation techniques, semantic segmentation goes beyond simply dividing an image into regions and requires a deep understanding and analysis of each pixel. Therefore, semantic segmentation requires the integration of advanced technologies such as deep learning and convolutional neural networks. Semantic segmentation is widely used in many application areas, such as autonomous driving, medical imaging diagnosis, intelligent transportation, and security monitoring.

[0103] In image retrieval technology, image retrieval often requires searching between a vast number of input images and a vast database of images. Typical image retrieval solutions involve two major steps: coarse sorting and fine sorting. Coarse sorting involves searching the database for multiple images that are similar to the input image, while fine sorting involves performing a more accurate sorting of these images to obtain a result that is more similar to the input image. A typical two-stage matching approach in the related art involves performing coarse sorting in the first stage and fine sorting in the second stage for image retrieval.

[0104] Related art approaches employ a first-stage coarse sorting followed by a second-stage fine sorting scheme for image retrieval. The first-stage coarse sorting relies solely on global features extracted from the entire search image and the underlying image for matching. However, in real-world scenarios, there are numerous difficult examples where matching the search image and underlying image is challenging. These examples often arise when the search image is not a simple graphical variation of the underlying image (e.g., rotation, flipping, or color change). Instead, the core subject of the search image and underlying image is the same object, but occupies only a portion of the image. When the object in the search image occupies a small area in the underlying image, the global feature similarity is relatively low, and the backgrounds of different images may differ significantly. These difficult examples have a significant impact on global feature matching algorithms, making them prone to missed detections. Once the key underlying image is not included in the coarse sorting candidate set, no matter how clever the second-stage fine sorting algorithm is, it will not produce a correct result, thereby reducing retrieval accuracy. Furthermore, fine sorting is performed based on the first-stage coarse sorting results, performing matching calculations between the search image and underlying image on the candidate set. At this point, all candidate results screened in the first stage need to be matched against the search image one by one, requiring a significant amount of computational time. Furthermore, since both the rough and fine sorting processes require matching the input image with multiple database images, repeated matching calculations may be performed on the same image, impacting image retrieval efficiency.

[0105] In order to improve the accuracy and efficiency of image retrieval, the embodiments of the present application provide an image retrieval method, an image retrieval device, an electronic device, a computer-readable storage medium, and a computer program product. First, multiple target images are segmented to obtain multiple target block images of each target image; then, for each target image, the attention thermal information of each target block image is obtained, and image features of each target block image are extracted based on the attention thermal information of each target block image to obtain block image features of each target block image. Since the block image features of the target block images are obtained by extracting image features based on their attention thermal information, the block image features can express the target block images more accurately; then, for each target image, the obtained The block image features of multiple target block images are integrated into the image features of the target image. Since the block image features of multiple target block images of the target image are integrated into its overall image features, these block image features can accurately express each block part of the target image. Therefore, when the image to be retrieved is received, the feature extraction of the image to be retrieved is first performed to obtain the feature of the image to be retrieved. Then, based on the matching of the feature of the image to be retrieved with the image features of multiple target images, the retrieval result image can be accurately determined among the multiple target images. There is no need to perform coarse sorting and then fine sorting as in the prior art. Therefore, the problems of unsatisfactory image retrieval accuracy and low image retrieval efficiency existing in the related art can be improved, thereby effectively improving the accuracy and efficiency of image retrieval.

[0106] Figure 1A This is a system architecture diagram of the image retrieval method of the embodiment of the present application. Figure 1A The system architecture includes a first user terminal 101, a second user terminal 102, and a first server 103. The first user terminal 101 and the first server 103 are directly or indirectly connected via wired or wireless communication, and the second user terminal 102 and the first server 103 are directly or indirectly connected via wired or wireless communication. In one example, the first user terminal 101, the second user terminal 102, and the first server 103 can be nodes in a blockchain, which is not specifically limited in this embodiment.

[0107] The first user terminal 101 may include but is not limited to smart phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft and other smart devices. Optionally, the first user terminal 101 may be installed with an image acquisition client. When the image acquisition client is in operation, the image acquisition client may capture a target image and send the target image to the first server 103 as a base image for image retrieval in the embodiment of the present application. The image acquisition client can capture many target images, so there will be a large number of base images. The target image may include a variety of simple graphic-level content or complex graphic-level content. For example, these graphic-level contents may include a large amount of background information, such as background information such as the sky and lawn, and may also include a variety of object information, such as cars, roads and pedestrians, etc., which are not specifically limited here.

[0108] The second user terminal 102 may include but is not limited to smart phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft and other smart devices. Optionally, the second user terminal 102 may be installed with an image acquisition client. When the image acquisition client is in operation, the image acquisition client may capture the image to be retrieved and send the image to be retrieved to the first server 103, wherein the image to be retrieved is the image to be retrieved. The image to be retrieved is compared with a plurality of target images (i.e., base library images) collected in advance. The target image that matches the image to be retrieved is the final retrieval result image. The image to be retrieved may include a variety of simple graphic-level contents or complex graphic-level contents. For example, these graphic-level contents may include a large amount of background information, such as background information such as the sky and lawn, and may also include a variety of object information, such as cars, roads and pedestrians, etc., which are not specifically limited here.

[0109] The first server 103 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0110] In this architecture, a first user terminal 101 collects a large number of target images (i.e., database images) as the basis for image retrieval. These database images are stored on a first server 103. First server 103 also receives images to be retrieved from a second user terminal 102. Once the image to be retrieved matches an image in the database, that database image becomes the retrieval result image.

[0111] Figure 1BThis is another system architecture diagram of the image retrieval method of the embodiment of the present application. Figure 1B The system architecture includes a third user terminal 104 and a second server 105. The third user terminal 104 and the second server 105 are directly or indirectly connected via wired or wireless communication. In one example, the third user terminal 104 and the second server 105 can both be nodes in the blockchain, which is not specifically limited in this embodiment.

[0112] The third user terminal 104 may include, but is not limited to, smart devices such as smartphones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, and aircraft. Optionally, the third user terminal 104 may be equipped with an image acquisition client. When the image acquisition client is in operation, the image acquisition client may capture an image to be retrieved and send it to the second server 105. The image to be retrieved is the image for image retrieval. The image to be retrieved may include a variety of simple or complex graphic-level content. For example, this graphic-level content may include a large amount of background information, such as the sky and lawn, or may include information about various objects, such as cars, roads, and pedestrians, without specific limitations herein.

[0113] The second server 105 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN networks, and big data and artificial intelligence platforms.

[0114] Figure 1B and Figure 1A The difference is that Figure 1A and Figure 1B It is clearly shown that the base image and the image to be retrieved come from two different user terminals. Figure 1B In the figure, only the sender of the image to be retrieved is shown, and the sender of the base image is not shown. In fact, the second server 105 can collect a large number of target images (i.e., base images) by itself. For example, it collects a large number of target images from public pictures on the Internet.

[0115] For example Figures 2A to 2E As shown, Figures 2A to 2E 1 is a schematic diagram of a scene interface in which the image retrieval method of an embodiment of the present application is applied to a scene of an image search engine. When a user searches for an image through the third user terminal 104, the third user terminal 104 may display the following: Figure 2AThe picture search interface shown in the figure shows a prompt message 201 of "Please enter the search picture" and a picture input control 202. When the user triggers the picture input control 202, the picture search interface will jump to the following display: Figure 2B The picture upload interface shown in FIG. 1 shows a local file directory 203 and a picture upload control 204 in the third user terminal 104. When the user selects an image to be retrieved in the local file directory 203 and triggers the picture upload control 204, the third user terminal 104 sends the image to be retrieved selected by the user to the second server 105. After the second server 105 receives the image to be retrieved, the second server 105 can display the following information to the backend reviewer: Figure 2C The image review interface shown in the figure shows a guide message 205 of "Please enter the base image" and an image input control 206. When the background reviewer triggers the image input control 206, the image review interface will jump to the following display: Figure 2D The image input interface shown in FIG. 2 shows the base file directory 207 and the image input control 208 in the second server 105. When the background reviewer selects the base image (i.e., the target image) in the base file directory 207 and triggers the image input control 208, the second server 105 will display the following image: Figure 2E The message notification interface shown in the figure displays a notification message notifying the backend reviewer that the base image has been entered. The notification message also informs the backend reviewer that they can continue to enter other base images after a certain period of time to perform a new round of image matching with the image to be retrieved. While the second server 105 displays the message notification interface, the second server 105 will perform image matching between the base image selected by the backend reviewer and the image to be retrieved. After the second server 105 switches from the message notification interface to the image review interface, the second server 105 can display the matching results between the base image and the image to be retrieved in the image review interface, so that the backend reviewer can see the matching results in the image review interface. If the matching result indicates that the base image and the image to be retrieved match, then the base image is the search result image. If the matching result indicates that the base image and the image to be retrieved do not match, the backend reviewer can continue to enter other base images through the image review interface, so that the second server 105 can continue to perform image retrieval processing for the image to be retrieved based on the other base images.

[0116] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on the target user's attribute information or attribute information set and other data related to the characteristics of the target user, the target user's permission or consent will be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the target user's attribute information, the target user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the target user's separate permission or consent, the target user's relevant data necessary to enable the normal operation of the embodiment of the present application will be obtained.

[0117] Figure 3 This is a flowchart of an image retrieval method provided in an embodiment of the present application. The image retrieval method can be executed by a server, or by a user terminal, or by both a user terminal and a server. In the embodiment of the present application, the method is described as being executed by a server. Figure 3 , the image retrieval method includes but is not limited to steps 310 to 350.

[0118] Step 310: performing image segmentation on the plurality of target images to obtain a plurality of target segmented images of each target image.

[0119] The target image, or base image, is the image used as the basis for comparison in image retrieval. In one embodiment, the target image can be a binary image, a grayscale image, an indexed image, or a true color RGB image. The target image can include multiple different main contents. For example, if the target image is a natural landscape image, the target image can include main contents such as trees, roads, and sky, without specific limitation.

[0120] By dividing a large-sized target image into a plurality of small-sized target block images, the computational complexity of subsequent image processing can be reduced, thereby facilitating improving the efficiency of subsequent image processing.

[0121] It should be noted that the specific process of performing image segmentation on the target image to obtain corresponding multiple target segmented images will be given in detail in subsequent content and will not be described here in detail.

[0122] Step 320: For each target image, obtain the attention thermal information of each target block image, and perform image feature extraction on each target block image based on the attention thermal information of each target block image to obtain the block image features of each target block image.

[0123] In one embodiment, the image's attention thermal information is information used to characterize the degree of attention or weight distribution of different regions within the image. This information can be used to analyze the importance of different features or regions within the image. Therefore, by extracting image features from the target segmented image based on its attention thermal information, it is possible to obtain segmented image features that more accurately represent the target segmented image, thereby facilitating the acquisition of more accurate image features for the target image in subsequent steps.

[0124] In one embodiment, the attention thermal information of the target block image can be an attention thermal matrix, and each element in the attention thermal matrix corresponds one-to-one to each pixel point in the target block image. If the value of an element in the attention thermal matrix is ​​a specific numerical value (for example, 1 or other non-zero numerical value), it can be considered that the pixel point corresponding to the element in the target block image is an image key point, and the feature corresponding to the image key point is more important; if the value of an element in the attention thermal matrix is ​​0, it can be considered that the pixel point corresponding to the element in the target block image is not an image key point, and the feature corresponding to the image key point is not very important.

[0125] It should be noted that the specific process of obtaining the attention thermal information of the target block image and the specific process of extracting image features of the target block image based on the attention thermal information of the target block image will be given in detail in the subsequent content and will not be repeated here.

[0126] Step 330: For each target image, integrate the block image features of multiple target block images into the image features of the target image.

[0127] In one embodiment, after obtaining the block image features of multiple target block images of the target image, the block image features of all target block images of the target image can be integrated together, and all these integrated block image features can be represented as the overall image feature of the target image. Since the block image features of multiple target block images of the target image are integrated into the overall image feature, these block image features can accurately represent each block portion of the target image.

[0128] Step 340: Receive the image to be retrieved, perform feature extraction on the image to be retrieved, and obtain features of the image to be retrieved.

[0129] In one embodiment, the image to be retrieved may be a binary image, a grayscale image, an indexed image, or a true color RGB image, etc. The image to be retrieved may include multiple different main contents. For example, when the image to be retrieved is a natural landscape image, the image to be retrieved may include main contents such as trees, roads, and sky, which are not specifically limited here.

[0130] In one embodiment, when feature extraction is performed on an image to be retrieved to obtain features of the image to be retrieved, a pre-trained feature extraction model can be called to perform feature extraction on the image to be retrieved, wherein the pre-trained feature extraction model can be composed of a conventional deep neural network model or a convolutional neural network model, etc. For the relevant model structure and model principles of the deep neural network model or the convolutional neural network model, please refer to the introduction of the relevant technology, which will not be repeated here.

[0131] Step 350: Determine a retrieval result image from the multiple target images based on the matching of the image features to be retrieved with the image features of the multiple target images.

[0132] In one embodiment, the number of retrieval result images determined from multiple target images based on the matching of features of the image to be retrieved with features of multiple target images can be one or more, and is not specifically limited herein. For example, when there is only one retrieval result image, the retrieval result image is the target image with the highest degree of match with the image to be retrieved among all target images. When there are multiple retrieval result images, the retrieval result images are multiple target images with a degree of match with the image to be retrieved greater than a preset match threshold, or multiple target images with the top N matching degrees, where N is a positive integer. These retrieval result images can be displayed in descending order of matching degree. It should be noted that the preset match threshold can be appropriately selected based on actual application needs and is not specifically limited herein. For example, the preset match threshold can be 0.9 or 0.95. For example, if the target image has a match degree of 0.98 with the image to be retrieved and the preset match threshold is 0.95, then this target image can be used as the retrieval result image.

[0133] In one embodiment, after feature matching is performed between the image to be retrieved and multiple target images, a matching degree list can be obtained. The matching degree list contains the matching degrees between the image to be retrieved and each target image. All matching degrees in the matching degree list are arranged in descending order. Therefore, one or more target images with the highest ranking in the matching degree list can be selected as retrieval result images.

[0134] In some cases, the main image content in the image to be retrieved may only correspond to part of the image content in the target image, while the rest of the target image may also contain a large amount of background information. For example, suppose that the main image content in the image to be retrieved is a vehicle (for example, the vehicle occupies 90% of the image area in the image to be retrieved), and in the target image, the main image content is also the vehicle, but the image content of the target image also includes the sky and the lawn (for example, the sky occupies 50% of the image area in the target image, the lawn occupies 20% of the image area in the target image, and the vehicle occupies 30% of the image area in the target image). Therefore, if the traditional feature matching method is used for image retrieval, since the sky and grass contained in the target image are both background information, and these background information are not related to the main image content of the image to be retrieved, when the image to be retrieved is matched with the target image, these background information has no benefit to the feature matching process, but may increase the computational complexity of feature matching, thereby reducing the efficiency of image retrieval. In addition, since these background information occupies most of the image area of ​​the target image, that is, among the image features of the target image, the image features corresponding to these background information occupy the majority, while the image features matching the main image content (i.e., the vehicle) in the image to be retrieved (i.e., the image features corresponding to the vehicle) only occupy a relatively small part. Therefore, when the image to be retrieved is matched with the target image, it is easy to cause the image to be retrieved and the target image to be judged as a match failure due to the small proportion of the matched image features, but in fact, the image to be retrieved and the target image should be matched (because the main image content of both images is the vehicle). Therefore, using the traditional feature matching method for image retrieval is prone to problems such as poor image retrieval efficiency and unsatisfactory image retrieval accuracy.In order to overcome these problems, in one embodiment, through the image retrieval method including the previous steps 310 to 350, the multiple target images are firstly segmented to obtain multiple target block images of each target image; then, for each target image, the attention thermal information of each target block image is obtained, and the image feature extraction of each target block image is performed on each target block image according to the attention thermal information of each target block image to obtain the block image feature of each target block image. Since the block image features of the target block images are obtained by extracting the image features according to their attention thermal information, the block image features can express the target block images more accurately; then, for each target image, the multiple target block images obtained are extracted. The block image features of the block image are integrated into the image features of the target image. Since the block image features of multiple target block images of the target image are integrated into its overall image features, and these block image features can accurately express each block part of the target image, when the image to be retrieved is received, the feature extraction of the image to be retrieved is first performed to obtain the feature of the image to be retrieved, and then based on the matching of the feature of the image to be retrieved with the image features of multiple target images, the retrieval result image can be accurately determined among the multiple target images. Therefore, the problems of unsatisfactory image retrieval accuracy and low image retrieval efficiency caused by the traditional feature matching method for image retrieval can be improved, thereby effectively improving the accuracy and efficiency of image retrieval.

[0135] In one embodiment, referring to Figure 4 As shown, step 310 may include but is not limited to steps 410 to 440.

[0136] Step 410: Perform object detection on each of the multiple target images to obtain multiple first sub-block images of each target image.

[0137] Object detection refers to detecting objects in an image. Such objects may include, for example, faces, school bags, apples, etc. The first sub-block image refers to the image of the object detected by object detection in the target image.

[0138] In one embodiment, a commonly used object detection algorithm such as Fast R-CNN (Fast Region Convolutional Neural Networks) or an algorithm based on YOLO (You Only Look Once) can be used to perform object detection on the target image, thereby obtaining multiple first sub-block images of the target image. Specifically, when these object detection algorithms are used to perform object detection on the target image, not only can all objects in the target image be identified, but also the positions and categories of these objects can be determined. For example, when these object detection algorithms are used to perform object detection on the target image, not only can the objects in the target image be identified, but also the positions of each object can be marked using rectangular frames. For example, assuming that the target image contains a car, a pedestrian, and a bicycle, when these object detection algorithms are used to perform object detection on the target image, the car, pedestrian, and bicycle will be marked in the target image using rectangular frames, respectively. The image areas framed by the rectangular frames are all first sub-block images of the target image.

[0139] Step 420: Perform subject object detection on each of the multiple target images to obtain multiple second sub-block images of each target image.

[0140] Subject object detection refers to the detection of the most dominant object in an image. For example, assuming the target image contains a car, a pedestrian, and a bicycle, the above object detection will detect the car, pedestrian, and bicycle, resulting in three first sub-block images. However, subject object detection may only detect the pedestrian as the dominant object, resulting in a single second sub-block image. Since each target image can generate a second sub-block image, multiple target images can generate multiple second sub-block images. Generally, the subject object is the primary focus or point of interest in the target image.

[0141] In one embodiment, a common object detection algorithm such as Fast R-CNN or a YOLO-based algorithm can be first used to perform object detection on the target image to identify all objects in the target image. Then, based on the size, color, position, shape of the identified object and its relationship with other objects in the target image (such as positional relationship, subordinate relationship, etc.), the main object is determined among all the identified objects. At this time, the image area framed by the rectangular frame corresponding to the main object is the second sub-block image of the target image. For example, assuming that the objects in the identified target image include a first person and a second person located at the center of the image, and multiple third people located at the edge of the image, wherein the image areas occupied by the first person and the second person are comparable and both larger than the image area occupied by the third person, it can be determined that the first person and the second person are the main objects, and all the third people are non-main objects. At this time, the image area framed by the rectangular frame corresponding to the first person and the image area framed by the rectangular frame corresponding to the second person are both the second sub-block images of the target image.

[0142] Step 430: Perform semantic segmentation on the multiple target images respectively to obtain multiple third sub-block images of each target image.

[0143] In one embodiment, a pre-trained semantic segmentation model can be called to perform semantic segmentation on the target image to obtain multiple objects in the target image, and then multiple third sub-block images of the target image can be obtained based on the obtained multiple objects. The pre-trained semantic segmentation model can be composed of a conventional deep neural network model or a convolutional neural network model. For the relevant model structure and model principles of the deep neural network model or the convolutional neural network model, please refer to the introduction of the relevant technology, which will not be repeated here.

[0144] It should be noted that when semantic segmentation is performed on the target image, each pixel in the target image will be classified to distinguish different objects from the background. In other words, semantic segmentation can not only distinguish objects in the target image, but also distinguish different parts of the same object (such as different leaves of the same tree), and each pixel in the target image will be labeled with a corresponding category. For example, assuming that the target image contains people, sheep, dogs, and grass, then after semantic segmentation of the target image, people, sheep, dogs, and grass can be distinguished, and each pixel in the target image will be labeled with a corresponding category, where all pixels related to people will be labeled with the category representing people, all pixels related to sheep will be labeled with the category representing sheep, and so on.

[0145] Step 440: For each target image, determine a plurality of target block images based on a plurality of first sub-block images, a plurality of second sub-block images, and a plurality of third sub-block images.

[0146] In one embodiment, after obtaining a plurality of first sub-block images, a plurality of second sub-block images, and a plurality of third sub-block images for each target image, a plurality of corresponding target block images may be determined for each target image based on the corresponding plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images. For example, all of the first sub-block images, the second sub-block images, and the third sub-block images may be determined as the plurality of target block images for the target image; or all of the first sub-block images, the second sub-block images, and the third sub-block images may be mixed and shuffled, and then a portion of the images therefrom may be selected as the plurality of target block images for the target image; or, a portion of the first sub-block images, the second sub-block images, and the third sub-block images may be selected from the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images, and then the selected first sub-block images, the second sub-block images, and the third sub-block images may be used as the plurality of target block images for the target image. By determining multiple target block images of the target image based on multiple first sub-block images, multiple second sub-block images and multiple third sub-block images, the content of the target block images can be enriched, which is conducive to obtaining richer image features of the target image in subsequent steps.

[0147] In one embodiment, in the process of determining multiple target block images based on multiple first sub-block images, multiple second sub-block images, and multiple third sub-block images, the confidence levels of the multiple first sub-block images, multiple second sub-block images, and multiple third sub-block images can be first obtained, and then the first sub-block images, second sub-block images, and third sub-block images whose confidence levels meet predetermined conditions are integrated into multiple target block images. It can be understood that the confidence level of an image refers to the degree of trustworthiness or certainty of the results of a certain analysis or prediction of the image. In computer vision tasks, such as target detection, recognition, or segmentation, the model typically assigns a confidence level or probability value to each prediction result to indicate the reliability of the result. Since the first and second sub-block images are both detected based on object detection algorithms such as Fast R-CNN or YOLO-based algorithms, and the third sub-block image is predicted based on a pre-trained semantic segmentation model, corresponding confidence levels are simultaneously obtained when the first, second, and third sub-block images are obtained. Therefore, confidence levels for multiple first, second, and third sub-block images can be obtained. After obtaining the confidence levels for the multiple first, second, and third sub-block images, the first, second, and third sub-block images whose confidence levels meet predetermined conditions can be integrated into multiple target block images. The advantage of integrating the first, second, and third sub-block images whose confidence levels meet predetermined conditions into multiple target block images is that the selected target block images can be made more reliable, thereby improving the accuracy of image retrieval.

[0148] In one embodiment, the predetermined condition may be that the confidence level is greater than a preset confidence threshold, or may be that the weighted value of the confidence levels is greater than the preset confidence threshold, which is not specifically limited herein. For example, when the predetermined condition is that the confidence level is greater than the preset confidence threshold, the first sub-block image, the second sub-block image, and the third sub-block image having a confidence level greater than the preset confidence threshold may be determined as the target sub-block image from the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images, respectively. For example, assuming that the number of first sub-block images, second sub-block images, and third sub-block images is 3, among which, only one of the 3 first sub-block images has a confidence level greater than a preset confidence threshold, the confidence levels of two of the 3 second sub-block images are greater than the preset confidence threshold, and the confidence levels of two of the 3 third sub-block images are also greater than the preset confidence threshold, then it can be determined that the number of target block images is 5, one of which is the first sub-block image, and among the remaining 4, 2 are second sub-block images, and the last 2 are third sub-block images.

[0149] In addition, when the predetermined condition is that the weighted value of the confidence level is greater than a preset confidence threshold, the weighted values ​​of the confidence levels of the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images can be first calculated, and then, among the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images, the first sub-block images, the second sub-block images, and the third sub-block images having weighted values ​​greater than the preset confidence threshold are determined as target block images. When calculating the weighted values ​​of the confidence levels, the weights corresponding to the respective confidence levels can be first determined, and then the confidence levels can be multiplied by the corresponding weights to obtain the weighted values ​​of the confidence levels. In one embodiment, the weights corresponding to the various confidence levels can be determined based on the image processing methods corresponding to the respective confidence levels. For example, since the first sub-block image is obtained based on object detection, the second sub-block image is obtained based on subject object detection, and the third sub-block image is obtained based on semantic segmentation, the confidence level of the first sub-block image can be assigned a first weight, the confidence level of the second sub-block image can be assigned a second weight, and the confidence level of the third sub-block image can be assigned a third weight, wherein the first, second, and third weights are different. For example, the first weight can be 0.2 or another value, the second weight can be 0.3 or another value, and the third weight can be 0.5 or another value, without specific limitation herein. Using a weighted value greater than a preset confidence threshold as the predetermined condition not only takes into account the confidence level of each sub-block image itself, but also the credibility of the method used to divide the sub-block images. This allows for a more comprehensive consideration of factors when selecting the target sub-block image, thereby improving the accuracy of image retrieval.

[0150] It should be noted that the preset confidence threshold can be appropriately selected according to actual application conditions and is not specifically limited here. For example, the value of the preset confidence threshold can be 0.8 or 0.9.

[0151] In one embodiment, in the process of determining multiple target block images based on multiple first sub-block images, multiple second sub-block images, and multiple third sub-block images, a first center position of each first sub-block image, a second center position of each second sub-block image, and a third center position of each third sub-block image may be determined. If the first center position of a first sub-block image, the second center position of a second sub-block image, and the third center position of a third sub-block image meet a first condition (e.g., overlap, or the distance between them is less than a predetermined distance), it indicates that the first sub-block image, the second sub-block image, and the third sub-block image correspond to the same area in the target image. For example, assuming that the target image includes blue sky, grass, and a person, regardless of whether object detection, subject object detection, or semantic segmentation is used, the first sub-block image, the second sub-block image, and the third sub-block image containing the person should roughly correspond to the same area in the target image. Therefore, if the first center position of a first sub-block image, the second center position of a second sub-block image, and the third center position of a third sub-block image meet the first condition, then the overlapping area of ​​the first, second, and third sub-block images can be determined, and then the overlapping area can be used as the target sub-block image. Because the overlapping area is the area commonly pointed to by the three different division methods, this division can help improve the accuracy of the block division, thereby improving the accuracy of image retrieval.

[0152] It is understood that the center position of an image generally refers to the coordinates of the center point of the image in the horizontal and vertical directions. For example, in a two-dimensional coordinate system, if the lower left corner of the image is used as the origin of the coordinate system, the coordinates of the center point of the image can be calculated using the following formula: Center point abscissa = image width / 2, Center point ordinate = image height / 2, where image width and image height represent the number of pixels in the image in the horizontal and vertical directions, respectively.

[0153] In addition to the above-mentioned overlap or the distance between each other being less than the predetermined distance, the first condition may also be that the first center position, the second center position, and the third center position are within a certain area range. The area range may be a rectangular area range, a circular area range, or a triangular area range. This can be appropriately selected according to the actual application situation and is not specifically limited here. For example, assuming that the first condition is that the first center position, the second center position, and the third center position are within a rectangular area range with a side length of 0.1 cm, then when the first center position of the first sub-block image, the second center position of the second sub-block image, and the third center position of the third sub-block image are indeed within a rectangular area range of 0.1 cm*0.1 cm, it means that the first sub-block image, the second sub-block image, and the third sub-block image correspond to the same area in the target image. At this time, the overlapping area can be used as the target block image. For example Figure 5 As shown, the first center position 511 of the first sub-block image 510, the second center position 521 of the second sub-block image 520, and the third center position 531 of the third sub-block image 530 are located within a rectangular area 540. In this case, the first sub-block image 510, the second sub-block image 520, and the third sub-block image 530 form an overlapping area 550, so the overlapping area 550 can be determined as the target block image.

[0154] In one embodiment, referring to Figure 6 As shown, in the process of performing image segmentation on a plurality of target images to obtain a plurality of target segmented images of each target image, the process may include but is not limited to steps 610 to 630.

[0155] Step 610: Call the first block model to perform image block division on the multiple target images respectively to obtain multiple fourth sub-block images of each target image.

[0156] In one embodiment, the first block model can be a semantic segmentation model or other models that can realize the image segmentation function. The first block model can be composed of a conventional deep neural network model or a convolutional neural network model. For the relevant model structure and model principles of the deep neural network model or the convolutional neural network model, please refer to the introduction of the relevant technology, which will not be repeated here.

[0157] Step 620: Call the second block model to perform image block division on the multiple target images respectively, and obtain multiple fifth sub-block images of each target image.

[0158] In one embodiment, the second block model can be a model with the same structure as the first block model, but when pre-training the first block model and the second block model, the first block granularity of the first block model is made larger than the second block granularity of the second block model. The block granularity refers to the granularity of dividing an image into different regions. It is understandable that in the semantic segmentation task, the block granularity can affect the number of blocks of the target image. For example, the larger the block granularity, the larger the image block will be, and the number of image blocks will be relatively smaller; and the smaller the block granularity, the smaller the image block will be, and therefore the number of image blocks will be relatively more. When pre-training the first block model and the second block model, by making the first block granularity of the first block model larger than the second block granularity of the second block model, the first block model and the second block model can be distinguished at the block granularity level, so that the multiple fourth sub-block images obtained by the first block model and the multiple fifth sub-block images obtained by the second block model can have different scales, thereby improving the richness of the target block images obtained after image segmentation of the target image, and further improving the accuracy of image retrieval.

[0159] In one embodiment, when pre-training the first block model and the second block model, the first block model and the second block model with the same model structure can be constructed first, and then the first block model and the second block model are trained accordingly using two image samples with the same image content but different block granularities, so that the first block granularity of the first block model is different from the second block granularity of the second block model (for example, the first block granularity is greater than the second block granularity). In one embodiment, the first block granularity and the second block granularity can be appropriately selected according to the actual application situation, and are not specifically limited here. For example, the first block granularity can be 2, and the second block granularity can be 1.

[0160] Step 630: For each target image, integrate the plurality of fourth sub-block images and the plurality of fifth sub-block images into a plurality of target block images.

[0161] The integration may refer to taking all fourth sub-block images and all fifth sub-block images as target block images, or may refer to selecting sub-block images that meet predetermined criteria (for example, a confidence level greater than a predetermined threshold) from all fourth sub-block images and all fifth sub-block images as target block images.

[0162] In one embodiment, the first block granularity of the first block model and the second block granularity of the second block model are different. Therefore, the multiple fourth sub-block images obtained based on the first block model and the multiple fifth sub-block images obtained based on the second block model can have different image scales. Therefore, integrating the multiple fourth sub-block images and the multiple fifth sub-block images into multiple target block images can provide more comprehensive and accurate image information for the target image.

[0163] In one embodiment, during the process of segmenting multiple target images to obtain multiple target segmented images for each target image, the multiple target images may be segmented to obtain multiple candidate segmented images for each target image. Then, for each target image, multiple target segmented images having image areas greater than a preset area threshold are identified from the multiple candidate segmented images. This embodiment has the advantage of reducing the impact of inconspicuous segmented images on the overall image matching comparison (larger target segmented images are more likely to be considered during image matching comparison), thereby improving image retrieval accuracy.

[0164] Figure 7 This is a schematic diagram of an implementation architecture for performing image segmentation on a target image to obtain multiple candidate segmented images in an embodiment of the present application. Figure 7 In [1], the architecture can resolve spatial ambiguity by predicting masks of multiple granularities and obtain the corresponding image semantic labels. Figure 7 As shown, the architecture can include a visual encoder and a mask decoder, wherein the visual encoder can be used to extract image features; the mask decoder can be used for both conventional image segmentation and promptable segmentation of various types of prompts. For the input point, the architecture can input the point into the mask decoder through the anchor box for processing to obtain the output mask of the corresponding point. Since the granularity of the input point is fuzzy, each point can be copied multiple times (for example, 6 times) and given different embedding levels, and the output mask of the point can be matched with multiple masks of different granularities. By matching with multiple masks of different granularities, it can help the model better understand the structure and semantic information of the image and improve the accuracy and robustness of image segmentation. Reference Figure 8 As shown, using Figure 7 The process of performing image segmentation on the target image to obtain multiple candidate segmented images may include but is not limited to steps 810 to 830.

[0165] Step 810: Use a text encoder to encode the text in the target image to obtain an image semantic label.

[0166] It is understood that a text encoder is a technology that can convert text into vector representation, which can map the text (such as characters, words or sentences, etc.) in the target image into a vector space to obtain the text vector corresponding to the text. By mapping text into text vectors, it can be more conveniently used in machine learning, natural language processing or other text-related tasks. Figure 7 The text encoder in the semantic segmentation model of the architecture encodes the text in the target image, which can achieve the purpose of converting the text information in the target image into the image semantic label corresponding to the target image, so as to better understand the content of the target image and improve the efficiency and accuracy of subsequent image retrieval.

[0167] Step 820: Segment the target image based on the image semantic label to obtain segmentation frames.

[0168] In this step, segmentation lines are used to mark the boundaries of each region or object in the image segmentation result. They can be represented as a series of continuous lines or outlines, accurately describing the outer shape of each region in the segmentation result. By segmenting the target image based on image semantic labels, it is possible to separate and understand the image content in the target image, thereby obtaining more accurate segmentation results and segmentation lines, which in turn provide a more reliable and accurate foundation for subsequent image processing and analysis.

[0169] Step 830: Use a visual encoder to visually encode the frame lines in the target image to obtain a frame line mask. If the segmentation frame lines match the frame line mask, use the segmentation frame lines to segment the target image into blocks to obtain multiple candidate block images.

[0170] It can be understood that the visual encoder is a technology that can convert an image into a vector representation. It can map the pixels in the target image into a vector space to obtain the vector representation of the pixels, and then obtain the image vector of the entire target image. Since the frame lines in the target image are actually composed of a series of pixels, the visual encoder can also be used to visually encode the frame lines in the target image. Figure 7 The visual encoder in the semantic segmentation model of the architecture visually encodes the frame lines in the target image to obtain the corresponding frame line mask. If the segmentation frame lines match the frame line mask, it means that the segmentation frame lines are accurate. Therefore, the segmentation frame lines can be used to segment the target image, thereby obtaining multiple corresponding candidate segmented images.

[0171] In one embodiment, in the process of using a text encoder to perform text encoding on the text in the target image to obtain an image semantic label, the text information in the target image can be first extracted, and the text information can be text, slogans, or titles, etc.; the extracted text information is then cleaned and segmented, wherein the cleaning process can include removing special characters, punctuation marks, and stop words, etc., and the segmentation process can be segmenting the text information according to parts of speech. For example, for the text information "beautiful scenery", when segmenting it according to parts of speech, since "beautiful" is an adjective and "scenery" is a noun, the text information obtained after segmentation includes the two words "beautiful" and "scenery"; then, the text information that has undergone cleaning and segmentation is converted into a format that can be processed by the text encoder; then, the text encoder is called to encode the text information after the conversion to obtain a semantic representation of the text information; at this time, the semantic representation of the text information can be used as an image semantic label of the target image to describe the image content of the target image.

[0172] In one embodiment, in the process of segmenting a target image based on an image semantic label to obtain a segmentation frame line, the image semantic label of the target image can be first used as an input parameter, and the target image can be segmented using an image segmentation algorithm to obtain a preliminary segmentation result corresponding to the image semantic label, wherein the image segmentation algorithm can include a commonly used semantic segmentation algorithm or an instance segmentation algorithm, etc., which is not specifically limited here; then, the preliminary segmentation result is post-processed, such as removing small areas or filling holes, etc., to eliminate possible noise and discontinuities, and obtain a smoother and more accurate target segmentation result; then, based on the target segmentation result, an edge detection algorithm or a contour extraction algorithm can be called to extract the frame line to obtain the corresponding segmentation frame line. It should be noted that edge detection algorithms or contour extraction algorithms are commonly used algorithms in this field, which can extract the edges or contours of the target segmentation result to form a segmentation frame line. Therefore, edge detection algorithms or contour extraction algorithms are not described in detail here.

[0173] In one embodiment, in the process of visually encoding the frame lines in the target image using a visual encoder to obtain a frame line mask, the frame lines in the target image can be first obtained, wherein the frame lines in the target image can be the edge, outline, or rectangular frame of the main object in the target image; then the frame lines are preprocessed, such as removing noise, smoothing edges, etc.; then, the preprocessed frame lines are input into the visual encoder for visual encoding to obtain a visual representation of the frame lines; then, the visual representation is used as the frame line mask corresponding to the frame lines to describe the position and shape of the frame lines. By using the visual encoder to convert the frame lines into a frame line mask, it can help to better describe and understand the content of the target image, thereby providing a more reliable and accurate data basis for subsequent image feature extraction.

[0174] In one embodiment, after obtaining a plurality of candidate block images of the target image, these candidate block images may be further processed, such as noise removal and edge smoothing, so as to improve the image quality of the candidate block images.

[0175] In one embodiment, after obtaining multiple candidate segmented images of the target image, multiple target segmented images whose image areas are greater than a preset area threshold value can be determined from these candidate segmented images. It is understandable that, among the multiple candidate segmented images of the target image, if the image area is too small, it means that the corresponding candidate segmented image is not an important segmented image, and the image features of the candidate segmented image will not significantly help the image features of the target image; if the image area is relatively large, it means that the corresponding candidate segmented image is a relatively important segmented image, and the image features of the candidate segmented image can significantly help the image features of the target image. Therefore, after obtaining multiple candidate segmented images of the target image, the multiple candidate segmented images can be screened by area according to the preset area threshold value to exclude candidate segmented images with too small an area, so as to obtain target segmented images that better meet the requirements, thereby improving the accuracy of image retrieval.

[0176] In one embodiment, the preset area threshold can be appropriately selected based on actual application conditions and is not specifically limited herein. For example, the preset area threshold can be a fixed constant or 1 / 10 of the image area of ​​the target image. For example, after segmenting a target image containing at least a road, a car, and a pedestrian, candidate segmented images containing roads, candidate segmented images containing cars, and candidate segmented images containing pedestrians can be obtained. In this case, when screening these candidate segmented images based on the preset area threshold, assuming the preset area threshold is 1 / 10 of the image area of ​​the target image, if the image area of ​​the candidate segmented image containing pedestrians is less than 1 / 10 of the image area of ​​the target image, the candidate segmented image containing pedestrians will be excluded; if the image area of ​​the candidate segmented image containing cars is greater than 1 / 10 of the image area of ​​the target image, the candidate segmented image containing cars can be determined as the target segmented image of the target image. By excluding candidate segmented images with too small an image area, a target segmented image that better meets the requirements can be obtained, thereby improving the accuracy of subsequent acquisition of image features of the target image.

[0177] In one embodiment, after obtaining multiple target segmented images of the target image, attention thermal information corresponding to each target segmented image can also be obtained, so that the attention thermal information can be used to guide the extraction process of the segmented image features of the target segmented images, thereby obtaining more accurate segmented image features of the target segmented images. Figure 9 As shown, the process of obtaining the attention thermal information of each target block image may include but is not limited to steps 910 to 920.

[0178] Step 910: Obtain image key points in each target block image.

[0179] It should be noted that image key points refer to pixels in the target block image that have significant features and can identify the image content in the target block image. Each target block image will have multiple image key points. In one embodiment, different key point detection algorithms such as SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded Up Robust Features) algorithm or ORB (Oriented FAST and Rotated BRIEF) algorithm can be used to obtain image key points in the target block image, which is not specifically limited here. Among them, the SIFT algorithm constructs a scale space pyramid and then detects image key points at multiple scales, so that image key points with uniqueness and stability can be extracted from the image. The SURF algorithm uses the Hessian matrix to detect image key points and uses a box filter to approximate the determinant of the Hessian matrix, so as to achieve fast key point detection. The ORB algorithm uses the directional FAST (Features from Accelerated Segment Test) algorithm to detect image key points, so that each image key point has a direction associated with it, making the algorithm rotation-invariant. In addition, the ORB algorithm also generates descriptors by using the rotated BRIEF (Binary Robust Independent Elementary Features) algorithm and further enhances the robustness of the algorithm by rotating the descriptors. By detecting important image key points in the target image, it is helpful to determine the pixels with significant features in the target block image, thereby facilitating the extraction of more accurate feature information of the target block image.

[0180] Step 920: Obtain attention thermal information of the target block image for the image key points.

[0181] Attention thermal information refers to information that reflects the degree of attention paid to each part of the image when performing image matching and comparison.

[0182] In one embodiment, the attention thermal information of the target block image is an attention thermal matrix, and each element in the attention thermal matrix corresponds one-to-one to each pixel point in the target block image. If the value of an element in the attention thermal matrix is ​​a specific numerical value (for example, 1 or other non-zero numerical value), it can be considered that the pixel point corresponding to the element in the target block image is an image key point, and the feature corresponding to the image key point is more important; if the value of an element in the attention thermal matrix is ​​0, it can be considered that the pixel point corresponding to the element in the target block image is not an image key point, and the feature corresponding to the image key point is not very important. Therefore, after obtaining the image key points in the target block image, since each element in the attention thermal information (i.e., the attention thermal matrix) corresponds one-to-one to each pixel point in the target block image, an initial attention thermal matrix can be initialized first, and the non-image key points in the target block image are determined based on the image key points. Then, the values ​​of the elements corresponding to the image key points in the initial attention thermal matrix are set to the specific values ​​(e.g., 1 or other non-zero values), and the values ​​of the elements corresponding to the non-image key points in the initial attention thermal matrix are set to 0, and the attention thermal information of the target block image can be obtained. Since the non-zero elements in the attention thermal information correspond one-to-one to the image key points in the target block image, the attention thermal information of the target block image can not only remove the background interference in the target block image, but also help to highlight the expression of the image key points in the target block image in the target block image, thereby facilitating the subsequent steps to obtain more accurate feature information of the target block image.

[0183] In one embodiment, in the process of obtaining image key points in each target segmented image, key point detection can be first performed on the target image to obtain an image key point distribution of the target image. Then, based on the position information of each target segmented image in the target image, the image key points of each target segmented image can be obtained from the image key point distribution. When key point detection is performed on the target image, multiple image key points in the target image can be obtained. These image key points in the target image form an image key point distribution, which can reflect the distribution of the image key points in the target image. Since the target segmented image is obtained based on image segmentation of the target image, the target segmented image is part of the target image. Therefore, in one embodiment, it is not necessary to perform key point detection on the target segmented image separately to obtain the image key points of the target segmented image. Instead, the image key points of the target segmented image can be obtained from the image key point distribution based on the position information of the target segmented image in the target image. For example, assuming that the obtained image key point distribution of the target image includes the central area and the lower left area of ​​the target image, and the target block image currently under consideration is located in the central area of ​​the target image, in this case, the image key points of the target block image can be obtained by obtaining multiple image key points located in the central area of ​​the target image from the image key point distribution of the target image. There is no need to perform key point detection on the target block image separately, thereby improving the efficiency of obtaining the image key points of the target block image. In one embodiment, different key point detection algorithms such as the SIFT algorithm, the SURF algorithm, or the ORB algorithm can be used to perform key point detection on the target image. Appropriate selection can be made based on actual application needs and is not specifically limited here.

[0184] In one embodiment, referring to Figure 10 As shown, the process of performing key point detection on the target image to obtain the image key point distribution of the target image may include but is not limited to steps 1010 to 1030.

[0185] Step 1010: Perform object recognition on the target image to obtain a recognized object.

[0186] In one embodiment, after performing object recognition on a target image, the category and location information of the objects contained in the target image can be obtained. For example, a common object detection algorithm such as Fast R-CNN or a YOLO-based algorithm can be used to identify the objects in the target image, although this is not specifically limited herein. For example, if the target image contains a tree and a person, the identified objects will include the tree and the person.

[0187] Step 1020: Determine key point positioning rules for the object based on the type of the identified object.

[0188] In one embodiment, key point localization rules refer to rules or methods used to determine the locations of key points on an object or human body during object detection or pose estimation tasks. By rationally designing and applying these rules, the accuracy and stability of key point localization can be improved. Because different objects are of different types, their representative points (i.e., key points) naturally differ; that is, their key point distributions also differ. For example, a car and a face have different representative points (i.e., key points), i.e., their key point distributions differ, so the key point localization rules for the two are different. Key point localization rules for a car are not applicable to a face. For example, for a car, the corresponding key point localization rule is to use the center of the wheel, the four corners of the door, and each inflection point of the front contour as key points; whereas for a face, the corresponding key point localization rule might be to use the corners of the mouth, upper lip, lower lip, nose tip, left sinus, right sinus, etc. as key points. Therefore, different objects can have corresponding key point localization rules. Invoking different key point localization rules based on the different object types can effectively improve the accuracy of key point acquisition.

[0189] In addition, in one embodiment, even for the same object, when its posture is different, the corresponding key point distribution will be different. For example, for a face, when the posture of the face is frontal, the corresponding key point distribution includes the front face edge points and some points on the facial features, while when the posture of the face is side face, the corresponding key point distribution includes some points in the facial features and the side face edge points. That is to say, even for the same object, when its posture is different, the corresponding key point positioning rules should also be different. Therefore, after determining the type of the identified object, the corresponding key point positioning rules can be further determined according to the posture of the object, so as to further improve the accuracy of obtaining key points.

[0190] Step 1030: Based on the key point positioning rule, key points are positioned on the object, thereby forming an image key point distribution of the target image.

[0191] In one embodiment, key points on an object are located using key point location rules, enabling more detailed and precise key point location, thereby more accurately obtaining the key point distribution of the target image. For example, for a frontal face, the corresponding key point location rule locates key points based on the frontal face edge points and certain points on the facial features. Therefore, based on this key point location rule, multiple key points can be located on the object based on the frontal face edge points and certain points on the facial features. The image key point distribution of the target image can then be obtained based on the distribution of these located key points in the target image.

[0192] In one embodiment, the attention thermal information of the target block image can be an attention thermal matrix, and each element in the attention thermal matrix corresponds one-to-one to each pixel in the target block image. Therefore, when obtaining the attention thermal information of the target block image based on the image key point, an initial attention thermal matrix can be established first, and then each pixel in the target block image can be traversed. If a pixel in the target block image is an image key point, the value of the element corresponding to the pixel in the initial attention thermal matrix is ​​set to the first element value; if a pixel in the target block image is not an image key point, the value of the element corresponding to the pixel in the initial attention thermal matrix is ​​set to 0. The first element value can be a preset value, such as 1 or other non-zero values, or the first element value can also be the pixel value of the pixel. For example, if a pixel in the target block image is an image key point, the value of the element corresponding to the pixel in the initial attention thermal matrix can be set to the pixel value of the pixel.

[0193] In one embodiment, referring to Figure 11 As shown, Figure 11 It is a schematic diagram of the process of obtaining the attention thermal information of the target block image based on the image key points of the target block image. Figure 11In the embodiment, for a target image 1100 (the squares in the target image 1100 represent pixel points in the target image 1100), the image is first divided into blocks to obtain a plurality of target block images 1110, and the position information of each target block image 1110 in the target image 1100 is determined. Then, key point detection is performed on the target image 1100 to obtain an image key point distribution 1120 of the target image 1100 (wherein the positions with black dots represent image key points, and the positions without black dots represent non-image key points). Then, for each target block image 1110, according to the position information of the target block image 1110 in the target image 1100, the image key point distribution 1120 is obtained. The image key points 1130 in the target block image 1110 are obtained from the key point distribution 1120. Then, an initial attention heat matrix is ​​constructed. Each element in the initial attention heat matrix corresponds one-to-one to each pixel in the target block image 1110. At this time, each pixel in the target block image 1110 is traversed. If a pixel in the target block image 1110 is an image key point 1130, the value of the element corresponding to the pixel in the initial attention heat matrix is ​​set to 1; if a pixel in the target block image 1110 is not an image key point 1130, the value of the element corresponding to the pixel in the initial attention heat matrix is ​​set to 0. After traversing all pixels in the target block image 1110 and assigning values ​​to the corresponding elements in the initial attention heat matrix, the assigned initial attention heat matrix is ​​the attention heat information 1111 of the target block image 1110.

[0194] In one embodiment, in order to find enough accurate key points and obtain more accurate attention thermal information, a key point detection model that decouples the detector and the descriptor generator can be called to detect key points of the target image. That is, in step 910, the image key points in each target block image can be obtained by Figure 12 The architecture is implemented. The architecture includes a detector and a descriptor generator. The detector uses the self-stabilizing properties of the detection points, so that the architecture can be trained based on unsupervised datasets without being limited by the complexity of data annotation. In addition, the architecture can also output a very large number of key points (for example, 10,000+) for a single image, so that accurate attention thermal information of the image can be obtained. Figure 12As shown, the architecture includes a detector and a descriptor generator that are decoupled from each other. The detector detects the key point position in the target image, and the descriptor generator generates a descriptor corresponding to the key point position for the key point position. At the same time, the motion trend of each key point position in the target image, that is, the three-dimensional motion trajectory, is learned from the SFM (Structure from Motion) model. Then, the descriptor corresponding to the key point position is matched with the three-dimensional motion trajectory to determine whether the key point position is on the three-dimensional motion trajectory. In this way, when performing key point detection on the target image, only one calculation is needed to obtain the image key point distribution of the target image, thereby effectively improving the efficiency of obtaining the image key point distribution of the target image, and further effectively improving the efficiency of obtaining the attention thermal information of the target block image.

[0195] In one embodiment, after obtaining the attention thermal information of each target block image, image feature extraction can be performed on each target block image based on the attention thermal information of each target block image, thereby obtaining the block image features of each target block image. For each target block image, in the process of extracting its image features based on its attention thermal information, basic feature extraction and local feature enhancement can be performed on the target block image first to obtain the basic image features and local image enhancement features of the target block image. Then, based on the attention thermal information, basic image features, and local image enhancement features of the target block image, the block image features of the target block image are generated. Generally, the image data in the target block image is high-dimensional data, which easily leads to difficulties in calculating and storing features. However, by performing basic feature extraction on the target block image, the image data of the target block image can be mapped from a high-dimensional space to a low-dimensional space, reducing the dimensionality of the image data of the target block image, thereby effectively simplifying the complexity of subsequent processing and analysis of the basic image features of the target block image. Furthermore, by performing local feature enhancement on the target segmented image, more important local features in the target segmented image can be enhanced, thereby improving the expression of the image content of the target segmented image. Furthermore, because the attentional thermal information of the target segmented image can enhance the feature importance of key points in the target segmented image, the segmented image features generated based on the attentional thermal information of the target segmented image, the basic image features, and the local image enhancement features can more accurately express the target segmented image, thereby facilitating the acquisition of more accurate image features of the target image.

[0196] In one embodiment, a commonly used feature extraction model can be used to extract basic features of the target block image, thereby obtaining the basic image features of the target block image. The commonly used feature extraction model can be composed of a conventional deep neural network model or a convolutional neural network model. For the relevant model structure and model principle of the deep neural network model or the convolutional neural network model, reference can be made to the introduction of the relevant technology, which will not be elaborated here. In addition, the local image enhancement features of the target block image can be obtained based on the normalized average pooling of the basic image features of the target block image. Among them, normalized average pooling is a common pooling operation that can improve the discrimination ability of image features to a certain extent.

[0197] In one embodiment, the basic image features of the target block image are the pixel value matrix of each pixel in the target block image, and the local image enhancement features of the target block image are the pixel enhancement value matrix of each pixel in the target block image. In addition, since the attention thermal information is an attention thermal matrix, and each element in the attention thermal matrix corresponds one-to-one to each pixel in the target block image, and if a pixel in the target block image is an image key point, the value of the element corresponding to the pixel is the first element value; if a pixel in the target block image is not an image key point, the value of the element corresponding to the pixel is 0. Therefore, in this case, when generating the block image features of the target block image based on the attention thermal information, basic image features and local image enhancement features of the target block image, the pixel value matrix can be first sensitively adjusted based on the attention thermal matrix to obtain a first matrix, and then the pixel enhancement value matrix can be background noise suppressed based on the attention thermal matrix to obtain a second matrix. Then, based on the first matrix and the second matrix, the total matrix is ​​determined, and the total matrix is ​​used as the block image features of the target block image. In the process of performing sensitivity adjustment on the pixel value matrix based on the attention thermal matrix to obtain a first matrix, the elements of the attention thermal matrix can be first multiplied by a first ratio and added by 1 to obtain a sensitivity adjustment coefficient matrix. The pixel value matrix and the sensitivity adjustment coefficient matrix are then multiplied together to obtain the first matrix. Since the attention thermal matrix can express the image key points in the target block image, the image key points therein are more sensitive to the target block image. Therefore, by multiplying the elements of the attention thermal matrix by the first ratio and adding 1, a sensitivity adjustment coefficient matrix for sensitivity adjustment can be obtained. At this point, the pixel value matrix and the sensitivity adjustment coefficient matrix are then multiplied together to achieve sensitivity adjustment of the pixel value matrix. The value of the first ratio can be appropriately selected based on the actual application and is not specifically limited here. For example, the value of the first ratio can be 0.3 or 0.5. In addition, in the process of performing background noise suppression on the pixel enhancement value matrix based on the attention thermal matrix to obtain a second matrix, the pixel enhancement value matrix can be multiplied by the attention thermal matrix to obtain the second matrix. Since the attention heat matrix can express the image key points in the target block image, when the pixel enhancement value matrix is ​​multiplied by the attention heat matrix, the feature information of more important pixels in the pixel enhancement value matrix can be further improved. Moreover, since the value of the elements corresponding to non-image key points in the attention heat matrix is ​​0, after multiplying the pixel enhancement value matrix by the attention heat matrix, the pixel values ​​related to background noise in the pixel enhancement value matrix can be changed to 0, thereby achieving the purpose of background noise suppression.

[0198] In one embodiment, in the process of determining a total matrix based on the first matrix and the second matrix, a first weight of the first matrix and a second weight of the second matrix can be first obtained, and then, based on the first weight and the second weight, a weighted sum of the first matrix and the second matrix is ​​performed to obtain a total matrix. Since the first matrix is ​​a matrix obtained by sensitivity adjustment and the second matrix is ​​a matrix obtained by background noise suppression, the first matrix can play an additional role in enhancing sensitive features in the target block image, while the second matrix can suppress background noise in the target block image. Therefore, using the total matrix obtained based on the first matrix and the second matrix as the block image feature of the target block image can effectively suppress background noise in the target block image and highlight the important content of the target block image, thereby effectively improving the quality of the block image feature of the target block image. In addition, the first weight and the second weight can be appropriately selected according to the actual application situation and are not specifically limited here. For example, the values ​​of the first weight and the second weight can both be 0.4 or 0.5.

[0199] In one embodiment, the total matrix of the block image features used as the target block image can be calculated using the following formula (1):

[0200] X=a*D*(1+Mask*c)+b*g r *Mask (1)

[0201] In formula (1), X is the total matrix of the block image features used as the target block image; Mask is the attention heat matrix; D is the pixel value matrix; g r is the pixel enhancement value matrix; a is the first weight; b is the second weight; c is the first ratio. Therefore, after obtaining the attention heat matrix, the pixel value matrix, and the pixel enhancement value matrix, the first ratio, the first weight, and the second weight can be determined first, and then the attention heat matrix, the pixel value matrix, the pixel enhancement value matrix, the first ratio, the first weight, and the second weight can be input as input parameters into formula (1) to calculate the total matrix used as the block image feature of the target block image. Alternatively, after determining the first ratio, the first weight, and the second weight, the first ratio, the first weight, and the second weight can be input into formula (1) first, and formula (1) can be fixed to the following formula (2) (assuming that the first ratio is 0.3 and the first weight and the second weight are both 0.5):

[0202] X=0.5*D*(1+Mask*0.3)+0.5*g r *Mask (2)

[0203] Then, the attention heat matrix, the pixel value matrix and the pixel enhancement value matrix are input as input parameters into formula (2), and the total matrix used as the block image feature of the target block image can be calculated.

[0204] In one embodiment, each element in the pixel enhancement value matrix is ​​a first value, wherein the first value can be determined by first performing enhancement processing on each pixel point in the target segmented image to obtain an enhanced pixel value of the pixel point, then determining an average value of the pixel value and the enhanced pixel value of the pixel point, and then determining the first value based on the average value of each pixel point in the target segmented image. The first value can be calculated using the following formula (3):

[0205]

[0206] In formula (3), D h,w Represents the pixel value of the pixel point in the target block image (i.e., pixel value matrix); M h,w Indicates D h,w Enhanced pixel values ​​of pixels obtained after performing enhancement processing (such as normalized average pooling); Represents the average value of the pixel value and the enhanced pixel value of the pixel point; W represents a weight value, which can be appropriately selected according to the actual application situation and is not specifically limited here; b is a constant used to adjust the degree of enhancement of the local features of the target block image; p represents the adjustment parameter for the degree of normalized average pooling processing of the basic image features. Therefore, after obtaining the pixel value of each pixel point in the target block image (i.e., the pixel value matrix), each pixel point in the target block image can be enhanced to obtain the enhanced pixel value of the pixel point, and then the pixel value of the pixel point (i.e., the pixel value matrix) and the enhanced pixel value are used as input parameters and input into formula (3) to calculate the first value, that is, the value of each element in the pixel enhancement value matrix can be obtained.

[0207] Reference Figure 13 As shown, Figure 13 This is a flow chart of obtaining the block image features of the target block image provided by the embodiment of the present application. Figure 13 In , before storing the target image in the database, the semantic segmentation model can be called to segment the target image into blocks, and multiple target block images of the target block image can be obtained, such as Figure 13The target block image M1, target block image M2 and target block image M3 shown in ; then call the key point detection model to obtain the image key point distribution of the target image, and obtain the image key point of each target block image in the image key point distribution according to the position information of each target block image in the target image; then, for each target block image, the attention thermal information of the target block image is obtained based on the image key point, as shown in FIG. Figure 13 The attention thermal information R1, attention thermal information R2 and attention thermal information R3 shown in , wherein the attention thermal information is an attention thermal matrix, and each element in the attention thermal matrix corresponds to each pixel point in the target block image one by one, if a pixel point in the target block image is an image key point, the value of the element corresponding to the pixel point is set to the first element value, if a pixel point in the target block image is not an image key point, the value of the element corresponding to the pixel point is set to 0; after obtaining the attention thermal information of each target block image, call the pre-trained feature extraction model, and extract image features of each target block image according to the attention thermal information of each target block image to obtain the block image features of each target block image, such as Figure 13 As shown in FIG, block image feature F1, block image feature F2, and block image feature F3. Since the attention thermal information corresponding to each target block image is used to extract the features of each target block image during the feature extraction process for each target block image, the obtained block image features of all target block images can accurately express each block portion of the target image. Therefore, when the image to be retrieved is received, the feature extraction of the image to be retrieved is first performed to obtain the feature of the image to be retrieved. Then, based on the matching of the feature of the image to be retrieved with the image features of multiple target images, the retrieval result image can be accurately determined from the multiple target images. Unlike the related art, there is no need to first perform a rough sorting process and then a fine sorting process. Therefore, the problems of unsatisfactory image retrieval accuracy and low image retrieval efficiency in the related art can be improved, thereby effectively improving the accuracy and efficiency of image retrieval.

[0208] As shown in Table 1 below, Table 1 exemplarily provides a comparison result of the effects achieved by the method for obtaining the image features of the target image provided by the embodiment of the present application and other feature extraction methods in the related art.

[0209] Table 1

[0210] Model selection ROxford dataset RParis dataset Baseline effect model 63.82 76.83 New Semantic-SAM framework 72.35 82.91 The method used in the embodiment of the present application 77.44 85.8

[0211] In Table 1, the results obtained by three models on the Roxford dataset and the Rparis dataset are compared. These three models include a baseline effect model, a model that adds a Semantic-SAM framework on the basis of the baseline effect model, and the method adopted in the embodiment of the present application. It should be noted that the baseline effect model usually refers to a simple model or method that serves as a comparison benchmark in a certain task or field. It is usually a simple and easy-to-implement model that can be used to compare and evaluate the performance of other more complex models. As can be seen from Table 1, compared with the baseline effect model, the model in the related art that adds a Semantic-SAM framework on the basis of the baseline effect model has significantly improved the scores for the Roxford dataset and the Rparis dataset when extracting features from the target image. Under the same conditions, the method adopted in the embodiment of the present application has a more significant improvement in the scores for the Roxford dataset and the Rparis dataset. This shows that the method adopted in the embodiment of the present application can more accurately obtain the image features of the target image, thereby helping to improve the accuracy and efficiency of subsequent image retrieval tasks.

[0212] Reference Figure 14 As shown, Figure 14 This is a detailed flow chart of an image retrieval method provided by a specific example. The image retrieval method can be executed by a server, or by a user terminal, or by both a user terminal and a server. In this specific example, the method is executed by a server as an example. Figure 14 In the embodiment, the image retrieval method may include but is not limited to steps 1401 to 1417.

[0213] Step 1401: performing object detection on a target image to obtain a plurality of first sub-block images, performing main object detection on the target image to obtain a plurality of second sub-block images, and performing semantic segmentation on the target image to obtain a plurality of third sub-block images.

[0214] Step 1402: Obtain confidence levels of the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images.

[0215] Step 1403: Integrate the first sub-block image, the second sub-block image, and the third sub-block image whose confidences meet a predetermined condition into a plurality of target block images.

[0216] Step 1404: Perform object recognition on the target image to obtain a recognized object.

[0217] Step 1405: Determine key point positioning rules for the object based on the type of the identified object.

[0218] Step 1406: Based on the key point positioning rule, locate the key points on the object, thereby forming an image key point distribution of the target image.

[0219] Step 1407: according to the position information of each target block image in the target image, obtain the image key points in each target block image in the image key point distribution.

[0220] Step 1408: Obtain attention thermal information of the target block image for the image key points.

[0221] In this step, the attention thermal information is an attention thermal matrix, and each element in the attention thermal matrix corresponds one-to-one to each pixel point in the target block image. If a pixel point in the target block image is an image key point, the value of the element corresponding to the pixel point is set to the first element value; if a pixel point in the target block image is not an image key point, the value of the element corresponding to the pixel point is set to 0.

[0222] Step 1409: extract basic features and enhance local features of each target block image to obtain basic image features and local image enhancement features of each target block image.

[0223] In this step, the basic image feature may be a pixel value matrix of each pixel point in the target segmented image, and the local image enhancement feature may be a pixel enhancement value matrix of each pixel point in the target segmented image.

[0224] Step 1410: Multiply the elements of the attention heat matrix by the first ratio and add 1 to obtain a sensitivity adjustment coefficient matrix, and multiply the pixel value matrix by the sensitivity adjustment coefficient matrix to obtain a first matrix.

[0225] Step 1411: Multiply the pixel enhancement value matrix and the attention heat matrix to obtain a second matrix.

[0226] Step 1412: Obtain a first weight of the first matrix and a second weight of the second matrix.

[0227] Step 1413: Based on the first weight and the second weight, perform weighted sum on the first matrix and the second matrix to obtain a total matrix, and use the total matrix as the block image feature of the target block image.

[0228] Step 1414: For each target image, integrate the block image features of multiple target block images into the image features of the target image.

[0229] Step 1415: Check whether the image to be retrieved is received. If so, execute step 1416; if not, execute step 1415.

[0230] Step 1416: Extract features of the image to be retrieved to obtain features of the image to be retrieved.

[0231] Step 1417: Based on the matching of the features of the image to be retrieved and the image features of the multiple target images, a retrieval result image is determined in the multiple target images.

[0232] In this embodiment, through the image retrieval method of steps 1401 to 1417 above, multiple target images are first segmented to obtain multiple target block images of each target image; then, for each target image, the attention thermal information of each target block image is obtained, and image features of each target block image are extracted based on the attention thermal information of each target block image to obtain block image features of each target block image. Since the block image features of the target block images are obtained by extracting image features based on their attention thermal information, the block image features can express the target block images more accurately; then, for each target image, the block image features of the multiple target block images obtained are integrated. The image features of the target image are combined. Since the block image features of multiple target block images of the target image are integrated into the overall image features, these block image features can accurately express each block part of the target image. Therefore, when the image to be retrieved is received, the feature extraction of the image to be retrieved is first performed to obtain the feature of the image to be retrieved. Then, based on the matching of the feature of the image to be retrieved with the image features of multiple target images, the retrieval result image can be accurately determined among the multiple target images. There is no need to perform coarse sorting and then fine sorting as in the prior art. Therefore, the problems of unsatisfactory image retrieval accuracy and low image retrieval efficiency in related technologies can be improved, thereby effectively improving the accuracy and efficiency of image retrieval.

[0233] It will be appreciated that, although the various steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless clearly stated in the present embodiment, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0234] Reference Figure 15 The present application also discloses an image retrieval device 1500, which can implement the image retrieval method in the previous embodiment. The image retrieval device 1500 includes:

[0235] An image blocking unit 1510 is configured to block the plurality of target images to obtain a plurality of target block images of the respective target images;

[0236] The first extraction unit 1520 is configured to obtain attention thermal information of each target block image for each target image, and perform image feature extraction on each target block image based on the attention thermal information of each target block image to obtain a block image feature of each target block image;

[0237] The feature processing unit 1530 is used to integrate the block image features of the plurality of target block images into the image features of the target image for each target image;

[0238] The second extraction unit 1540 is used to receive the image to be retrieved, perform feature extraction on the image to be retrieved, and obtain features of the image to be retrieved;

[0239] The image retrieval unit 1550 is configured to determine a retrieval result image from the multiple target images based on matching the features of the image to be retrieved with the features of the multiple target images.

[0240] In one embodiment, the image blocking unit 1510 is further configured to:

[0241] Performing object detection on the multiple target images respectively to obtain multiple first sub-block images of each target image;

[0242] Performing main object detection on the multiple target images respectively to obtain multiple second sub-block images of each target image;

[0243] Performing semantic segmentation on the multiple target images respectively to obtain multiple third sub-block images of each target image;

[0244] For each target image, a plurality of target block images are determined based on a plurality of first sub-block images, a plurality of second sub-block images, and a plurality of third sub-block images.

[0245] In one embodiment, the image blocking unit 1510 is further configured to:

[0246] Obtaining confidence scores of the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images;

[0247] The first sub-block image, the second sub-block image and the third sub-block image whose confidences meet the predetermined condition are integrated into a plurality of target block images.

[0248] In one embodiment, the image blocking unit 1510 is further configured to:

[0249] Determining a first center position of each first sub-block image, a second center position of each second sub-block image, and a third center position of each third sub-block image;

[0250] If a first center position of a first sub-block image, a second center position of a second sub-block image, and a third center position of a third sub-block image meet a first condition, an overlapping area of ​​the first sub-block image, the second sub-block image, and the third sub-block image is determined as a target block image.

[0251] In one embodiment, the image blocking unit 1510 is further configured to:

[0252] Calling the first block model to perform image block division on the plurality of target images respectively to obtain a plurality of fourth sub-block images of each target image;

[0253] Calling the second block model to perform image segmentation on the plurality of target images respectively to obtain a plurality of fifth sub-block images of each target image, wherein the first block model and the second block model are pre-trained so that a first block granularity of the first block model is greater than a second block granularity of the second block model;

[0254] For each target image, a plurality of fourth sub-block images and a plurality of fifth sub-block images are integrated into a plurality of target block images.

[0255] In one embodiment, the image blocking unit 1510 is further configured to:

[0256] Performing image segmentation on the multiple target images respectively to obtain multiple candidate segmented images of each target image;

[0257] For each target image, a plurality of target block images having image areas greater than a preset area threshold are determined from the plurality of candidate block images.

[0258] In one embodiment, the image blocking unit 1510 is further configured to:

[0259] Use the text encoder to encode the text in the target image to obtain the image semantic label;

[0260] Segment the target image based on the image semantic label to obtain the segmentation frame;

[0261] Use the visual encoder to visually encode the frame lines in the target image to obtain the frame line mask;

[0262] If the segmentation frame line matches the frame line mask, the target image is segmented using the segmentation frame line to obtain multiple candidate segmented images.

[0263] In one embodiment, the first extraction unit 1520 is further configured to:

[0264] Obtain image key points in each target block image;

[0265] For the key points of the image, the attention thermal information of the target block image is obtained.

[0266] In one embodiment, the first extraction unit 1520 is further configured to:

[0267] Perform key point detection on the target image to obtain the image key point distribution of the target image;

[0268] According to the position information of each target block image in the target image, the image key points in each target block image are obtained from the image key point distribution.

[0269] In one embodiment, the first extraction unit 1520 is further configured to:

[0270] Perform object recognition on the target image to obtain the recognized object;

[0271] Determine the key point positioning rules for the object based on the type of the identified object;

[0272] Based on the key point positioning rules, key points are located on the object to form the image key point distribution of the target image.

[0273] In one embodiment, the attention heat information is an attention heat matrix, and each element in the attention heat matrix corresponds one-to-one to each pixel in the target block image; the first extraction unit 1520 is further configured to:

[0274] If a pixel in the target block image is an image key point, the value of the element corresponding to the pixel is set to the first element value;

[0275] If a pixel in the target block image is not an image key point, the value of the element corresponding to the pixel is set to 0.

[0276] In one embodiment, the first extraction unit 1520 is further configured to:

[0277] Extract basic features and enhance local features of each target block image to obtain basic image features and local image enhancement features of each target block image;

[0278] According to the attention thermal information, basic image features and local image enhancement features of each target block image, the block image features of each target block image are generated.

[0279] In one embodiment, the basic image feature is a pixel value matrix of each pixel in the target block image, the local image enhancement feature is a pixel enhancement value matrix of each pixel in the target block image, and the attention thermal information is an attention thermal matrix. Each element in the attention thermal matrix corresponds one-to-one to each pixel in the target block image, wherein if a pixel in the target block image is an image key point, the value of the element corresponding to the pixel is the first element value; if a pixel in the target block image is not an image key point, the value of the element corresponding to the pixel is 0; the first extraction unit 1520 is further used to:

[0280] Based on the attention heat matrix, the pixel value matrix is ​​sensitively adjusted to obtain the first matrix;

[0281] Based on the attention heat matrix, background noise is suppressed on the pixel enhancement value matrix to obtain the second matrix;

[0282] Based on the first matrix and the second matrix, a total matrix is ​​determined, and the total matrix is ​​used as a block image feature of the target block image.

[0283] In one embodiment, the first extraction unit 1520 is further configured to:

[0284] Multiplying the elements of the attention heat matrix by the first ratio and adding 1 to obtain a sensitivity adjustment coefficient matrix; multiplying the pixel value matrix by the sensitivity adjustment coefficient matrix to obtain a first matrix;

[0285] Multiply the pixel enhancement value matrix by the attention heat matrix to obtain the second matrix.

[0286] In one embodiment, the first extraction unit 1520 is further configured to:

[0287] Obtain a first weight of the first matrix and a second weight of the second matrix;

[0288] Based on the first weight and the second weight, a weighted sum is performed on the first matrix and the second matrix to obtain a total matrix.

[0289] In one embodiment, each element of the pixel enhancement value matrix is ​​a first value. The image retrieval apparatus 1500 further includes a numerical calculation unit configured to:

[0290] Perform enhancement processing on each pixel in the target block image to obtain the enhanced pixel value of the pixel;

[0291] Determine the average value of the pixel value and the enhanced pixel value of the pixel point;

[0292] A first value is determined based on an average value of each pixel in the target block image.

[0293] It should be noted that since the image retrieval device 1500 of this embodiment can implement the image retrieval method of the previous embodiment, the image retrieval device 1500 of this embodiment and the image retrieval method of the previous embodiment have the same technical principles and the same beneficial effects. In order to avoid repetition, they will not be repeated here.

[0294] Reference Figure 16 The embodiment of the present application further discloses an electronic device, the electronic device 1600 including:

[0295] at least one processor 1601;

[0296] At least one memory 1602, configured to store at least one program;

[0297] When the at least one program is executed by the at least one processor 1601 , the above-mentioned image retrieval method is implemented.

[0298] An embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program executable by a processor. When the computer program executable by the processor is executed by the processor, it is used to implement the above-mentioned image retrieval method.

[0299] An embodiment of the present application also discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the electronic device performs the image retrieval method as described above.

[0300] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0301] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0302] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0303] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0304] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0305] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0306] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0307] The step numbers in the above method embodiment are only provided for the convenience of explanation and description, and do not limit the order of the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. An image retrieval method, characterized in that: The following steps are involved: Performing image segmentation on a plurality of target images respectively to obtain a plurality of target segmented images of each of the target images; For each of the target images, acquiring attention thermal information of each of the target block images, and performing image feature extraction on each of the target block images based on the attention thermal information of each of the target block images to obtain block image features of each of the target block images; For each of the target images, integrating the block image features of a plurality of the target block images into an image feature of the target image; Receiving an image to be retrieved, performing feature extraction on the image to be retrieved, and obtaining features of the image to be retrieved; Based on the matching of the to-be-retrieved image features with the image features of the plurality of target images, a retrieval result image is determined from the plurality of target images.

2. The method according to claim 1, characterized in that The step of performing image segmentation on the plurality of target images to obtain a plurality of target segmented images of the respective target images includes: Performing object detection on each of the plurality of target images to obtain a plurality of first sub-block images of each of the target images; Performing subject object detection on each of the plurality of target images to obtain a plurality of second sub-block images of each of the target images; Performing semantic segmentation on the plurality of target images respectively to obtain a plurality of third sub-block images of each of the target images; For each of the target images, a plurality of the target block images are determined based on a plurality of the first sub-block images, a plurality of the second sub-block images, and a plurality of the third sub-block images.

3. The method according to claim 2, characterized in that The determining of the plurality of target block images based on the plurality of the first sub-block images, the plurality of the second sub-block images, and the plurality of the third sub-block images comprises: Acquire confidence scores of the plurality of first sub-block images, the plurality of second sub-block images, and the plurality of third sub-block images; The first sub-block images, the second sub-block images, and the third sub-block images whose confidences meet a predetermined condition are integrated into a plurality of target block images.

4. The method according to claim 1, wherein The step of performing image segmentation on the plurality of target images to obtain a plurality of target segmented images of the respective target images includes: Calling the first block model to perform image block division on the plurality of target images respectively to obtain a plurality of fourth sub-block images of each of the target images; calling a second block model to perform image segmentation on each of the plurality of target images to obtain a plurality of fifth sub-block images of each of the target images, wherein the first block model and the second block model are pre-trained so that a first block granularity of the first block model is greater than a second block granularity of the second block model; For each of the target images, a plurality of the fourth sub-block images and a plurality of the fifth sub-block images are integrated into a plurality of the target block images.

5. The method according to claim 1, wherein The step of performing image segmentation on the plurality of target images to obtain a plurality of target segmented images of the respective target images includes: Performing image segmentation on the plurality of target images respectively to obtain a plurality of candidate segmented images of the respective target images; For each of the target images, a plurality of the target block images having image areas greater than a preset area threshold are determined from the plurality of candidate block images.

6. The method according to claim 5, characterized in that The process of performing image segmentation on the target image to obtain a plurality of candidate segmented images comprises the following steps: Using a text encoder to encode the text in the target image to obtain an image semantic label; Segmenting the target image based on the image semantic label to obtain segmentation frame lines; Using a visual encoder to visually encode the frame lines in the target image to obtain a frame line mask; If the segmentation frame line matches the frame line mask, the target image is segmented using the segmentation frame line to obtain a plurality of candidate segmented images.

7. The method according to claim 1, characterized in that The acquiring of the attention thermal information of each target block image includes: Obtaining image key points in each target block image; For the image key points, the attention thermal information of the target block image is obtained.

8. The method according to claim 7, characterized in that The obtaining of the image key points in each target block image includes: Performing key point detection on the target image to obtain image key point distribution of the target image; The image key points in each of the target block images are obtained in the image key point distribution according to the position information of each of the target block images in the target image.

9. The method according to claim 8, characterized in that The performing key point detection on the target image to obtain the image key point distribution of the target image includes: Performing object recognition on the target image to obtain a recognized object; Determining key point positioning rules for the object based on the identified type of the object; Based on the key point positioning rule, key points are positioned on the object, thereby forming an image key point distribution of the target image.

10. The method according to claim 7, characterized in that The attention thermal information is an attention thermal matrix, and each element in the attention thermal matrix corresponds one-to-one to each pixel in the target block image; The step of obtaining the attention thermal information of the target block image for the image key points includes: If one of the pixel points in the target block image is the image key point, setting the value of the element corresponding to the pixel point to the first element value; If one of the pixel points in the target block image is not the image key point, the value of the element corresponding to the pixel point is set to 0.

11. The method according to claim 1, wherein The step of extracting image features of each target block image based on the attention heat information of each target block image to obtain block image features of each target block image includes: Extracting basic features and enhancing local features of each target block image to obtain basic image features and local image enhancement features of each target block image; The block image features of each target block image are generated according to the attention thermal information, the basic image features and the local image enhancement features of each target block image.

12. The method according to claim 11, characterized in that The basic image feature is a pixel value matrix of each pixel in the target block image, the local image enhancement feature is a pixel enhancement value matrix of each pixel in the target block image, and the attention thermal information is an attention thermal matrix. Each element in the attention thermal matrix corresponds one-to-one to each pixel in the target block image, wherein if one of the pixel points in the target block image is the image key point, the value of the element corresponding to the pixel point is the first element value; if one of the pixel points in the target block image is not the image key point, the value of the element corresponding to the pixel point is 0; Generating the block image features of each target block image according to the attention heat information, the basic image features, and the local image enhancement features of each target block image includes: Based on the attention heat matrix, sensitively adjust the pixel value matrix to obtain a first matrix; Based on the attention heat matrix, background noise suppression is performed on the pixel enhancement value matrix to obtain a second matrix; Based on the first matrix and the second matrix, a total matrix is ​​determined, and the total matrix is ​​used as the block image feature of the target block image.

13. The method according to claim 12, characterized in that The step of performing sensitivity adjustment on the pixel value matrix based on the attention thermal matrix to obtain a first matrix includes: multiplying elements of the attention thermal matrix by a first ratio and adding 1 to obtain a sensitivity adjustment coefficient matrix; and multiplying the pixel value matrix by the sensitivity adjustment coefficient matrix to obtain the first matrix; The method of suppressing background noise on the pixel enhancement value matrix based on the attention thermal matrix to obtain a second matrix includes: multiplying the pixel enhancement value matrix by the attention thermal matrix to obtain the second matrix.

14. The method according to claim 12, characterized in that The determining of a total matrix based on the first matrix and the second matrix includes: Obtaining a first weight of the first matrix and a second weight of the second matrix; Based on the first weight and the second weight, a weighted sum is performed on the first matrix and the second matrix to obtain the total matrix.

15. An image retrieval device, characterized in that: include: An image blocking unit is used to block the multiple target images respectively to obtain multiple target block images of each target image; a first extraction unit, configured to obtain, for each target image, attention thermal information of each target block image, and perform image feature extraction on each target block image based on the attention thermal information of each target block image to obtain a block image feature of each target block image; a feature processing unit, configured to integrate the block image features of a plurality of the target block images into an image feature of the target image for each of the target images; A second extraction unit is used to receive the image to be retrieved, perform feature extraction on the image to be retrieved, and obtain features of the image to be retrieved; The image retrieval unit is configured to determine a retrieval result image from the plurality of target images based on matching the features of the image to be retrieved with the image features of the plurality of target images.

16. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the image retrieval method according to any one of claims 1 to 14 is implemented.

17. A computer-readable storage medium, characterized in that A computer program executable by a processor is stored therein, and when the computer program executable by the processor is executed by the processor, it is used to implement the image retrieval method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program or computer instructions, characterized in that The computer program or the computer instructions are stored in a computer-readable storage medium, the processor of the electronic device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the electronic device executes the image retrieval method as described in any one of claims 1 to 14.

Images