Image retrieval method and apparatus, electronic device, computer readable storage medium and computer program product

By combining the similarity of candidate images and reference images-modified text combinations and the similarity of candidate images and modified texts, the target image is determined, which solves the problem that image retrieval models ignore intention text information in the prior art, and improves the accuracy of image retrieval degree and performance.

WO2024234807A1PCT designated stage expired Publication Date: 2024-11-21TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
PCT/CN2024/081383
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2024-03-13
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

When the existing image retrieval model integrates reference images and intend text features, it is susceptible to the influence of reference images and ignores intention text information, resulting in inaccurate search results and poor accuracy.

Method used

By acquiring the combination of reference images and modified text, combining the similarity between candidate images and reference images-modified text combinations and the similarity between candidate images and modified text, determine the target image from multiple candidate images and strengthen the modified text and improve search accuracy.

Benefits of technology

It effectively improves the accuracy of image retrieval, enables the target image to highly match the expectations of modifying text, enhances the model's sensitivity to modifying text content, and improves the retrieval performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024081383_21112024_PF_FP_ABST
    Figure CN2024081383_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application provides an image retrieval method and apparatus, an electronic device, a computer readable storage medium and a computer program product. The method comprises: acquiring an image retrieval condition, the image retrieval condition comprising a reference image and a modification text, and the modification text being used for indicating the modification expectation for the reference image; combining the reference image and the modification text to obtain a reference image-modification text combination; acquiring a plurality of candidate images, and for each candidate image, determining a first similarity between the candidate image and the reference image-modification text combination, and a second similarity between the candidate image and the modification text; and according to the first similarity and the second similarity, determining at least one target image meeting the image retrieval condition among the plurality of candidate images.
Need to check novelty before this filing date? Find Prior Art

Description

Image retrieval method, device, electronic device, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202310538798.4 and application date May 12, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to an image retrieval method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0004] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0005] In related technologies, the corresponding target image is usually retrieved by matching the modified text with the labels of each candidate image. In this way, when the semantics of the modified text are complex, the retrieved target image is difficult to meet the modification expectations, resulting in poor accuracy of image retrieval.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an image retrieval method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of image retrieval.

[0008] The technical solution of the embodiment of the present application is implemented as follows:

[0009] The present invention provides an image retrieval method, including:

[0010] Acquiring an image retrieval condition, wherein the image retrieval condition includes a reference image and a modification text, wherein the modification text is used to indicate a modification expectation for the reference image;

[0011] combining the reference image and the modified text to obtain a reference image-modified text combination;

[0012] Acquire multiple candidate images, and determine a first similarity between each candidate image and the reference image-modified text combination, and a second similarity between each candidate image and the modified text;

[0013] At least one target image matching the image retrieval condition is determined from the plurality of candidate images by combining the first similarity and the second similarity.

[0014] The present invention provides an image retrieval device, comprising:

[0015] An acquisition module configured to acquire an image retrieval condition, wherein the image retrieval condition includes a reference image and a modification text, wherein the modification text is used to indicate a modification expectation for the reference image;

[0016] a combining module configured to combine the reference image and the modified text to obtain a reference image-modified text combination;

[0017] a similarity module configured to obtain a plurality of candidate images and determine a first similarity between each candidate image and the reference image-modified text combination, and a second similarity between each candidate image and the modified text;

[0018] The retrieval module is configured to determine at least one target image that matches the image retrieval condition from the multiple candidate images based on the first similarity and the second similarity.

[0019] An embodiment of the present application provides an electronic device, including:

[0020] a memory configured to store computer-executable instructions or computer programs;

[0021] The processor is configured to implement the image retrieval method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0022] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions configured to cause a processor to execute the instructions to implement the image retrieval method provided in the embodiment of the present application.

[0023] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image retrieval method described above in the present invention.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] By obtaining an image retrieval condition including a reference image and modified text, as well as multiple candidate images, and determining a first similarity between each candidate image and the reference image-modified text combination, and a second similarity between the candidate image and the modified text, a target image that matches the image retrieval condition is determined from the multiple candidate images by combining the first similarity and the second similarity. In this way, since the retrieved target image that meets the image retrieval condition is determined based on the first similarity between the candidate image and the reference image-modified text combination, it is possible to effectively ensure that the determined target image can simultaneously meet the retrieval requirements of the reference image and the modified text. At the same time, the second similarity between the candidate image and the modified text participates in the determination of the target image, which can strengthen the influence of the modification expectation of the modified text on the determined target image, so that the target image can be highly consistent with the modification expectation of the modified text, thereby effectively improving the accuracy of image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic diagram of the architecture of an image retrieval system provided in an embodiment of the present application;

[0027] FIG2 is a schematic diagram of the structure of an electronic device for image retrieval provided in an embodiment of the present application;

[0028] FIG3 is a flowchart of an image retrieval method according to an embodiment of the present application;

[0029] FIG4 is a second flow chart of the image retrieval method provided in an embodiment of the present application;

[0030] FIG5 is a third flow chart of the image retrieval method provided in an embodiment of the present application;

[0031] FIG6 is a fourth flow chart of the image retrieval method provided in an embodiment of the present application;

[0032] FIG7 is a schematic diagram showing the principle of image retrieval conditions provided in an embodiment of the present application;

[0033] FIG8 is a schematic diagram of the principle of a first prediction network provided in an embodiment of the present application;

[0034] FIG9 is a schematic diagram of the principle of a second prediction network provided in an embodiment of the present application;

[0035] FIG10 is a schematic diagram showing the principle of the image retrieval method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0037] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) Artificial Intelligence (AI): The theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics.

[0042] 2) Convolutional Neural Networks (CNN): These are a type of feed-forward neural network (FNN) that incorporates convolutional computations and possesses a deep structure. They are a representative algorithm for deep learning. CNNs possess representation learning capabilities and can perform shift-invariant classification of input images based on their hierarchical structure.

[0043] 3) Convolutional Layer: Each convolutional layer in a convolutional neural network consists of several convolutional units, whose parameters are optimized through a backpropagation algorithm. The purpose of the convolution operation is to extract different features from the input. The first convolutional layer may only extract low-level features such as edges, lines, and corners. A network with more layers can iteratively extract more complex features from these low-level features.

[0044] 4) Pooling Layer: After feature extraction in the convolutional layer, the output feature map is passed to the pooling layer for feature selection and information filtering. The pooling layer contains a pre-defined pooling function, which replaces the result of a single point in the feature map with the feature map statistics of its adjacent area. The pooling layer selects the pooling area in the same way as the convolution kernel scans the feature map, and is controlled by the pooling size, stride, and padding.

[0045] 5) Fully-Connected Layer: A fully-connected layer in a convolutional neural network is equivalent to a hidden layer in a traditional feedforward neural network. It is located at the end of the hidden layer of a convolutional neural network and only transmits signals to other fully-connected layers. Feature maps lose their spatial topology in fully-connected layers, becoming flattened into vectors and passing through activation functions.

[0046] 6) Game Program: A game program may be any of the following: a massively multiplayer online role-playing game (MMORPG), a first-person shooter (FPS), a third-person shooter, a multiplayer online battle arena (MOBA), a virtual reality application, a three-dimensional mapping program, a simulation program, or a multiplayer shooter survival game.

[0047] During the implementation of the embodiments of this application, the applicant discovered that the related technology has the following problems:

[0048] In recent years, with the rapid development of the internet, multimedia data in various forms, such as text, images, and audio, has exploded. With the emergence of a vast array of identical or similar content, multimedia retrieval has become a fundamental task for users to flexibly access information. Therefore, to meet users' increasingly complex retrieval needs, language-visual combined query-based image retrieval (LVCQ-IR) has been proposed and has received increasing attention. As one of the most popular multimedia retrieval models, LVCQ-IR uses a reference image and a textual description of the desired modification to the image as query input, then retrieves the corresponding target image through a database query. Because the input text expresses the intended modification of the reference image, existing retrieval models for LVCQ-IR primarily focus on designing a synthetic network. Specifically, they fuse the feature representation of the reference image with the feature representation of the intended text into a combined query representation. Through model training, the combined query representation is trained to be as close as possible to the feature representation of the target image.

[0049] However, during the model design process, the fused combined query representation is significantly influenced by the reference image, thus ignoring some or even all of the information in the intended context. For example, if a non-target image A is very similar to the reference image B used during training, but A does not match the description in the intended context, while the true target image C does match the description in the intended context, then during model training, the following situation is likely to occur: the fused combined query representation becomes increasingly closer to the feature representation of A than to the representation of the true target image C, leading to incorrect retrieval results and poor image retrieval accuracy.

[0050] To verify the effectiveness of the embodiments of the present application in actual scenarios, the embodiments of the present application were experimented on a public dataset. The public dataset was also used for interactive retrieval based on dialogue. 10,000 sets of triples were selected for training and 4,568 sets of triples were selected for testing. The embodiments of the present application used the following evaluation indicators to verify the performance of hash retrieval:

[0051] Accuracy (R@K): The ratio of correct queries returned in the top K results, where K is set to 10 and 50 in the Fashion-IQ dataset and 1, 10, and 50 in the Shoed dataset.

[0052] Performance parameter (Rmeans): The mean of all R@K values, used to evaluate the overall retrieval performance.

[0053] In the algorithmic framework, the embodiments of the present application use the CLIP pre-trained model to initialize the model, and use PyTorch to learn and optimize the model based on the Adam optimizer. The model is compared with the current best method CRR to verify the effectiveness of the invention. The results are shown in Table 1 below. As can be seen from Table 1, compared with the relevant technology, the embodiments of the present application have achieved at least a 3% performance improvement, achieving the best retrieval performance, which also proves the superiority of the embodiments of the present application on the LVCQ-IR task.

[0054] Table 1 Performance comparison between this application and related technologies

[0055] The embodiments of the present application provide an image retrieval method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the accuracy of image retrieval. The following describes an exemplary application of the image retrieval system provided by the embodiments of the present application.

[0056] See Figure 1, which is a schematic diagram of the architecture of the image retrieval system 100 provided in an embodiment of the present application. The terminal (terminal 400 is shown as an example) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] The terminal 400 is configured to allow a user to use a client 410 to display a target image on a graphical interface 410 - 1 (graphic interface 410 - 1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0058] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, a car terminal, etc., but is not limited to this. The electronic device provided in the embodiment of the present application can be implemented as a terminal or as a server. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application.

[0059] In some embodiments, the server 200 obtains an image retrieval condition, obtains a plurality of candidate images, determines a target image that matches the image retrieval condition, and sends the target image to the terminal 400 .

[0060] In other embodiments, the server 200 obtains image retrieval conditions and multiple candidate images, and determines a first similarity between each candidate image and the reference image-modified text combination, and a second similarity between each candidate image and the candidate text, and sends the first similarity and the second similarity to the terminal 400. The terminal 400 combines the first similarity and the second similarity to determine a target image that matches the image retrieval conditions from the multiple candidate images.

[0061] In other embodiments, the embodiments of the present application can be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize data calculation, storage, processing, and sharing.

[0062] Cloud technology is a general term for network, information, integration, management platform, and application technologies used in the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a key support. The backend services of technical network systems require a large amount of computing and storage resources.

[0063] Referring to Figure 2, Figure 2 is a schematic diagram of the structure of an electronic device 500 for image retrieval provided in an embodiment of the present application, wherein the electronic device 500 shown in Figure 2 can be the server 200 or the terminal 400 in Figure 1, and the electronic device 500 shown in Figure 2 includes: at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together through a bus system 440. It can be understood that the bus system 440 is configured to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 440 in Figure 2.

[0064] The processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 430.

[0066] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0067] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0068] Operating system 451, including system programs configured to handle various basic system services and perform hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., configured to implement various basic services and handle hardware-based tasks;

[0069] The network communication module 452 is configured to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0070] In some embodiments, the image retrieval device provided in the embodiments of the present application can be implemented in software. FIG2 shows an image retrieval device 455 stored in memory 450. The image retrieval device 455 can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 4551, a combination module 4552, a similarity module 4553, and a retrieval module 4554. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0071] In other embodiments, the image retrieval device provided in the embodiments of the present application can be implemented in hardware. As an example, the image retrieval device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image retrieval method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] In some embodiments, the terminal or server can implement the image retrieval method provided in the embodiments of the present application by running a computer program or computer executable instructions. For example, the computer program can be a native program (for example, a dedicated image retrieval program) or a software module in the operating system, for example, an image retrieval module that can be embedded in any program (such as an instant messaging client, a photo album program, an electronic map client, a navigation client); for example, it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0073] The image retrieval method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0074] Refer to Figure 3, which is a flow chart of the image retrieval method provided in an embodiment of the present application. It will be explained in conjunction with steps 101 to 105 shown in Figure 3. The image retrieval method provided in an embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will be explained using the server alone as an example.

[0075] In step 101, image retrieval conditions are obtained.

[0076] In some embodiments, the image retrieval condition includes a reference image and a modification text, where the modification text is used to indicate a modification expectation for the reference image.

[0077] In some embodiments, the image retrieval condition is used to retrieve a target image that meets the modification expectation of the reference image.

[0078] As an example, see FIG. 7 , which is a schematic diagram illustrating the principles of image retrieval conditions provided in an embodiment of the present application. Image retrieval condition 1 includes a reference image 11 and modification text 12. Modification text 12, "I want to open her up to her toes with double straps and trim," indicates a desired modification of reference image 11. Image retrieval condition 1 is used to retrieve target images that satisfy the desired modification of reference image 11.

[0079] As an example, see FIG. 7 , which is a schematic diagram illustrating the principles of image retrieval conditions provided in an embodiment of the present application. Image retrieval condition 2 includes a reference image 21 and modification text 22. Modification text 22, which reads, "I want to open her up below her toes with double straps and trim," indicates a desired modification of reference image 21. Image retrieval condition 2 is used to retrieve target images that satisfy the desired modification of reference image 21.

[0080] In some embodiments, referring to FIG. 4 , FIG. 4 is a second flow chart of the image retrieval method provided in an embodiment of the present application. Step 101 shown in FIG. 3 can be implemented by executing steps 1011 to 1012 shown in FIG. 4 .

[0081] In step 1011 , an image retrieval request is received.

[0082] In some embodiments, the terminal generates an image retrieval request based on the image retrieval instruction, and sends the image retrieval request to the server, and the server receives the image retrieval request.

[0083] In some embodiments, the image retrieval request carries an image retrieval condition. The image retrieval request is generated by the terminal based on an image retrieval instruction. The image retrieval instruction is used to instruct image retrieval based on an input reference image and modified text.

[0084] In step 1012, the image retrieval request is parsed to obtain image retrieval conditions.

[0085] In some embodiments, since the image retrieval request carries an image retrieval condition, the image retrieval condition carried in the image retrieval request can be obtained by parsing the image retrieval request.

[0086] In step 102, the reference image and the modified text are combined to obtain a reference image-modified text combination.

[0087] As an example, the above combination refers to a process of combining the objects to be combined (reference image and modified text).

[0088] In some embodiments, the above step 102 may be implemented in the following manner: determining the union of the reference image and the modified text as a reference image-modified text combination.

[0089] As an example, referring to FIG7 , the reference image 11 and the modified text 12 are combined to obtain a reference image-modified text combination; the reference image 21 and the modified text 22 are combined to obtain a reference image-modified text combination.

[0090] In step 103, a plurality of candidate images are obtained.

[0091] In some embodiments, the candidate images are images pre-stored in a database, and the candidate images include target images that meet the modification expectations for the reference image indicated by the modification text.

[0092] In step 104 , for each candidate image, a first similarity between the candidate image and the reference image-modified text combination, and a second similarity between the candidate image and the modified text are determined.

[0093] In some embodiments, the first similarity between the candidate image and the reference image-modified text combination is used to indicate the overall and local similarity between the candidate image and the reference image-modified text combination. The second similarity between the candidate image and the modified text is used to indicate the overall and local similarity between the candidate image and the modified text.

[0094] In some embodiments, the numerical values ​​of the first similarity and the second similarity are positively correlated with the corresponding similarity levels.

[0095] In this way, by determining the first similarity between each candidate image and the reference image-modified text combination, and the second similarity between each candidate image and the modified text, since the first similarity and the second similarity can accurately reflect the overall and local similarity between the candidate image and the reference image-modified text combination, and the overall and local similarity between the candidate image and the modified text, the similarity is comprehensively measured from multiple perspectives of overall and local, thereby effectively improving the accuracy of the determined first similarity and second similarity.

[0096] In some embodiments, referring to FIG. 5 , FIG. 5 is a flowchart diagram three of the image retrieval method provided in an embodiment of the present application. Step 104 shown in FIG. 3 can be implemented by executing steps 1041 to 1045 shown in FIG. 5 .

[0097] In step 1041, an image retrieval model is obtained, where the image retrieval model includes a first prediction network and a second prediction network.

[0098] In some embodiments, the image retrieval model may be a Linguistic-Visual Composed Query Based Image Retrieval (LVCQ-IR) model used to retrieve a target image that meets a modification expectation for a reference image.

[0099] In some embodiments, the first prediction network is used to predict the similarity between the candidate image and the reference image-modified text combination, and the second prediction network is used to predict the similarity between the candidate image and the modified text.

[0100] In some embodiments, the above-mentioned step 1041 can be implemented as follows: obtaining image retrieval condition samples and target image samples, the image retrieval condition samples include reference image samples and modified text samples, the modified text samples are used to indicate the modification expectations for the reference image samples, and the target image samples meet the modification expectations; combining the reference image samples and the modified text samples to obtain reference image-modified text combination samples; obtaining multiple candidate images, the candidate images include target image samples; for each candidate image, calling the initial image retrieval model, based on the candidate image, the reference image sample and the modified text sample, performing similarity prediction on the candidate image, obtaining a third similarity between the candidate image and the reference image-modified text combination sample, and a fourth similarity between the candidate image and the modified text sample; combining the third similarity and the fourth similarity to train the initial image retrieval model to obtain an image retrieval model.

[0101] In some embodiments, the above-mentioned initial image retrieval model includes a first initial prediction network and a second initial prediction network, the above-mentioned first initial prediction network has the same network structure as the first prediction network of the image retrieval model, and the above-mentioned second initial prediction network has the same network structure as the second prediction network of the image retrieval model.

[0102] In some embodiments, the above-mentioned calling of the initial image retrieval model, based on the candidate image, the reference image sample and the modified text sample, performs similarity prediction on the candidate image, obtains a third similarity between the candidate image and the reference image-modified text combination sample, and a fourth similarity between the candidate image and the modified text sample, can be achieved as follows: performing feature extraction on the reference image sample and the modified text sample respectively to obtain reference image features of the reference image sample and modified text features of the modified text sample, and combining the reference image features and the modified text features to determine the combined sample features of the reference image-modified text combination sample; performing feature extraction on the candidate image to obtain candidate image features of the candidate image; calling the first initial prediction network to predict the similarity between the candidate image and the reference image-modified text combination sample based on the candidate image features and the combined sample features to obtain the third similarity between the candidate image and the reference image-modified text combination sample; calling the second initial prediction network to predict the similarity between the candidate image and the modified text sample based on the candidate image features and the modified text features to obtain the fourth similarity between the candidate image and the modified text sample.

[0103] In some embodiments, the above-mentioned third similarity includes a third global similarity and a third local similarity, the above-mentioned first initial prediction network includes a first initial prediction layer and a second initial prediction layer, and the above-mentioned calling of the first initial prediction network to predict the similarity between the candidate image and the reference image-modified text combination sample based on the candidate image features and the combined sample features to obtain the third similarity between the candidate image and the reference image-modified text combination sample can be achieved as follows: calling the first initial prediction layer to predict the overall similarity between the candidate image and the reference image-modified text combination sample based on the candidate image features and the combined sample features to obtain the third global similarity corresponding to the candidate image; performing feature segmentation on the combined sample features to obtain multiple local sample features corresponding to the combined sample features, and performing feature segmentation on the modified text features of the modified text sample to obtain word features corresponding to each word in the modified text sample; calling the second initial prediction layer to predict the local similarity between the candidate image and the reference image-modified text combination sample based on the local sample features and the word features to obtain the third local similarity corresponding to the candidate image.

[0104] In some embodiments, the above-mentioned combination of reference image features and modified text features to determine the combined sample features of the reference image-modified text combination sample can be achieved as follows: adding the reference image features and the modified text features to obtain the sum feature; dividing the sum feature and the norm of the sum feature to obtain the combined sample features of the reference image-modified text combination sample.

[0105] In some embodiments, the above-mentioned calling of the first initial prediction layer, predicting the overall similarity between the candidate image and the reference image-modified text combination sample based on the candidate image features and the combined sample features, and obtaining the third global similarity corresponding to the candidate image can be achieved as follows: multiplying the transpose of the combined sample features and the candidate image features to obtain the sixth reference feature; calling the first initial prediction layer, predicting the overall similarity between the candidate image and the reference image-modified text combination sample based on the sixth reference feature, and obtaining the third global similarity corresponding to the candidate image.

[0106] In some embodiments, the above-mentioned calling of the second initial prediction layer, based on local sample features and word features, predicts the local similarity between the candidate image and the reference image-modified text combination sample to obtain the third local similarity corresponding to the candidate image, can be achieved as follows: multiplying the transpose of each word feature with the candidate image feature respectively to obtain the reference word feature corresponding to each word feature, and summing up the reference word features to obtain the seventh reference feature; multiplying the transpose of the local sample feature with the candidate image feature respectively to obtain the reference local sample feature corresponding to each local sample feature; and summing up the reference local sample features to obtain the eighth reference feature; calling the second initial prediction layer, based on the eighth reference feature, predicting the local similarity between the candidate image and the reference image-modified text combination sample to obtain the third local similarity corresponding to the candidate image.

[0107] In some embodiments, the fourth similarity includes a fourth global similarity and a fourth local similarity, and the second initial prediction network includes a third initial prediction layer and a fourth initial prediction layer.

[0108] In some embodiments, the above-mentioned fourth global similarity is used to indicate the overall similarity between the candidate image and the modified text sample, the fourth local similarity is used to indicate the local similarity between the candidate image and the modified text sample, the third initial prediction layer is used to predict the overall similarity between the candidate image and the modified text sample, and the fourth initial prediction layer is used to predict the local similarity between the candidate image and the modified text sample.

[0109] In some embodiments, the above-mentioned calling of the second initial prediction layer, based on local sample features and word features, predicts the local similarity between the candidate image and the reference image-modified text combination sample to obtain the third local similarity corresponding to the candidate image, can be achieved as follows: multiplying the transpose of each word feature with the candidate image feature respectively to obtain the reference word feature corresponding to each word feature, and summing up the reference word features to obtain the ninth reference feature; calling the second initial prediction layer, based on the ninth reference feature, predicts the local similarity between the candidate image and the reference image-modified text combination sample to obtain the third local similarity corresponding to the candidate image.

[0110] In some embodiments, since the candidate image includes the target image sample, the third similarity of the candidate image and the fourth similarity of the candidate image include the third similarity and fourth similarity of the target image sample. Since the target image sample is the candidate image with the highest similarity among the candidate images, the third similarity and fourth similarity of the target image sample can be used as training labels to train the initial image retrieval model, so that the obtained image retrieval model can effectively learn the feature representation of the target image sample and can accurately identify the corresponding target image from a large number of candidate images, thereby effectively improving the retrieval performance of the image retrieval model.

[0111] In some embodiments, the above-mentioned combination of the third similarity and the fourth similarity to train the initial image retrieval model to obtain the image retrieval model can be achieved as follows: based on the third similarity corresponding to the target image sample and the third similarity corresponding to each candidate image, determine the first loss value of the initial image retrieval model; based on the fourth similarity corresponding to the target image sample and the fourth similarity corresponding to each candidate image, determine the second loss value of the initial image retrieval model; perform weighted summation of the first loss value and the second loss value to obtain the target loss value, and based on the target loss value, train the initial image retrieval model to obtain the image retrieval model.

[0112] As an example, the expression of the above target loss value can be: L=L1+αL2 (1)

[0113] Among them, L is used to indicate the target loss value, L1 is used to indicate the first loss value, L2 is used to indicate the second loss value, and α is used to indicate the weight of the weighted summation.

[0114] In some embodiments, the above-mentioned third similarity includes a third global similarity and a third local similarity, wherein the third global similarity is used to indicate the overall similarity between the candidate image and the reference image-modified text combination sample, and the third local similarity is used to indicate the local similarity between the candidate image and the reference image-modified text combination sample.

[0115] In some embodiments, the above-mentioned determination of the first loss value of the initial image retrieval model based on the third similarity corresponding to the target image sample and the third similarity corresponding to each candidate image can be achieved in the following manner: summing the third global similarities corresponding to each candidate image to obtain a first summation result, and summing the third local similarities corresponding to each candidate image to obtain a second summation result; dividing the third global similarity corresponding to the target image sample by the first summation result to obtain a third loss value; dividing the third local similarity corresponding to the target image sample by the second summation result to obtain a fourth loss value; summing the third loss value and the fourth loss value to obtain a first loss value.

[0116] As an example, the expression of the third loss value may be:

[0117] Wherein, L3 is used to indicate the third loss value, Used to indicate the third global similarity corresponding to the target image sample, It is used to indicate the first summation result, and N is used to indicate the number of target image samples.

[0118] As an example, the expression of the fourth loss value may be:

[0119] Wherein, L4 is used to indicate the fourth loss value, Used to indicate the third global similarity corresponding to the target image sample, is used to indicate the second summation result, and N is used to indicate the number of target image samples.

[0120] As an example, the expression of the first loss value can be:

[0121] Among them, L1 is used to indicate the first loss value, L3 is used to indicate the third loss value, Used to indicate the third global similarity corresponding to the target image sample, is used to indicate the first summation result, L4 is used to indicate the fourth loss value, Used to indicate the third global similarity corresponding to the target image sample, is used to indicate the second summation result, and N is used to indicate the number of target image samples.

[0122] In some embodiments, the fourth similarity includes a fourth global similarity and a fourth local similarity, wherein the fourth global similarity is used to indicate the overall similarity between the candidate image and the modified text sample, and the fourth local similarity is used to indicate the local similarity between the candidate image and the modified text sample.

[0123] In some embodiments, the above-mentioned determination of the second loss value of the initial image retrieval model based on the fourth similarity corresponding to the target image sample and the fourth similarity corresponding to each candidate image can be achieved in the following manner: summing the fourth global similarities corresponding to each candidate image to obtain a third summation result, and summing the fourth local similarities corresponding to each candidate image to obtain a fourth summation result; dividing the fourth global similarity corresponding to the target image sample and the third summation result to obtain a fifth loss value; dividing the fourth local similarity corresponding to the target image sample and the fourth summation result to obtain a sixth loss value; summing the fifth loss value and the sixth loss value to obtain a second loss value.

[0124] As an example, the expression of the fifth loss value may be:

[0125] Wherein, L5 is used to indicate the fifth loss value, Used to indicate the fourth global similarity corresponding to the target image sample, is used to indicate the third summation result, and N is used to indicate the number of target image samples.

[0126] As an example, the expression of the sixth loss value may be:

[0127] Wherein, L6 is used to indicate the sixth loss value, Used to indicate the fourth local similarity corresponding to the target image sample, is used to indicate the fourth summation result, and N is used to indicate the number of target image samples.

[0128] As an example, the expression of the second loss value can be:

[0129] Among them, L2 is used to indicate the second loss value, L5 is used to indicate the fifth loss value, and L6 is used to indicate the sixth loss value. Used to indicate the fourth global similarity corresponding to the target image sample, is used to indicate the third summation result, N is used to indicate the number of target image samples, Used to indicate the fourth local similarity corresponding to the target image sample, Used to indicate the fourth summation result.

[0130] In this way, the first loss value of the initial image retrieval model is determined based on the third similarity corresponding to the target image sample and the third similarity corresponding to each candidate image; the second loss value of the initial image retrieval model is determined based on the fourth similarity corresponding to the target image sample and the fourth similarity corresponding to each candidate image; the first loss value and the second loss value are weightedly summed to obtain the target loss value, and the initial image retrieval model is trained based on the target loss value to obtain the image retrieval model. Since the trained image retrieval model makes full use of the third similarity corresponding to the target image sample and the third similarity corresponding to each candidate image, the retrieval performance of the image retrieval model is effectively improved.

[0131] In step 1042, feature extraction is performed on the reference image and the modified text respectively to obtain reference image features of the reference image and modified text features of the modified text, and the reference image features and the modified text features are combined to determine the combined features of the reference image-modified text combination.

[0132] In some embodiments, the above-mentioned feature extraction of the reference image and the modified text to obtain the reference image features of the reference image and the modified text features of the modified text can be achieved by calling the image coding model to extract features of the reference image to obtain the reference image features of the reference image; calling the text coding model to extract features of the modified text to obtain the modified text features of the modified text.

[0133] In some embodiments, the above-mentioned image coding model and text coding model can both be machine learning models. The image coding model is obtained by training the machine learning model based on image samples, and the text coding model can be obtained by training the machine learning model based on text samples.

[0134] In some embodiments, the above-mentioned combination of reference image features and modified text features to determine the combined features of the reference image-modified text combination can be achieved as follows: adding the reference image features and the modified text features to obtain the summed features; dividing the summed features by the norm of the summed features to obtain the combined features of the reference image-modified text combination.

[0135] As an example, the expression of the combined feature of the above reference image-modified text combination can be:

[0136] Among them, T1 is used to indicate the combined features of the reference image-modified text combination, Used to indicate additive characteristics, Used to indicate the norm of the summed features.

[0137] In some embodiments, the following steps 1043 to 1045 may be performed for each candidate image respectively. Steps 1043 to 1045 will be described below.

[0138] In step 1043 , feature extraction is performed on the candidate image to obtain candidate image features of the candidate image.

[0139] In some embodiments, the above step 1043 can be implemented in the following manner: calling an image coding model, performing feature extraction on the candidate image, and obtaining candidate image features of the candidate image.

[0140] In step 1044, the first prediction network is called to predict the similarity between the candidate image and the reference image-modified text combination based on the candidate image features and the combined features to obtain a first similarity corresponding to the candidate image.

[0141] In some embodiments, the first similarity includes a first global similarity and a first local similarity, and the first prediction network includes a first prediction layer and a second prediction layer. The first global similarity is used to indicate the overall similarity between the candidate image and the reference image-modified text combination, and the first local similarity is used to indicate the local similarity between the candidate image and the reference image-modified text combination. The first prediction layer is used to predict the overall similarity between the candidate image and the reference image-modified text combination, and the second prediction layer is used to predict the local similarity between the candidate image and the reference image-modified text combination.

[0142] In some embodiments, the above-mentioned step 1044 can be implemented as follows: calling the first prediction layer, based on the candidate image features and the combined features, predicting the overall similarity between the candidate image and the reference image-modified text combination, and obtaining the first global similarity corresponding to the candidate image; performing feature segmentation on the combined features to obtain multiple local features corresponding to the combined features, and performing feature segmentation on the modified text features to obtain word features corresponding to each word in the modified text; calling the second prediction layer, based on the local features and word features, predicting the local similarity between the candidate image and the reference image-modified text combination, and obtaining the first local similarity corresponding to the candidate image.

[0143] As an example, refer to Figure 8, which is a schematic diagram of the principle of the first prediction network provided in an embodiment of the present application. The first prediction layer 85 is called to predict the overall similarity between the candidate image and the reference image-modified text combination based on the candidate image features 81 and the combined features 82, and obtain the first global similarity 87 corresponding to the candidate image; the second prediction layer 86 is called to predict the local similarity between the candidate image and the reference image-modified text combination based on the local features 83 and the word features 84, and obtain the first local similarity 88 corresponding to the candidate image.

[0144] In some embodiments, the above-mentioned calling of the first prediction layer, based on the candidate image features and the combined features, predicts the overall similarity between the candidate image and the reference image-modified text combination to obtain the first global similarity corresponding to the candidate image, can be achieved as follows: multiplying the transpose of the combined features and the candidate image features to obtain the first reference features; calling the first prediction layer, based on the first reference features, predicts the overall similarity between the candidate image and the reference image-modified text combination to obtain the first global similarity corresponding to the candidate image.

[0145] As an example, the expression of the first reference feature can be:

[0146] Among them, T2 is used to indicate the first reference feature, T1 is used to indicate the combined feature, Used to indicate candidate image features, Used to indicate additive characteristics, Used to indicate the norm of the summed features.

[0147] In some embodiments, the above-mentioned calling of the second prediction layer predicts the local similarity between the candidate image and the reference image-modified text combination based on local features and word features to obtain the first local similarity corresponding to the candidate image, which can be achieved as follows: multiplying the transpose of each word feature with the candidate image feature respectively to obtain the reference word feature corresponding to each word feature, and summing up the reference word features to obtain the second reference feature; multiplying the transpose of each local feature with the candidate image feature respectively to obtain the reference local feature corresponding to each local feature, and summing up the reference local features to obtain the third reference feature; summing the second reference feature and the third reference feature to obtain the fourth reference feature; calling the second prediction layer to predict the local similarity between the candidate image and the reference image-modified text combination based on the fourth reference feature to obtain the first local similarity corresponding to the candidate image.

[0148] As an example, the expression of the fourth reference feature may be:

[0149] Among them, T3 is used to indicate the fourth reference feature, Used to indicate the second reference feature, Used to indicate a third reference feature, Used to indicate reference word features, Used to indicate reference local features, Used to indicate reference word features.

[0150] In step 1045, the second prediction network is called to predict the similarity between the candidate image and the modified text based on the candidate image features and the modified text features to obtain a second similarity corresponding to the candidate image.

[0151] In some embodiments, the second similarity includes a second global similarity and a second local similarity, and the second prediction network includes a third prediction layer and a fourth prediction layer.

[0152] In some embodiments, the second global similarity is used to indicate the overall similarity between the candidate image and the modified text, and the second local similarity is used to indicate the local similarity between the candidate image and the modified text. The third prediction layer is used to predict the overall similarity between the candidate image and the modified text, and the fourth prediction layer is used to predict the local similarity between the candidate image and the modified text.

[0153] In some embodiments, the above-mentioned step 1045 can be implemented as follows: calling the third prediction layer, based on the candidate image features and the modified text features, predicting the overall similarity between the candidate image and the modified text, and obtaining the second global similarity corresponding to the candidate image; performing feature segmentation on the modified text features to obtain word features corresponding to each word in the modified text; calling the fourth prediction layer, based on the word features and the candidate image features, predicting the local similarity between the candidate image and the modified text, and obtaining the second local similarity corresponding to the candidate image.

[0154] In some embodiments, the above-mentioned calling of the third prediction layer to predict the overall similarity between the candidate image and the modified text based on the candidate image features and the modified text features to obtain the second global similarity corresponding to the candidate image can be achieved as follows: multiplying the transpose of the modified text features and the candidate image features to obtain the fifth reference feature; calling the third prediction layer to predict the overall similarity between the candidate image and the modified text based on the fifth reference feature to obtain the second global similarity corresponding to the candidate image.

[0155] As an example, the expression of the second global similarity may be:

[0156] in, Used to indicate the second global similarity, Used to indicate the modification of text features. Used to indicate candidate image features.

[0157] In some embodiments, the above-mentioned calling of the fourth prediction layer to predict the local similarity between the candidate image and the modified text based on the word features and the candidate image features to obtain the second local similarity corresponding to the candidate image can be achieved as follows: multiplying the transpose of each word feature by the candidate image feature respectively to obtain the reference word feature corresponding to each word feature, and summing up the reference word features to obtain the sixth reference feature; calling the fourth prediction layer to predict the local similarity between the candidate image and the modified text based on the sixth reference feature to obtain the second local similarity corresponding to the candidate image.

[0158] As an example, the expression of the second local similarity can be:

[0159] in, Used to indicate the second local similarity, It is used to indicate the second reference feature, and K is used to indicate the number of word features.

[0160] As an example, refer to Figure 9, which is a schematic diagram of the principle of the second prediction network provided in an embodiment of the present application. The third prediction layer 95 is called to predict the overall similarity between the candidate image and the modified text based on the candidate image features 91 and the modified text features 92, and obtain the second global similarity 97 corresponding to the candidate image; the modified text features 92 are feature segmented to obtain word features 93 corresponding to each word in the modified text; the fourth prediction layer 96 is called to predict the local similarity between the candidate image and the modified text based on the word features 93 and the candidate image features 91, and obtain the second local similarity 98 corresponding to the candidate image.

[0161] In some embodiments, the above-mentioned calling of the fourth prediction layer to predict the local similarity between the candidate image and the modified text based on the word features and the candidate image features to obtain the second local similarity corresponding to the candidate image can be achieved as follows: multiplying the transpose of the word features and the candidate image features to obtain the second reference features; calling the fourth prediction layer to predict the local similarity between the candidate image and the modified text based on the second reference features to obtain the second local similarity corresponding to the candidate image.

[0162] In this way, by calling the third prediction layer, the overall similarity between the candidate image and the modified text is predicted based on the candidate image features and the modified text features, and the second global similarity corresponding to the candidate image is obtained; the modified text features are feature segmented to obtain the word features corresponding to each word in the modified text; the fourth prediction layer is called, and the local similarity between the candidate image and the modified text is predicted based on the word features and the candidate image features, and the second local similarity corresponding to the candidate image is obtained, thereby effectively determining the second similarity from two different dimensions, namely, the overall and local, through the determined second global similarity and the second local similarity, thereby effectively improving the validity of the determined second similarity.

[0163] In step 105 , the first similarity and the second similarity are combined to determine at least one target image that matches the image retrieval condition from the plurality of candidate images.

[0164] In some embodiments, the target image that matches the image retrieval condition is a candidate image among multiple candidate images that meets the modification expectation corresponding to the modified text.

[0165] In some embodiments, referring to FIG. 6 , FIG. 6 is a fourth flow chart of the image retrieval method provided in an embodiment of the present application. Step 105 shown in FIG. 3 can be implemented by executing steps 1051 to 1053 shown in FIG. 6 .

[0166] In step 1051 , the first similarity and the second similarity are combined to determine the target similarity corresponding to each candidate image.

[0167] In some embodiments, the first similarity includes a first global similarity for indicating the overall similarity between the candidate image and the reference image-modified text combination, and a first local similarity for indicating the local similarity between the candidate image and the reference image-modified text combination, and the second similarity includes a second global similarity for indicating the overall similarity between the candidate image and the modified text, and a second local similarity for indicating the local similarity between the candidate image and the modified text.

[0168] In some embodiments, the above-mentioned step 1051 can be implemented as follows: the following processing is performed on each candidate image respectively: the product of the first global similarity and the second global similarity is determined as the first target similarity; the product of the first local similarity and the second local similarity is determined as the second target similarity; the sum of the first target similarity, the second target similarity and the first local similarity is determined as the target similarity corresponding to the candidate image.

[0169] As an example, the expression of the first target similarity can be:

[0170] in, Used to indicate the first target similarity, Used to indicate the first global similarity, Used to indicate the second global similarity.

[0171] As an example, the expression of the second target similarity can be:

[0172] in, Used to indicate the second target similarity, Used to indicate the second local similarity, Used to indicate the first local similarity.

[0173] As an example, the expression of the above target similarity can be:

[0174] Among them, s ij Used to indicate target similarity, Used to indicate the first target similarity, Used to indicate the second target similarity, Used to indicate the first local similarity.

[0175] In step 1052 , the candidate images are sorted in descending order of target similarity to obtain a candidate image queue.

[0176] As an example, the candidate images include candidate image A, candidate image B and candidate image C. The target similarity corresponding to candidate image A is 0.8, the target similarity corresponding to candidate image B is 0.9, and the target similarity corresponding to candidate image C is 1. The candidate images are sorted in descending order of target similarity to obtain a candidate image queue {candidate image C, candidate image B, candidate image A}.

[0177] In step 1053 , starting from the head of the candidate image queue, at least one candidate image is selected as a target image that matches the image retrieval condition.

[0178] Continuing with the above example, starting from the head of the candidate image queue {candidate image C, candidate image B, candidate image A}, candidate image C is selected as the target image that matches the image retrieval condition.

[0179] Continuing with the above example, starting from the head of the candidate image queue {candidate image C, candidate image B, candidate image A}, candidate image C and candidate image B are selected as target images that match the image retrieval conditions.

[0180] In some embodiments, after executing step 105 above, the target image may be displayed in the following manner: at least one target image is sent to the terminal, so that the terminal displays the at least one target image.

[0181] In some embodiments, after the terminal displays at least one target image, the terminal determines the selected target image as the first target image and the unselected target image as the second target image in response to a selection operation for the at least one target image, the terminal cancels display of the second target image, and continues to display the first target image.

[0182] In this way, by obtaining an image retrieval condition including a reference image and modified text, as well as multiple candidate images, and determining a first similarity between each candidate image and the reference image-modified text combination, as well as a second similarity between the candidate image and the modified text, a target image that matches the image retrieval condition is determined from the multiple candidate images by combining the first similarity and the second similarity. In this way, since the retrieved target image that meets the image retrieval condition is determined based on the first similarity between the candidate image and the reference image-modified text combination, it is possible to effectively ensure that the determined target image can simultaneously meet the retrieval requirements of the reference image and the modified text. At the same time, by using the second similarity between the candidate image and the modified text to participate in the determination of the target image, the influence of the modification expectation of the modified text on the determined target image can be strengthened, so that the target image can be highly consistent with the modification expectation of the modified text, thereby effectively improving the accuracy of image retrieval.

[0183] The following describes an exemplary application of the embodiment of the present application in an actual image retrieval application scenario.

[0184] In recent years, with the rapid development of the internet, multimedia data in various forms, such as text, images, and audio, has exploded. With the emergence of a wide variety of identical or similar content, multimedia retrieval has become a fundamental task for users to flexibly obtain information. Therefore, to meet users' increasingly complex retrieval needs, language-visual combined query-based image retrieval (LVCQ-IR) has been proposed and has received increasing attention. As one of the most popular multimedia retrieval models today, LVCQ-IR uses a reference image and a textual description of a desired modification to the image as query inputs, and retrieves the corresponding target image through a database query. Because the input text expresses the intended modification of the reference image, existing retrieval models for LVCQ-IR primarily focus on designing a synthetic network. Specifically, they fuse the feature representation of the reference image with the feature representation of the intended text into a combined query representation. Through model training, the combined query representation is trained to be as close as possible to the feature representation of the target image.

[0185] However, during model design, the fused combined query representation is significantly influenced by the reference image, thus neglecting some or even all of the information in the intended context. For example, if a non-target image A is very similar to the reference image B used during training, but A does not match the description in the intended context, while the true target image C does match the description in the intended context, then during model training, the fused combined query representation will likely become closer and closer to the feature representation of A than to the representation of the true target image C, leading to incorrect retrieval results.

[0186] Therefore, since the target image matches the semantic information contained in the modified text, the similarity between the target image and the text should be very high, but the existing LVCQ-IR models have ignored this sign. In order to solve the above problem, the embodiment of the present application proposes an image retrieval method, which uses two loss functions to optimize the model during the training process, so that the learned features of the combined query are close to the learned features of its target image, and the features of the modified text are similar to the features of the target image, that is, the effect of the content of the modified text on the model optimization is enhanced. Finally, in the inference stage, the similarity between the modified text and the candidate image is used as a filter, and it is integrated into the similarity relationship between the combined query and the candidate image to obtain the final similarity result for target image retrieval.

[0187] Currently, using images to query images fails to meet user search intent. Combining text and images for image queries can further satisfy user search intent and retrieve accurate images. However, facing large-scale, complex data in real-world scenarios, combined text and image queries still present significant challenges in achieving accurate search results. Therefore, effectively ensuring the search accuracy of combined text and image queries is of great research significance and practical value.

[0188] In some embodiments, see Figure 10, which is a schematic diagram of the principle of the image retrieval method provided by the embodiment of the present application. The image retrieval method provided by the embodiment of the present application can be implemented by two major sub-modules: a feature extraction module and a similarity generator. The feature extraction module is used to convert the feature representations of the reference image sample, the modified text, and the target image sample into a set of feature sequences. According to the different forms of the information to be processed, it can be further divided into an image encoder and a text encoder, and both are designed based on the CLIP model; while the similarity generation module innovatively considers the similarity relationship between the target image and the reference image and the modified text, and not only designs a universal combined query feature Same target image features Similarities between A separate modified text feature is also calculated Same target image features Similarities between Therefore, based on the basic characteristic that "the target image contains the semantic information of the modified text, so the similarity between the modified text and the target image should be high", the m2t similarity is used as a filter to filter out false target images to improve the retrieval performance.

[0189] In some embodiments, referring to FIG10 , for the image encoder, the image encoder adopts the same structure as the 12-layer ViT in CLIP, aiming to process the reference image samples and the target image samples into a set of visual feature sequences (reference image sample features corresponding to the reference image samples, target image sample features corresponding to the target image samples), and for the input image Where H×W represents the resolution of the image, the encoder first converts It is divided into M sub-segments, and then all sub-segments are converted into a set of feature sequences, namely In the Before inputting the transformer, a learnable landmark feature [CLS] is added to the original feature sequence to facilitate better learning of feature representation. Finally, after being processed by the transformer, the feature sequence representation of the image feature is generated, that is, At the same time, in order to facilitate the subsequent similarity calculation, all feature sequence representations are l2 normalized, that is,

[0190] In some embodiments, referring to FIG10 , for a text encoder, the text encoder also utilizes the structure in the CLIP model and adopts a similar processing strategy, that is, First, it is marked as a set of feature sequences (modified text features corresponding to the modified text) and input into the transformer stack. Finally, after processing, the feature form of the input text (the modified text shown in Figure 10) can be expressed as Where K represents the total number of words in the input text, represents the feature representation of the jth word in the input text, and and They represent the representation of two special features in the CLIP model, namely [SOS] and [EOS]. Finally, similar to the above processing form, all text feature sequences are subjected to l2 normalization, namely

[0191] In some embodiments, for the first prediction network in the similarity generator, in the previous c2t similarity optimization, the model expects that for each j≠i, the similarity Should be higher than Finally, in the reasoning phase, for the combined query (xr (reference image),t m (modify text)), return the target image x t , so that x t Same as (x r ,t m ) has the highest c2t similarity, but as mentioned above, this similarity is affected by x r Therefore, in addition to calculating the c2t similarity, the modified text features are also calculated. Same target image features Similarities between That is, m2t similarity.

[0192] In some embodiments, referring to FIG10 , for the first prediction network in the similarity generator, the same structure as CLVC-Net is used. For a given combined query and candidate images Calculate the similarity from the global and local perspectives respectively, denoted as (first global similarity) and (first local similarity), where the global c2t similarity It is defined as the cosine similarity between the global features of the combined query and the global features of the candidate image, denoted as:

[0193] in, Represents image x * The global features after l2 normalization, That is, the [EOS] feature indicates that the text is modified The global features of the two are represented by the sum of the whole combination query. The global characteristics of Candidate images global characteristics.

[0194] In addition, in order to calculate the local similarity, the experiment first defines a local feature in the combined query and the candidate image The similarities between them are: Then based on the similarity relationship of candidate images with a single feature, the final local similarity It can be written as:

[0195] in, is the local feature of the combined query, To modify the local features of the text, M is the number of all local features of the candidate image, and K is the number of words in the modified text.

[0196] In some embodiments, referring to FIG10 , for the second prediction network in the similarity generator, similar to the c2t similarity, the embodiment of the present application also calculates the m2t similarity from both global and local perspectives, which is recorded as (second global similarity) and (Second local similarity), because m2t similarity only considers the similarity between the modified text and the target image, and has nothing to do with the reference image, the calculation formula is simpler than c2t similarity, which is recorded as:

[0197] In some embodiments, for the loss function, in order to make for each j≠i, the matching triples The c2t similarity of the triples is greater than that of the unmatched triples Therefore, InfoNCE loss is selected as the loss function to optimize the similarity relationship between the combined query and the target image, which is recorded as:

[0198] At the same time, because m2t similarity can be used as a filter to further improve the retrieval performance, and The m2t similarity between and The similarity between Not included To achieve this goal, a contrastive learning paradigm is adopted here, that is, Only with Similar to the candidate image, but not similar to other candidate images. On this basis, the loss function can be written as:

[0199] Therefore, the loss function in the final Text2Target Filter model is recorded as:

[0200] In some embodiments, during the inference phase, the present application first fuses the m2t similarity with the c2t similarity to construct a filtered global and local combined query to obtain the candidate image similarity, which is denoted as:

[0201] Therefore, for the combined query input and candidate images Four different types of similarity relationships were generated, namely: global c2t similarity Local c2t similarity C2t similarity after global filtering C2t similarity after local filtering After experimental verification, the final similarity s used for retrieval ij It can be written as:

[0202] It is understandable that in the embodiments of the present application, when data related to reference images, modified text, etc. is involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0203] The following continues to describe an exemplary structure of the image retrieval device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in Figure 2, the software modules stored in the image retrieval device 455 in the memory 450 may include: an acquisition module 4551, configured to acquire image retrieval conditions, the image retrieval conditions including a reference image and a modified text, the modified text being used to indicate a desire to modify the reference image; a combination module 4552, configured to combine the reference image and the modified text to obtain a reference image-modified text combination; a similarity module 4553, configured to acquire multiple candidate images and determine a first similarity between each candidate image and the reference image-modified text combination, and a second similarity between each candidate image and the modified text; a retrieval module 4554, configured to combine the first similarity and the second similarity to determine at least one target image that matches the image retrieval conditions from the multiple candidate images.

[0204] In some embodiments, the similarity module 4553 is further configured to obtain an image retrieval model, which includes a first prediction network and a second prediction network; perform feature extraction on the reference image and the modified text respectively to obtain reference image features of the reference image and modified text features of the modified text, and determine the combined features of the reference image-modified text combination by combining the reference image features and the modified text features; perform the following processing on each candidate image respectively: perform feature extraction on the candidate image to obtain candidate image features of the candidate image; call the first prediction network to predict the similarity between the candidate image and the reference image-modified text combination based on the candidate image features and the combined features to obtain the first similarity corresponding to the candidate image; call the second prediction network to predict the similarity between the candidate image and the modified text based on the candidate image features and the modified text features to obtain the second similarity corresponding to the candidate image.

[0205] In some embodiments, the first similarity includes a first global similarity and a first local similarity, and the first prediction network includes a first prediction layer and a second prediction layer; the above-mentioned similarity module 4553 is also configured to call the first prediction layer, and based on the candidate image features and the combined features, predict the overall similarity between the candidate image and the reference image-modified text combination to obtain the first global similarity corresponding to the candidate image; perform feature segmentation on the combined features to obtain multiple local features corresponding to the combined features, and perform feature segmentation on the modified text features to obtain word features corresponding to each word in the modified text; call the second prediction layer, and based on the local features and word features, predict the local similarity between the candidate image and the reference image-modified text combination to obtain the first local similarity corresponding to the candidate image.

[0206] In some embodiments, the above-mentioned similarity module 4553 is further configured to add the reference image feature and the modified text feature to obtain a sum feature; divide the sum feature by the norm of the sum feature to obtain a combined feature of the reference image-modified text combination; the above-mentioned similarity module 4553 is further configured to multiply the transpose of the combined feature and the candidate image feature to obtain a first reference feature; call the first prediction layer, and predict the overall similarity between the candidate image and the reference image-modified text combination based on the first reference feature to obtain a first global similarity corresponding to the candidate image.

[0207] In some embodiments, the similarity module 4553 is further configured to multiply the transpose of each word feature by the candidate image feature to obtain a reference word feature corresponding to each word feature, and sum up the reference word features to obtain a second reference feature; multiply the transpose of each local feature by the candidate image feature to obtain a reference local feature corresponding to each local feature, and sum up the reference local features to obtain a third reference feature; sum the second reference feature and the third reference feature to obtain a fourth reference feature; call the second prediction layer, and predict the local similarity between the candidate image and the reference image-modified text combination based on the fourth reference feature to obtain a first local similarity corresponding to the candidate image.

[0208] In some embodiments, the second similarity includes a second global similarity and a second local similarity, and the second prediction network includes a third prediction layer and a fourth prediction layer; the above-mentioned similarity module 4553 is also configured to call the third prediction layer, based on the candidate image features and the modified text features, predict the overall similarity between the candidate image and the modified text, and obtain the second global similarity corresponding to the candidate image; perform feature segmentation on the modified text features to obtain word features corresponding to each word in the modified text; call the fourth prediction layer, based on the word features and the candidate image features, predict the local similarity between the candidate image and the modified text, and obtain the second local similarity corresponding to the candidate image.

[0209] In some embodiments, the similarity module 4553 is further configured to obtain image retrieval condition samples and target image samples, the image retrieval condition samples including reference image samples and modified text samples, the modified text samples being used to indicate the modification expectation for the reference image samples, and the target image samples satisfying the modification expectation; combining the reference image samples and the modified text samples to obtain a reference image-modified text combination sample; obtaining multiple candidate images, the candidate images including the target image sample; calling the initial image retrieval model for each candidate image, and performing similarity prediction on the candidate image based on the candidate image, the reference image sample and the modified text sample to obtain a third similarity between the candidate image and the reference image-modified text combination sample, and a fourth similarity between the candidate image and the modified text sample; and training the initial image retrieval model in combination with the third similarity and the fourth similarity to obtain an image retrieval model.

[0210] In some embodiments, the above-mentioned similarity module 4553 is also configured to determine a first loss value of the initial image retrieval model based on the third similarity corresponding to the target image sample and the third similarity corresponding to each candidate image; determine a second loss value of the initial image retrieval model based on the fourth similarity corresponding to the target image sample and the fourth similarity corresponding to each candidate image; perform weighted summation of the first loss value and the second loss value to obtain a target loss value, and train the initial image retrieval model based on the target loss value to obtain an image retrieval model.

[0211] In some embodiments, the above-mentioned retrieval module 4554 is further configured to determine the target similarity corresponding to each candidate image by combining the first similarity and the second similarity; sort the candidate images in descending order of target similarity to obtain a candidate image queue; and starting from the head of the candidate image queue, select at least one candidate image as the target image that matches the image retrieval condition.

[0212] In some embodiments, the first similarity includes a first global similarity for indicating the overall similarity between the candidate image and the reference image-modified text combination, and a first local similarity for indicating the local similarity between the candidate image and the reference image-modified text combination, and the second similarity includes a second global similarity for indicating the overall similarity between the candidate image and the modified text, and a second local similarity for indicating the local similarity between the candidate image and the modified text; the above-mentioned retrieval module 4554 is also configured to perform the following processing for each candidate image respectively: determine the product of the first global similarity and the second global similarity as the first target similarity; determine the product of the first local similarity and the second local similarity as the second target similarity; determine the sum of the first target similarity, the second target similarity and the first local similarity as the target similarity corresponding to the candidate image.

[0213] In some embodiments, the above-mentioned retrieval module 4554 is also configured to receive an image retrieval request, which carries an image retrieval condition. The image retrieval request is generated by the terminal based on an image retrieval instruction, and the image retrieval instruction is used to instruct image retrieval based on the input reference image and modified text; the image retrieval request is parsed to obtain the image retrieval condition.

[0214] In some embodiments, the image retrieval device 455 further includes: a sending module configured to send at least one target image to the terminal, so that the terminal displays the at least one target image.

[0215] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image retrieval method described above in the present invention.

[0216] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the image retrieval method provided by an embodiment of the present application, for example, the image retrieval method shown in Figure 3.

[0217] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a CD-ROM; or it may be various electronic devices including one or any combination of the above memories.

[0218] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0219] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0220] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0221] In summary, the embodiments of the present application have the following beneficial effects:

[0222] (1) By obtaining an image retrieval condition including a reference image and a modified text, and a plurality of candidate images, and determining a first similarity between each candidate image and the reference image-modified text combination, and a second similarity between the candidate image and the modified text, a target image that matches the image retrieval condition is determined from the plurality of candidate images by combining the first similarity and the second similarity. In this way, since the retrieved target image that meets the image retrieval condition is determined based on the first similarity between the candidate image and the reference image-modified text combination, it is possible to effectively ensure that the determined target image can simultaneously meet the retrieval requirements of the reference image and the modified text. At the same time, by participating in the determination of the target image through the second similarity between the candidate image and the modified text, it is possible to strengthen the influence of the modification expectation of the modified text on the determined target image, so that the target image can be highly consistent with the modification expectation of the modified text, thereby effectively improving the accuracy of image retrieval.

[0223] (2) Since the candidate image includes the target image sample, the third similarity of the candidate image and the fourth similarity of the candidate image include the third similarity and the fourth similarity of the target image sample. Since the target image sample is the candidate image with the highest similarity among the candidate images, the third similarity and the fourth similarity of the target image sample can be used as training labels to train the initial image retrieval model, so that the obtained image retrieval model can effectively learn the feature representation of the target image sample and can accurately identify the corresponding target image from a large number of candidate images, thereby effectively improving the retrieval performance of the image retrieval model.

[0224] (3) By determining the first similarity between each candidate image and the reference image-modified text combination, and the second similarity between each candidate image and the modified text, since the first similarity and the second similarity can accurately reflect the overall and local similarity between the candidate image and the reference image-modified text combination, and the overall and local similarity between the candidate image and the modified text, the similarity is comprehensively measured from multiple perspectives of overall and local, thereby effectively improving the accuracy of the determined first similarity and second similarity.

[0225] (4) The first loss value of the initial image retrieval model is determined based on the third similarity corresponding to the target image sample and the third similarity corresponding to each candidate image; the second loss value of the initial image retrieval model is determined based on the fourth similarity corresponding to the target image sample and the fourth similarity corresponding to each candidate image; the first loss value and the second loss value are weightedly summed to obtain the target loss value, and the initial image retrieval model is trained based on the target loss value to obtain the image retrieval model. Since the trained image retrieval model makes full use of the third similarity corresponding to the target image sample and the third similarity corresponding to each candidate image, the retrieval performance of the image retrieval model is effectively improved.

[0226] (5) By calling the third prediction layer, the overall similarity between the candidate image and the modified text is predicted based on the candidate image features and the modified text features, and the second global similarity corresponding to the candidate image is obtained; the modified text features are feature segmented to obtain the word features corresponding to each word in the modified text; the fourth prediction layer is called, and the local similarity between the candidate image and the modified text is predicted based on the word features and the candidate image features, and the second local similarity corresponding to the candidate image is obtained, thereby effectively determining the second similarity from two different dimensions of overall and local through the determined second global similarity and the second local similarity, thereby effectively improving the validity of the determined second similarity.

[0227] (6) Since the candidate image includes the target image sample, the third similarity of the candidate image and the fourth similarity of the candidate image include the third similarity and the fourth similarity of the target image sample. Since the target image sample is the candidate image with the highest similarity among the candidate images, the third similarity and the fourth similarity of the target image sample can be used as training labels to train the initial image retrieval model, so that the obtained image retrieval model can effectively learn the feature representation of the target image sample and can accurately identify the corresponding target image from a large number of candidate images, thereby effectively improving the retrieval performance of the image retrieval model.

[0228] (7) The image retrieval method provided in the embodiment of the present application can be implemented by two major submodules: a feature extractor and a similarity generator. The feature extraction module is used to convert the feature representations of the reference image, the modified text and the target image into a set of feature sequences respectively. According to the different forms of the information to be processed, it can be further divided into an image encoder and a text encoder, and both are designed based on the CLIP model; while the similarity generation module innovatively considers the similarity relationship between the target image and the reference image and the modified text, and not only designs a universal combined query feature Same target image features Similarities between A separate modified text feature is also calculated Same target image features Similarities between Therefore, based on the basic characteristic that "the target image contains the semantic information of the modified text, so the similarity between the modified text and the target image should be high", the m2t similarity is used as a filter to filter out false target images to improve the retrieval performance.

[0229] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image retrieval method, the method comprising: Acquire an image retrieval condition, wherein the image retrieval condition includes a reference image and a modification text, wherein the modification text is used to indicate a modification expectation for the reference image; Combining the reference image and the modified text to obtain a reference image-modified text combination; Acquire multiple candidate images, and for each of the candidate images, determine a first similarity between the candidate image and the reference image-modified text combination, and a second similarity between the candidate image and the modified text; At least one target image matching the image retrieval condition is determined from the plurality of candidate images in combination with the first similarity and the second similarity.

2. The method according to claim 1, wherein: The determining of a first similarity between the candidate image and the reference image-modified text combination, and a second similarity between the candidate image and the modified text, comprises: Acquire an image retrieval model, wherein the image retrieval model includes a first prediction network and a second prediction network; Extracting features of the reference image and the modified text respectively to obtain reference image features of the reference image and modified text features of the modified text, and combining the reference image features and the modified text features to determine combined features of the reference image-modified text combination; Performing feature extraction on the candidate image to obtain candidate image features of the candidate image; Calling the first prediction network to predict the similarity between the candidate image and the reference image-modified text combination based on the candidate image feature and the combined feature, and obtaining the first similarity corresponding to the candidate image; The second prediction network is called to predict the similarity between the candidate image and the modified text based on the candidate image features and the modified text features, so as to obtain the second similarity corresponding to the candidate image.

3. The method according to any one of claims 1 to 2, wherein: The first similarity includes a first global similarity and a first local similarity, and the first prediction network includes a first prediction layer and a second prediction layer; The calling the first prediction network to predict the similarity between the candidate image and the reference image-modified text combination based on the candidate image feature and the combined feature to obtain the first similarity corresponding to the candidate image includes: Calling the first prediction layer to predict the overall similarity between the candidate image and the reference image-modified text combination based on the candidate image features and the combined features, and obtain the first global similarity corresponding to the candidate image; Performing feature segmentation on the combined feature to obtain a plurality of local features corresponding to the combined feature, and performing feature segmentation on the modified text feature to obtain word features corresponding to each word in the modified text; The second prediction layer is called to predict the local similarity between the candidate image and the reference image-modified text combination based on the local features and the word features, so as to obtain the first local similarity corresponding to the candidate image.

4. The method according to any one of claims 1 to 3, wherein: The combining the reference image feature and the modified text feature to determine the combined feature of the reference image-modified text combination includes: Adding the reference image feature and the modified text feature to obtain a sum feature; Dividing the sum feature by the norm of the sum feature to obtain a combined feature of the reference image-modified text combination; The calling the first prediction layer, predicting the overall similarity between the candidate image and the reference image-modified text combination based on the candidate image feature and the combined feature, to obtain the first global similarity corresponding to the candidate image, includes: Multiplying the transpose of the combined feature and the candidate image feature to obtain a first reference feature; The first prediction layer is called to predict the overall similarity between the candidate image and the reference image-modified text combination based on the first reference feature to obtain the first global similarity corresponding to the candidate image.

5. The method according to any one of claims 1 to 4, wherein: The calling the second prediction layer to predict the local similarity between the candidate image and the reference image-modified text combination based on the local features and the word features to obtain the first local similarity corresponding to the candidate image includes: Multiplying the transposition of each word feature by the candidate image feature to obtain a reference word feature corresponding to each word feature, and summing the reference word features to obtain a second reference feature; Multiplying the transpose of each of the local features by the candidate image feature to obtain a reference local feature corresponding to each of the local features, and summing the reference local features to obtain a third reference feature; summing the second reference feature and the third reference feature to obtain a fourth reference feature; The second prediction layer is called to predict the local similarity between the candidate image and the reference image-modified text combination based on the fourth reference feature to obtain the first local similarity corresponding to the candidate image.

6. The method according to any one of claims 1 to 5, wherein: The second similarity includes a second global similarity and a second local similarity, and the second prediction network includes a third prediction layer and a fourth prediction layer; The calling of the second prediction network to predict the similarity between the candidate image and the modified text based on the candidate image feature and the modified text feature to obtain the second similarity corresponding to the candidate image includes: Calling the third prediction layer to predict the overall similarity between the candidate image and the modified text based on the candidate image features and the modified text features, and obtain the second global similarity corresponding to the candidate image; Performing feature segmentation on the modified text features to obtain word features corresponding to each word in the modified text; The fourth prediction layer is called to predict the local similarity between the candidate image and the modified text based on the word features and the candidate image features, so as to obtain the second local similarity corresponding to the candidate image.

7. The method according to any one of claims 1 to 6, wherein: The calling the third prediction layer to predict the overall similarity between the candidate image and the modified text based on the candidate image features and the modified text features to obtain the second global similarity corresponding to the candidate image includes: multiplying the transpose of the modified text feature and the candidate image feature to obtain a fifth reference feature; The third prediction layer is called to predict the overall similarity between the candidate image and the modified text based on the fifth reference feature to obtain the second global similarity corresponding to the candidate image.

8. The method according to any one of claims 1 to 6, wherein: The calling the fourth prediction layer to predict the local similarity between the candidate image and the modified text based on the word feature and the candidate image feature to obtain the second local similarity corresponding to the candidate image includes: Multiplying the transposition of each word feature by the candidate image feature respectively to obtain a reference word feature corresponding to each word feature, and summing up the reference word features to obtain a sixth reference feature; The fourth prediction layer is called to predict the local similarity between the candidate image and the modified text based on the sixth reference feature to obtain a second local similarity corresponding to the candidate image.

9. The method according to any one of claims 1 to 2, wherein: The step of acquiring the image retrieval model comprises: Acquire an image retrieval condition sample and a target image sample, wherein the image retrieval condition sample includes a reference image sample and a modification text sample, wherein the modification text sample is used to indicate a modification expectation for the reference image sample, and the target image sample satisfies the modification expectation; Combining the reference image sample and the modified text sample to obtain a reference image-modified text combined sample; Acquire multiple candidate images, wherein the candidate images include the target image sample; For each of the candidate images, calling the initial image retrieval model, based on the candidate image, the reference image sample and the modified text sample, performing similarity prediction on the candidate image, obtaining a third similarity between the candidate image and the reference image-modified text combination sample, and a fourth similarity between the candidate image and the modified text sample; Combine the third similarity and the fourth similarity to train the initial image retrieval model to obtain the Image retrieval model.

10. The method according to any one of claims 1 to 9, wherein: The step of training the initial image retrieval model by combining the third similarity and the fourth similarity to obtain the image retrieval model comprises: Determining a first loss value of the initial image retrieval model based on a third similarity corresponding to the target image sample and a third similarity corresponding to each of the candidate images; Determining a second loss value of the initial image retrieval model based on a fourth similarity corresponding to the target image sample and a fourth similarity corresponding to each of the candidate images; The first loss value and the second loss value are weightedly summed to obtain a target loss value, and the initial image retrieval model is trained based on the target loss value to obtain the image retrieval model.

11. The method according to any one of claims 1 to 10, wherein: The combining the first similarity and the second similarity to determine at least one target image matching the image retrieval condition from the plurality of candidate images comprises: Determine a target similarity corresponding to each of the candidate images by combining the first similarity and the second similarity; Sorting the candidate images in descending order of the target similarity to obtain a candidate image queue; Starting from the head of the candidate image queue, at least one candidate image is selected as a target image that matches the image retrieval condition.

12. The method according to any one of claims 1 to 11, wherein: The first similarity includes a first global similarity for indicating the overall similarity between the candidate image and the reference image-modified text combination, and a first local similarity for indicating the local similarity between the candidate image and the reference image-modified text combination, and the second similarity includes a second global similarity for indicating the overall similarity between the candidate image and the modified text, and a second local similarity for indicating the local similarity between the candidate image and the modified text; The combining the first similarity and the second similarity to determine the target similarity corresponding to each of the candidate images includes: The following processing is performed for each candidate image: Determine the product of the first global similarity and the second global similarity as a first target similarity; Determine the product of the first local similarity and the second local similarity as a second target similarity; The sum of the first target similarity, the second target similarity and the first local similarity is determined as the target similarity corresponding to the candidate image.

13. The method according to any one of claims 1 to 12, wherein: The obtaining of image retrieval conditions includes: receiving an image retrieval request, the image retrieval request carrying the image retrieval condition, the image retrieval request being generated by the terminal based on an image retrieval instruction, the image retrieval instruction being used to instruct image retrieval based on the input reference image and the modified text; Parsing the image retrieval request to obtain the image retrieval condition; After determining at least one target image matching the image retrieval condition from the plurality of candidate images by combining the first similarity and the second similarity, the method further comprises: The at least one target image is sent to the terminal so that the terminal displays the at least one target image.

14. An image retrieval device, comprising: An acquisition module configured to acquire an image retrieval condition, wherein the image retrieval condition includes a reference image and a modification text, wherein the modification text is used to indicate a modification expectation for the reference image; a combining module, configured to combine the reference image and the modified text to obtain a reference image-modified text combination; a similarity module configured to obtain a plurality of candidate images, and for each of the candidate images, determine a first similarity between the candidate image and the reference image-modified text combination, and a second similarity between the candidate image and the modified text; The retrieval module is configured to determine at least one target image that matches the image retrieval condition from the multiple candidate images based on the first similarity and the second similarity.

15. An electronic device, comprising: a memory configured to store computer executable instructions or computer programs; The processor is configured to implement the image retrieval method according to any one of claims 1 to 13 when executing the computer executable instructions or computer programs stored in the memory.

16. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the image retrieval method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program or a computer executable instruction, wherein the computer program or the computer executable instruction, when executed by a processor, implements the image retrieval method according to any one of claims 1 to 13.

Citation Information

Patent Citations

  • Combined query image retrieval method based on multi-order adversarial feature learning

    CN112818157A

  • Image retrieval method and system based on multi-modal query, medium and equipment

    CN113239219A

  • Combined query image retrieval method based on cross-modal attention reservation

    CN115858847A

  • Image retrieval method and device, electronic equipment, storage medium and program product

    CN116975341A

  • Data compatibility for text-enhanced visual retrieval

    US20230073843A1

Cited By

  • Compound structure cross-modal search method and system

    CN120032757A

  • Cultural relic image retrieval method, system and equipment and storage medium

    CN120892597A

  • Rapid cross-modal retrieval method and system fusing fine-grained semantics

    CN121051254A