Image-text matching methods and electronic devices

Abstract

This application discloses an image-text matching method and electronic device, relating to the field of neural networks, for improving the accuracy of image-text matching processing. The image-text matching method includes: a display interface, the interface including controls; responding to operations on the controls, matching the search target according to a student image-text model, obtaining and displaying the search results; the training process of the student image-text model includes: inputting an image into a student image encoder to obtain a student image vector; inputting an image into a teacher image encoder to obtain a teacher image vector; inputting text into a student text encoder to obtain a student text vector; inputting text into a teacher text encoder to obtain a teacher text vector; fusing the student image vector and student text vector to obtain student image-text information; fusing the teacher image vector and teacher text vector to obtain teacher image-text information; using the teacher image-text information to perform knowledge distillation on the student image-text information, and updating the parameters of the student image-text model based on the distillation loss after knowledge distillation.

Description

Technical Field

[0001] This application relates to the field of neural networks, and more particularly to a text-image matching method and an electronic device. Background Technology

[0002] Image-text matching (ITM) is a fundamental form of vision-language (VL) processing. It allows users to search for corresponding images based on text, or vice versa. However, the limited processing resources of miniaturized electronic devices and the small scale of their neural network models result in low accuracy for ITM. Summary of the Invention

[0003] This application provides a text-image matching method and an electronic device to improve the accuracy of text-image matching processing by a neural network model.

[0004] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0005] Firstly, a text-image matching method is provided, comprising: displaying an interface including controls for obtaining a search target; responding to operations on the controls, matching the search target according to a student text-image model to obtain a search result; wherein the search target is text and the search result is visual media, or the search target is visual media and the search result is text; and displaying the search result; wherein the training process of the student text-image model includes: inputting images from multiple text-image pairs into a student image encoder in the student text-image model to obtain student image vectors; inputting images into a teacher image encoder in a teacher text-image model to obtain teacher image vectors; inputting text from multiple text-image pairs into a student text encoder in the student text-image model to obtain student text vectors; inputting text into a teacher text encoder in the teacher text-image model to obtain teacher text vectors; fusing student image vectors and student text vectors to obtain student text-image information; fusing teacher image vectors and teacher text vectors to obtain teacher text-image information; performing knowledge distillation on student text-image information using teacher text-image information, and updating the parameters of the student text-image model according to the distillation loss after knowledge distillation.

[0006] The image-text matching method provided in this application involves inputting image-text pairs into a student image-text model to obtain student image vectors and student text vectors, and inputting identical image-text pairs into a teacher image-text model to obtain teacher image vectors and teacher text vectors. The student image vectors and student text vectors are fused to obtain student image-text information; the teacher image vectors and teacher text vectors are fused to obtain teacher image-text information. On one hand, the teacher image-text information comes from a more complex teacher image-text model, possessing higher-level image-text semantic information compared to student image-text information from a simpler student image-text model. On the other hand, the fused image-text information, compared to simple image or text information, possesses higher-level image-text semantic information, resulting in higher knowledge representation capabilities when used for knowledge distillation. Then, the teacher image-text information is used to perform knowledge distillation on the student image-text information. The parameters of the student image-text model are updated based on the distillation loss, and the updated student image-text model is used for image-text matching. In this way, the updated student image-text model learns the knowledge of the complex teacher image-text model, improving the accuracy of the neural network model in image-text matching.

[0007] In one possible implementation, student image vectors and student text vectors are fused to obtain student image-text information; teacher image vectors and teacher text vectors are fused to obtain teacher image-text information, including: performing matrix multiplication of student image vectors and student text vectors to obtain a student image-text information matrix; and performing matrix multiplication of teacher image vectors and teacher text vectors to obtain a teacher image-text information matrix. The fused image-text information, compared to pure image or text information, possesses a higher level of semantic information and exhibits greater knowledge representation capability when used for knowledge distillation.

[0008] In one possible implementation, knowledge distillation is performed on student image-text information using teacher image-text information. The parameters of the student image-text model are then updated based on the distillation loss. This includes: obtaining image distillation loss and text distillation loss based on the student image-text information matrix and the teacher image-text information matrix; the image distillation loss represents the loss of the student image vector relative to the teacher image vector, and the text distillation loss represents the loss of the student text vector relative to the teacher text vector; obtaining the distillation loss based on the image distillation loss and the text distillation loss; obtaining the total loss based on the distillation loss and the training loss; the training loss represents the loss between the model output and the true value after training the student image-text model using image-text pairs; and updating the parameters of the student image-text model based on the total loss. Knowledge from the teacher image-text model can be used to update the parameters of the student image-text model.

[0009] In one possible implementation, the image distillation loss and text distillation loss are obtained based on the student's image-text information matrix and the teacher's image-text information matrix, including: obtaining the image distillation loss Distillation according to the following two formulas respectively.image and text distillation loss text :

[0010]

[0011]

[0012] Where, matrix S′ V S represents the student's text and image information matrix V The transpose of matrix T′ V Represents the teacher's text and image information matrix T V The transpose of , softmax() represents the exponential normalization function, log_softmax() represents taking the log of the result of softmax(), S V (i) represents the student graphic information matrix S V The i-th row of features, T V (i) represents the teacher's graphic information matrix T V The i-th row of features, S′ V (i) represents matrix S′ V The i-th row of features, T′ V (i) represents matrix T′ V The i-th row of features in the diagram, where N represents the number of image-text pairs.

[0013] The `softmax()` function has a normalization characteristic: larger values ​​result in a normalized value closer to 1, while smaller values ​​result in a normalized value closer to 0. Using `softmax()` to fuse the teacher image vectors and teacher text vectors in a teacher-text image model can significantly strengthen the knowledge on the diagonal of the teacher-text image information matrix, enhancing the distillation value of the knowledge. This is because the values ​​on the diagonal of the teacher-text image information matrix are much larger than those off-diagonal, meaning the knowledge on the diagonal of the teacher-text image information matrix is ​​more abundant and of higher value.

[0014] The log_softmax() function can map values ​​with large differences obtained by the softmax() function to the same level, effectively avoiding the overflow problem caused by excessively large values ​​of certain features in the student's text-image information matrix, and ensuring the stability of the knowledge received by the student's text-image model.

[0015] In one possible implementation, the distillation loss is obtained based on the image distillation loss and the text distillation loss, including: weighted summing of the image distillation loss and the text distillation loss. The loss from knowledge distillation includes the loss of the image after knowledge distillation and the loss of the text.

[0016] In one possible implementation, the total loss is obtained from the distillation loss and the training loss, including: weighted summing of the distillation loss and the training loss. The total loss includes the loss from knowledge distillation and the loss from model training.

[0017] In one possible implementation, updating the parameters of the student image-text model based on the total loss includes: iteratively updating the parameters of the student image-text model using gradient descent until the total loss converges to the minimum, and reaching a preset number of iterations.

[0018] In one possible implementation, the teacher-text model outperforms the student-text model in at least one of the following: number of model layers, number of hidden neurons, size of the multilayer perceptron, and number of heads. The teacher-text model is more complex and possesses higher-level semantic information compared to the simpler student-text model. By learning the more complex teacher-text model, the performance of the student-text model in text-text matching can be enhanced.

[0019] In a second aspect, an electronic device is provided, including a processor and a memory, wherein instructions are stored in the memory, and when the processor executes the instructions, the method described in the first aspect and any embodiment thereof is executed.

[0020] Thirdly, a computer-readable storage medium is provided, including instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in the first aspect and any embodiment thereof.

[0021] Fourthly, a computer program product containing instructions is provided, which, when executed on the aforementioned electronic device, cause the electronic device to perform the method as described in the first aspect and any embodiment thereof.

[0022] Fifthly, a chip system is provided, including a processor for supporting electronic devices in implementing the functions described in the first aspect above. In one possible design, the device further includes interface circuitry for receiving signals from other devices (e.g., memory) or sending signals to other devices (e.g., a communication interface). The chip system may include a chip and may also include other discrete devices.

[0023] The technical effects of the second to fifth aspects refer to the technical effects of the first aspect and any of its embodiments, and will not be repeated here. Attached Figure Description

[0024] Figure 1 A schematic diagram illustrating image-text matching provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of an interface for performing text search in a mobile phone's gallery application, provided as an embodiment of this application.

[0026] Figure 3 This is a schematic diagram of an interface for image-based text search in a mobile phone's intelligent recognition application, provided as an embodiment of this application.

[0027] Figure 4 This is a schematic diagram of a text-image matching system provided in an embodiment of this application;

[0028] Figure 5 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0030] Figure 7 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0031] Figure 8 A flowchart illustrating a text-image matching method provided in an embodiment of this application;

[0032] Figure 9 A schematic diagram of a student graphic information matrix provided in an embodiment of this application;

[0033] Figure 10 A schematic diagram of a teacher's graphic information matrix provided in an embodiment of this application;

[0034] Figure 11 A flowchart illustrating yet another image-text matching method provided in an embodiment of this application;

[0035] Figure 12 A schematic diagram of a softmax() function provided in an embodiment of this application;

[0036] Figure 13 A schematic diagram of a log_softmax() function provided in an embodiment of this application;

[0037] Figure 14 A schematic diagram illustrating the principle of a gradient descent method provided in an embodiment of this application;

[0038] Figure 15 A flowchart illustrating another image-text matching method provided in an embodiment of this application;

[0039] Figure 16 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0040] First, some concepts involved in this application will be described.

[0041] The terms "first" and "second" used in the embodiments of this application are only used to distinguish features of the same type and should not be construed as indicating relative importance, quantity, order, etc.

[0042] The terms "exemplary" or "for example" used in the embodiments of this application are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] The terms "coupling" and "connection" used in the embodiments of this application should be interpreted broadly. For example, they can refer to a physical direct connection or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0044] Text vectors: Text can be input into a text encoder to obtain text vectors, which are used to represent the semantic features of the entire text. The text encoder can adopt models such as transformers commonly used in natural language processing (NLP), and this application does not impose any restrictions.

[0045] Image vectors: Image frames from pictures or videos can be input into an image encoder to obtain visual vectors, which are used to represent the semantic features of the pictures or image frames. The image encoder can employ common convolutional neural network (CNN) models or vision transformer (VIT) models, etc., and this approach is not limited thereto. For ease of description, in this application, image frames from pictures and videos are collectively referred to as images, and pictures and videos are collectively referred to as visual media. Specifically, image frames from videos can be thumbnails of the video.

[0046] Training of the neural network model (specifically, the image-text model, hereinafter referred to as the model): Each training dataset of the neural network model includes the input of the model and its corresponding label. The label represents the true output value corresponding to the input data. The training process of the neural network model involves continuously updating its parameters, allowing the model to gradually fit the labels in each set of training data after receiving the input data. The goodness of fit is usually evaluated using a loss function, where the value of the loss function measures the difference between the output of the neural network model and the true output; a smaller loss function value indicates a better fit.

[0047] Knowledge distillation is a teacher-student training framework that employs transfer learning. It uses the output of a pre-trained, complex teacher model (in this application, a teacher-text / image model) as a supervisory signal to train a relatively simple student model (in this application, a student-text / image model). This effectively learns the student model from the teacher model, transferring knowledge from the complex teacher model to the simpler student model at a slight performance loss. The key to knowledge distillation lies in extracting rich knowledge from the teacher model and transferring that knowledge to guide the training of the student model. Knowledge distillation can be used for model compression and model augmentation. Model compression refers to using the teacher model to guide the training of the student model on the same labeled dataset, resulting in a simpler and more efficient student model. Model augmentation utilizes other resources (e.g., unlabeled, cross-domain, or cross-modal data) or optimization strategies (e.g., mutual learning and self-learning) to improve the accuracy of the student model. This application utilizes model enhancement to improve the performance of contrastive language-image pre-training (CLIP) models in image-text matching (ITM) applications (especially Chinese image-text matching).

[0048] Vision-language (VL) is a field formed at the intersection of computer vision (CV) and natural language processing (NLP), aiming to endow artificial intelligence (AI) systems with the ability to learn effective information from multimodal data (such as visual and linguistic modalities). Inspired by NLP pre-trained language models (such as bidirectional encoder representation from transformer (BERT) and generative pre-trained transformer (GPT), vision-language pre-training (VLP) has gradually gained attention and become a core training method for VL processing. This method first performs pre-training through self-supervised learning, using auxiliary tasks to automatically mine supervisory signals from large-scale unlabeled data to train the model, thereby obtaining a general-purpose trained model. Then, in specific applications, the trained model is fine-tuned using a small amount of manually labeled data, achieving excellent results.

[0049] As a fundamental visual language (VL) process, image-text matching (ITM) has gradually gained widespread attention. ITM primarily comprises two subtasks: image-to-text retrieval (TR) and text-to-image retrieval (IR). Image-to-text retrieval refers to searching for corresponding text based on an image (picture or video frame). For a video frame, it's equivalent to searching for corresponding text based on the video containing the image frame. Therefore, image-to-text retrieval can also refer to searching for corresponding text based on visual media (including images and videos). For example, searching for the text "puppy" based on a thumbnail of an image or video containing a puppy. Similarly, text-to-image retrieval refers to searching for corresponding images based on text. For a video frame, it's equivalent to searching for corresponding video containing the image frame based on the text. Therefore, text-to-image retrieval can also refer to searching for corresponding visual media (including images and videos) based on text. For example, searching for images or videos containing "puppy" based on the text "puppy". Image-text matching can bridge the semantic gap between visual and linguistic modalities, achieving accurate semantic alignment between multiple modalities, that is, establishing cross-modal semantic consistency between image and text modalities.

[0050] With the rise of Visual Language Processing (VLP), thanks to training on large-scale unlabeled data, trained models have significantly outperformed untrained models in image-text matching. A representative example is the CLIP model, which primarily consists of an image encoder and a text encoder. The CLIP model's image encoder is a visual transformer (ViT) or a residual network (ResNet), while the text encoder is the BERT mentioned earlier. The CLIP model's pre-training phase employs image-text contrastive learning, with optimization objectives including maximizing the cosine similarity score of image-text pairs and minimizing the score of mismatched image-text pairs. Through pre-training on massive amounts of data, it achieves excellent zero-shot learning results in multiple image-text matching processes. Zero-shot learning refers to training the model on a training set so that it can classify the test set, where there is no overlap between the training and test sets. In this application, an image-text pair refers to a data pair (in the form of <image, text>) consisting of an associated image and a text. For example, an image containing a puppy and the text "puppy" can form an image-text pair. For videos, image frames (e.g., thumbnails) are first extracted from the video as images in the image-text pair. Essentially, only the image frames of the video are used for training, not the entire video. Therefore, in this application, images represent the image frames of pictures or videos.

[0051] Applications of CLIP model in image-text matching, such as Figure 1 As shown. When CLIP model 11 is used for image-to-text search, the user inputs an image, and the image encoder 111 in the CLIP model extracts the image vector from the image, searches for the corresponding text vector in the image-to-text vector database, and thus returns the corresponding text to the user. When CLIP model is used for text-to-image search, the user inputs text into the CLIP model, and the text encoder 112 in the CLIP model extracts the text vector, searches for the corresponding image vector in the image-to-text vector database, and thus returns the corresponding image to the user.

[0052] The following is combined with Figure 2 This section explains the interface involved in text search for images in a mobile phone's gallery app:

[0053] like Figure 2 As shown in (a), the phone can display a main interface 201, which can also be called a desktop. The main interface 201 may include an icon 202 for the gallery application. The phone can receive a user's click on icon 202; in response to this action, the phone can launch the gallery application and display... Figure 2The interface 203 shown in (b) is an example. Interface 203 can be a photo album interface. It should be noted that, in response to the user clicking icon 202, the phone can launch the gallery application and display the gallery's image interface. The image interface includes thumbnails of images in the gallery or large images of a specific image. Within the image interface, in response to the user's interaction with the "Album" control, the aforementioned album interface 203 is displayed.

[0054] like Figure 2 As shown in (b), interface 203 includes multiple albums, including "All Pictures" containing 2023 pictures, "Camera" containing 1502 pictures and videos, "Screenshots and Screen Recordings" containing 102 pictures and videos, "My Favorites" containing 48 pictures and videos, "One Record, Multiple Views" containing 34 pictures and videos, "Video Editing" containing 65 videos, "Custom Albums" containing 57 pictures and videos, and "Shared Albums" containing 100 pictures and videos. Interface 203 may include a search box 204.

[0055] like Figure 2 As shown in (c), in response to the user entering the text "sky" in search box 204, the mobile phone can display interface 205, which can be referred to as the search interface. Interface 205 can display 100 visual media related to "sky," which can be images or videos (image frames in the video include sky elements) and 32 images or videos containing the text "sky." The 100 images related to "sky" can be searched because they are matched with the text "sky" using a text-image model (such as the CLIP model mentioned above). The 32 images related to the images containing the word "sky" can be searched because they are recognized as containing the word "sky" using optical character recognition (OCR) technology. In practical applications, the mobile phone can also perform predictive text generation based on the user's input, and then perform a search based on these predictive texts.

[0056] The following is combined with Figure 3 This document explains the interface involved in image-to-text search using a mobile phone's intelligent recognition application:

[0057] like Figure 3 As shown in (a), the mobile phone can display a search interface 301 on the negative one screen. The search interface 301 may include a smart recognition icon 302. The mobile phone can receive the user's action of clicking the smart recognition icon 302. In response to this action, the mobile phone can launch the smart recognition application, open the camera, and display as shown in (a). Figure 3 The shooting interface 303 shown in (b) is shown in the middle.

[0058] like Figure 3 As shown in (b), interface 303 includes a real-time captured image 304 (e.g., an image of a car), a camera button 305, and an album button 306. The phone can receive a user's click on the camera button 305. In response to this click, the intelligent recognition application searches for text corresponding to image 304 using a text-image model (e.g., the CLIP model described above). Alternatively, the phone can receive a user's click on the album button 306. In response to this click, the intelligent recognition application searches for text corresponding to images in the album using a text-image model.

[0059] like Figure 3 As shown in (c), the text 308 (“Car”) corresponding to image 304 is displayed in interface 307, and a “Search More” button 309 can also be displayed. The mobile phone can receive the user’s action of clicking the “Search More” button 309, and in response to this action, the mobile phone can call the search engine to search for more related content about “Car”.

[0060] Currently, miniaturized electronic devices (such as mobile phones, tablets, and laptops) have limited processing resources and run small-scale image-text models, resulting in very low accuracy in image-text matching processing (such as text-to-image search and image-to-text search).

[0061] Therefore, such as Figure 4 As shown, this application provides an image-text matching system for executing the image-text matching method of this application. The system mainly includes a server 21 and an electronic device 22. The server 21 executes steps S101-S104 to achieve high-quality data filtering, knowledge distillation and training of the student image-text model, and deploys the trained student image-text model on the electronic device 22. The electronic device 22 executes step S105 to perform image-text matching processing based on the student image-text model.

[0062] In S101, server 21 performs high-quality data filtering on a massive open-source image-text dataset, thereby selecting multiple high-quality image-text pairs. It should be noted that for images that are video frames, the video frames need to be extracted first for training. In S102, server 21 trains the student image-text model (which can be the CLIP model mentioned earlier) for these multiple image-text pairs. In S103, server 21 uses a complex teacher image-text model (which can be the CLIP model mentioned earlier) to perform knowledge distillation on the simple student image-text model. The distillation loss is used to update the parameters of the student image-text model, enabling the student model to learn the knowledge of the teacher model and thus improve the accuracy of image-text matching. In S104, server 21 deploys the trained student image-text model in the image-text matching application of the electronic device. In S105, the image-text matching application in electronic device 22 performs image-text matching processing based on this student image-text model.

[0063] like Figure 5 As shown, server 21 may include at least one processor 211, communication line 212, memory 213, and at least one communication interface 214. Communication line 212 may include a path for transmitting information between the aforementioned components. Communication interface 214 uses any transceiver-like device for communicating with other devices. Memory 213 stores instructions that, when executed by processor 211, perform the image-text matching method according to embodiments of this application.

[0064] The processor involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), application specific integrated circuit (ASIC), system on chip (SoC), central processor (CPU), network processor (NP), digital signal processor (DSP), micro controller unit (MCU), programmable logic device (PLD), or other integrated chip.

[0065] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0066] Electronic devices are devices with displays. They can be mobile or fixed. Electronic devices can be deployed on land (e.g., indoors or outdoors, handheld or vehicle-mounted), on water (e.g., ships), or in the air (e.g., airplanes, balloons, and satellites). These electronic devices can be referred to as user equipment (UE), access terminals, terminal units, subscriber units, terminal stations, mobile stations (MS), mobile stations, terminal agents, or terminal devices. For example, they can be mobile phones, tablets, laptops, smart bracelets, smart screens, smartwatches, virtual reality (VR) devices, augmented reality (AR) devices, terminals in industrial control, self-driving, remote medical care, smart grids, transportation safety, smart cities, and smart homes. This application does not limit the specific type and structure of the electronic device. The following describes one possible structure of the electronic device.

[0067] Taking mobile phones as an example, Figure 6 A possible structure of an electronic device 22 is shown. This electronic device 22 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a power management module 240, a battery 241, a wireless charging coil 242, antenna 1, antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Optionally, in some embodiments, it may also include an audio digital signal processor (ADSP) 243.

[0068] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 22. In other embodiments of this application, the electronic device 22 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0069] Processor 210 may include one or more processing units, such as: a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), an application processor (AP), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a baseband processor, and a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. For example, processor 210 may be an application processor (AP). Alternatively, processor 210 may be integrated into a system-on-chip (SoC). Or, processor 210 may be integrated into an integrated circuit (IC) chip. The processor 210 may include an analog front end (AFE) and a micro-controller unit (MCU) in an IC chip.

[0070] The processor 210 executes the display control method provided in this application embodiment by executing the program and computer instructions stored in the internal memory 221.

[0071] The processor 210 may also include a memory for storing computer instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store computer instructions or data that the processor 210 has just used or that are being used repeatedly. If the processor 210 needs to use the same computer instructions or data again, it can retrieve them directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves system efficiency.

[0072] In some embodiments, the processor 210 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, etc.

[0073] The ADSP 243 can be coupled to the audio module 270 and the sensor module 280. The ADSP 243 can process audio signals and sensor data. Even when the processor is in sleep mode, the ADSP 243 can remain operational, thereby reducing the power consumption of the electronic device.

[0074] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 22. In other embodiments of this application, the electronic device 22 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0075] The wireless communication function of electronic device 22 can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.

[0076] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 22 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0077] Mobile communication module 250 can provide wireless communication solutions including 2G / 3G / 4G / 5G for use on electronic device 22. Wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) for use on electronic device 22. In some embodiments, antenna 1 of electronic device 22 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling electronic device 22 to communicate with networks and other devices via wireless communication technology.

[0078] The external storage interface 220 can be used to connect an external memory card, such as a micro SanDisk (Micro SD) card, to expand the storage capacity of the electronic device 22. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0079] Internal memory 221 can be used to store computer executable program code, which includes computer instructions. Processor 210 executes various functional applications and data processing of electronic device 22 by running the computer instructions stored in internal memory 221. In addition, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0080] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0081] Electronic device 22 can implement audio functions such as music playback and recording through audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, and application processor.

[0082] Audio module 270 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. In some embodiments, audio module 270 may be located in processor 210, or some functional modules of audio module 270 may be located in processor 210. Speaker 270A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Receiver 270B, also called a "handpiece," is used to convert audio electrical signals into sound signals. Microphone 270C, also called a "microphone" or "microphone," is used to convert sound signals into electrical signals. Electronic device 22 may be equipped with at least one microphone 270C. Headphone jack 270D is used to connect wired headphones. Headphone jack 270D may be a USB interface 230, or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0083] Buttons 290 include a power button, volume buttons, etc. Buttons 290 can be mechanical buttons or touch buttons. The electronic device 22 can receive button inputs and generate key signal inputs related to user settings and function control of the electronic device 22. Motor 291 can generate vibration alerts. Motor 291 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 292 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to achieve contact and separation with the electronic device 22. The electronic device 22 can support one or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. In some embodiments, the electronic device 22 employs an embedded SIM (eSIM) card, which can be embedded in the electronic device 22 and cannot be separated from the electronic device 22.

[0084] The electronic device 22 can implement its shooting function through an ISP, a camera 293, a video codec, a GPU, a display 294, and an application processor. The ISP is used to process data fed back from the camera 293. In some embodiments, the ISP can be located within the camera 293. The camera 293 is used to capture still images or videos. In some embodiments, the electronic device 22 may include one or N cameras 293, where N is a positive integer greater than 1.

[0085] Electronic device 22 can implement display functions through a GPU, display screen 294, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute computer instructions to generate or modify display information.

[0086] The sensor module 280 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, an angle sensor, etc. When the display screen 294 is a foldable screen, the angle sensor can detect the folding angle of the display screen 294, which ranges from 0 to 180 degrees.

[0087] Battery 241 may include one or more batteries to power a load.

[0088] The power management module 240 is used to receive charging input from a charger. The charger can be a wireless charger, such as a wireless charging dock, or other electronic devices 22 with reverse wireless charging capabilities. The power management module 240 can receive wireless charging input via the wireless charging coil 242 of the electronic device. The charger can also be a wired charger; for example, the power management module 240 can receive charging input from a wired charger via a USB interface 230. The power management module 240 is also referred to as a charging chip.

[0089] The power management module 240 charges the battery 241 while simultaneously supplying power to the electronic devices. It receives input from the battery 241 and powers the processor 210, internal memory 221, external memory interface 220, display screen 294, camera 293, and wireless communication module 260. The power management module 240 can also monitor parameters such as the battery 241's capacity, voltage, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 240 may also be integrated into the processor 210.

[0090] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. In some embodiments, the electronic device 22 may include one or more display screens 294.

[0091] Figure 7This is a software structure block diagram of the electronic device 22 according to an embodiment of this application. The software system of the electronic device 22 can adopt a layered architecture, which divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. Taking the Android system as an example, in some embodiments, the Android system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0092] like Figure 7 As shown, the application layer can include applications such as image-to-text search applications and text-to-image search applications, which can use the image-to-text model described above to perform image-to-text matching processing.

[0093] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0094] The system library can include multiple functional modules. For example, it may include a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL for embedded systems, OpenGL ES), and 2D graphics engines (e.g., Skia graphics library, SGL). The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries can support various video and image encoding formats, such as Moving Pictures Experts Group 4 (MPEG4), Joint Photographic Experts Group (JPG), and Portable Network Graphics (PNG).

[0095] The kernel layer is the layer between hardware and software.

[0096] First of all Figure 4 The following sections provide a detailed explanation of S101 "High-Quality Data Screening" and S102 "Student Graphical Model Training":

[0097] Taking the student image-text pattern as an example, the accuracy of the CLIP model trained with datasets of different quality varies greatly when used for Chinese image-text matching. This is because the quality of open-source Chinese datasets varies considerably, and it is necessary to select a high-quality Chinese dataset for training.

[0098] To select high-quality datasets through experiments, we used two representative Chinese datasets from the open-source community—the Taisu dataset and the Wukong dataset—as examples. We randomly selected 11M image-text pairs from each dataset as training sets to train the CLIP model. Specifically, the CLIP model used Vit for the image encoder and BERT for the text encoder; all experimental conditions were identical except for the training dataset. In the specific experiments, the industry-standard Flickr30K-cn test set was used as the evaluation dataset. The evaluation metrics were text-to-image R1 and image-to-text R1. Recall rate, also known as the search coverage rate, is the ratio of the number of relevant targets found to the total number of actually relevant targets. A higher recall rate indicates a more comprehensive search. As shown in Table 1, the CLIP model trained using the Taisu dataset has better text search image recall and image search text recall than the CLIP model trained using the Wukong dataset. This means that the Taisu dataset is of higher quality than the Wukong dataset, so the Taisu dataset can be used to train the CLIP model.

[0099] Table 1

[0100] training set Image search recall rate Image search article recall rate Wukong Dataset - 11M 65.24 77.7 Taisu Dataset - 11M 68.68 83.2

[0101] The following is about Figure 4 The S103 section, "Knowledge Distillation," will be explained in detail. For example... Figure 8 As shown, S103 specifically includes:

[0102] S201. Input the images from multiple image-text pairs into the student image encoder in the student image-text model to obtain the student image vector; input the images from multiple image-text pairs into the teacher image encoder in the teacher image-text model to obtain the teacher image vector; input the text from multiple image-text pairs into the student text encoder in the student image-text model to obtain the student text vector; input the text from multiple image-text pairs into the teacher text encoder in the teacher image-text model to obtain the teacher text vector.

[0103] The teacher's text-image model can be an open-source Chinese pre-trained CLIP model based on ViT-Huge, and the student's text-image model can be an open-source Chinese pre-trained CLIP model based on ViT-Base. The teacher's text-image model has stronger text-image learning capabilities. Therefore, this application transfers the learning capabilities of the teacher's text-image model to the student's text-image model through text-image information fusion and knowledge distillation, thereby enhancing the student's learning capabilities. For example, as shown in Table 2, the teacher's text-image model outperforms the student's text-image model in at least one of the following aspects: number of model layers, number of hidden neurons, multilayer perceptron (MLP) size, and number of heads. Here, the number of model layers refers to the number of layers in the neural network model. The number of hidden neurons refers to the number of hidden neurons in the MLP. The MLP is the core of the model; the input of each neuron in the MLP is x, and the output is y = h. w,b (x) is used to fit the input-output relationship reflected by the existing training data (x,y) by adjusting the parameters weight (w) and bias (b). The MLP size represents the model size; the larger the MLP size, the larger the model size. Multi-head attention refers to multi-head attention, which helps the model learn the influence of different contexts on the search results. The more heads, the better it is to capture a wider range of features.

[0104] Table 2

[0105]

[0106] Assuming we use IV to represent an image vector and TV to represent a text vector, inputting the images from N image-text pairs into a student image encoder yields the student image vector IV. student for:

[0107] IV student ={IV student_1 IV student_2 ,...,IV student_N |IV student_i ∈R 1*J}

[0108] Input the text from N image-text pairs into the student text encoder to obtain the student text vector TV. student for:

[0109] TV student ={TV student_1 TV student_2 TV student_N TV student_i ∈R 1*J}

[0110] Input the images in N image-text pairs into the teacher image encoder to obtain the teacher image vector IV teacher It is:

[0111] IV teacher ={IV teacher_1 ,IV teacher_2 ,...,IV teacher_N |IV teacher_i ∈R 1*K}

[0112] [[ID=2l]]Input the texts in N image-text pairs into the teacher text encoder to obtain the teacher text vector TV teacher It is:

[0113] TV teacher ={TV teacher_1 ,TV teacher_2 ,...,TV teacher_N |TV teacher_i ∈R 1*K}

[0114] Where N represents the number of image-text pairs, and J < K. Exemplarily, J = 512 and K = 1024. The subscripts 1, 2,..., N in each vector represent the nth image-text pair. For example, IV student_1 represents the student image vector corresponding to the image in the 1st image-text pair, TV student_1 represents the student text vector corresponding to the text in the 1st image-text pair, IV teacher_1 represents the teacher image vector corresponding to the image in the 1st image-text pair, TV teacher_1 represents the teacher text vector corresponding to the text in the 1st image-text pair, and so on. R 1*J represents a 1*J-dimensional real vector, and R 1*K represents a 1*K-dimensional real vector.

[0115] S202. Fuse the student image vector and the student text vector to obtain the student image-text information; fuse the teacher image vector and the teacher text vector to obtain the teacher image-text information.

[0116] The cosine similarity between the student image vector and the student text vector can be used to fuse and obtain the student image-text information. Similarly, the cosine similarity between the teacher image vector and the teacher text vector can be used to fuse and obtain the teacher image-text information. The teacher image-text information comes from a more complex teacher image-text model and has higher-level image-text semantic information compared to the student image-text information from a simple student image-text model. In addition, the fused image-text information also has higher-level image-text semantic information compared to pure image information or text information, and thus has higher knowledge expression ability when used for knowledge distillation. [[ID=GO]] [[ID=Gl]]

[0117] Specifically, the student image vector and the student text vector are multiplied by matrix to obtain the student image-text information matrix S. V :

[0118] S V =L×IV student @TV student

[0119] Where L is a coefficient used to transform the student image and text information matrix S V The numerical values ​​are within a reasonable range; "×" indicates numerical product, and "@" indicates matrix product. Student graphic information matrix S V It is an N*N information fusion matrix, in the form of: Figure 9 As shown.

[0120] The teacher's image vector and teacher's text vector are multiplied by a matrix to obtain the teacher's image-text information matrix T. V :

[0121] T V =L×IV teacher @TV teacher

[0122] Where L is a coefficient used to integrate the teacher's text and image information matrix T V The numerical values ​​are within a reasonable range; "×" represents numerical product, and "@" represents matrix product. Teacher's graphic information matrix T V It is an N*N information fusion matrix, in the form of: Figure 10 As shown.

[0123] S203. Use teacher's text and image information to perform knowledge distillation on student's text and image information, and update the parameters of student's text and image model based on the distillation loss after knowledge distillation.

[0124] like Figure 11 As shown, step S203 includes S2031-S2034:

[0125] S2031, Based on the student graphic information matrix S V Teacher's graphic information matrix T V Obtaining image distillation loss Distill image and text distillation loss text .

[0126] Image distillation loss image The text distillation loss is used to represent the loss between the student image vector and the teacher image vector. text This represents the loss between the student's text vector and the teacher's text vector. Specifically, the image distillation loss (Distill) can be obtained using the following two formulas. imageand text distillation loss text .

[0127]

[0128]

[0129] Where, matrix S′ V S represents the student's text and image information matrix V The transpose of matrix T′ V Represents the teacher's text and image information matrix T V The transpose of , DistillLossFunction() represents the distillation loss function, softmax() represents the exponential normalization function, and log_softmax() represents taking the log of the result of softmax(). V (i) represents the student graphic information matrix S V The i-th row of features, T V (i) represents the teacher's graphic information matrix T V The i-th row of features, S′ V (i) represents matrix S′ V The i-th row of features, T′ V (i) represents matrix T′ V The i-th row of features.

[0130] like Figure 12 As shown, the softmax() function has a normalization characteristic; larger values ​​result in a value closer to 1 after normalization, while smaller values ​​result in a value closer to 0. Using the softmax() function to fuse the teacher image vector and teacher text vector in the teacher image-text model can significantly strengthen the knowledge on the diagonal of the teacher image-text information matrix, enhancing the distillation value of the knowledge. This is because the values ​​on the diagonal of the teacher image-text information matrix are much larger than the values ​​off-diagonally, meaning the knowledge on the diagonal of the teacher image-text information matrix is ​​more abundant and of higher value.

[0131] like Figure 13 As shown, the log_softmax() function can map values ​​with large differences obtained by the softmax() function to the same level, effectively avoiding the overflow problem caused by the excessively large values ​​of certain features in the student's text-image information matrix, and ensuring the stability of the knowledge received by the student's text-image model.

[0132] S2032, Distillation loss based on image image and text distillation loss text Distillation loss distill .

[0133] Distillation Loss distill This represents the loss of the student's image-text model relative to the teacher's image-text model after knowledge distillation. Specifically, the image distillation loss can be expressed by the following formula: image and text distillation loss text The weighted summation yields the distillation loss. distill .

[0134] Loss distill =θ×Distill image +μ×Distill text

[0135] Where θ and μ are coefficients, θ+μ=1, for example, θ=0.5, μ=0.5.

[0136] S2033, based on distillation loss distill and training loss train Total Loss total .

[0137] Training loss train This is used to represent the loss between the model output and the true value after training a student image-text model using image-text pairs. Specifically, the distillation loss can be calculated using the following formula: distill and training loss train The total loss is obtained by weighted summation. total .

[0138] Loss total =α×Loss train +β×Loss distill

[0139] Where α and β are coefficients, α+β=1, for example, α=0.5, β=0.5.

[0140] S2034, Based on the total loss total Update the parameters of the student graphic model.

[0141] Existing technologies use training loss as the basis. train This is used to update the parameters of the student image-text model. In this application, loss is achieved through distillation. distill Training loss train Compensation is made to obtain the total loss. total According to the total loss totalUpdating the parameters of the student graph-text model can be done by updating the parameters of the student graph-text model with knowledge from the teacher graph-text model. For example, gradient descent can be used to iteratively update the parameters of the student graph-text model until the total loss is reached. total The training process converges (ceases or slows down) to its minimum value and reaches the preset number of iterations, thus completing the final training. The parameters of the student image-text model are the weights and biases of the MLP mentioned earlier.

[0142] The principle of gradient descent is as follows: Figure 14 As shown, the loss of the model is a bowl-shaped curve with the model parameters as variables. Starting from point A on the curve, the gradient (partial derivative vector) at point A is calculated. The direction of the negative gradient is the direction that reduces the loss, thus determining the next point B on the curve. Then, starting from point B, points C and D on the curve are determined in the same way. The gradient at point D is approximately 0, which indicates that the convergence is at point D. The parameters at this point are the parameters after the model has converged.

[0143] The following is about Figure 4 The S105 "Image and text matching processing" section will be explained in detail.

[0144] Image-text matching processing includes, but is not limited to, the text-to-image and image-to-text matching methods mentioned above, and can also include more advanced image-text matching methods that lean towards artificial intelligence, such as text-to-image and image-to-text matching. Text-to-image matching refers to inputting a complex text into a student image-text model, which then generates an image. Image-to-text matching refers to inputting an image into a student image-text model, which then generates a complex text.

[0145] like Figure 15 As shown, step S105 includes:

[0146] S301, Electronic device display interface.

[0147] The interface includes controls for retrieving search targets. Search targets can be... Figure 2 The text shown or Figure 3 The visual media shown (including images and videos). Controls used to retrieve search targets include, but are not limited to, those shown. Figure 2 The search box in the middle is 204. Figure 3 The camera button in the image is 305, etc.

[0148] S302. The electronic device responds to the operation of the control, matches the search target according to the student's graphic model, and obtains the search results.

[0149] When the search target is text, the search results are visual media (including images and videos); when the search target is visual media, the search results are text. For example, operations on the control can be... Figure 2The user's action of entering text in search box 204 is shown, or it could be... Figure 3 The image shows a user's click on the camera button 305. Search results can be... Figure 2 The visual media shown in (c) can be, or can be Figure 3 The text shown in (c) is as follows.

[0150] S303, Electronic devices display search results.

[0151] For example, electronic devices can display search results, which can be... Figure 2 As shown in (c), visual media is displayed in the search interface 205, or, alternatively, it can be... Figure 3 The text shown in (c) is displayed in interface 307.

[0152] The image-text matching method provided in this application involves inputting image-text pairs into a student image-text model to obtain student image vectors and student text vectors, and inputting identical image-text pairs into a teacher image-text model to obtain teacher image vectors and teacher text vectors. The student image vectors and student text vectors are fused to obtain student image-text information; the teacher image vectors and teacher text vectors are fused to obtain teacher image-text information. On one hand, the teacher image-text information comes from a more complex teacher image-text model, possessing higher-level image-text semantic information compared to student image-text information from a simpler student image-text model. On the other hand, the fused image-text information, compared to simple image or text information, possesses higher-level image-text semantic information, resulting in higher knowledge representation capabilities when used for knowledge distillation. Then, the teacher image-text information is used to perform knowledge distillation on the student image-text information. The parameters of the student image-text model are updated based on the distillation loss, and the updated student image-text model is used for image-text matching. In this way, the updated student image-text model learns the knowledge of the complex teacher image-text model, improving the accuracy of the neural network model in image-text matching.

[0153] like Figure 16 As shown, this application embodiment also provides a chip system. The chip system 160 includes at least one processor 1601 and at least one interface circuit 1602. The at least one processor 1601 and at least one interface circuit 1602 are interconnected via lines. The processor 1601 is used to support electronic devices in implementing the various steps in the above method embodiments, for example... Figure 4 , Figure 8 , Figure 11 , Figure 15 The method shown allows at least one interface circuit 1602 to be used to receive signals from other devices (e.g., memory) or to send signals to other devices (e.g., a communication interface). The chip system may include a chip and may also include other discrete components.

[0154] This application also provides a computer-readable storage medium including instructions that, when executed on the aforementioned electronic device or server, cause the electronic device or server to perform the steps described in the method embodiments, such as executing... Figure 4 , Figure 8 , Figure 11 , Figure 15 The method shown.

[0155] This application also provides a computer program product including instructions, which, when executed on the aforementioned electronic device or server, cause the electronic device or server to perform the various steps in the above method embodiments, such as executing... Figure 4 , Figure 8 , Figure 11 , Figure 15 The method shown.

[0156] The technical effects of the chip system, computer-readable storage medium, and computer program product are described in the preceding method embodiments.

[0157] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0158] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0161] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.

[0163] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs), etc.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for matching images and text, characterized in that, The method includes: The display interface includes controls for obtaining the search target; In response to an operation on the control, the search target is matched according to the student image-text model to obtain a search result; the search target is text and the search result is visual media, or the search target is visual media and the search result is text. The search results are displayed; The training process of the student graphic model includes: The images from multiple image-text pairs are input into the student image encoder in the student image-text model to obtain student image vectors; the images are input into the teacher image encoder in the teacher image-text model to obtain teacher image vectors; the text from the multiple image-text pairs is input into the student text encoder in the student image-text model to obtain student text vectors; the text is input into the teacher text encoder in the teacher image-text model to obtain teacher text vectors. The student image vector and the student text vector are fused to obtain student image-text information; the teacher image vector and the teacher text vector are fused to obtain teacher image-text information. The teacher's text and image information is used to perform knowledge distillation on the student's text and image information, and the parameters of the student text and image model are updated based on the distillation loss after knowledge distillation.

2. The method according to claim 1, characterized in that, The process of fusing the student image vector and the student text vector to obtain student image-text information; and fusing the teacher image vector and the teacher text vector to obtain teacher image-text information, includes: The student image vector and the student text vector are multiplied by a matrix to obtain the student image and text information matrix; The teacher's image vector and the teacher's text vector are multiplied by matrix to obtain the teacher's image-text information matrix.

3. The method according to claim 2, characterized in that, The step of using the teacher's text-image information to perform knowledge distillation on the student's text-image information, and updating the parameters of the student text-image model based on the distillation loss after knowledge distillation, includes: Image distillation loss and text distillation loss are obtained based on the student image-text information matrix and the teacher image-text information matrix; the image distillation loss is used to represent the loss of the student image vector relative to the teacher image vector, and the text distillation loss is used to represent the loss of the student text vector relative to the teacher text vector; The distillation loss is obtained based on the image distillation loss and the text distillation loss; The total loss is obtained based on the distillation loss and the training loss; the training loss is used to represent the difference between the model output and the true value after training the student image-text model using the image-text pairs. The parameters of the student graph-text model are updated based on the total loss.

4. The method according to claim 3, characterized in that, The step of obtaining image distillation loss and text distillation loss based on the student image-text information matrix and the teacher image-text information matrix includes: The image distillation loss is obtained using the following two formulas. and text distillation loss : Among them, matrix Representing the student's graphic information matrix transpose of matrix Representing the teacher's text and image information matrix transpose, Represents the exponential normalization function, Indicates to Take the logarithm of the result. Representing the student's graphic information matrix The i-th row of features, Representing the teacher's text and image information matrix The i-th row of features, Representation matrix The i-th row of features, Representation matrix The i-th row of features in the diagram, where N represents the number of the image-text pairs.

5. The method according to claim 3, characterized in that, The process of obtaining the distillation loss based on the image distillation loss and the text distillation loss includes: The distillation loss is obtained by weighted summation of the image distillation loss and the text distillation loss.

6. The method according to claim 3, characterized in that, The process of obtaining the total loss based on the distillation loss and training loss includes: The total loss is obtained by weighted summation of the distillation loss and the training loss.

7. The method according to claim 3, characterized in that, The step of updating the parameters of the student image-text model based on the total loss includes: The parameters of the student image-text model are iteratively updated using the gradient descent method until the total loss converges to the minimum and the preset number of iterations is reached.

8. The method according to any one of claims 1-7, characterized in that, The teacher-text model is superior to the student-text model in at least one of the following aspects: number of model layers, number of hidden neurons, size of multilayer perceptron, and number of heads.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein instructions are stored in the memory, and when the processor executes the instructions, the method as described in any one of claims 1-8 is performed.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-8.

Images