Visual media search method and electronic device

Abstract

The application provides a visual media search method and an electronic device, and relates to the technical field of image processing. After receiving a search statement input by a user, the electronic device determines a text feature vector of the search statement. Then, the electronic device inputs the text feature vector of the search statement and an image feature vector of a visual media stored locally by the electronic device into a text-image matching model, so as to determine whether the search statement matches the visual media by using the text-image matching model, thereby realizing the search of the visual media. The text-image matching model is trained by using positive and negative samples, the positive and negative samples are determined by distinguishing sample text elements corresponding to sample images, the sample text elements corresponding to the sample images include one text element in a text set corresponding to each sample image, the text set corresponding to the sample images represents a set of description contents corresponding to the sample images, the training samples of the text-image matching model are determined quickly, and therefore the training efficiency of the text-image matching model can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a visual media search method and electronic device. Background Technology

[0002] With the development of electronic devices (such as mobile phones), the camera function of mobile phones has also developed rapidly. More and more users are using their mobile phones to take photos and videos and store them in their phone's gallery. In addition, users can also store downloaded pictures and screenshots in their phone's gallery.

[0003] When users want to search for visual media (such as photos and videos), they can enter a search query on their mobile phone, such as "photos taken on September 1st." The phone will respond to the user's search query, retrieve photos taken on September 1st, and obtain the corresponding search results. However, mobile phones have limited ability to understand search queries, and when the search query is complex, the phone may not be able to accurately obtain the corresponding search results. Summary of the Invention

[0004] In view of this, this application provides a visual media search method and electronic device to improve the accuracy of search results.

[0005] In a first aspect, this application provides a visual media search method applied to an electronic device, which can acquire P sample data pairs, the P sample data pairs including P sample images and descriptive text corresponding to the P sample images.

[0006] Next, for each of the P sample images, a text element is selected from the text set corresponding to that sample image to obtain the first text element (or text element 1) corresponding to that sample image. The text set corresponding to the sample image may include the descriptive text corresponding to that sample image and each subtext (or first subtext) within that descriptive text. The first text element may be either the descriptive text corresponding to the sample image or a subtext within the descriptive text.

[0007] Then, the electronic device can perform image-text matching on P sample images and the first text elements corresponding to the P sample images, that is, match any sample image with any first text element corresponding to any sample image to obtain a P*P sample matrix. Here, the elements in the i-th row of the sample matrix represent the sample text elements corresponding to the i-th sample image; any sample text element in the sample matrix and the sample image corresponding to the row containing that sample text element form a sample.

[0008] Afterwards, the electronic device can train a text-image matching model based on the sample matrix and sample images.

[0009] The electronic device can display a first interface; the first interface includes a search box;

[0010] Receives the search query entered in the search box; the search query includes one or more subtexts;

[0011] The search results are displayed, and the search results correspond to the first visual media; the first visual media refers to the visual media that matches the search query with the visual content determined by the image-text matching model.

[0012] In this application, the electronic device can acquire P pairs of sample data, where each pair represents a positive sample. The electronic device can then use these P pairs to generate a P*P sample matrix, resulting in P*P samples. This incremental increase in the number of samples allows for rapid generation of training samples, ensuring the training effectiveness of the image-text matching model. Subsequently, the electronic device can utilize the image-text matching model to search for visual media that match the user's input search query. This ensures the accuracy of visual media searches regardless of the complexity of the search query, meeting the user's search needs. Furthermore, in the context of image-text search, considering that the user's input search query conforms to natural language expression habits, it may contain multiple pieces of information (i.e., second sub-text), and this second sub-text is not a simple combination. Therefore, when training the image-text matching model, the sample images can also contain content describing the sample images, forming a text set corresponding to the sample images. This text set can include descriptions of the overall visual content of the sample images or descriptions of parts of the visual content of the sample images. This text set can be used to train the image-text matching model, so that the matching between the visual media and the search statement can be achieved regardless of whether the user's search statement is the overall visual content of the corresponding visual media or a part of the corresponding visual content.

[0013] In one possible design approach, the process of determining the sample matrix of P*P described above may include:

[0014] For each sample image, the first text element corresponding to the P sample images is taken as the sample text element corresponding to the sample image.

[0015] A sample matrix is ​​generated based on the sample text elements corresponding to each sample image. The elements in the i-th row of the sample matrix represent the sample text elements corresponding to the i-th sample image. Any sample text element in the sample matrix and the sample image corresponding to the row containing that sample text element form a sample. Based on this, matching P sample images with P first text elements is achieved, resulting in multiple samples and ensuring rapid sample generation.

[0016] In one possible design approach, the process of training the image-text matching model based on the sample matrix and the sample images may include:

[0017] Electronic devices can distinguish whether a sample text element in a sample matrix is ​​a positive or negative sample based on the sample matrix and the text set corresponding to the sample image.

[0018] Then, the electronic device can use the sample text elements and sample images in the sample matrix to obtain positive and negative samples. Based on these positive and negative samples, the electronic device can then train an image-text matching model to distinguish between the positive and negative samples.

[0019] In one possible design approach, the process of determining the positive and negative samples mentioned above may include:

[0020] For each sample text element in the sample matrix, the electronic device can determine whether the sample text element belongs to the text set corresponding to the sample image in the row where the sample text element is located, so as to determine whether the sample text element matches the sample image.

[0021] If a sample text element belongs to the text set corresponding to the sample image corresponding to the row where the sample text element is located, it indicates that the sample element matches the sample image, and the electronic device determines that the sample text element and the sample image corresponding to the row where the sample text element is located are positive samples.

[0022] If a sample text element does not belong to the text set corresponding to the sample image in the row where the sample text element is located, it indicates that the sample element matches the sample image. The electronic device can determine that the sample text element and the sample image in the row where the sample text element is located are negative samples, thus achieving the distinction between positive and negative samples.

[0023] In another possible design approach, the process of determining the positive and negative samples mentioned above may include:

[0024] The intersection of the first text intersection matrix (or text intersection matrix 1) and the sample matrix is ​​used to obtain the second text intersection matrix (or text intersection matrix 2); the first text intersection matrix is ​​determined based on the common elements in the text sets corresponding to any two sample images.

[0025] If the s-th intersection element in the t-th row of the second text intersection matrix is ​​empty, then the s-th intersection element in the t-th row of the sample matrix and the t-th sample image are determined to be negative samples.

[0026] If the s-th intersection element in the t-th row of the second text intersection matrix is ​​not empty, then the s-th intersection element in the t-th row of the sample matrix and the t-th sample image are determined to be positive samples. Based on this, by finding the intersection of the sample matrix and the first text intersection matrix, it is possible to quickly determine whether the text elements in the sample matrix belong to positive samples, thus improving the efficiency of determining positive and negative samples.

[0027] In one possible design approach, for each sample image, the corresponding descriptive text is segmented to obtain a first sub-text within the descriptive text; wherein the first sub-text includes nouns and / or phrases from the descriptive text. These phrases can be descriptive phrases, thereby determining the set of descriptive content corresponding to the sample image.

[0028] In one possible design approach, after obtaining the text sets corresponding to the sample images, the electronic device determines the union of the text sets corresponding to each sample image. Then, the electronic device assigns a number to each text element in the union of the text sets, such that different text elements in the text sets have different numbers, while the same text element has the same number.

[0029] Correspondingly, electronic devices can perform image-text matching on P sample images and the corresponding numbers of the first text elements of the P sample images to obtain a P*P sample matrix, thereby improving the efficiency of sample matrix generation.

[0030] In one possible design approach, the electronic device can randomly select text elements from the text set corresponding to each sample image according to a preset ratio; wherein the preset ratio includes the ratio of the selected first text element to the descriptive text and the ratio of the selected first text element to the first sub-text.

[0031] In this Chinese application, since the image encoder and text encoder are generally trained using the image and its corresponding text as a whole, the proportion of selected text elements that are descriptive text in the preset ratio will be greater than the proportion of selected text elements that are sub-text, so as to ensure the training effect of the model.

[0032] In one possible design approach, the first visual media refers to candidate visual media whose visual content matches the search query, as determined by the image-text matching model. Candidate visual media are those whose similarity to the search query is greater than a first threshold.

[0033] In this application, the image-text matching model can be used for secondary confirmation of candidate visual media, that is, to further determine whether the candidate visual media initially retrieved by the electronic device matches the search statement.

[0034] In one possible design approach, after obtaining the search query, the electronic device can filter the search query to obtain a filtered search query by filtering out non-visual semantic subjects within the search query. Then, the electronic device can utilize a text-image matching model to determine the matching value (e.g., a Boolean value) corresponding to the visual media based on the image information of the visual media and the information from the filtered search query.

[0035] The first visual medium is determined based on the matching value corresponding to the visual medium. Furthermore, the electronic device can determine a second visual medium (or visual medium 2) that matches the non-visual semantic subject in the search query.

[0036] Then, the electronic device can use the intersection between the first visual media and the second visual media as the target visual media to accurately determine the search results.

[0037] The matching value corresponding to the aforementioned visual media can include a Boolean value (or Boolean value 1). If the Boolean value corresponding to the visual media is true, then the visual media is determined to be the first visual media.

[0038] If the Boolean value corresponding to the visual medium is false, it is determined that the visual medium is not the primary visual medium.

[0039] In one possible design approach, the first interface is displayed in response to the operation of opening the gallery application;

[0040] Alternatively, in response to an operation to open the negative one screen triggered by the main screen of the electronic device, the first interface is displayed;

[0041] Alternatively, in response to a pull-down search operation triggered on the main screen of the electronic device, the first interface is displayed.

[0042] In one possible design approach, the electronic device can input the image feature vectors of each of the visual media in the electronic device and the text feature vectors of the search statement into the image-text matching model to obtain the matching results corresponding to each of the visual media. The image-text matching model is used to obtain the matching degree between the visual media and the search statement based on the image feature vectors of the visual media and the text feature vectors of the search statement, and compares the matching degree between the visual media and the search statement with a preset classification threshold to obtain the matching result corresponding to the visual media.

[0043] In a second aspect, this application provides an electronic device, the electronic device including a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the method described above.

[0044] Thirdly, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described above.

[0045] Fourthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described above.

[0046] It is understood that the beneficial effects achieved by the electronic device described in the second aspect, the computer storage medium described in the third aspect, and the computer program product described in the fourth aspect can be referred to the beneficial effects in the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0047] Figure 1A A schematic diagram of a visual media search interface provided in an embodiment of this application;

[0048] Figure 1B A second schematic diagram of a visual media search interface provided in an embodiment of this application;

[0049] Figure 1C Schematic diagram three of a visual media search interface provided for an embodiment of this application;

[0050] Figure 1D A schematic diagram of a visual media search interface provided in an embodiment of this application. Figure 4 ;

[0051] Figure 2A A structural block diagram of an electronic device provided in an embodiment of this application;

[0052] Figure 2B A software structure diagram of an electronic device provided in an embodiment of this application;

[0053] Figure 3A Schematic diagram five of a visual media search interface provided for an embodiment of this application;

[0054] Figure 3B A schematic diagram of a visual media search interface provided in an embodiment of this application. Figure 6 ;

[0055] Figure 4 A schematic diagram of a visual media search method provided in an embodiment of this application;

[0056] Figure 5A A schematic diagram illustrating the process of determining an image feature vector as provided in an embodiment of this application;

[0057] Figure 5B Schematic diagram 2 illustrating the process of determining an image feature vector as provided in an embodiment of this application;

[0058] Figure 5C A schematic diagram illustrating the process of determining a text feature vector as provided in an embodiment of this application;

[0059] Figure 5D A schematic diagram illustrating a similarity determination process provided in an embodiment of this application;

[0060] Figure 6 Schematic diagram 2 of a visual media search method provided in an embodiment of this application;

[0061] Figure 7 Schematic diagram three illustrating a visual media search method provided in an embodiment of this application;

[0062] Figure 8 A schematic diagram of a visual media search method provided in this application embodiment. Figure 4 ;

[0063] Figure 9A A schematic diagram of a secondary confirmation provided for an embodiment of this application;

[0064] Figure 9B A schematic diagram of a visual media provided for an embodiment of this application;

[0065] Figure 10 Schematic diagram five illustrating a visual media search method provided in an embodiment of this application;

[0066] Figure 11A A schematic diagram of a matrix provided for an embodiment of this application;

[0067] Figure 11B Schematic diagram 2 of a visual media provided in an embodiment of this application;

[0068] Figure 11C A second schematic diagram of a matrix provided for an embodiment of this application;

[0069] Figure 11D A schematic diagram illustrating a positive and negative sample differentiation process provided in an embodiment of this application;

[0070] Figure 11E This is a schematic diagram of a binary classification model training process provided in an embodiment of this application;

[0071] Figure 12 A second schematic diagram illustrating a secondary confirmation provided in this application embodiment;

[0072] Figure 13 Schematic diagram three illustrating a secondary confirmation provided in this application embodiment;

[0073] Figure 14A This application provides an illustration of a secondary confirmation process. Figure 4 ;

[0074] Figure 14B This is a schematic diagram of a secondary confirmation provided in an embodiment of this application. Detailed Implementation

[0075] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0076] To better understand the embodiments of this application, the terminology involved in this application will be explained below.

[0077] Visual media: refers to images or videos.

[0078] Semantic Subjects: Named entity recognition (NER) technology can identify entities with specific meanings in text, such as names of people and places. In this scheme, the identified entities with specific meanings are referred to as semantic subjects.

[0079] Visual content-related and visual content-independent: Visual content refers to the target presented by visual media and the relationships between them. Simply put, visual content can be understood as the content included in visual media. In the context of image search in this application, data that can be obtained from visual media files through the model's natural image understanding is called "visual content-related." This solution refers to data that is related to visual media files but can be obtained without the model's image understanding capabilities as "visual content-independent." For example, electronic devices can acquire and save information such as shooting location, shooting time, name, and file attributes when capturing visual media files.

[0080] For example, in "Photo taken in City 1 this year", "this year" (shooting time), "City 1" (shooting location), and "photo" (file attribute) are all data that electronic devices can acquire and save when capturing visual media files. Therefore, "this year", "City 1", and "photo" are not related to visual semantics. In "Sky taken in City 1 this year", "sky" needs to be understood by the model's image understanding ability to be perceived. Therefore, "sky" is related to visual semantics.

[0081] Text semantic vector: This is a vector that represents the semantic features of the entire sentence, obtained by feeding text into a text encoder. The text encoder can use a clip model or other models, such as the Transformer model commonly used in natural language processing (NLP). This approach does not impose any restrictions on this. In this application, the text semantic vector can also be referred to as the text feature vector.

[0082] Visual semantic vectors: These can be obtained by feeding visual media (such as images) into an image encoder. The image encoder can use a clip model or other models, such as a CNN model or a VIT model; this approach does not impose any restrictions. In this application, visual semantic vectors can also be referred to as image feature vectors.

[0083] Vector similarity: Describes the degree of similarity between two vectors (e.g., between a text semantic vector and a visual semantic vector). In this embodiment, the visual media matching the search query can be determined by comparing the similarity between the text semantic vector of the search query and the visual semantic vector of the visual media. Generally, vector similarity can be calculated using the cosine similarity formula; however, it can also be calculated using other methods.

[0084] Electronic devices (such as mobile phones) can manage users' images, videos, and other visual media through gallery applications. Taking a photo taken with a mobile phone as an example, after the photo is captured, the gallery application determines and stores attribute tags such as the shooting location, shooting time, and photo name, and can use these attribute tags as an index for the image. Once the gallery application has indexed the visual media, it can provide users with corresponding search services. Specifically, users can search for images or videos on their phones by entering keywords in the gallery application. For example, users can enter keywords such as "sky," "cat," or "time point 1" in the search box provided by the gallery application. The gallery application will then match the user's entered keywords with the index of images, videos, and other visual media within the gallery application to obtain search results.

[0085] Optionally, the aforementioned attribute tags may also include attributes such as the person's facial designation (ID), name, and relationship between that person and the mobile phone user. The facial designation of the person in the photo can be automatically generated by the photo library application, while the person's name and relationship can be manually entered by the user. In practice, the same person corresponds to the same facial designation, name, and relationship with the mobile phone user. Therefore, to simplify user operation, the user only needs to enter the person's name and relationship once for each person. Subsequently, the photo library application will automatically configure the person's facial designation, name, and relationship with the mobile phone user for images containing that person's face using facial recognition technology. Additionally, the aforementioned photo name and other attribute information can also be automatically generated by the photo library application or manually named by the user.

[0086] The following is an illustrative description of the interface involved in the search process of the gallery application, with reference to the accompanying drawings:

[0087] like Figure 1A As shown in (a), the mobile phone can display a main interface 101, which can also be referred to as the desktop. The main interface 101 may include an icon 102 for the gallery application. The mobile phone receives a user's click on icon 102; in response to this action, the mobile phone can launch the gallery application and display... Figure 1A The interface 103 shown in (b) can be a photo album interface. It should be noted that, in response to the user clicking icon 102, the phone can launch the gallery application and display the photo gallery interface. The photo gallery interface includes thumbnails of photos (i.e., images) in the gallery or a large image of a specific photo. Within the photo gallery interface, in response to the user's operation of the "Album" control, the aforementioned album interface 103 is displayed.

[0088] like Figure 1A As shown in (b), interface 103 includes multiple albums, including "All Photos" album containing 2023 photos, "Camera" album containing 1502 photos and videos, "Screenshot and Screen Recording" album containing 102 photos and videos, "My Favorites" album containing 48 photos and videos, "One Record, Multiple Views" album containing 34 photos and videos, "Video Editing" album containing 65 videos, "Custom Albums" album containing 57 photos and videos, and "Shared Albums" album containing 100 photos and videos.

[0089] like Figure 1A As shown in (b), interface 103 may include a search box 104. The mobile phone can receive the user's click on the search box 104, and in response to this action, the mobile phone can display as shown in Figure (b). Figure 1AInterface 105, shown in (c) of the diagram, can be referred to as a search interface. Interface 105 displays photo categorization information to the user. For example, in interface 105, the phone categorizes its photos by time, people, and objects. For instance, in the time dimension, the phone categorizes its photos by three time periods: "This Month," "Last Month," and "This Year." The "This Month" album includes photos or videos taken this month, the "Last Month" album includes photos or videos taken last month, and the "This Year" album includes photos or videos taken this year. In the people dimension, the phone categorizes its photos by different people, such as the four different people shown in interface 105. In the objects dimension, the phone categorizes its photos by "Landscape," "Animal," "Document," and "Architecture." It should be noted that the above categorization dimensions can also be other than those specified here. Users can see this categorization information in interface 105 without entering keywords.

[0090] Optionally, the interface 105 may also include search history 107 and a "clear" option 108. Search history includes keywords previously entered by the user, such as "flowers," "coffee," and "cat." The phone can receive the user's click on "clear" 108, and in response, the phone can clear the search history. After the phone clears the search history, the search interface 105 will no longer display the keywords previously entered by the user. For example, in response to the user's click on "clear" 108, such as... Figure 1B As shown, the search history 107 and the "clear" option 108 are no longer displayed on the search interface 105, and the content displayed below has moved up.

[0091] In response to the user entering the keyword "sky" on interface 105, the phone displays the following: Figure 1CThe interface 109 shown in (a) therein. Among them, the mobile phone can search for data related to the keyword "sky" on the local machine. Specifically, the mobile phone can associate with the keyword "sky" to obtain associative words such as "sky" and photos containing the word "sky". Then, search according to each associative word to obtain the search results of each associative word. For example: 100 photos related to "sky" and 32 photos related to photos containing the word "sky". Among them, the 100 photos related to "sky" can be recalled because the classification labels of each of these 100 photos match "sky" or "sky" and its associative words; the 32 photos related to photos containing the word "sky" can be recalled because through optical character recognition (OCR) technology, it is recognized that these 32 photos contain characters such as "sky". The union of the search results of each of these multiple associative words can be used as the search result of the keyword "sky".

[0092] The interface 109 also displays some search results of the keyword "sky" and a "More" option 110 corresponding to the search result of the keyword "sky". The mobile phone receives the click operation of the user on the "More" option 110 and displays as Figure 1C The interface 111 shown in (b) therein. Among them, the interface 111 is used to display photos and videos in the search results of the keyword "sky". Optionally, the photos and videos can be classified and displayed according to time. In addition, the interface 111 also includes a return key 112 and a title 113. In response to the user's operation on the return key 112, the mobile phone can redisplay the interface 109. The title 113 can include the keyword "sky".

[0093] That is to say, in the gallery application, when the user enters a simple search statement in the search box, such as a simple keyword, for example: sky, location 1, time 1, etc., corresponding search results can be obtained. However, because the mobile phone's ability to understand and associate with search statements is limited, if the user enters a more complex search statement in the search box, if the keywords in the search statement cannot match the attribute labels of the pictures or the text in the pictures, it may not be able to find any photos. That is to say, the existing gallery applications do not support the search function based on complex search statements. As Figure 1D shown, when the user enters a more complex search statement "warming oneself by the fire and brewing tea" in the search box of the interface 114, the mobile phone cannot understand the associative words of "warming oneself by the fire and brewing tea", and since the photos do not have labels that match "warming oneself by the fire and brewing tea" or its associative words, its search result shows "no pictures".

[0094] In some embodiments, an electronic device can utilize a text-image matching model to calculate the matching degree between visual media on the electronic device and a search query, allowing the device to select visual media with a matching degree higher than a threshold as search results. The electronic device then displays these search results to present the visual media desired by the user. However, before using the text-image matching model to search for visual media, it needs to be trained using training samples to obtain a model capable of accurately identifying whether visual media is relevant to the search query. However, training samples are typically obtained manually, resulting in long lead times and low training efficiency.

[0095] Therefore, to improve the efficiency of obtaining training samples for the image-text matching model and thus improve the training efficiency of the model, this application provides a scheme for determining the training samples (i.e., positive and negative samples) of the image-text matching model. The electronic device segments the descriptive text corresponding to the sample images in a preset sample data pair to obtain sub-texts within the descriptive text. Then, for each sample image, the electronic device can combine the descriptive text corresponding to that sample image and its sub-texts into a text set corresponding to that sample image. Next, for each sample image, the electronic device can randomly select a text element from the text set corresponding to that sample image as text element 1. Then, the electronic device can use text element 1 corresponding to each sample image as the sample text element corresponding to that sample image, so that multiple sample text elements corresponding to one sample image can be obtained, where one sample image and its corresponding sample text element can form one sample, thus obtaining multiple samples. Then, the electronic device can distinguish between positive and negative samples among these multiple samples, achieving rapid determination of a large number of positive and negative samples and shortening the time for obtaining training samples for the image-text matching model.

[0096] For example, the aforementioned electronic devices can be mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other devices capable of storing visual media.

[0097] For example, Figure 2AA schematic diagram of the structure of electronic device 200 is shown. Electronic device 100 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, antenna 1, antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0098] The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an accelerometer sensor 280E, a distance sensor 280F, a proximity sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0099] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0100] Processor 210 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0101] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0102] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0103] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0104] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0105] Electronic device 200 implements display functions through a GPU (Graphics Processing Unit), a display screen 294, and an application processor. The GPU connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0106] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 294, where N is a positive integer greater than 1.

[0107] Electronic device 200 can perform shooting functions through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0108] Camera 293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 293, where N is a positive integer greater than 1.

[0109] Video codecs are used to compress or decompress digital video. Electronic device 200 may support one or more video codecs. Thus, electronic device 200 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0110] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0111] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0112] Internal memory 221 can be used to store computer executable program code, which includes instructions. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 210 executes various functional applications and data processing of electronic device 200 by running instructions stored in internal memory 221 and / or instructions stored in memory disposed in the processor.

[0113] Figure 2B This is a software structure block diagram of an electronic device 200 according to an embodiment of this application. The software system of the electronic device 200 can adopt a layered architecture, which divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. Taking the Android system as an example, in some embodiments, the Android system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0114] like Figure 2B As shown, the application layer may include applications such as a gallery service module, a search module, a multimodal understanding module, a natural language understanding module, and a camera application.

[0115] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0116] The system library can include multiple functional modules, such as a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL). The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0117] The kernel layer is the layer between hardware and software.

[0118] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 200.

[0119] When the touch sensor 280K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, where the click corresponds to the camera application icon, the camera application calls the interface of the application framework layer to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 293.

[0120] Taking a mobile phone as an example, the visual media search method provided in this application embodiment will be introduced. The visual media search method provided in this application embodiment can be applied to applications such as gallery applications and file management applications.

[0121] The interface and search logic involved in the visual media search method provided in the embodiments of this application will be described exemplarily below with reference to the accompanying drawings.

[0122] like Figure 3A Interface 301, shown in (a), displays search history 303 and a "clear" option 304. Search history 303 includes previously entered search terms, such as "sunrise from the mountaintop" and "sky photographs." Other content displayed on interface 301 can be referenced from the related content on interface 105, and will not be repeated here. Interface 301 can be referred to as the search interface. The mobile phone can respond to user queries... Figure 1A The click operation on the search box 104 in the interface 103 shown in (b) in

[0123] As Figure 3A shown in (b) in Figure 3A interface 305 (i.e., the search result interface), the user inputs the search statement "warming the tea around the stove" in the search box 306 of interface 305, and the mobile phone searches for 239 pictures. The mobile phone displays some search results of the search statement "warming the tea around the stove" (e.g., thumbnails of 8 pictures) and the "More" option 307 corresponding to the search results of the search statement "warming the tea around the stove" on interface 305. In response to the user's operation on the "More" option 307, the mobile phone displays interface 308 as shown in (c) in

[0124] In addition, the mobile phone can also have a negative first screen, a pull-down search interface, etc. It can be understood that the negative first screen can be the leftmost split screen of the electronic device, which is used to provide functions such as search and quick services for users. Among them, the negative first screen can also be used to display notification messages to be pushed to users, such as application messages subscribed by users, real-time hot search messages, segment selection, itinerary information, etc. The pull-down search interface is an interface displayed in response to the user's pull-down operation on the main interface, and this interface is used to provide functions such as search and application suggestions for users. This interface and Figure 3B interface 315 in (c) below can be the same interface.

[0125] Next, the negative first screen will be taken as an example for introduction. When the user needs to view the negative first screen of the mobile phone, the user can slide the screen of the mobile phone to make the electronic device display the negative first screen.

[0126] Exemplarily, referring to Figure 3B shown in (a) in Figure 3B the mobile phone can receive operation 1 implemented by the user on the interface 309 (which can be called the desktop) of the mobile phone. Exemplarily, this operation 1 can be a Figure 3B rightward sliding operation as shown in (a) in

[0127] The mobile phone receives a click from the user on the search box 311 on the negative one screen 310. In response to this click, the mobile phone can display as follows: Figure 3B Interface 315 is shown in (c). Interface 315 may include: a search box 316 and application suggestions. Application suggestions include icons of suggested applications. Interface 315 may also include: search history 317 and its corresponding "clear" option 318. In response to the user's triggering operation on the "clear" option 318, the search history 317 and the "clear" option 318 will no longer be displayed on interface 315. In addition, the search box 316 may display trending news headlines, such as "marathon race".

[0128] like Figure 3B As shown in (d) of the diagram, interface 319 displays the user-inputted search term "mountain peaks photographed on weekends" in its search box. Interface 319 also displays a preview area 322 of the search results for "mountain peaks photographed on weekends" from the photo library app, as well as a "Search in app" option 323 from the photo library app. In response to the user's action on preview area 322, the phone can access the photo details interface provided by the photo library app, allowing the user to browse and view the search results for "mountain peaks photographed on weekends," including visual media such as photos or videos matching the search term. In response to the user's action on "Search in app" option 323, the phone displays... Figure 3B As shown in (e), interface 324, provided by the photo library application, displays partial search results for the search query "mountains photographed on weekends" and a "more" option corresponding to the search results for the same query. In response to a user's triggering of this "more" option, the phone can display a search results details interface, showing images from the search results for "mountains photographed on weekends." Additionally, interface 319 may also display an online search option 321. In response to a user's triggering of the online search option 321, the phone displays a search webpage and shows the online search results within that webpage.

[0129] The above section introduced the interface and search logic involved in the visual media search method. The following section will continue to combine the above... Figure 2B The software structure shown illustrates the specific implementation process of the visual media search method. For example... Figure 4 As shown, the implementation process may include S401-S423, where S401-S407 may belong to the index building stage, and S408-S423 may belong to the search stage.

[0130] S401. Add and / or modify visual media and their attributes in the Gallery Service module.

[0131] The attributes of visual media may include one or more of the following: capture location, capture time, visual media name, facial ID of a person in the visual media, person's name, and the relationship between that person and the mobile phone user. For example, in the case of a captured video or image, the capture location refers to the location where the video was taken, and the capture time refers to the time the video was taken; in the case of a screenshot, the capture location refers to the location where the screenshot was taken, and the capture time refers to the time the screenshot was taken; in the case of a downloaded video or image, the capture location refers to the location where the video was downloaded, and the capture time refers to the time the video was downloaded.

[0132] Users can add new visual media by taking photos, downloading, or taking screenshots. In addition, users can modify existing visual media to create new ones. These modifications include, but are not limited to, enhancements, custom naming, and adding watermarks.

[0133] S402, The image library service module stores visual media and their attributes.

[0134] The gallery service module can respond to user input regarding adding or modifying visual media, storing the visual media and its attributes locally on the phone. In practical applications, with user authorization, the phone can store the locally stored visual media and its attributes in the cloud to reduce the storage pressure on the phone's local storage.

[0135] S403, the image library service module sends request 1 to the multimodal understanding module. Request 1 is used to trigger visual semantic understanding of visual media.

[0136] S404. The multimodal understanding module returns the image feature vector of the visual media to the image library service module.

[0137] In this embodiment, the multimodal understanding module responds to request 1 above and determines the image feature vector of the visual media stored on the mobile phone. Since visual semantic understanding requires significant computing resources, to avoid impacting user experience, the gallery service module can request the multimodal understanding module to perform visual semantic understanding on the visual media stored locally on the mobile phone while the phone is charging and the screen is off. This includes performing visual semantic understanding on newly added or modified visual media to obtain the visual semantic vector (or image feature vector) of the visual media. This enables offline processing of the visual media, reducing the impact of visual semantic understanding on other services running on the mobile phone.

[0138] The multimodal understanding module can perform visual semantic understanding of visual media based on a multimodal model, obtaining image feature vectors of the visual media. Furthermore, the multimodal model can not only be used for visual semantic understanding of visual media to obtain visual semantic vectors, but also for semantic understanding of search statements to obtain text feature vectors (or sentence semantic vectors) of the search statements. The process of semantic understanding of search statements by the multimodal model can be referred to in the relevant description below, and will not be detailed here.

[0139] In some embodiments, the multimodal model can map visual media and text into vectors of the same dimension; that is, the dimension of the visual semantic vector of the visual media is the same as the dimension of the semantic vector of the text (e.g., the sentence semantic vector of a search query). Specifically, the multimodal model can be based on a contrastive language-image pre-training (CLIP) model. Mobile phones can use the CLIP model to map visual media and text into a unified vector space to understand the relationships between different modalities in both text and vision, thereby enabling image retrieval.

[0140] Among them, the CLIP model is either a standard CLIP model, that is, an existing CLIP model, or a custom CLIP model.

[0141] For example, the aforementioned custom CLIP model may include a base model and a fine-tuned model. The fine-tuned model is obtained by further training the base model using a training set for a specific scenario. Therefore, the fine-tuned model has a limited scope, capable of processing images and text in a specific scenario. The generalization ability of the fine-tuned model is less than that of the base model, while the retrieval ability of the base model in a specific scenario is less than that of the fine-tuned model.

[0142] In some embodiments, the base model and fine-tuning model reuse parts of the network to reduce resource consumption on mobile phones with a custom CLIP model installed. The base model and fine-tuning model need to process not only visual media but also text. The module in the base model that processes visual media can be called the visual encoding module, and the module in the fine-tuning model that processes visual media can be called the visual fine-tuning module. The image feature vectors mentioned above can include the image feature vectors corresponding to the base model and the image feature vectors corresponding to the fine-tuning model. The visual encoding module of the base model and the visual fine-tuning module of the fine-tuning model can reuse the image encoder, allowing the visual encoding module to use the feature vectors of the visual media output by the image encoder to further determine the first L2 norm of the visual media, thus determining the image feature vectors corresponding to the base model. The visual fine-tuning module can use the feature vectors of the visual media output by the image encoder to further determine the second L2 norm of the visual media, thus determining the image feature vectors corresponding to the fine-tuning model.

[0143] For example, upon receiving request 1 from the gallery service module, the response to request 1 is as follows: Figure 5A As shown, the multimodal understanding module can input k visual media from the mobile phone into the visual encoding module. The image encoder in the visual encoding module encodes each of the k visual media to obtain the image feature vector 1 of each visual media. This image feature vector can be 768-dimensional, i.e., X = {x1, x2, ..., x768}, where X represents the image feature vector 1. Then, on one hand, the visual encoding module can output this 768-dimensional image feature vector 1 so that it can continue to be used as the input parameter of the visual fine-tuning module. On the other hand, the visual encoding module continues to use the mapping matrix 1 and the image feature vector 1 to calculate and output the first L2 norm α1 of the image feature vectors of each of the k visual media. The mapping matrix 1 has a dimension of 768*512 to map the image feature vector 1 from 768 dimensions to 512 dimensions. Here, the image feature vector 1 and the first L2 norm of the visual media can be considered as the image feature vectors of the visual media corresponding to the base model.

[0144] Specifically, for each of the k visual media, the visual encoding module can adopt... Calculate the first L2 norm α1 of the image feature vectors of visual media. Here, X represents a 768-dimensional image feature vector 1, M1 represents a mapping matrix 1, and V1 represents an image feature vector 2, where V1 has a dimension of 512. This represents the i-th element in V1.

[0145] like Figure 5BAs shown, after receiving the image feature vector 1 of each of the k visual media, the visual fine-tuning module calculates and outputs the second L2 norm α2 of the image feature vector for each visual media using the mapping matrix 2 and the image feature vector 1. The mapping matrix 2 has a dimension of 768*512 to map the image feature vector 1 from 768 dimensions to 512 dimensions. Here, the image feature vector 1 and the second L2 norm of the visual media can be considered as the image feature vector of the visual media corresponding to the fine-tuning model.

[0146] Specifically, for each of the k visual media, the visual encoding module can adopt... Calculate the second L2 norm α2 of the image feature vectors of visual media. Here, X represents the 768-dimensional image feature vector 1, M2 represents the mapping matrix 2, and V2 represents the image feature vector 3, where V2 has a dimension of 512. This represents the i-th element in V2.

[0147] In some embodiments, the fine-tuning model described above (such as mapping matrix 2 in the visual fine-tuning module) is obtained by further training using a training set for a specific scene. When training the fine-tuning model, the device can freeze the image encoder and train only the last fully connected layer M2 of the fine-tuning model. The training set for this specific scene can be an image training set including visual content corresponding to a specific whitelist. This specific whitelist represents the support range of the fine-tuning model. The device can be the aforementioned mobile phone, or it can be something else; this application does not limit the device used to train the fine-tuning model.

[0148] After obtaining the image feature vectors of the visual media (such as the image feature vectors of the visual media corresponding to the base model and the image feature vectors of the visual media corresponding to the fine-tuning model), in order to facilitate the retrieval of the visual media required by the user using the image feature vectors of the visual media, the multimedia understanding module can save the image feature vectors of the visual media, namely the image feature vector 1, the first L2 norm α1, and the second L2 norm α2 mentioned above. This allows the storage of an image feature vector of a visual media to only require saving a 770-dimensional visual vector, instead of saving a 512*2-dimensional, i.e., 1024-dimensional visual feature vector, thus reducing the resources required to store the image feature vectors. Among them, the 512-dimensional vector is the image feature vector obtained by processing the 768-dimensional vector using mapping matrix 1 or mapping matrix 2 (i.e., the V1 and V2 mentioned above).

[0149] It should be noted that the image encoder described above, located in the visual encoding module, is merely an example. The image encoder could also be located in the visual fine-tuning module, and this application does not limit it. Furthermore, the image feature vector of the visual media described above, including the image feature vector of the visual media corresponding to the base model and the image feature vector of the visual media corresponding to the fine-tuning model, is only one example. The image feature vector of the visual media could also include only one image feature vector determined by a single model, which could be a clip model or not.

[0150] As described above, the image feature vectors of visual media on a mobile phone can be determined offline, while the text feature vectors of the search query can be determined online after the mobile phone receives the user's search query. This allows the mobile phone to use the text feature vectors of the search query and the image feature vectors of the visual media on the phone to determine the visual media that matches the search query. The process of determining the text feature vectors of the search query and using them to determine the matching visual media is detailed below and will not be elaborated on here. We will now continue by introducing the process of building an index for visual media.

[0151] S405, the image library service module stores the image feature vectors of visual media.

[0152] S406. The image library service module sends the attribute information of the visual media and its visual semantic vector to the search module.

[0153] In this embodiment, the image library service module can locally store the image feature vector of each of the k visual media on the mobile phone. Furthermore, the image library service module can send the attribute information and image feature vectors of the visual media in batches to the search module, so that the search module can construct an index of the visual media.

[0154] Optionally, the multi-image library service module may choose not to upload the image feature vector of the visual media to the cloud. Of course, it may also upload the image feature vector of the visual media to the cloud with the user's authorization. This application does not restrict this.

[0155] S407, The search module builds an index for visual media.

[0156] The index of visual media built by the search module may include: the attributes of the visual media and / or the visual semantic vector of the visual media.

[0157] S408, The image library service module receives the search query entered by the user.

[0158] Users can use the search interface provided by the gallery service module, such as the search results mentioned above. Figure 3AEnter "warming the tea around the stove" in the search box 306 on the interface 305 shown in (b) of []. This "warming the tea around the stove" is the search statement.

[0159] S409. The gallery service module sends the search statement to the search module.

[0160] S410. The search module determines whether the search statement includes visual content.

[0161] In the embodiments of the present application, the search module can determine whether the search statement includes visual content, that is, determine whether it is necessary to search for visual media using image feature vectors and text feature vectors. In other words, the search module determines whether it is necessary to use a multimodal model to determine visual media.

[0162] When the search statement does not include visual content, it indicates that visual media can be searched using the attributes of visual media, without the need to search for visual media using image feature vectors and text feature vectors. That is, it indicates that there is no need to use a multimodal model to determine visual media, and the search module can execute S411.

[0163] When the search statement includes visual content, it indicates that it is necessary to search for visual media using image feature vectors and text feature vectors. That is, it indicates that it is necessary to use a multimodal model to determine visual media, and the search module can execute S412.

[0164] In some embodiments, the search module can determine whether a search statement includes visual content by judging whether it includes a visual semantic subject. The search module can send request 2 to the natural language understanding module. The natural language understanding module can perform semantic subject recognition on the search statement to obtain the semantic subjects included in the search statement. The natural language understanding module performs semantic subject recognition based on a natural language understanding model. Specifically, the natural language understanding module can use named entity recognition (NER) technology to perform semantic subject recognition on the search statement to obtain the semantic subjects contained in the search statement. In this embodiment, a semantic subject can also be referred to as an entity, and a semantic subject may include one or more of the following: semantic subjects related to time, semantic subjects related to location, semantic subjects related to names, semantic subjects related to relationships between people, and semantic subjects related to visual content (or visual semantic subjects). Optionally, the semantic subjects related to visual content are determined from a preset set of M (M≥1) semantic subjects related to visual content using named entity recognition technology. These M semantic subjects can be configured by the developers of the image library service module according to actual needs. Generally, these M semantic subjects are all nouns. For example, semantic subject identification of the search query "sky taken in city 1 on September 1st" can yield three semantic subjects: "September 1st", "city 1", and "sky". Among them, "sky" is related to visual content and can be a visual semantic subject.

[0165] Next, the natural language understanding module can return the identified semantic subject to the search module. The search module then determines whether the semantic subject includes a visual semantic subject. If the semantic subject does not include a visual semantic subject, the search module can determine that the search query does not include visual content. If the semantic subject includes a visual semantic subject, the search module can determine that the search query includes visual content.

[0166] S411. The search module retrieves search results based on the index and search query of visual media.

[0167] For example, if the search query does not include a visual semantic subject, indicating that the search query does not include content related to visual semantics, the search module can directly query the visual media corresponding to the semantic subject based on the constructed index, that is, based on the attributes of each visual media, and use it as the search result. Here, the semantic subject refers to the attribute of the visual media. For example, if the search query is "September 1st" and does not include visual content, the search module can use the index to find visual media whose collection time is September 1st and obtain the search result. As another example, if the search query includes a photo of Zhang San but does not include visual content, the search module can match the names of people in each visual media with "Zhang San" to determine the visual media whose name attribute is "Zhang San" and use it as the search result.

[0168] S412. The search module filters the non-visual semantic subjects in the above search statement to obtain the filtered search statement.

[0169] For example, when the aforementioned semantic subjects include visual semantic subjects, it indicates that the search statement includes content related to visual semantics and may also include content unnecessary for visual semantics, i.e., non-visual semantic subjects. Since non-visual semantic subjects are irrelevant to the search for visual content, the search module can first filter the non-visual semantic subjects in the search statement to obtain a filtered search statement, in order to search for visual media that match the filtered search statement.

[0170] Optionally, non-visual semantic subjects refer to semantic subjects related to attributes of visual media, such as semantic subjects related to time, semantic subjects related to location, semantic subjects related to names, and semantic subjects related to relationships between people. It should be understood that since relationships between people have already been considered as attributes of visual media, they can be considered non-semantic subjects here.

[0171] For example, in the search query "sky photographed this year", "this year" is a non-visual semantic subject, so the filtered search query is "sky photographed".

[0172] In practical applications, after filtering semantic subjects unrelated to visual content, some redundant stop words may remain. For example, in the search query "sky photographed in city 1 this year," after deleting "this year" and "city 1," the stop word "in" becomes redundant and can therefore be filtered out by the search module. Specifically, the search module can filter semantic subjects unrelated to visual content and their associated stop words in the search query to obtain a filtered search query. For example, the filtered search query corresponding to the search query "sky photographed in city 1 this year" is "the sky photographed."

[0173] S413, The search module sends a filtered search statement to the multimodal understanding module.

[0174] S414 The multimodal understanding module performs semantic understanding on the filtered search statement to obtain the text feature vector of the filtered search statement.

[0175] For example, the multimodal understanding module can employ a multimodal model to determine the text feature vectors of the filtered search statements. Optionally, the multimodal model can be a CLIP model. The CLIP model can be a standard CLIP model, or the CLIP module can be a custom CLIP model.

[0176] In some embodiments, as described above, the image feature vector may include the image feature vector corresponding to the base model and the image feature vector corresponding to the fine-tuning model. The visual encoding module of the base model and the visual fine-tuning module of the fine-tuning model can reuse the image encoder. Accordingly, in order to maintain the similarity of the image-text pairs, this application introduces a text encoding module, which can reuse the text encoder to output the text feature vector of the filtered search statement corresponding to the base model and the text feature vector of the filtered search statement corresponding to the fine-tuning model, respectively. This allows for the calculation of the similarity between the search statement and the visual media using the text feature vector of the filtered search statement corresponding to the base model and the image feature vector of the visual media corresponding to the base model, and the calculation of the similarity between the search statement and the visual media using the text feature vector of the filtered search statement corresponding to the fine-tuning model and the image feature vector of the visual media corresponding to the fine-tuning model, thereby realizing the calculation of the similarity of the image pairs.

[0177] For example, such as Figure 5C As shown, the multimodal understanding model inputs the filtered search statement into the CLIP model. The text encoder in the CLIP model's text encoding module encodes the filtered search statement, obtaining and outputting its feature vector, which has 768 dimensions. Subsequently, the text encoding module can input the feature vector into the base model branch and the fine-tuning model branch, respectively. The base model branch and the fine-tuning model branch have similar structures.

[0178] For the base model branch: the text encoding module uses mapping matrix 3 and the feature vector of the filtered search statement to obtain a 512-dimensional intermediate variable 1. For example, the text encoding module calculates intermediate variable 1 according to T1 = YN1. Here, T1 represents intermediate variable 1, Y represents the feature vector of the filtered search statement, N1 represents mapping matrix 3, and the text encoding module uses intermediate variable 1 to calculate the first L2 norm of the filtered search statement. For example, the text encoding module can... Calculate the first L2 norm of the filtered search statement. Where β1 represents the first L2 norm of the filtered search statement. Let i represent the i-th element in T1.

[0179] Since the image feature vector of visual media has a dimension of 768, and the intermediate variable 1 of the filtering search statement has a dimension of 512, the text encoding module needs to map the intermediate variable 1 of the filtering search statement to a 768-dimensional form in order to perform calculations between the image feature vector and the text feature vector. Specifically, the text encoding module can use the aforementioned mapping matrix 1 and the first L2 norm of the filtering search statement to map the intermediate variable 1 into a 768-dimensional text feature vector, thereby obtaining the text feature vector of the filtering search statement corresponding to the base model. Calculate the text feature vector of the filtered search statement corresponding to the base model. Here, T′1 represents the text feature vector of the filtered search statement corresponding to the base model. It is the transpose of M1, where M1 represents the above mapping matrix 1.

[0180] For the fine-tuning model branch: the text encoding module uses mapping matrix 4 and the feature vectors of the filtered search statements to obtain a 512-dimensional intermediate variable 2. For example, the text encoding module calculates intermediate variable 2 based on T2 = YN2. Here, T2 represents intermediate variable 2, N2 represents mapping matrix 4, and the text encoding module uses intermediate variable 2 to calculate the second L2 norm of the filtered search statements. For example, the text encoding module can... Calculate the second L2 norm of the filtered search statement. Where β2 represents the second L2 norm of the filtered search statement. Let i represent the i-th element in T2.

[0181] Since the image feature vector of visual media has a dimension of 768, and the intermediate variable 2 of the filtering search statement has a dimension of 512, the text encoding module needs to map the intermediate variable 2 of the filtering search statement to a 768-dimensional form in order to perform calculations between the image feature vector and the text feature vector. Specifically, the text encoding module can use the aforementioned mapping matrix 2 and the first L2 norm of the filtering search statement to map the intermediate variable 2 into a 768-dimensional text feature vector, thereby obtaining the text feature vector of the filtering search statement corresponding to the fine-tuning model. Specifically, the text encoding module can utilize... Calculate the text feature vector of the filtered search statement corresponding to the base model. Here, T′2 represents the text feature vector of the filtered search statement corresponding to the fine-tuned model. M1 is the transpose of M2, and M1 represents the above mapping matrix 2.

[0182] It should be noted that the text feature vectors of the filtered search statements for the fine-tuning model and the filtered search statements for the base model can be output simultaneously by the text encoding module.

[0183] In this embodiment, the base model and fine-tuning model in the custom CLIP model reuse the text encoder, so that the custom CLIP model only needs to calculate the feature vector of the filtered search statement once. Then, it can use the feature vector of the filtered search statement to determine the text feature vector of the filtered search statement corresponding to the base model and the text feature vector of the filtered search statement corresponding to the fine-tuning model, respectively, without the base model and the fine-tuning model calculating the feature vector of the filtered search statement separately, thereby improving the computational efficiency of the text feature vector of the filtered search statement.

[0184] In this embodiment, to ensure the generalization ability and the ability to recognize specific images and text of the multimodal model, the multimodal model includes a base model and a fine-tuning model. Since the base model and the fine-tuning model have the same model weights, meaning some parts of the network are identical, directly setting two models would waste mobile phone memory. Therefore, this application reuses the image encoder and text encoder for both the base model and the fine-tuning model, allowing them to determine their respective text feature vectors and image feature vectors using the feature vectors output by the image encoder and text encoder, respectively. Overall, this reduces the determination time of image feature vectors and text feature vectors by nearly half.

[0185] S415. The multimodal understanding module performs recall based on text feature vectors and image feature vectors of visual media to obtain candidate visual media.

[0186] In some embodiments, the image feature vector of the aforementioned visual media may be sent by the search module to the multimodal understanding module, or it may be obtained by the multimodal understanding module from the local mobile phone.

[0187] In this embodiment, for each visual medium on the mobile phone (such as each of the k visual media mentioned above), the multimodal understanding module can calculate the vector similarity (or simply similarity) between the image feature vector of the visual medium and the text feature vector of the filtering search statement. That is, based on the image feature vector of the visual medium and the text feature vector of the filtering search statement, the similarity between the visual medium and the filtering search statement is calculated. Then, based on the vector similarity, the multimodal understanding module determines the visual medium matching the filtering search statement from the k visual media and uses the determined visual medium as candidate visual media.

[0188] For example, the multimodal understanding module can identify visual media with a vector similarity greater than or equal to a threshold of 1 as visual media matching the filtering search statement. Optionally, if the number of visual media with a vector similarity greater than or equal to the threshold of 1 is greater than one, the multimodal understanding module can sort the visual media with a vector similarity greater than or equal to the threshold of 1 in descending order of vector similarity. Then, the multimodal understanding module can identify the top n sorted visual media as visual media matching the filtering search statement. Here, n is a positive integer.

[0189] In some embodiments, the image feature vector of the visual media may include the image feature vector of the visual media corresponding to the base model and the image feature vector of the visual media corresponding to the fine-tuning model. The text feature vector of the filtered search statement includes the text feature vector of the filtered search statement corresponding to the base model and the text feature vector of the filtered search statement corresponding to the fine-tuning model. In one case, such as... Figure 5D As shown, the multimodal understanding module can calculate the vector similarity 1 between the text feature vector of the filter search statement corresponding to the base model and the image feature vector of the visual media corresponding to the base model (that is, calculate the similarity between the visual media corresponding to the base model and the filter search statement), and calculate the vector similarity 2 between the text feature vector of the filter search statement corresponding to the fine-tuning model and the image feature vector of the visual media corresponding to the fine-tuning model (that is, calculate the similarity between the visual media corresponding to the fine-tuning model and the filter search statement). Then, the multimodal understanding module can determine visual media with vector similarity 1 or vector similarity 2 greater than or equal to threshold 1 as visual media matching the filter search statement.

[0190] In another scenario, the multimodal understanding module can determine whether the filtered search statement falls within the support range of the fine-tuning model by checking if it hits whitelist 1, thus determining whether to use the fine-tuning model branch to identify candidate visual media. The following will combine... Figure 6 This paper introduces the process by which the multimodal understanding module uses whitelist 1 to determine candidate visual media.

[0191] S501, The multimodal understanding module determines whether the filtered search statement belongs to the whitelist.

[0192] In this embodiment of the application, if the filtered search statement is not in the whitelist, it indicates that the filtered search statement is within the support range of the base model. The multimodal understanding module can use the base model branch to determine the candidate visual media, and the multimodal understanding module can execute S502.

[0193] If the filtered search query is on the whitelist, it indicates that the search query is within the support range of the fine-tuning model. The multimodal understanding module can then determine the candidate visual media by making the fine-tuning model branch, and the multimodal understanding module can execute S504.

[0194] In some embodiments, the multimodal understanding module determines whether each semantic subject (i.e., visual semantic subject) in the filtered search statement belongs to whitelist 1. Considering that user-input search statements are generally phrases, the multimodal understanding module can determine that the filtered search statement belongs to the whitelist if all semantic subjects in the filtered search statement belong to whitelist 1. If any semantic subject does not belong to whitelist 1, the multimodal understanding module determines that the filtered search statement does not belong to the whitelist. For example, if the filtered search statement is "a boy holding a basket," the semantic subjects include "basket" and "boy." The multimodal understanding module can determine whether the basket and the boy belong to whitelist 1 respectively. If both "blue" and "boy" belong to whitelist 1, the multimodal understanding module can determine that the filtered search statement belongs to the whitelist. If neither "blue" nor "boy" belongs to whitelist 1, the multimodal understanding module can determine that the filtered search statement does not belong to the whitelist.

[0195] S502. For each visual medium, the multimodal understanding module calculates the vector similarity 1 between the image feature vector of the visual medium corresponding to the base model and the text feature vector of the filter search statement corresponding to the base model.

[0196] In the embodiments of this application, such as Figure 7 As shown, when the filtered search statement does not belong to the whitelist 1, the multimodal understanding module can call the text feature vector of the filtered search statement output by the base model branch of the text encoding module and the image feature vector of the visual media corresponding to the base model to calculate the score of the visual media corresponding to the base model, that is, calculate the vector similarity 1 between the visual media corresponding to the base model and the filtered search statement.

[0197] Specifically, the multimodal understanding module can be achieved through... Calculate the score corresponding to the visual media of the base model. Here, S1 represents the score corresponding to the visual media of the base model, α1 represents the first L2 norm of the image feature vector of the aforementioned visual media, X represents the image feature vector 1, and T′1 represents the text feature vector of the filtered search statement corresponding to the aforementioned base model.

[0198] S503, the multimodal understanding module selects visual media with a vector similarity of 1 greater than the threshold 1 as candidate visual media.

[0199] S504. For each visual medium, the multimodal understanding module calculates the vector similarity 2 between the image feature vector of the visual medium corresponding to the fine-tuning model and the text feature vector of the filter search statement corresponding to the fine-tuning model.

[0200] In the embodiments of this application, as described above Figure 7 As shown, when the filtered search statement belongs to whitelist 1, the multimodal understanding module can call the text feature vector of the filtered search statement output by the fine-tuning model branch of the text encoding module and the image feature vector of the visual media corresponding to the fine-tuning model to calculate the score corresponding to the visual media corresponding to the fine-tuning model, that is, to calculate the vector similarity 2 between the visual media corresponding to the fine-tuning model and the filtered search statement.

[0201] Specifically, the multimodal understanding module can be achieved through... Calculate the score corresponding to the visual media of the base model. Here, S2 represents the score corresponding to the visual media of the fine-tuned model, and α2 represents the second L2 norm of the image feature vector of the aforementioned visual media. X represents the aforementioned image feature vector 1, and T′2 represents the text feature vector of the filtered search statement corresponding to the aforementioned fine-tuned model.

[0202] S505, the multimodal understanding module selects visual media with a vector similarity of 2 greater than the threshold of 1 as candidate visual media.

[0203] In some embodiments, the operations performed by the multimodal understanding module can be performed by the multimodal model. Additionally, S501-S505 can be performed by the multimodal understanding module, i.e., the model output module within the multimodal model.

[0204] In this embodiment, the multimodal understanding module can determine the image feature vector of visual media offline, while only needing to determine the text feature vector of the search query online. This improves the computation time for vector similarity between image and text feature vectors, effectively reducing user retrieval time. Furthermore, the multimodal understanding module can determine whether to use a fine-tuning model to identify candidate visual media based on whether the filtered search query belongs to whitelist 1, thereby improving the accuracy of candidate visual media identification and ultimately enhancing the accuracy of search results.

[0205] Optionally, the whitelist 1 mentioned above is expandable, meaning the scope of support for the fine-tuning model is expandable. In other words, the scenarios supported by the fine-tuning model are expandable; the fine-tuning model can be trained using the training set corresponding to the expanded scenarios. It should be understood that training the fine-tuning model actually involves training the fine-tuning model branches in the fine-tuning module and the text encoding module, without needing to train the reused parts (such as the image encoder and text encoder mentioned above).

[0206] In some embodiments, after determining candidate visual media, the multimodal understanding module can directly use these candidate visual media as search results. The multimodal understanding module can then return the search results to the search module. The search module can then send the search results to the image library service module for display.

[0207] Furthermore, to improve the accuracy of search results, after obtaining the aforementioned candidate visual media, the mobile phone can perform a secondary verification of these candidate visual media to further filter them and obtain visual media that match the filtered search statement, i.e., the search statement. Optionally, the mobile phone can include both visual media with a vector similarity of 1 greater than a threshold of 1 and visual media with a vector similarity of 2 greater than a threshold of 1 as candidate visual media. In other words, the mobile phone can use the visual media determined by the base model and the fine-tuning model as candidate visual media, i.e., it performs two searches. The process of secondary verification of candidate visual media will be described below.

[0208] S416. The multimodal understanding module sends the information of the filtered search statement and the image information of the candidate visual media to the secondary confirmation module.

[0209] The information for filtering search statements includes one or more of the following: the filtering search statement, the word segmentation results of the filtering search statement, the tag 1 included in the filtering search statement, and the text feature vector of the filtering search statement.

[0210] The image information of the candidate visual media includes one or more of the following: the similarity between the candidate visual media and the filtering search statement, the label 2 of the candidate visual media, and the image feature vector of the candidate visual media.

[0211] In some embodiments, the image feature vectors and text feature vectors described above are determined based on a multimodal model including a base model and a fine-tuning model. Correspondingly, the text feature vector of the filtered search statement may include the text feature vector of the filtered search statement corresponding to the base model and the text feature vector of the filtered search statement corresponding to the fine-tuning model. The image feature vector of the candidate visual media may include the image feature vector of the candidate visual media corresponding to the base model and the image feature vector of the candidate visual media corresponding to the fine-tuning model.

[0212] The similarity between the candidate visual media and the filtered search statement can include the similarity between the candidate visual media and the filtered search statement corresponding to the base model (or similarity 1), and the similarity between the candidate visual media and the filtered search statement corresponding to the fine-tuned model (or similarity 2).

[0213] Of course, the above image feature vectors, text feature vectors, and similarity can also be determined by only one model (such as the above base model or fine-tuned model), and this application does not limit it.

[0214] In some embodiments, the multimodal understanding module can perform word segmentation extraction on the filtered search statement to obtain the word segmentation result of the filtered search statement. For example, if the filtered search statement is "The child is playing at the seaside", the word segmentation result is: child, at, seaside, and playing. Exemplarily, the multimodal understanding module can perform word segmentation extraction on the filtered search statement through the natural language understanding engine service (the natural language understanding, NLU). Optionally, the multimodal understanding module can perform word segmentation extraction on the filtered search statement through the natural language understanding module.

[0215] After obtaining the word segmentation result of the filtered search statement, the multimodal understanding module can perform label mapping on the word segmentation result to obtain the label included in the filtered search statement, that is, label 1. Specifically, for each word in the word segmentation result, the filtered search module determines whether the word exists in the preset label table. In the case where the word does not exist in the preset label table, the multimodal understanding module can determine that the word does not have a corresponding label, that is, the filtered search statement does not include the label corresponding to the word.

[0216] In the case where the word exists in the preset label table, the multimodal understanding module can use the label corresponding to the word in the preset label table as the label 1 included in the filtered search statement. For example, the word segmentation includes "cat", and the label corresponding to "cat" in the preset label table is "cat", so the label included in the filtered search statement includes "cat". It should be noted that the words in the word segmentation result of the filtered search statement and the labels corresponding to the words may be the same or different.

[0217] In some embodiments, the above-mentioned label 2 of the candidate visual media is obtained from the label library. The label 2 of the visual media in the label library represents the classification label of the visual media, which can be determined by the mobile phone (such as the multimodal understanding module in the mobile phone) using an image classification model to identify the visual media. Exemplarily, the classification label of the visual media can also be determined offline by the mobile phone.

[0218] S417. The secondary confirmation module determines the Boolean value corresponding to each candidate visual media based on the information of the filtered search statement and the image information of the candidate visual media.

[0219] S418. For each candidate visual media, when the Boolean value corresponding to the candidate visual media is true, the secondary confirmation module determines that the candidate visual media is visual media 1.

[0220] In this embodiment, the multimodal understanding module inputs the information of the filtering search statement and the image information of the candidate visual media to the secondary confirmation module, so that the secondary confirmation module determines the Boolean value corresponding to each candidate visual media. The Boolean value corresponding to the candidate visual media indicates whether the candidate visual media is a visual media that matches the filtering search statement, thereby realizing the secondary confirmation of the candidate visual media.

[0221] In some embodiments, the aforementioned secondary confirmation module may include a binary classification module, a dynamic threshold module, a label confirmation module, and a whitelist threshold module. The binary classification module may employ a binary classification model to determine whether there is a correlation between the filtered search statement and the candidate visual media, thereby obtaining the Boolean value of the candidate visual media. The dynamic threshold module may employ a dynamic threshold model to determine a threshold 2 that matches the length of the filtered search statement, thereby comparing the similarity between the candidate visual media and the filtered search statement with this threshold 2 to obtain the Boolean value of the candidate visual media. The label confirmation module may compare the label 2 of the candidate visual media with the label 1 included in the filtered search statement to obtain the Boolean value of the candidate visual media. The whitelist threshold module may determine a threshold 3 that matches the entire filtered search statement based on whether the filtered search statement hits a preset dictionary, thereby comparing the similarity between the candidate visual media and the filtered search statement with this threshold 3 to obtain the Boolean value of the candidate visual media. The detailed process by which the secondary confirmation module determines the candidate visual media through the binary classification module, dynamic threshold module, label confirmation module, and whitelist threshold module, i.e., the implementation process of S417 described above, can be referred to the relevant description below, and will not be introduced here.

[0222] S419, The secondary confirmation module returns visual media 1 to the search module.

[0223] S420. The search module obtains visual media 2 based on the index of non-visual semantic subjects and visual media in the above search statement.

[0224] S421, The search module uses the intersection of visual media 1 and visual media 2 as the search result.

[0225] In this embodiment of the application, the search module searches for visual media (or visual media 2) that correspond to the filtered non-visual semantic subjects in the search statement input by the user from the constructed index. That is, it searches for visual media whose attributes match the non-visual semantic subject and uses it as visual media 2.

[0226] The search module then determines the intersection of visual media 2 and visual media 1 to obtain the target visual media. The attributes of the target visual media match the non-visual semantic subject of the search query, and the visual content of the target search media corresponds to the visual semantic subject of the search query. For example, if the search query is "sky photographed this year," then visual media 1 includes sky content, visual media 2 was captured this year, therefore the target visual media includes sky content, and the target visual media was captured this year.

[0227] S422. The search module sends the search results to the image library service module.

[0228] S423, The image gallery service module displays search results.

[0229] In this embodiment of the application, the image gallery service module can display search results to the user, for example: through the above... Figure 3A Interface 305, the above Figure 3A The search results are displayed in interface 308 as described above. Figure 3A As shown in interface 305, the search results include 239 images, but interface 305 only displays thumbnails of 8 of these images. To view all 239 images, the user can click the "More" option in interface 305. In response to this action, the phone displays... Figure 3A Interface 308.

[0230] In some embodiments, the gallery service module can sort and display target visual media based on their acquisition time in the search results. For example, the target visual media can be sorted in order of acquisition time from earliest to latest, so that those acquired earlier are displayed first.

[0231] In other embodiments, the gallery service module can sort the target visual media in the search results based on the similarity between the target visual media and the filtered search query. For example, the target visual media can be sorted in descending order of similarity, so that the target visual media with higher similarity is displayed earlier. That is, the similarity between the target visual media displayed earlier and the filtered search query is greater than or equal to the similarity between the target visual media displayed later and the filtered search query. Here, the similarity between the target visual media and the filtered search query can be similarity 1 or similarity 2, or a similarity calculated based on similarity 1 and similarity 2.

[0232] Optionally, if the filtered search statement belongs to whitelist 1 above, the similarity between the target visual media and the filtered search statement can be similarity 2, that is, the similarity between the visual media corresponding to the fine-tuning model and the filtered search statement. If the filtered search statement does not belong to whitelist 1 above, the similarity between the target visual media and the filtered search statement can be similarity 1, that is, the similarity between the visual media corresponding to the base model and the filtered search statement.

[0233] Optionally, the similarity calculated based on similarity 1 and similarity 2 can be the average of similarity 1 and similarity 2. Alternatively, a weighted sum of similarity 1 and similarity 2 can be calculated; this application does not limit the specific calculation method.

[0234] The following section will continue to describe one possible implementation process of the above S417, such as... Figure 8 As shown, the process may include S417a-S417g.

[0235] S417a, The secondary confirmation module determines whether the preset fine-tuning model vocabulary contains the filtered search statement.

[0236] In this embodiment, the secondary confirmation module can determine whether the preset fine-tuning model vocabulary contains a filtering search statement, so as to determine whether the filtering search statement corresponds to the branch of the base model or the branch of the fine-tuning model. That is, it determines whether to perform secondary confirmation through the module corresponding to the base model or through the module corresponding to the fine-tuning model.

[0237] If the preset fine-tuning vocabulary does not contain a filtering search statement, it indicates that the filtering search statement is the branch corresponding to the base model. In other words, it indicates that the candidate visual media is reconfirmed by the module corresponding to the base model in order to select visual media that better matches the filtering search statement from the candidate visual media. The reconfirmation module can execute S417b.

[0238] If the above-mentioned preset fine-tuning vocabulary contains a filtering search statement, it indicates that the filtering search statement is the branch corresponding to the fine-tuning model. That is, it indicates that the candidate visual media is reconfirmed by the module corresponding to the fine-tuning model in order to select the visual media that is more compatible with the filtering search statement from the candidate visual media. The reconfirmation module can execute S417f.

[0239] S417b, the secondary confirmation module inputs the text feature vector of the filtered search statement and the image feature vector of each candidate visual medium into the binary classification module, and obtains the Boolean value 1 corresponding to each candidate visual medium output by the binary classification module.

[0240] In this embodiment of the application, it is assumed that the number of candidate visual media is m. For example... Figure 9AAs shown, the secondary confirmation module inputs the image feature vectors of each of the m candidate visual media and the text feature vector of the filtered search statement into the binary classification model in the binary classification module. For each candidate visual media, the binary classification model determines the matching degree between the candidate visual media and the search statement based on the image feature vector of the candidate visual media and the text feature vector of the filtered search statement. This matching degree represents the degree of correlation between the candidate visual media and the filtered search statement; the higher the matching degree, the more relevant the candidate visual media and the filtered search statement are. Then, the binary classification model compares the matching degree between the candidate visual media and the filtered search statement with a classification threshold to determine whether the candidate visual media matches the search statement, obtaining a Boolean value (bool) 1 for the candidate visual media, and outputting the Boolean value 1 for each of the m candidate visual media. If the matching degree is greater than the classification threshold, it indicates that the candidate visual media matches the filtered search statement, and the Boolean value 1 for the candidate visual media is true. If the matching degree is less than or equal to the classification threshold, it indicates that the candidate visual media does not match the filter search statement, and the Boolean value 1 corresponding to the candidate visual media is false.

[0241] The binary classification model is a pre-trained model using a training sample set. It can determine whether image feature vectors and text feature vectors match. This binary classification model can be trained on the aforementioned mobile phone or other devices. The following will use the example of training the binary classification model on a mobile phone to introduce the training process of the binary classification model.

[0242] In some embodiments, if the preset fine-tuning model vocabulary contains a filtering search statement, it indicates that the filtering search statement corresponds to the branch of the fine-tuning model. Since the branch of the fine-tuning model does not include the binary classification module, the secondary confirmation module does not need to execute the above S417b.

[0243] If the word segmentation of the filtering search statement is not present in the above-mentioned preset fine-tuning model vocabulary, it indicates that the filtering search statement corresponds to the branch of the base model. The text feature vector of the filtering search statement in S417b can include the text feature vector of the filtering search statement corresponding to the base model. The image feature vector of the candidate visual media in S417b can include the image feature vector of the candidate visual media corresponding to the base model.

[0244] It is understandable that if the input parameters of the secondary confirmation module only include an image feature vector of the candidate visual media and a text feature vector of the filtering search statement, but not the image feature vectors of the candidate visual media corresponding to the fine-tuning model and the candidate visual media corresponding to the base model, then the image feature vector of the candidate visual media in S417b above is the image feature vector of the input candidate visual media, and the text feature vector of the filtering search statement is the text feature vector of the input filtering search statement.

[0245] For example, the training sample set of the binary classification model described above includes multiple training data sets, each of which includes a sample image and its corresponding descriptive text. The training sample set may include a positive sample training set (referred to as positive samples) and a negative sample training set (referred to as negative samples). The sample images in the training data included in the positive samples are matched with the descriptive text corresponding to the sample images. For example, as... Figure 9B The image shown, with the corresponding descriptive text, is of a little boy holding a basket. Because... Figure 9B The visual content depicted is a little boy holding a basket, which matches its corresponding descriptive text. Therefore, the image and its corresponding descriptive text can be used as a training data point in the positive sample.

[0246] Negative samples are training data where the sample images do not match the corresponding descriptive text. For example, ... Figure 9B The image shown has the descriptive text "flower". Because... Figure 9B The visual content expressed does not match the descriptive text. Therefore, the sample image and its corresponding descriptive text can be used as a training data point in the negative samples.

[0247] It should be noted that since users generally use natural language when searching for visual media, the descriptive text corresponding to the sample images is also in natural language to match the actual search situation.

[0248] To improve training accuracy, it's necessary to ensure a sufficient amount of training data in both positive and negative samples. Considering the low efficiency of manually obtaining positive and negative samples, mobile phones can automatically generate them. The following will use the aforementioned binary classification model, an image-text matching model, as an example to further explain the process of generating positive and negative samples. Figure 10 As shown, the process is as follows:

[0249] S1. The binary classification module obtains P sample data pairs. Each of the P sample data pairs includes a sample image and its corresponding descriptive text.

[0250] S2, the binary classification module extracts subtext from the descriptive text of each sample data.

[0251] For example, the subtext mentioned above may include descriptive phrases and / or nouns from the descriptive text. For example, subtext generally does not include adverbs, verbs, or other words in the descriptive text that do not correspond to actual visual content. The binary classification module can segment the descriptive text to identify descriptive phrases and nouns, thus obtaining the subtext within the descriptive text. For example, if the descriptive text is "a little boy sitting in a basket," the subtexts are "basket," "little boy," and "boy." Another example is "a little boy sitting in a basket sucking his thumb." After segmenting this descriptive text, the subtexts obtained are "basket," "boy," "little boy," and "little boy sucking his thumb," where "little boy sucking his thumb" can represent a descriptive phrase.

[0252] In some embodiments, the binary classification module can use the Han Language Processing (HanLP) package to segment the descriptive text corresponding to the sample images.

[0253] S3. For each sample image, the binary classification module generates a text set corresponding to the sample image based on the description text corresponding to the sample image and the subtext in the description text.

[0254] Here, the text set corresponding to the sample image represents the set of descriptions of the visual content corresponding to the sample image. The text set corresponding to the sample image can include the descriptive text corresponding to the sample image and each subtext within the descriptive text. For example, the descriptive text corresponding to the sample image is "a little boy holding a basket," and the subtexts within the descriptive text include "basket," "little boy," and "boy." Accordingly, the text set corresponding to this image is {basket, little boy, boy, little boy holding a basket}.

[0255] S4. The binary classification module randomly selects a text element from the text set corresponding to each sample image to obtain text element 1 corresponding to each sample image.

[0256] The text elements in the text collection can be subtext or descriptive text.

[0257] In some embodiments, the binary classification module can randomly select text elements from the text sets corresponding to each sample image according to a preset ratio. This preset ratio includes the proportion of selected text elements that are descriptive text and the proportion of selected text elements that are sub-text. For example, if the preset ratio is 8:2, the probability of selecting a text element that is descriptive text is 80%, and the probability of selecting a text element that is sub-text is 20%. Since the image encoder and text encoder are generally trained using the image and its corresponding text as a whole, the proportion of selected text elements that are descriptive text in the preset ratio will be greater than the proportion of selected text elements that are sub-text, ensuring the training effect of the model.

[0258] S5. For each sample image, the binary classification module takes the text element 1 corresponding to the P sample images as the sample text element corresponding to that sample image.

[0259] In this model, one sample image from a set of P sample images corresponds to P sample text elements. A sample image and its corresponding text element constitute one sample, resulting in multiple samples. Since the model needs to be trained using both positive and negative samples, it is necessary to distinguish between positive and negative samples after obtaining them, thus training a binary classification model using both positive and negative samples. The process of distinguishing between positive and negative samples will be described below.

[0260] In this embodiment, the binary classification module initially obtains P sample data pairs. All P sample data pairs are positive samples. However, training the binary classification model also requires negative samples. Therefore, the binary classification module can perform image-text matching between the P sample images and a text element (i.e., text element 1) from the corresponding text set of the P sample images. That is, for each sample image in the P sample images, the binary classification module can use all P text elements 1 as the corresponding sample text element. Each sample image and each text element 1 in the P text elements 1 form a sample, thereby obtaining P*P samples, increasing the number of samples, and thus achieving rapid sample generation. To train the binary classification model, the model needs to distinguish between positive and negative samples among the P*P samples. The process of distinguishing between positive and negative samples will be described below.

[0261] S6. For each sample text element corresponding to the sample image, the binary classification module determines whether the sample text element belongs to the text set corresponding to the sample image.

[0262] In this embodiment of the application, for each sample text element corresponding to the sample image, the binary classification module can determine whether the sample text element corresponds to the visual content of the sample image by judging whether the sample text element belongs to the text set corresponding to the sample image, thereby determining whether the sample text element and the sample image constitute a positive sample.

[0263] If the sample text element is in the text set corresponding to the sample image, it indicates that the sample text element corresponds to (i.e., matches) the visual content of the sample image. In other words, the sample consisting of the sample text element and the sample image is a positive sample. Therefore, the binary classification module can execute S7.

[0264] If the sample text element is not part of the text set corresponding to the sample image, it indicates that the sample text element does not correspond to the visual content of the image (i.e., they do not match). In other words, the sample formed by the sample text element and the sample image is not a positive sample. Therefore, the binary classification module can execute S8.

[0265] S7. The binary classification module treats the sample image and the sample text element as positive samples.

[0266] S8, the binary classification module treats the image and the text element of the sample as negative samples.

[0267] In some embodiments, after obtaining the sample text elements corresponding to each sample image, the binary classification module can generate a corresponding sample matrix. The elements in the i-th row of the sample matrix represent the sample text elements corresponding to the i-th sample image, and the elements in the j-th column of the sample matrix represent the text element 1 corresponding to the j-th sample image, which is a text element randomly selected from the text set corresponding to the j-th sample image.

[0268] For example, the sample images include image a, image b, image c, and image d. Figure 11A In the sample matrix shown, "1" represents the text element 1 corresponding to image a (i.e., the first sample image). Similarly, "6" represents the text element 1 corresponding to image d (i.e., the fourth sample image). The first row element 50 (i.e., 1, 2, 5, 6) represents the sample text element corresponding to image a. The second column element 51 (i.e., 2, 2, 2, 2) represents the text element 1 corresponding to image b.

[0269] In this sample matrix, a sample text element and the sample image corresponding to the row containing that text element constitute a sample. For example, as described above. Figure 11A The "1" in the first row and first column shown forms a sample with image a.

[0270] In this embodiment, the binary classification model generates a corresponding sample matrix by matching P sample images with P text elements 1. This allows the binary classification module to distinguish whether the sample text elements in the sample matrix belong to positive or negative samples, thus achieving rapid determination of positive and negative samples and improving the efficiency of positive and negative sample determination. Furthermore, by generating a label matrix, the comprehensiveness of image-text matching can be guaranteed, avoiding the omission of sample images or text elements 1, such as avoiding the situation where a certain sample image is not matched with a certain text element 1.

[0271] After obtaining the sample matrix, for each sample text element in the sample matrix, the binary classification module needs to determine whether the sample text element matches the sample image corresponding to the row containing the sample text element. To improve matching efficiency, the binary classification module can use the text intersection matrix 1 to match the sample matrix to obtain positive and negative samples. The text intersection matrix 1 represents the intersection of the text sets corresponding to the sample images. For example, the process of determining the text intersection matrix 1 may include:

[0272] The binary classification module determines the overlapping elements in the text sets corresponding to any two sample images. Then, the binary classification module generates the corresponding text intersection matrix 1. In this matrix, the s-th element in the t-th row represents the overlapping element in the text set between the t-th and s-th sample images.

[0273] For example, image a (such as Figure 11B The text set corresponding to image a is {1,2,3}, the text set corresponding to image b is {2,3,4}, the text set corresponding to image c is {5,6}, and the text set corresponding to image d is {6,7}. It should be understood that these numbers actually correspond to text elements (such as the subtext and descriptive text mentioned above).

[0274] Then, the binary classification module can determine the overlapping elements (i.e., the same elements) in the text sets corresponding to any two sample images from images a, b, c, and d. For example, the same elements in the text sets corresponding to images a and b are 2 and 3.

[0275] Afterwards, the binary classification module generates, as follows: Figure 11C The text intersection matrix 1 shown includes overlapping elements between the sample images and the text sets corresponding to each sample image (i.e., image a, image b, image c, and image d). Figure 11C The elements in the first row and first column of the text intersection matrix 1 shown represent the overlapping elements in the text sets between image a and image b (i.e., the text set corresponding to image a). The elements in the first row and second column represent the overlapping elements in the text sets between image a and image b. And so on. Figure 11C The element in the 4th row and 4th column represents the overlapping elements in the text set between image d and image d.

[0276] In some embodiments, after obtaining the text sets corresponding to each of the P sample images, the binary classification module can calculate the union of the text sets corresponding to each sample image. This union can include all text elements from all text sets. Then, the binary classification module can assign a number (as described above) to each text element in the union of the text sets, such that different text elements in one text set have different numbers, and the same text element in different text sets has the same number. By assigning numbers to text elements, the efficiency of image-text matching can be improved, thereby increasing the efficiency of generating positive and negative samples.

[0277] The process of determining the text intersection matrix 1 has been introduced above. The process of matching the text intersection matrix 1 with the sample matrix to determine the positive and negative samples will be introduced below.

[0278] The binary classification module intersects the text intersection matrix 1 and the sample matrix to obtain the text intersection matrix 2.

[0279] Then, for each intersection element in the text intersection matrix 2, the binary classification module determines whether the intersection element is empty.

[0280] If the intersection element is not empty, it indicates that the visual content of the intersection element matches that of the sample image corresponding to its row. Therefore, the binary classification module can confirm that the intersection element and the sample image corresponding to its row are positive samples.

[0281] If the intersection element is empty, it indicates that the visual content of the intersection element and the sample image corresponding to its row do not match. Therefore, the binary classification module can confirm that the element and the sample image corresponding to its row are negative samples, thus achieving rapid determination of positive and negative samples. The element corresponding to the sample image can also be referred to as the text corresponding to the sample image.

[0282] Optionally, the binary classification module can distinguish whether the intersection elements in the text intersection matrix 2 belong to positive or negative samples using 1 and 0, that is, to distinguish whether the sample text elements in the sample matrix belong to positive or negative samples. When the intersection elements in the text intersection matrix 2 are empty, the binary classification module can set the label corresponding to the intersection element to 0, that is, label the sample text element corresponding to the intersection element with 0.

[0283] If the intersection element in the text intersection matrix 2 is not empty, the binary classification module can set the label corresponding to that element to 1, that is, label the sample text element corresponding to that intersection element with label 1, thus obtaining the positive and negative sample label matrix, and realizing the labeling of the sample text elements in the sample matrix. The position of the intersection element in the text intersection matrix 2 is the same as the position of the sample text element corresponding to the intersection element in the sample matrix.

[0284] Based on this, the binary classification module can use the sample text element corresponding to label 0 and the image corresponding to the row where the sample text element is located as a training data in the negative sample, and use the sample text element corresponding to label 1 and the image corresponding to the row where the sample text element is located as a training data in the positive sample, so as to realize the batch determination of training data for multiple positive and negative samples.

[0285] For example, such as Figure 11D As shown, the intersection of text intersection matrix 1 and sample matrix is ​​obtained as text intersection matrix 2. Then, the binary classification module can determine whether the intersection elements in text intersection matrix 2 are empty, thereby determining the positive and negative sample label matrix corresponding to the sample matrix. Figure 11DThe label in the first row and first column of the positive and negative sample label matrix is ​​1. The element in the sample matrix corresponding to this label is the sample text element "1" in the first row and first column. This "1" and image a together form a positive sample. Figure 11D The label in the second row and first column of the positive and negative sample label matrix is ​​0. The sample text element in the second row and first column of the sample matrix corresponding to this label is "1". This "1" and the image b together form a negative sample.

[0286] It should be noted that, generally speaking, the elements on the diagonal of the sample matrix and the sample images corresponding to their respective rows constitute the positive samples.

[0287] In this embodiment, the binary classification module obtains a text intersection matrix 2 by intersecting the text intersection matrix 1 and the sample matrix. The positive and negative samples can be quickly determined by checking whether the elements in the text intersection matrix 2 are empty, thereby improving the efficiency of determining positive and negative samples and ensuring the comprehensiveness and completeness of the positive and negative sample distinction, so as to enable the model to be trained quickly.

[0288] The process of determining positive and negative samples has been introduced above. The process of training a binary classification model using positive and negative samples will be introduced below.

[0289] S9. The binary classification module uses positive and negative samples to train the image-text matching model, resulting in the trained image-text matching model.

[0290] For example, the image-text matching model can be a multilayer perceptron (MLP) model. Figure 11E As shown, the binary classification module uses a text encoder to process the sample text elements in both positive and negative samples, obtaining the text feature vectors of the sample text elements. Simultaneously, the binary classification module uses an image encoder to process the sample images in both positive and negative samples, obtaining the image feature vectors of the sample images. Then, for each sample in both positive and negative samples, the binary classification module concatenates the text feature vectors of the sample text elements and the image feature vectors of the sample image. The binary classification module then inputs the concatenated text feature vector and image feature vector into the MLP model to train the MLP model. During training, the binary classification module can use a loss function to test the difference between the predicted values ​​and actual values ​​output by the trained MLP model. The predicted value indicates whether the predicted sample image matches the text. The actual value indicates the actual matching situation between the sample image and the text.

[0291] In some embodiments, the loss function described above can be a focal loss function. Specifically, the focal loss function can be FL(p t )=-at1(1-pt ) γ log(p t Here, pt represents the probability that the image-text matching model predicts a match between a sample image and its corresponding text, i.e., the probability that the predicted sample image and its corresponding text are positive samples. at1 is a factor that adjusts the weights of positive and negative samples, which can be set according to the number of positive and negative samples to adjust for imbalances in the number of positive and negative samples; for example, at1 is 0.1. γ is an adjustment factor used to reduce the loss contribution of easily distinguishable samples. It should be understood that the larger pt is, the smaller the value of the loss function.

[0292] Optionally, the image encoder described above can be a stacked autoencoder, and the text encoder can be a counting vector.

[0293] It should be noted that the focus loss function described above is only one example of a loss function. This loss function can also be other types of loss functions, such as a BCE-type loss function. Furthermore, the MLP model described above is only one example of an image-text matching model. This image-text matching model can also be other deep learning models, and this application does not limit it.

[0294] In some embodiments, the aforementioned positive and negative sample label matrix can be used when calculating the value of the loss function.

[0295] In some embodiments, the above-described image-text matching model for secondary verification of candidate visual media is merely an example; the image-text matching model can also be directly used to search for visual media. For instance, after a user enters a search query, the image-text matching model can directly utilize the text feature vector of the search query (or the filtered search query) and the image feature vectors of various visual media on the mobile phone to determine whether the visual media matches the search query and obtain the corresponding Boolean value.

[0296] The above describes the process of determining whether candidate visual media matches the filter search statement using a binary classification module. The following section will continue to describe the process of determining whether candidate visual media matches the filter search statement using a dynamic threshold module, in conjunction with S417c.

[0297] S417c, the binary classification module inputs the filtering search statement and the similarity between each candidate visual media and the filtering search statement into the dynamic threshold module, and obtains the Boolean value 2 corresponding to each candidate visual media output by the dynamic threshold module.

[0298] In this embodiment, generally, the similarity threshold 1 corresponding to different lengths of search statements is different. The longer the search statement, the higher the similarity between the visual content of the visual media and the search statement needs to be, and correspondingly, the similarity threshold 1 needs to be higher. Therefore, the secondary confirmation module can use the dynamic threshold module to determine the dynamic threshold matching the length of the filtering search statement, that is, to determine the similarity threshold 1 corresponding to the filtering search statement, and then use the similarity threshold 1 corresponding to the filtering search statement to determine the Boolean value 2 corresponding to each candidate visual media.

[0299] For example, such as Figure 12 As shown, the process by which the dynamic thresholding module determines the Boolean value 2 corresponding to each candidate visual media can include: First, the dynamic thresholding module can determine the length of the filtering search statement through the filtering search statement. Then, based on the length of the filtering search statement, the dynamic thresholding model, combined with t = parameter 1 * L + parameter 2, determines the similarity threshold 1 corresponding to the filtering search statement. Here, parameter 1 and parameter 2 are pre-set parameters; for example, parameter 1 is 0.05 and parameter 2 is 0.33. Optionally, the dynamic thresholding module can also determine the length of the filtering search statement through the word segmentation results of the filtering search statement.

[0300] Subsequently, for each of the m candidate visual media, the dynamic thresholding module compares the similarity between the candidate visual media and the filter search statement with the similarity threshold 1 corresponding to the filter search statement. If the similarity between the candidate visual media and the filter search statement is less than the similarity threshold 1, it indicates that the visual content of the candidate visual media has a low degree of similarity to the filter search statement, and the dynamic thresholding module determines that the Boolean value 2 corresponding to the candidate visual media is false.

[0301] If the similarity between the candidate visual media and the filter search statement is greater than or equal to the similarity threshold 1, it indicates that the visual content of the candidate visual media is highly similar to the filter search statement, and the dynamic threshold module can determine that the Boolean value 2 corresponding to the candidate visual media is true.

[0302] Optionally, as described above Figure 12 As shown, the range of values ​​for t can be greater than or equal to parameter 3, meaning min(t, parameter 3), and the range of values ​​for t can be less than or equal to parameter 4, meaning max(t, parameter 4). Parameters 3 and 4 are preset values.

[0303] In some embodiments, since the dynamic threshold module corresponding to S417c belongs to the branch corresponding to the base model, the similarity between the candidate visual media and the filter search statement in S417c includes the similarity between the candidate visual media and the filter search statement corresponding to the base model (i.e., similarity 1). Accordingly, the dynamic threshold module can determine whether the similarity 1 between the candidate visual media and the filter search statement is less than the similarity threshold 1. If the similarity 1 is greater than or equal to the similarity threshold 1, the dynamic threshold module can determine that the Boolean value 2 corresponding to the candidate visual media is true. If the similarity 1 is less than the similarity threshold 1, the dynamic threshold module can determine that the Boolean value 2 corresponding to the candidate visual media is false.

[0304] Understandably, if the input parameters of the secondary confirmation module only include a similarity between the candidate visual media and the filter search statement, and do not include the similarity between the candidate visual media and the filter search statement corresponding to the fine-tuning model and the candidate visual media and the filter search statement corresponding to the base model, then the image feature vector of the candidate visual media in S417c above is the similarity between the input candidate visual media and the filter search statement.

[0305] In addition, the similarity between the aforementioned candidate visual media and the filtered search statement can also be determined by other multimodal models, and this application does not limit it.

[0306] The above describes the process of determining the Boolean value corresponding to candidate visual media using the dynamic threshold module. The following section will continue to introduce the process of determining whether candidate visual media matches the filter search statement using the label confirmation module, in conjunction with S417d.

[0307] S417d, the secondary confirmation module inputs the word segmentation results of the above filtering search statement, the tag 1 included in the filtering search statement, and the tag 2 of each candidate visual medium into the tag confirmation module, and obtains the Boolean value 3 corresponding to each candidate visual medium output by the tag confirmation module.

[0308] In this embodiment, the secondary confirmation module can use the tag confirmation module to determine whether tag 2 of the candidate visual media exists in the word segmentation of the filtering search statement or tag 1 included in the filtering search statement, so as to determine whether the candidate visual media matches the filtering search statement, thereby determining the Boolean value 3 corresponding to the candidate visual media. For example, as shown... Figure 13 As shown, the process by which the label confirmation model determines the Boolean value 3 corresponding to the candidate visual media can include:

[0309] First, for each of the m candidate visual media, the tag confirmation module can determine whether the tag 2 of the candidate visual media includes the word segmentation result of the filtered search statement, and determine whether the tag 2 of the candidate visual media includes the tag 1 included in the filtered search statement, that is, determine whether the candidate visual media includes the visual content corresponding to the filtered search statement.

[0310] If the candidate visual media's tag 2 includes at least one word of the filtered search query, or if the candidate visual media's tag 2 includes at least one tag 1 included in the filtered search query, it indicates that the candidate visual media matches the filtered search query. The tag confirmation module can determine that the candidate visual media may be the visual media needed by the user. Therefore, the tag confirmation module can determine the Boolean value 3 corresponding to the candidate visual media as true. For example, if the filtered search query's tag 1 includes "child" and "flower," and the filtered search query's word segmentation results include the words "child," "holding," and "flower," then if the candidate visual media's tag 2 includes "child," "holding," or "flower," or if the candidate visual media's tag 2 includes "child" or "flower," then the Boolean value 3 corresponding to the candidate visual media is determined to be true.

[0311] If the tag 2 of the above candidate visual media does not include all the words of the filter search statement, and the tag 2 of the candidate visual media does not include all the tags 1 included in the filter search statement, it indicates that the candidate visual media does not match the filter search statement, and the candidate visual media may not be the visual media required by the user. Therefore, the tag confirmation module can determine the Boolean value 3 corresponding to the candidate visual media as false, thereby obtaining the Boolean value 3 of m candidate visual media.

[0312] Optionally, to improve the accuracy of search results, the tag confirmation module may determine that the Boolean value 3 corresponding to the candidate visual media is true if the tag 2 of the candidate visual media includes all the words of the filtered search statement, or includes all the tags 1 included in the filtered search statement.

[0313] If the candidate visual media's tag 2 does not include at least one word of the filtering search statement, and the filtering search statement does not include at least one tag 1, then the Boolean value 3 corresponding to the candidate visual media is determined to be false.

[0314] The above describes the process by which the secondary confirmation module uses the tag confirmation module to determine the Boolean value 3 corresponding to each candidate visual media. The following will continue to describe the process by which the secondary confirmation module uses the whitelist threshold module to determine whether the candidate visual media matches the filtering search statement.

[0315] S417e, the secondary confirmation module inputs the text feature vector of the filtered search statement and the similarity between the candidate visual media and the filtered search statement into the whitelist threshold module, and obtains the Boolean value 4 corresponding to each candidate visual media output by the whitelist threshold module.

[0316] In this embodiment, the secondary confirmation module can determine whether the filtered search statement hits the dictionary through the whitelist threshold module, thereby determining whether a similarity threshold 2 corresponding to the filtered search statement exists, and accurately determining the similarity threshold. For example, the whitelist threshold module can determine whether the dictionary contains the filtered search statement. If it does, the whitelist threshold module takes effect, using the similarity threshold corresponding to the filtered search statement in the dictionary as the similarity threshold 2, and thus using the similarity threshold 2 to filter candidate visual media matching the filtered search statement. If it does not exist, the whitelist threshold module does not take effect, meaning there is no need to use the whitelist threshold module to determine the Boolean value 4 corresponding to each candidate visual media. The dictionary includes a key and its corresponding value. The key represents a preset search text, and the value is the similarity threshold corresponding to that preset search text. The key in the dictionary can be search text frequently entered by the user, i.e., a pre-tested search statement.

[0317] In some embodiments, the branches corresponding to the fine-tuning model and the base model have corresponding dictionaries. For example... Figure 14A As shown, when the filtering search statement is the branch corresponding to the base model, the whitelist threshold module can determine whether the dictionary corresponding to the base model includes the filtering search statement, that is, whether the dictionary corresponding to the base model has the same key as the filtering search statement.

[0318] If the dictionary corresponding to the base model includes the filter search statement, it indicates that the filter search statement hits the dictionary corresponding to the base model, that is, the dictionary corresponding to the base model contains the same key as the filter search statement. Then the whitelist threshold module can use the value corresponding to the filter search statement in the dictionary corresponding to the base model as the similarity threshold 2.

[0319] Subsequently, for each of the m candidate visual media, the whitelist threshold module compares the similarity between the candidate visual media corresponding to the base model and the filtering search threshold with a similarity threshold 2. If the similarity between the candidate visual media corresponding to the base model and the filtering search threshold is less than the similarity threshold 2, the whitelist threshold module determines that the corresponding Boolean value 4 for the candidate visual media is false. If the similarity is greater than or equal to the similarity threshold 2, it indicates that the visual content of the candidate visual media is highly similar to the filtering search statement, and the whitelist threshold module determines that the corresponding Boolean value for the candidate visual media is true.

[0320] Accordingly, when the filtering search statement corresponds to the branch of the base model, S418 above can be: for each candidate visual medium, if the Boolean value corresponding to the candidate visual medium is true, the secondary confirmation module (such as the fusion module in the secondary confirmation module) can treat the candidate visual medium as visual medium 1. The Boolean values ​​corresponding to the candidate visual medium include Boolean value 1, Boolean value 2, Boolean value 3, and Boolean value 4. That is, the secondary confirmation module can take the union of Boolean values ​​1, 2, 3, and 4 corresponding to the candidate visual medium to obtain the Boolean value corresponding to the candidate visual medium.

[0321] If all Boolean values ​​corresponding to the candidate visual medium are false, it means that Boolean values ​​1, 2, 3 and 4 corresponding to the candidate visual medium are all false, and the secondary confirmation module (such as the fusion module in the secondary confirmation module) can choose not to use the candidate visual medium as visual medium 1.

[0322] It should be noted that since both the whitelist threshold module and the dynamic threshold module confirm the similarity threshold corresponding to the filtering search statement, the similarity threshold determined by the whitelist threshold module is more accurate. Therefore, when the whitelist threshold module is in effect, that is, when the Boolean value corresponding to the candidate visual media is determined using the similarity threshold 2 corresponding to the filtering search statement, the dynamic threshold module may not be in effect. The secondary confirmation module may not use the similarity threshold 1 corresponding to the filtering search statement to determine the Boolean value corresponding to the candidate visual media, thus avoiding unnecessary determination of similarity thresholds and unnecessary screening of candidate visual media.

[0323] In some embodiments, the binary classification module, dynamic threshold module, label confirmation module, and whitelist threshold module in the secondary confirmation module can determine the Boolean value corresponding to the candidate visual media in parallel or sequentially. Regardless of whether the determination is parallel or sequential, if one module in the secondary confirmation module determines that the Boolean value corresponding to the candidate visual media is true, other modules do not need to continue determining the Boolean value corresponding to that candidate visual media. That is, there is no need to input the relevant information of that candidate visual media into other models, avoiding unnecessary Boolean value determination and ensuring low transmission costs.

[0324] Furthermore, the aforementioned secondary confirmation module, including a binary classification module, a dynamic threshold module, a label confirmation module, and a whitelist threshold module, is merely an example. The secondary confirmation module can include one or more of these modules to improve the search efficiency for visual media. For instance, if the secondary confirmation module includes one of these modules, such as a binary classification module, a dynamic threshold module, a label confirmation module, and a whitelist threshold module, and this module includes a binary classification module, the corresponding Boolean value for the candidate visual media can include Boolean value 1. As another example, if the secondary confirmation module includes two of these modules, such as a binary classification module, a dynamic threshold module, a label confirmation module, and a whitelist threshold module, and this module includes both a binary classification module and a dynamic threshold module, the corresponding Boolean value for the candidate visual media can include Boolean value 1 and Boolean value 2. As yet another example, if the secondary confirmation module includes three of these modules, such as a binary classification module, a dynamic threshold module, a label confirmation module, and a whitelist threshold module, and this module includes a binary classification module, a dynamic threshold module, and a label confirmation module, the corresponding Boolean value for the candidate visual media can include Boolean value 1, Boolean value 2, and Boolean value 3.

[0325] Furthermore, the content included in the aforementioned candidate visual media image information and filtering search statement information, i.e., the input parameters of the secondary confirmation module, is merely an example and can be adaptively set according to the modules included in the secondary confirmation module. For example, the secondary confirmation module includes a binary classification model, the aforementioned candidate visual media image information may include the image feature vectors of the candidate visual media, and the aforementioned filtering search statement information may include the text feature vectors of the filtering search statement.

[0326] The above describes how, when the filtered search statement corresponds to a branch of the base model, the secondary confirmation module can sequentially use the binary classification module, dynamic threshold module, label confirmation module, and whitelist threshold module to determine the Boolean value corresponding to the candidate visual media. The following will continue to describe how, when the filtered search statement corresponds to a branch of the fine-tuning model, the secondary confirmation module can sequentially use the label confirmation module and whitelist threshold module to determine the Boolean value corresponding to the candidate visual media.

[0327] S417f, the secondary confirmation module inputs the word segmentation result of the above filtering search statement, the tag 1 included in the filtering search statement, and the tag 2 of each candidate visual media to the tag confirmation module, and obtains the Boolean value 5 corresponding to each candidate visual media output by the tag confirmation module.

[0328] The implementation process of S417f can be referred to the implementation process of S417d mentioned above, and will not be repeated here.

[0329] S417g, the secondary confirmation module inputs the similarity between the filtered search statement and the candidate visual media to the whitelist threshold module, and obtains the Boolean value 4 corresponding to each candidate visual media output by the whitelist threshold module.

[0330] The implementation process of S417g can be referenced from the implementation process of S417e described above. For example... Figure 14B As shown, when the filtering search statement corresponds to the branch of the fine-tuning model, the whitelist threshold module can determine whether the dictionary corresponding to the fine-tuning model includes the filtering search statement, that is, whether the dictionary corresponding to the fine-tuning model has the same key as the filtering search statement, thereby determining whether the whitelist threshold module is effective.

[0331] After the whitelist threshold module takes effect, the secondary confirmation module can compare the similarity between the candidate visual media corresponding to the fine-tuning model and the filter search statement with the value corresponding to the filter search statement in the dictionary corresponding to the fine-tuning model to determine the boolean value corresponding to the candidate visual media.

[0332] Accordingly, when the filtering search statement corresponds to the branch of the fine-tuning model, S418 above can be: for each candidate visual medium, if the boolean value corresponding to the candidate visual medium is true, the secondary confirmation module (such as the fusion module in the secondary confirmation module) can treat the candidate visual medium as visual medium 1. The boolean values ​​corresponding to the candidate visual medium include the aforementioned boolean value 5 and boolean value 6. That is, the secondary confirmation module can take the union of the boolean values ​​5 and 6 corresponding to the candidate visual medium to obtain the boolean value corresponding to the candidate visual medium.

[0333] In this embodiment, after obtaining candidate visual media, the mobile phone uses a secondary confirmation module to further determine the Boolean value corresponding to the candidate visual media in order to determine whether the candidate visual media matches the filter search statement. In this way, candidate visual media that matches the filter search statement can be selected from the candidate visual media to obtain the corresponding search results, ensuring the accuracy of the search results and thus ensuring user satisfaction.

[0334] It should be noted that the operations performed by the above-described module or model are merely examples, and the operations performed by the above-described module could also be performed by other modules in the mobile phone; this application does not limit them. Furthermore, the operations performed by the above-described module or model are actually performed by the mobile phone itself.

[0335] In some embodiments, when determining candidate visual media or performing secondary confirmation on candidate visual media, the mobile phone may not first filter non-visual semantic subjects in the search statement, but directly use the text feature vector of the search statement to determine candidate visual media or perform secondary confirmation on candidate media.

[0336] In some embodiments, the search and storage of the aforementioned visual media are carried out under the authorization of the user, including but not limited to notifying and reminding the user to read the relevant user agreement (notification) and sign the agreement (authorization) which includes the authorization of relevant user information before the user uses the function.

[0337] In some embodiments, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described above.

[0338] In some embodiments, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described above.

[0339] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated.

[0340] It should be understood that the term "embodiment" used throughout the specification means that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, various embodiments throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0341] It should also be understood that in this application, “when…”, “if” and “if” all refer to the UE or base station taking corresponding actions under certain objective circumstances, and are not time-limited, nor do they require the UE or base station to perform a judgment action, nor do they imply any other limitations.

[0342] Those skilled in the art will understand that the various numerical designations such as "first," "second," etc., involved in this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application, nor do they indicate the order of sequence.

[0343] In this application, the use of singular pronouns to denote "one or more" rather than "one and only one," unless otherwise specified. In this application, unless otherwise specified, "at least one" is intended to mean "one or more," and "more than" is intended to mean "two or more."

[0344] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. Here, A can be singular or plural, and B can be singular or plural.

[0345] In this document, the terms "at least one of..." or "at least one of..." refer to all or any combination of the listed items. For example, "at least one of A, B, and C" can mean: A exists alone, B exists alone, C exists alone, A and B exist simultaneously, B and C exist simultaneously, and A, B, and C exist simultaneously. A can be singular or plural, B can be singular or plural, and C can be singular or plural.

[0346] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0347] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0348] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0349] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0350] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0351] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0352] The same or similar parts between the various embodiments in this application can be referred to mutually. In the various embodiments of this application, and the various implementation methods / methods / implementations within each embodiment, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments and between the various implementation methods / methods / implementations within each embodiment are consistent and can be mutually referenced. The technical features in different embodiments and the various implementation methods / methods / implementations within each embodiment can be combined according to their inherent logical relationships to form new embodiments, implementation methods, methods, or implementation approaches. The above-described embodiments of this application do not constitute a limitation on the scope of protection of this application.

[0353] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims. In conclusion, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A visual media search method, suitable for an electronic device, characterized by, The method comprises: acquiring P sample data pairs, wherein each of the P sample data pairs comprises a sample image and corresponding description text; for each sample image, selecting a first text element from a text set corresponding to the sample image to obtain a first text element corresponding to the sample image; wherein the text set corresponding to the sample image comprises the description text corresponding to the sample image and a first subtext in the description text; performing image-text matching on the P sample images and the first text elements corresponding to the P sample images to obtain a P*P sample matrix; determining positive and negative samples according to the sample matrix, the text set corresponding to the sample image and the sample image, comprising: performing intersection operation on a first text intersection matrix and the sample matrix to obtain a second text intersection matrix; the first text intersection matrix is determined based on the same elements in the text sets corresponding to any two sample images; in a case where a s-th intersection element in a t-th row in the second text intersection matrix is empty, determining that an s-th intersection element in the t-th row in the sample matrix and a t-th sample image belong to a negative sample; in a case where the s-th intersection element in the t-th row in the second text intersection matrix is not empty, determining that the s-th intersection element in the t-th row in the sample matrix and the t-th sample image belong to a positive sample; training to obtain an image-text matching model based on the positive and negative samples; displaying a first interface; the first interface comprises a search box; receiving a search statement input in the search box; the search statement comprises one or more subtexts; displaying a search result corresponding to a first visual media; the first visual media represents a visual media determined by the image-text matching model and matching the search statement.

2. The method of claim 1, wherein, The image-text matching on the P sample images and the first text elements corresponding to the P sample images to obtain a P*P sample matrix comprises: for each sample image, taking the first text elements corresponding to the P sample images as sample text elements corresponding to the sample image; generating the sample matrix based on the sample text elements corresponding to each sample image; wherein an i-th row element in the sample matrix represents each sample text element corresponding to an i-th sample image; any sample text element in the sample matrix and a sample image corresponding to a row where the sample text element is located form a sample.

3. The method of claim 1, wherein, The determining of the positive and negative samples according to the sample matrix, the text set corresponding to the sample image and the sample image comprises: for each sample text element in the sample matrix, in a case where the sample text element belongs to a text set corresponding to a sample image corresponding to a row where the sample text element is located, determining that the sample text element and the sample image corresponding to the row where the sample text element is located belong to a positive sample; in a case where the sample text element does not belong to the text set corresponding to the sample image corresponding to the row where the sample text element is located, determining that the sample text element and the sample image corresponding to the row where the sample text element is located belong to a negative sample.

4. The method according to any one of claims 1 to 3, characterized in that, Before the image-text matching of the P sample images and the first text elements corresponding to the P sample images to obtain the P*P sample matrix, the method further comprises: determining the union set of the text sets corresponding to the respective sample images; allocating numbers to the respective text elements in the union set of the text sets; the image-text matching of the P sample images and the first text elements corresponding to the P sample images to obtain the P*P sample matrix comprises: image-text matching of the numbers corresponding to the P sample images and the first text elements corresponding to the P sample images to obtain the P*P sample matrix.

5. The method according to any one of claims 1 to 3, characterized in that, The selecting of the first text element from the text set corresponding to each sample image comprises: randomly selecting text elements from the text set corresponding to each sample image according to a preset proportion; wherein the preset proportion comprises a proportion of the selected first text elements being description text and a proportion of the selected first text elements being first sub-text.

6. The method according to any one of claims 1 to 3, characterized in that, The first visual media representation is determined by an image-text matching model, and the visual content of the candidate visual media matches the search statement; the similarity between the candidate visual media representation and the search statement is greater than a first threshold.

7. The method according to any one of claims 1 to 3, characterized in that, The first visual media representation is determined by an image-text matching model, and the visual content of the visual media matches the filtered search statement; the filtered search statement is obtained by filtering non-visual semantic subjects in the search statement, and the non-visual semantic subjects do not correspond to visual content.

8. An electronic device, comprising: The electronic device comprises a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display images generated by the processor, the memory is used to store computer program code, and the computer program code comprises computer instructions; when the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1 to 7.

9. A computer storage medium, characterized in that The computer instructions are executed on the electronic device, so that the electronic device executes the method as claimed in any one of claims 1 to 7.

Images