CROSS-MODAL SEARCH METHOD, CROSS-MODAL SEARCH APPARATUS, COMPUTER DEVICE, AND COMPUTER PROGRAM

The cross-modal search method and device address the limitations of single-dimensional searches by integrating content and semantic information, enhancing search efficiency and comprehensiveness through multi-dimensional data retrieval.

JP2024541145AInactive Publication Date: 2024-11-07TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024532539
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-03-07
Filing Date
2022-11-29
Publication Date
2024-11-07
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Current search technologies lack support for cross-modal searches, resulting in low efficiency and incomplete search results due to single-dimensional searches.

Method used

A cross-modal search method and device that performs searches across multiple dimensions by matching content and semantic information, merging results to enhance search efficiency and comprehensiveness.

Benefits of technology

The method and device improve search efficiency and diversity by supporting cross-modal searches, enabling faster and more accurate retrieval of relevant data across different modalities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541145000001_ABST
    Figure 2024541145000001_ABST
Patent Text Reader

Abstract

The present application discloses a cross-modal search method and a related device, the cross-modal search method including: acquiring first modal data; searching a second modal database based on content information of the first modal data to acquire a first set, the first set including at least one second modal data matching the content information of the first modal data; searching a second modal database based on semantic information of the first modal data to acquire a second set, the second set including at least one second modal data matching the semantic information of the first modal data; and merging the first set and the second set to acquire a cross-modal search result corresponding to the first modal data. The present application can improve the efficiency of the cross-modal search and the diversity and comprehensiveness of the cross-modal search result.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application filed with the China Patent Office on March 7, 2022, bearing application number 2022102220890 and entitled "Cross-modal search method and related device," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of computers, and more particularly to cross-modal search techniques. [Background technology]

[0003] With the rapid development of Internet technology, searching for data through computer devices has become an essential function for people in productive life and work-learning. Through practice, it has been found that current searching generally has problems such as not supporting cross-modal searching, single search dimension, low search efficiency, incomplete search results, etc. Summary of the Invention [Problem to be solved by the invention]

[0004] The embodiments of the present application provide a cross-modal search method and related device that can improve the efficiency of cross-modal search and the diversity and comprehensiveness of cross-modal search results. [Means for solving the problem]

[0005] One aspect of an embodiment of the present application provides a method for cross-modal search executed by a computing device, comprising: acquiring first modal data; searching in a second modal database based on content information of the first modal data to obtain a first set, the first set including at least one second modal data matching the content information of the first modal data; searching in a second modal database based on semantic information of the first modal data to obtain a second set, the second set including at least one second modal data matching the semantic information of the first modal data; Merging the first and second sets to obtain cross-modal search results corresponding to the first modal data.

[0006] An aspect of an embodiment of the present application provides another cross-modal search method executed by a computing device, comprising: displaying a session interface for the social session; displaying a session record details interface in response to checking against a historical session record for the social session, the session record details interface including second modal data in the historical session record for the social session; and in response to first modal data input into the session record details interface, outputting cross-modal search results corresponding to the first modal data, the cross-modal search results being obtained by employing a cross-modal search method according to an embodiment of the present application.

[0007] According to one aspect of the present invention, there is provided a cross-modal search device, an acquisition module for acquiring first modal data; a search module for searching in a second modal database based on content information of the first modal data to obtain a first set, the first set including at least one second modal data matching the content information of the first modal data; The search module is further used to search in a second modal database based on the semantic information of the first modal data to obtain a second set, the second set including at least one second modal data matching the semantic information of the first modal data; A merge module is included for merging the first and second sets to obtain cross-modal search results corresponding to the first modal data.

[0008] According to an aspect of the present invention, there is provided another cross-modal search apparatus, a display module for displaying a session interface for the social session; The display module is further adapted to display a session record detail interface in response to checking against the historical session record of the social session, the session record detail interface including the second modal data in the historical session record of the social session; and an output module for outputting cross-modal search results corresponding to the first modal data input into the session record details interface in response to the first modal data, the cross-modal search results being obtained by employing the cross-modal search method of an embodiment of the present application.

[0009] One aspect of an embodiment of the present application provides a computing device including a processor, a memory, and a network interface, the processor being connected to the memory and the network interface, the network interface being used to provide a network communication function, the memory being used to store program code, and the processor being used to call the program code, thereby executing a cross-modal search method in an embodiment of the present application.

[0010] One aspect of the present application provides a computer-readable storage medium having a computer program stored therein, the computer program including program instructions that, when executed by a processor, perform a cross-modal search method in the present application.

[0011] An aspect of an embodiment of the present application provides a computer program product, the computer program product including a computer program or computer instructions, which, when executed by a processor, implements a cross-modal search method provided by an aspect of an embodiment of the present application.

[0012] In the embodiment of the present application, based on the content information of the first modal data, second modal data matching the content information of the first modal data can be searched for, and based on the semantic information of the first modal data, second modal data matching the semantic information of the first modal data can be searched for. In this way, the embodiment of the present application not only supports cross-modal search, but also supports comprehensive search from two dimensions, namely content and semantic, respectively, so that the dimension covered by the search is no longer single. In addition, the second modal data searched in two dimensions can be merged as a cross-modal search result, and search results in multiple dimensions can be obtained in one search process, thereby improving the search efficiency of the cross-modal search. In addition, since the cross-modal search results are obtained by merging search results in two dimensions, the cross-modal search results are more diversified and more comprehensive. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is an architecture diagram of a cross-modal search system provided by an embodiment of the present application. [Diagram 2] 1 is a first schematic flowchart of a cross-modal search method provided by an embodiment of the present application; [Diagram 3]2 is a second schematic flowchart of a cross-modal search method provided by an embodiment of the present application; [Figure 4a] FIG. 2 is a structural schematic diagram of a first modal processing network in a cross-modal search model provided by an embodiment of the present application. [Figure 4b] FIG. 2 is a structural schematic diagram of a second modal processing network in a cross-modal retrieval model provided by an embodiment of the present application. [Diagram 5] FIG. 1 is a schematic diagram of training a cross-modal retrieval model provided by an embodiment of the present application. [Figure 6] 1 is a schematic flowchart of a cross-modal search algorithm provided by an embodiment of the present application. [Figure 7] 1 is a third schematic flowchart of a cross-modal search method provided by an embodiment of the present application; [Figure 8a] FIG. 2 is a schematic diagram of the operation of checking against historical session records provided by an embodiment of the present application; [Figure 8b] FIG. 1 is a schematic diagram of the operation of cross-modal search provided by an embodiment of the present application. [Figure 8c] FIG. 1 is a schematic diagram of the effect of outputting cross-modal search results provided by an embodiment of the present application; [Figure 9] FIG. 2 is a structural schematic diagram of a cross-modal search device provided by an embodiment of the present application; [Figure 10] FIG. 2 is a structural schematic diagram of another cross-modal search device provided by an embodiment of the present application; [Figure 11] FIG. 2 is a structural schematic diagram of a computing device provided by an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the technical means in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application, and obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments that can be obtained by those skilled in the art without spending creative labor shall fall within the scope of protection of the present application.

[0015] In order to better understand the implementation of the embodiments of the present application, the following first introduces some potentially relevant terms and concepts related to the embodiments of the present application.

[0016] Chat Photo Wall: This is a display page for all photos sent and received within each chat in an application program (APP).

[0017] Multimodal learning: refers to mapping two different modal data into the same feature space (e.g., semantic space) so that the two different modal data are semantically related, where modal data with similar semantics have similar features in the feature space, and the two different modal data may be, for example, images, text.

[0018] Based on the above terms and concepts, the following introduces the architecture of a cross-modal search system provided by an embodiment of the present application in conjunction with the accompanying drawings.

[0019] Referring to Fig. 1, Fig. 1 is an architecture schematic diagram of a cross-modal search system provided by an embodiment of the present application. As shown in Fig. 1, the architecture diagram includes a database 101 and a cross-modal search device 102. The cross-modal search device 102 can establish a communication connection with the database 101 through a wired or wireless manner, and the database 101 may be a local database of the cross-modal search device 102 or a cloud database accessible by the cross-modal search device 102. The cross-modal search device 102 may specifically be a computer device such as a server or a terminal.

[0020] In the embodiments of the present application, the server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, etc., but is not limited thereto. The terminal may be a device such as a smartphone, a tablet computer, a smart wearable device, a smart voice interactive device, a smart home appliance, a personal computer, an in-vehicle terminal, etc., but is not limited thereto.

[0021] The database 101 may include a second modal database and a second modal feature library associated with the second modal database, and the second modal database is used to store the second modal data and attribute information of the second modal data. Here, in one embodiment, the attribute information of the second modal data may be information included in the second modal data itself, for example, the second modal data is an image, and the attribute information may be text in the image. In another embodiment, the attribute information of the second modal data may further be information associated with the second modal data, for example, the second modal data is an image, and the attribute information may be a category label annotated to the image. The second modal feature library is used to store semantic features of the second modal data, and each semantic feature of the second modal data is provided with a feature index, which can assist in quickly searching the second modal data from the second modal database.

[0022] The cross-modal search device 102 searches for second modal data according to the first modal data, and further generates a cross-modal search result, and the specific process is as follows: (1) Obtain first modal data. The first modal data may be any one of text, voice, image, etc. (2) Based on the content information and semantic information of the first modal data, search for second modal data matching the content information and second modal data matching the semantic information from the database 101 (specifically, the second modal database), respectively. Here, the content information refers to the content contained in the first modal data itself, and the semantic information refers to the abstract meaning expressed by the first modal data. For example, the first modal data is text, the content information is the characters in the text, and the semantic information is the meaning expressed by the text. If the first modal data is an image, the content information may be the content contained in the image, for example, characters, and the semantic information may be semantic features extracted from the image. In one embodiment, based on the content information of the first modal data, the second modal data matching the content information can be directly searched for in the second modal database, but based on the semantic information of the first modal data, it is necessary to go through the second modal feature library, search for second modal features matching the semantic information of the first modal data in the second modal feature library, and further determine corresponding second modal data in the second modal database according to the second modal features, which is the second modal data matching the semantic information. (3) These second modal data are merged as a cross-modal search result matching the first modal data.

[0023] The cross-modal search device 102 may further output a cross-modal search result corresponding to the first modal data according to the input first modal data. A specific process includes the following steps: (1) displaying a session interface of a social session; (2) displaying a session record detail interface in response to a check operation on a historical session record of the social session, where the session record detail interface displays second modal data, and the second modal data belongs to the historical session record of the social session; and (3) outputting a cross-modal search result corresponding to the first modal data according to the input first modal data on the session record detail interface. Optionally, the session record detail interface may provide a search box for the session object to manually input the first modal data, or may recommend the first modal data for the session object to select, so as to quickly trigger a search function to perform a cross-modal search. In one embodiment, the cross-modal search may be performed according to a specified search rule, for example, the input text may be searched according to the description of the image, or may be searched according to the characters in the image. In this manner, the displayed cross-modal search results can be associated with a search dimension, for example, outputting second modal data that matches content information of the first modal data in the cross-modal search results, or outputting second modal data that matches semantic information of the first modal data in the cross-modal search results.

[0024] As can be seen from the above, the cross-modal search system supports two kinds of cross-modal search means, one is general-purpose cross-modal search in technical aspect, the other is cross-modal search in historical session records in product aspect, the latter outputs cross-modal search results, and the cross-modal search results are obtained by implementing the cross-modal search means in technical aspect. The cross-modal search devices supporting these two means can be the same computer device or different computer devices. If the cross-modal search devices supporting these two means are different computer devices, for example, computer device A and computer device B, computer device B receives the input first modal data and sends the first modal data to computer device A, computer device A searches the database according to the acquired first modal data, obtains the cross-modal search results, and then sends the cross-modal search results to computer device B, and computer device B outputs the cross-modal search results. When the cross-modal search device supporting these two means is the same computer device, for example computer device A, computer device A can automatically recognize the input first modal data, search for matching second modal data in a database based on the first modal data, and obtain the cross-modal search result to output on computer device A.

[0025] In this way, the cross-modal search system provided by the embodiment of the present application can support searching for second modal data matching with first modal data in the second modal database based on the content information of the first modal data and the semantic information of the first modal data, respectively, which is a cross-modal search method, and by comprehensively searching from two dimensions, namely content and semantic, respectively, the dimension covered by the search is not single, and all second modal data associated with the first modal data can be covered and searched, so that the search results can be obtained faster and more accurately. In addition, by merging the second modal data searched in two dimensions as a cross-modal search result, search results in multiple dimensions can be obtained in one search process, which significantly improves the efficiency of the cross-modal search and provides sufficiently rich and comprehensive search results. In addition, the cross-modal search system can further provide a search function based on the historical session record of the social session, which can search for second modal data in the historical session record and display all cross-modal search results therefor, or can search according to a specified dimension and display cross-modal search results of a specified dimension. With the technical support of the above-mentioned cross-modal search means, when using the search function to search for second modal data in the historical session record, the freedom and complexity of inputting first modal data can be effectively increased.

[0026] In order to better understand the cross-modal search method provided by the embodiment of the present application, the following describes the scenarios in which the cross-modal search method can be applied. Specifically, the cross-modal search method of the embodiment of the present application can be applied to, but is not limited to, the following scenarios: Scenario 1 and Scenario 2. Below, Scenario 1 and Scenario 2 are introduced respectively.

[0027] Scene 1: the first modal data is text data, the second modal data is image data, and image data and text data are searched and matched. In the historical session record of a social session, there are many types of session messages, such as photos, videos, files, links, music, etc., and searching the historical session record is a way to quickly reach the historical session messages contained in the historical session record. When searching the historical session record in the form of photos or videos, the manually entered text or the selected recommendation text can be used as the search text, and image data including photos or videos that match it can be output. In addition, when searching for photos or videos in the system album of a smart terminal, the cross-modal search method of the embodiment of the present application can also be adopted, and similarly, the text can be used as the input of a query, and the corresponding image data can be output by matching the image features in the album and the text information or associated text description information contained in the image.

[0028] Scene 2: the first modal data is audio data, the second modal data is image data, and image data and voice data are searched and matched. Take smartphones as an example, currently, most smartphones are equipped with the function of smart voice, and the terminal device can be controlled by smart voice to automatically perform corresponding operations. When faced with a large number of photos or videos in a smartphone, the problem of cross-modal search can be queried by voice, that is, by recognizing and understanding the voice, mapping the voice and the image into the same feature comparison space to match the corresponding photos, and further converting the voice into text, and comparing the text with the image category label, text description information, etc., to match the corresponding photos or videos. The cross-modal search method of the embodiment of the present application can take the voice as the query input, and automatically output the image matching the voice by matching the image content in the album of the mobile phone.

[0029] The specific implementation of the cross-modal search method proposed in the embodiments of the present application will be described in detail below in conjunction with the drawings.

[0030] 2, which is a schematic flowchart of a cross-modal search method provided in an embodiment of the present application, may be performed by a computer device (such as the cross-modal search device 102 shown in FIG. 1). The cross-modal search method includes, but is not limited to, steps S201 to S204.

[0031] S201, first modal data is acquired. Modality may refer to the source or form of information. For example, humans have hearing, vision, smell, and touch, and information media include voice, video, text, photos, etc., each of which can be considered as a modality. In the embodiments of the present application, cross-modal search mainly relates to processing on information media, and modal data may specifically be data of different forms, such as images, videos, and audio. The acquired first modal data may be modal data input by a user through a computer device, and the first modal data may be, for example, text data or image data input by an assisted manner such as a physical keyboard, a virtual keyboard, or cursor selection, or may be audio data recognized through a smart voice device, or may be selected from recommended first modal data (e.g., recommended text).

[0032] S202, searching the second modal database according to content information of the first modal data to obtain a first set.

[0033] The first set includes at least one second modal data that matches the content information of the first modal data. The content information of the first modal data is data information for describing essential content included in the first modal data. For example, the first modal data is a text, and the corresponding content information may be the text characters themselves or keywords extracted based on the text; or, for example, the first modal data is an image, and the corresponding content information may be other modal information or basic features included in the image, such as one or more of the geometric shapes, textures, colors, object category labels, and text description information included in the image. Based on the dimension of the content information of the first modal data, the second modal database may be searched for all second modal data that match the content information of the first modal data, and the matching second modal data may be added to the first set.

[0034] In one embodiment, the second modal database stores N pieces of second modal data and attribute information of each of the N pieces of second modal data, where N is a positive integer. The second modal data and the first modal data are data of two different modalities, and the second modal data may be any one of modal data such as text, image, audio, video, etc., and the second modal data stored in the second modal database is different for different business scenarios. For example, in searching the history session record of a social session, the second modal data may be an image sent or received in the session. The attribute information of the second modal data is information describing the attribute of the second modal data, and may be related information recognized from the second modal data or generated from other data, and the attribute information and the content information of the first modal data may be data of the same recording format, for example, both are text description information. The content information of the first modal data can be matched with the attribute information of the second modal data, so that the matching second modal data is searched for in the second modal database to obtain a first set.

[0035] Optionally, a specific implementation manner of step S202 includes the following steps S2021 and S2022: S2021, for each second modal data among the N second modal data, determine the matching degree between the content information of the first modal data and the attribute information of the second modal data as the matching degree corresponding to the second modal data; S2022, add the second modal data whose corresponding matching degree satisfies the matching condition to the first set.

[0036] The content information of the first modal data is matched with the attribute information of each of the N pieces of second modal data in the second modal database, and a corresponding matching degree can be obtained. The matching degree here can indicate whether the content information of the first modal data and the attribute information of the second modal data are similar or coincident. The matching degree between the content information of the first modal data and the attribute information of the second modal data may be measured by the similarity of the modal data (e.g., text similarity) or abstract semantic similarity, or other methods may be adopted, and is not limited here. By determining whether the matching degree satisfies the matching condition, the second modal database can be searched for second modal data matching the content information of the first modal data. The matching condition here may be set so that the matching degree is equal to or greater than a matching degree threshold, or may be set so that the matching degree is in the top y rank, where y is a positive integer. The specific setting contents of the matching condition are not limited.

[0037] Optionally, the attribute information includes one or two of first modal description information and category label, and the first modal description information refers to description information recorded in the first modal format, for example, the first modal data is text, in which case the first modal description information is text description information, or further, for example, the first modal data is image, in which case the first modal description information is image description information. The first modal description information can be matched to the content information of the first modal data as the attribute information of the second modal data, and when the content information of the first modal data and the attribute information of the second modal are both recorded in the same modal format, it is a matching of the same modal information, and thus, by comparing the content information of the first modal data with the first modal description information of the second modal data, the second modal data matching the content information of the first modal data can be more conveniently selected. The category label is information annotated for dividing the second modal data into categories, and may be information manually annotated on the second modal data, or may be information obtained by inputting the second modal data into a classification model and performing multi-label classification. The category label of the second modal data may be matched with content information of the first modal data to search for second modal data that satisfies a matching condition.

[0038] According to the attribute information, the detailed implementation steps of steps S2021 and S2022 are also different. Specifically, the following two embodiments can be referred to. For convenience of description, any one of N second modal data is represented as the i-th second modal data, where i is a positive integer and is less than or equal to N.

[0039] In one embodiment, the attribute information includes first modal description information, and the implementation methods corresponding to steps S2021 and S2022 may include a step of determining the semantic similarity between content information of the first modal data and the first modal description information of the i-th number of second modal data as a matching degree corresponding to the i-th number of second modal data, and a step of adding the i-th number of second modal data to the first set if the semantic similarity between the content information of the first modal data and the first modal description information of the i-th number of second modal data is greater than a first similarity threshold.

[0040] Specifically, the degree of matching between the content information of the first modal data and the attribute information of the second modal data can adopt the semantic similarity mentioned above, and the method of obtaining the semantic similarity can be a method of extracting semantic features corresponding to the content information of the first modal data and semantic features corresponding to the first modal description information of the i-th number of second modal data, and determining the similarity between the semantic features corresponding to each of the content information of the first modal data and the first modal description information of the i-th number of second modal data, and using this as the semantic similarity. Then, by determining whether the semantic similarity is greater than a first similarity threshold, it can be determined whether the i-th piece of second modal data satisfies the matching condition; if the semantic similarity is greater than the first similarity threshold, it indicates that the matching degree between the attribute information of the i-th piece of second modal data and the content information of the first modal data satisfies the matching condition; and if the attribute information of the i-th piece of second modal data indicates that it matches the content information of the first modal data, the i-th piece of second modal data can be added to the first set; otherwise, the i-th piece of second modal data is not added to the first set.

[0041] By calculating the semantic similarity between the content information of the first modal data and the first modal description information of the second modal data, it is possible to know the consistency between the content information of the first modal data and the semantics expressed by the first modal description information of the second modal data, and further to determine whether the second modal data and the first modal data match.

[0042] For example, the first modal data is text data, the second modal data is image data, the specific content of the first modal data is "blue sky and white clouds", the content information is the text content, and the first modal description information of the second modal data is text description information for the image content, the text description information is associated with the image, and may be character information contained in the image or text description information associated with the image. If the text description content associated with the image is "The sky is very beautiful today", the keyword "sky" can be the first modal description information, and the semantic similarity corresponding to each of the two texts "sky" and "blue sky and white clouds" is determined to determine whether they match, and thus determine whether the corresponding image is an image matching the text.

[0043] In another embodiment, the attribute information includes a category label, and the implementation methods corresponding to steps S2021 and S2022 may include a step of determining the similarity between content information of the first modal data and the category label of the i-th number of second modal data as a matching degree corresponding to the i-th number of second modal data, and a step of adding the i-th number of second modal data to the first set if the similarity between the content information of the first modal data and the category label of the i-th number of second modal data is greater than a second similarity threshold.

[0044] When the attribute information includes a category label, the matching degree specifically refers to the similarity between the content information of the first modal data and the category label of the i-th second modal data, and may be, for example, text similarity, where the similarity may represent the degree of match between the category label of the second modal data and the content information of the first modal data. When the content information of the first modal data is completely equivalent to the category label of the i-th second modal data, the i-th second modal data is the second modal data that satisfies the matching condition, or the content information of the first modal data is sufficiently similar to the category label, and for the same reason, the i-th second modal data may be determined as the second modal data that satisfies the matching condition. Whether the i-th piece of second modal data satisfies the matching condition can be determined, specifically, by whether the similarity between the content information of the first modal data and the category label of the i-th piece of second modal data is greater than a second similarity threshold. If the similarity is greater than the second similarity threshold, it indicates that the matching degree between the content information of the first modal data and the category label of the i-th piece of second modal data satisfies the matching condition, and further indicates that the category information of the i-th piece of second modal data matches the content information of the first modal data, and the i-th piece of second modal data is added to the first set; otherwise, the i-th piece of second modal data is not added to the first set.

[0045] For example, the first modal data is a search text, the second modal data is an image, and the i-th second modal data is a target image, which is divided into two category labels, “person” and “scenery”, by a classification model. In this case, when the input of the search text is “person” or “scenery”, the category label and the search text are completely equivalent, so the photo is matched. The similarity used here may specifically be text similarity.

[0046] Furthermore, if text-based image search is realized using only category labels, and related photos can only be matched when the search term is completely equivalent to the category label, the requirements for search terms are high, the supported search terms are limited, and the dimension is single, so it is easy to encounter a situation where no results are found after searching.By searching by integrating information from other dimensions, specifically searching from two dimensions of semantics and content, the matching conditions can be reduced at the same time, for example, the inclusion of the category label can also be considered as a match, which not only improves search efficiency but also reduces the probability of the search results being empty.

[0047] In addition, the above two embodiments can also be applied to any one of the N pieces of second modal data. In this way, after the N pieces of second modal data stored in the second modal database are all matched with the content information of the first modal data according to the above method, the finally obtained first set can be part of the cross-modal search result described below.

[0048] S203, searching in a second modal database according to the semantic information of the first modal data to obtain a second set.

[0049] The second set includes at least one second modal data that matches the semantic information of the first modal data. The semantic information of the first modal data as another information expression form may specifically refer to the meaning represented by the real-world object corresponding to the first modal data. The semantic information may be used to characterize a shallow or deep semantic understanding of the first modal data, and the semantic information may be very rich, for example, when the first modal data is text, the same semantic may have many different text expressions and is very flexible.

[0050] Searching the second modal database according to the dimension of the semantic information of the first modal data can specifically match the semantic information of the second modal data with the semantic information of the first modal data, and then searching all the second modal data that match the semantic information of the first modal data from the second modal database to obtain a second set. The semantic information can be represented by semantic features, specifically, a semantic feature vector. Based on multi-modal learning, the semantic features of the first modal data and the semantic features of the second modal data are respectively extracted, so that the semantic features of the two different modal data can be mapped to the same semantic feature space to compare the similarity, and the second modal data having similar semantics is searched for based on the similar semantic features. For a specific implementation manner of this step, the introduction of the embodiment corresponding to FIG. 3 below can be referred to, and will not be described in detail here.

[0051] Under the condition that the first modal data is text and the second modal data is an image, this step is a character-based image retrieval method based on cross-modal features, that is, by respectively extracting the text feature vector of the search term and the image feature vector of the photo, the similarity of the feature vectors of two different modalities is compared in the same semantic feature space, so that images with similar semantics can be directly searched for cross-modally through the text description, thus supporting more and more complex text descriptions, and realizing the search for target photos by inputting free and diverse text to describe images.

[0052] S204, merging the first set and the second set to obtain a cross-modal search result corresponding to the first modal data.

[0053] By searching the N second modal data stored in the second modal database according to the above steps, a first set matching the content of the first modal data and a second set matching the semantics of the first modal data can be obtained. By merging the first set and the second set, all the second modal data matching the first modal data can be obtained, including the second modal data matching the content information of the first modal data and the second modal data matching the semantic information of the first modal data, that is, the cross-modal search result corresponding to the first modal data, and the cross-modal search result obtained thereby includes search results of multiple dimensions, and is a diverse and comprehensive search result.

[0054] The cross-modal search means provided in the embodiments of the present application can search for second modal data matching the content information of the first modal data in the second modal database based on the content information of the first modal data, and can search for second modal data matching the semantic information of the first modal data in the second modal database based on the semantic information of the first modal data. This search method is not limited to a certain dimension, but is a method of comprehensively searching from multiple dimensions, so that the dimension covered by the search is not single, and search results of multiple dimensions can be obtained in one search, thus improving the efficiency of cross-modal search. By improving the quality of the search, and by merging second modal data that respectively match the two dimensions of the first modal data as the cross-modal search result, it is possible to obtain richer and more diverse cross-modal search results. Specifically, searching based on the content information of the first modal data is based on the degree of matching between the content information of the first modal data and the attribute information of the second modal data (which may be first modal description information or category labels). Since the attribute information often describes the content contained in the second modal data, the first modal data may accordingly support more diverse and complex expressions rather than being limited to fixed expressions.

[0055] 3, which is a second schematic flowchart of a cross-modal search method provided by an embodiment of the present application, which may be performed by a computer device (such as the cross-modal search device 102 shown in FIG. 1). The cross-modal search method of the present embodiment is a detailed introduction to the corresponding implementation manner of step S203 corresponding to FIG. 2, which is to search in a second modal database based on semantic information of the first modal data to obtain a second set.

[0056] The second modal database stores N pieces of second modal data. A second modal feature library is associated with the second modal data, and the second modal feature library stores semantic features of the N pieces of second modal data. A specific implementation of the step of searching the second modal database based on the semantic information of the first modal data to obtain a second set includes the following steps S301 to S304.

[0057] S301, obtaining semantic features of the first modal data. In one embodiment, the semantic features of the first modal data can be obtained by processing a cross-modal search model, specifically, the cross-modal search model includes a first modal processing network, and a specific implementation manner of this step may be a manner of performing feature extraction processing on the first modal data through the first modal processing network in the cross-modal search model to obtain the semantic features of the first modal data. The first modal processing network is a processing network for the first modal data. For example, when the first modal data is text, the first modal processing network may be a text processing network, and the text processing network may be a BERT (Bidirectional Encoder Representation from Transformers, a pre-trained linguistic feature model) model or various variant models related to BERT, or other natural language processing (NLP) models. As shown in FIG. 4a, a schematic diagram of a text encoder process is shown, where text is input, and a text encoder can output a text feature vector.

[0058] S302, based on the semantic features of the first modal data, searching for target semantic features matching the semantic features of the first modal data in a second modal feature library.

[0059] Whether the semantic features of the first modal data and the semantic features of the second modal data match can be determined by judging whether the similarity between the semantic features of the two modal data is greater than a similarity threshold. Specifically, the feature similarity between the N semantic features of the second modal data stored in the second modal feature library and the semantic features of the first modal data is calculated, and the semantic features of the second modal data whose feature similarity is greater than the similarity threshold can be determined as the semantic features of the second modal data matching the semantic features of the first modal data, i.e., the target semantic features. According to the above method, one or more target semantic features can be searched for in the second modal feature library.

[0060] For example, the first modal data is text, the second modal data is an image, a semantic feature corresponding to the first modal data is a text feature vector, and a semantic feature corresponding to the second modal data is an image feature vector. The text feature vector is used to search for a similar image feature vector from an image feature library. A specific search method may be a method in which feature similarity is calculated using the text feature vector and the image feature vector, and an image feature vector having feature similarity higher than a threshold is set as a target image feature vector matching the text feature vector.

[0061] S303, determining second modal data in a second modal database that matches the semantic information of the first modal data according to the target semantic feature.

[0062] Because the second modal feature library is associated with the second modal database, utilizing the target semantic feature found in the second modal feature library can determine second modal data from the second modal database that corresponds to the target semantic feature, which can then be the second modal data that matches the semantic information of the first modal data.

[0063] In one embodiment, the second modal feature library and the second modal database are associated by a feature index, and the implementation manner of step S303 may specifically include: (1) determining a feature index corresponding to a target semantic feature; and (2) determining corresponding second modal data of the feature index corresponding to the target semantic feature in the second modal database based on the feature index corresponding to the target semantic feature.

[0064] Each semantic feature of the second modal data in the second modal feature library is associated with a feature index, and each feature index is unique. There is also an association relationship between the feature index and the second modal data in the second modal database. In this way, the second modal data in the second modal database and the semantic features of the second modal data in the second modal feature library can be associated one by one by the feature index. Thus, based on the feature index corresponding to the searched target semantic feature, second modal data corresponding to the feature index can be selected from the second modal database, and the second modal data matching the semantic information of the first modal data can be obtained.

[0065] S304, adding second modal data matching the semantic information of the first modal data to a second set.

[0066] The second modal data determined from the second modal database and matching the semantic information of the first modal data can be added to a second set, and all the second modal data stored in the second modal database can be processed according to the above steps, and all the second modal data matching the semantic information of the first modal data can be determined and added one by one to the second set, and the final second set can be part of the cross-modal search results.

[0067] The cross-modal search method provided in the embodiment of the present application searches from the dimension of semantic information of the first modal data, extracts semantic features corresponding to each of the two modal data, i.e., the first modal data and the second modal data, and performs feature comparison processing on the semantic features of the first modal data and the semantic features of the second modal data in the same semantic space, searches for target semantic features that match the semantic features of the first modal data from the second modal feature library, and determines second modal data that matches the semantic information of the first modal data from the second modal database based on the searched target semantic features to obtain cross-modal search results. This method is essentially a search method based on cross-modal features, and the cross-modal features in the semantic aspect can search for search results that match the first modal data more quickly and accurately, and can also increase the diversity of cross-modal search results to a certain extent.

[0068] As can be seen from the above, searching for second modal data based on semantic information of first modal data needs to go through the second modal feature library, so next we will introduce in detail the method of acquiring the semantic features of the second modal data stored in the second modal feature library.

[0069] In one embodiment, the cross-modal search model includes a second modal processing network, and the semantic features of the N pieces of second modal data stored in the second modal database are obtained by performing feature extraction on each of the N pieces of second modal data through the second modal processing network in the cross-modal search model. The second modal processing network is a processing network for the second modal data, and may include various networks with different functions. For example, the second modal data is an image, and the second modal processing network may specifically be an image processing network.

[0070] Optionally, the second modal processing network includes a feature extraction network, a pooling processing network, and a feature integration network, and for convenience of description, any one of the N second modal data is denoted as the i-th second modal data, where i is a positive integer and is less than or equal to N, that is, all the N second modal data are processed according to the following steps to obtain corresponding semantic features. Based on this, the step of performing feature extraction processing on each of the N second modal data through the second modal processing network in the cross-modal retrieval model and obtaining the semantic features of the N second modal data may specifically include: extracting initial features of the i-th second modal data through the feature extraction network in the second modal processing network; performing pooling processing on the initial features through the pooling processing network in the second modal processing network to obtain pooled features of the i-th second modal data; and performing integration processing on the pooled features through the feature integration network to obtain the semantic features of the i-th second modal data.

[0071] The feature extraction network may be a deep model for image processing, for example, a normal Convolutional Neural Network (CNN) model or a Vision Transformer (VIT) model for feature extraction. The feature extraction network is a backbone network in the second modal processing network, and mainly extracts initial features of the second modal data for use in the subsequent network. The pooling processing network performs pooling processing on the initial features output from the feature extraction network, and may be, specifically, a global average pooling (GAP) processing. In this case, the pooling processing network may be called a global average pooling layer. Global average pooling can not only reduce the amount of parameters and prevent overfitting, but also integrate global spatial information to make the features of the second modal data more robust. After that, a feature integration network is called to perform integration processing on the pooling features output from the pooling processing network, and the i-th semantic features of the second modal data can be obtained. The feature integration network may specifically be a feature fully connected layer, and since the fully connected layer requires the input object to be one-dimensional, the pooling features need to be flattened into one-dimensional features before being input to the feature integration network for processing, and then the one-dimensional features are further processed by the feature integration network to obtain the semantic features of the second modal data.

[0072] The above method is based on the principle of processing any one of the N pieces of second modal data using a cross-modal retrieval model, that is, for each piece of second modal data among the N pieces of second modal data, the same processing steps can be adopted to extract semantic features of the second modal data, which can then be stored in a second modal feature library.

[0073] In one possible embodiment, the second modal processing network may further include a classification network, and the classification network may perform classification prediction processing based on the pooling features to obtain category labels of the i-th second modal data, and add the category labels of the i-th second modal data to the second modal database. The classification network may be a classification fully connected layer, and similar to the feature fully connected layer, the pooling features processed by the classification fully connected layer are also one-dimensional features after flattening, and the output of the classification fully connected layer is passed through an activation function (e.g., Sigmoid function) to obtain the scores of the i-th second modal data belonging to each category, thereby obtaining the corresponding category labels. All of the category labels of the N second modal data in the second modal database can be obtained by adopting the above classification network to perform multi-classification processing, and all of the category labels of the second modal data can be added to the second modal database, so that when processing the first modal data, it is convenient to search for second modal data matching the first modal data according to the similarity between the category labels of each second modal data and the content information of the first modal data.

[0074] Based on the above description of the second modal processing network, the specific structure of the second modal processing network can be known. If the second modal data is an image, the second modal processing network is specifically an image encoder, including a feature extraction network, a pooling network, a feature integration network and a classification network, specifically a backbone network, a global average pooling layer, a feature fully connected layer and a classification fully connected layer, respectively, the second modal database is specifically an image library, and the second modal feature library is specifically an image feature vector search set. The process flow of processing the second modal data of the second modal processing network is combined with the structure of the image encoder in the cross-modal search model as shown in Figure 4b to provide an exemplary description as follows:

[0075] Specifically, the image encoder in the cross-modal retrieval model as shown in FIG. 4b includes a backbone network Backbone, a global average pooling layer, a classification fully connected layer, and a feature fully connected layer. If the image XI sent by the session object in the session is XI, the image XI is input to the image encoder, and the image encoder outputs a multi-label classification result C I = {c1, c2, …, c n} and the image feature vector f I (or called image feature vector), the specific processing steps are as follows: First, the image is used as the input of the image encoder, and the image feature map (i.e., initial features) is obtained through the backbone network (e.g., CNN or VIT) of the image encoder. The image feature map is then flattened into a one-dimensional vector through a global average pooling process, and the flattened one-dimensional vector is then input into the classification fully connected layer (Cls FC) to output a one-dimensional vector of length C, and the score of each category is obtained through a Sigmoid function to obtain the corresponding category label C. I = {c1, c2, …, c n} is obtained. At the same time, the one-dimensional vector is further input to the feature fully connected layer (Feature FC) to output a vector of length d (assuming it is 512), which is then subjected to L2 normalization to obtain the image feature vector f I Let the image feature vector f I is stored in the image feature vector search set (corresponding to the second modal feature library), while the category labels of the multi-label classification corresponding to the image can be stored in the second modal database, and finally, the image feature vector f I According to the,index of the corresponding image feature vector is newly increased and the image,feature vector search set G I , which conveniently assists in quickly searching for the target image from the image library.

[0076] By combining the first modal processing network in the cross-modal retrieval model shown in FIG. 4a above and the second modal processing network in the cross-modal retrieval model shown in FIG. 4b above, when using the cross-modal retrieval model for retrieval, the specific processing process for the first modal data is as follows. For convenience of description, it is taken as an example that the first modal data is text, and the first modal processing network is a text encoder correspondingly. The output text feature vector and image feature vector are feature vectors that are mapped to the same semantic feature space and have the same dimensions. The processing for the text specifically includes the following steps: first, search in the image library from the content information of the text, specifically, query the image library for images that contain a label that completely matches the text query, and recall the matched images as an image set A. At the same time, the text is input to the text encoder, and the text encoder outputs a vector of length d, and the text feature vector f is obtained through L2 normalization. T Then, we obtain the text feature vector f T Using the image feature vector search set G I (The image feature vectors contained in it are obtained by processing the images with an image encoder as shown in Figure 4b.) Search for similar image feature vectors from the image set B and recall the corresponding image set B. The specific search method is to search for the text feature vector f T , image feature vector f in the search set I We use the feature similarity, i.e., S=f T f I Then, images with feature similarity S higher than a threshold θ are searched for as image set B, and finally image set A and image set B are merged to obtain cross-modal search results.

[0077] In one embodiment, combining the above description of the structure and function of the cross-modal retrieval model, the cross-modal retrieval model includes a first modal processing network and a second modal processing network, and the specific training process may be as follows: 1) obtaining a cross-modal training dataset, the cross-modal training dataset including a plurality of sets of cross-modal sample data, each set of cross-modal sample data including a second modal sample data, a first modal sample data, and a matching result between the second modal sample data and the first modal sample data; 2) performing a feature extraction process on the first modal sample data of the cross-modal sample data through a first modal processing network in the cross-modal search model to obtain semantic features of the first modal sample data; and performing a feature extraction process on the second modal sample data of the cross-modal sample data through a second modal processing network in the cross-modal search model to obtain semantic features of the second modal sample data; 3) iteratively training the cross-modal search model according to the cross-modal contrast loss between the semantic features of the first modal sample data and the semantic features of the second modal sample data, to obtain a trained cross-modal search model.

[0078] In the training data preparation stage, a cross-modal training dataset can be obtained from business data generated by corresponding scenes, and the cross-modal training dataset is a collection of two different modal sample data, and for training the cross-modal retrieval model, each set of cross-modal sample data can be input as a unit into the cross-modal retrieval model for processing. For example, the first modal sample data and the second modal sample data are text and image, respectively, and each set of cross-modal sample data can be an image-text pair, that is, an image and a text description corresponding to the image can constitute an image-text pair, and a large number of image-text pairs can constitute the cross-modal training dataset.

[0079] The training process of the cross-modal retrieval model is specifically, a hybrid training is performed on a first modal processing network and a second modal processing network. K sets of cross-modal sample data can be input simultaneously, and the first modal sample data of the i-th set of cross-modal sample data is processed through the first modal processing network to obtain the semantic features of the first modal sample data, and the second modal sample data of the i-th set of cross-modal sample data is processed through the second modal processing network to obtain the semantic features of the second modal sample data, and a cross-modal contrast loss is calculated according to the semantic features of the two different modal sample data, and the cross-modal retrieval model is iteratively trained based on the cross-modal contrast loss, and the model parameters are continuously updated until convergence, so that a trained model can be obtained.

[0080] When the second modal processing network includes a classification processing network, the cross-modal training dataset may further include a category label corresponding to the second modal sample data. Specifically, the training process may further include: performing classification prediction processing on the second modal sample data of the cross-modal sample data through the second modal processing network in the cross-modal retrieval model to obtain category prediction information of the second modal sample data; determining a classification loss of the second modal sample data according to the category prediction information and the category label; and iteratively training the cross-modal retrieval model according to the classification loss and the cross-modal contrast loss to obtain a trained cross-modal retrieval model. The category prediction information may include a predicted probability that the second modal sample data belongs to each category, and for classification loss, a cross-entropy loss may be used, and then the classification loss and the cross-modal contrast loss may be combined to obtain a total loss. For example, a weighted sum is performed on the classification loss and the cross-modal contrast loss to obtain a total loss, and an optimizer (e.g., a Stochastic Gradient Descent (SGD) optimizer) is used to update the model parameters of the cross-modal search model, and the above training process is continuously repeated until the model parameters converge to obtain a trained cross-modal search model. In this way, the cross-modal search model is applied to the semantic feature extraction process of the first modal data and the second modal data, and can not only detect the matching degree between the first modal data and the second modal data based on the cross-modal features, but also has a multi-label classification function, and generates category labels for the second modal data and stores them in the second modal database.

[0081] In order to better understand the principle of the training stage, the training process of the cross-modal retrieval model is described below by taking the first modal processing network as a text encoder and the second modal processing network as an image encoder as an example. Referring to FIG. 5, there is shown a schematic diagram of the training of the cross-modal retrieval model provided by the embodiment of the present application. If the cross-modal training data set includes K sets of image-text pairs (or referred to as image-text pairs), during training, the K sets of image-text pairs are input simultaneously, and the image feature vectors are generated by the image encoder and the text encoder, respectively.

number

number

number

[0082] In addition,

number

number

number

number

[0083] Image classification loss L cls can be calculated using Cross Entropy Loss (CEL), and L cls and L infoNCE The total loss is calculated by combining the two and updating the model parameters with the SGD optimizer until convergence is achieved.

[0084] Based on the introduction of the above embodiment, the algorithm flow adopted by the cross-modal search means is described by taking the first modal data as text and the second modal data as an example of an image, and specifically, the algorithm flow chart shown in FIG. 6 can be referred to. When an image is input to the image encoder for multi-label classification, a category label can be obtained, and if the search text (referred to as query) is completely equivalent to the category label, related images can be matched. In addition, the image encoder can further output an image feature vector, and add the image feature vector and the newly increased index to the image feature vector search set. The search text query can be input to the text encoder, output a text feature vector, and then search for a similar image feature vector from the image feature vector search set based on the text feature vector, and recall the corresponding image set based on the similar image feature vector. Such a means for searching based on cross-modal features can support more diverse and complex text descriptions because it can match based on the features of different modal data without relying on a fixed class label system of a classification model, and can increase the freedom of input of search terms, and can find target photos faster, more accurately, and more comprehensively.

[0085] 7, which is a third schematic flowchart of a cross-modal search method provided by an embodiment of the present application, may be performed by a computer device (for example, the cross-modal search device 102 shown in FIG. 1, the cross-modal search device 102 may specifically be a terminal). The cross-modal search method includes, but is not limited to, steps S701 to S703.

[0086] S701, displaying a session interface of a social session. The social session here may be a person-to-person session or a group session. In the session interface of the social session, the session object can send and receive session messages, such as images, text, voice, etc. If the session message received in the session interface includes second modal data, the second modal processing network in the cross-modal search model can be invoked to process the second modal data, output category labels of multi-label classification and semantic features of the second modal data, and further store the category labels in a second modal database and store the semantic features of the second modal data (e.g., image feature vectors) in a second modal feature library.

[0087] The session interface of the social session can provide a function of checking historical session records. Specifically, a session interface can be entered into a session details interface, which includes a check entry of historical session records, and the session object can initiate a check operation through the check entry to check and search specific historical session records. Specifically, see steps S702 to S703 below.

[0088] S702, in response to checking against a historical session record of the social session, displaying a session record details interface. The session record details interface includes second modal data in the historical session record of the social session. The historical session record of the social session may include different modal data, such as images, videos, text, audio, etc., and the session object can select and check the different modal data, where the check on the historical session record is mainly the check on the second modal data, so that the second modal data generated in the historical session record is displayed in the session record details interface.

[0089] If the number of second modal data is small, the second modal data can be fully displayed in the session record details interface, and if the number of second modal data is large, only a part of the second modal data is displayed in the current session record details interface. For example, the second modal data is an image, and the session record details interface is specifically a chat photo wall, and 12 images can be displayed in the same size, and if there are 10 images in the history session record, they can be fully displayed in the session record details interface, and if there are more than 12 images, at most 12 images are displayed. If other images are to be checked, for example, a sliding down operation needs to be performed to display them, and then the session record details interface supports searching the second modal data with the first modal data, and outputting the second modal data that matches the first modal data, i.e., the cross-modal search result.

[0090] Referring to Fig. 8a, an operation diagram of checking historical session records provided by an embodiment of the present application is shown. A session interface 810 (1) in Fig. 8a provides an entry for searching historical session records, i.e., "search chat contents", and when the entry is triggered, a user can enter a historical session record search interface 811 as shown in Fig. 8a (2), in which the user can select a corresponding search type and display all historical session records of the corresponding search type. For example, when selecting photos and videos, the session record details interface 812 displays a chat photo wall, and the chat photo wall is all photos and videos displayed according to date, specifically as shown in Fig. 8a (3), and the session record details interface 812 provides a search box 8120 to facilitate the user to search for photos or videos.

[0091] In one embodiment, the second modal data in the historical session record of the social session is stored in a second modal database, and attribute information of the second modal data is stored in the second modal database. By storing the second modal data in the historical session record separately in the second modal database, when a search for the second modal data is started, the second modal data can be directly searched in the second modal database without searching in the global historical session record, which is advantageous to improve the search efficiency of the second modal data; at the same time, attribute information of the second modal data is stored in the second modal database, and the search dimension can be further expanded according to the difference of the attribute information.

[0092] The attribute information includes at least one of a category label, first modal description information associated with the second modal data, and first modal description information recognized from the second modal data. The category label may be annotation information generated by classifying the second modal data manually or by a device (e.g., a classification model), and the first modal description information is description information related to the second modal data, specifically, may be recognized from the second modal data, or may be generated from and associated with the historical session record. Exemplarily, when the second modal data is an image and the image in the historical session record is an image including text, the text in the image can be obtained by recognizing the image and can be used as the first modal description information. If the session object of the social session transmits an image and then transmits text description information for the image, for example, information such as "Look, A Park has changed a lot," then a description related to the image can be generated according to the text description information, and for example, a keyword "A Park" is extracted as the first modal description information of the image.

[0093] S703, in response to the first modal data input into the session record detail interface, outputting cross-modal search results corresponding to the first modal data.

[0094] The cross-modal search results are obtained by employing the cross-modal search method introduced in the above embodiment, and the output cross-modal search results include all second modal data that match the first modal data entered in the session record details interface.

[0095] In one embodiment, the first modal data is text and the second modal data is an image, the session record details interface includes a search box, and the first modal data is obtained by inputting into the search box, or the session record details interface further includes at least one recommendation text, and the first modal data is obtained by selecting from the at least one recommendation text. That is, the first modal data input into the session record details interface may be manually input into the search box via an input device (e.g., a physical / virtual keyboard, a smart voice device), etc., or may be selected from the recommendation text provided by the session record details interface. Optionally, the selected recommendation text may be automatically filled into the search box and automatically launch a search function. The recommendation text in the session record details interface may be randomly generated or may be generated according to attribute information of the second modal data or semantic features of the second modal data. With the technical support of the above cross-modal search method, the text input into the search box may be an image description that meets an intuitive expression. In simple terms, taking character-based image search as an example, when a session object searches in a search box, images with category labels matching the search text can be queried and recalled in the second modal database, and at the same time, the search text can be processed through a text encoder in the cross-modal search model to output corresponding text feature vectors, search for similar image feature vectors from the image feature vector search set, recall the corresponding image set, and finally merge all the recalled images to be exhibited in the session object.

[0096] For example, referring to FIG. 8b, an operation schematic diagram of a cross-modal search provided by an embodiment of the present application is shown. As shown in FIG. 8b (1), the session record details interface provides a search box 8220, which supports input of image description or in-picture text for search, where the image description is a semantic interpretation of the content contained in the image, and the in-picture text belongs to the content information of the image. In addition, automatically generated recommendation texts such as "tickets", "screenshots", etc. in FIG. 8b are also displayed in the session record details interface, and the recommendation texts can provide more reference and convenient operation. When a search function is triggered by inputting a query text in the search box 8220, a search result interface is output, and photos matching the query text can be displayed in the search result interface. As shown in FIG. 8b (2), the search result interface 823 displays three photos matching the input query text "food", which belong to the cross-modal search result.

[0097] In one embodiment, in response to a selection of the first search rule, second modal data from the cross-modal search results that matches content information of the first modal data may be output, or in response to a selection of the second search rule, second modal data from the cross-modal search results that matches semantic information of the first modal data may be output.

[0098] The first search rule and the second search rule are rules for searching from different dimensions, and can search according to different dimensions, and can display all cross-modal search results separately according to different search dimensions. Searching according to the first search rule can obtain and output second modal data that matches the content information of the first modal data, and searching according to the second search rule can obtain and output second modal data that matches the semantic information of the first modal data. That is, a single search dimension can be specified. For example, when the first modal data is text and the second modal data is an image, search by image and search by character can be performed. Search by image specifically refers to searching according to the description of the image, that is, searching by matching the dimension of the semantic information of the image, and search by character specifically refers to searching according to the characters in the image, that is, searching by matching the dimension of the content information of the image.

[0099] For example, referring to FIG. 8c, it is a schematic diagram of the effect of outputting a cross-modal search result provided by an embodiment of the present application. As shown in FIG. 8c, all photos matching the query text are displayed based on the cross-modal search result provided by (2) in FIG. 8b, and are obtained by searching according to different search dimensions. When searching by image or by text is selected, different cross-modal search results are presented, as shown in (1) in FIG. 8c and (2) in FIG. 8c, respectively, and in the search result interface, photos whose semantic information of the text matches the semantic information of the image, or photos whose content information of the text matches the attribute information of the image (e.g., the category label of the image) can be displayed. This means can be applied to various scenes, and in addition to the cross-modal search based on the history session record of the social session introduced in this embodiment, it can also be applied to other scenes supporting multimedia data search, such as the scene of short video search, and is not limited thereto.

[0100] The cross-modal search means provided in the embodiments of the present application can support cross-modal search scenes in historical session records of social sessions, and can be specifically applied to the scene of graphic and text cross-modal search, that is, a target photo is searched for by inputting a search term in a search box, and since cross-modal search searches from multiple dimensions of the search term, the corresponding photo can be found even if the search term does not completely match the category label of the photo. Therefore, searching for target photos by inputting more diverse and complex image descriptions that meet more intuitive expressions can not only increase the freedom of input, but also greatly increase the probability of searching for target photos and improve the diversity of cross-modal search results. In addition, by providing recommended text (e.g., recommended search terms), the search efficiency can also be improved to a certain extent.

[0101] Referring to FIG. 9, FIG. 9 is a structural schematic diagram of a cross-modal search device provided by an embodiment of the present application. The cross-modal search device may be a computer program (including program code) executed on a computer device, for example, the cross-modal search device is an application software, and the cross-modal search device may be used to perform corresponding steps of the method provided by an embodiment of the present application. As shown in FIG. 9, the cross-modal search device 900 may include an acquisition module 901, a search module 902, and a merge module 903. The acquisition module 901 is used for acquiring first modal data; The search module 902 is used to search in the second modal database according to the content information of the first modal data to obtain a first set, the first set including at least one second modal data matching the content information of the first modal data; The search module 902 is further used to search in a second modal database based on the semantic information of the first modal data to obtain a second set, the second set including at least one second modal data matching the semantic information of the first modal data; The merge module 903 is used to merge the first and second sets to obtain cross-modal search results corresponding to the first modal data.

[0102] In one embodiment, the second modal database stores N pieces of second modal data and attribute information of each of the N pieces of second modal data, where N is a positive integer. The search module 902 is specifically used for determining, for each piece of second modal data among the N pieces of second modal data, the matching degree between the content information of the first modal data and the attribute information of the second modal data as a matching degree corresponding to the second modal data, and adding the second modal data whose corresponding matching degree meets the matching condition to the first set.

[0103] In one embodiment, the attribute information includes first modal description information, and any one of the N second modal data is indicated as the i-th second modal data, where i is a positive integer and is less than or equal to N, and the search module 902 specifically determines the semantic similarity between the content information of the first modal data and the first modal description information of the i-th second modal data as a matching degree corresponding to the i-th second modal data, and is used to add the i-th second modal data to the first set when the semantic similarity between the content information of the first modal data and the first modal description information of the i-th second modal data is greater than a first similarity threshold.

[0104] In one embodiment, the attribute information includes a category label, and any one of the N second modal data is indicated as the i-th second modal data, where i is a positive integer and is less than or equal to N. The search module 902 specifically determines the similarity between the content information of the first modal data and the category label of the i-th second modal data as a matching degree corresponding to the i-th second modal data, and is used to add the i-th second modal data to the first set when the similarity between the content information of the first modal data and the category label of the i-th second modal data is greater than a second similarity threshold.

[0105] In one embodiment, the second modal database stores N pieces of second modal data, and the second modal database is associated with a second modal feature library in which semantic features of each of the N pieces of second modal data are stored, and the search module 902 is specifically further used for obtaining semantic features of the first modal data, searching for target semantic features matching the semantic features of the first modal data in the second modal feature library based on the semantic features of the first modal data, determining second modal data matching the semantic information of the first modal data in the second modal database according to the target semantic features, and adding the second modal data matching the semantic information of the first modal data to the second set.

[0106] In one embodiment, the second modal feature library and the second modal database are associated by a feature index, and the search module 902 is specifically used to determine a feature index of the target semantic feature, and determine second modal data corresponding to the feature index of the target semantic feature in the second modal database based on the feature index of the target semantic feature.

[0107] In one embodiment, the semantic features of each of the N pieces of second modal data stored in the second modal feature library are obtained by performing feature extraction processing on each of the N pieces of second modal data through a second modal processing network in the cross-modal search model, the cross-modal search model further includes a first modal processing network, and the search module 902 is specifically used to perform feature extraction processing on the first modal data through the first modal processing network in the cross-modal search model to obtain the semantic features of the first modal data.

[0108] In one embodiment, the second modal processing network includes a feature extraction network, a pooling processing network, and a feature integration network, and any one of the N second modal data is denoted as the i-th second modal data, where i is a positive integer and is less than or equal to N, and the search module 902 is specifically used for extracting initial features of the i-th second modal data through the feature extraction network in the second modal processing network, performing a pooling process on the initial features through the pooling processing network in the second modal processing network to obtain pooled features of the i-th second modal data, and performing an integration process on the pooled features through the feature integration network to obtain semantic features of the i-th second modal data.

[0109] In one embodiment, the second modal processing network further includes a classification network, and the search module 902 is specifically further used for performing classification prediction processing based on the pooling features through the classification network, obtaining category labels of the i-th second modal data, and adding the category labels of the i-th second modal data to the second modal database.

[0110] In one embodiment, the cross-modal search apparatus further includes a training module 904, which is used to acquire a cross-modal training dataset, the cross-modal training dataset including a plurality of sets of cross-modal sample data, each set of cross-modal sample data including a second modal sample data, a first modal sample data, and a matching result between the second modal sample data and the first modal sample data; performing a feature extraction process on the first modal sample data of the cross-modal sample data through a first modal processing network in the cross-modal search model to obtain semantic features of the first modal sample data; performing a feature extraction process on the second modal sample data of the cross-modal sample data through a second modal processing network in the cross-modal search model to obtain semantic features of the second modal sample data; and iteratively training the cross-modal search model according to a cross-modal contrast loss between the semantic features of the first modal sample data and the semantic features of the second modal sample data to obtain a trained cross-modal search model.

[0111] The functions of each functional module of the cross-modal search device described in the embodiments of the present application can be specifically realized according to the method in the above method embodiments, and the specific realization process can be referred to the relevant description of the above method embodiments, and will not be described in detail here, and the description of the beneficial effects of the same method will not be described in detail here either.

[0112] Referring to Fig. 10, Fig. 10 is a structural schematic diagram of another cross-modal search device provided by an embodiment of the present application. The cross-modal search device may be a computer program (including program code) executed on a computer device, for example, the cross-modal search device is an application software, and the cross-modal search device may be used to perform corresponding steps of the method provided by the embodiment of the present application. As shown in Fig. 10, the cross-modal search device 1000 may include a display module 1001 and an output module 1002. The display module 1001 is used to display a session interface of a social session; The display module 1001 is further adapted to display a session record detail interface in response to checking against the historical session record of the social session, the session record detail interface including second modal data in the historical session record of the social session; The output module 1002 is used for outputting cross-modal search results corresponding to the first modal data input into the session record details interface in response to the first modal data, the cross-modal search results being obtained by employing the cross-modal search method described in the embodiments of the present application.

[0113] In one embodiment, the second modal data in the historical session record of the social session is stored in a second modal database, and attribute information of the second modal data is stored in the second modal database, the attribute information including at least one of a category label, first modal descriptive information associated with the second modal data, and first modal descriptive information recognized from the second modal data.

[0114] In one embodiment, the first modal data is text and the second modal data is an image, and the session record details interface includes a search box and the first modal data is obtained by entering data into the search box; or the session record details interface further includes at least one recommendation text and the first modal data is obtained by selecting from the at least one recommendation text.

[0115] In one embodiment, the output module 1002 is specifically used to output second modal data from the cross-modal search results that matches content information of the first modal data in response to a selection for a first search rule, or to output second modal data from the cross-modal search results that matches semantic information of the first modal data in response to a selection for a second search rule.

[0116] The functions of each functional module of the cross-modal search device described in the embodiments of the present application can be specifically realized according to the method in the above method embodiments, and the specific realization process can be referred to the relevant description of the above method embodiments, and will not be described in detail here, and the description of the beneficial effects of the same method will not be described in detail here either.

[0117] The cross-modal search device of Fig. 9 and the cross-modal search device of Fig. 10 may be arranged in the same computer device or in different computer devices. If arranged in the same computer device, the computer device can automatically search for second modal data matching the first modal data from a database according to the input first modal data, obtain the cross-modal search result, and output the cross-modal search result in the computer device. If arranged in different computer devices, if the cross-modal search device of Fig. 9 is arranged in computer device A and the cross-modal search device of Fig. 10 is arranged in computer device B, the computer device B plays a role of receiving the input first modal data and transmitting the first modal data to computer device A, and the computer device A searches for second modal data matching the first modal data in a second modal database according to the first modal data, obtains the cross-modal search result, and transmits the cross-modal search result to computer device B, and the computer device B displays the cross-modal search result.

[0118] Referring to Fig. 11, Fig. 11 is a structural schematic diagram of a computer device provided by an embodiment of the present application. The computer device 1100 may include an independent device (e.g., one or more of a server, a node, a terminal, etc.), or may include a component inside an independent device (e.g., a chip, a software module, or a hardware module, etc.). The computer device 1100 may include at least one processor 1101 and a communication interface 1102, and optionally, the computer device 1100 may further include at least one memory 1103 and a bus 1104. The processor 1101, the communication interface 1102, and the memory 1103 are connected via the bus 1104.

[0119] The processor 1101 is a module that performs arithmetic and / or logical operations, and may be one or more of a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (which helps the central processing unit complete corresponding processes and applications), a microcontroller unit (MCU), etc.

[0120] The communication interface 1102 may be used to provide input or output of information for at least one processor and / or may be used to receive data transmitted from the outside and / or transmit data to the outside, and may be a wired link interface including an Ethernet cable or the like, or a wireless link (such as Wi-Fi, Bluetooth, general-purpose wireless transmission, in-vehicle short-range communication technology, and other short-range wireless communication technology) interface.

[0121] The memory 1103 is used to provide storage space, in which data such as an operating system and computer programs may be stored, and may be one or more combinations of random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable compact disc read-only memory (CD-ROM), etc.

[0122] To execute the above cross-modal search methods, for example the cross-modal search methods described in the embodiments shown in Figures 2, 3 and 7 above, at least one processor 1101 of the computing device 1100 is used to call a computer program stored in at least one memory 1103.

[0123] The computer device 1100 described in the embodiment of the present application may execute the description of the cross-modal search method in the embodiment corresponding to the above, or the description of the cross-modal search device 900 in the embodiment corresponding to Fig. 9 in the above, or the description of the cross-modal search device 1000 in the embodiment corresponding to Fig. 10, which will not be described here in detail, and the description of the beneficial effects of the same method will not be described in detail here either.

[0124] In addition, an exemplary embodiment of the present application further provides a storage medium, in which a computer program of the above-mentioned cross-modal search method is stored, the computer program including program instructions, when one or more processors load and execute the program instructions, the description of the cross-modal search method of the embodiment can be realized, and no further description will be given here, and no further description will be given here of the beneficial effects of the same method, and the program instructions may be located and executed in one or more computing devices that can communicate with each other.

[0125] The computer-readable storage medium may be the cross-modal search device provided by any of the above embodiments or an internal storage unit of the computer device, such as a hard disk or internal memory of the computer device. The computer-readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk mounted on the computer device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Furthermore, the computer-readable storage medium may further include an internal storage unit of the computer device as well as an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has already been output or is scheduled to be output.

[0126] One aspect of the present application provides a computer program product or computer program, the computer program product or computer program including computer instructions stored in a computer-readable storage medium, a processor of a computing device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computing device to perform a method provided by one aspect of an embodiment of the present application.

[0127] One aspect of the present application provides another computer program product, the computer program product including a computer program or computer instructions that, when executed by a processor, implements steps of a cross-modal search method provided by an embodiment of the present application.

[0128] The above disclosure is merely a preferred embodiment of the present application, and naturally does not limit the scope of the present application. Therefore, any equivalent changes made according to the claims of the present application are within the scope of the present application. [Explanation of symbols]

[0129] 101 Database 102 Cross-modal search device 810 Session Interface 811 Historical Session Record Search Interface 812 Session Recording Details Interface 823 Search Results Interface 900 Cross-modal search device 901 Acquisition Module 902 Search Module 903 Merge Module 904 Training Module 1000 Cross-modal search device 1001 Display module 1002 Output Module 1100 Computer Devices 1101 Processor 1102 Communication Interface 1103 Memory 1104 Bus 8120 Search Box 8220 Search box

Claims

1. 1. A method of cross-modal search performed by a computing device, comprising: acquiring first modal data; searching a second modal database based on content information of the first modal data to obtain a first set, the first set including at least one second modal data matching the content information of the first modal data; searching the second modal database based on semantic information of the first modal data to obtain a second set, the second set including at least one second modal data matching the semantic information of the first modal data; and merging the first set and the second set to obtain a cross-modal search result corresponding to the first modal data. A method comprising:

2. The second modal database stores N pieces of second modal data and attribute information of each of the N pieces of second modal data, where N is a positive integer, and the step of searching the second modal database based on content information of the first modal data to obtain a first set includes: determining, for each of the N pieces of second modal data, a matching degree between content information of the first modal data and attribute information of the second modal data as a matching degree corresponding to the second modal data; adding second modal data whose corresponding matching degree satisfies a matching condition to the first set; 2. The method of claim 1 .

3. The attribute information includes first modal description information, and any one of the N pieces of second modal data is indicated as an i-th piece of second modal data, where i is a positive integer and is equal to or smaller than N; The step of determining, for each of the N pieces of second modal data, a matching degree between content information of the first modal data and attribute information of the second modal data as a matching degree corresponding to the second modal data, includes: determining a semantic similarity between content information of the first modal data and first modal description information of the i number of second modal data as a matching degree corresponding to the i number of second modal data; The step of adding second modal data whose corresponding matching degree satisfies a matching condition to the first set includes: adding the i-th second modal data to the first set when a semantic similarity between content information of the first modal data and first modal description information of the i-th second modal data is greater than a first similarity threshold; 3. The method of claim 2 .

4. The attribute information includes a category label, and any one of the N pieces of second modal data is indicated as an i-th piece of second modal data, where i is a positive integer and is equal to or smaller than N; The step of determining, for each of the N pieces of second modal data, a matching degree between content information of the first modal data and attribute information of the second modal data as a matching degree corresponding to the second modal data, includes: determining a similarity between content information of the first modal data and category labels of the i number of second modal data as a matching degree corresponding to the i number of second modal data; The step of adding second modal data whose corresponding matching degree satisfies a matching condition to the first set includes: adding the i-th second modal data to the first set when a similarity between content information of the first modal data and a category label of the i-th second modal data is greater than a second similarity threshold; 3. The method of claim 2 .

5. The second modal database stores N pieces of second modal data, and the second modal database is associated with a second modal feature library in which semantic features of each of the N pieces of second modal data are stored; The step of searching in the second modal database based on semantic information of the first modal data to obtain a second set includes: obtaining semantic features of the first modal data; searching, based on the semantic features of the first modal data, for target semantic features in the second modal feature library that match the semantic features of the first modal data; determining second modal data in the second modal database that matches semantic information of the first modal data according to the target semantic features; and adding second modal data matching semantic information of the first modal data to the second set.

2. The method of claim 1 .

6. the second modal feature library and the second modal database are associated by a feature index; The step of determining second modal data in the second modal database that matches semantic information of the first modal data according to the target semantic features includes: determining a feature index for the target semantic feature; determining second modal data corresponding to the feature index of the target semantic feature in the second modal database based on the feature index of the target semantic feature; 6. The method of claim 5 .

7. The semantic features of each of the N pieces of second modal data stored in the second modal feature library are obtained by performing feature extraction processing on each of the N pieces of second modal data through a second modal processing network in a cross-modal search model, and the cross-modal search model further includes a first modal processing network; The step of obtaining semantic features of the first modal data further comprises: performing a feature extraction process on the first modal data through the first modal processing network in the cross-modal retrieval model to obtain semantic features of the first modal data; 6. The method of claim 5 .

8. The second modal processing network includes a feature extraction network, a pooling processing network, and a feature integration network, and any one of the N second modal data is denoted as an i-th second modal data, where i is a positive integer and is less than or equal to N; The step of performing feature extraction processing on each of the N second modal data through a second modal processing network in the cross-modal retrieval model to obtain semantic features of each of the N second modal data includes: extracting initial features of the i second modal data through the feature extraction network in the second modal processing network; performing a pooling process on the initial features through the pooling process network in the second modal processing network to obtain pooling features of the i second modal data; and performing an integration process on the pooled features through the feature integration network to obtain semantic features of the i second modal data.

8. The method of claim 7.

9. The second modal processing network further comprises a classification network, and the method further comprises: performing a classification prediction process based on the pooling features through the classification network to obtain category labels of the i second modal data; adding category labels of the i pieces of second modal data to the second modal database; 9. The method of claim 8.

10. The method further comprises: acquiring a cross-modal training data set, the cross-modal training data set including a plurality of sets of cross-modal sample data, each set of the cross-modal sample data including second modal sample data, first modal sample data, and matching results between the second modal sample data and the first modal sample data; performing a feature extraction process on a first modal sample data of the cross-modal sample data through a first modal processing network in the cross-modal retrieval model to obtain semantic features of the first modal sample data; and performing a feature extraction process on a second modal sample data of the cross-modal sample data through a second modal processing network in the cross-modal retrieval model to obtain semantic features of the second modal sample data. and iteratively training the cross-modal retrieval model according to a cross-modal contrast loss between the semantic features of the first modal sample data and the semantic features of the second modal sample data to obtain a trained cross-modal retrieval model.

8. The method of claim 7.

11. 1. A method of cross-modal search performed by a computing device, comprising: displaying a session interface for the social session; displaying a session record details interface in response to checking against a historical session record for the social session, the session record details interface including second modal data in the historical session record for the social session; in response to first modal data entered into the session record details interface, outputting a cross-modal search result corresponding to the first modal data, the cross-modal search result being obtained by employing a cross-modal search method according to any one of claims 1 to 10. A method comprising:

12. The second modal data in the historical session record of the social session is stored in a second modal database, and attribute information of the second modal data is stored in the second modal database, the attribute information including at least one of a category label, first modal descriptive information associated with the second modal data, and first modal descriptive information recognized from the second modal data.

12. The method of claim 11 .

13. the first modal data is text and the second modal data is an image, the session record details interface includes a search box, and the first modal data is obtained by entering data into the search box; or the session record detail interface further includes at least one recommendation text, and the first modal data is obtained by selecting from the at least one recommendation text.

12. The method of claim 11 .

14. The method further comprises: outputting second modality data from the cross-modal search results that matches content information of the first modality data in response to a selection of a first search rule; or outputting second modal data from the cross-modal search results that matches semantic information of the first modal data in response to a selection of a second search rule; 12. The method of claim 11 .

15. A cross-modal search device, comprising: an acquisition module for acquiring first modal data; a search module for searching a second modal database based on content information of the first modal data to obtain a first set, the first set including at least one second modal data matching the content information of the first modal data; The search module is further used to search the second modal database based on the semantic information of the first modal data to obtain a second set, the second set including at least one second modal data matching the semantic information of the first modal data; a merging module for merging the first set and the second set to obtain cross-modal search results corresponding to the first modal data. An apparatus comprising:

16. A cross-modal search device, comprising: a display module for displaying a session interface for the social session; the display module is further adapted to display a session record detail interface in response to checking against a historical session record of the social session, the session record detail interface including second modal data in the historical session record of the social session; an output module for outputting a cross-modal search result corresponding to the first modal data inputted into the session record detail interface, the cross-modal search result being obtained by employing a cross-modal search method according to any one of claims 1 to 10; An apparatus comprising:

17. A computing device comprising a processor, a memory, and a network interface; The processor is connected to the memory and the network interface, the network interface is used to provide a network communication function, the memory is used to store a program code, and the processor is used to call the program code to execute the cross-modal search method according to any one of claims 1 to 14. A computing device comprising:

18. A computer readable storage medium having a computer program stored thereon, the computer program comprising program instructions which, when executed by a processor, perform the cross-modal search method according to any one of claims 1 to 14. A computer-readable storage medium comprising:

19. A computer program product comprising a computer program or computer instructions which, when executed by a processor, implements the steps of the cross-modal search method according to any one of claims 1 to 14.

23. A computer program product comprising: