Cross-modal index processing method, cross-modal search method, cross-modal index processing device, cross-modal search device and related equipment

Abstract

The embodiment of the invention provides a cross-modal index processing method, a cross-modal search method, a cross-modal search device and related equipment, and relates to the technical field of computers. The cross-modal index processing method comprises the steps of obtaining a plurality of service fields by obtaining a to-be-processed service object, mapping the service fields into platform fields according to a mapping rule of a search platform, extracting feature vectors based on service data corresponding to the platform fields, and obtaining the service data corresponding to the platform fields. The index of the business object in the search platform can be determined according to the platform field and the extracted feature vector. In this way, the business fields of different types of business objects can be converted into unified platform fields of the search platform, and then the universal index is generated for the business data of multiple modals of the business objects based on the index design rule of the search platform, so that the repeated development condition in the cross-modal index can be reduced; therefore, the index development workload can be reduced, and the overall development efficiency of cross-modal search is improved.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of data search, and more specifically, the embodiments of this disclosure relate to a cross-modal indexing processing method, a cross-modal search method, an apparatus, and related equipment. Background Technology

[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] A modality refers to the form in which data exists, such as file formats like text, audio, images, and video. Data in different modalities may describe the same object; for example, a music album can contain songs in the audio modality and album art in the image modality. Cross-modal search allows users to search and match between different modalities (such as text, images, audio, and video), retrieving content in another modality (such as images or videos) by inputting data in one modality (such as a text description).

[0004] Currently, search platform databases offer a large number of business objects for users to search. When providing cross-modal search solutions, customized development is usually carried out based on business needs, requiring the construction of custom indexes for each type of object. This makes cross-modal search designs difficult to reuse, resulting in a large amount of redundant development. Summary of the Invention

[0005] This disclosure provides a cross-modal indexing processing method, a cross-modal search method, an apparatus, and related equipment to solve the problem of the difficulty in reusing cross-modal search designs.

[0006] In a first aspect of this disclosure, a cross-modal indexing processing method is provided, comprising:

[0007] Obtain the business object to be processed, the business object includes multiple business fields, and the business fields have multiple modalities;

[0008] The business fields are mapped to platform fields in the search platform according to the mapping rules of the search platform;

[0009] Extract feature vectors based on the business data corresponding to the platform fields;

[0010] The index of the business object in the search platform is determined based on the platform field and the feature vector.

[0011] In another embodiment of this disclosure, the platform field includes a general field, and mapping the business field to the platform field in the search platform according to the mapping rules of the search platform includes:

[0012] A first mapping relationship is determined based on the business type of the business object and the mapping rule. The first mapping relationship is used to indicate the correspondence between the business field and the general field.

[0013] Based on the first mapping relationship, the attributes and data of the business fields are mapped to the attributes and data of the general fields.

[0014] In another embodiment of this disclosure, the attributes of the general field include field name, modality, and index path. Mapping the attributes and data of the business field to the attributes and data of the general field according to the first mapping relationship includes:

[0015] Map the field names of the business fields to the field names of the general fields;

[0016] The modality of the business field is determined as the modality of the general field;

[0017] The index path of the general field is determined based on the modality of the general field.

[0018] In another embodiment of this disclosure, the platform field includes a custom field, and mapping the business field to the platform field in the search platform according to the mapping rules of the search platform includes:

[0019] The attributes of the custom field are determined based on the business type of the business object and the attributes of the business field;

[0020] Map the data of the business field to the data of the custom field.

[0021] In another embodiment of this disclosure, determining the attributes of the custom field based on the business type of the business object and the attributes of the business field includes:

[0022] The field name of the custom field is determined based on the business type of the business object;

[0023] The modality of the business field is determined as the modality of the custom field;

[0024] The index path of the custom field is determined based on the modality of the custom field.

[0025] In another embodiment of this disclosure, the platform field includes an index path, which is generated based on data or feature vectors of the platform field. Determining the index of the business object in the search platform based on the platform field and the feature vector includes:

[0026] The data of the platform field is stored in the database of the search platform to obtain the storage path of the platform field;

[0027] Generate the index field corresponding to the platform field based on the index path and the storage path;

[0028] The index of the business object is obtained by combining the index fields corresponding to multiple platform fields.

[0029] In another embodiment of this disclosure, the step of extracting feature vectors based on business data corresponding to the platform fields includes:

[0030] Obtain the modality of the platform field;

[0031] If the modality of the platform field is a preset modality, then feature vectors are extracted based on the data of the platform field, where the preset modality is text, image, video, or audio.

[0032] In another embodiment of this disclosure, the step of extracting feature vectors based on the platform field data includes:

[0033] The data from the platform field is input into a pre-trained feature extraction model to obtain the feature vector output by the feature extraction model.

[0034] In a second aspect of this disclosure, a cross-modal search method is provided, which can be applied to a search platform. The cross-modal search method includes:

[0035] Obtain the search criteria provided by the user;

[0036] Extract the feature vector to be searched from the data that conforms to the preset modality in the search conditions;

[0037] The query statement of the search platform is determined based on the feature vector to be searched;

[0038] The system searches for an index that matches the query statement in the database associated with the search platform and returns the data corresponding to the index, wherein the index is generated according to the cross-modal indexing method as described in any of the first aspects.

[0039] In another embodiment of this disclosure, obtaining the search criteria provided by the user includes:

[0040] Display multiple cross-modal search fields, which are used to indicate the modality of the input search content;

[0041] In response to a selection operation for any cross-modal search field, retrieve the search terms entered by the user.

[0042] In another embodiment of this disclosure, the step of extracting the feature vector to be searched based on data that conforms to a preset modality in the search conditions includes:

[0043] The cross-modal search field selected by the user is determined based on the search conditions, and the selected cross-modal search field and the search content are input into a pre-trained feature extraction model to obtain the search feature vector output by the feature extraction model.

[0044] In another embodiment of this disclosure, determining the query statement of the search platform based on the feature vector to be searched includes:

[0045] Based on the feature vector to be searched and the cross-modal search field selected by the user, a query statement is generated using a domain-specific language (DSL).

[0046] In another embodiment of this disclosure, the preset modality is text, image, video, or audio.

[0047] In a third aspect of this disclosure, a cross-modal indexing processing apparatus is provided, comprising:

[0048] The first acquisition module is used to acquire a business object to be processed. The business object includes multiple business fields, and the business fields have multiple modalities.

[0049] The mapping module is used to map the business fields to platform fields in the search platform according to the mapping rules of the search platform;

[0050] The first extraction module is used to extract feature vectors based on the business data corresponding to the platform fields;

[0051] An index determination module is used to determine the index of the business object in the search platform based on the platform field and the feature vector.

[0052] In a fourth aspect of this disclosure, a cross-modal search apparatus is provided for use in a search platform, the cross-modal search apparatus comprising:

[0053] The second acquisition module is used to acquire the search criteria provided by the user.

[0054] The second extraction module is used to extract the feature vector to be searched based on the data that conforms to the preset modality in the search conditions;

[0055] The query determination module is used to determine the query statement of the search platform based on the feature vector to be searched;

[0056] A query processing module is configured to search for an index matching the query statement in the database associated with the search platform and return the data corresponding to the index, wherein the index is generated according to the cross-modal index processing apparatus as described in any of the second aspects.

[0057] In a fifth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of the first aspects.

[0058] In a sixth aspect of this disclosure, a computing device is provided, comprising: at least one processor;

[0059] and a memory communicatively connected to the at least one processor;

[0060] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the computing device to perform the method as described in any one of the first aspects.

[0061] In a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in any of the first aspects.

[0062] According to the cross-modal indexing processing method, cross-modal search method, apparatus, and related equipment of this disclosure, multiple business fields can be obtained by acquiring the business object to be processed. Based on the mapping rules of the search platform, the business fields can be mapped to platform fields. Then, feature vectors are extracted based on the business data corresponding to the platform fields. The index of the business object in the search platform can be determined based on the platform fields and the extracted feature vectors. In this way, business fields of different types of business objects can be converted into unified platform fields of the search platform. Then, based on the index design rules of the search platform, a general index can be generated for the business data of the business object across multiple modalities. This reduces redundant development in cross-modal indexing, thereby reducing the workload of index development and improving the overall development efficiency of cross-modal search. Attached Figure Description

[0063] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0064] Figure 1 A schematic diagram illustrating an application scenario of the cross-modal indexing processing method according to an embodiment of the present disclosure is shown.

[0065] Figure 2 A schematic flowchart of a cross-modal indexing processing method according to an embodiment of the present disclosure is shown.

[0066] Figure 3A schematic diagram illustrating a mapping platform field according to an embodiment of the present disclosure is shown.

[0067] Figure 4 A schematic diagram illustrating another mapping platform field according to an embodiment of the present disclosure is shown.

[0068] Figure 5 A schematic diagram illustrating a process for determining a business object index according to an embodiment of the present disclosure is shown.

[0069] Figure 6 A schematic flowchart of a cross-modal search method according to an embodiment of the present disclosure is shown.

[0070] Figure 7 A schematic diagram of a cross-modal search architecture according to an embodiment of the present disclosure is shown.

[0071] Figure 8 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown;

[0072] Figure 9 A schematic diagram of the structure of a cross-modal indexing processing apparatus according to an embodiment of the present disclosure is shown.

[0073] Figure 10 A schematic diagram of the structure of a cross-modal search device according to an embodiment of the present disclosure is shown.

[0074] Figure 11 A schematic diagram of a computing device according to an embodiment of the present disclosure is shown.

[0075] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0076] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0077] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0078] According to embodiments of this disclosure, a cross-modal indexing processing method, a cross-modal search method, an apparatus, and related equipment are proposed.

[0079] In this article, it is important to understand that the terms used have the following meanings:

[0080] Modality: refers to the form in which data or information exists. Some data may exist in different forms, but they may describe the same thing or event. For example, the same thing can be represented in different forms such as text, audio, image, and video.

[0081] Cross-modal search: Retrieval refers to the technique of searching between data of different modalities (such as text, images, video, audio, etc.). It allows you to retrieve corresponding data of other modalities by querying data of one modality. For example, cross-modal search can be performed between images and text.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0083] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0084] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0086] Many application platforms offer cross-modal search capabilities, allowing them to retrieve relevant business objects from databases based on user-input queries and return them to the user. For example, in audio applications, users can enter lyrics in text format to search for songs, or album art in image format to search for albums.

[0087] Currently, cross-modal search is typically developed custom-based based on business needs. Before development, information such as the type of business object and the use case of cross-modal search needs to be obtained. Then, a custom index is built based on this information to search for business objects. For example, a business requirement is to search for videos with a similarity greater than 0.8 based on text. The typical process is: develop a large business table -> encode video information in the business data into feature vectors -> store the data in a search engine that supports vector search. This custom development model is often only suitable for specific needs. Once the business objects or use cases involved change, the already built indexes and related index search work cannot be reused, resulting in a large amount of redundant development for the application platform.

[0088] Through research on the development process of cross-modal search, the inventors have discovered that the reason why current cross-modal search designs are difficult to reuse is that different types of business objects are not stored in the same way in the database, and the design of indexes lacks a unified standard, resulting in poor usability.

[0089] The inventors thus conceived of providing an efficient indexing method for cross-modal search, which involves uniformly defining different types of business objects and establishing relationships between different modal data in vector form, thereby designing a universal cross-modal index. This approach improves the usability of index design, enables the construction of a one-stop cross-modal content search platform, and solves the problems of non-reusability and repetitive development in cross-modal search.

[0090] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0091] Application Scenarios Overview

[0092] First refer to Figure 1 , Figure 1 This is a schematic diagram illustrating one application scenario provided in this disclosure. For example... Figure 1 As shown, terminal 102 communicates with server 101 via a network. A data storage system can store the data that server 101 needs to process. The data storage system can be integrated onto server 101 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 101 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0093] For cross-modal indexing, server 101 can integrate resources from different modalities to create a resource pool containing several business objects. For example, for an audio application platform, different types of business objects such as music albums, songs, and playlists can be stored uniformly, and a general index for these business objects can be created based on data from different modalities such as album names, album covers, song lyrics, and song music videos.

[0094] For the cross-modal search process, the user can input query conditions through terminal 102, the server 101 can obtain the query conditions through communication with terminal 102 and search for the corresponding business object in the relevant database, and then output the relevant content of the business object to the user through terminal 102.

[0095] The application scenarios mentioned above are only partial examples. Those skilled in the art can extend the applications based on the cross-modal indexing and cross-modal search process. This disclosure does not impose specific limitations on these applications.

[0096] Exemplary methods

[0097] The following is combined with Figure 1 Application scenarios, refer to Figures 2 to 6 This document describes a cross-modal indexing processing method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0098] Figure 2 This is a flowchart illustrating a cross-modal indexing processing method according to an embodiment of this disclosure. Figure 2 As shown, the method may include:

[0099] Step S201: Obtain the business object to be processed.

[0100] The business object includes multiple business fields, and these business fields have multiple modalities.

[0101] The cross-modal indexing processing method of this disclosure can be used to construct an index for a search platform, the server of which can run on... Figure 1 Server 101 in the middle.

[0102] Optionally, multiple business fields corresponding to the business object that needs to be indexed can be obtained through data upload or import into an external database. These business fields can be used to store data of different modalities; for example, some business fields can store text data, some can store audio data, and some can store video data.

[0103] For example, the business object is a song, which may include multiple business fields such as song name, lyrics, music video, and release time. The song name field can store string data, the lyrics field can store text data, and the music video field can store video data.

[0104] Step S202: Map the business fields to platform fields in the search platform according to the mapping rules of the search platform.

[0105] The search platform can predefine standardized object attributes, which can be used to represent the attributes of platform fields used by the platform. Mapping rules can map the original business field attributes of business objects to the object attributes defined by the search platform.

[0106] Optionally, after obtaining the business fields of the business object, the attributes of the business fields can be mapped to the attributes of the platform fields to convert the business fields of the business object into fields used by the platform. For example, the mapping rule can include the mapping relationship between the field names of the business fields and the field names of the platform fields. For instance, the field name of the business field representing the song release time is "Song Release Date". According to the mapping relationship, the name of the platform field corresponding to the name of the business field is "Resource Release Date". If the data stored in the "Song Release Date" business field of a song is [20220101], then the platform field named "Resource Release Date" for the song in the search platform can be established according to the mapping relationship, and the data content [20220101] can be synchronized to the platform field.

[0107] Step S203: Extract feature vectors based on the business data corresponding to the platform fields.

[0108] Optionally, for cross-modal related business data in each business field of a business object, feature vectors can be extracted from this business data using a vector extraction model or related data processing algorithms, and the extracted feature vectors can be synchronized to the corresponding platform fields. The vector extraction model can include models such as clip (Contrastive Language-Image Pre-training, a pre-trained model based on contrastive text-image comparison).

[0109] Step S204: Determine the index of the business object in the search platform based on the platform fields and feature vectors.

[0110] Optionally, the search platform can pre-set an index template, which may include the platform fields that the index in the search platform should include, as well as the fields corresponding to different modalities. By filling the fields in the template with business data and feature vectors in the platform fields of the business object, the index corresponding to the business object can be obtained.

[0111] In the above embodiments, multiple business fields can be obtained by acquiring the business object to be processed. According to the mapping rules of the search platform, these business fields can be mapped to platform fields. Then, feature vectors are extracted based on the business data corresponding to the platform fields. The index of the business object in the search platform can be determined based on the platform fields and the extracted feature vectors. In this way, business fields of different types of business objects can be converted into unified platform fields of the search platform. Then, based on the index design rules of the search platform, a general index can be generated for the business data of the business object in multiple modalities. This reduces redundant development in cross-modal indexing, thereby reducing the workload of index development and improving the overall development efficiency of cross-modal search.

[0112] In one example, the business object to be processed is a song. This song may include multiple business fields such as song name, song release date, and lyrics. The business fields of the song can be referred to in Table 1 below. The search platform can predefine the platform fields used by objects in the platform. Taking the song as the business object in this example, the business fields of the song are mapped to platform fields according to the search platform's rules. The mapping results can be referred to in Table 2 below.

[0113] Field Name Field data Data types Song Title Alphabet Song String Song release date 20210101 Date type lyrics abcdefg text ··· ··· ···

[0114] Table 1

[0115] Field Name Field data Data types Resource Name Alphabet Song String Resource release date 20210101 Date type Resource Text 1 abcdefg text ··· ··· ···

[0116] Table 2

[0117] In another example, the business object to be processed is an album, which may include multiple business fields such as album name, album release date, and album cover. These business fields can be found in Table 3 below. The search platform can predefine the platform fields used by objects within the platform. Taking the album as an example in this case, the album's business fields are mapped to platform fields according to the search platform's rules. The mapping results can be found in Table 4 below.

[0118] Field Name Field data Data types Album Title Alphabet Song String Album release date 20220202 Date type Album cover Cover.jpg image ··· ··· ···

[0119] Table 3

[0120] Field Name Field data Data types Resource Name Alphabet Song String Resource release date 20210101 Date type Resource Image 1 Cover.jpg image ··· ··· ···

[0121] Table 4

[0122] In one embodiment, the platform fields include generic fields, such as Figure 3 As shown, mapping business fields to platform fields in the search platform according to the platform's mapping rules can include:

[0123] Step S301: Determine the first mapping relationship based on the business type and mapping rules of the business object.

[0124] The first mapping relationship indicates the correspondence between business fields and general fields. General fields can be fields in the platform fields that are applicable to all types of business objects.

[0125] Optionally, the mapping rules may include several mapping relationships. Each mapping relationship can be used to map the business fields of a business object of a certain business type to the corresponding general fields in the search platform. Based on the business type of the business object, the corresponding first mapping relationship can be matched from these mapping relationships. For example, the mapping rules may include mapping relationships that map the business fields of songs to general fields, mapping relationships that map the business fields of albums to general fields, etc.

[0126] Step S302: Map the attributes and data of the business fields to the attributes and data of the general fields according to the first mapping relationship.

[0127] The attributes must include at least the field name and the data type, which can be any modality such as text, image, and video.

[0128] For example, Table 1 and Table 2 can be referenced. Table 2 includes platform fields such as resource name, resource release date, and custom field 1. Resource name and resource release date can be general fields. The first mapping relationship is used to indicate the correspondence between song name and resource name and song release date and resource release date. According to this correspondence, the field data and data type of the business field can be synchronized to the platform field.

[0129] In some possible implementations, the attributes of the generic field include field name, modality, and index path. The attributes and data of the business field are mapped to the attributes and data of the generic field according to the first mapping relationship, including:

[0130] Map the field names of business fields to the field names of general fields; determine the modality of business fields to the modality of general fields; determine the index path of general fields based on the modality of general fields.

[0131] The mapping between field names and modalities can be referenced from the mapping process in Tables 1 and 2. The index path can be used by the search platform to match user-input query conditions, serving as a field for the corresponding search engine.

[0132] For generic fields, the index path can be determined based on the modality. If the generic field is a data type that is easy to retrieve, such as a string or date, the field name of the generic field can be used as the index path.

[0133] In one embodiment, the platform fields include custom fields, such as... Figure 4 As shown, mapping business fields to platform fields in the search platform according to the platform's mapping rules can include:

[0134] Step S401: Determine the attributes of the custom field based on the business type and attributes of the business field of the business object.

[0135] Optionally, in addition to general fields, platform fields may also include custom fields related to the business type of the business object. For example, referring to the mapping processes of Tables 1 to 2 and Tables 3 to 4, "Resource Text 1" in Table 2 is a custom field mapped to a business object of the song type, and "Resource Image 1" in Table 4 is a custom field mapped to a business object of the album type. Mapping rules may include mapping relationships that map business fields of songs to custom fields, mapping relationships that map business fields of albums to custom fields, etc.

[0136] In some possible implementations, the field name of the custom field can be determined based on the business type of the business object; the modality of the business field can be determined as the modality of the custom field; and the index path of the custom field can be determined based on the modality of the custom field.

[0137] The mapping between field names and modalities can refer to the mapping process from the lyrics field in Table 1 to resource text 1 in Table 2, or the mapping process from the album cover field in Table 3 to resource image 1 in Table 4.

[0138] For custom fields, the index path can be determined based on the modality. If the custom field is a data type such as text or image, the index path can be generated based on the modality of the custom field. For example, if the custom field corresponds to text data, the index path can be named text1; if the custom field corresponds to image data, the index path can be named picture1. It is understandable that if multiple custom fields are all of image data type, the index paths for these custom fields can be picture1, picture2, picture3, ..., picture10, etc.

[0139] Step S402: Map the data of the business field to the data of the custom field.

[0140] For example, after determining the mapping relationship between business fields and custom fields, the field data of the business fields can be synchronized to the corresponding custom fields.

[0141] In one embodiment, the platform field includes an index path, which is generated based on data or feature vectors of the platform field, such as... Figure 5As shown, determining the index of a business object in the search platform based on platform fields and feature vectors can include:

[0142] Step S501: Store the platform field data in the database of the search platform to obtain the storage path of the platform field.

[0143] Optionally, after mapping the business fields of the business object to the platform fields of the search platform, the data of all business fields can be stored in the database associated with the search platform, and the storage path of the data corresponding to each platform field can be recorded.

[0144] Step S502: Generate the index field corresponding to the platform field based on the index path and storage path.

[0145] Optionally, for the platform field obtained after mapping the business object, the index path of the platform field can be used as the field name and the storage path can be used as the field data to generate the index field corresponding to the platform field.

[0146] In some possible implementations, if the business data corresponding to the platform field is extracted to obtain a feature vector, the extracted feature vector can be updated to the index field so that the index field contains the feature vector of the platform field.

[0147] Step S503: Obtain the index of the business object by combining the index fields corresponding to multiple platform fields.

[0148] Specifically, the index fields corresponding to the various platform fields of a business object in the search platform can be integrated into a general index for that business object.

[0149] In one embodiment, extracting feature vectors based on business data corresponding to platform fields may include:

[0150] Obtain the modality of the platform field; if the modality of the platform field is a preset modality, extract the feature vector based on the data of the platform field.

[0151] The preset modalities are text, images, videos, or audio.

[0152] Specifically, it can be determined whether feature vectors need to be extracted based on the modality of the platform field. If the modality is any of text, image, video, or audio, then feature vectors can be extracted from the business data of that platform field.

[0153] In some possible implementations, feature vectors are extracted based on platform field data, including: inputting platform field data into a pre-trained feature extraction model and obtaining the feature vectors output by the feature extraction model. The feature extraction model can be a clip model.

[0154] For example, if the business object is a song, the platform fields corresponding to the song include lyrics and MV video, etc. The lyrics are text data and the MV video is video data. Then the clip model can be used to extract feature vectors from the data of these two fields respectively.

[0155] According to another aspect of this disclosure, a cross-modal search method is also provided. For example... Figure 6 As shown, this cross-modal search method may include:

[0156] Step S601: Obtain the search criteria provided by the user.

[0157] The search criteria can be data in different modalities, such as voice, text, video, or audio, input by the user.

[0158] Optionally, an application client running on terminal 102 can provide a human-computer interaction interface to the user, allowing the user to input search criteria such as song lyrics and song titles through input boxes.

[0159] The cross-modal search method disclosed herein can be applied to search platforms.

[0160] Step S602: Extract the feature vector to be searched based on the data that meets the preset modality in the search conditions.

[0161] The preset modalities are text, images, videos, or audio.

[0162] Optionally, after obtaining the search criteria, it can be detected whether the search criteria contain data of preset modalities such as text, images, videos or audio. If they do, feature vectors can be extracted from the data to obtain the feature vector to be searched.

[0163] For example, if the user inputs a piece of lyrics, the lyrics can be processed by the clip model to obtain the feature vector output by the model as the feature vector to be searched.

[0164] It should be noted that the feature vector extraction can refer to the process of extracting feature vectors from business data of business objects in the above embodiments.

[0165] Step S603: Determine the query statement of the search platform based on the feature vector to be searched.

[0166] Optionally, the extracted feature vectors to be searched can be used as database query conditions, and corresponding query statements (such as SQL statements) can be generated according to the query rules of the search platform.

[0167] Step S604: Search the database associated with the search platform for an index that matches the query statement and return the data corresponding to the index.

[0168] The index is generated according to the cross-modal indexing processing method in any of the above embodiments.

[0169] Optionally, based on the query, the search platform can match indexes that meet the similarity criteria in its associated database, and then find the data stored in the corresponding location based on the index to return the data to the user. For example, the feature vector to be searched is extracted based on the lyrics text input by the user. The database stores a large number of indexes of platform fields corresponding to the lyrics. The similarity between the feature vector to be searched and the relevant indexes in the search platform can be calculated, and the index corresponding to the user's search criteria can be matched based on the similarity.

[0170] In one embodiment, obtaining the search criteria provided by the user includes:

[0171] Displays multiple cross-modal search fields; in response to a selection action for any cross-modal search field, retrieves the search terms entered by the user.

[0172] The cross-modal search field is used to indicate the modality of the search content.

[0173] For example, cross-modal search fields can be pre-configured through the search platform's server, such as adding search fields like album art or song lyrics. These cross-modal search fields can be associated with the index paths in the platform's fields.

[0174] For example, multiple options can be displayed through the human-computer interaction interface provided by terminal 102. Each option corresponds to a cross-modal search field (such as album cover). Users can select any cross-modal search field and enter the corresponding search content, such as selecting an album cover and entering search content in image format.

[0175] In some possible implementations, extracting the feature vector to be searched based on data that conforms to a preset modality in the search criteria may include:

[0176] The cross-modal search field selected by the user is determined based on the search criteria, and the selected cross-modal search field and search content are input into the pre-trained feature extraction model to obtain the search feature vector output by the feature extraction model.

[0177] In some possible implementations, the query statement of the search platform is determined based on the feature vector to be searched, including: generating the query statement through a domain-specific language (DSL) based on the feature vector to be searched and the cross-modal search fields selected by the user.

[0178] Among them, DSL (Domain-Specific Language) is a programming language or description language that can be used for designing specific application domains.

[0179] Figure 7 This is a schematic diagram illustrating a cross-modal search architecture provided as an example of this disclosure. Figure 7 As shown, the search platform can define platform field attributes and integrate business objects of different business types into a resource pool. This resource pool can be a database associated with the search platform. For data in text, image, video, and audio modalities, feature vectors can be extracted as cross-modal index fields. These vectors can be combined with regular data in the resource pool of other modalities besides text, images, videos, and audio to obtain a general index. Users can input search terms into the search platform. For data in modalities other than text, images, videos, and audio, such as strings, regular search methods can be used. Conversely, for data in any of the text, image, video, and audio modalities, feature vectors can be extracted and assembled with data in other modalities, such as strings, to generate a query statement. Finally, the query statement is matched with the general index in the search platform to obtain and return the data needed by the user.

[0180] Exemplary media

[0181] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 7 The storage medium of the exemplary embodiments of this disclosure will be described.

[0182] refer to Figure 8 As shown, the storage medium 80 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0183] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0184] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0185] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0186] Exemplary device

[0187] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 9 The cross-modal indexing processing apparatus of the exemplary embodiments of this disclosure is described to implement the method in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0188] like Figure 9 As shown, the cross-modal indexing processing device 900 may include:

[0189] The first acquisition module 901 is used to acquire a business object to be processed. The business object includes multiple business fields, and the business fields have multiple modalities.

[0190] The mapping module 902 is used to map the business field to the platform field in the search platform according to the mapping rules of the search platform.

[0191] The first extraction module 903 is used to extract feature vectors based on the business data corresponding to the platform fields.

[0192] The index determination module 904 is used to determine the index of the business object in the search platform based on the platform field and the feature vector.

[0193] In yet another embodiment of this disclosure, the mapping module 902 may include:

[0194] The general mapping determination unit can be used to determine a first mapping relationship based on the business type of the business object and the mapping rule. The first mapping relationship is used to indicate the correspondence between the business field and the general field.

[0195] The first mapping unit can be used to map the attributes and data of the business field to the attributes and data of the general field according to the first mapping relationship.

[0196] In yet another embodiment of this disclosure, the first mapping unit is further configured to: map the field name of the business field to the field name of the general field; determine the modality of the business field as the modality of the general field; and determine the index path of the general field based on the modality of the general field.

[0197] In yet another embodiment of this disclosure, the mapping module 902 may further include:

[0198] The custom mapping determination unit can be used to determine the attributes of a custom field based on the business type of the business object and the attributes of the business field.

[0199] The second mapping unit can be used to map the data of the business field to the data of the custom field.

[0200] In yet another embodiment of this disclosure, the custom mapping determination unit is further configured to: determine the field name of the custom field according to the business type of the business object; determine the modality of the business field as the modality of the custom field; and determine the index path of the custom field according to the modality of the custom field.

[0201] In yet another embodiment of this disclosure, the index determination module 904 may include:

[0202] The platform field storage unit can be used to store the platform field data in the database of the search platform to obtain the storage path of the platform field.

[0203] The index generation unit can be used to generate the index field corresponding to the platform field based on the index path and the storage path.

[0204] The index combination unit can be used to obtain the index of the business object by combining the index fields corresponding to multiple platform fields.

[0205] In yet another embodiment of this disclosure, the first extraction module 903 may include:

[0206] The modality acquisition unit can be used to acquire the modality of the platform field.

[0207] The vector extraction unit can be used to extract feature vectors based on the data of the platform field when the modality of the platform field is a preset modality, wherein the preset modality is text, image, video or audio.

[0208] In yet another embodiment of this disclosure, the vector extraction unit is further configured to: input the data of the platform field into a pre-trained feature extraction model, and obtain the feature vector output by the feature extraction model.

[0209] According to another aspect of this disclosure, a cross-modal search device 1000 is also provided. For example... Figure 10 As shown, the cross-modal search device 1000 may include:

[0210] The second acquisition module 1001 is used to acquire the search conditions provided by the user.

[0211] The second extraction module 1002 is used to extract the feature vector to be searched based on the data that conforms to the preset mode in the search conditions.

[0212] The query determination module 1003 is used to determine the query statement of the search platform based on the feature vector to be searched.

[0213] The query processing module 1004 is used to find an index that matches the query statement in the database associated with the search platform and return the data corresponding to the index, wherein the index is generated according to the cross-modal index processing device 900 in any of the above embodiments.

[0214] In yet another embodiment of this disclosure, the second acquisition module 1001 may include:

[0215] The display unit can be used to display multiple cross-modal search fields, which are used to indicate the modality of the input search content.

[0216] The input acquisition unit can be used to acquire the search content entered by the user in response to a selection operation for any cross-modal search field.

[0217] In yet another embodiment of this disclosure, the second extraction module 1002 may include: generating a query statement using a domain-specific language (DSL) based on the feature vector to be searched and the cross-modal search field selected by the user.

[0218] In yet another embodiment of this disclosure, the preset modality is text, image, video, or audio.

[0219] Exemplary computing device

[0220] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 11A computing device according to an exemplary embodiment of the present disclosure will be described.

[0221] Figure 11 The computing device 110 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0222] like Figure 11 As shown, the computing device 110 is presented in the form of a general-purpose computing device. The components of the computing device 110 may include, but are not limited to: at least one processing unit 1101, at least one storage unit 1102, and a bus 1103 connecting different system components (including the processing unit 1101 and the storage unit 1102). The at least one storage unit 1102 stores computer-executable instructions; the at least one processing unit 1101 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0223] Bus 1103 includes a data bus, a control bus, and an address bus.

[0224] Storage unit 1102 may include readable media in the form of volatile memory, such as random access memory (RAM) 11021 and / or cache memory 11022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 11023.

[0225] Storage unit 1102 may also include a program / utility 11025 having a set (at least one) of program modules 11024, such program modules 11024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0226] The computing device 110 can also communicate with one or more external devices 1104 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 1105. Furthermore, the computing device 110 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1106. Figure 11 As shown, network adapter 1106 communicates with other modules of computing device 110 via bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 110, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0227] It should be noted that although several units / modules or sub-units / modules of the cross-modal indexing processing apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0228] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0229] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A cross-modal indexing processing method, characterized in that, include: Obtain the business object to be processed, the business object includes multiple business fields, and the business fields have multiple modalities; The business fields are mapped to platform fields in the search platform according to the mapping rules of the search platform; Extract feature vectors based on the business data corresponding to the platform fields; The index of the business object in the search platform is determined based on the platform field and the feature vector.

2. The processing method according to claim 1, characterized in that, The platform fields include general fields, and mapping the business fields to platform fields in the search platform according to the search platform's mapping rules includes: A first mapping relationship is determined based on the business type of the business object and the mapping rule. The first mapping relationship is used to indicate the correspondence between the business field and the general field. Based on the first mapping relationship, the attributes and data of the business fields are mapped to the attributes and data of the general fields.

3. The processing method according to claim 2, characterized in that, The attributes of the general field include field name, modality, and index path. Mapping the attributes and data of the business field to the attributes and data of the general field according to the first mapping relationship includes: Map the field names of the business fields to the field names of the general fields; The modality of the business field is determined as the modality of the general field; The index path of the general field is determined based on the modality of the general field.

4. The processing method according to claim 1, characterized in that, The platform fields include custom fields, and mapping the business fields to platform fields in the search platform according to the search platform's mapping rules includes: The attributes of the custom field are determined based on the business type of the business object and the attributes of the business field; Map the data of the business field to the data of the custom field.

5. The processing method according to claim 4, characterized in that, The step of determining the attributes of the custom field based on the business type of the business object and the attributes of the business field includes: The field name of the custom field is determined based on the business type of the business object; The modality of the business field is determined as the modality of the custom field; The index path of the custom field is determined based on the modality of the custom field.

6. A cross-modal search method, characterized in that, The cross-modal search method, applied to a search platform, includes: Obtain the search criteria provided by the user; Extract the feature vector to be searched from the data that conforms to the preset modality in the search conditions; The query statement of the search platform is determined based on the feature vector to be searched; The system searches for an index that matches the query statement in the database associated with the search platform and returns the data corresponding to the index, wherein the index is generated according to the cross-modal indexing processing method as described in any one of claims 1 to 5.

7. A cross-modal indexing processing device, characterized in that, include: The first acquisition module is used to acquire a business object to be processed. The business object includes multiple business fields, and the business fields have multiple modalities. The mapping module is used to map the business fields to platform fields in the search platform according to the mapping rules of the search platform; The first extraction module is used to extract feature vectors based on the business data corresponding to the platform fields; An index determination module is used to determine the index of the business object in the search platform based on the platform field and the feature vector.

8. A cross-modal search device, characterized in that, The cross-modal search device, applied to a search platform, includes: The second acquisition module is used to acquire the search criteria provided by the user. The second extraction module is used to extract the feature vector to be searched based on the data that conforms to the preset modality in the search conditions; The query determination module is used to determine the query statement of the search platform based on the feature vector to be searched; The query processing module is used to find an index that matches the query statement in the database associated with the search platform and return the data corresponding to the index, wherein the index is generated according to the cross-modal index processing device as described in claim 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 6.

10. A computing device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the computing device to perform the method as described in any one of claims 1 to 6.

Images