Data search method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to a data search method and device, computer equipment and a computer readable storage medium. The method comprises the steps of obtaining multi-modal service data corresponding to a data search request in response to the data search request; extracting single-mode service data characteristics of the single-mode service data; fusing the plurality of single-mode business data features and the plurality of attention weights into a multi-mode attention weight feature; fusing the multi-modal attention weight feature with each single-modal service data feature to obtain a target single-modal service data feature corresponding to each single-modal service data feature; for each data mode, searching similar service data corresponding to the single-mode service data in the data mode according to the target single-mode service data features, and detecting the similarity between the similar service data and the single-mode service data; and screening out the target similar business data with the highest similarity, and outputting the target similar business data. By adopting the method, data search can be accurately carried out.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data search method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of science and technology, the processing and analysis of multimodal data has become an important research direction in the field of data processing. Multimodal data refers to information from different sources and in different forms of expression, such as text, voice, image, etc. This type of data has rich semantic information and can provide more comprehensive and accurate information support for various application scenarios. However, when it is necessary to search for similar data based on a certain data, and the data is multimodal data, due to the heterogeneity and complexity between different modal data, how to achieve effective multimodal data search has become a problem to be solved.

[0003] In traditional technology, when searching for multimodal data, the multimodal data is often fused into a target data through feature splicing or weighted averaging, and then similar data of the target data is searched from the database through machine learning and other methods to obtain the search results of the multimodal data.

[0004] However, the current data search method is not accurate enough. Summary of the invention

[0005] Based on this, it is necessary to provide an accurate data search method, device, computer equipment, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a data search method, comprising:

[0007] In response to the data search request, obtaining multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data;

[0008] Extracting the unimodal business data features of each unimodal business data, and obtaining the attention weight corresponding to the data modality of each unimodal business data feature;

[0009] Fusion of multiple unimodal business data features and multiple attention weights into multimodal attention weight features;

[0010] The multimodal attention weight features are respectively fused with each unimodal business data feature to obtain the target unimodal business data feature corresponding to each unimodal business data feature;

[0011] For each data modality, according to the characteristics of the target unimodal business data, search for similar business data corresponding to the unimodal business data under the data modality, and detect the similarity between the similar business data and the unimodal business data;

[0012] Filter out the target similar business data with the highest similarity from the similar business data searched under all data modes;

[0013] The target similar business data is output as the data search result corresponding to the data search request.

[0014] In one embodiment, a plurality of unimodal business data features and a plurality of attention weights are fused into a multimodal attention weight feature, including:

[0015] For each data modality, a unimodal attention weight feature is generated based on the unimodal business data features and the attention weight under the data modality;

[0016] The unimodal attention weight features under all data modalities are fused into multimodal attention weight features.

[0017] In one embodiment, extracting the unimodal service data features of each unimodal service data includes:

[0018] For each data mode, encoding the single-mode service data to generate single-mode service encoded data;

[0019] Feature extraction is performed on the unimodal service coded data to obtain unimodal service data features.

[0020] In one embodiment, feature extraction is performed on the unimodal service coded data to obtain unimodal service data features, including:

[0021] Detecting a query vector, a key vector, and a value vector corresponding to unimodal service encoding data;

[0022] Generate a correlation vector between the query vector and the key vector, and process the correlation vector through an activation function to obtain a target correlation vector;

[0023] Based on the target relevance vector and the value vector, a unimodal business data feature is generated.

[0024] In one embodiment, searching for similar service data corresponding to the unimodal service data in a data modality according to the target unimodal service data features includes:

[0025] According to the characteristics of the target unimodal business data, the data search model is used to search for similar business data corresponding to the unimodal business data under the modality;

[0026] The iterative optimization process of the data search model includes:

[0027] Data acquisition step: acquiring historical multimodal business data, wherein the historical multimodal business data includes historical single-modal business data under multiple data modes;

[0028] Extracting actual similar business data corresponding to each data modality of historical multimodal business data and historical unimodal business data features of each historical unimodal business data, and obtaining the attention weight corresponding to the data modality of each historical unimodal business data feature;

[0029] Fusion of multiple historical unimodal business data features and multiple attention weights into historical multimodal attention weight features;

[0030] The historical multimodal attention weight features are respectively fused with each historical unimodal business data feature to obtain the historical target unimodal business data features corresponding to each historical unimodal business data feature;

[0031] For each data modality, based on the historical target unimodal business data characteristics, the data search model is used to search for search result business data similar to the historical unimodal business data under the data modality;

[0032] Based on the difference between the actual similar business data and the search result business data under each data modality, the data search model is iteratively optimized.

[0033] In one embodiment, based on the difference between the actual similar business data and the search result business data under each data modality, iteratively optimizing the data search model includes:

[0034] Generate a model loss for each data modality based on the difference between the actual similar business data and the search result business data for each data modality;

[0035] According to multiple attention weights, the model loss under each data modality is fused to obtain the total model loss;

[0036] If the total model loss is less than the preset model loss threshold, it is determined that the iterative optimization of the data search model is completed;

[0037] If the total model loss is not less than the preset model loss threshold, the data search model is updated according to the total model loss, and the data acquisition step is returned until the calculated total model loss is less than the preset model loss threshold.

[0038] In a second aspect, the present application also provides a data search device, the device comprising:

[0039] A data acquisition module, configured to acquire, in response to a data search request, multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data;

[0040] An extraction module, used to extract the unimodal business data features of each unimodal business data, and obtain the attention weight corresponding to the data modality of each unimodal business data feature;

[0041] A weight feature generation module, used to fuse multiple unimodal business data features with multiple attention weights into a multimodal attention weight feature;

[0042] A feature fusion module is used to fuse the multimodal attention weight feature with each unimodal business data feature to obtain the target unimodal business data feature corresponding to each unimodal business data feature;

[0043] A search module is used to search for similar business data corresponding to the unimodal business data under the data modality for each data modality according to the characteristics of the target unimodal business data, and detect the similarity between the similar business data and the unimodal business data;

[0044] A screening module is used to screen out target similar business data with the highest similarity from similar business data searched under all data modes;

[0045] The search result output module is used to output the target similar business data as the data search result corresponding to the data search request.

[0046] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0047] In response to the data search request, obtaining multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data;

[0048] Extracting the unimodal business data features of each unimodal business data, and obtaining the attention weight corresponding to the data modality of each unimodal business data feature;

[0049] Fusion of multiple unimodal business data features and multiple attention weights into multimodal attention weight features;

[0050] The multimodal attention weight features are respectively fused with each unimodal business data feature to obtain the target unimodal business data feature corresponding to each unimodal business data feature;

[0051] For each data modality, according to the characteristics of the target unimodal business data, search for similar business data corresponding to the unimodal business data under the data modality, and detect the similarity between the similar business data and the unimodal business data;

[0052] Filter out the target similar business data with the highest similarity from the similar business data searched under all data modes;

[0053] The target similar business data is output as the data search result corresponding to the data search request.

[0054] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0055] In response to the data search request, obtaining multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data;

[0056] Extracting the unimodal business data features of each unimodal business data, and obtaining the attention weight corresponding to the data modality of each unimodal business data feature;

[0057] Fusion of multiple unimodal business data features and multiple attention weights into multimodal attention weight features;

[0058] The multimodal attention weight features are respectively fused with each unimodal business data feature to obtain the target unimodal business data feature corresponding to each unimodal business data feature;

[0059] For each data modality, according to the characteristics of the target unimodal business data, search for similar business data corresponding to the unimodal business data under the data modality, and detect the similarity between the similar business data and the unimodal business data;

[0060] Filter out the target similar business data with the highest similarity from the similar business data searched under all data modes;

[0061] The target similar business data is output as the data search result corresponding to the data search request.

[0062] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0063] In response to the data search request, obtaining multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data;

[0064] Extracting the unimodal business data features of each unimodal business data, and obtaining the attention weight corresponding to the data modality of each unimodal business data feature;

[0065] Fusion of multiple unimodal business data features and multiple attention weights into multimodal attention weight features;

[0066] The multimodal attention weight features are respectively fused with each unimodal business data feature to obtain the target unimodal business data feature corresponding to each unimodal business data feature;

[0067] For each data modality, according to the characteristics of the target unimodal business data, search for similar business data corresponding to the unimodal business data under the data modality, and detect the similarity between the similar business data and the unimodal business data;

[0068] Filter out the target similar business data with the highest similarity from the similar business data searched under all data modes;

[0069] The target similar business data is output as the data search result corresponding to the data search request.

[0070] The above-mentioned data search method, device, computer equipment, computer-readable storage medium and computer program product, in response to a data search request, obtains the multimodal business data corresponding to the data search request, and extracts the unimodal business data features of each unimodal business data, and generates a multimodal attention weight feature of the multimodal business data according to the attention weight corresponding to the data modality of each unimodal business data feature and multiple unimodal business data features. At this time, the multimodal attention weight feature is a feature that is associated with multiple unimodal business data features, and then the multimodal attention weight feature and the multiple unimodal business data features are respectively Row fusion is performed to generate target unimodal business data features corresponding to each unimodal business data feature. The generated target unimodal business data features are the data obtained after effective fusion of data features between all data modalities. Therefore, for each data modality, according to the target unimodal business data features, the process of searching for similar business data corresponding to the unimodal business data under the data modality is also more accurate. Furthermore, the target similar business data with the highest similarity can be screened out from the similar business data of all data modalities, which further improves the accuracy of data search and makes the final output data search result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 An application environment diagram of a data search method in an embodiment;

[0073] Figure 2 A schematic diagram of a flow chart of a data search method in one embodiment;

[0074] Figure 3 A schematic diagram of a flow chart of a data search method in another embodiment;

[0075] Figure 4 It is an overall interaction diagram of the iterative optimization phase of a data search model in one embodiment;

[0076] Figure 5 A module interaction diagram of a model in an iterative optimization phase of a data search model in one embodiment;

[0077] Figure 6 is a structural block diagram of a data search device in one embodiment;

[0078] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are used to explain the present application and are not used to limit the present application.

[0080] The data search method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the user through the network.

[0081] The user interacts with the data search interface in the terminal 102, uploads the multimodal business data to the terminal 102, and triggers the data search button on the data search interface to generate a data search request to the terminal 102. At this time, the terminal 102 responds to the data search request and obtains the multimodal business data corresponding to the data search request, wherein the multimodal business data is composed of multiple single-modal business data; extracts the single-modal business data features of each single-modal business data, and obtains the attention weight corresponding to the data modality of each single-modal business data feature; merges the multiple single-modal business data features with the multiple attention weights into a multimodal Attention weight feature; fuse the multimodal attention weight feature with each unimodal business data feature to obtain the target unimodal business data feature corresponding to each unimodal business data feature; for each data modality, search for similar business data corresponding to the unimodal business data under the data modality according to the target unimodal business data feature, and detect the similarity between the similar business data and the unimodal business data; select the target similar business data with the highest similarity from the similar business data searched under all data modalities; output the target similar business data as the data search result corresponding to the data search request. Furthermore, the data search results can also be displayed to the user.

[0082] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0083] In an exemplary embodiment, Figure 2 As shown, a data search method is provided, which is applied to Figure 1 The terminal 102 in FIG. 1 is used as an example for explanation.

[0084] S100, in response to a data search request, obtaining multimodal service data corresponding to the data search request.

[0085] Among them, multimodal business data is composed of multiple unimodal business data, and unimodal business data is obtained by representing business data through different data representation forms. Multimodality refers to the use of two or more different information representation or communication modes to convey information in information processing and communication. These modes can include text, images, audio, video, gestures, touch and other sensory information. In the field of artificial intelligence and machine learning, multimodal technology allows the system to fuse information from different modes, thereby providing richer and more comprehensive understanding and interpretation.

[0086] Specifically, the user interacts with the data search interface in the terminal, uploads the multimodal business data to the terminal, and triggers the data search button on the data search interface to generate a data search request to the terminal. At this time, the terminal responds to the data search request and obtains the multimodal business data corresponding to the data search request. Among them, the multimodal business data can be obtained by collecting business data of different modes through different channels and methods. The data of different modes include but are not limited to text, voice, and images. These business data of different modes can come from different fields, such as social media, news, forums, and blogs.

[0087] Furthermore, after collecting multimodal business data, it is necessary to preprocess the collected multimodal business data. The means of preprocessing include but are not limited to data cleaning, format conversion, and labeling. Among them, data cleaning is mainly to remove irrelevant, duplicate, and erroneous data; format conversion is to convert data from different sources into a unified format for subsequent processing; labeling is to classify or mark the data to provide a basis for subsequent classification or identification. In addition, a variety of random noises can be added to perform data enhancement processing on the collected data, and then normalization is performed to update the multimodal business data.

[0088] In an exemplary embodiment, each unimodal service data in the multimodal service data corresponds to a data representation type, and the data representation form of the unimodal service data may be text data, voice data, image data or other types of data.

[0089] S200, extracting unimodal business data features of each unimodal business data, and obtaining an attention weight corresponding to the data modality of each unimodal business data feature.

[0090] Among them, the attention weight refers to the weight of each unimodal business data feature in all unimodal business data features during feature fusion, which can be obtained through a certain attention mechanism in combination with specific scenarios to make it more suitable for actual application scenarios.

[0091] Specifically, feature extraction is performed on each unimodal service data to obtain unimodal service data features corresponding to each unimodal service data.

[0092] More specifically, the method constructs a multi-head input and multi-head output model, which aims to simultaneously process data of different modalities and realize effective fusion of information. Therefore, the present application first performs multi-head feature extraction on multiple unimodal business data, that is, performs separate feature extraction on each unimodal business data, so that the machine can notice the correlation between different parts of various types of unimodal business data itself, so as to obtain the unimodal business data features corresponding to the unimodal business data under different data modes. Generally, the feature extraction method is feature extraction based on the self-attention mechanism, that is, through the self-attention mechanism, feature extraction is performed on each unimodal business data separately to obtain the unimodal business data features corresponding to each unimodal business data. Furthermore, the attention weight of each unimodal business data feature at the corresponding feature position can also be obtained.

[0093] S300, multiple unimodal business data features and multiple attention weights are merged into a multimodal attention weight feature.

[0094] Specifically, a data set consisting of multiple unimodal business data features is sent to a feature fusion module based on an attention mechanism, so that multiple unimodal business data features are fused. In the process of feature fusion, it is not a simple feature splicing, but it is necessary to pay more attention to certain unimodal business data features, that is, to perform weighted feature fusion on multiple unimodal business data features.

[0095] To achieve this goal, first, we need to obtain the attention weights G of multiple unimodal business data features at the corresponding feature positions. i d , where i represents the i-th unimodal business data feature. Based on all unimodal business data features and the attention weights corresponding to all unimodal business data features, a multimodal attention weight feature is generated. The multimodal attention weight feature is the weight information that integrates all unimodal business data features.

[0096] S400, fusing the multimodal attention weight feature with each unimodal business data feature respectively to obtain the target unimodal business data feature corresponding to each unimodal business data feature.

[0097] Specifically, when the multimodal attention weight feature is obtained, the multimodal attention weight feature is fused with each unimodal business data feature respectively, so that each unimodal business data feature can be associated with other unimodal business data features, thereby enhancing the correlation between multimodal data features and realizing effective fusion between different modal data. Finally, the target unimodal business data feature corresponding to each unimodal business data feature is generated.

[0098] S500, for each data modality, according to the target unimodal business data characteristics, search for similar business data corresponding to the unimodal business data under the data modality, and detect the similarity between the similar business data and the unimodal business data.

[0099] Specifically, the present application can perform independent data searches for target unimodal business data features of each data modality. More specifically, for each data modality, based on the target unimodal business data features under the data modality, similar business data similar to the unimodal business data under the data modality is searched from the database, that is, the business data features in the similar business data are similar or identical to the target unimodal business data features. Further, the data modality of the similar business data is the same as the data modality of the target unimodal business data features. Furthermore, it is also necessary to detect the similarity between the similar business data and the unimodal business data.

[0100] For example, in the case where multimodality includes text data, voice data, and image data, the target unimodal business data features corresponding to the text data, voice data, and image data can be predicted separately to generate text search results, voice search results, image search results, etc., where the text search results represent similar business text data similar to the text data, the voice search results represent similar business voice data similar to the voice data, and the image search results represent similar business image data similar to the image data.

[0101] In one embodiment, based on the characteristics of the target unimodal business data, searching for similar business data corresponding to the unimodal business data in the data modality can be achieved through a machine learning method, that is, building a data search model.

[0102] S600, screening out target similar business data with the highest similarity from similar business data searched under all data modalities.

[0103] Specifically, the final target similar business data is screened from the similar business data searched under all data modalities. The target similar business data is the data with the highest similarity to the corresponding single-modal business data among the similar business data under each data modality.

[0104] That is to say, based on the similarity between the unimodal business data and the similar business data corresponding to the detected unimodal business data, the similar business data are sorted according to the degree of similarity. For example, all similar business data can be sorted according to the degree of similarity from high to low, or all similar business data can be sorted according to the degree of similarity from low to high.

[0105] Finally, based on the sorted similar business data, the target similar business data is screened out. If the sorting method is to sort from high to low by similarity, the similar business data with the highest ranking in the sorting result is determined as the target similar business data; if the sorting method is to sort from low to high by similarity, the similar business data with the lowest ranking in the sorting result is determined as the target similar business data. This design enables the most relevant target similar business data to be selected as the final output according to different task requirements, which improves the flexibility and practicality of data search.

[0106] S700: Output the target similar business data as the data search result corresponding to the data search request.

[0107] Specifically, ultimately, the target similar business data is output as the data search result corresponding to the data search request. Further, the output data search result can be displayed to the user so that the user can clearly obtain the data search result corresponding to the data search request.

[0108] In the above data search method, in response to a data search request, the multimodal business data corresponding to the data search request is obtained, and the unimodal business data features of each unimodal business data are extracted. According to the attention weight corresponding to the data modality of each unimodal business data feature and multiple unimodal business data features, the multimodal attention weight feature of the multimodal business data is generated. At this time, the multimodal attention weight feature is a feature that is associated with multiple unimodal business data features. Then, the multimodal attention weight feature and the multiple unimodal business data features are fused respectively to generate a target unimodal business data feature corresponding to each unimodal business data feature. The generated target unimodal business data feature is the data obtained after the effective fusion of the data features between all data modalities is considered. Therefore, for each data modality, according to the target unimodal business data feature, the process of searching for similar business data corresponding to the unimodal business data under the data modality is also more accurate. Furthermore, the target similar business data with the highest similarity can be screened out from the similar business data of all data modalities, which further improves the accuracy of the data search and makes the final output data search result more accurate.

[0109] In an exemplary embodiment, Figure 3 As shown, S300 includes:

[0110] S320: For each data modality, a unimodal attention weight feature is generated based on the unimodal business data feature and the attention weight under the data modality.

[0111] S340, merging the unimodal attention weight features under all data modalities into multimodal attention weight features.

[0112] Specifically, for each data modality, based on the unimodal business data features and the corresponding attention weights under the data modality, a unimodal attention weight feature corresponding to each data modality is generated. Furthermore, based on the unimodal attention weight features of the unimodal business data features under all data modalities, a multimodal attention weight feature is generated. Furthermore, in the case where there are multiple input multimodal business data, the generated multimodal attention weight feature can be a set of multimodal attention weight features.

[0113] Furthermore, based on the unimodal business data features and the attention weights under the data modality, a unimodal attention weight feature is generated, including: multiplying the attention weights under the data modality by the unimodal business data features under the data modality element by element to generate a unimodal attention weight feature.

[0114] The unimodal attention weight features under all data modalities are fused into multimodal attention weight features, including: generating multimodal attention weight features according to the sum of the unimodal attention weight features under all data modalities.

[0115] In an exemplary embodiment, let the multimodal attention weight feature be F p , G i d is the attention weight under the i-th data modality, W s is a learnable weight matrix, which is automatically learned during the iterative optimization process of the model. i s is the i-th unimodal business data feature. At this time, the generation expression of the multimodal attention weight feature is:

[0116]

[0117] In the above embodiment, a unimodal attention weight feature is generated by element-by-element multiplication of the unimodal business data feature and the attention weight under the corresponding data modality, and then a multimodal attention weight feature is accurately generated based on the sum of the unimodal attention weight features under all data modalities. That is, the correlation weight features between the unimodal attention weight features under different data modalities are accurately obtained, thereby achieving effective fusion of multiple unimodal business data. Furthermore, this method can fully consider the inherent correlation and complementarity between business data of different modalities, thereby improving the fusion effect of business data.

[0118] In an exemplary embodiment, extracting the unimodal service data features of each unimodal service data includes:

[0119] For each data modality, the unimodal service data is encoded to generate unimodal service encoded data, and features are extracted from the unimodal service encoded data to obtain features of the unimodal service data.

[0120] Specifically, for each data mode, the single-modal service data is encoded to generate single-modal service coded data, and feature extraction is performed on the encoded single-modal service coded data to obtain single-modal service data features. Further, according to this step, single-modal service data features of single-modal service data under all data modes are obtained.

[0121] In an exemplary embodiment, before searching for similar business data corresponding to the unimodal business data under the data modality based on the target unimodal business data characteristics, it is also necessary to decode the target unimodal business data characteristics to obtain the decoded target unimodal business data characteristics, and use the decoded target unimodal business data characteristics as the updated target unimodal business data characteristics.

[0122] In the above embodiment, by encoding the unimodal business data and then performing feature extraction, important features in the data can be efficiently learned to perform accurate feature extraction, thereby achieving accurate data search operations.

[0123] In an exemplary embodiment, feature extraction is performed on unimodal service coded data to obtain unimodal service data features, including:

[0124] Detect the query vector, key vector and value vector corresponding to the unimodal service encoded data, generate a correlation vector between the query vector and the key vector, and process the correlation vector through an activation function to obtain a target correlation vector. Based on the target correlation vector and the value vector, generate unimodal service data features.

[0125] Specifically, through the self-attention mechanism, all unimodal service encoding data are separately feature extracted, so that the machine can notice the correlation between different parts of various types of input unimodal service encoding data itself, and obtain multiple unimodal service data features. Among them, the self-attention mechanism, also known as the internal attention mechanism, as the name suggests, is an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence.

[0126] The expression of the self-attention mechanism is as follows:

[0127] Where, let Q be the query vector, K T is the key vector, V is the value vector, d k is the scaling factor, which can prevent the result from being too large or too small.

[0128] First, for each unimodal service coded data, the unimodal service coded data is processed through the self-attention mechanism. At this time, Q, K and V in the self-attention mechanism are all unimodal service coded data, then K T The inverted matrix of the coded data for unimodal services.

[0129] Furthermore, based on the inner product of each row vector of Q and K, the correlation vector between the query vector and the key vector is generated , and in order to prevent the inner product from being too large, the inner product can also be divided by the scaling factor to make the inner product within a preset range.

[0130] Secondly, through the activation function, the correlation vector Processing is performed to normalize the numerical vector into a probability distribution vector to obtain a target relevance vector. The target relevance vector is a matrix. Generally, the activation function used is the softmax function.

[0131] Finally, the final output is obtained by multiplying the target relevance vector and the value vector, and the final output is determined as the unimodal business data feature.

[0132] In this embodiment, multi-head feature extraction is performed on business data of different modalities through the self-attention mechanism, that is, separate feature extraction is performed on the business data of each modality, so that each head can focus on specific aspects of data of different modalities, thereby achieving comprehensive and more fine-grained processing of multimodal business data, which helps to deeply explore the potential information of business data of different modalities and improve the accuracy of data search.

[0133] In an exemplary embodiment, searching for similar service data corresponding to the unimodal service data in a data modality according to the target unimodal service data features includes:

[0134] According to the target unimodal business data characteristics, a data search model is used to search for similar business data corresponding to the unimodal business data under the modality; wherein, the iterative optimization process of the data search model includes: a data acquisition step: obtaining historical multimodal business data, wherein the historical multimodal business data includes historical unimodal business data under multiple data modalities; extracting actual similar business data corresponding to each data modality of the historical multimodal business data, as well as historical unimodal business data characteristics of each historical unimodal business data, and obtaining the attention weight corresponding to the data modality of each historical unimodal business data feature; fusing multiple historical unimodal business data features with multiple attention weights into historical multimodal attention weight features; fusing the historical multimodal attention weight features with each historical unimodal business data feature respectively, and obtaining the historical target unimodal business data features corresponding to each historical unimodal business data feature; for each data modality, based on the historical target unimodal business data characteristics, using the data search model to search for search result business data similar to the historical unimodal business data under the data modality; based on the difference between the actual similar business data and the search result business data under each data modality, iteratively optimizing the data search model.

[0135] Among them, the historical multimodal business data includes historical unimodal business data under multiple data modalities.

[0136] Specifically, according to the characteristics of the target unimodal business data, searching for similar business data in the data modality that is similar to the unimodal business data of the data modality is achieved through a machine learning method, that is, according to the characteristics of the target unimodal business data, using the data search model to search for similar business data in the modality that is similar to the unimodal business data. Among them, the data search model is a model generated after iterative optimization, and the iterative optimization process of the data search model includes:

[0137] First, execute the data acquisition step, which includes: acquiring historical multimodal business data, wherein the collected historical multimodal business data can be integrated into a unified data set to form a historical multimodal data set, and the historical multimodal business data in the historical multimodal business data set is divided according to a preset training ratio to obtain training data and test data. For example, the historical multimodal data set is randomly divided into training data and test data in a ratio of 7:3. The training data is used for the iterative optimization of the subsequent data search model, and the test data is used to evaluate the performance of the data search model after the iterative optimization.

[0138] Secondly, the actual similar business data corresponding to multiple historical unimodal business data in the historical multimodal business data are extracted, and the historical unimodal business data features of each historical unimodal business data are extracted through the self-attention mechanism.

[0139] Obtain the attention weight corresponding to the data modality of each historical unimodal business data feature, and for each data modality, generate a historical unimodal attention weight feature based on the historical unimodal business data feature and the attention weight under the data modality; merge the historical unimodal attention weight features under all data modalities into a historical multimodal attention weight feature.

[0140] The historical multimodal attention weight features are point-multiplied with each historical unimodal business data feature to obtain the historical target unimodal business data features corresponding to each historical unimodal business data feature.

[0141] Then, for each data modality, based on the characteristics of the historical unimodal business data, search result business data similar to the historical unimodal business data is searched from the database.

[0142] Finally, based on the difference between the actual similar business data and the search result business data under each data modality, the data search model is iteratively optimized to obtain an iteratively optimized data search model.

[0143] The optimizer corresponding to the data search model can be pre-selected, for example, Adme (Adaptive moment estimation) and SGD (Stochastic Gradient Descent). In addition, according to the characteristics of the data search model and the scale of training data, the appropriate optimizer, model learning rate, momentum and other hyperparameters can be selected to update the data search model.

[0144] In an exemplary embodiment, after the iterative optimization process of the data search model is completed, the data search model completed by the iterative optimization can be evaluated through test data, and appropriate evaluation indicators such as accuracy, recall rate, F1 value, etc. are selected according to the data search requirements, and the data search model completed by the iterative optimization is tuned according to the evaluation results, such as adjusting the model structure, optimizing hyperparameters, and adding data enhancement, etc. Among them, the F1 value, also known as the F1 score, is a statistical indicator used to measure the accuracy of the binary classification model.

[0145] In an exemplary embodiment, after the iterative optimization of the data search model is completed, the data search model needs to be stored. Storing the model is a key step to ensure that subsequent work can continue. By storing the iteratively optimized model for subsequent use or deployment, the model can be saved in a specific file format during the storage process, such as a PyTorch pth file or a TensorFlow ckpt file.

[0146] Furthermore, after the model is stored, it can be deployed. Model deployment is the process of applying the iteratively optimized model to the actual scenario. Before deployment, it is necessary to select the appropriate hardware and software environment according to actual needs and perform corresponding configuration.

[0147] Environment preparation: According to the requirements of the model, prepare the corresponding hardware environment, such as GPU (Graphics Processing Unit), CPU (Central Processing Unit), etc., as well as the software environment, such as operating system, deep learning framework, etc., to ensure that these environments can support the operation of the model.

[0148] Model loading: When deploying, the previously saved model structure and weights need to be loaded into memory, which is usually achieved through the loading function provided by the deep learning framework. After loading, the model can be used for prediction or analysis.

[0149] Service construction: In order to facilitate users to access and use the model, the model can be deployed to an online environment and corresponding services can be built. These services can receive users' data search requests and return the data search results of the model. The service construction method can be selected according to actual needs, such as using Web frameworks such as Flask and Django to build a RESTful API interface. In other words, data search can be achieved by building an interface.

[0150] Furthermore, after the model is deployed, the deployed data search model can be used to process and analyze multimodal business data.

[0151] In the process of model application, the following scenarios can be applied:

[0152] The first type is cross-modal data retrieval: users can retrieve relevant multimodal data by inputting data in different modalities such as text, images, or videos. The data search model will output multimodal data similar or related to the input data from the data source based on the content and characteristics of the input data, and return it to the user.

[0153] The second type is multimodal data recommendation: the data search model can be applied to the multimodal data recommendation system. By analyzing the user's historical behavior and preferences, multimodal content related to the user's interests, such as movies, music, news, etc., can be searched from the data source.

[0154] Furthermore, the present application is not only applicable to the above-mentioned model application scenarios, but also to various scenarios such as sentiment analysis, intelligent question and answer, and image recognition. By adjusting the parameters and structure of the model, it is possible to process and analyze different types of multimodal data to meet the needs of different scenarios.

[0155] In an exemplary embodiment, based on the difference between the actual similar business data and the search result business data under each data modality, iteratively optimizing the data search model includes:

[0156] Based on the difference between the actual similar business data and the search result business data under each data modality, the model loss under each data modality is generated; according to multiple attention weights, the model loss under each data modality is fused to obtain the total model loss; if the total model loss is less than the preset model loss threshold, it is determined that the iterative optimization of the data search model is completed; if the total model loss is not less than the preset model loss threshold, the data search model is updated according to the total model loss, and the data acquisition step is returned until the calculated total model loss is less than the preset model loss threshold.

[0157] Specifically, according to the multi-head output characteristics of the data search model, a suitable loss function is designed to measure the difference between the model output label and the true label. Considering that business data of different modalities may have different importance, the loss function can include weighted terms for different head outputs. In addition, commonly used loss functions such as cross entropy loss and mean square error loss can also be considered.

[0158] Furthermore, the optimizer corresponding to the data search model can be used to update the parameters of the model according to the gradient of the loss function to minimize the loss function value.

[0159] Specifically, based on the difference between the actual similar business data and the search result business data under each data modality, the model loss of the data search model under each data modality is generated. Furthermore, the model loss under each data modality is fused to obtain the total model loss. The method of fusing the model loss under each data modality includes: according to multiple attention weights, the model loss under each data modality is fused to obtain the total model loss.

[0160] When the total model loss is less than the preset model loss threshold, the loss of the data search model converges and the iterative optimization of the data search model is completed; when the total model loss is not less than the preset model loss threshold, the loss of the data search model has not converged and the data search model needs to be iteratively optimized.

[0161] In other words, it is also necessary to update the model parameters of the data search model according to the total model loss to update the data search model, and return to the step of obtaining new historical multimodal business data, and repeat the above steps to iteratively optimize the data search model until the calculated total model loss is less than the preset model loss threshold during a certain iteration of the data search model.

[0162] In an exemplary embodiment, if the model losses under each data modality are fused according to multiple attention weights to obtain the total model loss, the data search model is updated according to the total model loss, and the data search effect of the updated data search model is not good, then other target weights can be used to fuse the model losses under each data modality to obtain the total model loss, and the target weight is a weight related to the attention weight. For example, a mapping function between a target weight and the attention weight can be set in advance. Through the mapping function, the attention weight is mapped to the target weight to obtain the target weight corresponding to the attention weight under each data modality, and based on the target weight under each data modality, the model loss under each data modality is fused to obtain the total model loss to update the data search model.

[0163] In an exemplary embodiment, the method for determining whether the iterative optimization of the data search model is completed may also be to determine whether the number of iterative optimizations reaches a preset number. If the number of iterative optimizations reaches the preset number, it is determined that the iterative optimization of the data search model is completed; if the number of iterative optimizations does not reach the preset number, it is necessary to continue iteratively optimizing the data search model.

[0164] In an exemplary embodiment, in the iterative optimization process of the data search model, a mini-batch gradient descent method is used for iterative optimization. Each batch contains a certain number of samples, and the model calculates the gradient of the loss function based on these samples and updates the parameters. After each iteration, the performance of the model on the validation set can be evaluated to adjust the hyperparameters or take other measures to prevent overfitting.

[0165] In the above embodiment, by designing a loss function, the model parameters of the data search model can be accurately updated, so that the prediction effect of the data search model completed by iterative optimization is more accurate.

[0166] In an exemplary embodiment, the data modalities of the multimodal business data include but are not limited to voice data, text data, image data, and other types of data. The following will use a detailed embodiment to describe the data search method in detail from the iterative optimization stage of the data search model and the data search stage.

[0167] The first stage is the iterative optimization stage of the data search model, such as Figure 4 and Figure 5 As shown, Figure 4 is the overall interaction diagram of the iterative optimization phase. Figure 5 This is the module interaction diagram of the model in the iterative optimization phase:

[0168] Step 1: Construct the dataset.

[0169] 1.1 Data collection: Collect business data of different data modes through different channels and methods, including text, voice, images, etc. These data can come from different fields, such as social media, news, forums, blogs, etc.

[0170] 1.2 Data preprocessing: Preprocess the collected business data, including data cleaning, format conversion, labeling, etc.

[0171] 1.3 Data partitioning: The preprocessed business data of different modes are integrated into a unified data set to form a multimodal business data set, and the business data set is randomly divided into a training set and a test set in a ratio of 7:3.

[0172] Step 2: Model construction.

[0173] This method constructs a multi-head input and multi-head output model, which aims to process data of different modalities simultaneously and realize effective information fusion. The model adopts a multi-head attention mechanism, which performs independent feature extraction and attention weight calculation by inputting data of different modalities into multiple heads respectively. In this way, each head can focus on specific aspects of business data of different modalities, thereby realizing comprehensive processing of multimodal business data.

[0174] 2.1: Training data processing.

[0175] The historical multimodal business data within the historical time period and the actual similar business data corresponding to multiple historical unimodal business data in the historical multimodal business data are used as training data. The training data are divided into four types, namely text, voice, image and other types of data, and the training data of different modalities are encoded and processed respectively as historical unimodal business encoded data.

[0176] Step 2.2: Self-attention mechanism feature extraction module.

[0177] In order to fuse data of different modalities, this method designs a multimodal attention fusion module based on the attention fusion mechanism. This module first uses the self-attention mechanism module to process the three-head input data of text, speech, and image, that is, to extract features from the historical unimodal business encoding data and obtain the processed historical unimodal business data features T S 、V S ,I S .

[0178] Step 2.3: Attention feature fusion module.

[0179] T S 、V S ,I SThe historical data feature set composed of other types of data is sent to the feature fusion module based on the attention mechanism, T S 、V S ,I S It performs feature fusion with other types of data by calculating the attention weight of each modality at each feature position, and then multiplies this weight element by element with the corresponding historical unimodal business data feature. Finally, it sums the weighted features of all data modalities to obtain the fused historical multimodal attention weight feature Fp.

[0180] Furthermore, Fp and T S 、V S ,I S Perform point multiplication fusion to obtain the historical target single-modal business data features T of different modalities a 、V a ,I a That is, the historical target unimodal business data feature T is generated by the following expression a 、V a ,I a .

[0181]

[0182]

[0183]

[0184] Step 2.4: Long prediction module.

[0185] The historical target unimodal business data features of different modalities after being processed by the attention fusion mechanism T a 、V a ,I a It is sent to the multi-head prediction module for decoding prediction.

[0186] This module performs independent data searches from the data source for each modality. That is to say, based on the characteristics of the historical target unimodal business data, it searches for search result business data that is similar to the historical unimodal business data in the same modality, such as Tp (text prediction results), Vp (voice prediction results), Ip (image prediction results), and other types of prediction results.

[0187] Furthermore, the data search model may be iteratively optimized based on the difference between the actual similar business data and the search result business data under each data modality.

[0188] Step 3: Iterative optimization of data search model.

[0189] Step 3.1: Data processing.

[0190] The training data obtained in step 1 is processed by first adding various random noises to various types of data to perform data enhancement, then normalizing the data and updating the training data.

[0191] Step 3.1: Loss function design.

[0192] According to the multi-head output characteristics of the model, a suitable loss function is designed to measure the difference between the model prediction results and the true labels. Considering that different modal data may have different importance, the loss function can include weighted terms for different head outputs. In addition, commonly used loss functions such as cross entropy loss and mean square error loss can also be considered.

[0193] Step 3.2: Optimizer selection.

[0194] Select an optimizer suitable for the data search model, such as Adam, SGD, etc. The optimizer is used to update the model parameters according to the gradient of the loss function to minimize the loss function value. According to the characteristics of the data search model and the scale of the training data, select a suitable optimizer and its hyperparameters such as learning rate and momentum.

[0195] Step 3.3: Iterative optimization process of data search model.

[0196] In the iterative optimization process of the data search model, mini-batch gradient descent is used for iterative optimization. Each batch contains a certain number of samples, and the model calculates the gradient of the loss function based on these samples and updates the parameters. After each iteration, the performance of the model on the validation set can be evaluated in order to adjust the hyperparameters or take other measures to prevent overfitting.

[0197] Step 3.4: Model evaluation and tuning of data search model.

[0198] After the iterative optimization of the data search model is completed, the data search model is evaluated using the test data. Select appropriate evaluation indicators according to the task requirements, such as accuracy, recall, and F1 value. Based on the evaluation results, the model can be tuned, such as adjusting the model structure, optimizing hyperparameters, and adding data enhancement.

[0199] Step 4: Save, deploy and apply the data search model.

[0200] 4.1: Preservation of data search model.

[0201] After the model iteration optimization is completed, saving the model is a key step to ensure that subsequent work can continue. Save the iteratively optimized model for subsequent use or deployment. The model can be saved in a specific file format, such as PyTorch's pth file or TensorFlow's ckpt file.

[0202] 4.2: Deployment of data search model.

[0203] Model deployment is the process of applying the iteratively optimized model to the actual scenario. Before deployment, you need to select the appropriate hardware and software environment according to actual needs and perform corresponding configuration.

[0204] Environment preparation: Prepare the corresponding hardware environment (such as GPU, CPU, etc.) and software environment (such as operating system, deep learning framework, etc.) according to the requirements of the model. Ensure that these environments can support the operation of the model.

[0205] Model loading: When deploying, the previously saved model structure and weights need to be loaded into memory. This is usually achieved through the loading function provided by the deep learning framework. After loading, the model can be used for prediction or analysis.

[0206] Service construction: In order to facilitate users to access and use the model, the model can be deployed to an online environment and corresponding services can be built. These services can receive user query requests and return the prediction results or analysis results of the model. The service construction method can be selected according to actual needs, such as using Web frameworks such as Flask and Django to build a RESTful API interface.

[0207] The second stage, data search stage:

[0208] After the model is deployed, it can be used to process and analyze multimodal data.

[0209] That is to say, the user sends a data search request at the terminal, and the terminal responds to the data search request, obtains the multimodal business data within the current time period, and detects the unimodal business data features of multiple unimodal business data in the multimodal business data, obtains the attention weight corresponding to each unimodal business data feature, and fuses the multiple unimodal business data features with the multiple attention weights into a multimodal attention weight feature, and fuses the multimodal attention weight feature with each unimodal business data feature respectively to obtain the target unimodal business data feature corresponding to each unimodal business data feature; for each data modality, according to the target unimodal business data feature, use the data search model to search for similar business data corresponding to the unimodal business data under the data modality, and detect the similarity between the similar business data and the unimodal business data. The specific process is the same as the iterative optimization stage of the data search model, which will not be repeated here; further, according to specific task requirements, the target similar business data with the highest similarity can be screened out from the similar business data of all data modalities. The target similar business data is multimodal data that is similar or related to the multimodal business data, such as movies, music, news, etc. The target similar business data is output to the user as a data search result corresponding to the data search request.

[0210] In this application, by introducing a multi-head attention mechanism and a feature fusion module based on the attention mechanism, effective fusion of multimodal data is achieved, which has beneficial effects such as fine-grained feature extraction, efficient feature fusion, flexible multi-head prediction and strong adaptability.

[0211] Furthermore, the present application has the following beneficial effects, including but not limited to:

[0212] A. Improve data processing capabilities: The high-performance computing and storage capabilities provided by the cloud computing platform, combined with multimodal attention fusion technology, can significantly improve the processing capabilities of multimodal data and meet the needs of complex business scenarios.

[0213] B Enhance user experience: Through multimodal attention fusion technology, companies can provide users with more natural and diverse interaction methods, improving user experience and satisfaction.

[0214] C Promote technological innovation and application expansion: The combination of multimodal attention fusion technology and cloud computing will bring new solutions and ideas to multiple fields, promoting technological innovation and application expansion. At the same time, as the technology continues to improve and mature, its application scenarios and value will continue to expand and deepen.

[0215] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0216] Based on the same inventive concept, the embodiment of the present application also provides a data search device for implementing the above-mentioned data search method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations of one or more data search device embodiments provided below can refer to the limitations of the data search method above, and will not be repeated here.

[0217] In an exemplary embodiment, Figure 6 As shown, a data search device is provided, comprising: a data acquisition module 100, an extraction module 200, a weight feature generation module 300, a feature fusion module 400, a search module 500, a screening module 600 and a search result output module 700, wherein:

[0218] The data acquisition module 100 is used to respond to the data search request and acquire the multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data;

[0219] An extraction module 200 is used to extract the unimodal business data features of each unimodal business data, and obtain the attention weight corresponding to the data modality of each unimodal business data feature;

[0220] A weight feature generation module 300, for fusing a plurality of unimodal business data features with a plurality of attention weights into a multimodal attention weight feature;

[0221] A feature fusion module 400 is used to fuse the multimodal attention weight feature with each unimodal service data feature to obtain a target unimodal service data feature corresponding to each unimodal service data feature;

[0222] A search module 500 is used to search for similar business data corresponding to the unimodal business data under the data modality for each data modality according to the characteristics of the target unimodal business data, and detect the similarity between the similar business data and the unimodal business data;

[0223] A screening module 600 is used to screen out target similar business data with the highest similarity from similar business data searched under all data modes;

[0224] The search result output module 700 is used to output the target similar business data as the data search result corresponding to the data search request.

[0225] In one embodiment, the weight feature generation module 300 is also used to generate a unimodal attention weight feature for each data modality based on the unimodal business data feature and the attention weight under the data modality; and to fuse the unimodal attention weight features under all data modalities into a multimodal attention weight feature.

[0226] In one of the embodiments, the extraction module 200 is further used to encode the unimodal service data for each data modality to generate unimodal service encoded data; and perform feature extraction on the unimodal service encoded data to obtain features of the unimodal service data.

[0227] In one embodiment, the extraction module 200 is also used to detect the query vector, key vector and value vector corresponding to the unimodal service encoded data; generate a correlation vector between the query vector and the key vector, and process the correlation vector through an activation function to obtain a target correlation vector; based on the target correlation vector and the value vector, generate unimodal service data features.

[0228] In one embodiment, the search module 500 is further used to search for similar service data corresponding to the unimodal service data in a modality using a data search model according to the target unimodal service data characteristics;

[0229] Among them, the iterative optimization process of the data search model includes: data acquisition step: acquiring historical multimodal business data, wherein the historical multimodal business data includes historical unimodal business data under multiple data modalities; extracting the actual similar business data corresponding to each data modality of the historical multimodal business data, and the historical unimodal business data features of each historical unimodal business data, and obtaining the attention weight corresponding to the data modality of each historical unimodal business data feature; fusing multiple historical unimodal business data features with multiple attention weights into historical multimodal attention weight features; fusing the historical multimodal attention weight features with each historical unimodal business data feature respectively to obtain the historical target unimodal business data features corresponding to each historical unimodal business data feature; for each data modality, based on the historical target unimodal business data features, using the data search model to search for search result business data similar to the historical unimodal business data under the data modality; based on the difference between the actual similar business data and the search result business data under each data modality, iteratively optimizing the data search model.

[0230] In one embodiment, the search module 500 is also used to generate a model loss under each data modality based on the difference between the actual similar business data and the search result business data under each data modality; according to multiple attention weights, the model loss under each data modality is fused to obtain a total model loss; if the total model loss is less than a preset model loss threshold, it is determined that the iterative optimization of the data search model is completed; if the total model loss is not less than the preset model loss threshold, the data search model is updated according to the total model loss, and the data acquisition step is returned until the calculated total model loss is less than the preset model loss threshold.

[0231] Each module in the above data search device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0232] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as multimodal business data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data search method is implemented.

[0233] Those skilled in the art will understand that Figure 7 The structure shown in the figure is a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0234] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0235] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0236] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0238] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0239] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0240] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data search method, characterized in that: The method comprises: In response to a data search request, obtaining multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data; Extracting unimodal business data features of each unimodal business data, and obtaining an attention weight corresponding to the data modality of each of the unimodal business data features; Fusion of the plurality of unimodal business data features with the plurality of attention weights into a multimodal attention weight feature; The multimodal attention weight feature is respectively integrated with each of the unimodal business data features to obtain a target unimodal business data feature corresponding to each of the unimodal business data features; For each of the data modalities, according to the target unimodal service data characteristics, searching for similar service data corresponding to the unimodal service data under the data modality, and detecting the similarity between the similar service data and the unimodal service data; Filtering out target similar business data with the highest similarity from similar business data searched under all the data modalities; The target similar business data is output as the data search result corresponding to the data search request.

2. The method according to claim 1, characterized in that The fusing of the plurality of the unimodal business data features and the plurality of the attention weights into a multimodal attention weight feature comprises: For each of the data modalities, generating a unimodal attention weight feature based on the unimodal service data feature and the attention weight under the data modality; The unimodal attention weight features under all the data modalities are fused into multimodal attention weight features.

3. The method according to claim 1, characterized in that The extracting the single-mode service data features of each single-mode service data includes: For each of the data modes, encoding the single-mode service data to generate single-mode service encoded data; Feature extraction is performed on the unimodal service encoded data to obtain unimodal service data features.

4. The method according to claim 3, characterized in that The extracting features of the unimodal service coded data to obtain unimodal service data features includes: Detecting a query vector, a key vector, and a value vector corresponding to the unimodal service coded data; Generate a relevance vector between the query vector and the key vector, and process the relevance vector through an activation function to obtain a target relevance vector; Based on the target relevance vector and the value vector, a unimodal business data feature is generated.

5. The method according to claim 1, characterized in that The step of searching, according to the target unimodal service data feature, similar service data corresponding to the unimodal service data in the data modality comprises: According to the target unimodal business data characteristics, using a data search model to search for similar business data corresponding to the unimodal business data in the modality; The iterative optimization process of the data search model includes: Data acquisition step: acquiring historical multimodal service data, wherein the historical multimodal service data includes historical single-modal service data under multiple data modes; Extracting actual similar business data corresponding to each data modality of the historical multimodal business data and historical unimodal business data features of each of the historical unimodal business data, and obtaining an attention weight corresponding to the data modality of each of the historical unimodal business data features; Merging the plurality of the historical unimodal business data features with the plurality of the attention weights into a historical multimodal attention weight feature; The historical multimodal attention weight feature is respectively integrated with each of the historical unimodal business data features to obtain the historical target unimodal business data feature corresponding to each of the historical unimodal business data features; For each of the data modalities, based on the characteristics of the historical target unimodal business data, searching for search result business data similar to the historical unimodal business data in the data modality using the data search model; Based on the difference between the actual similar business data and the search result business data under each of the data modalities, the data search model is iteratively optimized.

6. The method according to claim 5, characterized in that The iterative optimization of the data search model based on the difference between the actual similar business data and the search result business data under each of the data modalities includes: Generate a model loss for each data modality based on the difference between the actual similar business data and the search result business data for each data modality; According to multiple attention weights, the model loss under each data modality is fused to obtain the total model loss; If the total model loss is less than a preset model loss threshold, it is determined that the iterative optimization of the data search model is completed; If the total model loss is not less than the preset model loss threshold, the data search model is updated according to the total model loss, and the data acquisition step is returned until the calculated total model loss is less than the preset model loss threshold.

7. A data search device, characterized in that: The device comprises: A data acquisition module, configured to, in response to a data search request, acquire multimodal service data corresponding to the data search request, wherein the multimodal service data is composed of a plurality of single-modal service data; An extraction module, used to extract the unimodal business data features of each unimodal business data, and obtain the attention weight corresponding to the data modality of each of the unimodal business data features; A weight feature generation module, used for fusing the plurality of unimodal business data features with the plurality of attention weights into a multimodal attention weight feature; A feature fusion module, used to fuse the multimodal attention weight feature with each of the unimodal business data features to obtain a target unimodal business data feature corresponding to each of the unimodal business data features; A search module, configured to search, for each of the data modalities, similar service data corresponding to the unimodal service data under the data modality according to the target unimodal service data characteristics, and detect the similarity between the similar service data and the unimodal service data; A screening module, used to screen out target similar business data with the highest similarity from similar business data searched under all the data modalities; The search result output module is used to output the target similar business data as the data search result corresponding to the data search request.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images