Method, device and computer device for matching data information and event information

Abstract

The application relates to a data information and event information matching method and device, a computer device, a storage medium and a computer program product. The method comprises the following steps: obtaining to-be-matched data information and to-be-matched event information, the to-be-matched data information comprising one or more modalities of data; obtaining to-be-matched event information features corresponding to the to-be-matched event information, and obtaining to-be-matched data information features corresponding to the to-be-matched data information; based on the to-be-matched data information features and the to-be-matched event information features, obtaining a matching result through an information matching model, the matching result being used to describe the matching degree between the to-be-matched data information and the to-be-matched event information. The method can obtain multi-dimensional image, audio and text details in the data information, thereby improving the accuracy of the matching result, that is, ensuring the accuracy of matching between the event information and the data information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, and computer device for matching data information with event information. Background Technology

[0002] With the rapid development of internet applications, massive amounts of data are presented to people in various forms such as text, video, and images. Therefore, when people want to understand relevant data about an event, they need to perform related searches and then view the data related to the event through the search results.

[0003] Currently, considering various data formats such as text, video, and images, cross-modal retrieval methods combining video and text can be employed. Specifically, this involves learning a joint embedding space to obtain cross-modal similarity between video and text, thereby generating search results based on this similarity. However, event information matching requires high semantic relevance, while joint embedding struggles to capture the details of multi-dimensional data, thus reducing the accuracy of matching event information with data information. Therefore, ensuring the accuracy of matching event information with data information is a pressing issue that needs to be addressed. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, and computer device for matching data information and event information that can ensure the accuracy of matching event information and data information, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training a pre-trained model. The method includes:

[0006] Acquire the data information to be matched and the event information to be matched. The data information to be matched includes one or more modal data.

[0007] Obtain the features of the event information to be matched corresponding to the event information to be matched, and obtain the features of the data information to be matched corresponding to the data information to be matched;

[0008] Based on the features of the data information to be matched and the features of the event information to be matched, the matching result is obtained through the information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning the initial information matching model obtained by pre-training based on the training samples of each data information and the event information training samples that match each data information training sample. The data information training samples include one or more modal data, and the event information training samples are obtained by manual annotation based on the data information training samples.

[0009] The methods for obtaining the initial information matching model include:

[0010] Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data.

[0011] Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair. The multimodal data features include multiple modal features.

[0012] Based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model.

[0013] In one embodiment, obtaining the features of the event information to be matched corresponding to the event information to be matched includes:

[0014] Based on the information of the event to be matched, the features of the event information to be matched are obtained through an information matching model;

[0015] Obtain the features of the data to be matched that correspond to the data to be matched, including:

[0016] The text features to be matched are obtained from the information matching model, and the information features to be matched are obtained from a pre-set cache database. The information features to be matched include at least one of the image features to be matched and the audio features to be matched.

[0017] In one embodiment, obtaining the features of the data to be matched corresponding to the data to be matched includes:

[0018] If the data to be matched is text data, the features of the data to be matched are obtained through the information matching model based on the data to be matched, and the features of the data to be matched are the text features to be matched.

[0019] In one embodiment, obtaining the features of the data to be matched corresponding to the data to be matched includes:

[0020] If the data to be matched is video data, the text features to be matched are obtained through the information matching model based on the data to be matched.

[0021] Based on the data to be matched, the image features and audio features to be matched are obtained from the cached database.

[0022] Among them, the features of the data information to be matched include the text features to be matched, the image features to be matched, and the audio features to be matched;

[0023] The cache database includes: image features obtained by extracting image features from each video data information in the video data information database using an image feature extraction model, and audio features obtained by extracting audio features from each video data information using an audio feature extraction model.

[0024] In one embodiment, obtaining the features of the data to be matched corresponding to the data to be matched includes:

[0025] If the data to be matched is image and text data, the text features to be matched are obtained through the information matching model based on the data to be matched.

[0026] Based on the data to be matched, the features of the image to be matched are obtained from the cached database;

[0027] Among them, the features of the data information to be matched include the features of the text to be matched and the features of the image to be matched;

[0028] The cache database includes image features obtained by extracting image features from each image and text data in the image and text data information database using an image feature extraction model.

[0029] In one embodiment, obtaining a multimodal sample set includes:

[0030] Obtain data content information, first data text information, and second data text information. The degree of matching between the data content information and the first data text information is greater than the matching degree threshold, and the degree of matching between the data content information and the second data text information is less than the matching degree threshold.

[0031] The data content information is masked to obtain the masked data content information;

[0032] The first data text information is masked to obtain the masked first data text information.

[0033] The multimodal sample set includes at least: a first multimodal sample pair including data content information and first data text information; a second multimodal sample pair including data content information and second data text information; a third multimodal sample pair including masked data content information and first data text information; and a fourth multimodal sample pair including data content information and masked first data text information.

[0034] In one embodiment, obtaining the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair, includes:

[0035] Obtain the first multimodal data features corresponding to the data content information in the first multimodal sample pair, and the first text data features corresponding to the first data text information;

[0036] Obtain the first multimodal data features corresponding to the data content information in the second multimodal sample pair, and the second text data features corresponding to the second data text information;

[0037] Obtain the second multimodal data features corresponding to the data content information after the third multimodal sample centering mask processing, and the first text data features corresponding to the first data text information;

[0038] Obtain the first multimodal data feature corresponding to the data content information in the fourth multimodal sample pair, and the third text data feature corresponding to the first data text information after masking.

[0039] In one embodiment, the initial pre-trained model is trained based on the multimodal data features and text data features in each multimodal sample pair, including:

[0040] During the training process of the initial pre-trained model:

[0041] Based on the first multimodal data features and the first text data features, the matching result of the first multimodal sample pair is obtained. The matching result is used to describe the degree of matching between the data content information and the first data text information.

[0042] Based on the matching results of the first multimodal sample pair, update the model parameters of the initial pre-trained model.

[0043] In one embodiment, the multimodal sample pair specifically includes video content information and video text information, wherein the video content information includes text data, image data, and audio data;

[0044] Multimodal data features specifically include text features, image features, and audio features.

[0045] In one embodiment, the initial pre-trained model is a pre-trained text modality model. The text modality model is obtained after training based on text content information and text information, and is used to calculate the probability that text content information and text information belong to the same text data information.

[0046] In one embodiment, the information matching model is obtained by means of:

[0047] Acquire training samples for each data information, as well as training samples for event information that match each data information training sample;

[0048] Based on the training samples of each data information, the data information features corresponding to each data information training sample are obtained, and based on the training samples of each event information, the event information features corresponding to each event information training sample are obtained.

[0049] Based on the features of each data information and each event information, the predicted matching degree between each data information feature and each event information feature is obtained, and the initial information matching model is fine-tuned based on each actual matching degree and each predicted matching degree to obtain the information matching model.

[0050] In one embodiment, the data information training sample is a video data information training sample, which consists of text data, image data, and audio data. The image data consists of multiple image frames, and the audio data consists of multiple audio frames.

[0051] Based on the training samples of each data information, the data information features corresponding to each training sample are obtained, including:

[0052] Text segmentation is performed on the text data in the training samples of each video data information to obtain the text sequence corresponding to the text data in the training samples of each video data information. Based on each text sequence, the text features corresponding to the text data in the training samples of each video data information are generated. The text sequence includes multiple text tokens.

[0053] Image recognition processing is performed on the image data in each video data training sample to obtain the image features corresponding to the image data in each video data training sample;

[0054] Audio recognition processing is performed on the audio data in each video data training sample to obtain the audio features corresponding to the audio data in each video data training sample.

[0055] The data information features include text features, image features, and audio features.

[0056] Secondly, this application also provides a device for matching data information with event information. The device includes:

[0057] The acquisition module is used to acquire data information to be matched and event information to be matched. The data information to be matched includes one or more modal data.

[0058] The feature processing module is used to obtain the features of the event information to be matched corresponding to the event information to be matched, and to obtain the features of the data information to be matched corresponding to the data information to be matched.

[0059] The information matching module is used to obtain matching results based on the features of the data information to be matched and the features of the event information to be matched through the information matching model. The matching results are used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning the initial information matching model obtained by pre-training based on the training samples of each data information and the event information training samples that match each data information training sample. The data information training samples include one or more modal data, and the event information training samples are obtained by manual annotation based on the data information training samples.

[0060] The methods for obtaining the initial information matching model include:

[0061] Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data.

[0062] Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair. The multimodal data features include multiple modal features.

[0063] Based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model.

[0064] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0065] Acquire the data information to be matched and the event information to be matched. The data information to be matched includes one or more modal data.

[0066] Obtain the features of the event information to be matched corresponding to the event information to be matched, and obtain the features of the data information to be matched corresponding to the data information to be matched;

[0067] Based on the features of the data information to be matched and the features of the event information to be matched, the matching result is obtained through the information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning the initial information matching model obtained by pre-training based on the training samples of each data information and the event information training samples that match each data information training sample. The data information training samples include one or more modal data, and the event information training samples are obtained by manual annotation based on the data information training samples.

[0068] The methods for obtaining the initial information matching model include:

[0069] Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data.

[0070] Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair. The multimodal data features include multiple modal features.

[0071] Based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model.

[0072] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0073] Acquire the data information to be matched and the event information to be matched. The data information to be matched includes one or more modal data.

[0074] Obtain the features of the event information to be matched corresponding to the event information to be matched, and obtain the features of the data information to be matched corresponding to the data information to be matched;

[0075] Based on the features of the data information to be matched and the features of the event information to be matched, the matching result is obtained through the information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning the initial information matching model obtained by pre-training based on the training samples of each data information and the event information training samples that match each data information training sample. The data information training samples include one or more modal data, and the event information training samples are obtained by manual annotation based on the data information training samples.

[0076] The methods for obtaining the initial information matching model include:

[0077] Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data.

[0078] Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair. The multimodal data features include multiple modal features.

[0079] Based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model.

[0080] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0081] Acquire the data information to be matched and the event information to be matched. The data information to be matched includes one or more modal data.

[0082] Obtain the features of the event information to be matched corresponding to the event information to be matched, and obtain the features of the data information to be matched corresponding to the data information to be matched;

[0083] Based on the features of the data information to be matched and the features of the event information to be matched, the matching result is obtained through the information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning the initial information matching model obtained by pre-training based on the training samples of each data information and the event information training samples that match each data information training sample. The data information training samples include one or more modal data, and the event information training samples are obtained by manual annotation based on the data information training samples.

[0084] The methods for obtaining the initial information matching model include:

[0085] Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data.

[0086] Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair. The multimodal data features include multiple modal features.

[0087] Based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model.

[0088] The aforementioned data and event information matching method, apparatus, computer equipment, storage medium, and computer program product acquire data information to be matched and event information to be matched. The data information to be matched includes one or more modal data. Then, the features of the event information to be matched corresponding to the event information to be matched are acquired, and the features of the data information to be matched corresponding to the data information to be matched are also acquired. Based on the features of the data information to be matched and the event information to be matched, a matching result is obtained through an information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning an initial information matching model obtained through pre-training, based on training samples of each data information and training samples of event information that match each training sample of data information. The training samples of data information include one or more modal data, and the training samples of event information are obtained by manual annotation based on the training samples of data information. Therefore, a multimodal sample set is obtained, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modalities of data. The multimodal data features corresponding to the data content information in each multimodal sample pair and the text data features corresponding to the data text information in each multimodal sample pair are obtained. The multimodal data features include multiple modalities of data. Based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model. By using multimodal samples, a task is designed to match data content information with text information from multiple modalities for model pre-training. Then, training samples with manually labeled data and event information are used to fine-tune the initial information matching model obtained from the pre-training. The resulting information matching model is compatible with both single-modal and multimodal data information. Therefore, in the process of matching data information with time information, it can more accurately and efficiently obtain multi-dimensional image, audio, and text details in the data information, thereby improving the accuracy of the matching results, that is, ensuring the accuracy of matching event information with data information. Attached Figure Description

[0089] Figure 1 This is an application environment diagram of a data information and event information matching method in one embodiment;

[0090] Figure 2 This is a schematic diagram illustrating the practical application of a data and event information matching method in one embodiment.

[0091] Figure 3 This is a flowchart illustrating a method for matching data information and event information in one embodiment;

[0092] Figure 4This is an example diagram illustrating the data information to be matched and the event information to be matched in one embodiment.

[0093] Figure 5 This is a schematic diagram of the information feature acquisition process in one embodiment;

[0094] Figure 6 This is a schematic diagram illustrating the acquisition of information features of an event to be matched through an information matching model in one embodiment.

[0095] Figure 7 This is a schematic diagram of the coding layer structure of an information matching model in one embodiment;

[0096] Figure 8 This is a schematic diagram illustrating the actual application process of a data information and event information matching method in one embodiment.

[0097] Figure 9 This is a partial flowchart illustrating the information feature acquisition process in one embodiment;

[0098] Figure 10 This is a schematic diagram of the structure for obtaining the features of the data to be matched in one embodiment;

[0099] Figure 11 This is a partial flowchart illustrating the information feature acquisition process in another embodiment;

[0100] Figure 12 This is a partial flowchart illustrating the information feature acquisition process in yet another embodiment;

[0101] Figure 13 This is a schematic diagram of the process for obtaining a multimodal sample set in one embodiment;

[0102] Figure 14 This is a flowchart illustrating the process of obtaining an initial information matching model in one embodiment;

[0103] Figure 15 This is a schematic diagram illustrating the acquisition of multimodal data features in one embodiment;

[0104] Figure 16 This is a flowchart illustrating the process of obtaining the initial information matching model in another embodiment;

[0105] Figure 17 This is a flowchart illustrating the process of obtaining an information matching model in one embodiment;

[0106] Figure 18 This is a schematic diagram of the information matching model in one embodiment;

[0107] Figure 19 This is a partial structural diagram of a MobileNetV2 network in one embodiment;

[0108] Figure 20 This is a schematic diagram of the complete structure of a MobileNetV2 network in one embodiment;

[0109] Figure 21 This is a flowchart illustrating the process of obtaining the information matching model in another embodiment;

[0110] Figure 22 This is a schematic diagram of feature processing in an information matching model in one embodiment;

[0111] Figure 23 This is a schematic diagram illustrating the complete process of a method for matching data information and event information in one embodiment.

[0112] Figure 24 This is a structural block diagram of a data information and event information matching device in one embodiment;

[0113] Figure 25 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0114] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0115] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0116] The solutions provided in this application relate to Artificial Intelligence as a Service (AIaaS) in cloud technology. AIaaS is also commonly referred to as "AI as a Service." This is a mainstream service model for artificial intelligence platforms. Specifically, AIaaS platforms break down several common AI services and provide them as independent or packaged services in the cloud. This service model is similar to opening an AI-themed marketplace: all developers can access and use one or more AI services provided by the platform through API interfaces. Some experienced developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and maintain their own dedicated cloud AI services. The following embodiments illustrate this further:

[0117] The data and event information matching method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another server.

[0118] Specifically, taking server 104 as an example, before matching data information and event information, an information matching model needs to be trained. Terminal 102 can send instructions to server 104 to train the model, or server 104 can directly start model training; this is not limited here. Therefore, to match data information and event information, server 104 can obtain the data information and event information to be matched from the data storage system, or obtain them through communication with terminal 102; this is not limited here. Based on this, server 104 obtains the features of the event information to be matched corresponding to the event information to be matched, and obtains the features of the data information to be matched corresponding to the data information to be matched. Then, based on the features of the data information to be matched and the event information to be matched, it obtains the matching result through the information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched.

[0119] Secondly, taking a high-computing-power terminal 102 as an example, before matching data information and event information, an information matching model needs to be trained. Terminal 102 can train the model itself or obtain it through communication with server 104; this is not limited here. Based on this, terminal 102 obtains the data information to be matched and the event information to be matched, and obtains the features of the event information to be matched and the features of the data information to be matched. Then, based on the features of the data information to be matched and the event information to be matched, it obtains the matching result through the information matching model.

[0120] It should be understood that the aforementioned information matching model can be obtained by: acquiring training samples of each data information and training samples of event information that match each training sample of data information. Based on this, data information features corresponding to each training sample of data information are obtained, and event information features corresponding to each training sample of event information are obtained. Finally, based on each data information feature and each event information feature, the predicted matching degree between each data information feature and each event information feature is obtained, and the initial information matching model is fine-tuned based on each actual matching degree and each predicted matching degree to obtain the information matching model. The information matching model is used to calculate the matching degree between the data information and the event information.

[0121] Secondly, the initial information matching model can be obtained by: acquiring a multimodal sample set, which includes multiple multimodal sample pairs, each including data content information and data text information, the data content information including multiple modal data; acquiring the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair, the multimodal data features including multiple modal features; and training the initial pre-trained model based on the multimodal data features and text data features in each multimodal sample pair to obtain the target pre-trained model, and using the target pre-trained model as the initial information matching model.

[0122] The initial information matching model is used to calculate the probability that data content information and data text information come from the same data information. Specifically, multimodal sample pairs can include: video content information and video text information, where video content information includes text data, image data, and audio data; and multimodal data features specifically include text features, image features, and audio features. Alternatively, multimodal sample pairs can include: image-text content information and image-text information, where image-text content information includes text data and image data; and multimodal data features specifically include text features and image features. It should be understood that the foregoing examples are only for understanding this scheme and should not be construed as limiting this scheme.

[0123] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and aircraft. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers. This invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0124] Furthermore, the data and event information matching method provided in this application embodiment can be applied to scenarios such as news recommendation, searching for articles, images, or videos based on event information, and ranking and recommending video data information. Taking its application to news recommendation as an example, the specific display method of the same event information differs in different news articles, that is, the types of data information included are different. For example, data information can include multiple types such as image and text data information, text data information, and video data information. Based on this, in the news recommendation process, the data and event information matching method provided in this application embodiment first obtains the event information to be matched, then matches each data information in the data information database with the event information to be matched, that is, selects a data information from the data information database as the data information to be matched, then obtains the features of the event information to be matched corresponding to the event information to be matched, and obtains the features of the data information to be matched corresponding to the data information to be matched, and then obtains the matching result through the information matching model based on the features of the data information to be matched and the event information to be matched. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched. By performing similar operations on each piece of data in the database, the matching results between each piece of data and the event information to be matched can be obtained. Based on the matching results between each piece of data and the event information to be matched, the data information to be recommended (i.e., news) can be determined.

[0125] Based on this, in practical applications, such as scenarios where users search for events in real time, obtaining the event information to be matched can be achieved by: when a user needs to perform information matching, inputting information about various elements, and generating the event information to be matched based on this information. Secondly, in scenarios involving data information recommendation, the distribution of various data information can be obtained, and clustering can be performed based on this distribution to generate multiple event information to be matched. Then, based on the information matching method provided in this application, each event information to be matched is matched with each data information to be matched, thereby obtaining the degree of matching between each event information to be matched and each data information to be matched. Based on each degree of matching, a matching relationship can be established between each event information to be matched and each data information to be matched, or the recommended data information corresponding to each event information to be matched can be determined.

[0126] It should be understood that the data to be recommended can be data on the degree of matching with the event information to be matched, or it can be multiple data points with the highest degree of matching with the event information to be matched, such as... Figure 2 As shown, specific event information 201 includes person A, action B, and location C. Using the data information and event information matching method provided in this application embodiment, data information 202 with a high degree of matching with person A, action B, and location C is determined. Data information 202 specifically includes multiple data information titles, related media information, and text content. It should be understood that in practical applications, event information may also include person and action, or action and location, or person, action, and time, etc. Figure 2 The examples are for understanding this solution only and should not be construed as limiting the solution.

[0127] In one embodiment, such as Figure 3 As shown, a method for matching data information with event information is provided, which can be applied to... Figure 1 Taking a terminal as an example, it can be understood that this method can also be applied to a server, and to a system that includes both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0128] Step 302: Obtain the data information to be matched and the event information to be matched. The data information to be matched includes one or more modal data.

[0129] The data information includes one or more modalities, meaning it can be, but is not limited to, text, video, and graphic data. If the data is text, it includes text data; similarly, if it is video, it includes text, image, and audio data; and if it is graphic data, it includes both text and image data. Secondly, the event information is textual description information, and it describes at least one of the following elements: location, people, time, and action.

[0130] Specifically, the terminal can obtain the data and event information to be matched from local storage, or it can obtain the data and event information to be matched through communication with the server. The method of obtaining the data to be processed is not limited here. Furthermore, the aforementioned event information to be matched may also include matching element information, which may include, but is not limited to, at least one of location, person, time, and action. For example, the event information to be matched may include person A, action B, location C, and time D, and this event information specifically describes that person A performed action B at location C and time D. Alternatively, the event information to be matched may include person A, action B, and time D, and this event information specifically describes that person A performed action B at time D. It should be understood that the foregoing examples are only for understanding this solution and should not be construed as limiting this solution.

[0131] Secondly, if the data to be matched is video data, then the video data can include, but is not limited to, video title (text data), video screen content (image data), video audio content (audio data), video subtitles (text data), and video audio-to-text conversion (text data). Similarly, if the data to be matched is text data, then the text data can include, but is not limited to, text title (text data) and text content (text data). Furthermore, if the data to be matched is image-text data, then the image-text data can include, but is not limited to, image-text title (text data), image-text content (text data), and accompanying images (image data).

[0132] For ease of understanding, such as Figure 4 As shown, Figure 4 Figure (A) shows that the text information of the event information 401 to be matched is: Rare species appear in a certain location. The data information to be matched is specifically image and text data information 402, which includes an image and text title 4021 and an image and text illustration 4022. The text information of the image and text title 4021 is: A rare species appears in a certain county of a certain city. Secondly, Figure 4Figure (B) shows that the text information of the event information 403 to be matched is: "Someone did something." The data information to be matched is video data information 404, which includes a video title 4041 and video content 4042. The text information of the video title 4041 is: "Someone went to do this!" The result...

[0133] Step 304: Obtain the features of the event information to be matched corresponding to the event information to be matched, and obtain the features of the data information to be matched corresponding to the data information to be matched.

[0134] Specifically, the terminal performs text feature extraction processing on the elements to be matched in the event information to be matched, so as to obtain the features of the event information to be matched corresponding to each event information. It also obtains the features of the data information to be matched corresponding to the data information to be matched.

[0135] Since the information to be matched is specifically textual description, the corresponding event information features to be matched are specifically textual features. Secondly, since the data information can be any of the following: textual, video, or image-text data, or multimodal data including text, video, and audio data, the features corresponding to the data to be matched can also be textual, image, or audio features, or multimodal features including textual, image, and audio features. The specific features to be matched need to be determined based on the data to be matched, therefore no limitation is made here.

[0136] Step 306: Based on the features of the data information to be matched and the features of the event information to be matched, obtain the matching result through the information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning the initial information matching model obtained by pre-training based on the training samples of each data information and the event information training samples that match each data information training sample. The data information training samples include one or more modal data, and the event information training samples are obtained by manual annotation based on the data information training samples.

[0137] Specifically, the aforementioned matching degree refers to the correlation between the data information to be matched and the event information to be matched. The matching result describes the degree of matching between the data information to be matched and the event information to be matched; that is, the matching result can directly describe whether the data information to be matched and the event information to be matched match or not. Alternatively, the matching result can describe the matching score between the data information to be matched and the event information to be matched, such as a matching score of 40 or a matching score of 90, etc. Then, based on different application scenario requirements, it is determined whether the data information to be matched and the event information to be matched are actually matched.

[0138] Secondly, the information matching model is obtained by fine-tuning the initial information matching model obtained through pre-training, based on training samples of each data information and training samples of event information that match each training sample of the data information. The training samples of the data information include one or more modal data, and the training samples of the event information are obtained by manual annotation based on the training samples of the data information. Furthermore, the information matching model is used to calculate the degree of matching between the data information and the event information.

[0139] To ensure that the obtained information matching model is compatible with multimodal data and achieves good training results with as little manually labeled data as possible, a model pre-training process based on multimodal data is performed before training the information matching model. The pre-trained model is then used as the initial information matching model. Therefore, the information matching model can be obtained by fine-tuning the pre-trained initial information matching model based on the training samples of each data information and the event information training samples that match each data information training sample.

[0140] The initial information matching model is obtained by: acquiring a multimodal sample set, which includes multiple multimodal sample pairs, each including data content information and data text information, with the data content information including multiple modal data.

[0141] Specifically, when the data content information is video content information, the data text information is video text information. Therefore, the data content information (video content information) includes, but is not limited to: video screen content (image data), video audio content (audio data), video subtitles (text data), and the data text information (video text information) specifically includes the video title (text data). Similarly, when the data content information is graphic and text content information, the data text information is graphic and text information. Therefore, the data content information (graphic and text content information) includes, but is not limited to, graphic and text content (text data), graphic and text illustrations (image data), and the data text information (graphic and text information) specifically includes graphic and text titles (text data).

[0142] Based on this, the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair are further obtained. The multimodal data features include multiple modal features. Specifically, when the data content information is video content information, the multimodal data features include text features, image features, and audio features. Similarly, when the data content information is image-text content information, the multimodal data features include text features and image features. Then, based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model. The initial information matching model is used to calculate the probability that the data content information and the data text information come from the same data information.

[0143] Specifically, the terminal obtains matching results through an information matching model. It predicts the matching result based on the features of the data to be matched and the event to be matched, and then determines the degree of matching between the data and the event based on the matching result. Therefore, by determining the degree of matching between the data and the event, it can be determined whether they match. In practical applications, all data in the database is matched one-to-one with the event information, resulting in different matching results. Then, a response operation is performed based on the specific scenario requirements.

[0144] For example, taking news (data information)-based recommendation as an example, the news recommendation process first involves obtaining the event information to be matched. Then, each news item in the news database (data information repository) is matched with the event information to be matched, resulting in matching results for each news item in the database. If the news database includes news 1, news 2, news 3, and news 4, and based on the aforementioned similar operations, the matching result 1 for news 1, the matching result 2 for news 2, the matching result 3 for news 3, and the matching result 4 for news 4 are obtained, and matching result 1 is a match, matching result 2 is a non-match, matching result 3 is a non-match, and matching result 4 is a match, then news 1 corresponding to matching result 1 and news 4 corresponding to matching result 4 can be used as the news to be recommended. That is, news 1 and news 4 are displayed on the user's search recommendation interface, thus completing the news recommendation. It should be understood that the examples in this embodiment are for understanding this solution and should not be construed as limiting this solution.

[0145] In the above-mentioned method for matching data and event information, a task is designed to match data content information and text information from multiple modalities using multimodal samples for model pre-training. Then, training samples of manually labeled data and event information are used to fine-tune the initial information matching model obtained from the pre-training. The resulting information matching model is compatible with both single-modal and multimodal data information. Therefore, in the process of matching data and time information, it can more accurately and efficiently obtain multi-dimensional image, audio, and text details in the data information, thereby improving the accuracy of the matching results and ensuring the accuracy of the matching between event information and data information.

[0146] In one embodiment, such as Figure 5 As shown, the features of the event information to be matched, corresponding to the event information to be matched, are obtained, including:

[0147] Step 502: Based on the event information to be matched, obtain the features of the event information to be matched through the information matching model.

[0148] Specifically, the terminal uses the event information to be matched as input to the information matching model. The encoding layer in the information matching model extracts text features from the elements to be matched in the event information to obtain the event information features. Secondly, since the event information to be matched is specifically textual description information, the encoding layer in the information matching model specifically performs text segmentation on the event information to be matched, obtaining multiple text tokens. Then, based on the multiple tokens, it obtains the text sequence corresponding to the event information to be matched, and generates the event information features corresponding to the event information to be matched based on each text sequence. The event information features are specifically text features.

[0149] For ease of understanding, such as Figure 6 As shown, the event information 602 to be matched is input, and the event information 602 to be matched is divided into multiple text tags 6041 to 6042, thereby forming the text sequence 606 corresponding to the event information 602 to be matched. Then, the text sequence 606 is processed to obtain the event information feature 608 to be matched.

[0150] Furthermore, the encoding layer of the information matching model specifically adopts a Transformer encoding structure, such as... Figure 7As shown, the encoding layer specifically consists of a multi-head attention mechanism layer 701 and a first residual summation layer 702, followed by a cascaded forward propagation layer 703 and a second residual summation layer 704. Therefore, when the event information to be matched is input into the encoding layer, the encoding layer encodes the event information through the multi-head attention mechanism layer 701, the first residual summation layer 702, the forward propagation layer 703, and the second residual summation layer 704, thereby extracting features from the event information to obtain the features of the event information to be matched. In this embodiment, a 12-layer Transformer is specifically used, which means that... Figure 7 The Transformer encoding structure shown is stacked 12 times.

[0151] Obtain the features of the data to be matched that correspond to the data to be matched, including:

[0152] Step 504: Obtain the text features to be matched from the data information features to be matched through the information matching model, and obtain the data information features to be matched from the preset cache database. The data information features to be matched include at least one of the image features to be matched and the audio features to be matched.

[0153] The data to be matched includes multiple modalities. Specifically, the terminal uses the data to be matched as input to the information matching model. The encoding layer in the information matching model extracts text features from the data to be matched to obtain the text features to be matched. The specific structure of the encoding layer is as follows: Figure 7 The types will not be elaborated here.

[0154] Secondly, in practical applications, the correlation calculation (i.e., matching degree calculation) between the same data information and different event information may be involved. Therefore, repeatedly extracting the data information features of the same data information will increase the matching time and resource consumption. Thus, when the data information is video data information or image and text data information, it involves downloading video data information and image and text data information, and the processing of image data and audio data will consume more feature extraction time.

[0155] Based on this, before matching data and event information, each video data point is acquired in advance. An image feature extraction model is used to obtain the image features of the video data, and an audio feature extraction model is used to obtain the audio features. These image and audio features are then cached in a pre-set cache database. Similarly, before matching data and event information, each text and image data point can also be acquired in advance. An image feature extraction model is used to obtain the image features of the text and image data, and these image features are also cached in a pre-set cache database.

[0156] Furthermore, when a user needs to perform information matching, an information matching request is initiated. This request instructs the matching of data information for the event information to be matched. After the terminal obtains the data information to be matched and the event information to be matched, it needs to further determine the specific data type of the data information to be matched. If it is video data information or image and text data information, the terminal obtains the image features to be matched, or the image features and audio features to be matched, from a pre-set cache database, and then obtains the text features to be matched from the data information features through the information matching model, thereby completing the acquisition of data information features.

[0157] For ease of understanding, such as Figure 8 As shown, a video data information database 801 and an image and text data information database 802 are generated. The audio feature extraction model extracts audio features from all video data information in the video data information database 801 and writes the obtained audio features into the cache database 805 for caching. Similarly, the image feature extraction model 805 extracts image features from all video data information in the video data information database 801 and from all image and text data information in the image and text data information database 802, and writes the obtained image features into the cache database 805 for caching. Based on this, after obtaining the information matching request 806, the request scheduling and response service 807 schedules the information matching request 806, and determines through data type judgment 808 whether to directly obtain the text features to be matched from the data information features to be matched through the information matching model, or to obtain image features, or image features and audio features 809 from the cache database 805. Information matching 710 is then performed based on the obtained data information features, and the matching result is fed back to the request scheduling and response service 807. The request scheduling and response service 807 responds to the information matching request 806 based on the matching result obtained in information matching 710. Furthermore, to improve computational speed, in practical applications, the model in this embodiment can be accelerated using the Forward tool.

[0158] In practical applications, to facilitate the retrieval of corresponding data information features based on data information, each data information in this embodiment carries a data information identifier. Therefore, when the image features, or image features and audio features corresponding to the data information are written into the cache database for storage, the image features, or image features and audio features will carry the data information identifier corresponding to the data information.

[0159] In this embodiment, since the data information features corresponding to the same data information usually do not change, the extraction of text features is decoupled from the acquisition of image features and audio features to be matched. In scenarios where the data information to be matched includes multiple modal data, the text features to be matched can be obtained through the information matching model, but there is no need to obtain the image features and audio features to be matched through the information matching model. Instead, at least one of the image features and audio features to be matched is directly obtained from the preset cache database. While ensuring the accuracy of the image features and audio features to be matched, the time spent processing image data and audio data to obtain image features and audio features is reduced, that is, the time spent on feature extraction of image features and audio features to be matched is reduced. Thus, while ensuring the reliability of matching data information and event information, the matching efficiency of data information and event information is improved.

[0160] In one embodiment, such as Figure 9 As shown, the features of the data to be matched, corresponding to the data to be matched, are obtained, including:

[0161] Step 902: If the data information to be matched is text data information, based on the data information to be matched, the features of the data information to be matched are obtained through the information matching model. The features of the data information to be matched are the text features to be matched.

[0162] As described in the preceding embodiments, the data to be matched can be text data, video data, or image and text data. This embodiment first describes how to obtain the features of the data to be matched corresponding to the text data when the data to be matched is text data.

[0163] Specifically, after the terminal obtains the data information to be matched, it is necessary to further determine the data type of the data information to be matched. If the data information to be matched is text data, the data information to be matched is used as the input of the information matching model. The encoding layer of the information matching model processes the data information to be matched to obtain the features of the data information to be matched, and the aforementioned features of the data information to be matched are the text features to be matched.

[0164] Secondly, the encoding layer in the information matching model specifically performs text segmentation on the data information to be matched, obtaining multiple text tokens. Then, based on these multiple tokens, it obtains the text sequence corresponding to the data information to be matched, and generates the features of the data information to be matched based on each text sequence. Specifically, the text sequence corresponding to the data information to be matched includes multiple tokens. For example, if the data information to be matched includes a text title, and the specific text information of the text title is "Xiao Meng arrived in City A on October 15th of this year", then after segmenting the text information of the aforementioned text title, multiple text tokens can be obtained: [Xiao], [Meng], [Zai], [Jin], [Nian], [Shi], [Yue], [Shi], [Wu], [Ri], [Dui], [Da], [A], [Cheng]. The brackets "[]" and their contents represent text tokens. Thus, the text sequence corresponding to the text title in the data information to be matched can be obtained as: [Xiao][Meng][Zai][Jin][Nian][Shi][Yue][Shi][Wu][Ri][Dui], [Da][A][Cheng]. It should be understood that the foregoing examples are only for understanding the text sequences described in this scheme and should not be construed as limiting this scheme.

[0165] Furthermore, for ease of understanding, such as Figure 10 As shown, the data information to be matched 1002 is input, and the data information to be matched 1002 is divided into multiple text tags 10041 to 10042, thereby forming the text sequence 1006 corresponding to the data information to be matched 1002. Then, the text sequence 1006 is processed to obtain the text feature 1009 to be matched (i.e. the data information feature to be matched).

[0166] It should be understood that in practical applications, the event information to be matched and the data information to be matched are jointly input into the encoding layer of the information matching model. The encoding layer of the information matching model will input the features of the data information to be matched (i.e., text features) and the features of the event information to be matched, and the specific structure of the encoding layer of the information matching model is as follows: Figure 7 As shown, it will not be elaborated further here.

[0167] In this embodiment, the encoding layer in the information matching model specifically performs text segmentation on the data information to be matched to obtain the text sequence corresponding to the data information to be matched. The resulting features of the data information to be matched can accurately reflect the position of each token in the data information to be matched, as well as the text features corresponding to each token itself, thereby improving the reliability and feature completeness of the obtained features of the data information to be matched, and thus improving the reliability and accuracy of information matching.

[0168] In one embodiment, such as Figure 11As shown, the features of the data to be matched, corresponding to the data to be matched, are obtained, including:

[0169] Step 1102: If the data information to be matched is video data information, obtain the text features to be matched through the information matching model based on the data information to be matched.

[0170] If the data to be matched is video data, the features of the data to be matched include text features, image features, and audio features.

[0171] Specifically, after the terminal acquires the data information to be matched, it needs to further determine the data type. If the data information is video data, it is used as input to the information matching model. The encoding layer of the information matching model processes the data information to obtain its features, which are text features to be matched. Next, the encoding layer in the information matching model performs text segmentation on the data information to be matched, obtaining multiple tokens. Then, based on these tokens, it obtains the text sequence corresponding to the data information to be matched, and generates the text features corresponding to each text sequence. The specific implementation is similar to the aforementioned embodiments and will not be repeated here.

[0172] Step 1104: Based on the data to be matched, obtain the image features and audio features to be matched from the cache database.

[0173] pass Figure 8 It can be seen that the cache database includes: image features obtained by extracting image features from each video data information in the video data information database through the image feature extraction model, and audio features obtained by extracting audio features from each video data information through the audio feature extraction model.

[0174] Secondly, in order to facilitate the search of corresponding data information features based on data information, each video data information in this embodiment carries a data information identifier. Therefore, when the image features and audio features corresponding to the video data information are written into the cache database for storage, the image features and audio features will carry the data information identifier corresponding to the video data information.

[0175] Specifically, after the terminal determines that the data information to be matched is video data information, and uses the data information to be matched as the input of the information matching model, it further obtains the image features and audio features corresponding to the data information identifier from the cache database based on the data information identifier corresponding to the data information information, and determines the image features and audio features corresponding to the data information identifier as the image features and audio features to be matched.

[0176] Therefore, through steps 1102 and 1104, the matching data information features, including the text features to be matched, the image features to be matched, and the audio features to be matched, can be obtained.

[0177] In this embodiment, the extraction of text features is decoupled from the acquisition of image and audio features. While ensuring the accuracy of data information features, it can reduce the time spent on feature extraction of the image and audio features to be matched in practical applications, thereby improving the matching efficiency of data information and event information while ensuring the reliability of matching data information and event information.

[0178] In one embodiment, such as Figure 12 As shown, the features of the data to be matched, corresponding to the data to be matched, are obtained, including:

[0179] Step 1202: If the data to be matched is image and text data, obtain the text features to be matched through the information matching model based on the data to be matched.

[0180] If the data to be matched is text and image data, the features of the data to be matched include the text features to be matched and the image features to be matched.

[0181] Specifically, the data to be matched is used as input to the information matching model. The encoding layer of the information matching model processes the data to be matched to obtain features of the data to be matched, and these features are text features to be matched. Secondly, the encoding layer in the information matching model performs text segmentation on the data to be matched, obtaining multiple tokens. Then, based on these multiple tokens, it obtains the text sequence corresponding to the data to be matched, and generates the text features to be matched based on each text sequence. The specific implementation method is similar to the aforementioned embodiments and will not be repeated here.

[0182] Step 1204: Based on the data to be matched, obtain the features of the image to be matched from the cache database.

[0183] pass Figure 8 It can be seen that the cache database includes: image features obtained by extracting image features from each image and text data information in the image and text data information database through an image feature extraction model.

[0184] Secondly, in order to facilitate the search for corresponding data information features based on data information, each image and text data information in this embodiment carries a data information identifier. Therefore, when the image features corresponding to the image and text data information are written into the cache database for storage, the image features will carry the data information identifier corresponding to the data information.

[0185] Specifically, after the terminal determines that the data information to be matched is image and text data information, and uses the data information to be matched as the input of the information matching model, it further obtains the image features corresponding to the data information identifier from the cache database based on the data information identifier corresponding to the data information information to be matched, and determines the image features corresponding to the data information identifier as the image features to be matched.

[0186] Therefore, through steps 1202 and 1204, the data information features to be matched, including the text features to be matched and the image features to be matched, can be obtained.

[0187] In this embodiment, the extraction of text features is decoupled from the acquisition of image features. While ensuring the accuracy of data information features, it can reduce the time spent on feature extraction of the image to be matched in practical applications. Thus, while ensuring the reliability of matching data information and event information, it further improves the matching efficiency of data information and event information.

[0188] In one embodiment, during the process of obtaining the initial information matching model, each multimodal sample pair is acquired as follows: (e.g.) Figure 13 As shown, the multimodal sample set is obtained, including:

[0189] Step 1302: Obtain data content information, first data text information, and second data text information. The matching degree between the data content information and the first data text information is greater than the matching degree threshold, and the matching degree between the data content information and the second data text information is less than the matching degree threshold.

[0190] The degree of matching is specifically the relevance, which is expressed as a percentage. The matching threshold can be 50% or 60%, and there is no specific limitation.

[0191] Since most of the data content is highly correlated with the data titles, when acquiring the multimodal sample set, we first acquire the data content information, the first set of data text information that is highly correlated with the data content information, and the first set of data text information that is less correlated with the data content information. Specifically, the aforementioned text data refers to the data titles corresponding to the data content information.

[0192] Specifically, the data content information is acquired, along with first data text information whose matching degree with the data content information is greater than a matching degree threshold, and second data text information whose matching degree with the data content information is less than a matching degree threshold. For example, if the data content information is video content information, and the video footage specifically shows rare birds, the first data text information with a high matching degree is "rare species appeared in a certain city and county," while the first data text information with a low matching degree is "someone did this."

[0193] Step 1304: Mask the data content information to obtain the masked data content information.

[0194] Specifically, the data content information is masked to obtain the masked data content information. Since the data content information can be video content information, when the video content information specifically includes video frame content (image data), video audio content (audio data), and video subtitles (text data), the masking of the data content information (video content information) can be performed by: masking the video frame content, or masking the video audio content, or masking the video subtitles, or partially masking the video frame content, video audio content, and video subtitles; the specific method is not limited here. Similarly, if the data content information is graphic content information, when the graphic content information specifically includes graphic content (text data) and graphic images (image data), the masking of the data content information (graphic content information) can be performed by: masking the graphic content, or masking the graphic images, or partially masking both the graphic content and graphic images; the specific method is not limited here.

[0195] Step 1306: Mask the first data text information to obtain the masked first data text information.

[0196] Furthermore, the first data text information is masked to obtain the masked first data text information. For example, if the first data text information is "A rare species has appeared in a certain city and county", then the masked first data text information can be "A rare species has appeared in xxxx" or "A rare xxxx has appeared in a certain city and county".

[0197] Therefore, the multimodal sample set includes at least: a first multimodal sample pair including data content information and first data text information, a second multimodal sample pair including data content information and second data text information, a third multimodal sample pair including masked data content information and first data text information, and a fourth multimodal sample pair including data content information and masked first data text information.

[0198] Based on this, the multimodal sample set specifically includes multiple positive and negative multimodal samples. After the construction of the multimodal samples is completed, a task of matching data content information with data text information is designed for model pre-training. The model pre-training task is to input data content information and data text information, use the model to determine whether the data content information and data text information come from the same data information, and perform backpropagation to correct the model parameters of the initial pre-trained model.

[0199] In this embodiment, a model is pre-trained by designing a task to match data content information with data text information. This enables the information matching model to learn multi-dimensional and multimodal feature information from the pre-trained data content information during the matching process, further improving the extraction of multi-dimensional details such as images, audio, and text, and further improving the accuracy of the matching results.

[0200] In one embodiment, the acquisition methods for multimodal data features and text data features during the process of obtaining the initial information matching model are as follows: (e.g.) Figure 14 As shown, the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair, are obtained, including:

[0201] Step 1402: Obtain the first multimodal data features corresponding to the data content information in the first multimodal sample pair, and the first text data features corresponding to the first data text information.

[0202] The data content information includes multiple modalities, with the most modalities involved when the data content information is specifically video content information. That is, when the data content information is video content information, it specifically includes text data, image data, and audio data, where image data consists of multiple image frames and audio data consists of multiple audio frames. Based on this, when acquiring the multimodal data features and text data features from each multimodal sample pair, the data content information of the first multimodal sample pair is input into the encoding layer of the initial pre-trained model. The encoding layer encodes the text data, image data, and audio data to obtain the first multimodal data features corresponding to the data content information.

[0203] For ease of understanding, let's take video content information as an example, such as... Figure 15 As shown, taking video content information 1501 as input, the video content information is first divided into text data 1502, image data 1503, and audio data 1504. The image data 1503 is further divided into multiple image frames 1505, and image frame feature sequences are calculated for the multiple image frames 1505. Then, a fully connected layer is used to perform feature dimension transformation processing on the image frame feature sequences. Similarly, the audio data 1504 is divided into multiple audio frames 1506, and audio frame feature sequences are calculated for the multiple audio frames 1506. Then, a fully connected layer is used to perform feature dimension transformation processing on the audio frame feature sequences. This results in multimodal data features 1510, including text features 1507, image data 1508, and audio data 1509.

[0204] It should be understood that in practical applications, if the data content after masking does not include any of the text, image, or audio data, the encoding layer in the initial pre-trained model can set the corresponding feature part as a placeholder special character. Furthermore, if other types of data information modalities are included in the practical application, the data information modalities can be directly added after the current feature input, and the corresponding feature processing flow and feature part positions can be set.

[0205] Secondly, the first data text information in the first multimodal sample pair is used as the input of the initial information matching model. The encoding layer in the initial information matching model performs text segmentation on the first data text information to obtain multiple tokens of the first data text information. Then, based on the multiple tokens of the first data text information, the text sequence corresponding to the first data text information is obtained, and based on the text sequence corresponding to the first data text information, the first text data feature corresponding to the first data text information is generated.

[0206] Based on this, the first multimodal data features corresponding to the data content information in the first multimodal sample pair, and the first text data features corresponding to the first data text information can be obtained.

[0207] Step 1404: Obtain the first multimodal data features corresponding to the data content information in the second multimodal sample pair, and the second text data features corresponding to the second data text information.

[0208] Similar to step 1402, when the data content information is video content information, it specifically includes text data, image data, and audio data, where the image data consists of multiple image frames and the audio data consists of multiple audio frames. Based on this, the data content information (i.e., video content information) in the second multimodal sample pair is input into the encoding layer of the initial pre-trained model. The encoding layer encodes the text data, image data, and audio data to obtain the first multimodal data features corresponding to the data content information. Next, the second data text information in the second multimodal sample pair is used as input to the initial information matching model. The encoding layer in the initial information matching model performs text segmentation on the second data text information, obtaining multiple tokens for the second data text information. Then, based on the multiple tokens of the second data text information, the text sequence corresponding to the second data text information is obtained, and based on the text sequence corresponding to the second data text information, the second text data features corresponding to the second data text information are generated.

[0209] Based on this, the first multimodal data features corresponding to the data content information in the second multimodal sample pair, and the second text data features corresponding to the second data text information can be obtained.

[0210] Step 1406: Obtain the second multimodal data features corresponding to the data content information after the masking process of the third multimodal sample pair, and the first text data features corresponding to the first data text information.

[0211] Similar to step 1402, when the data content information is video content information, it specifically includes text data, image data, and audio data, where the image data consists of multiple image frames and the audio data consists of multiple audio frames. Based on this, the data content information (i.e., video content information) after the third multimodal sample alignment and masking process is input into the encoding layer of the initial pre-trained model. The encoding layer encodes the text data, image data, and audio data to obtain the second multimodal data features corresponding to the masked data content information. It should be understood that since any of the text data, image data, and audio data may be masked during the masking process, the second multimodal data features include at least one of the text features, image features, and audio features.

[0212] Secondly, the first data text information in the third multimodal sample pair is used as the input of the initial information matching model. The encoding layer in the initial information matching model performs text segmentation on the first data text information to obtain multiple tokens of the first data text information. Then, based on the multiple tokens of the first data text information, the text sequence corresponding to the first data text information is obtained, and based on the text sequence corresponding to the first data text information, the first text data feature corresponding to the first data text information is generated.

[0213] Based on this, we can obtain the second multimodal data features corresponding to the data content information after the masking process of the third multimodal sample, and the first text data features corresponding to the first data text information.

[0214] Step 1408: Obtain the first multimodal data feature corresponding to the data content information in the fourth multimodal sample pair, and the third text data feature corresponding to the first data text information after masking.

[0215] Similar to step 1402, when the data content information is video content information, it specifically includes text data, image data, and audio data, where the image data consists of multiple image frames and the audio data consists of multiple audio frames. Based on this, the data content information (i.e., video content information) from the fourth multimodal sample pair is input into the encoding layer of the initial pre-trained model. The encoding layer encodes the text data, image data, and audio data to obtain the first multimodal data feature corresponding to the data content information. Next, the masked first data text information from the fourth multimodal sample pair is used as the input to the initial information matching model. The encoding layer in the initial information matching model performs text segmentation on the masked first data text information, obtaining multiple tokens for the masked first data text information. Then, based on the multiple tokens of the masked first data text information, a text sequence corresponding to the masked first data text information is obtained. Finally, based on the text sequence corresponding to the masked first data text information, a third text data feature corresponding to the masked first data text information is generated.

[0216] Based on this, we can obtain the first multimodal data feature corresponding to the data content information in the fourth multimodal sample pair, and the third text data feature corresponding to the first data text information after masking.

[0217] In this embodiment, by acquiring multimodal data features of both the data content information and the masked data content information, the feature extraction capability of positive and negative samples during pre-training is improved, thereby ensuring that features of various modalities can be acquired more accurately in subsequent model training. Secondly, the extraction of textual features can further ensure the acquisition of textual features during feature decoupling in practical applications, thereby improving the feasibility of this solution and the reliability of information matching.

[0218] In one embodiment, such as Figure 16 As shown, the initial pre-trained model is trained based on the multimodal data features and text data features in each multimodal sample pair, including:

[0219] During the training process of the initial pre-trained model:

[0220] Step 1602: Based on the first multimodal data features and the first text data features, obtain the matching result of the first multimodal sample pair. The matching result is used to describe the degree of matching between the data content information and the first data text information.

[0221] Specifically, the first multimodal data features and the first text data features are input into the initial pre-trained model. The initial pre-trained model outputs the matching results of the first multimodal sample pairs. The matching results of the first multimodal sample pairs are used to describe the degree of matching between the data content information and the first data text information. That is, the matching results of the first multimodal sample pairs can determine whether the data content information and the first data text information come from the same video. If the matching results of the first multimodal sample pairs tend to be true matching results, that is, the matching results of the first multimodal sample pairs should describe a high degree of matching between the data content information and the first data text information.

[0222] Step 1604: Update the model parameters of the initial pre-trained model based on the matching results of the first multimodal sample pair.

[0223] Specifically, the matching results of the first multimodal sample pair are compared with the actual matching degree between the video data information and the first data text information, thereby performing back gradient propagation to correct the model parameters of the initial pre-trained model.

[0224] It should be understood that steps 1602 and 1604 only describe the process of adjusting the model parameters of the initial pre-trained model once during the training process of the initial pre-trained model. In practical applications, the training process of the initial pre-trained model also includes at least: inputting the first multimodal data features and the second text data features from the second multimodal sample pair into the initial pre-trained model; the initial pre-trained model outputting the matching result of the second multimodal sample pair; the matching result of the second multimodal sample pair being used to describe the degree of matching between the data content information and the second data text information; and then comparing the matching result of the second multimodal sample pair with the actual degree of matching between the video data information and the second data text information, thereby performing backpropagation to correct the model parameters of the initial pre-trained model.

[0225] Similarly, the training process of the initial pre-trained model may also include: inputting the second multimodal data features and the first text data features from the third multimodal sample pair into the initial pre-trained model; the initial pre-trained model outputting the matching result of the third multimodal sample pair; the matching result of the third multimodal sample pair being used to describe the degree of matching between the masked data content information and the first data text information; and then comparing the matching result of the third multimodal sample pair with the actual degree of matching between the masked data content information and the first data text information, thereby performing backpropagation to correct the model parameters of the initial pre-trained model.

[0226] Furthermore, the training process of the initial pre-trained model may also include: inputting the first multimodal data features and the third text data features from the fourth multimodal sample pair into the initial pre-trained model; the initial pre-trained model outputting the matching result of the fourth multimodal sample pair; the matching result of the fourth multimodal sample pair being used to describe the degree of matching between the data content information and the masked first data text information; and then comparing the matching result of the fourth multimodal sample pair with the actual degree of matching between the data content information and the masked first data text information, thereby performing backpropagation to correct the model parameters of the initial pre-trained model.

[0227] In this embodiment, the model determines whether the video data and text data come from the same video and compares them with the actual results. Then, back gradient propagation is performed to correct the model parameters of the initial pre-trained model to ensure the reliability of the pre-training.

[0228] In one embodiment, to reduce the cost of annotating training samples and event information training samples that match each training sample, the initial pre-trained model uses a pre-trained text modality model. This text modality model is trained based on text content information and text information, and it is used to calculate the probability that text content information and text information belong to the same text data. Based on this, the pre-trained text modality model is loaded as the initial pre-trained model during the pre-training process, and further pre-trained based on a multi-modal sample set. Therefore, during the pre-training of the pre-trained text modality model, text features can be learned more quickly, with a focus on acquiring image and audio detail features. This allows the information matching model to learn multi-dimensional details such as text, images, and audio using a small number of training samples and event information samples, improving both the training effect and efficiency of the information matching model.

[0229] In one embodiment, such as Figure 17 As shown, the information matching model is obtained in the following ways:

[0230] Step 1702: Obtain training samples for each data information and training samples for event information that match each data information training sample.

[0231] To ensure the information matching model can simultaneously accept three different types of data inputs—text, image and text, and video—and to guarantee the model's effectiveness, the training samples used during model training include text, image and text, and video data training samples.

[0232] Based on this, training samples of various data types are obtained, along with event information training samples that match each training data sample. The actual matching degree between the training data samples and the event information training samples can be greater than a matching degree threshold, or it can be less than the matching degree threshold, meaning there are positive and negative samples. For example, the actual matching degree between text training sample A1 and event information training sample B1 is greater than the matching degree threshold, the actual matching degree between image and text training sample A2 and event information training sample B2 is greater than the matching degree threshold, and the actual matching degree between video training sample A3 and event information training sample B3 is less than the matching degree threshold.

[0233] Step 1704: Based on each data information training sample, obtain the data information features corresponding to each data information training sample, and based on each event information training sample, obtain the event information features corresponding to each event information training sample.

[0234] Specifically, the method described in the foregoing embodiments is used to obtain the data information features corresponding to each data information training sample based on each data information training sample, and to obtain the event information features corresponding to each event information training sample based on each event information training sample. Further details will not be elaborated here.

[0235] Step 1706: Based on the features of each data information and the features of each event information, obtain the predicted matching degree between each data information feature and each event information feature, and fine-tune the initial information matching model based on each actual matching degree and each predicted matching degree to obtain the information matching model.

[0236] Specifically, data information features and event information features are used as inputs to the initial information matching model. The initial information matching model then outputs the predicted matching degree between the data information features and the event information features. Based on each actual matching degree and each predicted matching degree, the parameters of the initial information matching model are adjusted to obtain the information matching model.

[0237] In this embodiment, the information matching model structure adopts a multi-modal Transformer structure. It processes text data and event information in the same way as BERT, dividing the text data and event information into multiple tokens and finding the initial vector of each token, thereby obtaining the text features corresponding to the text data and the event information features corresponding to the event information.

[0238] For ease of understanding, using input event information and video data as an example, the specific structure of the information matching model is as follows: Figure 18As shown, event information and video data are taken as input, and the event information features corresponding to the event information, as well as the text features, image features, and audio features corresponding to the video data, are obtained through a multi-modal Transformer structure. In this embodiment, the image frame feature sequence is calculated using the MobileNetV2 network, and in order to convert the 1280-dimensional feature vector output by the MobileNetV2 network to the same length as the token vector, as shown... Figure 18 As shown, after the MobileNetV2 network outputs the image frame feature sequence, a fully connected layer is used to perform feature dimension transformation processing, thereby ensuring that the length of the obtained text features is consistent with that of the image features.

[0239] Furthermore, since the image frame feature sequence in this invention is calculated using the MobileNetV2 network, and the basic unit of the MobileNetV2 network is the Bottleneck residual block structure, as shown below... Figure 19 As shown, h is the image height, w is the image width, k is the number of feature channels, s is the stride, and t is the dilation factor. The overall structure of the Bottleneck residual block is as follows: Figure 20 As shown, column t represents the expansion factor, column c represents the number of channels, column n represents the stacking number of the corresponding basic structure, and column s represents the step size. Figure 19 As can be seen, the first convolutional layer of MobileNetV2 is a standard convolution with 32 kernels, followed by stacked Bottleneck residual blocks. Based on this, in this embodiment, the output of the penultimate layer of the entire MobileNetV2 network is taken as the image frame feature sequence, that is, the representation of each image is a 1280-dimensional vector.

[0240] In one embodiment, such as Figure 21 As shown, the training samples are video data, which consist of text, image, and audio data. Image data comprises multiple image frames, and audio data comprises multiple audio frames. Based on this, the data features corresponding to each training sample are obtained, including:

[0241] Step 2102: Perform text segmentation on the text data in each video data training sample to obtain the text sequence corresponding to the text data in each video data training sample, and generate the text features corresponding to the text data in each video data training sample based on each text sequence. The text sequence includes multiple text tokens.

[0242] The text features include: first position embedding features, first expert embedding features, and text tag features. The first position embedding features are used to characterize the position of each token in the text sequence. The first expert embedding features are used to characterize the type of text features. The text tag features include the text features corresponding to each token.

[0243] Step 2104: Perform image recognition processing on the image data in each video data training sample to obtain the image features corresponding to the image data in each video data training sample.

[0244] The image features include: second position embedding features, second expert embedding features, and image frame features. The second position embedding features are used to characterize the position of each image in the image data, the second expert embedding features are used to characterize the type of image features, and the image frame features include the image frame features corresponding to each image frame.

[0245] Step 2106: Perform audio recognition processing on the audio data in each video data training sample to obtain the audio features corresponding to the audio data in each video data training sample.

[0246] The audio features include: third position embedding features, third expert embedding features, and audio frame features. The third position embedding features are used to characterize the position of each audio frame in the audio data. The third expert embedding features are used to characterize the type of audio features. The audio frame features include the audio frame features corresponding to each audio frame.

[0247] Furthermore, the final data information features include text features, image features, and audio features.

[0248] Specifically, in the feature processing part of the information matching model, such as Figure 22 As shown, the features of each token consist of three parts: position embedding, expert embedding, and data features. The position embedding features include: a first position embedding feature 2201 for representing the position of each text token in the text sequence, a second position embedding feature 2202 for representing the position of each image token in the image data, and a third position embedding feature 2203 for representing the position of each audio token (or audio Emb) in the audio data.

[0249] Secondly, expert embedding features are used to distinguish between text data, image data, and audio data. Therefore, expert embedding features include: a first expert embedding feature 2204 for representing the type of text features, a second expert embedding feature 2205 for representing the type of image features, and a third expert embedding feature 2206 for representing the type of audio features. Data features are used to represent the initial feature vector, that is, data features include text features 2207 corresponding to each token, image frame features 2208 corresponding to each image frame, and audio frame features 2209 corresponding to each audio frame.

[0250] In this embodiment, during the process of training the information matching model, training is performed using text data training samples, image and text data training samples, and video data training samples. This ensures the reliability of the model matching and also enables the information matching model to be compatible with multimodal data information.

[0251] Based on the aforementioned embodiments, the complete process of matching data information with event information will be described below, such as... Figure 23 As shown, this method is applied to Figure 1 Taking a terminal as an example, it can be understood that this method can also be applied to a server, and to a system that includes both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0252] Step 2301: Obtain the multimodal sample set.

[0253] Specifically, since most of the data content is highly correlated with the data titles, when acquiring the multimodal sample set, we first acquire the data content information, the first data text information that is highly correlated with the data content information, and the first data text information that is less correlated with the data content information. The aforementioned text data specifically refers to the data titles corresponding to the data content information.

[0254] Based on this, we obtain data content information, as well as first data text information whose matching degree with the data content information is greater than a matching degree threshold, and second data text information whose matching degree with the data content information is less than a matching degree threshold. For example, if the data content information is video content information, and the video footage specifically shows rare birds, the first data text information with a high matching degree is "rare species appeared in a certain city and county," while the first data text information with a low matching degree is "someone did this."

[0255] Furthermore, the data content information is masked to obtain the masked data content information. Since the data content information can be video content information, when the video content information specifically includes video frame content (image data), video audio content (audio data), and video subtitles (text data), the masking of the data content information (video content information) can be performed by: masking the video frame content, or masking the video audio content, or masking the video subtitles, or partially masking the video frame content, video audio content, and video subtitles; the specific method is not limited here. Similarly, if the data content information is graphic content information, when the graphic content information specifically includes graphic content (text data) and graphic images (image data), the masking of the data content information (graphic content information) can be performed by: masking the graphic content, or masking the graphic images, or partially masking both the graphic content and graphic images; the specific method is not limited here.

[0256] Similarly, the first data text information is then masked to obtain the masked first data text information. For example, if the first data text information is "A rare species has appeared in a certain city and county", then the masked first data text information can be "A rare species has appeared in xxxx" or "A rare xxxx has appeared in a certain city and county".

[0257] Therefore, the multimodal sample set includes at least: a first multimodal sample pair including data content information and first data text information, a second multimodal sample pair including data content information and second data text information, a third multimodal sample pair including masked data content information and first data text information, and a fourth multimodal sample pair including data content information and masked first data text information.

[0258] Based on this, the multimodal sample set specifically includes multiple positive and negative multimodal samples. After the construction of the multimodal samples is completed, a task of matching data content information with data text information is designed for model pre-training. The model pre-training task is to input data content information and data text information, use the model to determine whether the data content information and data text information come from the same data information, and perform backpropagation to correct the model parameters of the initial pre-trained model.

[0259] Step 2302: Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair.

[0260] The data content information includes multiple modalities, with the most modalities involved when the data content information is specifically video content information. That is, when the data content information is video content information, it specifically includes text data, image data, and audio data, where image data consists of multiple image frames and audio data consists of multiple audio frames. Based on this, when acquiring the multimodal data features and text data features from each multimodal sample pair, the data content information of the first multimodal sample pair is input into the encoding layer of the initial pre-trained model. The encoding layer encodes the text data, image data, and audio data to obtain the first multimodal data features corresponding to the data content information.

[0261] Secondly, the first data text information in the first multimodal sample pair is used as the input to the initial information matching model. The encoding layer in the initial information matching model performs text segmentation on the first data text information, obtaining multiple tokens for the first data text information. Then, based on the multiple tokens of the first data text information, the text sequence corresponding to the first data text information is obtained, and based on the text sequence corresponding to the first data text information, the first text data feature corresponding to the first data text information is generated. Based on this, the first multimodal data feature corresponding to the data content information in the first multimodal sample pair, as well as the first text data feature corresponding to the first data text information, can be obtained.

[0262] Similarly, based on the aforementioned methods, we can also obtain the first multimodal data features corresponding to the data content information in the second multimodal sample pair, and the second text data features corresponding to the second data text information. We can also obtain the second multimodal data features corresponding to the masked data content information in the third multimodal sample pair, and the first text data features corresponding to the first data text information. Finally, we can obtain the first multimodal data features corresponding to the data content information in the fourth multimodal sample pair, and the third text data features corresponding to the masked first data text information.

[0263] Step 2303: Train the initial pre-trained model based on the multimodal data features and text data features in each multimodal sample pair.

[0264] Specifically, during the training process of the initial pre-trained model: First multimodal data features and first text data features are input to the initial pre-trained model. The initial pre-trained model outputs the matching results of the first multimodal sample pairs. These matching results describe the degree of matching between the data content information and the first data text information. Based on this, the matching results of the first multimodal sample pairs are compared with the actual degree of matching between the video data information and the first data text information, thereby performing backpropagation to correct the model parameters of the initial pre-trained model. Specific implementation methods are as follows... Figure 16The corresponding implementation examples will not be described in detail here.

[0265] Optionally, to reduce the cost of labeling training samples and event information training samples matched with each training sample, the initial pre-trained model uses a pre-trained text modality model. This text modality model is obtained after training on text content information and text information, and it is used to calculate the probability that text content information and text information belong to the same text data. Based on this, the pre-trained text modality model is loaded as the initial pre-trained model during the pre-training process, and further pre-trained on the initial pre-trained model based on a multi-modal sample set. Therefore, during the pre-training of the pre-trained text modality model, text features can be learned more quickly, with a focus on acquiring image and audio detail features. This allows the information matching model to learn multi-dimensional details such as text, images, and audio using a small number of training samples and event information samples, thus improving the training efficiency of the information matching model while ensuring its training effect.

[0266] Step 2304: Obtain training samples for each data information and training samples for event information that match each data information training sample.

[0267] To ensure the information matching model can simultaneously accept three different types of data inputs—text, image and text, and video—and to guarantee the model's effectiveness, the training samples used during model training include text, image and text, and video data training samples.

[0268] Based on this, training samples of data information including multiple data information types are obtained, as well as training samples of event information that match each training sample of data information. The actual matching degree between the training samples of data information and the training samples of event information is greater than the matching degree threshold.

[0269] Step 2305: Based on each data information training sample, obtain the data information features corresponding to each data information training sample, and based on each event information training sample, obtain the event information features corresponding to each event information training sample.

[0270] Specifically, the method described in the foregoing embodiments is used to obtain the data information features corresponding to each data information training sample based on each data information training sample, and to obtain the event information features corresponding to each event information training sample based on each event information training sample. Further details will not be elaborated here.

[0271] Step 2306: Based on the features of each data information and the features of each event information, obtain the predicted matching degree between each data information feature and each event information feature, and fine-tune the initial information matching model based on each actual matching degree and each predicted matching degree to obtain the information matching model.

[0272] Specifically, data information features and event information features are used as inputs to the initial information matching model. The initial information matching model then outputs the predicted matching degree between the data information features and the event information features. Based on each actual matching degree and each predicted matching degree, the initial information matching model is fine-tuned to obtain the information matching model.

[0273] Step 2307: Obtain the data information to be matched and the event information to be matched. The data information to be matched includes one or more modal data.

[0274] The data information includes one or more modalities, meaning it can be, but is not limited to, text, video, and graphic data. If the data is text, it includes text data; similarly, if it is video, it includes text, image, and audio data; and if it is graphic data, it includes both text and image data. Secondly, the event information is textual description information, and it describes at least one of the following elements: location, people, time, and action.

[0275] Specifically, the terminal can obtain the data and event information to be matched from local storage, or it can obtain the data and event information to be matched through communication with the server. The method of obtaining the data to be processed is not limited here. Furthermore, the aforementioned event information to be matched may also include matching element information, which may include, but is not limited to, at least one of location, person, time, and action. For example, if a user inputs person A, action B, location C, and time D in real time, the generated event information to be matched includes person A, action B, location C, and time D, and this event information specifically describes that person A performed action B at location C and time D. Alternatively, the event information to be matched may include person A, action B, and time D, and this event information specifically describes that person A performed action B at time D. It should be understood that the foregoing examples are only for understanding this solution and should not be construed as limiting this solution.

[0276] Secondly, if the data to be matched is video data, then the video data can include, but is not limited to, video title (text data), video screen content (image data), video audio content (audio data), video subtitles (text data), and video audio-to-text conversion (text data). Similarly, if the data to be matched is text data, then the text data can include, but is not limited to, text title (text data) and text content (text data). Furthermore, if the data to be matched is image-text data, then the image-text data can include, but is not limited to, image-text title (text data), image-text content (text data), and accompanying images (image data).

[0277] Step 2308: Based on the event information to be matched, obtain the features of the event information to be matched through the information matching model.

[0278] Specifically, the terminal uses the event information to be matched as input to the information matching model. The encoding layer in the information matching model extracts text features from the elements to be matched in the event information to obtain the event information features. Secondly, since the event information to be matched is specifically textual description information, the encoding layer in the information matching model specifically performs text segmentation on the event information to be matched, obtaining multiple text tokens. Then, based on the multiple tokens, it obtains the text sequence corresponding to the event information to be matched, and generates the event information features corresponding to the event information to be matched based on each text sequence. The event information features are specifically text features.

[0279] Step 2309: Obtain the text features to be matched from the information features of the data to be matched through the information matching model, and obtain the information features of the data to be matched from the preset cache database.

[0280] The data to be matched includes multiple modalities. Specifically, the terminal uses the data to be matched as input to the information matching model. The encoding layer in the information matching model extracts text features from the data to be matched to obtain the text features to be matched. The specific structure of the encoding layer is as follows: Figure 7 The types will not be elaborated here.

[0281] Secondly, the information matching model is obtained by fine-tuning the initial information matching model obtained through pre-training, based on training samples of each data information and training samples of event information that match each training sample of the data information. The training samples of the data information include one or more modal data, and the training samples of the event information are obtained by manual annotation based on the training samples of the data information. Furthermore, the information matching model is used to calculate the degree of matching between the data information and the event information.

[0282] In practical applications, the correlation calculation (i.e., matching degree calculation) between the same data information and different event information may be involved. Therefore, repeatedly extracting the data information features of the same data information will increase the matching time and resource consumption. Thus, when the data information is video data information or image and text data information, it involves downloading video data information and image and text data information, and the processing of image data and audio data will consume more feature extraction time.

[0283] Based on this, before matching data and event information, each video data point is acquired in advance. An image feature extraction model is used to obtain the image features of the video data, and an audio feature extraction model is used to obtain the audio features. These image and audio features are then cached in a pre-set cache database. Similarly, before matching data and event information, each text and image data point can also be acquired in advance. An image feature extraction model is used to obtain the image features of the text and image data, and these image features are also cached in a pre-set cache database.

[0284] Furthermore, when a user needs to perform information matching, an information matching request is initiated. This request instructs the matching of data information for the event information to be matched. After the terminal obtains the data information to be matched and the event information to be matched, it needs to further determine the specific data type of the data information to be matched. If it is video data information or image and text data information, the terminal obtains the image features to be matched, or the image features and audio features to be matched, from a pre-set cache database, and then obtains the text features to be matched from the data information features through the information matching model, thereby completing the acquisition of data information features.

[0285] Step 2310: Based on the information features of the data to be matched and the information features of the event to be matched, obtain the matching result through the information matching model.

[0286] Specifically, the terminal obtains matching results through an information matching model. It predicts the matching result based on the features of the data to be matched and the event to be matched, and then determines the degree of matching between the data and the event based on the matching result. Therefore, by determining the degree of matching between the data and the event, it can be determined whether they match. In practical applications, all data in the database is matched one-to-one with the event information, resulting in different matching results. Then, a response operation is performed based on the specific scenario requirements.

[0287] For example, taking news (data information)-based recommendation as an example, the news recommendation process first involves obtaining the event information to be matched. Then, each news item in the news database (data information repository) is matched with the event information to be matched, resulting in matching results for each news item in the database. If the news database includes news 1, news 2, news 3, and news 4, and based on the aforementioned similar operations, we obtain matching result 1 for news 1, matching result 2 for news 2, matching result 3 for news 3, and matching result 4 for news 4, and if matching result 1 is a match, matching result 2 is a non-match, matching result 3 is a non-match, and matching result 4 is a match, then news 1 (matching result 1) and news 4 (matching result 4) can be selected as the news to be recommended. In other words, news 1 and news 4 are displayed on the user's search recommendation interface, thus completing the news recommendation. It should be understood that, in practical applications, the data information and event information matching method provided in the embodiments of this application can also be applied to scenarios such as searching for articles, images, or videos based on event information, and sorting and recommending video data information. Therefore, the foregoing examples are only for understanding this solution and should not be construed as limiting this solution.

[0288] It should be understood that the specific implementation methods of steps 2301 to 2310 have been described in detail in the foregoing embodiments, and therefore will not be repeated here.

[0289] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0290] Based on the same inventive concept, this application also provides a data information and event information matching device for implementing the data information and event information matching method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data information and event information matching device embodiments provided below can be found in the limitations of the data information and event information matching method described above, and will not be repeated here.

[0291] In one embodiment, such as Figure 24 As shown, a device for matching data information and event information is provided, including: an acquisition module 2402, a feature processing module 2404, and an information matching module 2406, wherein:

[0292] The acquisition module 2402 is used to acquire data information to be matched and event information to be matched. The data information to be matched includes one or more modal data.

[0293] The feature processing module 2404 is used to obtain the features of the event information to be matched corresponding to the event information to be matched, and to obtain the features of the data information to be matched corresponding to the data information to be matched;

[0294] The information matching module 2406 is used to obtain matching results based on the features of the data information to be matched and the features of the event information to be matched through the information matching model. The matching results are used to describe the degree of matching between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning the initial information matching model obtained by pre-training based on the training samples of each data information and the event information training samples that match each data information training sample. The data information training samples include one or more modal data, and the event information training samples are obtained by manual annotation based on the data information training samples.

[0295] The methods for obtaining the initial information matching model include:

[0296] Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data.

[0297] Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair. The multimodal data features include multiple modal features.

[0298] Based on the multimodal data features and text data features in each multimodal sample pair, the initial pre-trained model is trained to obtain the target pre-trained model, which is then used as the initial information matching model.

[0299] In one embodiment, the feature processing module 2404 is further configured to obtain the features of the event information to be matched through an information matching model based on the event information to be matched; and to obtain the text features to be matched from the data information features to be matched through the information matching model, and to obtain the data information features to be matched from a preset cache database, wherein the data information features to be matched include at least one of image features to be matched and audio features to be matched.

[0300] In one embodiment, the feature processing module 2404 is further configured to, if the data information to be matched is text data information, obtain the features of the data information to be matched through an information matching model based on the data information to be matched, wherein the features of the data information to be matched are the text features to be matched.

[0301] In one embodiment, the feature processing module 2404 is further configured to, if the data information to be matched is video data information, obtain the text features to be matched through an information matching model based on the data information to be matched; and obtain the image features to be matched and the audio features to be matched from a cache database based on the data information to be matched; wherein, the features of the data information to be matched include the text features to be matched, the image features to be matched, and the audio features to be matched; the cache database includes: image features obtained by extracting image features from each video data information in the video data information database through an image feature extraction model, and audio features obtained by extracting audio features from each video data information through an audio feature extraction model.

[0302] In one embodiment, the feature processing module 2404 is further configured to, if the data information to be matched is image and text data information, obtain the text features to be matched through an information matching model based on the data information to be matched; and obtain the image features to be matched from a cache database based on the data information to be matched; wherein, the features of the data information to be matched include the text features to be matched and the image features to be matched; the cache database includes: image features obtained after extracting image features from each image and text data information in the image and text data information database through an image feature extraction model.

[0303] In one embodiment, the initial information matching model is obtained by acquiring data content information, first data text information, and second data text information, wherein the matching degree between the data content information and the first data text information is greater than a matching degree threshold, and the matching degree between the data content information and the second data text information is less than a matching degree threshold; and the data content information is masked to obtain masked data content information; and the first data text information is masked to obtain masked first data text information; wherein the multimodal sample set includes at least: a first multimodal sample pair including data content information and first data text information, a second multimodal sample pair including data content information and second data text information, a third multimodal sample pair including masked data content information and first data text information, and a fourth multimodal sample pair including data content information and masked first data text information.

[0304] In one embodiment, the method for obtaining the initial information matching model further includes: obtaining the first multimodal data feature corresponding to the data content information in the first multimodal sample pair, and the first text data feature corresponding to the first data text information; obtaining the first multimodal data feature corresponding to the data content information in the second multimodal sample pair, and the second text data feature corresponding to the second data text information; obtaining the second multimodal data feature corresponding to the masked data content information in the third multimodal sample pair, and the first text data feature corresponding to the first data text information; and obtaining the first multimodal data feature corresponding to the data content information in the fourth multimodal sample pair, and the third text data feature corresponding to the masked first data text information.

[0305] In one embodiment, the method of obtaining the initial information matching model further includes: during the training process of the initial pre-trained model: based on the first multimodal data features and the first text data features, obtaining the matching result of the first multimodal sample, the matching result being used to describe the degree of matching between the data content information and the first data text information; and updating the model parameters of the initial pre-trained model based on the matching result of the first multimodal sample.

[0306] In one embodiment, the multimodal sample pair specifically includes video content information and video text information, wherein the video content information includes text data, image data, and audio data; and the multimodal data features specifically include text features, image features, and audio features.

[0307] In one embodiment, the initial pre-trained model is a pre-trained text modality model. The text modality model is obtained by training based on text content information and text information, and is used to calculate the probability that text content information and text information belong to the same text data information.

[0308] In one embodiment, the information matching model is obtained by: acquiring training samples of each data information and training samples of event information that match each training sample of data information; obtaining data information features corresponding to each training sample of data information based on each training sample of data information, and obtaining event information features corresponding to each training sample of event information based on each training sample of event information; obtaining the predicted matching degree between each data information feature and each event information feature based on each data information feature and each event information feature; and fine-tuning the initial information matching model based on each actual matching degree and each predicted matching degree to obtain the information matching model.

[0309] In one embodiment, the data information training samples are video data information training samples, which consist of text data, image data, and audio data. The image data consists of multiple image frames, and the audio data consists of multiple audio frames. The method of obtaining the information matching model further includes: performing text segmentation on the text data in each video data information training sample to obtain the text sequence corresponding to the text data in each video data information training sample, and generating text features corresponding to the text data in each video data information training sample based on each text sequence, wherein the text sequence includes multiple text tokens; performing image recognition processing on the image data in each video data information training sample to obtain the image features corresponding to the image data in each video data information training sample; and performing audio recognition processing on the audio data in each video data information training sample to obtain the audio features corresponding to the audio data in each video data information training sample; wherein the data information features include text features, image features, and audio features.

[0310] Each module in the aforementioned data and event information matching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0311] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 25As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for matching data information with event information. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0312] Those skilled in the art will understand that Figure 25 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0313] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0314] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0315] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0316] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0317] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0318] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0319] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for matching data information with event information, characterized in that, The method includes: Obtain data information to be matched and event information to be matched. The data information to be matched includes one or more modal data, and the event information is used to describe at least one of the following elements: location, person, time, and action. Based on the event information to be matched, the event information to be matched is segmented into text using an information matching model to obtain text tags corresponding to at least one matching element in the event information to be matched. Based on the text tags, a text sequence corresponding to the event information to be matched is obtained. Based on the text sequence, the event information features corresponding to the event information to be matched are obtained. And the matching data information features corresponding to the data information to be matched are acquired. Based on the features of the data information to be matched and the features of the event information to be matched, a matching result is obtained through an information matching model. The matching result is used to describe the degree of matching between the data information to be matched and the event information to be matched, and the degree of matching is the correlation between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning an initial information matching model obtained through pre-training based on training samples of each data information and training samples of event information that match each training sample of the data information. The training samples of the data information include one or more modal data, and the training samples of the event information are obtained by manual annotation based on the training samples of the data information. The information matching model is used to calculate the degree of matching between the data information and the event information. The initial information matching model is obtained through the following methods: Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data. Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair, wherein the multimodal data features include multiple modal features; Based on the multimodal data features and text data features in each multimodal sample pair, the probability that the data content information and the data text information come from the same data information is calculated using an initial pre-trained model. Then, back gradient propagation is performed to correct the model parameters of the initial pre-trained model to obtain a target pre-trained model, which is then used as the initial information matching model.

2. The method according to claim 1, characterized in that, The step of obtaining the features of the data to be matched corresponding to the data to be matched includes: The matching model obtains the text features to be matched from the data information features to be matched, and retrieves the data information features to be matched from a preset cache database. The data information features to be matched include at least one of image features to be matched and audio features to be matched.

3. The method according to claim 2, characterized in that, The step of obtaining the features of the data to be matched corresponding to the data to be matched includes: If the data to be matched is text data, the features of the data to be matched are obtained through the information matching model based on the data to be matched, and the features of the data to be matched are the text features to be matched.

4. The method according to claim 2, characterized in that, The step of obtaining the features of the data to be matched corresponding to the data to be matched includes: If the data to be matched is video data, the text features to be matched are obtained based on the data to be matched through the information matching model; Based on the data to be matched, the image features and audio features to be matched are obtained from the cache database. The data information features to be matched include the text features to be matched, the image features to be matched, and the audio features to be matched. The cache database includes: image features obtained by extracting image features from each video data information in the video data information database using an image feature extraction model, and audio features obtained by extracting audio features from each of the video data information using an audio feature extraction model.

5. The method according to claim 2, characterized in that, The step of obtaining the features of the data to be matched corresponding to the data to be matched includes: If the data to be matched is image and text data, the text features to be matched are obtained through the information matching model based on the data to be matched. Based on the data to be matched, the features of the image to be matched are obtained from the cache database; The features of the data information to be matched include the text features to be matched and the image features to be matched; The cache database includes image features obtained by extracting image features from each image and text data in the image and text data information database using an image feature extraction model.

6. The method according to claim 1, characterized in that, The acquisition of the multimodal sample set includes: The data content information, the first data text information, and the second data text information are obtained, wherein the degree of matching between the data content information and the first data text information is greater than the matching degree threshold, and the degree of matching between the data content information and the second data text information is less than the matching degree threshold; The data content information is masked to obtain the masked data content information; The first data text information is masked to obtain the masked first data text information. The multimodal sample set includes at least: a first multimodal sample pair including the data content information and the first data text information; a second multimodal sample pair including the data content information and the second data text information; a third multimodal sample pair including the masked data content information and the first data text information; and a fourth multimodal sample pair including the data content information and the masked first data text information.

7. The method according to claim 6, characterized in that, The step of obtaining the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair, includes: Obtain the first multimodal data features corresponding to the data content information in the first multimodal sample pair, and the first text data features corresponding to the first data text information; Obtain the first multimodal data feature corresponding to the data content information in the second multimodal sample pair, and the second text data feature corresponding to the second data text information; Obtain the second multimodal data feature corresponding to the masked data content information in the third multimodal sample pair, and the first text data feature corresponding to the first data text information; Obtain the first multimodal data feature corresponding to the data content information in the fourth multimodal sample pair, and the third text data feature corresponding to the first data text information after masking.

8. The method according to claim 7, characterized in that, The training of the initial pre-trained model based on the multimodal data features and text data features in each multimodal sample pair includes: During the training process of the initial pre-trained model: Based on the first multimodal data features and the first text data features, a matching result for the first multimodal sample pair is obtained, and the matching result is used to describe the degree of matching between the data content information and the first data text information. Based on the matching results of the first multimodal sample pair, the model parameters of the initial pre-trained model are updated.

9. The method according to claim 1, characterized in that, The multimodal sample pairs specifically include video content information and video text information, wherein the video content information includes text data, image data, and audio data; The multimodal data features specifically include text features, image features, and audio features.

10. The method according to claim 1, characterized in that, The initial pre-trained model is a text modality model that has been trained. The text modality model is obtained after training based on text content information and text information, and the text modality model is used to calculate the probability that the text content information and text information belong to the same text data information.

11. The method according to claim 1, characterized in that, The information matching model is obtained through the following methods: Acquire training samples for each data information, and training samples for event information that match each of the data information training samples; Based on the training samples of each of the data information, the data information features corresponding to each of the data information training samples are obtained, and based on the training samples of each of the event information, the event information features corresponding to each of the event information training samples are obtained. Based on the data information features and event information features, the predicted matching degree between the data information features and event information features is obtained, and the initial information matching model is fine-tuned based on the actual matching degree and the predicted matching degree to obtain the information matching model.

12. The method according to claim 11, characterized in that, The training samples are video data training samples, which consist of text data, image data, and audio data. The image data consists of multiple image frames, and the audio data consists of multiple audio frames. The step of obtaining data information features corresponding to each training sample based on the data information includes: Text segmentation is performed on the text data in each of the video data training samples to obtain the text sequence corresponding to the text data in each of the video data training samples, and text features corresponding to the text data in each of the text sequences are generated based on the text sequences, wherein the text sequence includes multiple text tokens. Image recognition processing is performed on the image data in each of the video data training samples to obtain the image features corresponding to the image data in each of the video data training samples; Audio recognition processing is performed on the audio data in each of the video data training samples to obtain the audio features corresponding to the audio data in each of the video data training samples; The data information features include the text features, the image features, and the audio features.

13. A device for matching data information with event information, characterized in that, The device includes: The acquisition module is used to acquire data information to be matched and event information to be matched. The data information to be matched includes one or more modal data, and the event information is used to describe at least one of the following elements: location, person, time, and action. The feature processing module is used to perform text segmentation on the event information to be matched based on the event information to be matched using an information matching model, to obtain text tags corresponding to at least one element information to be matched in the event information to be matched, to obtain a text sequence corresponding to the event information to be matched based on the text tags, to obtain the event information features corresponding to the event information to be matched based on the text sequence, and to obtain the data information features to be matched corresponding to the data information to be matched. An information matching module is used to obtain matching results based on the features of the data information to be matched and the features of the event information to be matched, through an information matching model. The matching results describe the degree of matching between the data information to be matched and the event information to be matched, where the degree of matching is the correlation between the data information to be matched and the event information to be matched. The information matching model is obtained by fine-tuning an initial information matching model obtained through pre-training, based on training samples of each data information and training samples of event information that match each training sample of the data information. The training samples of the data information include one or more modal data, and the training samples of the event information are obtained by manual annotation based on the training samples of the data information. The information matching model is used to calculate the degree of matching between the data information and the event information. The initial information matching model is obtained through the following methods: Obtain a multimodal sample set, which includes multiple multimodal sample pairs. Each multimodal sample pair includes data content information and data text information. The data content information includes multiple modal data. Obtain the multimodal data features corresponding to the data content information in each multimodal sample pair, and the text data features corresponding to the data text information in each multimodal sample pair, wherein the multimodal data features include multiple modal features; Based on the multimodal data features and text data features in each multimodal sample pair, the probability that the data content information and the data text information come from the same data information is calculated using an initial pre-trained model. Then, back gradient propagation is performed to correct the model parameters of the initial pre-trained model to obtain a target pre-trained model, which is then used as the initial information matching model.

14. The apparatus according to claim 13, characterized in that, The feature processing module is further configured to obtain the text features to be matched from the data information features to be matched through the information matching model, and to obtain the data information features to be matched from a preset cache database. The data information features to be matched include at least one of image features to be matched and audio features to be matched.

15. The apparatus according to claim 14, characterized in that, The feature processing module is further configured to, if the data information to be matched is text data information, obtain the features of the data information to be matched through the information matching model based on the data information to be matched, wherein the features of the data information to be matched are the text features to be matched.

16. The apparatus according to claim 14, characterized in that, The feature processing module is further configured to, if the data information to be matched is video data information, obtain the text features to be matched through the information matching model based on the data information to be matched; and obtain the image features to be matched and the audio features to be matched from the cache database based on the data information to be matched; wherein, the features of the data information to be matched include the text features to be matched, the image features to be matched, and the audio features to be matched; the cache database includes: image features obtained by extracting image features from each video data information in the video data information database through an image feature extraction model, and audio features obtained by extracting audio features from each video data information through an audio feature extraction model.

17. The apparatus according to claim 14, characterized in that, The feature processing module is further configured to, if the data information to be matched is image-text data information, obtain the text features to be matched through the information matching model based on the data information to be matched; and obtain the image features to be matched from the cache database based on the data information to be matched; wherein, the features of the data information to be matched include the text features to be matched and the image features to be matched; the cache database includes: image features obtained by extracting image features from each image-text data information in the image-text data information database through an image feature extraction model.

18. The apparatus according to claim 13, characterized in that, The method for obtaining the initial information matching model further includes acquiring the data content information, the first data text information, and the second data text information, wherein the matching degree between the data content information and the first data text information is greater than the matching degree threshold, and the matching degree between the data content information and the second data text information is less than the matching degree threshold; and performing masking processing on the data content information to obtain masked data content information; The first data text information is masked to obtain the masked first data text information; wherein, the multimodal sample set includes at least: a first multimodal sample pair including the data content information and the first data text information, a second multimodal sample pair including the data content information and the second data text information, a third multimodal sample pair including the masked data content information and the first data text information, and a fourth multimodal sample pair including the data content information and the masked first data text information.

19. The apparatus according to claim 18, characterized in that, The method for obtaining the information matching model further includes: obtaining the first multimodal data feature corresponding to the data content information in the first multimodal sample pair, and the first text data feature corresponding to the first data text information; obtaining the first multimodal data feature corresponding to the data content information in the second multimodal sample pair, and the second text data feature corresponding to the second data text information; obtaining the second multimodal data feature corresponding to the masked data content information in the third multimodal sample pair, and the first text data feature corresponding to the first data text information; obtaining the first multimodal data feature corresponding to the data content information in the fourth multimodal sample pair, and the third text data feature corresponding to the masked first data text information.

20. The apparatus according to claim 19, characterized in that, The method for obtaining the initial information matching model further includes: during the training process of the initial pre-trained model: based on the first multimodal data features and the first text data features, obtaining the matching result of the first multimodal sample pair, wherein the matching result is used to describe the degree of matching between the data content information and the first data text information; and updating the model parameters of the initial pre-trained model based on the matching result of the first multimodal sample pair.

21. The apparatus according to claim 13, characterized in that, The multimodal sample pairs specifically include video content information and video text information. The video content information includes text data, image data, and audio data. The multimodal data features specifically include text features, image features, and audio features.

22. The apparatus according to claim 13, characterized in that, The initial pre-trained model is a text modality model that has been trained. The text modality model is obtained after training based on text content information and text information, and the text modality model is used to calculate the probability that the text content information and text information belong to the same text data information.

23. The apparatus according to claim 13, characterized in that, The method for obtaining the information matching model includes: acquiring training samples of each data information and training samples of event information that match each training sample of the data information; obtaining data information features corresponding to each training sample of the data information based on each training sample of the data information, and obtaining event information features corresponding to each training sample of the event information based on each training sample of the event information; obtaining the predicted matching degree between each data information feature and each event information feature based on each data information feature and each event information feature, and fine-tuning the initial information matching model based on each actual matching degree and each predicted matching degree to obtain the information matching model.

24. The apparatus according to claim 23, characterized in that, The training samples are video data training samples, which consist of text data, image data, and audio data. The image data consists of multiple image frames, and the audio data consists of multiple audio frames. The method for obtaining the information matching model further includes: performing text segmentation on the text data in each of the video data training samples to obtain text sequences corresponding to the text data in each of the video data training samples, and generating text features corresponding to the text data in each of the video data training samples based on the text sequences, wherein the text sequences include multiple text tokens; performing image recognition processing on the image data in each of the video data training samples to obtain image features corresponding to the image data in each of the video data training samples; performing audio recognition processing on the audio data in each of the video data training samples to obtain audio features corresponding to the audio data in each of the video data training samples; wherein the data features include the text features, the image features, and the audio features.

25. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

26. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images