Data processing method, electronic device, program product, and medium

Abstract

The embodiments of the present application disclose a data processing method, an electronic device, a program product, and a medium, which can be applied to the field of data processing technology. The method includes: obtaining a sample data set; calling a prediction model to generate text features for each text data and image features for each image data; respectively generating a first feature difference for each first feature pair in a plurality of first feature pairs; respectively generating a second feature difference for each second feature pair in a plurality of second feature pairs; based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair, the model parameters of the prediction model are corrected to obtain a trained prediction model. The use of the embodiments of the present application helps to improve the accuracy of the prediction of the events indicated by the text data. The embodiments of the present application can also be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, electronic equipment, program product, and medium. Background Art

[0002] At present, natural language processing technology (NLP) has been widely used in various fields. In a subtask of natural language processing technology, the event indicated by the text data can be predicted based on the text data. At present, predicting the event indicated by the text data based on the text data usually involves training an event prediction model based on the text data, and then predicting the event indicated by the input text data based on the trained event prediction model. In practice, the inventors found that text data is greatly affected by writing habits. When using a single text data to train an event prediction model, the accuracy of the trained event information extraction model in predicting the event indicated by the input text data is low. Summary of the Invention

[0003] Embodiments of the present application provide a data processing method, electronic device, program product, and medium, which help improve the accuracy of prediction of events indicated by text data.

[0004] In one aspect, an embodiment of the present application discloses a data processing method, the method comprising:

[0005] Obtain a sample data set; the sample data in the sample data set includes N text data and M image data, any text data and any image data has an event label, and N and M are positive integers;

[0006] Call the prediction model to generate text features for each text data and image features for each image data;

[0007] Generating first feature differences between text features and image features included in each of the plurality of first feature pairs; wherein text data to which the text features included in any first feature pair belong and image data to which the image features included in the text features belong have the same event label;

[0008] Generating second feature differences between text features and image features included in each of the plurality of second feature pairs; wherein text data to which the text features included in any second feature pair belong and image data to which the image features included in the second feature pair belong have different event labels;

[0009] The model parameters of the prediction model are corrected based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model; the trained prediction model is used to predict the event indicated by the input text data based on the input text data.

[0010] Optionally, the method further includes:

[0011] Combining the plurality of first feature pairs and the plurality of second feature pairs to obtain a first combined feature pair and a second combined feature pair; the first feature pair and the second feature pair in the first combined feature pair contain the same text feature, and the first feature pair and the second feature pair in the second combined feature pair contain the same image feature;

[0012] Generate a first prediction loss value of the prediction model for the sample feature according to a first feature difference corresponding to the first feature pair and a second feature difference corresponding to the second feature pair in the first combined feature pair;

[0013] generating a second prediction loss value of the prediction model for the sample feature according to a first feature difference corresponding to the first feature pair and a second feature difference corresponding to the second feature pair in the second combined feature pair;

[0014] A first characteristic prediction deviation of the prediction model is determined according to the first prediction loss value and the second prediction loss value, and a model parameter of the prediction model is corrected according to the first characteristic prediction deviation to obtain a trained prediction model.

[0015] Optionally, the method further includes:

[0016] Calling the prediction model to predict the events indicated by the N text data respectively, and generating event prediction deviations based on the events indicated by the predicted N text data and the event labels carried by the N text data respectively;

[0017] Based on the first feature prediction deviation and the event prediction deviation, the model parameters of the prediction model are corrected to obtain a trained prediction model.

[0018] In one aspect, an embodiment of the present application discloses a data processing device, comprising:

[0019] An acquisition unit is used to acquire a sample data set; the sample data in the sample data set includes N text data and M image data, any text data and any image data has an event label, and N and M are positive integers;

[0020] A processing unit, configured to call a prediction model to generate text features for each text data and image features for each image data;

[0021] The processing unit is further configured to respectively generate a first feature difference between a text feature and an image feature included in each of the plurality of first feature pairs; wherein the text data to which the text feature included in any first feature pair belongs and the image data to which the image feature included in the text feature pair belongs have the same event label;

[0022] The processing unit is further configured to respectively generate a second feature difference between the text feature and the image feature included in each of the plurality of second feature pairs; wherein the text data to which the text feature included in any second feature pair belongs and the image data to which the image feature included in the text feature pair belongs have different event labels;

[0023] The processing unit is also used to correct the model parameters of the prediction model based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model; the trained prediction model is used to predict the event indicated by the input text data based on the input text data.

[0024] In one aspect, an embodiment of the present application provides an electronic device, the electronic device including a processor and a memory, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to perform the following steps:

[0025] Obtain a sample data set; the sample data in the sample data set includes N text data and M image data, any text data and any image data has an event label, and N and M are positive integers;

[0026] Call the prediction model to generate text features for each text data and image features for each image data;

[0027] Generating first feature differences between text features and image features included in each of the plurality of first feature pairs; wherein text data to which the text features included in any first feature pair belong and image data to which the image features included in the text features belong have the same event label;

[0028] Generating second feature differences between text features and image features included in each of the plurality of second feature pairs; wherein text data to which the text features included in any second feature pair belong and image data to which the image features included in the second feature pair belong have different event labels;

[0029] The model parameters of the prediction model are corrected based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model; the trained prediction model is used to predict the event indicated by the input text data based on the input text data.

[0030] In one aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a processor, the computer program instructions are used to perform the following steps:

[0031] Obtain a sample data set; the sample data in the sample data set includes N text data and M image data, any text data and any image data has an event label, and N and M are positive integers;

[0032] Call the prediction model to generate text features for each text data and image features for each image data;

[0033] Generating first feature differences between text features and image features included in each of the plurality of first feature pairs; wherein text data to which the text features included in any first feature pair belong and image data to which the image features included in the text features belong have the same event label;

[0034] Generating second feature differences between text features and image features included in each of the plurality of second feature pairs; wherein text data to which the text features included in any second feature pair belong and image data to which the image features included in the second feature pair belong have different event labels;

[0035] The model parameters of the prediction model are corrected based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model; the trained prediction model is used to predict the event indicated by the input text data based on the input text data.

[0036] In one aspect, embodiments of the present application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in various optional embodiments, including the aforementioned aspect.

[0037] The embodiment of the present application proposes a data processing solution that can call a prediction model to generate text features for each text data and image features for each image data, and then based on the first feature differences corresponding to multiple first feature pairs in which the text data to which the text features belong and the image data to which the image features belong have the same event labels, and the second feature differences corresponding to second feature pairs in which the text data to which the text features belong and the image data to which the image features belong have different event labels, the model parameters of the prediction model are corrected to obtain a trained prediction model. In this way, the feature differences between the features corresponding to text data and image data with the same event can be compared with the feature differences between the features corresponding to text data and image data with different events. By adding image data to train the prediction model, the influence of writing habits on the text features corresponding to the text data is reduced, so that the prediction model can generate text features that predict events more accurately, which helps to improve the accuracy of the prediction of the events indicated by the text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0040] Figure 2 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of the results of a trigger word extractor provided by an embodiment of the present application;

[0042] Figure 4 Schematic diagram of a training framework of a prediction model provided in an embodiment of the present application;

[0043] Figure 5 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0044] Figure 6 This is a schematic diagram of the results of a trigger word extractor provided by an embodiment of the present application;

[0045] Figure 7 Schematic diagram of a training framework of a prediction model provided in an embodiment of the present application;

[0046] Figure 8This is a flow chart of an application scenario provided by an embodiment of the present application;

[0047] Figure 9 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0048] Figure 10 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0050] The embodiment of the present application proposes a data processing solution that can call a prediction model to generate text features for each text data and image features for each image data, and then based on the first feature differences corresponding to multiple first feature pairs in which the text data to which the text features belong and the image data to which the image features belong have the same event labels, and the second feature differences corresponding to second feature pairs in which the text data to which the text features belong and the image data to which the image features belong have different event labels, the model parameters of the prediction model are corrected to obtain a trained prediction model. In this way, the feature differences between the features corresponding to text data and image data with the same event can be compared with the feature differences between the features corresponding to text data and image data with different events. By adding image data to train the prediction model, the influence of writing habits on the text features corresponding to the text data is reduced, so that the prediction model can generate text features that predict events more accurately, which helps to improve the accuracy of the prediction of the events indicated by the text data.

[0051] It should be noted that this application can display a prompt interface or pop-up window before collecting the user's relevant data (such as the above-mentioned text data and image data) and during the process of collecting the user's relevant data. The prompt interface or pop-up window is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining the user's relevant data after obtaining the user's confirmation operation on the prompt interface or pop-up window. Otherwise (that is, when the user's confirmation operation on the prompt interface or pop-up window is not obtained), the relevant steps of obtaining the user's relevant data are terminated, that is, the user's relevant data is not obtained. In other words, all user data collected by this application are collected with the user's consent and authorization, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0052] The technical solution of the present application can be applied to electronic devices. The electronic device can be a terminal, a server, or other devices for data processing, which is not limited in the present application. Optional. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc.

[0053] In one possible implementation, the embodiments of the present application can be used in the field of artificial intelligence. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to have the capabilities of perception, reasoning, and decision-making. Specifically, it can be applied to the field of natural language processing technology. Natural language processing (NLP) is an important area in both computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. Natural language processing is a discipline that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language, the language people use in daily life, and is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robotic question answering, knowledge graphs, and other technologies.

[0054] In one possible implementation, the embodiments of the present application can be applied to the field of blockchain technology, such as events indicated by extracted text data and associated information of events can be stored based on blockchain nodes. Blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0055] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operation management. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between the user's real identity and the blockchain address (authority management), etc., and under authorization, it supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records valid requests to storage after consensus is reached. For a new business request, the basic service first adapts the interface for parsing and authentication (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management). The smart contract module is responsible for the registration, issuance, triggering and execution of contracts. Developers can define the contract logic in a programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, the contract logic is triggered by calling keys or other events to trigger execution. The contract logic is completed, and the contract upgrade and cancellation functions are also provided. The operation management module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and real-time status visualization output of the product during the product release process, such as alarms, network management, and node device health status.

[0056] The platform's product service layer provides the basic capabilities and implementation framework for typical applications. Developers can build on these basic capabilities, overlay business features, and complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for business participants to use.

[0057] It is understood that the above scenarios are merely examples and do not limit the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, those skilled in the art will appreciate that with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application will also be applicable to similar technical problems.

[0058] Based on the above description, the present application embodiment proposes a data processing method. Figure 1 , Figure 1 1 is a flow chart of a data processing method provided in an embodiment of the present application. The method can be executed by the above-mentioned electronic device. The data processing method can include the following steps.

[0059] S101: Obtain a sample data set.

[0060] The sample data in the sample dataset includes N text data and M image data, each of which has an event label, and N and M are positive integers. The sample dataset is the dataset used to train the prediction model. M and N can be the same or different, and are not limited here.

[0061] The event labels of the sample data in the sample data set are annotated based on the event set, which contains multiple events. That is to say, the event labels of the sample data are annotated based on the events in the event set of the sample data. The events included in the event set can be some preset events that need to be predicted, also known as event types. For example, the events in the event set can be 8 event types and 33 subtypes defined by ACE2005 (a method of defining event types). The events in the event set can be defined by themselves according to the field in which the prediction model needs to be trained. For example, if the prediction model to be trained is applied to the medical field, some events related to disease types, causes of symptoms, etc. can be preset, such as events can be "jaundice", "water accumulation", "fever", etc., which are not limited here.

[0062] In one possible implementation, each event in the event set has a corresponding event number. Sample data can then be labeled based on the event tag associated with the event set. For example, if the event "jaundice" in the event set has an event number of "1," then if sample data contains the event "jaundice," the sample data will also have an event tag of "1."

[0063] In one possible embodiment, to obtain a sample data set, an event sample set can be first obtained, and then a sample data set is generated based on the event sample set, where each event sample in the event sample set includes text data and image data. The event sample can be text data and image data describing the same event information. For example, the event sample can be an inspection report containing text data and image data, which can include text data describing symptoms, the causes of symptoms, and other information, as well as corresponding image data showing information such as events (such as symptoms), such as electrocardiograms, magnetic resonance imaging, etc. The text data and image data in the event sample indicated by the inspection report are used to express the event information of the event sample. Then, based on the events present in the text data of the event sample, event tags of the text data and image data in the event sample can be determined, and the text data and image data with event tags can be determined as sample data to obtain a sample data set. It is understood that the collection of the above-mentioned inspection report is carried out with the consent and authorization of the user, and the collection, use and processing of relevant user data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0064] For example, in an event example, the text data is "Two months ago, Xiao Ming was injured playing basketball and was found to have effusion in his knee," and the image data is an MRI image of the knee. If the event "effusion" exists in the event set, then the event "effusion" can also exist in the text data. The text data in the event example is annotated with the event label corresponding to the event "effusion," and the image data in the event example is annotated with the event label corresponding to the event "effusion." This results in one text data set and one image data set in the sample dataset. Similarly, multiple text data sets and image data sets can be obtained based on multiple event examples as sample data in the sample dataset, each of which has a corresponding event label.

[0065] S102: Calling a prediction model to generate text features for each text data and image features for each image data.

[0066] The text feature may be a feature used to represent the semantics of text data, and the image feature may be a feature used to represent the content in image data.

[0067] In a possible embodiment, the prediction model may include a text encoder, and the text encoder in the prediction model is called to generate text features for each text data, and the text features of each text data can be generated based on the text encoder in the prediction model. The text encoder can be a pre-trained BERT (language representation model) model, and the text features of the text data can be obtained based on the pre-trained BERT (language representation model). In a possible embodiment, the sequence representation of the text data is directly obtained based on the BERT model. In order to facilitate the subsequent comparative learning based on image features and text features, it is necessary to make the dimensions of the text features and image features the same, so that the sequence representation directly obtained based on the BERT model can be average pooled to obtain the text features of the text data.

[0068] In one possible implementation, the prediction model may include an image encoder, which then calls the image encoder in the prediction model to generate image features for each image data. The image features for each image data can be generated based on the image encoder in the prediction model. The image encoder may include a Resnet50 (a neural network model) model for generating overall features of image data, such as Resnet50 (a neural network model), and a model for performing entity recognition and entity region recognition in images, such as Faster-RCNN (a neural network model). An entity in an image can also be referred to as an objectively existing and distinguishable object in the image, such as a skull in an MRI image of a skull, which can be considered an entity. The Resnet50 model may be pre-trained on the 1k ImageNet (an image dataset) dataset and can extract overall features of image data; the Faster-RCNN model may be pre-trained on the Visual Genome (an image dataset) dataset and can extract entity features and entity location features of image data. The entity features, entity location features, and overall features of the image data can then be combined to obtain the image features of the aforementioned image data. In one possible implementation, when generating image features of image data, existing open source tools can be used to perform fine-grained information extraction on the image. For example, by performing scene recognition tasks (events are strongly associated with scenes) on the graph modality, the model can collect more information about the event. In addition, by using graph-based semantic tools to semantically annotate the information obtained from scene recognition, while collecting event-related information, semantic information can also be obtained, thereby helping the prediction model to inject image modality information, thereby improving the accuracy of the trained prediction model in recognizing events.

[0069] S103: Generate first feature differences between text features and image features included in each of the plurality of first feature pairs.

[0070] The text data to which the text feature contained in any first feature pair belongs and the image data to which the image feature contained in any first feature pair belongs have the same event label.

[0071] In one possible implementation, any text data may have at least one event tag, and any image data may have at least one event tag, and having the same event tags means that at least one event tag of the text data and at least one event tag of the image data are the same.

[0072] For example, the event labels of a text data S1 include "1" and "2", and the event labels of an image data V1 include "1" and "2". Obviously, the event labels of the text data S1 and the image data V1 are the same, so the text features of the text data S1 and the image features of the image data V1 can be combined into a first feature pair.

[0073] The first feature difference is used to characterize the difference between the text feature and the image feature contained in the first feature pair. The first feature difference can be a feature distance or feature similarity between the text feature and the image feature. Among them, the feature distance between the text feature and the image feature can be Euclidean distance, etc., and the feature similarity between the text feature and the image feature can be cosine similarity, etc. If the first feature difference is characterized by feature distance, then the larger the first feature difference (i.e., feature distance), the larger the difference between the text feature and the image feature, and the smaller the first feature difference (i.e., feature distance), the smaller the difference between the text feature and the image feature. If the first feature difference is characterized by feature similarity, then the larger the first feature difference (i.e., feature similarity), the smaller the difference between the text feature and the image feature, and the smaller the first feature difference (i.e., feature similarity), the smaller the difference between the text feature and the image feature.

[0074] S104: Generate second feature differences between the text features and the image features included in each of the plurality of second feature pairs.

[0075] The text data to which the text feature included in any second feature pair belongs and the image data to which the image feature included in any second feature pair belong have different event labels.

[0076] In a possible implementation manner, having different event tags means that at least one event tag of the text data and at least one event tag of the image data are different from each other.

[0077] For example, the event labels of a text data S2 include "1" and "2", and the event labels of an image data V2 include "3" and "4". Obviously, the event labels of the text data S2 are different from those of the image data V2, that is, the text data S2 and the image data V2 have different event labels. In this case, the text features of the text data S2 and the image features of the image data V2 can be combined into a second feature pair. For another example, the event labels of a text data S3 include "1" and "2", and the event labels of the image data V3 include "1". Obviously, the text data S3 has one more event label "2" than the image data V3, that is, the text data S3 and the image data V3 have different event labels. In this case, the text features of the text data S3 and the image features of the image data V3 can be combined into a second feature pair.

[0078] The second feature difference is used to characterize the difference between the text features and the image features contained in the second feature pair. Similar to the first feature difference mentioned above, the second feature difference can be a feature distance or feature similarity between the text features and the image features. Among them, the feature distance between the text features and the image features can be Euclidean distance, etc., and the feature similarity between the text features and the image features can be cosine similarity, etc. If the second feature difference is characterized by feature distance, then the larger the second feature difference (i.e., feature distance), the larger the difference between the event features and the image features of the second feature pair, and the smaller the second feature difference (i.e., feature distance), the smaller the difference between the event features and the image features of the second feature pair. If the second feature difference is characterized by feature similarity, then the larger the second feature difference (i.e., feature similarity), the smaller the difference between the event features and the image features of the second feature pair, and the smaller the second feature difference (i.e., feature similarity), the smaller the difference between the event features and the image features of the second feature pair.

[0079] S105 . Modify the model parameters of the prediction model based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model.

[0080] The trained prediction model is used to predict the event indicated by the input text data based on the input text data.

[0081] It can be understood that during the training process, the model parameters of the prediction model are corrected based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair. It is necessary to gradually reduce the difference between the text features and the image features corresponding to each first feature pair (i.e., the feature distance decreases and the feature similarity increases), and gradually increase the difference between the text features and the image features corresponding to each second feature pair (i.e., the feature distance increases and the feature similarity decreases). This enables the text encoder in the trained prediction model to generate text features that can better represent the events in the text data. This is because, under normal circumstances, the style of the image data in an event sample is relatively fixed and will not be affected by writing habits like text data, and the embodiment of event information is more intuitive and comprehensive. For example, when the event sample is an examination report, the image is usually an electrocardiogram, magnetic resonance imaging, etc. generated by a machine, which will not change according to the change of the doctor and can also represent the examination report information in the examination report. Therefore, by introducing the feature difference between the image data and the text data for comparative learning, the accuracy of feature extraction of the text data can be improved, thereby preventing the uncertainty caused by the change of the description style of the text data from affecting the prediction effect of the event, and improving the accuracy of event prediction for the text data.

[0082] In one possible implementation, calling a trained prediction model to predict the event indicated by the input text data may specifically include the following steps: ① Obtaining target text data. ② Calling a trained prediction model to generate text features of the target text data. ③ Calling a trained prediction model to predict the event indicated by the target text data based on the text features of the target text data. The target text data refers to the text data for which event prediction is required. The text features of the target text data can be determined based on a text encoder in a trained prediction model. After comparative learning of the above-mentioned various text data and image data, the text encoder can better characterize events in the text, and then the event indicated by the target text data can be determined based on the text features.

[0083] The embodiment of the present application proposes a data processing solution that can call a prediction model to generate text features for each text data and image features for each image data, and then based on the first feature differences corresponding to multiple first feature pairs in which the text data to which the text features belong and the image data to which the image features belong have the same event labels, and the second feature differences corresponding to second feature pairs in which the text data to which the text features belong and the image data to which the image features belong have different event labels, the model parameters of the prediction model are corrected to obtain a trained prediction model. In this way, the feature differences between the features corresponding to text data and image data with the same event can be compared with the feature differences between the features corresponding to text data and image data with different events. By adding image data to train the prediction model, the influence of writing habits on the text features corresponding to the text data is reduced, so that the prediction model can generate text features that predict events more accurately, which helps to improve the accuracy of the prediction of the events indicated by the text data.

[0084] See Figure 2 , Figure 2 1 is a flow chart of a data processing method provided in an embodiment of the present application. The method can be executed by the above-mentioned electronic device. The data processing method can include the following steps.

[0085] S201: Obtain a sample data set.

[0086] S202: Call the prediction model to generate text features for each text data and image features for each image data.

[0087] Among them, steps S201-S202 can refer to the relevant description of the above steps S101-S102, and will not be repeated here.

[0088] S203 : Generate first feature differences between the text features and the image features included in each of the plurality of first feature pairs.

[0089] In a possible embodiment, as described above, the first feature difference can be characterized by feature similarity, and step S203 may include the following steps: ① Generate the feature similarity between the text features and image features contained in each first feature pair respectively. The feature similarity can be the cosine similarity between the text features and the image features, etc. ② Determine the feature similarity between the text features and the image features contained in each first feature pair as the first feature difference corresponding to each first feature pair. In this way, the feature difference of the first feature pair can be characterized by the feature similarity between the text features and the image features contained in the first feature pair.

[0090] In a possible implementation, as described above, the first feature difference can be characterized by a feature distance, and step S203 may include the following steps: ① Generate the feature distance between the text feature and the image feature contained in each first feature pair respectively. The feature distance may be the Euclidean distance between the text feature and the image feature, etc. ② Determine the feature distance between the text feature and the image feature contained in each first feature pair as the first feature difference corresponding to each first feature pair. Thus, the feature difference of the first feature pair can be characterized by the feature distance between the text feature and the image feature contained in the first feature pair.

[0091] S204: Generate second feature differences between the text features and the image features included in each of the plurality of second feature pairs.

[0092] In a possible embodiment, as described above, the second feature difference can be characterized by feature similarity, and step S203 may include the following steps: ① Generate the feature similarity between the text feature and the image feature contained in each second feature pair respectively. The feature similarity can be the cosine similarity between the text feature and the image feature, etc. ② Determine the feature similarity between the text feature and the image feature contained in each second feature pair as the second feature difference corresponding to each second feature pair. In this way, the feature difference of the second feature pair can be characterized by the feature similarity between the text feature and the image feature contained in the second feature pair.

[0093] In a possible implementation, as described above, the second feature difference can be characterized by a feature distance, and step S203 may include the following steps: ① Generate the feature distance between the text feature and the image feature contained in each second feature pair respectively. The feature distance may be the Euclidean distance between the text feature and the image feature, etc. ② Determine the feature distance between the text feature and the image feature contained in each second feature pair as the second feature difference corresponding to each second feature pair. Thus, the feature difference of the second feature pair can be characterized by the feature distance between the text feature and the image feature contained in the second feature pair.

[0094] It can be understood that the first feature pair and the second feature pair are determined by whether the text data to which the text feature belongs and the image data to which the image feature belongs have the same event label, that is, the event label is used to characterize whether the text data to which the text feature belongs and the image data to which the image feature belongs have the same event content. In one possible embodiment, the event content can also be characterized by a combination of trigger words and events. An event may correspond to a variety of different trigger words, and the trigger words may also have different word forms. Therefore, existing open source tools can be used to restore the word form of the trigger word, and then spliced ​​with the event indicated by the event label to represent a specific event content, and then the first feature pair and the second feature pair are determined by comparing whether the event content is the same.

[0095] S205 , combining multiple first feature pairs and multiple second feature pairs to obtain first combined feature pairs and second combined feature pairs.

[0096] The first and second feature pairs in the first combined feature pair contain the same text features, and the first and second feature pairs in the second combined feature pair contain the same image features. There can be multiple first combined feature pairs and multiple second combined feature pairs.

[0097] For example, multiple first feature pairs may include (s1, v1), (s2, v2), (s3, v3), (s4, v4), etc., and multiple second feature pairs may include (s1, v2), (s2, v3), (s1, v4), (s4, v3), etc., where s1-s4 represent text features and v1-v4 represent image features. By combining the multiple first feature pairs and the multiple second feature pairs, multiple first combined feature pairs (s1, v1)-( s1,v2), (s1,v1)-(s1,v4), (s2,v2)-(s2,v3) and (s4,v4)-(s4,v3), etc., and multiple second combined feature pairs containing the same image features can be obtained, such as (s2,v2)-(s1,v2), (s3,v3)-(s2,v3), (s4,v4)-(s1,v4), (s3,v3)-(s4,v3), etc., thereby obtaining multiple first combined feature pairs and multiple second combined feature pairs.

[0098] It can be understood that the same text features included in the first combined feature pair can be called anchor sample features, the image features included in the first feature pair in the first combined feature pair can be called positive sample features of the anchor sample features, and the image features included in the second feature pair in the first combined feature pair can be called negative sample features of the anchor sample features. The same image features included in the second combined feature pair can also be called anchor sample features, the text features included in the first feature pair in the second combined feature pair can be called positive sample features of the anchor sample features, and the text features included in the second feature pair in the second combined feature pair can be called negative sample features of the anchor sample features. Furthermore, during the training process of the prediction model, it is necessary to gradually reduce the difference between the anchor sample features and the positive sample features corresponding to the anchor sample features, and gradually increase the difference between the anchor sample features and the negative sample features corresponding to the anchor sample features, thereby achieving comparative learning of text data and image data.

[0099] S206 : Generate a first prediction loss value of the prediction model for the sample feature based on the first feature difference corresponding to the first feature pair and the second feature difference corresponding to the second feature pair in the first combined feature pair.

[0100] The sample features refer to the features corresponding to the sample data, namely, the text features and image features. The first prediction loss value for the sample features can represent the difference between the first feature difference and the second feature difference corresponding to the first combined feature. During the training process of the prediction model, the first prediction loss value should be gradually reduced until convergence.

[0101] In a possible implementation, if both the first feature difference and the second feature difference are characterized by feature similarity, the first predicted loss value can be obtained by subtracting the first feature difference from the second feature difference corresponding to the first combined feature pair. If there can be multiple first combined feature pairs, the second feature difference corresponding to each first combined feature pair can be subtracted from the first feature difference to obtain the feature pair difference value corresponding to each first combined feature pair, and the expectation of the feature pair difference value corresponding to each first combined feature pair can be calculated to obtain the first predicted loss value. Calculating the expectation of the feature pair difference value corresponding to each first combined feature pair is also called calculating the mean of the feature pair difference values ​​corresponding to each combined feature pair.

[0102] In one possible embodiment, for any second feature pair, the event label of the text data corresponding to the second feature pair is different from the event label of the image data. Obviously, the smaller the proportion of the same event label in the union of the event labels of the text data and the event labels of the image data, the greater the degree of difference between the events described by the text data and the image data. Therefore, during the training process, the difference in features of the second feature pair corresponding to the text data and image data with a larger degree of event difference should be smaller than the difference in features of the second feature pair corresponding to the text data and image data with a smaller degree of event difference. Furthermore, when calculating the first prediction loss value, different weights can be assigned to the second feature differences corresponding to the second feature pairs in different first combination features to achieve the above purpose, that is, weighting the second feature differences according to the repetition of the event labels corresponding to the text features and image features in the second feature pair.

[0103] Specifically, step S206 may include the following steps: ① determining the text feature included in the second feature pair in the first combined feature pair as the first text feature, and determining the image feature included in the second feature pair in the first combined feature pair as the first image feature.

[0104] ② Calling the prediction model to generate a first label weight for the event label of the text data to which the first text feature belongs, and a second label weight for the event label of the image data to which the first image feature belongs, based on the first image feature. The first label weight refers to the weight corresponding to the event label of the text data to which the first text feature belongs, and the second label weight refers to the weight corresponding to the event label of the text data to which the first image feature belongs. It is understood that each of the at least one event label corresponding to the first text feature corresponds to a first label weight, and each of the at least one event label corresponding to the first image feature corresponds to a second label weight. To determine the first and second label weights, the weights of each event in the event set can be determined based on the first image feature. The first label weight is then determined based on the weight of the event indicated by the event label of the text data to which the first text feature belongs, and the second label weight is determined based on the weight of the event indicated by the event label of the image data to which the first image feature belongs. It is understood that the weights of each event in the event set are determined based on the first image feature because image data better expresses the salience of events. Salience, also known as statistical significance, indicates the ability to distinguish between groups, and in this case, the ability to distinguish between different events. For example, if the image data is an MRI image of a skull, it indicates that the event causing a head injury is more likely, i.e., more significant. In this case, the weight of each event in the event set can be measured based on the image data in the second feature pair. Specifically, the event weight of each event obtained based on the first image feature can be determined according to the following formula (i.e., Formula 1).

[0105] φ(v o )=Wv o +b Formula 1

[0106] Among them, φ(v o ) represents the event weight of each event obtained based on the first image feature, φ(v o ) corresponds to an event in the event set, and each row of data is equivalent to the event weight of the corresponding event. W can be a parameter matrix learned in the process of training the prediction model, and the dimension of W is |E a |×d, where E a Represents the above event set, d is a constant with the same number of columns as the image feature, that is, each row of data in W corresponds to an event in the event set. o represents the first image feature, that is, the image feature in the second feature pair, and b represents a constant.

[0107] After obtaining the event weights of each event based on the first image feature, the first label weight of at least one event label corresponding to the first text feature and the second label weight of at least one event label corresponding to the first image feature can be calculated based on the following formula (i.e., Formula 2).

[0108]

[0109] Among them, the represents the label weight of event label k. φ(v o ) k represents the weight of the event indicated by the event label k, exp( ) refers to the exponential function with the natural constant e as the base, exp(φ(v o ) k ) represents a value with base e and the weight of the event indicated by event label k as the exponent. E represents the set of events indicated by each event label of the image data to which the first image feature belongs. The value of l can be 1 to E, that is, l can be the event indicated by each event label corresponding to the first image feature. It can be the sum of the values ​​with base e and the weight of the event indicated by each event label corresponding to the first image feature as the exponent. It can be understood that if the first label weight corresponding to the first text feature is to be calculated, the weight of the event indicated by each event label of the first text feature can be used as φ(v o ) k To calculate the second label weight corresponding to the first image feature, the weight of the event indicated by each event label of the first image feature can be used as φ(v o ) kPerform calculations.

[0110] ③ Generate a first difference weight for the second feature difference corresponding to the first combined feature pair based on the first label weight and the second label weight. This first difference weight indicates the weight assigned to the second feature difference corresponding to the first combined feature pair. When determining the first difference weight, the second feature difference can be weighted by comparing the difference in event labels between the negative feature and the anchor feature in the first combined feature pair. This can be calculated using the following formula (i.e., Formula 3).

[0111]

[0112] Among them, μ T (i, j) represents the first difference weight corresponding to the second feature difference between the text feature with sample number i and the image feature with sample number j. represents the weight of any first label corresponding to the first text feature i, E i represents a set of events indicated by at least one event tag in the text data to which the first text feature belongs, E j represents a set of events indicated by at least one event label of the image data to which the first image feature belongs, then E i \E j Indicates E i Remove E j With E i Events other than the same event, Indicates the event labels corresponding to the first text feature except E j With E i The sum of the first label weights corresponding to the event labels of events other than the same event; E j \E i Indicates E j Except E i With E j Events other than the same event, Indicates the event labels corresponding to the first image feature except E i With E j The sum of the second label weights corresponding to the event labels of events other than the same event can be obtained by, and To express the difference in event labels between the negative sample features and the anchor sample features, δ is a preset constant. Represents the sum of the first label weights corresponding to each event label corresponding to the first text feature, Represents the sum of the second label weights corresponding to each event label corresponding to the first image feature.

[0113] ④ Generate a first prediction loss value based on the first difference weight, the first feature difference corresponding to the first feature pair in the first combined feature pair, and the second feature difference corresponding to the second feature pair in the first combined feature pair. If both the first feature difference and the second feature difference are characterized by feature similarity, the first prediction loss value can be obtained by multiplying the first difference weight by the second feature difference corresponding to the first combined feature pair, and then subtracting the first feature difference corresponding to the first combined feature pair.

[0114] S207 : Generate a second prediction loss value of the prediction model for the sample feature based on the first feature difference corresponding to the first feature pair and the second feature difference corresponding to the second feature pair in the second combined feature pair.

[0115] The second prediction loss value for the sample feature may represent the difference between the first feature difference and the second feature difference corresponding to the second combined feature pair. During the training of the prediction model, the second prediction loss value should be gradually reduced until convergence.

[0116] The method for generating the second predicted loss value may refer to the above description of generating the first predicted loss value.

[0117] In one possible implementation, if both the first feature difference and the second feature difference are characterized by feature similarity, the second predicted loss value can be obtained by subtracting the first feature difference from the second feature difference corresponding to the second combined feature pair. If there can be multiple second combined feature pairs, the first feature difference can be subtracted from the second feature difference corresponding to each second combined feature pair to obtain a feature pair difference value corresponding to each second combined feature pair, and the expectation of the feature pair difference value corresponding to each second combined feature pair can be calculated to obtain the second predicted loss value.

[0118] In one possible embodiment, when calculating the second prediction loss value, different weights can be assigned to the second feature differences corresponding to the second feature pairs in different second combination features. Specifically, the second feature differences are weighted based on the repetition of the event labels corresponding to the text features and image features in the second feature pairs, such that the feature differences of the second feature pairs corresponding to text data and image data with a greater degree of event difference are smaller than the feature differences of the second feature pairs corresponding to text data and image data with a smaller degree of event difference. Specifically, the method for determining the second prediction loss value can refer to the above description of determining the first prediction loss value based on the first difference weight, that is, it can specifically include the following steps: ① Determining the text features included in the second feature pair in the second combination feature pair as the second text feature, and determining the image features included in the second feature pair in the first combination feature pair as the second image feature. ② Invoking the prediction model to generate a third label weight for the event label of the text data to which the second text feature belongs, and generating a fourth label weight for the event label of the image data to which the second image feature belongs based on the second image feature. ③ Generating a second difference weight for the second feature differences corresponding to the second combination feature pair based on the third label weight and the fourth label weight. ④ Generate a second prediction loss value based on the second difference weight, the first feature difference corresponding to the first feature pair in the second combined feature pair, and the second feature difference corresponding to the second feature pair in the second combined feature pair. The method for determining the third and fourth label weights can refer to the description of the first and second label weights above, and the method for determining the second difference weight can refer to the description of the first difference weight above, which will not be repeated here.

[0119] S208. Determine a first characteristic prediction deviation of the prediction model according to the first prediction loss value and the second prediction loss value, and correct the model parameters of the prediction model according to the first characteristic prediction deviation to obtain a trained prediction model.

[0120] The first feature prediction deviation may be a deviation value used to correct model parameters of the prediction model based on the comparison between the text feature and the image feature. During the training of the prediction model, the first feature prediction deviation should be gradually reduced until convergence.

[0121] In a possible implementation, the first characteristic prediction deviation may be the sum of the first predicted loss value and the second predicted loss value, which may be determined according to the following formula (Formula 4).

[0122]

[0123] in, is the first feature prediction deviation determined based on each text feature and image feature, s represents the text feature, and v represents the image feature. S(s,v′) represents the feature similarity between the text feature s and the image feature v′ in the second feature pair of the first combined feature pair, and S(s,v) represents the feature similarity between the text feature s and the image feature v in the first feature pair of the first combined feature pair, μ T (i, j) represents the first difference weight mentioned above, that is, the first difference weight corresponding to the second feature difference between the text feature s with sample number i and the image feature v′ with sample number j, i represents the sample number of the text feature s, j represents the sample number of the image feature v′, E v′ [ ] + represents the expectation of the value in [ ], then Represents the first prediction loss value mentioned above. Where ∈ represents the boundary value, which can make the text features and image features in the first feature pair not completely the same, and the text features and image features in the second feature pair not completely different during the training of the prediction model, and may also make the first feature prediction deviation converge. S(s′,v) represents the feature similarity between the text feature s′ and the image feature v in the second feature pair of the second combined feature pair, S(s,v) represents the feature similarity between the text feature s and the image feature v in the first feature pair of the second combined feature pair, μ T (k,l)S(s′,v) represents the second difference weight mentioned above, that is, the second difference weight corresponding to the second feature difference between the text feature s′ with sample number k and the image feature v with sample number l, k represents the sample number of the text feature s′, l represents the sample number of the image feature v, then E s′ [μ T (k,l)S(s′,v)-S(s,v)+∈] + The first characteristic prediction deviation can be obtained based on the sum of the first prediction loss value and the second prediction loss value.

[0124] In a possible implementation, in the process of training the model parameters of the prediction model, in addition to correcting the model parameters based on the above-mentioned first feature prediction deviation, the model parameters can also be corrected by the events indicated by the predicted text data and the events indicated by the event labels, so that the trained prediction model can accurately predict the events indicated by the input text data. Specifically, correcting the model parameters of the prediction model according to the first feature prediction deviation to obtain a trained prediction model can specifically include the following steps: ① Calling the prediction model to predict the events indicated by N text data respectively, and generating event prediction deviations based on the events indicated by the predicted N text data and the event labels carried by the N text data. ② Based on the first feature prediction deviation and the event prediction deviation, correct the model parameters of the prediction model to obtain a trained prediction model.

[0125] The event prediction bias can be determined based on the difference between the events indicated by the N predicted text data and the event labels carried by the N text data. During training, the event prediction bias needs to be gradually reduced until convergence occurs, so that the trained prediction model can accurately determine the event indicated by the text data.

[0126] In a possible embodiment, the prediction model may include a trigger word extractor, and N text data respectively carry trigger word labels. Among them, the trigger word label is used to indicate which text in the text data triggers the corresponding event. Then, the text features corresponding to the text data can be obtained based on the text encoder of the prediction model, and the trigger word extractor is called to determine the probability that each token in the text data belongs to each event in the event set based on the text features. If the probability of an event corresponding to a token is greater than a threshold, it can be determined that the token is the text in the trigger word corresponding to the event (that is, the token can trigger the event), and then determine which events in the event set each token can trigger, so as to obtain the event prediction deviation based on the predicted triggered event, event label and trigger word label of each token, and then correct the model parameters based on the event prediction deviation. During the training process, the event triggered by each token should be gradually consistent with the trigger word label and event label, so that the trained prediction model can accurately determine the event indicated in the text data. If a segmentation word can trigger a corresponding event, then the segmentation word is the text in the trigger word, and consecutive segmentations that trigger the same event can form a corresponding trigger word. The one or more events triggered by the trigger word are the one or more events indicated by the text data. For example, for the text data "Two months ago, Xiao Ming was injured while playing basketball, and a checkup showed that he had effusion in his knee", the probability that the segmentation word "knee" corresponds to the event "effusion" is greater than the threshold, and the probability that the segmentation word "effusion" corresponds to the event "effusion" is greater than the threshold, then effusion in the knee can be the trigger word for the event "effusion", and the predicted events indicated by the text data include the event "effusion". It can be understood that the text features input to the trigger word extractor here can be the sequence representation directly obtained by the text encoder, and the above-mentioned average pooling processing does not need to be performed.

[0127] In a possible implementation, the trigger word extractor in the prediction model can use a sequence tagging method. Figure 3 , Figure 3 This is a schematic diagram of the result of a trigger word extractor provided by an embodiment of the present application, such as Figure 3 As shown, in the trigger word extractor, the following Figure 3 The text feature number shown in 301 is input into Figure 3 In the linear layer shown in 302, a low-dimensional feature with the dimension of the number of events in the event set is obtained. Then, a conditional random field (CRF) is used to perform a joint probability representation calculation on each segmentation to obtain the probability of each segmentation corresponding to each event. Then, based on the probability, the event corresponding to each segmentation and the trigger word of the event corresponding to each segmentation are determined (such as Figure 3), the event corresponding to the trigger word is the event indicated by the predicted text data (such as Figure 3 305 in the example). Thus, the model parameters of the prediction model can be optimized based on the event indicated by the text data and the event label predicted by the trigger word extractor, so that the trained prediction model can accurately predict the event indicated by the text data.

[0128] Here we take a prediction model as an example to describe the training process of the prediction model. Figure 4 , Figure 4 This is a schematic diagram of a training framework for a prediction model provided in an embodiment of the present application. First, a sample data set is obtained, i.e. Figure 4 Multiple text data shown in 401 and Figure 4 The multiple image data shown in 402 can then be based on a text encoder (such as Figure 4 403 in the figure) to obtain the text features of each text data, based on the image encoder (as shown in FIG. Figure 4 404 in the figure) to obtain the image features of each image data, so as to determine the first feature prediction deviation (as shown in FIG. Figure 4 405 in ). And call the trigger word extractor (as shown in Figure 4 406 in the example) determines the event indicated by the text data predicted by each text feature based on the text features of each text data (e.g. Figure 4 407 in ), thereby determining the event prediction deviation (as shown in FIG4 ) based on the event indicated by the predicted text data and the event label carried by the text data. Figure 4 408 in the flowchart). Then, based on the event prediction deviation and the first feature prediction deviation, the model parameters of the optimized prediction model are determined, such as the model parameters in the text encoder and the trigger word extractor. It is understandable that when correcting the model parameters based on the first feature prediction deviation, the difference in features between text data and image data with the same event label should be reduced, and the difference in features between text data and image data with different event labels should be increased. In this way, comparative learning can be performed based on the text data and image data, so that the text features can better represent the events, thereby improving the accuracy of the prediction of events in the text data.

[0129] In one possible implementation, based on the prediction model trained in the embodiments of the present application, when performing event prediction on input text data, the following steps may be included: Specifically, the following steps may be included: ① Obtaining target text data. ② Calling the text encoder in the trained prediction model to generate text features of the target text data. ③ Calling the trigger word extractor of the trained prediction model to predict the event and trigger word indicated by the target text data based on the text features of the target text data. The target text data refers to the text data for which event prediction is required. Calling the trigger word extractor of the trained prediction model to predict the event and trigger word indicated by the target text data based on the text features of the target text data. The probability of each event in the event set corresponding to each segmentation in the target text data can be determined based on the trigger word extractor. Events with probabilities greater than a threshold are determined as the events corresponding to the segmentation, thereby obtaining the event and trigger word indicated by the target text data. The text features of the target text data can be determined based on the text encoder in the trained prediction model. After comparative learning of the above-mentioned text data and image data, the text encoder can better represent events in the text, and the event indicated by the target text data can be determined based on the text features.

[0130] The embodiment of the present application proposes a data processing solution that can call a prediction model to generate text features for each text data and image features for each image data, and then based on the first feature differences corresponding to multiple first feature pairs in which the text data to which the text features belong and the image data to which the image features belong have the same event labels, and the second feature differences corresponding to second feature pairs in which the text data to which the text features belong and the image data to which the image features belong have different event labels, the model parameters of the prediction model are corrected to obtain a trained prediction model. In this way, the feature differences between the features corresponding to text data and image data with the same event can be compared with the feature differences between the features corresponding to text data and image data with different events. By adding image data to train the prediction model, the influence of writing habits on the text features corresponding to the text data is reduced, so that the prediction model can generate text features that predict events more accurately, which helps to improve the accuracy of the prediction of the events indicated by the text data.

[0131] See Figure 5 , Figure 5 1 is a flow chart of a data processing method provided in an embodiment of the present application. The method can be executed by the above-mentioned electronic device. The data processing method can include the following steps.

[0132] S501: Obtain a sample data set.

[0133] S502: Call the prediction model to generate text features for each text data and image features for each image data.

[0134] S503: Generate first feature differences between the text features and the image features included in each of the plurality of first feature pairs.

[0135] S504: Generate second feature differences between the text features and the image features included in each of the plurality of second feature pairs.

[0136] Steps S501 to S504 may refer to the relevant descriptions of the above steps S101 to S104 and are not repeated here.

[0137] S505 : Generate a first feature prediction deviation of the prediction model based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair.

[0138] The relevant description of step S505 can be found in Figure 2 The description of steps S205 to S208 in the embodiment is not repeated here.

[0139] S506: Call the information extractor to generate event features for each event in the event set.

[0140] The event feature may be a semantic feature of the text corresponding to the event. For example, the event feature of the event "jaundice" is used to represent the semantic feature of the text "jaundice".

[0141] It is understood that the prediction model may also include an information extractor, which can be used to extract event-related information from text data. Event-related information may include arguments and argument roles associated with the events. An argument refers to a participant in an event, and an argument role refers to the role the argument plays within the event. It is understood that each event has several corresponding argument roles. Therefore, when determining argument-related information, it is necessary to determine the arguments for each argument role corresponding to each event in the text data. For example, in the text data "Two months ago, Xiaoming was injured playing basketball, and an examination revealed effusion in the knee," there is the event "effusion." The argument roles for the event "effusion" may include argument roles such as cause and time. In this text data, it can be extracted that for the event "effusion," the argument corresponding to the argument role "cause" is "injured playing basketball," and the argument corresponding to the argument role "time" is "two months ago."

[0142] In one possible implementation, the information extractor of the prediction model may include an event encoder, which can then be called to generate event features corresponding to each event. The event encoder can be an embedding layer (an embedding coding network), and the event features corresponding to each event can be obtained based on the embedding layer.

[0143] S507: Generate a second feature prediction deviation of the information extractor based on the event feature of each event and the image feature of each image data.

[0144] The second prediction deviation may be a deviation value used to correct model parameters of the prediction model based on the comparison between the image features and the event features.

[0145] In a possible implementation, step S507 may include the following steps: ① Calling the information extractor to generate a third feature difference between the event features and image features contained in each of the multiple third feature pairs. The event to which the event features contained in any third feature pair belong is the same as the event indicated by the event label corresponding to the contained image feature, that is, the event indicated by the event label corresponding to the image feature in the third feature pair is the event to which the event features in the third feature pair belong. For example, the event to which event feature e1 belongs is event 1. If the event label corresponding to image feature v1 only includes the event label corresponding to event 1, then event feature e1 and image feature v1 can be combined into a third feature pair.

[0146] The third feature difference is used to characterize the difference between the event feature and the image feature contained in the third feature pair. Similar to the first feature difference mentioned above, the third feature difference can be a feature distance or feature similarity between the event feature and the image feature. Among them, the feature distance between the event feature and the image feature can be Euclidean distance, etc., and the feature similarity between the event feature and the image feature can be cosine similarity, etc. If the third feature difference is characterized by feature distance, the larger the third feature difference (i.e., feature distance), the larger the difference between the event feature and the image feature, and the smaller the third feature difference (i.e., feature distance), the smaller the difference between the event feature and the image feature. If the third feature difference is characterized by feature similarity, the larger the third feature difference (i.e., feature similarity), the smaller the difference between the event feature and the image feature, and the smaller the third feature difference (i.e., feature similarity), the smaller the difference between the event feature and the image feature.

[0147] ② Call the information extractor to generate the fourth feature difference between the event features and image features contained in each of the fourth feature pairs in the plurality of fourth feature pairs. The event to which the event feature contained in any fourth feature pair belongs is different from the event indicated by the event label corresponding to the contained image feature, that is, the event indicated by the event label corresponding to the image feature is not exactly the same as the event to which the event feature belongs. For example, the event to which event feature e1 belongs is event 1. If the event label corresponding to image feature v2 includes the event labels corresponding to event 1 and event 2, then event feature e1 and image feature v2 can be combined into a fourth feature pair; if the event label corresponding to image feature v3 includes the event labels corresponding to event 3 and event 4, then event feature e1 and image feature v3 can be combined into a fourth feature pair.

[0148] The fourth feature difference is used to characterize the difference between the event feature and the image feature included in the fourth feature pair. Similar to the third feature difference described above, the fourth feature difference can be a feature distance or feature similarity between the event feature and the image feature, which will not be described in detail here.

[0149] ③ Calling the information extractor to generate a second feature prediction deviation for the information extractor based on the third feature difference corresponding to each third feature pair and the fourth feature difference corresponding to each fourth feature pair. The second prediction deviation may be a deviation value used to correct model parameters of the prediction model based on the comparison of image features and event features. During the training process of the prediction model, the second feature prediction deviation should gradually decrease until convergence.

[0150] In one possible embodiment, generating a second feature prediction bias for an information extractor may include the following steps: ① combining multiple third feature pairs and multiple fourth feature pairs to obtain a third combined feature pair and a fourth combined feature pair. The third feature pair and the fourth feature pair in the third combined feature pair contain the same event features, and the third feature pair and the fourth feature pair in the second combined feature pair contain the same image features. It can be understood that the same event features contained in the third combined feature pair can be referred to as anchor sample features, the image features contained in the third feature pair in the third combined feature pair can be referred to as positive sample features of the anchor sample features, and the image features contained in the fourth feature pair in the third combined feature pair can be referred to as negative sample features of the anchor sample features. The same image features contained in the fourth combined feature pair can also be referred to as anchor sample features, the event features contained in the third feature pair in the fourth combined feature pair can be referred to as positive sample features of the anchor sample features, and the event features contained in the fourth feature pair in the fourth combined feature pair can be referred to as negative sample features of the anchor sample features. Furthermore, during the training process of the prediction model, it is necessary to gradually reduce the difference between the anchor sample features and the positive sample features corresponding to the anchor sample features, and gradually increase the difference between the anchor sample features and the negative sample features corresponding to the anchor sample features, thereby realizing comparative learning of event and image data.

[0151] ② Based on the third feature difference corresponding to the third feature pair in the third combination feature pair and the fourth feature difference corresponding to the fourth feature pair, generate the third prediction loss value of the prediction model for the sample feature. Among them, the third prediction loss value for the sample feature can represent the difference between the third feature difference and the fourth feature difference corresponding to the third combination feature. During the training process of the prediction model, the third prediction loss value should be gradually reduced. The calculation method for the third prediction loss value can refer to the relevant description of the first prediction loss value mentioned above. In a possible implementation, if the third feature difference and the fourth feature difference are both characterized by feature similarity, the third prediction loss value can be obtained by subtracting the third feature difference from the fourth feature difference corresponding to the third combination feature pair.

[0152] ③ Based on the third feature difference corresponding to the first feature pair in the fourth combination feature pair and the fourth feature difference corresponding to the fourth feature pair, generate the fourth prediction loss value of the prediction model for the sample feature. Among them, the fourth prediction loss value for the sample feature can represent the difference between the third feature difference and the fourth feature difference corresponding to the fourth combination feature. During the training process of the prediction model, the fourth prediction loss value should be gradually reduced. The calculation method for the fourth prediction loss value can refer to the relevant description of the third prediction loss value mentioned above. In a possible implementation, if the third feature difference and the fourth feature difference are both characterized by feature similarity, the fourth prediction loss value can be obtained by subtracting the third feature difference from the fourth feature difference corresponding to the fourth combination feature pair.

[0153] ④ Determine the second characteristic prediction deviation of the prediction model based on the third predicted loss value and the fourth predicted loss value. The second characteristic prediction deviation may be the sum of the third predicted loss value and the fourth predicted loss value. Specifically, this may be determined according to the following formula (Formula 5).

[0154]

[0155] Among them, among them, is the second feature prediction deviation determined based on each event feature and image feature, e represents the event feature, and v represents the image feature. S(e, v′) represents the feature similarity between the event feature e and the image feature v′ in the fourth feature pair of the third combined feature pair, and S(e, v) represents the feature similarity between the event feature e and the image feature v in the third feature pair of the third combined feature pair, μ E (e, j) represents the weight assigned to the fourth feature difference between the event feature e and the image feature v′ with sample number j in the third combined feature pair, e represents the event feature e, j represents the sample number of the image feature v′, E v ′[ ] + represents the expectation of the value in [ ], then E v′ [μ E (e,j)S(e,v′)-S(e,v)+∈] + Represents the third prediction loss value mentioned above. Where ∈ represents the boundary value, by which the event features and image features in the third feature pair do not have to be completely identical, and the event features and image features in the fourth feature pair do not have to be completely different, and the second feature prediction deviation may also converge. S(e′, v) represents the feature similarity between the event feature e′ and the image feature v in the fourth feature pair in the fourth combined feature pair, S(s, v) represents the feature similarity between the event feature s and the image feature v in the third feature pair in the fourth combined feature pair, μ E(e′, i) represents the weight assigned to the fourth feature difference between the event feature e′ and the image feature v with sample number i in the fourth combined feature pair, e′ represents the event feature e′, j represents the sample number of the image feature v, then E e′ [μ E (e′,i)S(e′,v)-S(e,v)+∈] + The fourth predicted loss value is represented by . Thus, the second characteristic prediction deviation can be obtained based on the sum of the third predicted loss value and the fourth predicted loss value.

[0156] In one possible embodiment, for any fourth feature pair, the event to which the event feature contained in the fourth feature pair belongs is different from the event indicated by the event label corresponding to the image feature contained in the fourth feature pair. Obviously, the smaller the proportion of the event to which the event feature belongs in the event indicated by at least one event label corresponding to the image feature, the greater the difference between the image feature and the event feature should be. Therefore, when calculating the second prediction loss value, different weights can be assigned to the fourth feature differences corresponding to different fourth feature pairs to achieve the above purpose, that is, the fourth feature differences are weighted according to the repetition of the event labels corresponding to the event features and image features in the fourth feature pair. Then, the third prediction loss value of the prediction model for the sample feature is generated based on the third feature difference corresponding to the third feature pair in the third combined feature pair and the fourth feature difference corresponding to the fourth feature pair. Specifically, the following steps may be included: ① Determining the event feature contained in the fourth feature pair in the third combined feature pair as the third event feature, and determining the image feature contained in the fourth feature pair in the third combined feature pair as the third image feature. ② Calling the prediction model to generate a fifth label weight for the event to which the third event feature belongs based on the third image feature, and generating a sixth label weight for the event label of the image data to which the third image feature belongs. Among them, the method for determining the fifth label weight and the sixth label weight can refer to the relevant description of determining the first label weight and the second label weight above, that is, referring to Formula 1 and Formula 2 to calculate the fifth label weight corresponding to the event feature and the sixth label weight of the event label corresponding to the image feature, which will not be repeated here. ③ Based on the fifth label weight and the sixth label weight, the third difference weight of the fourth feature difference corresponding to the third combination feature pair is generated. Among them, when determining the third difference weight, it can be determined according to the following formula.

[0157]

[0158] Among them, μ E (e k , j) represents the event feature e k The third difference weight corresponding to the fourth feature difference between the image feature with sample number j is the weight assigned to the fourth feature difference corresponding to the third combined feature pair. Indicates the sixth label weight corresponding to the second image feature, E j -e k Indicates E j Except event feature e k Events outside the event to which they belong, Indicates that the event labels corresponding to the third image feature are all except the event feature e k The sum of the sixth label weights corresponding to the event labels of events other than the event to which it belongs, Represents event feature e k The fifth label weight corresponding to the event, δ represents a preset constant.

[0159] ④ Generate a third predicted loss value based on the third difference weight, the third feature difference corresponding to the third feature pair in the third combined feature pair, and the fourth feature difference corresponding to the fourth feature pair in the third combined feature pair. The method for generating the third predicted loss value can be determined by referring to Formula 5 above and is not further described here.

[0160] Similarly, when calculating the fourth prediction loss value, different weights can be assigned to the fourth feature differences corresponding to the fourth feature pairs in different fourth combined feature pairs. Thus, generating the fourth prediction loss value for the sample feature using the prediction model based on the third feature differences corresponding to the first feature pair and the fourth feature differences corresponding to the fourth feature pair in the fourth combined feature pair can specifically include the following steps: 1. Determining the event feature included in the fourth feature pair in the fourth combined feature pair as the fourth event feature, and determining the image feature included in the fourth feature pair in the fourth combined feature pair as the fourth image feature. 2. Invoking the prediction model to generate, based on the fourth image feature, a seventh label weight for the event to which the fourth event feature belongs, and an eighth label weight for the event label of the image data to which the fourth image feature belongs. 3. Generating a fourth difference weight for the fourth feature difference corresponding to the fourth combined feature pair based on the seventh and eighth label weights. 4. Generating a fourth prediction loss value based on the fourth difference weight, the third feature difference corresponding to the third feature pair in the fourth combined feature pair, and the fourth feature difference corresponding to the fourth feature pair in the fourth combined feature pair. Among them, the method for determining the seventh label weight and the eighth label weight can refer to the relevant description of the fifth label weight and the sixth label weight mentioned above, and the method for determining the fourth difference weight can refer to the relevant description of the third difference weight mentioned above, which will not be repeated here.

[0161] S508 : Calling the prediction model to predict the events indicated by the N text data respectively, and calling the information extractor to extract the prediction association information of the events indicated by the predicted N text data respectively from the N text data.

[0162] Each of the N text data carries a correlation information tag, which indicates correlation information in the text data, i.e., the arguments indicated by each argument role corresponding to an event in the text data. The predicted correlation information indicates the arguments indicated by each argument role corresponding to the event indicated by the text data extracted based on the prediction model.

[0163] In a possible embodiment, the information extractor of the prediction model may include an argument extractor, which can be used to extract the associated information of the event indicated by the text data. Then, when extracting the predicted associated information of the event indicated by the text data, the event in the text data can be determined based on the above-mentioned trigger word extractor, and then the event features of the event indicated by the text data can be determined based on the event encoder in the information extractor. Then, the features of the trigger word, the text features and the event features obtained based on the trigger word extractor are spliced ​​and input into the above-mentioned argument extractor, so as to determine the probability of each argument role corresponding to the event to which the input event features belong for each segmentation in the text data. If the probability of an argument role corresponding to a segmentation is greater than a threshold, it can be determined that the segmentation is the argument of the argument role corresponding to the event, and then the segmentations that are continuously predicted to belong to the same argument role are formed into corresponding arguments, thereby obtaining the predicted associated information of the event indicated by the text data.

[0164] In one possible implementation, the argument extractor in the prediction model can use a sequence labeling method. Figure 6 , Figure 6 This is a schematic diagram of the result of a trigger word extractor provided by an embodiment of the present application, such as Figure 6 As shown, in the trigger word extractor (such as Figure 6 ), the event indicated by the text data and the trigger word corresponding to the event (such as Figure 6 ), and then the event features of the predicted event (such as Figure 6 ), trigger word features corresponding to the trigger word, and text features (such as Figure 6 603 in the input argument extractor (as shown in Figure 6 As shown in 602 in FIG, in the argument extractor, the event feature trigger word feature and the text feature can be spliced ​​together first, and then input into a linear layer (such as Figure 6606 in ), and then the conditional random field (CRF) is used to perform a joint probability representation calculation on the output of the linear layer to obtain the probability that each segmentation corresponds to each argument role of the event. Then, based on the probability, the argument role corresponding to each segmentation is determined, and the segmentation predicted to have the corresponding argument role can be determined as the argument of the argument role of the event. The argument and the argument role are the association information of the event, as shown in Figure 3 As shown in 308, in the text data, the event triggered by trigger word 1 has arguments 1-1 and 1-2, and the event triggered by trigger word 2 has arguments 2-1 and 2-2. The event triggered by trigger word 2 has no corresponding argument in the text data. During the training process, the model parameters of the prediction model can be optimized based on the associated information of the event indicated by the text data predicted by the information extractor and the associated information label, so that the trained prediction model can accurately predict the associated information of the event indicated by the text data.

[0165] It is understandable that the above-mentioned method of predicting the events indicated by the text data and the associated information of the events can be called a pipeline method (an event extraction method). The embodiment of the present application can also adopt a joint extraction method to realize the prediction of events based on text features. Specifically, the event extraction task can be converted into a question-answering task to realize the extraction of event information. In addition, it is also possible to consider converting the event extraction task into a graph network generation task, that is, to realize the task of generating graphs from text data, so that the event structured information can be retained to the greatest extent in a joint manner, and events with the same trigger words or arguments can be linked together through potential connections in the form of graphs, thereby retaining more information.

[0166] S509 : Generate an information extraction bias of the information extractor based on the associated information labels and predicted associated information of the N text data.

[0167] The information extraction bias can be determined by the difference between the predicted context information and the context information label based on N text data. During training, the information extraction bias needs to be gradually reduced until convergence occurs. Specifically, the argument role corresponding to each predicted word should gradually become consistent with the context information label. This allows the trained prediction model to accurately determine the context information of the event indicated by the text data.

[0168] S510: Correct the model parameters of the prediction model based on the first feature prediction deviation, the second feature prediction deviation, and the information extraction deviation to obtain a trained prediction model.

[0169] Among them, the information extractor in the trained prediction model is used to extract the associated information of the event indicated by the input text data, and the event indicated by the text data and the associated information of the indicated event, that is, the arguments corresponding to each argument role corresponding to the indicated event can be determined through the trained prediction model. It can be understood that in the process of training the prediction model, the first feature prediction deviation, the second feature prediction deviation, and the information extraction deviation can be gradually reduced until convergence. In a possible embodiment, the model parameters of the prediction model can also be corrected based on the first feature prediction deviation, the second feature prediction deviation, the information extraction deviation and the above-mentioned event prediction deviation to obtain a trained prediction model.

[0170] Here we take a prediction model as an example to describe the training process of the prediction model. Figure 7 , Figure 7 This is a schematic diagram of a training framework for a prediction model provided in an embodiment of the present application. First, a sample data set is obtained, i.e. Figure 7 The multiple text data shown in 701 and Figure 7 The multiple image data shown in 702 can then be based on a text encoder (such as Figure 7 703 in ) to obtain the text features of each text data, based on the image encoder (as shown in Figure 7 704 in the figure) to obtain the image features of each image data, so as to determine the first feature prediction deviation (as shown in FIG. Figure 7 705 in ), and based on the event encoder (as Figure 7 709 in the example) to obtain event features of each event in the event set, so as to determine the second feature prediction deviation (such as Figure 7 706 in ). And call the trigger word extractor (as shown in Figure 7 706 in FIG) determines the event indicated by the text data predicted by each text feature based on the text features of each text data (such as Figure 7 707 in ), and determine the event features of the event indicated by the predicted text data based on the event encoder, thereby Figure 7 708 in the figure), the information extractor composed of the predicted associated information of the event indicated by the text data (such as Figure 7 710 in ), thereby determining the information extraction bias based on the predicted associated information and the associated information label carried by the text data (as shown in Figure 7711 in ), the model parameters of the prediction model can be corrected based on the first feature prediction deviation, the second feature prediction deviation and the information extraction deviation, so that the prediction model can predict the event indicated by the text data and the associated information of the event indicated by the text data.

[0171] It can be understood that when correcting the model parameters based on the first feature prediction deviation, the difference in features between text data and image data with the same event label should be reduced, and the difference in features between text data and image data with different event labels should be increased. When correcting the model parameters based on the second feature prediction deviation, the difference in features between events that are the same as the events indicated by the event labels of the image data should be reduced, and the difference in features between events that are different from the events indicated by the event labels of the image data should be increased. In this way, comparative learning can be performed based on text data and image data, and comparative learning can be performed based on events and image data, so that text features can better represent events, and better event representations can be learned, thereby improving the extraction effect of arguments corresponding to events, thereby improving the accuracy of prediction of events in text data, and the accuracy of prediction of associated information corresponding to events.

[0172] In one possible scenario, the embodiments of the present application can be applied to online consultation scenarios in the field of medical technology. Figure 8 , Figure 8 This is a flow chart of an application scenario provided by an embodiment of the present application. Figure 8 As shown, if a user needs to conduct an online consultation, the homepage of the consultation website can be opened (as shown in step S801), so as to determine the department that needs to be registered based on the intelligent navigation of the text website (as shown in step S802). After completing the registration, the user can enter the pre-consultation process (as shown in step S803), and then obtain the user's relevant basic information based on some preset consultation questions, such as current medical history, family history, cause of disease, etc. Based on the prediction model, the event indicated by the text data in the pre-consultation process and the associated information of the event can be determined (as shown in step S804), and then an electronic examination report can be generated based on the event and the event associated information, and pushed to the doctor assigned to the user for consultation (as shown in step S805), so that the doctor can have a preliminary understanding of the user's situation before diagnosing the user, thereby optimizing the entire diagnosis and treatment process. It can be understood that the embodiments of the present application are not limited to application in the field of medical technology, but can also be applied to other fields, which are not limited here.

[0173] The embodiment of the present application proposes a data processing solution that can call a prediction model to generate text features for each text data and image features for each image data, and then based on the first feature differences corresponding to multiple first feature pairs in which the text data to which the text features belong and the image data to which the image features belong have the same event labels, and the second feature differences corresponding to second feature pairs in which the text data to which the text features belong and the image data to which the image features belong have different event labels, the model parameters of the prediction model are corrected to obtain a trained prediction model. In this way, the feature differences between the features corresponding to text data and image data with the same event can be compared with the feature differences between the features corresponding to text data and image data with different events. By adding image data to train the prediction model, the influence of writing habits on the text features corresponding to the text data is reduced, so that the prediction model can generate text features that predict events more accurately, which helps to improve the accuracy of the prediction of the events indicated by the text data.

[0174] See Figure 9 , Figure 9 Schematic diagram of a data processing device provided in an embodiment of the present application. Optionally, the data processing device may be provided in the above-mentioned electronic device. Figure 9 As shown, the data processing device described in this embodiment may include:

[0175] The acquisition unit 901 is used to acquire a sample data set; the sample data in the sample data set includes N text data and M image data, any text data and any image data has an event label, and N and M are positive integers;

[0176] Processing unit 902, configured to call a prediction model to generate text features for each text data and image features for each image data;

[0177] The processing unit 902 is further configured to respectively generate a first feature difference between a text feature and an image feature included in each of the plurality of first feature pairs; wherein the text data to which the text feature included in any first feature pair belongs and the image data to which the image feature included in the text feature pair belongs have the same event label;

[0178] The processing unit 902 is further configured to respectively generate a second feature difference between the text feature and the image feature included in each of the plurality of second feature pairs; wherein the text data to which the text feature included in any second feature pair belongs and the image data to which the image feature included in the text feature pair belongs have different event labels;

[0179] The processing unit 902 is also used to correct the model parameters of the prediction model based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model; the trained prediction model is used to predict the event indicated by the input text data based on the input text data.

[0180] In one implementation, the processing unit 902 is specifically configured to:

[0181] Combining the plurality of first feature pairs and the plurality of second feature pairs to obtain a first combined feature pair and a second combined feature pair; the first feature pair and the second feature pair in the first combined feature pair contain the same text feature, and the first feature pair and the second feature pair in the second combined feature pair contain the same image feature;

[0182] Generate a first prediction loss value of the prediction model for the sample feature according to a first feature difference corresponding to the first feature pair and a second feature difference corresponding to the second feature pair in the first combined feature pair;

[0183] generating a second prediction loss value of the prediction model for the sample feature according to a first feature difference corresponding to the first feature pair and a second feature difference corresponding to the second feature pair in the second combined feature pair;

[0184] A first characteristic prediction deviation of the prediction model is determined according to the first prediction loss value and the second prediction loss value, and a model parameter of the prediction model is corrected according to the first characteristic prediction deviation to obtain a trained prediction model.

[0185] In one implementation, the processing unit 902 is specifically configured to:

[0186] Calling the prediction model to predict the events indicated by the N text data respectively, and generating event prediction deviations based on the events indicated by the predicted N text data and the event labels carried by the N text data respectively;

[0187] Based on the first feature prediction deviation and the event prediction deviation, the model parameters of the prediction model are corrected to obtain a trained prediction model.

[0188] In one implementation, the processing unit 902 is specifically configured to:

[0189] Determine the text feature included in the second feature pair in the first combined feature pair as the first text feature, and determine the image feature included in the second feature pair in the first combined feature pair as the first image feature;

[0190] Calling the prediction model to generate a first label weight of the event label of the text data to which the first text feature belongs based on the first image feature, and generating a second label weight of the event label of the image data to which the first image feature belongs;

[0191] Generating a first difference weight of a second feature difference corresponding to the first combined feature pair according to the first label weight and the second label weight;

[0192] A first prediction loss value is generated according to the first difference weight, a first feature difference corresponding to the first feature pair in the first combined feature pair, and a second feature difference corresponding to the second feature pair in the first combined feature pair.

[0193] In one implementation, the processing unit 902 is specifically configured to:

[0194] Determining the text feature included in the second feature pair in the second combined feature pair as the second text feature, and determining the image feature included in the second feature pair in the first combined feature pair as the second image feature;

[0195] Calling the prediction model to generate a third label weight for the event label of the text data to which the second text feature belongs, and generating a fourth label weight for the event label of the image data to which the second image feature belongs, based on the second image feature;

[0196] Generating a second difference weight of the second feature difference corresponding to the second combined feature pair according to the third label weight and the fourth label weight;

[0197] A second prediction loss value is generated according to the second difference weight, a first feature difference corresponding to the first feature pair in the second combined feature pair, and a second feature difference corresponding to the second feature pair in the second combined feature pair.

[0198] In one implementation, event labels of sample data in a sample data set are annotated based on an event set, where the event set includes multiple events; N text data all carry associated information labels; the prediction model includes an information extractor; and the processing unit 902 is specifically configured to:

[0199] generating a first feature prediction deviation of the prediction model based on a first feature difference corresponding to each first feature pair and a second feature difference corresponding to each second feature pair;

[0200] Calling the information extractor to generate event features for each event in the event set;

[0201] generating a second feature prediction bias of the information extractor based on the event feature of each event and the image feature of each image data;

[0202] Calling the prediction model to predict the events indicated by the N text data respectively, and calling the information extractor to extract the predicted related information of the events indicated by the N text data respectively from the N text data;

[0203] Generate information extraction bias of an information extractor based on associated information labels and predicted associated information of N text data;

[0204] Based on the first feature prediction deviation, the second feature prediction deviation, and the information extraction deviation, the model parameters of the prediction model are corrected to obtain a trained prediction model; the information extractor in the trained prediction model is used to extract related information of the event indicated by the input text data.

[0205] In one implementation, the processing unit 902 is specifically configured to:

[0206] Invoking an information extractor to generate a third feature difference between an event feature and an image feature included in each of a plurality of third feature pairs; an event to which the event feature included in any third feature pair belongs is the same as an event indicated by an event label corresponding to the included image feature;

[0207] Invoking an information extractor to generate a fourth feature difference between an event feature and an image feature included in each of a plurality of fourth feature pairs; an event to which the event feature included in any fourth feature pair belongs is different from an event indicated by an event label corresponding to the included image feature;

[0208] The information extractor is called to generate a second feature prediction deviation for the information extractor based on the third feature difference corresponding to each third feature pair and the fourth feature difference corresponding to each fourth feature pair.

[0209] In one implementation, the processing unit 902 is further configured to:

[0210] Get target text data;

[0211] Call the trained prediction model to generate text features of the target text data;

[0212] The trained prediction model is called to predict the event indicated by the target text data based on the text features of the target text data.

[0213] See Figure 10 , Figure 10 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device described in this embodiment includes: a processor 1001 and a memory 1002. Optionally, the electronic device may also include a network interface or a power supply module. The processor 1001 and the memory 1002 can exchange data.

[0214] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0215] The network interface may include an input device and / or an output device. For example, the input device may be a control panel, a microphone, a receiver, etc., and the output device may be a display screen, a transmitter, etc., which are not listed here. For example, in an embodiment of the application, the network interface may include a receiver and a transmitter.

[0216] The memory 1002 may include a read-only memory and a random access memory, and provide program instructions and data to the processor 1001. A portion of the memory 1002 may also include a non-volatile random access memory. When the processor 1001 calls a program instruction, it is used to execute:

[0217] Obtain a sample data set; the sample data in the sample data set includes N text data and M image data, any text data and any image data has an event label, and N and M are positive integers;

[0218] Call the prediction model to generate text features for each text data and image features for each image data;

[0219] Generating first feature differences between text features and image features included in each of the plurality of first feature pairs; wherein text data to which the text features included in any first feature pair belong and image data to which the image features included in the text features belong have the same event label;

[0220] Generating second feature differences between text features and image features included in each of the plurality of second feature pairs; wherein text data to which the text features included in any second feature pair belong and image data to which the image features included in the second feature pair belong have different event labels;

[0221] The model parameters of the prediction model are corrected based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model; the trained prediction model is used to predict the event indicated by the input text data based on the input text data.

[0222] In one implementation, the processor 1001 is specifically configured to:

[0223] Combining the plurality of first feature pairs and the plurality of second feature pairs to obtain a first combined feature pair and a second combined feature pair; the first feature pair and the second feature pair in the first combined feature pair contain the same text feature, and the first feature pair and the second feature pair in the second combined feature pair contain the same image feature;

[0224] Generate a first prediction loss value of the prediction model for the sample feature according to a first feature difference corresponding to the first feature pair and a second feature difference corresponding to the second feature pair in the first combined feature pair;

[0225] generating a second prediction loss value of the prediction model for the sample feature according to a first feature difference corresponding to the first feature pair and a second feature difference corresponding to the second feature pair in the second combined feature pair;

[0226] A first characteristic prediction deviation of the prediction model is determined according to the first prediction loss value and the second prediction loss value, and a model parameter of the prediction model is corrected according to the first characteristic prediction deviation to obtain a trained prediction model.

[0227] In one implementation, the processor 1001 is specifically configured to:

[0228] Calling the prediction model to predict the events indicated by the N text data respectively, and generating event prediction deviations based on the events indicated by the predicted N text data and the event labels carried by the N text data respectively;

[0229] Based on the first feature prediction deviation and the event prediction deviation, the model parameters of the prediction model are corrected to obtain a trained prediction model.

[0230] In one implementation, the processor 1001 is specifically configured to:

[0231] Determine the text feature included in the second feature pair in the first combined feature pair as the first text feature, and determine the image feature included in the second feature pair in the first combined feature pair as the first image feature;

[0232] Calling the prediction model to generate a first label weight of the event label of the text data to which the first text feature belongs based on the first image feature, and generating a second label weight of the event label of the image data to which the first image feature belongs;

[0233] Generating a first difference weight of a second feature difference corresponding to the first combined feature pair according to the first label weight and the second label weight;

[0234] A first prediction loss value is generated according to the first difference weight, a first feature difference corresponding to the first feature pair in the first combined feature pair, and a second feature difference corresponding to the second feature pair in the first combined feature pair.

[0235] In one implementation, the processor 1001 is specifically configured to:

[0236] Determining the text feature included in the second feature pair in the second combined feature pair as the second text feature, and determining the image feature included in the second feature pair in the first combined feature pair as the second image feature;

[0237] Calling the prediction model to generate a third label weight for the event label of the text data to which the second text feature belongs, and generating a fourth label weight for the event label of the image data to which the second image feature belongs, based on the second image feature;

[0238] Generating a second difference weight of the second feature difference corresponding to the second combined feature pair according to the third label weight and the fourth label weight;

[0239] A second prediction loss value is generated according to the second difference weight, a first feature difference corresponding to the first feature pair in the second combined feature pair, and a second feature difference corresponding to the second feature pair in the second combined feature pair.

[0240] In one implementation, event labels of sample data in a sample data set are annotated based on an event set, where the event set includes multiple events; N text data all carry associated information labels; the prediction model includes an information extractor; and the processor 1001 is specifically configured to:

[0241] generating a first feature prediction deviation of the prediction model based on a first feature difference corresponding to each first feature pair and a second feature difference corresponding to each second feature pair;

[0242] Calling the information extractor to generate event features for each event in the event set;

[0243] generating a second feature prediction bias of the information extractor based on the event feature of each event and the image feature of each image data;

[0244] Calling the prediction model to predict the events indicated by the N text data respectively, and calling the information extractor to extract the predicted related information of the events indicated by the N text data respectively from the N text data;

[0245] Generate information extraction bias of an information extractor based on associated information labels and predicted associated information of N text data;

[0246] Based on the first feature prediction deviation, the second feature prediction deviation, and the information extraction deviation, the model parameters of the prediction model are corrected to obtain a trained prediction model; the information extractor in the trained prediction model is used to extract related information of the event indicated by the input text data.

[0247] In one implementation, the processor 1001 is specifically configured to:

[0248] Invoking an information extractor to generate a third feature difference between an event feature and an image feature included in each of a plurality of third feature pairs; an event to which the event feature included in any third feature pair belongs is the same as an event indicated by an event label corresponding to the included image feature;

[0249] Invoking an information extractor to generate a fourth feature difference between an event feature and an image feature included in each of a plurality of fourth feature pairs; an event to which the event feature included in any fourth feature pair belongs is different from an event indicated by an event label corresponding to the included image feature;

[0250] The information extractor is called to generate a second feature prediction deviation for the information extractor based on the third feature difference corresponding to each third feature pair and the fourth feature difference corresponding to each fourth feature pair.

[0251] In one implementation, the processor 1001 is further configured to:

[0252] Get target text data;

[0253] Call the trained prediction model to generate text features of the target text data;

[0254] The trained prediction model is called to predict the event indicated by the target text data based on the text features of the target text data.

[0255] Optionally, when the program instructions are executed by the processor, other steps of the method in the above embodiment may also be implemented, which will not be repeated here.

[0256] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the above method, such as executing the method executed by the above electronic device, which is not repeated here.

[0257] Optionally, the storage medium involved in this application, such as computer-readable storage medium, may be non-volatile or volatile.

[0258] Optionally, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created based on the use of blockchain nodes, etc. Among them, the blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain may include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0259] It should be noted that for the aforementioned various method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0260] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0261] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps performed in the above-described method embodiments. For example, the computer device may be a terminal or a server.

[0262] The above is a detailed introduction to a data processing method, electronic device, program product and medium provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain a sample data set; the sample data in the sample data set includes N text data and M image data, any text data and any image data has an event label, and N and M are positive integers; Call the prediction model to generate text features for each text data and image features for each image data; Generating first feature differences between text features and image features included in each of the plurality of first feature pairs; wherein text data to which the text features included in any first feature pair belong and image data to which the image features included in the text features belong have the same event label; Generating second feature differences between text features and image features included in each of the plurality of second feature pairs; wherein text data to which the text features included in any second feature pair belong and image data to which the image features included in the second feature pair belong have different event labels; Correcting the model parameters of the prediction model based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model; the trained prediction model is used to predict the event indicated by the input text data based on the input text data; Among them, the method of correcting the model parameters of the prediction model based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model includes: combining the multiple first feature pairs and the multiple second feature pairs to obtain a first combined feature pair and a second combined feature pair; the first feature pair and the second feature pair in the first combined feature pair contain the same text features, and the first feature pair and the second feature pair in the second combined feature pair contain the same image features; generating a first prediction loss value of the prediction model for the sample feature based on the first feature difference corresponding to the first feature pair and the second feature difference corresponding to the second feature pair in the first combined feature pair; generating a second prediction loss value of the prediction model for the sample feature based on the first feature difference corresponding to the first feature pair and the second feature difference corresponding to the second feature pair in the second combined feature pair; determining the first feature prediction deviation of the prediction model based on the first prediction loss value and the second prediction loss value, and correcting the model parameters of the prediction model based on the first feature prediction deviation to obtain a trained prediction model.

2. The method according to claim 1, characterized in that Generating a first prediction loss value of the prediction model for the sample feature according to a first feature difference corresponding to the first feature pair and a second feature difference corresponding to the second feature pair in the first combined feature pair includes: Determining the text feature included in the second feature pair in the first combined feature pair as a first text feature, and determining the image feature included in the second feature pair in the first combined feature pair as a first image feature; Calling the prediction model to generate a first label weight for an event label of text data to which the first text feature belongs, and generating a second label weight for an event label of image data to which the first image feature belongs, based on the first image feature; Generating a first difference weight of a second feature difference corresponding to the first combined feature pair according to the first label weight and the second label weight; A first prediction loss value is generated according to the first difference weight, a first feature difference corresponding to the first feature pair in the first combined feature pair, and a second feature difference corresponding to the second feature pair in the first combined feature pair.

3. The method according to claim 1, characterized in that Generating a second prediction loss value of the prediction model for the sample feature according to the first feature difference corresponding to the first feature pair and the second feature difference corresponding to the second feature pair in the second combined feature pair includes: Determining the text feature included in the second feature pair in the second combined feature pair as a second text feature, and determining the image feature included in the second feature pair in the first combined feature pair as a second image feature; Calling the prediction model to generate a third label weight of the event label of the text data to which the second text feature belongs, and generating a fourth label weight of the event label of the image data to which the second image feature belongs, based on the second image feature; Generating a second difference weight of the second feature difference corresponding to the second combined feature pair according to the third label weight and the fourth label weight; A second prediction loss value is generated according to the second difference weight, a first feature difference corresponding to the first feature pair in the second combined feature pair, and a second feature difference corresponding to the second feature pair in the second combined feature pair.

4. The method according to claim 1, characterized in that The event labels of the sample data in the sample data set are annotated based on an event set, wherein the event set includes multiple events; the N text data all carry associated information labels; the prediction model includes an information extractor; and the model parameters of the prediction model are corrected based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair to obtain a trained prediction model, including: generating a first feature prediction deviation of the prediction model based on the first feature difference corresponding to each first feature pair and the second feature difference corresponding to each second feature pair; calling the information extractor to generate event features for each event in the event set; generating a second feature prediction bias of the information extractor based on the event feature of each event and the image feature of each image data; Calling the prediction model to predict the events indicated by the N text data respectively, and calling the information extractor to extract the predicted associated information of the events indicated by the N text data respectively from the N text data; generating an information extraction bias of the information extractor based on the associated information labels and predicted associated information of the N text data; Based on the first feature prediction deviation, the second feature prediction deviation, and the information extraction deviation, the model parameters of the prediction model are corrected to obtain a trained prediction model; the information extractor in the trained prediction model is used to extract related information of the event indicated by the input text data.

5. The method according to claim 4, characterized in that: Generating a second feature prediction deviation of the information extractor based on the event feature of each event and the image feature of each image data includes: The information extractor is called to generate a third feature difference between an event feature and an image feature included in each of a plurality of third feature pairs; the event to which the event feature included in any third feature pair belongs is the same as the event indicated by the event label corresponding to the included image feature; calling the information extractor to generate a fourth feature difference between an event feature and an image feature included in each of a plurality of fourth feature pairs; wherein the event to which the event feature included in any fourth feature pair belongs is different from the event indicated by the event label corresponding to the included image feature; The information extractor is called to generate the second feature prediction deviation for the information extractor based on the third feature difference corresponding to each third feature pair and the fourth feature difference corresponding to each fourth feature pair.

6. The method according to claim 1, characterized in that The method further comprises: Get target text data; Calling the trained prediction model to generate text features of the target text data; The trained prediction model is called to predict the event indicated by the target text data based on the text features of the target text data.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 6.

8. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 6 when executed by a processor.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6.

Images