Data processing method and device, electronic equipment and computer readable storage medium

Abstract

This disclosure provides a data processing method and apparatus, an electronic device, and a computer-readable storage medium. The method includes: obtaining a hidden feature vector of a target object based on time-series data of the target object; obtaining an entity feature vector and a global feature vector of the target object based on text data of the target object; wherein the entity feature vector is used to characterize the features of entities in the text data, and the global feature vector is used to characterize the features of the text data; fusing the entity feature vector with the hidden feature vector to obtain an entity-fused feature vector, and fusing the hidden feature vector with the entity feature vector to obtain a hidden-fused feature vector; determining a multimodal fusion feature of the target object based on the hidden-fused feature vector, the entity-fused feature vector, and the global feature vector; and predicting the probability of the target object exhibiting a target state based on the multimodal fusion feature of the target object. Embodiments of this disclosure can improve the accuracy of prediction.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method and apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Natural Language Processing (NLP) is a science integrating linguistics, computer science, and mathematics. It is a crucial core technology of artificial intelligence, applicable to areas such as machine translation, text classification, question answering, speech recognition, and text-to-semantic comparison. Neural networks are mathematical models that mimic the behavioral characteristics of animal neural networks, performing distributed parallel information processing. Based on the principles of biological neural networks and the needs of practical applications, practical artificial neural network models are built. Through appropriate learning algorithms, they simulate certain intellectual activities of the human brain, technically solving real-world problems. With technological advancements and application demands, the ability to combine NLP and neural network technologies to solve practical problems is receiving increasing attention. Summary of the Invention

[0003] This disclosure provides a data processing method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, this disclosure provides a data processing method, the method comprising:

[0005] The hidden feature vector of the target object is obtained based on the time series data of the target object; wherein, the hidden feature vector is used to characterize the features of the time series data at different times;

[0006] Based on the text data of the target object, obtain the entity feature vector and global feature vector of the target object; wherein, the entity feature vector is used to characterize the features of the entities in the text data, and the global feature vector is used to characterize the features of the text data;

[0007] The entity feature vector is fused with the hidden feature vector to obtain an entity fused feature vector, and the hidden feature vector is fused with the entity feature vector to obtain a hidden fused feature vector;

[0008] The multimodal fusion features of the target object are determined based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector;

[0009] The probability of the target object appearing in the target state is predicted based on the multimodal fusion features of the target object.

[0010] Secondly, this disclosure provides a data processing apparatus, the apparatus comprising:

[0011] The first acquisition module is used to acquire the hidden feature vector of the target object based on the time series data of the target object; wherein, the hidden feature vector is used to characterize the features of the time series data at different times;

[0012] The second acquisition module is used to acquire the entity feature vector and global feature vector of the target object based on the text data of the target object; wherein, the entity feature vector is used to characterize the features of the entities in the text data, and the global feature vector is used to characterize the features of the text data;

[0013] The fusion module is used to fuse the entity feature vector with the hidden feature vector to obtain an entity fused feature vector, and to fuse the hidden feature vector with the entity feature vector to obtain a hidden fused feature vector;

[0014] The determination module is used to determine the multimodal fusion features of the target object based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector;

[0015] The prediction module is used to predict the probability of the target object appearing in the target state based on the multimodal fusion features of the target object.

[0016] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the data processing method described above.

[0017] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processor core, implements the above-described data processing method.

[0018] The embodiments provided in this disclosure obtain a hidden feature vector of the target object based on time-series data of the target object, and obtain an entity feature vector and a global feature vector of the target object based on text data of the target object. The entity feature vector is then fused with the hidden feature vector to obtain an entity fusion feature vector, which contains information from both the text data and the time-series data. Similarly, the hidden feature vector is fused with the entity feature vector to obtain a hidden fusion feature vector, which also contains information from both the time-series data and the text data. Based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector, a multimodal fusion feature of the target object is determined. This means that the multimodal fusion feature of the target object contains information from both the time-series data and the entity and global information from the text data. Therefore, predicting the probability of the target object exhibiting a target state based on the multimodal fusion feature of the target object can improve the accuracy of the prediction.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 The diagram illustrates an application scenario of the data processing method and apparatus provided in the embodiments of this disclosure.

[0022] Figure 2 A flowchart of a data processing method provided in an embodiment of this disclosure;

[0023] Figure 3 A schematic diagram illustrating the construction of a target object correlation diagram provided in an embodiment of this disclosure;

[0024] Figure 4 A schematic diagram illustrating a data processing method provided in an embodiment of this disclosure;

[0025] Figure 5 A schematic diagram of a data processing apparatus provided in an embodiment of this disclosure;

[0026] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0029] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0031] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0032] Medical devices can be used to monitor a user's physical condition. The devices store the monitored data on a server, which then analyzes the user's health status (e.g., healthy, sub-healthy) based on this data. However, the monitoring data is time-series data collected by the medical device and can only partially reflect the user's health condition. On the other hand, when a user seeks medical attention, doctors typically create clinical notes, which record the doctor's analysis of the user's health condition based on their professional knowledge. In related technologies, monitoring data and clinical notes are independent of each other; that is, if a user predicts their health condition solely based on monitoring data or clinical notes, the resulting predictions will be inaccurate.

[0033] Figure 1 The diagram illustrates an application scenario of the data processing method and apparatus provided in the embodiments of this disclosure.

[0034] like Figure 1 As shown, the application scenarios of this disclosure embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a data transmission link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0035] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 are used to obtain information about target objects, such as physical condition information, business information for handling various transactions, credit information, etc.

[0036] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to testing devices, smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the data processing method provided in this embodiment can be executed by server 105. Correspondingly, the data processing apparatus provided in this embodiment can be located in server 105. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the data processing apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The data processing method according to embodiments of this disclosure can be executed by electronic devices such as terminal devices or servers. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The server can be a standalone physical server, a server cluster consisting of multiple physical servers, or a cloud server capable of cloud computing. This method can be implemented by a processor calling computer-readable program instructions stored in memory.

[0041] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this disclosure. (Refer to...) Figure 2 The method includes:

[0042] Step S201: Obtain the hidden feature vector of the target object based on the time series data of the target object.

[0043] Hidden feature vectors (HMRs) are used to characterize the features of time series data at different times. For example, when the time series data is an electrocardiogram (ECG) result, the ECG results measured at different times will change over time. For instance, ECG results change with age each year. HMRs can be extracted using LSTM to obtain features that reflect time characteristics; that is, HMRs can characterize the characteristics of ECG changes over time. Similarly, the physiological characteristics of a person taking hypoglycemic drugs will change with the duration of medication. HMRs can represent the characteristics at different times after taking the medication.

[0044] This disclosure does not limit the method for extracting the hidden feature vector of the target object. For example, the hidden feature vector of the target object can be extracted using a Bidirectional Gate Recurrent Unit (BiGRU) model or a Long Short-Term Memory (LSTM) model.

[0045] In this embodiment of the disclosure, the target object can be an object from different fields. For example, in the medical field, the target object can be a patient, and in the financial field, the target object can be a borrower. Time series data is data with time characteristics, that is, data that changes over time. When the target object is a patient, the time series data can be physiological data that changes over time, such as physiological data such as heart rate, blood pressure, and blood oxygen.

[0046] When the target is a borrower, the time series data represents the relationship between loan delinquency time and call status. Call status includes, but is not limited to, connected, hung up, and unanswered. For example, when the loan delinquency is short (within one week), the call status is connected; when the loan delinquency is relatively long (between two and four weeks), the call status is hung up; and when the loan delinquency is very long (more than one month), the call status is unreachable.

[0047] Assume that the embodiments of this disclosure include i objects, and the i objects constitute an object set. Among them, X (i) The time series data of the i-th object, C (i) y represents the text data of the i-th object. (i) N represents the label of the i-th object. s This represents the total number of objects. Time-series data of the target objects can be represented as... Where, N t N represents the time step. v Representing a time series, N represents t ×N v The eigenvector matrix. The entity set in the text data is represented as... Where, N w This indicates the number of entities, where c represents an entity.

[0048] In some embodiments, before extracting the hidden feature vector, entity feature vector, and global feature vector of the target object, multiple target objects are preprocessed to remove target objects with incomplete information.

[0049] In some embodiments, BiGRU is used to process the time series data of the target object to extract hidden feature vectors. For example, hidden feature vectors are extracted using formula (1).

[0050] H = BiGRU(X) (1)

[0051] Where H represents the hidden feature vector and X represents the time series data.

[0052] In some embodiments, the hidden feature vector H can be represented as Where, N t N represents the time step. h Let h represent the dimension of the hidden feature vector, and h represent the hidden feature vector. N represents t ×N h The eigenvector matrix.

[0053] Step S202: Obtain the entity feature vector and global feature vector of the target object based on the text data of the target object.

[0054] Entity feature vectors are used to represent the features of entities in text data; that is, entity feature vectors represent local features in text data. Global feature vectors are used to represent the features of text data; that is, global feature vectors represent the overall features of text data, such as representing important information in text data.

[0055] In this embodiment of the disclosure, the text data can be text information from different fields. In the medical field, the text data can be clinical notes. Entities are medical information within the clinical notes, such as diseases, medications, and dosages. In the financial field, the text data can be call records between agents and customers.

[0056] Each piece of text data has a corresponding textual intent. The global feature vector is used to represent the features of the text data, that is, to represent the textual intent corresponding to the text data. Each piece of text data includes one or more entities, and each entity has a corresponding intent. The entity feature vector is used to represent the features of the entity, that is, the entity features correspond to the entity's intent.

[0057] This disclosure does not limit the extraction methods for entities, entity feature vectors, and global feature vectors. For example, entities can be extracted from clinical notes using the Med7 model, entity feature vectors can be extracted using a pre-trained speech model (such as BioBERT), and global feature vectors can be extracted using the term frequency-inverse document frequency (TF-IDF) method.

[0058] When using the Med7 model to extract entities, the Med7 model can extract various types of entities, resulting in an entity set. Where, N e It refers to the number of entities, that is, the number of entities in the entity set N. e Individual entities.

[0059] Entity set C e It can reflect different pieces of knowledge about the status of the target object. For example, in the medical field, entity set C e It can reflect the relationship between the target subject's physiological condition and medical information. For example, after a target subject receives drug treatment, physiological signals will change in time series data.

[0060] In the medical field, clinical notes contain detailed patient information and medical knowledge involved in physician inferences. Furthermore, clinical notes may contain redundant information, but embodiments of this disclosure extract only representative features to highlight key patient information. Therefore, this disclosure extracts only multi-granular features from clinical notes, such as entity feature vectors and global feature vectors.

[0061] In some embodiments, the global feature vector has a high dimensionality, making it impossible to represent the target object densely and write it entirely into memory. Therefore, embodiments of this disclosure employ Principal Component Analysis (PCA) to reduce the dimensionality of the global feature vector. The dimensionality-reduced global feature vector can be represented as follows: Among them, C g N represents the global feature vector. g This represents the dimension of the global feature vector.

[0062] Step S203: Merge the entity feature vector with the hidden feature vector to obtain the entity fused feature vector, and merge the hidden feature vector with the entity feature vector to obtain the hidden fused feature vector.

[0063] In this embodiment of the disclosure, feature vectors from two different modalities are fused, namely, entity feature vectors and hidden feature vectors are fused to obtain entity fused feature vectors. and hidden fusion feature vector Among them, entity fusion feature vector The hidden feature vector is fused, and the hidden fused feature vector is fused. It incorporates entity feature vectors.

[0064] In some embodiments, step S203 fuses the entity feature vector with the hidden feature vector to obtain the entity fused feature vector. And, fusing the hidden feature vector with the entity feature vector to obtain the hidden fused feature vector; including:

[0065] Step S31: Determine the attention score of the entity feature vector and the attention score of the hidden feature vector based on the entity feature vector and the hidden feature vector.

[0066] The attention score is a weight that represents different feature vectors.

[0067] In some embodiments, step S31, determining the attention score of the entity feature vector and the attention score of the hidden feature vector based on the entity feature vector and the hidden feature vector, includes:

[0068] Step S311: Adjust the dimension of the entity feature vector and the dimension of the hidden feature vector to obtain a standard entity feature vector C with consistent dimensions. b ′ and the standard hidden feature vector H′.

[0069] In some embodiments, the dimensions of the entity feature vector and the hidden feature vector are not always the same. Therefore, before calculating the attention score, it is necessary to adjust the dimensions of the entity feature vector and the hidden feature vector, that is, to unify the dimensions of the entity feature vector and the hidden feature vector.

[0070] The dimensions of the entity feature vector and the hidden feature vector are adjusted using formulas (2) and (3).

[0071] H′=HW a (2)

[0072] C b ′=C b W b (3)

[0073] Where H′ represents the standard hidden feature vector, H represents the hidden feature vector, and C b ' represents the entity feature vector, C b W represents the entity feature vector. a and W b This indicates that the parameters are being adjusted.

[0074] In some embodiments, and Where, N h N represents the dimension of the hidden feature vector. d N represents the number of hidden feature vectors. b The dimension of the entity feature vector.

[0075] Step S312: Calculate the correlation (association degree) between the standard hidden feature vector and the standard entity feature vector.

[0076] In some embodiments, the standard hidden feature vector H′ and the standard entity feature vector C are calculated using formula (4). b The relevance of ′.

[0077] S=tanh(H′W s C b ′ T (4)

[0078] Where S represents the correlation between the standard hidden feature vector and the standard entity feature vector, which can be calculated using the shared feature space. H′ represents the standard hidden feature vector, and C b ′ represents the standard entity feature vector, W s Indicates trainable parameters, Where, N d This indicates the number of hidden feature vectors.

[0079] In some embodiments, the shared feature matrix Where, N t N represents the time step. e This indicates the number of entity feature vectors.

[0080] Step S313: Based on the correlation between the standard hidden feature vector and the standard entity feature vector, the standard hidden feature vector, and the standard entity feature vector, the fused hidden feature vector and the fused entity feature vector are determined using an activation function.

[0081] This disclosure does not limit the specific form of the activation function. For example, the activation function can be the tanh() function.

[0082] In some embodiments, the fused hidden feature vector H is calculated using formula (5). f The fused entity feature vector is calculated using formula (6).

[0083] H f =tanh(W o H′ T +W e C b ′ T S T (5)

[0084]

[0085] Among them, H f This indicates the fusion of hidden feature vectors. H' represents the fused entity feature vector, and C represents the standard hidden feature vector. b ′ represents the standard entity feature vector, S represents the relevance, and W e and W o These represent the entity weight parameters and the hidden vector weight parameters, respectively. Where, N k N represents the number of entity feature vectors. d This indicates the number of hidden feature vectors.

[0086] Step S314: Based on the fused hidden feature vector, the attention score of the hidden feature vector is determined using an attention function; and based on the fused entity feature vector, the attention score of the entity feature vector is determined using an attention function.

[0087] The embodiments disclosed herein do not limit the attention function; for example, the attention function may be the softmax() function.

[0088] In some embodiments, the attention score of the hidden feature vector H is calculated using formula (7), and the attention score of the entity feature vector Cb is calculated using formula (8).

[0089] α o =softmax(W ho H f (7)

[0090]

[0091] Where, α o α represents the attention score of the hidden feature vector. e H represents the attention score of the entity feature vector. f This indicates the fusion of hidden feature vectors. W represents the fused entity feature vector. ho and W he This represents the trainable parameters for predicting attention scores. and Where, N k This indicates the number of entity feature vectors.

[0092] Step S32: Determine the entity fusion feature vector based on the entity feature vector and the attention score of the entity feature vector.

[0093] In some embodiments, the entity fusion feature vector is calculated using formula (9).

[0094]

[0095] in, Represents the entity fusion feature vector. Let e ​​represent the attention score of the i-th entity. i Let N represent the i-th entity. e Indicates the number of entities.

[0096] Step S33: Determine the hidden fusion feature vector based on the hidden feature vector and the attention score of the hidden feature vector.

[0097] In some embodiments, the hidden fusion feature vector is calculated using formula (10).

[0098]

[0099] in, This represents the hidden fusion feature vector. h represents the attention score of the i-th hidden vector. i Let N represent the i-th hidden feature vector. t Indicates the time step.

[0100] Step S204: Determine the multimodal fusion features of the target object based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector.

[0101] Multimodal refers to different data structures, such as time series data and text data, which are two different data structures. The features of data with different structures are fused together to form multimodal fusion features.

[0102] In some embodiments, step S204 is based on hidden fusion feature vectors Entity fusion feature vector Obtaining multimodal fusion features of the target object from the global feature vector Cg, including: hiding the fusion feature vector Entity fusion feature vector By concatenating the global feature vector Cg, the multimodal fusion features of the target object are obtained.

[0103] This disclosure does not limit the concatenation method. For example, the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector can be directly concatenated. For instance, assuming the hidden fusion feature vector is 1101, the entity fusion feature vector is 1011, and the global feature vector is 0101, the multimodal fusion feature obtained by concatenating the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector is 110110110101.

[0104] Step S205: Predict the probability of the target object appearing in the target state based on the multimodal fusion features of the target object.

[0105] The target object has different states, and the target state is the state that the user wants to predict. For example, in the medical field, the target state could be the hospital mortality rate, the ICU mortality rate, or the length of stay in the ICU (such as the probability of staying for more than 3 days or more than 7 days). In the financial field, the target state could be the probability of repayment or the probability of being complained about.

[0106] The embodiments of this disclosure can predict the probability of a target object exhibiting a target state using a fully connected network model.

[0107] In other embodiments, the data processing method considers both the probability of predicting the occurrence of the target state from different data (internal perspective) of the same target object, and the probability of predicting the occurrence of the target state by combining time series data and text data of other objects.

[0108] In some embodiments, step S205 predicts the probability of the target object exhibiting a target state based on the multimodal fusion features of the target object, including:

[0109] Step S51: Determine the relevance between the target object and its neighboring objects based on the text data of the target object and the text data of its neighboring objects.

[0110] In this embodiment of the disclosure, the target object and the adjacent objects belong to the same business domain, wherein the business domain includes, but is not limited to, the financial field and the medical field.

[0111] In some embodiments, step S51, determining the relevance between the target object and its neighboring objects based on the text data of the target object and the text data of its neighboring objects, includes:

[0112] Extract target object entities from the text data of the target object to obtain a target object entity set; wherein the target object entity set includes one or more target object entities; extract neighboring object entities from the text data of neighboring objects to obtain a neighboring object entity set; wherein the neighboring object entity set includes one or more neighboring object entities; determine the relevance between the target object and neighboring objects based on the target object entities in the target object entity set and the neighboring object entities in the neighboring object entity set.

[0113] For example, textual data in the medical field includes clinical notes. Entities were extracted from clinical notes using the Med7 model to obtain an entity set. Where, N s This indicates the number of clinical notes.

[0114] Construct a relational graph of different objects based on the entity set, such as Figure 3 As shown, entities from the clinical notes of each target object are extracted using the Med7 model. Each target object corresponds to one or more entities, and these entities form an entity set. A correlation graph is constructed based on the target object entity set and the entity sets of neighboring objects, and this correlation graph is initialized as an identity matrix. In a matrix, elements excluding those on the diagonal represent relevance, such as {A}. ij |i,j∈{1,2,3,…,N s}}, where A ij This represents the relevance between the i-th target object and the j-th target object.

[0115] The correlation between the target object entity set and the adjacent object entity sets can be calculated using formula (11).

[0116]

[0117] in, This represents the entity set of the i-th target object. This represents the entity set of the j-th target object.

[0118] Step S52: Obtain the multimodal fusion features of adjacent objects.

[0119] Among them, multimodal fusion features refer to the fusion of features from multiple types of data, such as the fusion of features from time series data and text data.

[0120] In some embodiments, obtaining multimodal fusion features of neighboring objects includes: obtaining hidden feature vectors of neighboring objects based on time series data of neighboring objects; obtaining entity feature vectors and global feature vectors of neighboring objects based on text data of neighboring objects; and fusing the entity feature vectors and hidden feature vectors to obtain hidden fusion feature vectors and entity fusion feature vectors.

[0121] The extraction methods for the hidden feature vectors of neighboring objects are the same as those for the target object. The extraction methods for the entity feature vectors and global feature vectors of neighboring objects are the same as those for the target object. The extraction methods for the hidden fusion feature vectors and entity fusion feature vectors of neighboring objects are the same as those for the target object. These will not be elaborated further here.

[0122] Step S53: Determine the multimodal aggregation features of the target object based on the multimodal fusion features of the target object, the multimodal fusion features of neighboring objects, and the correlation between the target object and neighboring objects.

[0123] In some embodiments, the present disclosure uses a graph convolutional neural network (GCN) model to aggregate multimodal features of neighboring objects, for example, by using formula (12).

[0124]

[0125] in, Let σ() represent the multimodal aggregated feature, A represent the relevance, P represent the multimodal fusion feature, and W represent the multimodal aggregated feature. g and b g This represents the variable parameters in a graph convolutional neural network model.

[0126] In some embodiments, the multimodal features of the i-th target object can be calculated using formula (13).

[0127]

[0128] Where, p (i) This represents the multimodal features of the i-th target object. This represents the hidden fusion feature vector of the i-th target object. This represents the entity fusion feature vector of the i-th target object. This represents the global characteristics of the i-th target object. N represents the hidden feature vector of the i-th target object. t W represents the time step. p and bp This indicates a variable parameter.

[0129] The multimodal aggregated features of the target object are represented as follows: Where i represents the i-th target object. N represents the multimodal aggregated feature of the i-th target object. s Indicates the amount of text data.

[0130] Step S54: Predict the probability of the target object appearing in the target state based on the multimodal aggregated features of the target object.

[0131] The embodiments disclosed herein predict the probability of a target state occurring by combining multiple modal fusion features from multiple perspectives (adjacent objects), which further improves the accuracy of prediction compared to prediction from a single perspective.

[0132] In some embodiments, step S54, predicting the probability of the target object appearing in the target state based on the multimodal aggregated features of the target object, includes: concatenating the hidden feature vector and the entity feature vector of the target object to obtain the concatenated feature vector of the target object; and predicting the probability of the target object appearing in the target state based on the multimodal aggregated features of the target object and the concatenated feature vector of the target object through a neural network model.

[0133] In this embodiment of the disclosure, the hidden feature vector and the entity feature vector of the target object are concatenated, and the multimodal aggregated features and the concatenated feature vector of the target object are input into the neural network model. The neural network model uses formula (14) to predict the probability of the occurrence of the target state.

[0134]

[0135] in, This represents the probability of the target state occurring. Let C represent the hidden feature vector of the i-th target object. g Represents the global characteristics of the target object. N represents the multimodal aggregated feature of the i-th target object. t Indicates the time step.

[0136] In some embodiments, the training steps of the neural network model include: obtaining the concatenated feature vectors and multimodal aggregated features of the training samples; wherein, the concatenated feature vectors of the training samples are determined based on the hidden feature vectors and entity feature vectors of the training samples; the multimodal aggregated features of the training samples are determined based on the multimodal fusion features of multiple training samples and the correlation between different training samples; inputting the concatenated feature vectors and multimodal aggregated features of the training samples into a fully connected network model to predict the probability of the training samples reaching the target state; updating the model parameters in the fully connected network model using the stochastic gradient descent method based on the cross-entropy loss and the predicted probability of the training samples reaching the target state; and obtaining the neural network model when the termination condition is met.

[0137] The concatenated feature vectors of the training samples and the target object are obtained in the same way, as are the multimodal aggregated features of the training samples and the target object, and will not be repeated here.

[0138] In some embodiments, the cross-entropy loss is calculated using formula (15).

[0139]

[0140] Where L represents the cross-entropy loss, N s Indicates the amount of text data. y represents the probability that the i-th target object will exhibit the target state. (i) This represents the connection result of the i-th target object.

[0141] In this embodiment, Gradient Descent (SGD) is used for learning linear classifiers with loss functions such as Support Vector Machines and Logistic Regression. Randomness refers to the fact that the samples are randomly shuffled in each iteration. The SGD algorithm randomly selects a set of samples, trains it, updates it once according to the gradient, then selects another set and updates it again. With a very large sample size, it may be possible to obtain a model with an acceptable loss value without training all the samples.

[0142] In some embodiments, the multimodal aggregation features of training samples are determined based on the multimodal fusion features of the target training sample, the multimodal fusion features of neighboring training samples, and the correlation between the target training sample and neighboring training samples; wherein, neighboring training samples and target training samples belong to different training samples in the same business domain.

[0143] In some embodiments, the step of obtaining the multimodal fusion features of the training samples includes: obtaining the hidden feature vector of the training samples based on the time series data of the training samples; wherein the hidden feature vector is used to characterize the features of the training samples at different times; obtaining the entity feature vector and the global feature vector of the training samples based on the text data of the training samples; wherein the entity feature vector is used to characterize the features of entities in the text data, and the global feature vector is used to characterize the features of the training samples; fusing the entity feature vector and the hidden feature vector to obtain the hidden fusion feature vector and the entity fusion feature vector; and determining the multimodal fusion features of the training samples based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector.

[0144] In some embodiments, entity feature vectors and hidden feature vectors are fused together to obtain hidden fused feature vectors and entity fused feature vectors; including:

[0145] The attention scores of entity feature vectors and hidden feature vectors are determined based on entity feature vectors and hidden feature vectors; the entity fusion feature vector is determined based on entity feature vectors, hidden feature vectors and the attention scores of entity feature vectors; and the hidden fusion feature vector is determined based on hidden feature vectors, entity feature vectors and the attention scores of hidden feature vectors.

[0146] In some embodiments, determining the attention score of the entity feature vector and the attention score of the hidden feature vector based on the entity feature vector and the hidden feature vector includes:

[0147] Adjust the dimensions of the entity feature vector and the hidden feature vector to obtain standard entity feature vectors and standard hidden feature vectors with consistent dimensions; calculate the correlation between the standard hidden feature vectors and the standard entity feature vectors; based on the correlation between the standard hidden feature vectors and the standard entity feature vectors, the standard hidden feature vectors, and the standard entity feature vectors, determine the fused hidden feature vectors and fused entity feature vectors using an activation function; based on the fused hidden feature vectors, determine the attention score of the hidden feature vectors using an attention function, and based on the fused entity feature vectors, determine the attention score of the entity feature vectors using an attention function.

[0148] In some embodiments, determining the entity fusion feature vector based on the entity feature vector, the hidden feature vector, and the attention score of the entity feature vector includes: multiplying the entity feature vector of the training sample with the attention score of the entity feature vector to obtain a first multiplication result, and summing the first multiplication result with the hidden feature vector to obtain the entity fusion feature vector of the training sample.

[0149] In some embodiments, determining the hidden fusion feature vector based on the hidden feature vector, the entity feature vector, and the attention score of the hidden feature vector includes: multiplying the hidden feature vector of the training sample and the attention score of the hidden feature vector to obtain a second multiplication result, and summing the second multiplication result with the entity feature vector to obtain the hidden fusion feature vector.

[0150] In some embodiments, the step of determining the correlation between training samples and neighboring objects includes:

[0151] Training sample entities are extracted from the text data of the training samples to obtain a training sample entity set; wherein the training sample entity set includes one or more training sample entities; adjacent object entities are extracted from the text data of adjacent objects to obtain an adjacent object entity set; wherein the adjacent object entity set includes one or more adjacent object entities; the relevance between the training samples and adjacent objects is determined based on the training sample entities in the training sample entity set and the adjacent object entities in the adjacent object entity set.

[0152] In some embodiments, the concatenated feature vector is obtained by concatenating the hidden feature vector and the entity feature vector of the training sample; wherein the hidden feature vector of the training sample is obtained based on the time series data of the training sample, and the entity feature vector of the training sample is obtained based on the text data of the training sample.

[0153] Figure 4 This is a schematic diagram illustrating a data processing method provided in an embodiment of this disclosure. Figure 4 As shown, the data processing methods include:

[0154] Step S401: Obtain the hidden feature vector of the target object using GRU based on the time series data of the target object.

[0155] Step S402: Based on the text data of the target object, obtain the global feature vector of the target object through TF-IDF.

[0156] Step S403: Extract entities from the text data of the target objects using the Med7 model to obtain an entity set for each target object.

[0157] Step S404: Extract entity feature vectors from the entity set using pre-trained BioBERT.

[0158] Step S405: Through common attention, the entity feature vector is fused with the hidden feature vector to obtain the entity fused feature vector, and the hidden feature vector is fused with the entity feature vector to obtain the hidden fused feature vector.

[0159] Step S406: Determine the multimodal fusion features of the target object based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector.

[0160] Step S407: Construct a correlation graph based on the text data of the target object and the text data of adjacent objects, and determine the correlation between the target object and adjacent objects based on the text data of the target object and the text data of adjacent objects.

[0161] Step S408: Determine the multimodal aggregation features of the target object based on the multimodal fusion features of the target object, the multimodal fusion features of neighboring objects, and the correlation between the target object and neighboring objects.

[0162] Step S409: Concatenate the hidden feature vector and the entity feature vector of the target object to obtain the concatenated feature vector of the target object;

[0163] Step S410: Based on the multimodal aggregated features of the target object and the concatenated feature vector of the target object, the probability of the target object appearing in the target state is predicted by the FFNN model.

[0164] For example, taking the consumer finance sector and targeting customers to predict repayment probability, the call status between the customer and the agent is time-series data, while the call log is text data. The call status exhibits certain patterns of change with the overdue period. For instance, call status includes connected, hung up, and unanswered. When the overdue period is short, the customer usually answers, resulting in a connected status. When the overdue period is long, the customer usually hangs up, resulting in a hung-up status. The longer the overdue period, the less likely the customer is to answer, resulting in an unanswered status. Furthermore, there is also a certain pattern between the total overdue amount and the call status. For example, the higher the total overdue amount, the higher the probability of a hung-up or unanswered call. The call log consists of the content of the conversation between the customer and the agent.

[0165] Hidden feature vectors are extracted from call states with temporal characteristics using a bidirectional GRU, and global feature vectors are extracted from call records using TF-IDF. Entities such as "promised repayment," "refused repayment," "excuse for work," "excuse for business trip," and "excuse for illness" are extracted from call records using a Med7 model, and entity feature vectors are obtained based on these entities using a BioBERT model. A co-attention model is used to fuse the hidden and entity feature vectors to obtain entity-fused feature vectors and hidden-fused feature vectors. Then, the multimodal fusion features of the target object are determined based on the hidden-fused feature vectors, entity-fused feature vectors, and global feature vectors. A customer relevance graph is constructed based on the entities. The multimodal aggregation features of the target object are determined based on the multimodal fusion features of the target object, the multimodal fusion features of neighboring objects, and the relevance between the target object and neighboring objects. Based on the multimodal aggregation features, global feature vectors, and hidden feature vectors, the probability of customer repayment is obtained using an FFNN model. In some embodiments, based on the output of the FFNN model, the probability of customer repayment is predicted by combining basic customer information, contract information, and credit information (obtained through legal means).

[0166] For example, taking the consumer finance sector and targeting customers, let's consider predicting the probability of customer complaints. Debt collectors will take different measures depending on the degree of delinquency, such as mentioning the impact of delinquency on credit, early full repayment, and legal risks. During communication with customers, they may express dissatisfaction with the lending company or debt collectors, or even file complaints. Using the data processing method provided in this disclosure, the probability of customer complaints can be predicted, allowing for proactive reassurance or adjustment of collection strategies for customers with a tendency to complain.

[0167] The call status between the customer and the agent is presented as time-series data, while the call log is presented as text data. The call status exhibits certain patterns of change with the overdue period. For example, call status includes connected, hung up, and unanswered. When the overdue period is short, the customer usually answers, and the call status is "connected." When the overdue period is long, the customer usually hangs up, and the call status is "hung up." The longer the overdue period, the less likely the customer is to answer, and the call status becomes "unanswered." Furthermore, there is a certain pattern between the total overdue amount and the call status. For instance, the higher the total overdue amount, the higher the probability of the call status being "hung up" or "unanswered." The call log consists of the content of the conversation between the customer and the agent.

[0168] Hidden feature vectors are extracted from call states with temporal characteristics using a bidirectional GRU, and global feature vectors are extracted from call logs using TF-IDF. Entities such as "protesting violent debt collection," "threatening to report," "threatening to complain," and "complaining to the China Banking Regulatory Commission" are extracted from call logs using a Med7 model, and entity feature vectors are obtained based on these entities using a BioBERT model. The hidden feature vectors and entity feature vectors are fused using a co-attention model to obtain entity fused feature vectors and hidden fused feature vectors. Then, the multimodal fused features of the target object are determined based on the hidden fused feature vectors, entity fused feature vectors, and global feature vectors.

[0169] Based on entity-based customer relevance graphs, customers with a tendency to complain will mention relevant terms such as "regulatory," "CBRC," "report to the police," "not accepting," and "violent debt collection." Customers with a high probability of complaints exhibit similar multimodal characteristics during conversations. Therefore, based on the multimodal fusion features of the target object, the multimodal fusion features of neighboring objects, and the relevance between the target object and neighboring objects, the multimodal aggregation features of the target object are determined. Based on the multimodal aggregation features, global feature vectors, and hidden feature vectors, the probability of customer complaints is obtained through an FFNN model. In some embodiments, based on the output of the FFNN model, the probability of customer complaints is predicted by combining basic customer information, contract information, and credit information (obtained through legal means).

[0170] For example, in the medical field, targeting patients, we can predict the probability of a patient's length of stay in the ICU. Physiological data from medical equipment is used as time-series data, and clinical notes are used as text data.

[0171] Hidden feature vectors were extracted from temporally sequenced physiological data using a bidirectional GRU, and global feature vectors were extracted from clinical notes using TF-IDF. Entities in the clinical notes were extracted using a Med7 model, and entity feature vectors were obtained based on these entities using a BioBERT model. A co-attention model was used to fuse the hidden and entity feature vectors to obtain entity-fused feature vectors and hidden-fused feature vectors. Then, the patient's multimodal fusion features were determined based on the hidden-fused feature vectors, entity-fused feature vectors, and global feature vectors.

[0172] A patient correlation map is constructed based on entities in the clinical notes of different patients. The correlation between the target patient and adjacent patients is determined based on the clinical notes of the target patient and adjacent patients. Then, the multimodal aggregation features of the target patient are determined based on the multimodal fusion features of the target patient, the multimodal fusion features of adjacent patients, and the correlation between the target patient and adjacent patients. Based on the multimodal aggregation features, global feature vectors, and hidden feature vectors, the probability of the target patient's stay in the ICU (3 days or 5 days) is determined by an FFNN model.

[0173] Table 1 shows the probability of predicting a target state using the data processing method and baseline method provided in embodiments of this disclosure. Table 1 shows the inpatient mortality rate, ICU mortality rate, LOS>3, and LOS>7.

[0174] 1. In-hospital mortality: Used to predict whether a patient will die before being discharged from the hospital.

[0175] 2. In-ICU mortality: Used to detect patients whose physical condition is declining and to predict their mortality rate within 24 hours.

[0176] 3. LOS>3 days: Used to predict whether a patient's stay in the ICU will exceed 3 days.

[0177] 4. LOS>7 days: Used to predict patients who stay in the ICU for more than 7 days.

[0178] Table 1

[0179]

[0180] As shown in Table 1, the experimental results obtained by the data processing method provided in this disclosure achieve the best results on the four prediction tasks compared to the baseline method. The Area Under the Receiver Operating Characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) scores are increased by 1.1%-3.7% and 0.4%-10.5% respectively compared to the baseline for the four tasks. The embodiments of this disclosure fully utilize the fusion of multi-dimensional and multi-modal features from patients and use a Co-Attention module to effectively fuse these features. Furthermore, clinical notes are used to construct patient correlation maps, and GCN is used to aggregate multi-modal features, improving the generalization ability in predicting clinical outcomes.

[0181] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0182] The embodiments provided in this disclosure obtain a hidden feature vector of the target object based on the time series data of the target object, and obtain an entity feature vector and a global feature vector of the target object based on the text data of the target object. The entity feature vector is fused with the hidden feature vector to obtain an entity fused feature vector, which contains information from both the text data and the time series data. The hidden feature vector is fused with the entity feature vector to obtain a hidden fused feature vector, which contains information from both the time series data and the text data. The multimodal fused features of the target object are determined based on the hidden fused feature vector, the entity fused feature vector, and the global feature vector. That is, the multimodal fused features of the target object contain information from both the time series data and the entity and global information from the text data. Therefore, the probability of the target object appearing in the target state is predicted based on the multimodal fused features of the target object.

[0183] Secondly, embodiments of this disclosure also provide a data processing apparatus.

[0184] Figure 5 This is a schematic diagram of a data processing apparatus provided in an embodiment of this disclosure. Figure 5 As shown, the data processing device 500 includes:

[0185] The first acquisition module 501 is used to acquire the hidden feature vector of the target object based on the time series data of the target object; wherein, the hidden feature vector is used to characterize the features of the time series data at different times;

[0186] The second acquisition module 502 is used to acquire the entity feature vector and global feature vector of the target object based on the text data of the target object; wherein, the entity feature vector is used to characterize the features of the entities in the text data, and the global feature vector is used to characterize the features of the text data.

[0187] The fusion module 503 is used to fuse the entity feature vector with the hidden feature vector to obtain an entity fused feature vector, and to fuse the hidden feature vector with the entity feature vector to obtain a hidden fused feature vector.

[0188] The determination module 504 is used to determine the multimodal fusion features of the target object based on the hidden fusion feature vector, the entity fusion feature vector and the global feature vector;

[0189] The prediction module 505 is used to predict the probability of the target object appearing in the target state based on the multimodal fusion features of the target object.

[0190] In some embodiments, the probability of a target object appearing in a target state is predicted by multimodal fusion features of the target object and neighboring objects, wherein the target object and neighboring objects belong to the same business domain.

[0191] The prediction module 505 is also used to determine the correlation between the target object and its neighboring objects based on the text data of the target object and the text data of its neighboring objects; to obtain the multimodal fusion features of the neighboring objects; to determine the multimodal aggregation features of the target object based on the multimodal fusion features of the target object, the multimodal fusion features of the neighboring objects, and the correlation between the target object and its neighboring objects; and to predict the probability of the target object appearing in the target state based on the multimodal aggregation features of the target object.

[0192] In some embodiments, the prediction module 505 is further configured to extract target object entities from the text data of the target object to obtain a target object entity set; wherein the target object entity set includes one or more target object entities; extract neighboring object entities from the text data of neighboring objects to obtain a neighboring object entity set; wherein the neighboring object entity set includes one or more neighboring object entities; and determine the relevance between the target object and neighboring objects based on the target object entities in the target object entity set and the neighboring object entities in the neighboring object entity set.

[0193] In some embodiments, the prediction module 505 is further configured to obtain the hidden feature vectors of neighboring objects based on the time series data of neighboring objects; obtain the entity feature vectors and global feature vectors of neighboring objects based on the text data of neighboring objects; and fuse the entity feature vectors and hidden feature vectors to obtain hidden fused feature vectors and entity fused feature vectors.

[0194] In some embodiments, the prediction module 505 is further configured to concatenate the hidden feature vector and the entity feature vector of the target object to obtain the concatenated feature vector of the target object; and based on the multimodal aggregated features of the target object and the concatenated feature vector of the target object, predict the probability of the target object appearing in the target state through a neural network model.

[0195] In some embodiments, the data processing apparatus further includes: a training module for a neural network model, the training module being used to acquire multiple training samples; wherein each training sample includes a concatenated feature vector and a multimodal aggregated feature; wherein the concatenated feature vector of the training sample is determined based on the hidden feature vector and the entity feature vector of the training sample; the multimodal aggregated feature of the training sample is determined based on the multimodal fusion feature of multiple training samples and the correlation between multiple training samples; the concatenated feature vector and the multimodal aggregated feature of the training samples are input into a fully connected network model; the model parameters in the fully connected network model are updated using a stochastic gradient descent method based on the cross-entropy loss and the predicted probability of the training sample appearing in the target state; and a neural network model is obtained when the termination condition is met.

[0196] In some embodiments, the fusion module 503 is further configured to determine the attention score of the entity feature vector and the attention score of the hidden feature vector based on the entity feature vector and the hidden feature vector; determine the entity fusion feature vector based on the entity feature vector and the attention score of the entity feature vector; and determine the hidden fusion feature vector based on the hidden feature vector and the attention score of the hidden feature vector.

[0197] In some embodiments, the fusion module 503 is further configured to adjust the dimension of the entity feature vector and the dimension of the hidden feature vector to obtain a standard entity feature vector and a standard hidden feature vector with consistent dimensions; calculate the correlation between the standard hidden feature vector and the standard entity feature vector; determine the fused hidden feature vector and the fused entity feature vector based on the correlation between the standard hidden feature vector and the standard entity feature vector, the standard hidden feature vector, and the standard entity feature vector, and using an activation function; determine the attention score of the hidden feature vector based on the fused hidden feature vector and using an attention function, and determine the attention score of the entity feature vector based on the fused entity feature vector and using an attention function.

[0198] In some embodiments, the determining module 504 is further configured to concatenate the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector to obtain the multimodal fusion feature of the target object.

[0199] The data processing apparatus provided in this embodiment includes a first acquisition module that acquires a hidden feature vector of the target object based on time-series data of the target object, a second acquisition module that acquires an entity feature vector and a global feature vector of the target object based on text data of the target object, a fusion module that fuses the entity feature vector with the hidden feature vector to obtain an entity fusion feature vector, such that the entity fusion feature vector contains information from both the text data and the time-series data; a determination module that fuses the hidden feature vector with the entity feature vector to determine the hidden fusion feature vector, such that the hidden fusion feature vector contains information from both the time-series data and the text data; and a multimodal fusion feature of the target object is determined based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector, meaning that the multimodal fusion feature of the target object contains information from both the time-series data and the entity and global information from the text data, enabling the prediction module to predict the probability of the target object appearing in the target state based on the multimodal fusion feature of the target object, thereby improving the accuracy of the prediction.

[0200] In addition, this disclosure also provides an electronic device and a computer-readable storage medium, all of which can be used to implement any of the data processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section and will not be repeated here.

[0201] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0202] Reference Figure 6 This disclosure provides an electronic device, which includes: at least one processor 601; at least one memory 602; and one or more I / O interfaces 603 connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs that can be executed by the at least one processor 601, and the one or more computer programs are executed by the at least one processor 601 to enable the at least one processor 601 to perform the above-described data processing method.

[0203] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the data processing method described above. The computer-readable storage medium may be volatile or non-volatile.

[0204] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described data processing method.

[0205] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0206] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0207] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0208] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0209] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0210] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0211] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0212] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0213] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0214] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A data processing method, characterized in that, include: The hidden feature vector of the target object is obtained based on the time series data of the target object; wherein, the hidden feature vector is used to characterize the features of the time series data at different times; Based on the text data of the target object, obtain the entity feature vector and global feature vector of the target object; wherein, the entity feature vector is used to characterize the features of the entities in the text data, and the global feature vector is used to characterize the features of the text data; The entity feature vector is fused with the hidden feature vector to obtain an entity fused feature vector, and the hidden feature vector is fused with the entity feature vector to obtain a hidden fused feature vector; The multimodal fusion features of the target object are determined based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector; The probability of the target object appearing in the target state is predicted based on the multimodal fusion features of the target object.

2. The method according to claim 1, characterized in that, The prediction of the probability of the target object exhibiting a target state based on the multimodal fusion features of the target object includes: The relevance between the target object and its neighboring objects is determined based on the text data of the target object and the text data of its neighboring objects; the target object and its neighboring objects belong to the same business domain. Obtain the multimodal fusion features of the adjacent objects; The multimodal aggregation features of the target object are determined based on the multimodal fusion features of the target object, the multimodal fusion features of the neighboring objects, and the correlation between the target object and the neighboring objects. Predict the probability of the target object exhibiting the target state based on the multimodal aggregated features of the target object.

3. The method according to claim 2, characterized in that, Determining the relevance between the target object and its neighboring objects based on the text data of the target object and the text data of its neighboring objects includes: Extract target object entities from the text data of the target object to obtain a target object entity set; wherein, the target object entity set includes one or more target object entities; The adjacent object entities are extracted from the text data of the adjacent objects to obtain the set of adjacent object entities; wherein, the set of adjacent object entities includes one or more adjacent object entities; The relevance between the target object and the adjacent objects is determined based on the target object entities in the target object entity set and the adjacent object entities in the adjacent object entity set.

4. The method according to claim 2, characterized in that, The acquisition of the multimodal fusion features of the adjacent objects includes: The hidden feature vectors of the neighboring objects are obtained based on the time series data of the neighboring objects. Based on the text data of the adjacent objects, obtain the entity feature vector and global feature vector of the adjacent objects; The entity feature vector and the hidden feature vector are fused together to obtain a hidden fused feature vector and an entity fused feature vector.

5. The method according to claim 2, characterized in that, The prediction of the probability of the target object exhibiting a target state based on the multimodal aggregated features of the target object includes: The hidden feature vector and the entity feature vector of the target object are concatenated to obtain the concatenated feature vector of the target object; Based on the multimodal aggregated features and the concatenated feature vector of the target object, the probability of the target object appearing in the target state is predicted by a neural network model.

6. The method according to claim 5, characterized in that, The training steps of the neural network model include: Obtain the concatenated feature vector of the training samples and the multimodal aggregated feature of the training samples; wherein, the concatenated feature vector of the training samples is determined based on the hidden feature vector and the entity feature vector of the training samples; the multimodal aggregated feature of the training samples is determined based on the multimodal fusion feature of multiple training samples and the correlation between different training samples; The concatenated feature vector of the training sample and the multimodal aggregated feature of the training sample are input into a fully connected network model to predict the probability of the training sample appearing in the target state; The model parameters in the fully connected network model are updated using the stochastic gradient descent method based on the cross-entropy loss and the predicted probability of the training sample appearing in the target state. The neural network model is obtained when the termination condition is met.

7. The method according to claim 1, characterized in that, The step of fusing the entity feature vector with the hidden feature vector to obtain an entity fused feature vector, and fusing the hidden feature vector with the entity feature vector to obtain a hidden fused feature vector, includes: The attention score of the entity feature vector and the attention score of the hidden feature vector are determined based on the entity feature vector and the hidden feature vector. The entity fusion feature vector is determined based on the entity feature vector, the hidden feature vector, and the attention score of the entity feature vector; The hidden fusion feature vector is determined based on the hidden feature vector, the entity feature vector, and the attention score of the hidden feature vector.

8. The method according to claim 7, characterized in that, The step of determining the attention score of the entity feature vector and the attention score of the hidden feature vector based on the entity feature vector and the hidden feature vector includes: Adjust the dimension of the entity feature vector and the dimension of the hidden feature vector to obtain a standard entity feature vector and a standard hidden feature vector with the same dimension; Calculate the correlation between the standard hidden feature vector and the standard entity feature vector; Based on the correlation between the standard hidden feature vector and the standard entity feature vector, the standard hidden feature vector, and the standard entity feature vector, the fused hidden feature vector and the fused entity feature vector are determined using an activation function; Based on the fused hidden feature vector, the attention score of the hidden feature vector is determined using an attention function, and based on the fused entity feature vector, the attention score of the entity feature vector is determined using an attention function.

9. A data processing apparatus, characterized in that, include: The first acquisition module is used to acquire the hidden feature vector of the target object based on the time series data of the target object; wherein, the hidden feature vector is used to characterize the features of the time series data at different times; The second acquisition module is used to acquire the entity feature vector and global feature vector of the target object based on the text data of the target object; wherein, the entity feature vector is used to characterize the features of the entities in the text data, and the global feature vector is used to characterize the features of the text data; The fusion module is used to fuse the entity feature vector with the hidden feature vector to obtain an entity fused feature vector, and to fuse the hidden feature vector with the entity feature vector to obtain a hidden fused feature vector; The determination module is used to determine the multimodal fusion features of the target object based on the hidden fusion feature vector, the entity fusion feature vector, and the global feature vector; The prediction module is used to predict the probability of the target object appearing in the target state based on the multimodal fusion features of the target object.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the data processing method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1-8.

Images