Task prediction methods, devices, equipment and storage media

Abstract

This application discloses a task prediction method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology. The method includes: acquiring training samples of a task prediction model, the training samples including feature data of n modalities of a sample individual; extracting feature representation information of the sample individual based on the feature data of the n modalities using the task prediction model, and determining the task prediction result corresponding to the sample individual based on the feature representation information; determining the model training loss based on the mutual information between the n modalities and the accuracy of the task prediction result; adjusting the parameters of the task prediction model based on the model training loss to obtain a trained task prediction model, which is then used for task prediction. In this embodiment, the task prediction model is oriented towards feature data of multiple modalities, enabling the embedding of more complete semantic information during the task prediction process, thus helping to improve the accuracy of task prediction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a task prediction method, apparatus, device, and storage medium. Background Technology

[0002] Artificial intelligence (AI) technology has been researched and applied in many fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service.

[0003] Machine learning (ML) is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications span all areas of artificial intelligence. By training machine learning models with massive amounts of data and then using these models to make predictions for specific tasks, it can assist or replace human judgment on specific tasks, effectively avoiding errors caused by insufficient human knowledge or limited human effort, and improving the accuracy of task judgment. In related technologies, because data from different modalities comes from different sources or in different formats, it is usually impossible to process data from different modalities uniformly. Therefore, machine learning is used to perform predictions on specific tasks based on individual, single-modal data.

[0004] However, since unimodal data often cannot fully represent the semantics of an individual, the accuracy of predictions for tasks performed on unimodal data is also greatly reduced. Summary of the Invention

[0005] This application provides a task prediction method, apparatus, device, and storage medium that are designed for feature data of multiple modalities. It enables the embedding of more complete semantic information during task prediction, thereby improving the accuracy of task prediction. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a task prediction method, the method comprising:

[0007] Obtain training samples for the task prediction model, wherein the training samples include feature data of n modalities of an individual sample, where n is an integer greater than 1;

[0008] Using the task prediction model, based on the feature data of the n modalities, feature representation information of the sample individual is extracted, and based on the feature representation information, the task prediction result corresponding to the sample individual is determined; the feature representation information includes feature representation vectors corresponding to the n modalities respectively;

[0009] The model training loss is determined based on the mutual information among the n modalities and the accuracy of the task prediction results;

[0010] Based on the training loss of the model, the parameters of the task prediction model are adjusted to obtain a fully trained task prediction model, which is then used for task prediction.

[0011] On the other hand, embodiments of this application provide a task prediction method, the method comprising:

[0012] Obtain feature data of m modalities of the target individual, where m is an integer greater than 1;

[0013] Using a task prediction model, feature representation information of the target individual is extracted based on the feature data of the m modalities. The feature representation information includes feature representation vectors corresponding to the m modalities respectively.

[0014] A global max pooling operation is used to fuse the feature representation vectors corresponding to the m modalities to obtain the fused feature representation vector of the target individual;

[0015] Based on the fused feature representation vector, the task prediction result corresponding to the target sample is determined using the task prediction model.

[0016] Furthermore, embodiments of this application provide a task prediction apparatus, the apparatus comprising:

[0017] The sample acquisition module is used to acquire training samples for the task prediction model. The training samples include feature data of n modalities of individual samples, where n is an integer greater than 1.

[0018] The task prediction module is used to extract feature representation information of the sample individual based on the feature data of the n modalities through the task prediction model, and to determine the task prediction result corresponding to the sample individual based on the feature representation information; the feature representation information includes feature representation vectors corresponding to the n modalities respectively;

[0019] The loss determination module is used to determine the model training loss based on the mutual information between the n modalities and the accuracy of the task prediction results;

[0020] The parameter adjustment module is used to adjust the parameters of the task prediction model based on the model training loss to obtain a completed task prediction model, which is used for task prediction.

[0021] In another aspect, embodiments of this application provide a task prediction apparatus, the apparatus comprising:

[0022] The data acquisition module is used to acquire feature data of m modalities of the target individual, where m is an integer greater than 1;

[0023] The feature extraction module is used to extract feature representation information of the target individual based on the feature data of the m modalities through the task prediction model. The feature representation information includes feature representation vectors corresponding to the m modalities respectively.

[0024] The feature fusion module is used to fuse the feature representation vectors corresponding to the m modalities by using a global max pooling operation to obtain the fused feature representation vector of the target individual.

[0025] The task prediction module is used to determine the task prediction result corresponding to the target sample based on the fused feature representation vector through the task prediction model.

[0026] In another aspect, embodiments of this application provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-described task prediction method.

[0027] In another aspect, embodiments of this application provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described task prediction method.

[0028] Furthermore, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned task prediction method.

[0029] The technical solutions provided in this application have at least the following beneficial effects:

[0030] By training a task prediction model, a fully trained task prediction model is obtained. This model is then used for subsequent task prediction, realizing a method for task prediction using artificial intelligence. This method can assist or replace manual task identification, helping to reduce labor costs. Furthermore, in this embodiment, the task prediction model is designed for feature data from multiple modalities, enabling the embedding of more complete semantic information during task prediction, thus improving accuracy. Additionally, in this embodiment, the training of the task prediction model references mutual information between multiple modalities, ensuring consistency in data representation across modalities. It also embeds useful features into the feature representation vector while discarding useless features, enhancing robustness against interference. Moreover, in this embodiment, the task prediction model is trained using a joint approach of multimodal mutual information and the accuracy of task prediction results, achieving the embedding of task-related semantic features into the feature representation vector while preserving individual features as much as possible. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of a task prediction system provided in one embodiment of this application;

[0033] Figure 2 This is a flowchart of a task prediction method provided in one embodiment of this application;

[0034] Figure 3 This is a schematic diagram of the prediction result of a task prediction model provided in one embodiment of this application;

[0035] Figure 4 This is a schematic diagram of a task prediction method provided in one embodiment of this application;

[0036] Figure 5 This is a flowchart of a task prediction method provided in another embodiment of this application;

[0037] Figure 6 This is a block diagram of a task prediction model provided in one embodiment of this application;

[0038] Figure 7 This is a block diagram of a task prediction model provided in another embodiment of this application;

[0039] Figure 8This is a block diagram of a task prediction model provided in another embodiment of this application;

[0040] Figure 9 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0042] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0043] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0044] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0045] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0046] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0047] The technical solutions provided in this application involve artificial intelligence technologies such as natural language processing and machine learning / deep learning, which are specifically illustrated in the following embodiments.

[0048] Before introducing the technical solutions provided in this application, some technical terms involved in the embodiments of this application will be briefly introduced.

[0049] Electronic Health Records (EHRs): A medical record is the original record of a patient's entire diagnostic and treatment process in a hospital. It includes a cover sheet, progress notes, examination and test results, doctor's orders, surgical records, nursing records, etc. An EHR is information about an individual's lifelong health status and healthcare behaviors managed electronically, encompassing all processes related to the collection, storage, transmission, processing, and use of patient information. In some examples, an EHR can be defined as: an electronic patient record based on a specific system that provides users with access to complete and accurate data, alerts, prompts, and clinical decision-making capabilities.

[0050] Modal: Data is the foundation of AI, and data from different industries comes from a wide range of sources and takes many forms. Each source or form can be considered a modal, such as video, images, voice, and sensor data, infrared, and acoustic spectra in industrial scenarios.

[0051] Representation Learning: In machine learning, representation learning is a set of techniques that allow systems to automatically discover the representations needed for feature detection or classification from raw data. Representation learning replaces manual feature engineering, allowing machine learning to use features and perform specific tasks. It typically involves transforming high-dimensional data into low-dimensional vector representations.

[0052] Mutual information: Mutual information is a useful information measure in information theory. It can be seen as the amount of information contained in one random variable about another random variable, or the amount of uncertainty of one random variable reduced by knowing another random variable.

[0053] Please refer to Figure 1 This illustration shows an embodiment of a task prediction system provided in one embodiment of the present application. The task prediction system may include: a first computer device 110 and a second computer device 120.

[0054] Computer equipment refers to devices with data analysis and processing functions and / or storage functions, such as PCs (Personal Computers) and servers with data storage and / or data processing capabilities, or terminals such as mobile phones, tablets, multimedia playback devices, and wearable devices. It can also be other computer equipment, and this application does not limit the specific type. Optionally, when the computer equipment is a server, it can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0055] In this embodiment, a computer device is used to train and / or use a task prediction model for task prediction. Optionally, the task prediction model is the aforementioned machine learning model / deep learning model. It should be understood that the training device and the usage device of the task prediction model can be the same computer device; that is, the trained task prediction model is used to perform task prediction on the computer device where the task prediction model is trained, or the task prediction model is trained on the computer device where task prediction needs to be performed. Alternatively, the training device and the usage device of the task prediction model can be at least two different computer devices; that is, the training device of the task prediction model is at least one computer device, and the usage device of the task prediction model is another one or more computer devices.

[0056] Typically, training a task prediction model requires analyzing and processing massive amounts of data, thus placing high demands on the performance of the computer equipment. However, using the task prediction model does not require such high performance. Therefore, in this embodiment, the training and usage devices for the task prediction model are implemented using at least two different computer devices, but this does not constitute a limitation of this application. It should be understood that the computer device used to train the task prediction model can also use the trained task prediction model to perform task prediction.

[0057] like Figure 1 As shown, the computer equipment includes a first computer device 110 and a second computer device 120; wherein the first computer device 110 is used to train a task prediction model, and the second computer device 120 is used to perform task prediction using the trained task prediction model. Optionally, the first computer device 110 and the second computer device 120 are implemented as the same type of computer device, such as both being implemented as personal computers; or, the first computer device 110 and the second computer device 120 are implemented as different types of computer devices, such as the first computer device 110 being implemented as a server and the second computer device 120 being implemented as a terminal.

[0058] In one example, such as Figure 1 As shown, the first computer device 110 can acquire a training sample set containing multiple training samples, and train the task prediction model using the training sample set to obtain a trained task prediction model; the second computer device 120 can acquire the trained task prediction model, and use the trained task prediction model to perform task prediction to obtain a task prediction result. Optionally, the first computer device 110 and the second computer device 120 communicate with each other through a network, which can be a wired network or a wireless network.

[0059] It should be noted that the method for performing task prediction using a task prediction model provided in this application can be applied to multiple fields, such as smart healthcare, autonomous driving, smart marketing, robotics, education, gaming, driverless cars, smart homes, and smart customer service. For example, in the field of smart healthcare, the task prediction model can predict tasks such as a patient's condition, hospitalization duration, mortality risk, and treatment plan based on the patient's electronic medical records; in the field of autonomous driving, the task prediction model can predict tasks such as vehicle driving time, driving plan, driving route, and congestion risk based on vehicle performance parameters and destination information; and in the field of smart marketing, the task prediction model can predict tasks such as marketing direction, expected returns, and marketing risks based on marketing data. The examples in the following embodiments are only illustrative of the smart healthcare field and should not be construed as limiting the technical solutions provided in this application.

[0060] Please refer to Figure 2 The diagram illustrates a flowchart of a task prediction method provided in one embodiment of this application. This task prediction method can be applied to... Figure 1 In the task prediction system shown, if applied to Figure 1 In the first computer device 110 shown. For example... Figure 2 As shown, the task prediction method includes at least some of the following steps (steps 210 to 240).

[0061] Step 210: Obtain training samples for the task prediction model. The training samples include feature data of n modalities of the sample individuals, where n is an integer greater than 1.

[0062] This application embodiment achieves task prediction through a task prediction model, which is a machine learning model / neural network model. For example, the task prediction model is applied in the field of intelligent healthcare, and the task predictions performed by the task prediction model include, but are not limited to: disease progression prediction, hospitalization duration prediction, mortality risk prediction, and treatment plan prediction. To ensure the accuracy of the task prediction model, it needs to be trained.

[0063] The computer device first needs to acquire training samples for training the task prediction model. Optionally, the computer device acquires a training sample set corresponding to the task prediction model, which includes at least one training sample. This application embodiment does not limit the number of training samples acquired by the computer device. In practical applications, the number of training samples can be determined based on the accuracy requirements of the task prediction model, the processing power of the computer device, etc. Optionally, one training sample corresponds to one individual sample.

[0064] The training samples include feature data for n modalities of individual samples, where n is an integer greater than 1. For example, in the application of the task prediction model to the field of intelligent healthcare, the individual samples include patients' electronic medical records, and the feature data for multiple modalities includes at least one of the following: feature data for text modalities (e.g., text data entered by doctors), feature data for numerical modalities (e.g., examination and test indicators), and feature data for image modalities (e.g., medical images). Optionally, the training samples also include the task identification results corresponding to the individual samples, which typically refer to the results obtained by manually performing the task. For example, in the application of the task prediction model to the field of intelligent healthcare, if the task to be performed includes hospitalization duration assessment, the task identification result includes hospitalization duration; if the task to be performed includes mortality risk prediction, the task identification result includes the magnitude of mortality risk or whether there is a mortality risk; if the task to be performed includes disease diagnosis, the task identification result includes the disease condition.

[0065] Step 220: Using the task prediction model, based on the feature data of n modalities, extract the feature representation information of individual samples, and determine the task prediction result corresponding to the individual samples based on the feature representation information; the feature representation information includes the feature representation vectors corresponding to the n modalities respectively.

[0066] After obtaining training samples for training the task prediction model, the computer device can call the task prediction model to process the training samples. In this embodiment, the computer device uses the task prediction model to extract feature representation vectors corresponding to the n modalities based on the feature data of the n modalities, thereby obtaining the feature representation information of the individual sample; and further determines the task prediction result corresponding to the individual sample based on the feature representation information of the individual sample using the task prediction model.

[0067] Based on the steps performed by the task prediction model, it can be seen that in this embodiment, the task prediction model mainly performs two processes: extracting feature representation information and performing task prediction. Optionally, the task prediction model includes two parts of a learning network: one part is used to extract feature representation information, and the other part is used to perform task prediction. Of course, the task prediction model may also include only one learning network, which is used not only to extract feature representation information but also to perform task prediction; this embodiment does not limit this. For other descriptions of the model structure of the task prediction model, please refer to the following embodiments, which will not be repeated here.

[0068] Optionally, the computer device directly inputs the feature data of the n modalities of the individual samples in the training samples into the task prediction model to obtain the feature representation information of the individual samples; or, the computer device first preprocesses the feature data of the n modalities of the individual samples in the training samples and inputs the preprocessed data into the task prediction model to obtain the feature representation information of the individual samples. Optionally, after extracting the feature representation information of the individual samples, the computer device can directly input the feature representation information of the individual samples into the task prediction model to obtain the task prediction result corresponding to the individual samples; or, the computer device first reprocesses the feature representation information of the individual samples and inputs the reprocessed data into the task prediction model to obtain the task prediction result corresponding to the individual samples. The specific implementation of step 220 is not limited in this application embodiment. For other descriptions of step 220, please refer to the following embodiments, which will not be elaborated here.

[0069] Step 230: Determine the model training loss based on the mutual information among the n modalities and the accuracy of the task prediction results.

[0070] During the training of a task prediction model, the parameters of the model need to be adjusted based on its training loss. Therefore, the computer device needs to determine the training loss of the task prediction model. In this embodiment, the training loss of the task prediction model is determined based on at least two pieces of information: the mutual information among the n modalities and the accuracy of the task prediction result. Optionally, the computer device can perform one or more processing on the mutual information among the n modalities and the accuracy of the task prediction result, such as summation, weighted summation, averaging, difference, or modulo, to obtain the training loss of the task prediction model. This embodiment does not limit this process. For further explanation regarding the calculation of the training loss, please refer to the following embodiments, which will not be elaborated here.

[0071] As explained above, mutual information can be viewed as the amount of information contained in one random variable about another, or the reduced uncertainty of one random variable due to the knowledge of another. Therefore, by combining the mutual information among n modalities to train the task prediction model, the consistency of data representation across the n modalities can be ensured. Furthermore, useful features can be embedded into the feature representation vector while useless features are discarded, enhancing robustness against interference. This application's embodiments also achieve the goal of embedding task-related semantic features into the feature representation vector while preserving individual features as much as possible by combining the mutual information among the n modalities and the accuracy of the task prediction results.

[0072] Step 240: Based on the model training loss, adjust the parameters of the task prediction model to obtain a completed task prediction model, which is then used for task prediction.

[0073] The computer device adjusts the parameters of the task prediction model based on the model training loss until the model training loss converges, resulting in a fully trained task prediction model. The computer device can then use the fully trained task prediction model to perform task prediction; alternatively, other computer devices can use the fully trained task prediction model obtained from the computer device to perform task prediction. Optionally, if the task prediction model includes multiple learning networks, the computer device jointly adjusts the parameters of the multiple learning networks based on the model training loss.

[0074] To further improve the prediction accuracy of the task prediction model, in one example, the computer device can also validate the trained task prediction model using test samples. For example, the computer device acquires test samples of the task prediction model, which include feature data of n modalities for a sample individual. Using the trained task prediction model, based on the feature data of the n modalities, it extracts feature representation information of the sample individual and determines the task prediction result corresponding to the sample individual based on the feature representation information. Based on the task prediction result and the task identification result corresponding to the sample individual, the accuracy of the trained task prediction model is validated. Optionally, the computer device acquires test samples and training samples in a certain ratio, for example, in any ratio of 8:2, 7:3, or 6:4. Optionally, the computer device can perform multiple rounds of training on the task prediction model, validating the trained task prediction model after each round of training. If the validation results do not improve after x consecutive validations (x is a positive integer, such as 5), the model parameters corresponding to the best validation result among these x validations are retained.

[0075] In summary, the technical solution provided in this application, by training a task prediction model to obtain a trained task prediction model, and then using this trained model for subsequent task prediction, realizes a method for task prediction using artificial intelligence. This method can assist or replace manual task identification, helping to reduce labor costs. Furthermore, in this application, the task prediction model is oriented towards feature data from multiple modalities, enabling the embedding of more complete semantic information during task prediction, thus improving the accuracy of task prediction. Additionally, in this application, the training of the task prediction model references mutual information between multiple modalities, ensuring consistency in data representation across multiple modalities. It also embeds useful features into the feature representation vector while discarding useless features, enhancing anti-interference capabilities. Moreover, in this application, the task prediction model is trained using a joint approach of mutual information between multiple modalities and the accuracy of task prediction results, achieving the embedding of task-related semantic features into the feature representation vector while preserving individual features as much as possible.

[0076] In one example, the task prediction model includes a feature encoding network and a task prediction network. Combining this with step 220 above, it can be seen that the computer device can extract feature representation information of individual samples based on feature data from n modalities using the feature encoding network; and determine the task prediction result corresponding to the individual sample based on the feature representation information of the individual sample using the task prediction network. Therefore, step 220 above includes at least some of the following steps (steps 222 to 228).

[0077] Step 222: Discretize the feature data of each of the n modes to obtain the discretized feature data corresponding to each of the n modes.

[0078] Computer equipment can first discretize the feature data of n modalities of a sample individual to obtain discretized feature data corresponding to each of the n modalities. This application does not limit the discretization method; optionally, the discretization process includes one or more of the following: natural language processing, normalization, histogram equalization, etc. Optionally, the discretization method can also differ depending on the modality, thereby achieving targeted discretization of the modality's feature data and effectively extracting useful information from the feature data. Several possible discretization methods for different modalities are shown below.

[0079] In one example, when there are n modalities, including a text modality, natural language processing is performed on the feature data of the text modality to obtain the discretized feature data corresponding to the text modality. For example, in the field of intelligent healthcare, the feature data of the text modality includes text data input by the doctor, such as "The patient started coughing 3 days ago, resembling a dog barking, accompanied by hoarseness and stridor. The symptoms are milder during the day and worsen after falling asleep at night, with difficulty breathing when the symptoms worsen." By analyzing the structured discrete features through natural language processing techniques (such as named entity recognition), discretized feature data such as "dog barking," "hoarseness," "stridor," and "difficulty breathing" can be obtained.

[0080] In one example, when there are n modalities, including numerical modalities, the feature data of the numerical modalities are normalized to obtain the discrete feature data corresponding to the numerical modalities. For example, in the field of intelligent healthcare, the feature data of numerical modalities include numerical values ​​from examinations and tests, such as "platelet count is 200". By analyzing the discrete features corresponding to the numerical values ​​through normalization, discrete feature data such as "low", "normal", and "high" can be obtained.

[0081] In one example, when there are n modalities including image modalities, the feature data of the image modalities are normalized and / or histogram equalized to obtain the discrete feature data corresponding to the image modalities. For example, in the field of intelligent healthcare, the feature data of the image modalities includes medical images, such as "magnetic resonance imaging" (MRI). By analyzing the discrete features corresponding to the images through normalization and / or histogram equalization, discrete feature data such as "lesion" and "lumbar degeneration" can be obtained.

[0082] Step 224: Through the feature encoding network, the discretized feature data corresponding to each of the n modes are mapped to obtain the feature representation vectors corresponding to each of the n modes.

[0083] After preprocessing the feature data of n modalities in a discretized manner, the computer device calls the feature encoding network in the task prediction model to map the discretized feature data corresponding to each of the n modalities, obtaining feature representation vectors corresponding to each of the n modalities. The feature encoding network can be any suitable neural network encoder, such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Deep Neural Network (DNN). Through the feature encoding network, the discretized feature data of the n modalities can be mapped into low-dimensional vector representations.

[0084] Optionally, each of the n modes can have its corresponding feature encoding network. Based on this, the feature encoding network includes n feature encoding sub-networks, and the n feature encoding sub-networks correspond one-to-one with the n modes. Thus, step 224 above includes: for the k-th mode among the n modes, the discretized feature data corresponding to the k-th mode is mapped through the feature encoding sub-network corresponding to the k-th mode to obtain the feature representation vector corresponding to the k-th mode; where k is a positive integer less than or equal to n.

[0085] Step 226: Use global max pooling to fuse the feature representation vectors corresponding to the n modalities to obtain the fused feature representation vector of the sample individual.

[0086] The feature representation information of a sample individual obtained through the feature encoding network in the task prediction model includes feature representation vectors corresponding to n modalities. To facilitate subsequent processing by the task prediction network in the task prediction model, the computer device needs to fuse the feature representation vectors corresponding to the n modalities to obtain a fused feature representation vector for the sample individual. In this embodiment, the computer device employs global max pooling to fuse the feature representation vectors corresponding to the n modalities. Global max pooling can loosely fuse feature representation vectors corresponding to different modalities, without requiring a specific number of input modalities; while training the task prediction model based on the mutual information between different modalities emphasizes the consistency of data representation across different modalities for the same individual. The combination of these two operations allows the task prediction model to better address the modality missing problem in real-world application scenarios.

[0087] Optionally, the dimension of the feature representation vector is d, where d is a positive integer. Based on this, the above uses global max pooling to fuse the feature representation vectors corresponding to n modalities, including: extracting the i-th element of the feature representation vectors corresponding to the n modalities to obtain n elements; where i is a positive integer less than or equal to d; taking the maximum value of the n elements as the i-th element of the fused feature representation vector; the dimension of the fused feature representation vector is d. In other words, for each dimension, the computer device takes the element with the largest value among the elements corresponding to that dimension in the feature representation vectors corresponding to the n modalities as the element of the fused feature representation vector corresponding to that dimension.

[0088] Step 228: Determine the task prediction result based on the fused feature representation vector through the task prediction network.

[0089] The fusion feature representation vector is a vector that fuses features from n modalities of a sample individual. The computer device calls the task prediction network in the task prediction model to process the fusion feature representation vector, thereby obtaining the task prediction result corresponding to the sample individual. This application embodiment does not limit the dimension of the task prediction result for the sample individual; optionally, the dimension of the task prediction result is related to the task type. For example, if the task type is a regression problem (such as a hospital stay duration prediction task), the dimension of the task prediction result is one-dimensional; if the task type is a classification problem (such as a mortality risk assessment task), the dimension of the task prediction result is y-dimensional, where y is an integer greater than 1, such as y = 2.

[0090] In summary, the technical solution provided in this application, by employing global max pooling to fuse feature representation vectors corresponding to multiple modalities, can loosely fuse feature representation vectors corresponding to different modalities without requiring a specific number of input modalities. Furthermore, training the task prediction model based on mutual information between different modalities can emphasize the consistency of data representations of the same individual across different modalities. The combination of these two operations enables the task prediction model to better address the modality missing problem in real-world application scenarios.

[0091] In one example, step 230 above includes at least some of the steps (steps 232 to 238) below.

[0092] Step 232: Construct modal positive samples and modal negative samples corresponding to the modal positive samples; wherein, modal positive samples include feature representation vectors of different modalities of the same sample individual, and modal negative samples include feature representation vectors of different modalities of different sample individuals.

[0093] To emphasize the consistency of data representation across different modalities for the same individual, it is necessary to collect modal negative samples and modal positive samples from different modalities. Modal positive samples include feature representation vectors of different modalities for the same individual, while modal negative samples include feature representation vectors of different modalities for different individuals. In this embodiment, the computer device constructs modal positive samples and corresponding modal negative samples based on the feature representation vectors corresponding to the n modalities of multiple individuals.

[0094] Optionally, the mutual information among the n modalities includes the set of mutual information between any two modalities. That is, the computer calculates the mutual information between each pair of modalities within the n modalities. Therefore, the positive and negative modal samples also consist of the feature representation vectors of each pair of modalities. The construction process of positive and negative modal samples is described below, taking the first and second modalities within the n modalities as examples.

[0095] The computer device constructs a first-mode positive sample based on the feature representation vector corresponding to the first modality of the first sample individual and the feature representation vector corresponding to the second modality of the first sample individual; constructs a first-mode negative sample based on the feature representation vector corresponding to the first modality of the first sample individual and the feature representation vector corresponding to the second modality of the second sample individual; constructs a second-mode positive sample based on the feature representation vector corresponding to the second modality of the first sample individual and the feature representation vector corresponding to the first modality of the first sample individual; and constructs a second-mode negative sample based on the feature representation vector corresponding to the second modality of the first sample individual and the feature representation vector corresponding to the first modality of the second sample individual.

[0096] Here, a first-mode positive sample refers to a positive modality sample between the first and second modes when the first mode is used as the anchor point; a first-mode negative sample refers to a negative modality sample between the first and second modes when the first mode is used as the anchor point; a second-mode positive sample refers to a positive modality sample between the second and first modes when the second mode is used as the anchor point; and a second-mode negative sample refers to a negative modality sample between the second and first modes when the second mode is used as the anchor point. In other words, constructing samples between the first and second modes requires constructing positive and negative modality samples, respectively, using the first and second modes as anchor points.

[0097] For example, the feature representation vector corresponding to the first modality of the i-th sample individual is: The feature representation vector corresponding to the second modality of the i-th sample individual is: The feature representation vector corresponding to the first modality of the j-th sample individual is: The feature representation vector corresponding to the second modality of the j-th sample individual is: Where i ≠ j. When the first mode is used as the anchor point... It can be viewed as a modal positive sample. This can be viewed as a modal negative sample; when the second mode is used as the anchor point, It can be viewed as a modal positive sample. It can be viewed as a modal negative sample.

[0098] Step 234: Based on the modal positive samples and modal negative samples, determine the modal mutual information loss. The modal mutual information loss is used to indicate the mutual information between n modes.

[0099] The computer device determines the modal mutual information loss based on the constructed modal positive and negative samples. The modal mutual information loss indicates the mutual information among n modalities. The computer device can maximize the mutual information between modalities by minimizing the modal mutual information loss, thereby achieving consistency in the representation of data from different modalities of the same individual. Based on step 232, the process of determining the modal mutual information loss is described below, taking the first and second modalities among the n modalities as an example.

[0100] The computer device determines the first cosine similarity between the first and second modes based on positive samples of the first mode; determines the second cosine similarity between the first and second modes based on negative samples of the first mode; determines the mutual information loss of the first mode based on the first and second cosine similarities between the first and second modes; determines the first cosine similarity between the second mode and the first mode based on positive samples of the second mode; determines the second cosine similarity between the second mode and the first mode based on negative samples of the second mode; and determines the mutual information loss of the second mode based on the first and second cosine similarities between the second and first modes.

[0101] Specifically, the first modal mutual information loss indicates the mutual information between the first and second modes when the first mode is used as the anchor point; the second modal mutual information loss indicates the mutual information between the second mode and the first mode when the second mode is used as the anchor point. In other words, the modal mutual information loss between the first and second modes needs to be determined separately using the first and second modes as anchor points.

[0102] For example, based on the example of step 232, assume that for each modality positive sample, N is determined. neg Given a corresponding modal negative sample, the modal mutual information loss is calculated as follows.

[0103] With the first mode as the anchor point, the modal mutual information loss between the first mode and the second mode can be expressed as:

[0104]

[0105] With the second mode as the anchor point, the modal mutual information loss between the second mode and the first mode can be expressed as:

[0106]

[0107] The sim function calculates the cosine similarity, and τ is a temperature parameter that can be set to values ​​such as 0.1 or 0.01.

[0108] Step 236: Based on the task prediction results and the task identification results corresponding to the sample individuals, determine the task prediction loss. The task prediction loss is used to indicate the accuracy of the task prediction results.

[0109] On the one hand, the training samples include the task identification results corresponding to individual samples; on the other hand, the computer device obtains the task prediction results corresponding to individual samples based on the task prediction model. Therefore, the computer device can determine the task prediction loss based on the task prediction results and the task identification results. This task prediction loss indicates the accuracy of the task prediction results, or in other words, it indicates the deviation between the task prediction results and the task identification results.

[0110] For example, suppose the fusion feature representation vector of the i-th sample individual is c. i The task prediction result for the i-th sample individual is The task assignment result for the i-th sample individual is y. i The task prediction loss can then be expressed as:

[0111]

[0112] Where, N b The number of individuals in the sample; CE stands for Cross-Entropy, which is applicable to classification problems; SE stands for Squared Error, which is applicable to regression problems.

[0113] Step 238: Determine the model training loss based on modal mutual information loss and task prediction loss.

[0114] The computer device can determine the model training loss by combining the modal mutual information loss and the task prediction loss. Optionally, the computer device can perform one or more processing on the modal mutual information loss and the task prediction loss, such as summation, difference, weighted summation, or modulo operation, to obtain the model training loss.

[0115] For example, based on the examples of steps 234 and 236 above, and expanding the n modalities to more than two modalities, the model training loss can be expressed as:

[0116]

[0117] Where, λ s , λ t , λ z They are respectively L task The corresponding balance parameters can be flexibly set according to the actual application requirements.

[0118] In summary, the technical solution provided in this application, by jointly training the task prediction model using mutual information between multimodal models and the accuracy of task prediction results, achieves the goal of embedding task-related semantic features into the feature representation vector while preserving individual features as much as possible. Figure 3 As shown, Figure 3 (a) is a schematic diagram of the process of individual samples by a trained model in related technologies. It can be seen that the model only has a boundary between positive and negative samples and loses individual characteristics. Figure 3 (b) is a schematic diagram obtained by processing individual samples through a task prediction model that has been trained in this application embodiment. It can be seen that the feature representation obtained through this application embodiment can cluster all individual samples into multiple clusters. Individuals with similar features are clustered together to form a cluster, and each cluster has obvious boundaries, effectively preserving individual features.

[0119] Please refer to Figure 4 This illustration shows a schematic diagram of a task prediction method provided in one embodiment of this application. This task prediction method can be applied to... Figure 1 In the task prediction system shown, if applied to Figure 1 In the first computer device 110 shown.

[0120] like Figure 4 As shown, the computer device acquires a training sample set corresponding to the task prediction model. This training sample set includes multiple training samples, each containing feature data of multiple modalities for an individual sample. First, the computer device preprocesses the feature data of the individual sample in a discretized manner, obtaining discrete feature data corresponding to each modality. Next, the computer device maps the discrete feature data of each modality using feature encoding networks corresponding to each modality, obtaining feature representation vectors corresponding to each modality. Then, the computer device uses global max pooling to fuse the feature representation vectors corresponding to each modality, obtaining a fused feature representation vector for the individual sample. Finally, the computer device processes the fused feature representation vector of the individual sample using a task prediction network to obtain the task prediction result for that individual sample.

[0121] When calculating losses, such as Figure 4As shown, on the one hand, the computer device obtains the mutual information loss among multiple modalities based on the feature representation vectors corresponding to each modality; on the other hand, the computer device determines the task prediction loss based on the task identification result and task prediction result corresponding to the individual sample. The computer device combines the mutual information loss among multiple modalities and the task prediction loss to obtain the model training loss, and adjusts the parameters of the feature encoding network and the task prediction network based on the model training loss to obtain the trained feature encoding network and the trained task prediction network.

[0122] Please refer to Figure 5 The diagram illustrates a flowchart of a task prediction method provided in one embodiment of this application. This task prediction method can be applied to... Figure 1 In the task prediction system shown, if applied to Figure 1 In the second computer device 120 shown. For example... Figure 5 As shown, the task prediction method includes at least some of the following steps (steps 510 to 540).

[0123] Step 510: Obtain feature data of m modalities of the target individual, where m is an integer greater than 1.

[0124] Step 520: Using the task prediction model, based on the feature data of m modalities, extract the feature representation information of the target individual. The feature representation information includes the feature representation vectors corresponding to the m modalities respectively.

[0125] Step 530: Use global max pooling to fuse the feature representation vectors corresponding to m modalities to obtain the fused feature representation vector of the target individual.

[0126] Optionally, the dimension of the feature representation vector is d, where d is a positive integer; the above step 530 includes: extracting the i-th element of the feature representation vector corresponding to each of the m modes to obtain m elements; i is a positive integer less than or equal to d; taking the maximum value of the m elements as the i-th element of the fused feature representation vector; the dimension of the fused feature representation vector is d.

[0127] Step 540: Using the task prediction model, based on the fused feature representation vector, determine the task prediction result corresponding to the target sample.

[0128] Optionally, the target individual includes an electronic medical record; the m modalities include at least one of the following modalities: text modality, numerical modality, and image modality; and / or, the task prediction result includes at least one of the following: disease diagnosis result, risk assessment result, duration assessment result, and protocol assessment result.

[0129] related Figure 5 For any content not described in the embodiments, please refer to the above. Figure 2The specific implementation examples will not be detailed here. One point that needs to be noted is that... Figure 5 The task prediction model in the embodiment uses Figure 2 The method used to train the task prediction model in the embodiment is as follows.

[0130] In summary, the technical solution provided in this application embodiment, by employing global max pooling to fuse feature representation vectors corresponding to multiple modalities, can loosely fuse feature representation vectors corresponding to different modalities without requiring a specific number of input modalities. Furthermore, the task prediction model is trained based on mutual information between different modalities, which can emphasize the consistency of data representations of the same individual across different modalities. The combination of these two operations can better address the modality missing problem in practical application scenarios.

[0131] Please refer to Figure 6 The diagram illustrates a block diagram of a task prediction apparatus according to an embodiment of this application. The apparatus has the following features: Figure 2 The task prediction method described in the embodiment has the following functions: these functions can be implemented in hardware or by hardware executing corresponding software. The device can be a computer device or can be installed within a computer device. The device 600 may include: a sample acquisition module 610, a task prediction module 620, a loss determination module 630, and a parameter adjustment module 640.

[0132] The sample acquisition module 610 is used to acquire training samples for the task prediction model. The training samples include feature data of n modalities of individual samples, where n is an integer greater than 1.

[0133] The task prediction module 620 is used to extract feature representation information of the sample individual based on the feature data of the n modalities through the task prediction model, and to determine the task prediction result corresponding to the sample individual based on the feature representation information; the feature representation information includes feature representation vectors corresponding to the n modalities respectively.

[0134] The loss determination module 630 is used to determine the model training loss based on the mutual information between the n modalities and the accuracy of the task prediction results.

[0135] The parameter adjustment module 640 is used to adjust the parameters of the task prediction model based on the model training loss to obtain a completed task prediction model, which is used for task prediction.

[0136] In one example, the task prediction model includes a feature encoding network and a task prediction network; such as Figure 7As shown, the task prediction module 620 includes: a discretization unit 622, used to discretize the feature data of the n modalities respectively to obtain the discretized feature data corresponding to the n modalities respectively; a feature encoding unit 624, used to map the discretized feature data corresponding to the n modalities respectively through the feature encoding network to obtain the feature representation vectors corresponding to the n modalities respectively; a feature fusion unit 626, used to fuse the feature representation vectors corresponding to the n modalities respectively using global max pooling operation to obtain the fused feature representation vector of the sample individual; and a task prediction unit 628, used to determine the task prediction result based on the fused feature representation vector through the task prediction network.

[0137] In one example, the feature coding network includes n feature coding sub-networks, and the n feature coding sub-networks correspond one-to-one with the n modes; for example... Figure 7 As shown, the feature encoding unit 624 is used to: for the kth mode among the n modes, map the discretized feature data corresponding to the kth mode through the feature encoding sub-network corresponding to the kth mode to obtain the feature representation vector corresponding to the kth mode; wherein, k is a positive integer less than or equal to n.

[0138] In one example, the dimension of the feature representation vector is d, where d is a positive integer; for example... Figure 7 As shown, the feature fusion unit 626 is used to: extract the i-th element of the feature representation vector corresponding to the n modes respectively, to obtain n elements; where i is a positive integer less than or equal to d; take the maximum value of the n elements as the i-th element of the fused feature representation vector; the dimension of the fused feature representation vector is d.

[0139] In one example, such as Figure 7 As shown, the discretization unit 622 is configured to: perform natural language processing on the feature data of the text modality when the n modalities include a text modality, to obtain discretized feature data corresponding to the text modality; perform normalization processing on the feature data of the numerical modality when the n modalities include a numerical modality, to obtain discretized feature data corresponding to the numerical modality; and perform normalization processing and / or histogram equalization processing on the feature data of the image modality when the n modalities include an image modality, to obtain discretized feature data corresponding to the image modality.

[0140] In one example, such as Figure 7As shown, the loss determination module 630 includes: a sample construction unit 632, used to construct modal positive samples and modal negative samples corresponding to the modal positive samples; wherein, the modal positive samples include feature representation vectors of different modalities of the same sample individual, and the modal negative samples include feature representation vectors of different modalities of different sample individuals; a mutual information loss determination unit 634, used to determine the modal mutual information loss based on the modal positive samples and the modal negative samples, wherein the modal mutual information loss is used to indicate the mutual information between the n modalities; a prediction loss determination unit 636, used to determine the task prediction loss based on the task prediction result and the task identification result corresponding to the sample individual, wherein the task prediction loss is used to indicate the accuracy of the task prediction result; and a training loss determination unit 638, used to determine the model training loss based on the modal mutual information loss and the task prediction loss.

[0141] In one example, the n modes include a first mode and a second mode; such as Figure 7 As shown, the sample construction unit 632 is used to: construct a first modality positive sample based on the feature representation vector corresponding to the first modality of the first sample individual and the feature representation vector corresponding to the second modality of the first sample individual; the first modality positive sample refers to the modality positive sample between the first modality and the second modality when the first modality is used as the anchor point; and construct a first modality negative sample based on the feature representation vector corresponding to the first modality of the first sample individual and the feature representation vector corresponding to the second modality of the second sample individual; the first modality negative sample refers to the modality negative sample between the first modality and the second modality when the first modality is used as the anchor point. The modal negative samples; based on the feature representation vector corresponding to the second modality of the first sample individual, and the feature representation vector corresponding to the first modality of the first sample individual, a second modal positive sample is constructed; the second modal positive sample refers to the modal positive sample between the second modality and the first modality when the second modality is used as the anchor point; based on the feature representation vector corresponding to the second modality of the first sample individual, and the feature representation vector corresponding to the first modality of the second sample individual, a second modal negative sample is constructed; the second modal negative sample refers to the modal negative sample between the second modality and the first modality when the second modality is used as the anchor point.

[0142] In one example, such as Figure 7As shown, the mutual information loss determination unit 634 is configured to: determine a first cosine similarity between the first mode and the second mode based on positive samples of the first mode; determine a second cosine similarity between the first mode and the second mode based on negative samples of the first mode; determine a first mode mutual information loss based on the first cosine similarity and the second cosine similarity between the first mode and the second mode; the first mode mutual information loss is used to indicate the mutual information between the first mode and the second mode when the first mode is used as the anchor point; determine a first cosine similarity between the second mode and the first mode based on positive samples of the second mode; determine a second cosine similarity between the second mode and the first mode based on negative samples of the second mode; determine a second mode mutual information loss based on the first cosine similarity and the second cosine similarity between the second mode and the first mode; the second mode mutual information loss is used to indicate the mutual information between the second mode and the first mode when the second mode is used as the anchor point.

[0143] In summary, the technical solution provided in this application, by training a task prediction model to obtain a trained task prediction model, and then using this trained model for subsequent task prediction, realizes a method for task prediction using artificial intelligence. This method can assist or replace manual task identification, helping to reduce labor costs. Furthermore, in this application, the task prediction model is oriented towards feature data from multiple modalities, enabling the embedding of more complete semantic information during task prediction, thus improving the accuracy of task prediction. Additionally, in this application, the training of the task prediction model references mutual information between multiple modalities, ensuring consistency in data representation across multiple modalities. It also embeds useful features into the feature representation vector while discarding useless features, enhancing anti-interference capabilities. Moreover, in this application, the task prediction model is trained using a joint approach of mutual information between multiple modalities and the accuracy of task prediction results, achieving the embedding of task-related semantic features into the feature representation vector while preserving individual features as much as possible.

[0144] Please refer to Figure 8 The diagram illustrates a block diagram of a task prediction apparatus according to an embodiment of this application. The apparatus has the following features: Figure 5 The task prediction method described in the embodiment has the following functions: these functions can be implemented in hardware or by hardware executing corresponding software. The device can be a computer device or can be installed within a computer device. The device 800 may include: a data acquisition module 810, a feature fusion module 830, and a task prediction module 840.

[0145] The data acquisition module 810 is used to acquire feature data of m modalities of the target individual, where m is an integer greater than 1.

[0146] The feature extraction module 820 is used to extract feature representation information of the target individual based on the feature data of the m modalities through a task prediction model. The feature representation information includes feature representation vectors corresponding to the m modalities respectively.

[0147] The feature fusion module 830 is used to fuse the feature representation vectors corresponding to the m modalities by using a global max pooling operation to obtain the fused feature representation vector of the target individual.

[0148] The task prediction module 840 is used to determine the task prediction result corresponding to the target sample based on the fused feature representation vector through the task prediction model.

[0149] In one example, the dimension of the feature representation vector is d, where d is a positive integer; the feature fusion module 830 is used to: extract the i-th element of the feature representation vector corresponding to each of the m modalities to obtain m elements; where i is a positive integer less than or equal to d; and take the maximum value of the m elements as the i-th element of the fused feature representation vector; the dimension of the fused feature representation vector is d.

[0150] In one example, the target individual includes an electronic medical record; the m modalities include at least one of the following modalities: text modality, numerical modality, and image modality; and / or, the task prediction result includes at least one of the following: disease diagnosis result, risk assessment result, duration assessment result, and plan assessment result.

[0151] In summary, the technical solution provided in this application embodiment, by employing global max pooling to fuse feature representation vectors corresponding to multiple modalities, can loosely fuse feature representation vectors corresponding to different modalities without requiring a specific number of input modalities. Furthermore, the task prediction model is trained based on mutual information between different modalities, which can emphasize the consistency of data representations of the same individual across different modalities. The combination of these two operations can better address the modality missing problem in practical application scenarios.

[0152] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0153] Please refer to Figure 9 This diagram illustrates the structural block diagram of a computer device provided in an embodiment of this application, which can be used to implement the functions of the above-described task prediction method example. Specifically:

[0154] The computer device 900 includes a processing unit (such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), and FPGA (Field Programmable Gate Array)) 901, a system memory 904 including RAM (Random-Access Memory) 902 and ROM (Read-Only Memory) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The computer device 900 also includes an I / O system 906 that facilitates information transfer between various devices within the computer device, and a mass storage device 907 for storing the operating system 913, application programs 914, and other program modules 915.

[0155] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 for user input, such as a mouse or keyboard. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include the input / output controller 910 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0156] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. That is, the mass storage device 907 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0157] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 904 and mass storage device 907 described above can be collectively referred to as memory.

[0158] According to an embodiment of this application, the computer device 900 can also be connected to a remote computer on a network, such as the Internet, for operation. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to the system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0159] The memory also includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory and configured to be executed by one or more processors to implement the task prediction method described above.

[0160] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the task prediction method described above.

[0161] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0162] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the task prediction method described above.

[0163] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0164] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A task prediction method, characterized by, The method includes: Obtain training samples for the task prediction model. The training samples include feature data of n modalities of individual samples, where n is an integer greater than 1. The n modalities include at least one of the following modalities: text modality, numerical modality, and image modality. Using the task prediction model, based on the feature data of the n modalities, feature representation information of the sample individual is extracted, and based on the feature representation information, the task prediction result corresponding to the sample individual is determined; the feature representation information includes feature representation vectors corresponding to the n modalities respectively; Based on the feature representation vectors corresponding to the first mode among the n modes of the first sample individual, and the feature representation vectors corresponding to the second mode among the n modes of the first sample individual, a first mode positive sample is constructed; based on the feature representation vectors corresponding to the first mode of the first sample individual, and the feature representation vectors corresponding to the second mode of the second sample individual, a first mode negative sample is constructed; based on the feature representation vectors corresponding to the second mode of the first sample individual, and the feature representation vectors corresponding to the first mode of the first sample individual, a second mode positive sample is constructed; based on the feature representation vectors corresponding to the second mode of the first sample individual, and the feature representation vectors corresponding to the first mode of the second sample individual, a second mode negative sample is constructed. Based on modal positive samples and modal negative samples, modal mutual information loss is determined. The modal mutual information loss is used to indicate the mutual information between the n modes. The modal positive samples include the first modal positive samples and the second modal positive samples. The modal negative samples include the first modal negative samples and the second modal negative samples. Based on the task prediction results and the task identification results corresponding to the sample individuals, the task prediction loss is determined; Based on the modal mutual information loss and the task prediction loss, the model training loss is determined; Based on the training loss of the model, the parameters of the task prediction model are adjusted to obtain a fully trained task prediction model, which is then used for task prediction.

2. The method of claim 1, wherein, The task prediction model includes a feature encoding network and a task prediction network; The step of extracting feature representation information of the sample individual based on the feature data of the n modalities using the task prediction model, and determining the task prediction result corresponding to the sample individual based on the feature representation information, includes: The feature data of the n modes are discretized respectively to obtain the discretized feature data corresponding to the n modes respectively; The feature encoding network is used to map the discrete feature data corresponding to the n modes to obtain the feature representation vectors corresponding to the n modes. A global max pooling operation is used to fuse the feature representation vectors corresponding to the n modalities to obtain the fused feature representation vector of the sample individual; The task prediction result is determined using the task prediction network based on the fused feature representation vector.

3. The method of claim 2, wherein, The feature coding network includes n feature coding sub-networks, and the n feature coding sub-networks correspond one-to-one with the n modes; The step of mapping the discretized feature data corresponding to the n modalities through the feature encoding network to obtain the feature representation vectors corresponding to the n modalities includes: For the kth mode among the n modes, the discretized feature data corresponding to the kth mode is mapped through the feature encoding sub-network corresponding to the kth mode to obtain the feature representation vector corresponding to the kth mode; where k is a positive integer less than or equal to n.

4. The method of claim 2, wherein, The dimension of the feature representation vector is d, where d is a positive integer; The process employs global max pooling to fuse the feature representation vectors corresponding to the n modalities, resulting in the fused feature representation vector for each sample individual, including: Extract the i-th element of the feature representation vector corresponding to each of the n modes to obtain n elements; where i is a positive integer less than or equal to d; The maximum value of the n elements is taken as the i-th element of the fused feature representation vector; the dimension of the fused feature representation vector is d.

5. The method of claim 2, wherein, The feature data of the n modes are discretized respectively to obtain the discretized feature data corresponding to the n modes, including: When the n modalities include a text modality, natural language processing is performed on the feature data of the text modality to obtain the discretized feature data corresponding to the text modality; When the n modes include numerical modes, the feature data of the numerical modes are normalized to obtain the discretized feature data corresponding to the numerical modes. When the n modalities include image modalities, the feature data of the image modalities are normalized and / or histogram equalized to obtain the discretized feature data corresponding to the image modalities.

6. The method of claim 1, wherein, The determination of modal mutual information loss based on modal positive and modal negative samples includes: Based on the positive samples of the first modality, determine the first cosine similarity between the first modality and the second modality; Based on the negative samples of the first mode, determine the second cosine similarity between the first mode and the second mode; Based on the first cosine similarity and the second cosine similarity between the first mode and the second mode, the mutual information loss of the first mode is determined; the first modal mutual information loss is used to indicate the mutual information between the first mode and the second mode when the first mode is used as the anchor point; Based on the positive samples of the second modality, determine the first cosine similarity between the second modality and the first modality; Based on the negative samples of the second modality, determine the second cosine similarity between the second modality and the first modality; The mutual information loss of the second mode is determined based on the first cosine similarity and the second cosine similarity between the second mode and the first mode; the mutual information loss of the second mode is used to indicate the mutual information between the second mode and the first mode when the second mode is used as the anchor point.

7. A task prediction method characterized by, The method includes: Obtain feature data of m modalities of the target individual, where m is an integer greater than 1; Using a task prediction model, feature representation information of the target individual is extracted based on the feature data of the m modalities. The feature representation information includes feature representation vectors corresponding to the m modalities respectively. A global max pooling operation is used to fuse the feature representation vectors corresponding to the m modalities to obtain the fused feature representation vector of the target individual; Based on the fused feature representation vector, the task prediction result corresponding to the target individual is determined using the task prediction model. The task prediction model is obtained by training the method described in any one of claims 1 to 6.

8. The method of claim 7, wherein, The dimension of the feature representation vector is d, where d is a positive integer; The method employs global max pooling to fuse the feature representation vectors corresponding to the m modalities, thereby obtaining the fused feature representation vector of the target individual, including: Extract the i-th element of the feature representation vector corresponding to each of the m modes to obtain m elements; where i is a positive integer less than or equal to d; The maximum value of the m elements is taken as the i-th element of the fused feature representation vector; the dimension of the fused feature representation vector is d.

9. The method of claim 7, wherein, The target individuals include electronic medical records; The m modalities include at least one of the following modalities: text modality, numerical modality, and image modality; And / or, The task prediction results include at least one of the following: disease diagnosis results, risk assessment results, duration assessment results, and plan assessment results.

10. A task prediction apparatus characterized by comprising: The device includes: The sample acquisition module is used to acquire training samples for the task prediction model. The training samples include feature data of n modalities of individual samples, where n is an integer greater than 1, and the n modalities include at least one of the following modalities: text modality, numerical modality, and image modality. The task prediction module is used to extract feature representation information of the sample individual based on the feature data of the n modalities through the task prediction model, and to determine the task prediction result corresponding to the sample individual based on the feature representation information; the feature representation information includes feature representation vectors corresponding to the n modalities respectively; The loss determination module is used to construct a first-mode positive sample based on the feature representation vector corresponding to the first mode among the n modes of the first sample individual and the feature representation vector corresponding to the second mode among the n modes of the first sample individual; construct a first-mode negative sample based on the feature representation vector corresponding to the first mode of the first sample individual and the feature representation vector corresponding to the second mode of the second sample individual; construct a second-mode positive sample based on the feature representation vector corresponding to the second mode of the first sample individual and the feature representation vector corresponding to the first mode of the first sample individual; and construct a second-mode negative sample based on the feature representation vector corresponding to the second mode of the first sample individual and the feature representation vector corresponding to the first mode of the second sample individual. The loss determination module is further configured to determine modal mutual information loss based on modal positive samples and modal negative samples. The modal mutual information loss is used to indicate the mutual information between the n modes. The modal positive samples include the first modal positive samples and the second modal positive samples. The modal negative samples include the first modal negative samples and the second modal negative samples. The loss determination module is further configured to determine the task prediction loss based on the task prediction result and the task identification result corresponding to the sample individual. The loss determination module is further configured to determine the model training loss based on the modal mutual information loss and the task prediction loss; The parameter adjustment module is used to adjust the parameters of the task prediction model based on the model training loss to obtain a completed task prediction model, which is used for task prediction.

11. A task prediction apparatus characterized by comprising: The device includes: The data acquisition module is used to acquire feature data of m modalities of the target individual, where m is an integer greater than 1; The feature extraction module is used to extract feature representation information of the target individual based on the feature data of the m modalities through the task prediction model. The feature representation information includes feature representation vectors corresponding to the m modalities respectively. The feature fusion module is used to fuse the feature representation vectors corresponding to the m modalities by using a global max pooling operation to obtain the fused feature representation vector of the target individual. The task prediction module is used to determine the task prediction result corresponding to the target individual based on the fused feature representation vector through the task prediction model. The task prediction model is obtained by training the method described in any one of claims 1 to 6.

12. A computer device, comprising: The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the task prediction method as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the task prediction method as described in any one of claims 1 to 9.

14. A computer program product, characterised in that, The computer program product includes a computer program that is loaded and executed by a processor to implement the task prediction method as described in any one of claims 1 to 9.

Images