Model training method, prediction method based on multiple user data, and device

Abstract

The present disclosure provides a model training method, a prediction method based on multiple user data, and an apparatus and device. The method comprises: obtaining multiple first sample groups corresponding to multiple users respectively, wherein the first sample groups comprise multiple sample data, and different sample data are used to describe different attributes of user features; based on the multiple first sample groups, iteratively pre-training a sub-network to be trained in an unsupervised manner with a preset target as a constraint; the preset target is that multiple sample features of the same user are gathered to the same center; a parent network comprising the trained sub-network is constructed; the parent network is used to predict a target based on sample features extracted by the sub-network; based on multiple second sample groups corresponding to the multiple users respectively, the parent network is iteratively trained in a supervised manner to obtain a target model; the second sample groups comprise multiple sample data and carry prediction labels corresponding to the target prediction.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a model training method, a prediction method based on multiple user data, an apparatus and device. Background Technology

[0002] With the development of deep learning technology, it has been applied in various fields. For example, in the medical field, deep learning technology can be used for image segmentation of tumor sites and prediction of patient prognosis and survival.

[0003] In related technologies, deep learning tasks typically involve training a model for a specific task, often requiring the preparation of a large amount of labeled data beforehand. This data is then used to train a model for a single prediction task. When there are multiple prediction tasks, this necessitates preparing a large amount of labeled data for each task and training a separate model for each. In practice, this not only consumes significant computing resources, but the preparation of labeled data is also extremely costly in terms of manpower and resources. For example, in the medical field, providing a large amount of labeled data requires repeated evaluation by multiple professionals before final annotation, making the process very difficult. Furthermore, training multiple prediction tasks requires substantial computing resources, resulting in high expenditures of human and material resources. Summary of the Invention

[0004] This disclosure provides a model training method, the method comprising:

[0005] Obtain the first unlabeled sample group corresponding to each of the multiple users. The first sample group includes multiple types of sample data, and the different types of sample data are used to describe user characteristics with different attributes.

[0006] Based on multiple first sample groups, the sub-network to be trained is subjected to unsupervised iterative pre-training with a preset objective as a constraint; wherein, the sub-network is used to extract features for each sample data to obtain sample features corresponding to each sample data, and the preset objective is: multiple sample features of the same user converge to the same center.

[0007] Construct a parent network that includes the trained sub-networks; wherein the parent network is used to perform target prediction based on the sample features extracted by the sub-networks, and the target prediction includes at least a classification prediction of the user's category;

[0008] Based on the second sample groups corresponding to multiple users, the parent network is subjected to supervised iterative training to obtain the target model; wherein, the second sample group includes multiple sample data and carries prediction labels corresponding to the target prediction.

[0009] Optionally, the multiple sample data includes at least two of the following: nuclear magnetic resonance (MRI) sample images, clinical sample data, and genetic sample data;

[0010] The target prediction further includes at least one of prognostic prediction, image segmentation, and gene mutation prediction; wherein, the image segmentation is used to segment the image region belonging to the target type in the MR sample image.

[0011] Optionally, based on multiple first sample groups, unsupervised iterative training is performed on the sub-network to be trained under the constraint of a preset objective, including:

[0012] In each iteration of training, a first loss value is determined based on the distance between multiple sample features; wherein, the distance between the multiple sample features includes the distance between sample features of different users, and the distance between sample features of the same user;

[0013] Based on the first loss value, the parameters of the sub-network are adjusted.

[0014] Optionally, determining the first loss value based on the distance between multiple sample features includes:

[0015] For each pair of attribute sample features, obtain the first distance between the sample features of the two attributes belonging to the same user, and the second distance between the sample features of the two attributes belonging to different users;

[0016] The first loss value is determined based on the first distance and the second distance corresponding to the sample features of each of the two attributes.

[0017] Optionally, determining the first loss value based on the first distance and the second distance corresponding to the sample features of each of the two attributes includes:

[0018] Based on the first distance and the second distance corresponding to the sample features of each of the two attributes, the following loss function is constructed:

[0019]

[0020] The first loss value is determined based on the loss value obtained in the loss function for each of the two attributes.

[0021] Where L(v,g) represents the loss between sample features v and sample features g, and b represents the batch of the first sample group input into the subnetwork;

[0022] A sample feature representing an attribute of the j-th user. Let Zv represent a sample feature that is different from Zv in another attribute of the j-th user. This represents a sample feature of the k-th user that is different from Zv in another attribute;

[0023] τ represents the temperature parameter, and cos represents the cosine distance.

[0024] Optionally, the multiple sample data includes MRI sample images, clinical sample data, and gene sample data. In each iteration of training, determining a first loss value based on the distance between the multiple sample features includes:

[0025] In each iteration of training, the first sub-loss value is determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the clinical sample data.

[0026] A second sub-loss value is determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the gene sample data; wherein, the distance includes the distance between sample features of different users and the distance between sample features of the same user;

[0027] The first loss value is determined based on the first sub-loss value and the second sub-loss value.

[0028] Optionally, a parent network is constructed that includes the trained sub-networks, including:

[0029] The parent network is obtained by splicing the sub-networks obtained through training with the network layers corresponding to the target prediction; wherein the network layers include at least fully connected layers according to the target prediction.

[0030] Alternatively, an initial network corresponding to the target prediction can be obtained, and the parameters of the trained sub-network can be transferred to the initial network to obtain the parent network.

[0031] Optionally, based on the second sample groups corresponding to multiple users, the parent network is subjected to supervised iterative training, including:

[0032] During each iteration of training the parent network, the target prediction result output by the parent network is obtained;

[0033] Based on the target prediction result and the prediction label corresponding to the second sample group, a second loss value is determined;

[0034] The parameters of the parent network are adjusted based on the second loss value.

[0035] Optionally, the multiple sample data includes nuclear magnetic resonance (MR) sample images, and the step of extracting features from each type of sample data to obtain sample features corresponding to each type of sample data includes:

[0036] After performing convolution operations at multiple scales on the MR sample images in sequence, the extracted radiomics features are subjected to dimensionality transformation to obtain initial radiomics features.

[0037] The initial radiomics features are subjected to self-attention mechanism feature extraction to enhance the distinction between lesion features and normal tissue features in the MR sample image, thereby obtaining the sample features corresponding to the MR sample image.

[0038] Optionally, the multiple sample data includes gene sample data, and the feature extraction for each type of sample data to obtain the sample features corresponding to each type of sample data includes:

[0039] The gene sample data is transformed into a feature vector to obtain the corresponding gene feature vector;

[0040] The gene feature vector is mapped to the sample features corresponding to the gene sample data.

[0041] Optionally, the multiple sample data includes clinical sample data, and the feature extraction for each type of sample data to obtain the sample features corresponding to each type of sample data includes:

[0042] Convert each piece of clinical information in the clinical sample data into a clinical feature vector;

[0043] The clinical feature vector is subjected to self-attention mechanism feature extraction to cross-integrate various clinical information in the clinical sample data to obtain the sample features corresponding to the clinical sample data.

[0044] Optionally, the sub-network includes: a radiomics feature extraction branch, a clinical feature extraction branch, and a gene feature extraction branch, wherein:

[0045] The image omics feature extraction branch includes a convolution module, a residual module, a dimension transformation module, a first self-attention module, and a first perceptron module connected in sequence.

[0046] The clinical feature extraction branch includes a vector transformation module, at least one second self-attention module, a first self-attention module, and a second perceptron module connected in sequence.

[0047] The gene feature extraction branch includes a vector transformation module, a long short-term memory network module, and a third perceptron module connected in sequence.

[0048] This application also provides a prediction method based on multiple user data, the method comprising:

[0049] Obtain various user data for the user to be predicted, wherein the various user data are used to describe user characteristics with different attributes;

[0050] The various user data are input into the target model corresponding to the target prediction, and the target model is trained by the model training method described above.

[0051] Obtain the prediction results output by the target model; wherein, the prediction results include the classification prediction results of the user's category.

[0052] This application also provides a model training apparatus, the apparatus comprising:

[0053] The first sample acquisition module is used to acquire the first unlabeled sample group corresponding to each of the multiple users. The first sample group includes multiple types of sample data, and different types of sample data are used to describe user characteristics with different attributes.

[0054] The pre-training module is used to perform unsupervised iterative pre-training on the sub-network to be trained based on multiple first sample groups and with a preset target as a constraint; wherein, the sub-network is used to extract features for each sample data to obtain sample features corresponding to each sample data, and the preset target is: multiple sample features of the same user converge to the same center.

[0055] A network construction module is used to construct a parent network containing trained sub-networks; wherein the parent network is used to perform target prediction based on sample features extracted by the sub-networks, and the target prediction includes at least a classification prediction of the user's category;

[0056] The fine-tuning training module is used to perform supervised iterative training on the parent network based on the second sample groups corresponding to multiple users to obtain the target model; wherein, the second sample group includes multiple sample data and carries prediction labels corresponding to the target prediction.

[0057] This application also provides a prediction device based on multiple user data, the device comprising:

[0058] The data acquisition module is used to acquire various user data of the user to be predicted, and the various user data are used to describe user characteristics with different attributes;

[0059] The input module is used to input various user data into the target model corresponding to the target prediction, wherein the target model is trained by the model training method described above.

[0060] The result acquisition module is used to acquire the prediction results output by the target model; wherein, the prediction results include the classification prediction results of the user's category.

[0061] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing, implements the model training method described above, or implements the prediction method based on multiple user data described above.

[0062] By adopting the technical solution of this disclosure embodiment, a first sample group corresponding to each of multiple users can be obtained, and an unsupervised iterative pre-training of the sub-network to be trained can be performed based on the multiple first sample groups with a preset target as a constraint; then, a parent network containing the trained sub-network is constructed, and a supervised iterative training of the parent network is performed based on the second sample group corresponding to each of the multiple users, thereby obtaining a target model for target prediction.

[0063] On the one hand, since the sub-network is trained using an unlabeled first sample group, and a parent network containing the sub-network is then constructed after the sub-network is pre-trained, with the parent network using a labeled second sample group for training, the target model for target prediction can be obtained by fine-tuning the parent network based on a small amount of labeled sample data. This greatly reduces the need for labeled sample data, thus eliminating the limitation on the acquisition of labeled data in deep learning practices in the medical field and avoiding the problem of the difficulty in obtaining labeled data restricting the application of deep learning. Furthermore, since the sub-network is pre-trained, it can be shared for multiple prediction tasks. Only the sub-network needs to be pre-trained; subsequent fine-tuning based on labeled data is required for different prediction tasks, eliminating the need for separate full-process training for each prediction task, thereby saving computer resources.

[0064] On the other hand, the sub-network is used to extract features for each type of sample data. When training the sub-network, the preset training objective is to make the multiple sample features of the same user converge towards the same center, and the sample features of different users move away from the same center. In this way, through pre-training, the sub-network can extract multiple sample features of the same user, and the extracted multiple sample features can highly represent the same user. This can improve the accuracy of downstream tasks (target prediction).

[0065] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the dimensions and shapes of the figures in the drawings do not reflect actual proportions and are only intended to illustrate the content of the present invention. The same or similar reference numerals in the drawings represent the same or similar elements or elements having the same or similar functions.

[0067] Figure 1 A flowchart illustrating the steps of the model training method is shown schematically.

[0068] Figure 2 A schematic diagram of a subnetwork structure is shown.

[0069] Figure 3 The schematic diagram illustrates the steps of feature extraction from MR sample images according to this disclosure.

[0070] Figure 4 A flowchart illustrating the steps of a prediction method based on multiple user data is shown schematically.

[0071] Figure 5 An exemplary flowchart illustrating the steps of a model training method is shown schematically.

[0072] Figure 6 A schematic diagram of a model training device is shown.

[0073] Figure 7 A schematic diagram of the structure of a prediction device based on multiple user data is shown.

[0074] Figure 8 A schematic block diagram of an electronic device is shown. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0076] In the medical field, deep learning technology can be applied to tasks such as prognosis prediction and image segmentation. Generally, it is necessary to train a neural network model for these tasks. However, the training of neural network models usually depends on labeled sample data.

[0077] For example, in tumor image segmentation tasks, it is necessary to annotate the tumor region in medical image samples. However, this annotation is different from image annotation in other fields. It requires experienced medical experts to work together to annotate. Therefore, the cost of annotating a medical image sample is high, and it is difficult and the amount of annotated data is also small.

[0078] Of course, the above only uses the annotation of medical image samples as an example. In practice, other types of annotation are involved in the medical field, such as the annotation of tumor staging for patients. These annotations all suffer from high annotation costs and limited labeled data. This increases the difficulty of applying deep learning in the medical field.

[0079] On the other hand, in the field of medical prognosis prediction, the current model is generally trained based on medical image samples and labeled prognostic survival (labeling prognostic survival is also quite difficult). However, this approach not only has the problem of insufficient labeled data leading to the model not achieving the expected results, but also the problem of inaccurate predictions due to the single prognostic factor (based solely on medical image samples).

[0080] In view of this, this application provides a technical solution to address the aforementioned technical problems. The core concept of this solution is as follows: First, unsupervised pre-training of the sub-network responsible for feature extraction is performed using unlabeled sample data. To ensure the sub-network has strong feature extraction capabilities, during pre-training, the preset goal is to make multiple sample features of the same user cluster towards the same center, while sample features of different users are far from the same center, so that the multiple sample features extracted by the sub-network highly represent the same user. Next, a parent network containing this sub-network is constructed, and supervised training is performed on the parent network using a second sample group, thereby achieving fine-tuning of the parameters of the parent network. In this way, the expected model training effect can be achieved with a small amount of labeled data, resulting in a target model capable of performing target prediction, thus reducing the requirement for a large amount of labeled data and overcoming the bottleneck problem in the medical field where the limited amount of labeled data results in narrow coverage and difficulty in application of deep learning.

[0081] Reference Figure 1 As shown, a flowchart illustrating the steps of a model training method according to this application is illustrated. Figure 1As shown, this model training method can be applied to smart terminals, such as servers and computers. It can also be applied in the medical field, specifically in scenarios such as predicting user prognosis and survival, tumor segmentation, and gene mutation prediction based on multiple data. Of course, the multiple data can include at least two types of data such as medical image data, clinical data, gene data, and user identity data. Specifically, two or more types can be selected according to actual needs, without any special restrictions.

[0082] like Figure 1 As shown, the specific steps may include:

[0083] Step S101: Obtain the first unlabeled sample group corresponding to each of the multiple users.

[0084] The first sample group includes multiple types of sample data, with different types of sample data used to describe user characteristics of different attributes.

[0085] In this embodiment, to ensure the diversity of training samples, the age distribution of multiple users can be broad, covering various age groups such as the elderly, middle-aged, young, and children. The gender of users can include both males and females. Each user's first sample group does not require labeling; instead, an identifier is generated for each user using their identity code, allowing the subnetwork to distinguish between different users. This process can be automatically generated by the computer when generating sample data, eliminating the need for manual labeling.

[0086] Each user's first sample group can include multiple sample data sets, specifically at least two different sets. Each set of sample data sets describes a user characteristic of one attribute. Depending on the application scenario, the described multiple user characteristics can include the following: facial features, user identity features, body shape features, etc. Specifically, in the medical field, the described multiple user characteristics can include the following: lesion tissue features, clinical features, genetic features, etc.

[0087] Different types of sample data can have different formats. For example, sample data can include text, image, and even video formats. Different formats reflect user characteristics from different dimensions and describe a user from different perspectives. For instance, image-type sample data can describe a user's image features; if the sample data is a facial image, it can describe the user's facial features.

[0088] For example, in the medical field, various types of sample data can include image-type lesion tissue images, hospitalization data, diagnosis and treatment data, and users' historical medical records.

[0089] It should be noted that multiple sample data can be obtained depending on the specific application scenario. It should be noted that although the multiple sample data can have different formats, it does not necessarily mean that they must have different formats. In some cases, it may only include multiple sample data in image format. The only difference is that these multiple sample data are obtained by collecting images of different parts of the user's body. The image acquisition method can be camera acquisition or medical imaging equipment acquisition.

[0090] For example, in one alternative example, the various sample data may include facial and body images captured by a camera, as well as medical images obtained from medical imaging equipment capturing images of the user's tissues and organs. In this case, by combining images from different body parts with the user's physical characteristics and medical images, the prognosis of the disease can be assessed, along with other medical assessment scenarios, to provide a reference for subsequent diagnosis.

[0091] Step S102: Based on multiple first sample groups, perform unsupervised iterative pre-training on the sub-network to be trained with a preset target as a constraint.

[0092] The sub-network is used to extract features for each type of sample data to obtain the sample features corresponding to each type of sample data. The preset goal is to make multiple sample features of the same user converge to the same center, and to make sample features of different users move away from the same center.

[0093] In this embodiment, a sub-network can be trained. The sub-network is used to extract features for each type of sample data, thereby obtaining the sample features corresponding to each type of sample data. In specific implementation, the sub-network can be a CNN network. In practice, a ResNet network can be selected because the ResNet network is a series of residual modules, which has the advantage of not losing gradients and can improve the feature extraction effect of the sub-network.

[0094] In one embodiment, multiple first sample groups can be input into the sub-network in batches for training, with each batch containing first sample groups from multiple users. Since the sub-network requires unsupervised training—meaning no ground truth is provided as a reference, i.e., the first sample groups do not carry ground truth labels—a constraint needs to be imposed on the sub-network. In this application, the constraint on the sub-network is: to cause multiple sample features from the same user to converge towards the same center, and for sample features from different users to move away from the same center.

[0095] In this context, the convergence of multiple sample features of the same user toward the same center can mean that the cosine distance between multiple sample features of the same user should approach the minimum value; the dispersion of sample features of different users away from the same center can mean that the cosine distance between sample features of the same user should approach the maximum value.

[0096] In other words, during the training process, when updating the parameters of the subnetwork, it is necessary to determine the distance between various sample features, and based on the distance between various sample features, update the parameters of the subnetwork according to the preset target.

[0097] The distance between multiple sample features can be cosine distance, representing the similarity between sample features. Generally, although a single user may have multiple sample data sets, these multiple sample data sets describe the same user, and therefore they possess a certain degree of similarity. For example, in the medical field, medical imaging images and clinical data both describe the same patient and the same disease, so they should contain a large amount of consistent information about the disease; therefore, the features extracted from these two parts of data should be close.

[0098] In this way, during the training of the subnetwork, the distance between the various sample features output by the subnetwork is directly used to update the parameters of the subnetwork. This not only helps the subnetwork to extract consistent features that reflect the user from multiple data sets of the same user, but also, as training progresses, it can extract as many common features as possible that reflect a user group from a large amount of unlabeled data.

[0099] Training the subnetwork is an iterative training method. Generally, the training of the subnetwork can end when the iteration termination condition is met. This termination condition can be that the number of iterations has reached a preset number, or that the subnetwork has met a preset objective. That is, the various sample features output by the subnetwork have satisfied the following: the distance between sample features of the same user is less than a first preset distance, while the distance between different sample features of different users is greater than a second preset distance. The second preset distance is greater than the first preset distance.

[0100] Step S103: Construct a parent network that includes the trained sub-networks.

[0101] The parent network inherits the parameters of the trained sub-networks. The parent network is used to predict targets based on the sample features extracted by the sub-networks. The target prediction includes at least the classification prediction of the user's category.

[0102] In this embodiment, after training the sub-network, a parent network can be constructed based on the sub-network. Specifically, the parent network needs to contain the sub-network, and thus inherits the parameters of the trained sub-network. The sub-network serves as the feature extraction layer of the parent network to perform the upstream task of feature extraction in the parent network. The parent network also includes a downstream network layer connected in series with the sub-network. This downstream network layer can perform downstream tasks based on the output of the sub-network. The downstream task can be understood as the target prediction of this application.

[0103] The process of constructing the parent network can be detailed in subsequent embodiments.

[0104] In this embodiment, target prediction may include classification prediction, which is used to determine the category to which the user belongs. This category may vary depending on the application scenario. For example, in the medical field, classification prediction is used to predict the stage of the user's disease, such as tumor staging. Alternatively, in another scenario, it can be used to predict the gene mutation category of the user's disease, that is, to assess which gene mutation group the user belongs to.

[0105] The classification prediction results described above can serve as a medical parameter for diagnostic reference in the medical field. Of course, in practice, their application is not limited to classification prediction. For example, in the field of image segmentation, they can also be used for image segmentation. In the case of image segmentation as the target prediction, multiple sample data can be multiple image data, and these multiple image data are based on images of different dimensions of the object to be segmented. For example, if the object to be segmented is a tumor, the multiple image data can be images of the tumor tissue using different image sequences, such as t1-weighted t1 sequences, t2-weighted t2 sequences, etc.

[0106] Step S104: Based on the second sample groups corresponding to each of the multiple users, perform supervised iterative training on the parent network to obtain the target model.

[0107] The second sample group includes multiple sample data and carries prediction labels corresponding to the target prediction.

[0108] In this embodiment, after the parent network is constructed, since the parent network contains the trained sub-networks, it inherits the parameters of the trained sub-networks. In this way, the parent network can be trained using the labeled second sample group, thereby fine-tuning the parameters of the parent network (including fine-tuning the parameters of the sub-networks and the parameters of the downstream network layers in the parent network).

[0109] The second sample group may contain various types of sample data that are the same as those in the first sample group. For example, if the first sample group includes image sample data and text sample data, then the second sample group may also include image sample data and text sample data. Alternatively, in some cases, the data may not be of the same type. In practical applications, the user attributes described by the sample data in the first sample group and the sample data in the second sample group may be different or the same.

[0110] For example, in the medical field, the multiple sample data in the first sample group includes medical imaging images and clinical data used to describe the user's tissues and organs, and the multiple sample data in the second sample group may also include medical imaging images obtained from medical image acquisition of the user's tissues and organs. The tissues and organs may be of the same type, such as both being the brain; or, in some cases, they may be of different types, such as the first sample group targeting the lungs and the second sample group targeting the brain.

[0111] For example, in other fields, such as identity verification, it is necessary to classify whether a user is a suspicious user. The multiple sample data in the first sample group include facial images used to describe the user's face and video data used to describe body posture. The multiple sample data in the second sample group may also include facial images used to describe the user's face and video data used to describe body posture. However, in some cases, the multiple sample data in the second sample group may also include facial images used to describe the user's face and posture images used to describe human posture.

[0112] Since the training of the parent network is supervised training, a ground truth value is required in supervised training to determine the difference between the parent network's predicted value and the ground truth value. Therefore, the second sample group can carry the user's predicted label. When the target prediction is a classification prediction, this predicted label can be a classification label, used to characterize the ground truth value of the user's category. For example, as described in the above embodiment, the predicted label can be a disease stage label, such as a tumor stage label; or, the predicted label can be a gene mutation category label.

[0113] In practice, during the training of the parent network, its parameters can be updated based on the predicted labels and the output target predictions. That is, when training the sub-networks, parameter updates are based on the distances between the various sample features output by the sub-networks, with the preset goal of bringing the sample features of the same user closer together. However, when training the parent network, parameter updates are based on the prediction results output by the parent network, with the training goal of making the prediction results more accurate. Therefore, different loss functions can be constructed for the two processes; that is, the loss functions used in the pre-training and training processes are different.

[0114] The pre-trained subnetwork can be shared for model training of various target prediction tasks. In practice, the subnetwork can be uploaded to the platform database for sharing. Then, users who need training can download the subnetwork from the platform database. Next, the parent network is built based on the downloaded subnetwork, and then the parent network is trained using a small amount of labeled data, thus obtaining multiple target models applicable to different target predictions.

[0115] In other words, in actual implementation, the second sample group prepares sample data for the target prediction task, but it is not necessary for the sample data of the second sample group to correspond to the same type as the sample data of the first sample group. This is because the main performance of the sub-network after training is feature extraction, and multiple sample features extracted for the same user can focus on the same user. Thus, in practice, as long as multiple user data of a user are input into the sub-network, the sub-network will perform feature extraction with the aim of clustering the extracted multiple sample features towards the same center. Therefore, the sub-network trained in this application can be shared for multiple target prediction tasks.

[0116] Of course, if the sample data of the second sample group corresponds to the same type as the sample data of the first sample group, such as MR image samples, clinical data and gene data, then better training results will be achieved, a better target model will be obtained, and more accurate target prediction results will be obtained.

[0117] The technical solution of this application adopts unsupervised training for the training of the sub-network. After the sub-network is pre-trained, a parent network containing the sub-network is constructed. The training samples of the parent network are trained in a supervised manner. Thus, the parent network can be fine-tuned based on a small amount of labeled sample data to obtain the target model for target prediction. This greatly reduces the need for labeled sample data, thereby no longer being limited by the acquisition of labeled data and avoiding the problem of limiting the application of deep learning due to the difficulty in obtaining labeled data.

[0118] Furthermore, since the subnetwork is fed with multiple sample data, it means that the subnetwork can extract sample features of multiple attributes of the same user. In practice, this can be called multi-dimensional features. Thus, through pre-training, the subnetwork can extract multiple sample features of the same user, and extract multi-dimensional feature expressions of the user. This can accurately and comprehensively describe the features of a user, thereby improving the accuracy of downstream tasks (target prediction).

[0119] In one embodiment, the multiple sample data includes at least two of the following: MRI sample images, clinical sample data, and gene sample data. Target prediction also includes at least one of prognostic prediction, image segmentation, and gene mutation prediction; wherein image segmentation is used to segment image regions in the MRI sample images that belong to the target type.

[0120] The MR sample images are obtained by capturing MR images of the user's lesion tissue. Within the first sample group, MR sample images from different users may correspond to the same disease; for example, if they are all for brain tumors, then each user will have their own brain MR sample images. Similarly, clinical sample data and genetic sample data also contain clinical and genetic data for the same disease, reflecting both individual and common data among different users when they suffer from that disease.

[0121] In practice, in one scenario, if MRI sample images can be included as mandatory data in the first sample group, then either clinical sample data or gene sample data can be chosen as the other sample data in the first sample group, or both clinical sample data and gene sample data can be included as sample data in the first sample group.

[0122] In this embodiment, when multiple sample data include at least two of the following: MRI sample images, clinical sample data, and genetic sample data, target prediction can be used to make the following predictions based on multiple sample data of a user:

[0123] Prognostic prediction, whereby prognostic prediction is used to predict the user's prognostic survival;

[0124] Image segmentation, where image segmentation is used to segment lesion sites from MR sample images; in image segmentation, since it involves segmenting an image, clinical data and genetic data can be used to help determine the degree and area of ​​lesion invasion into normal tissue. For example, in tumor diseases, the degree and area of ​​tumor invasion into normal tissue can be reflected in clinical treatment and gene mutations. For example, the more severe the invasion, the more significantly the clinical data indicators will be different. Therefore, it can be used to help determine the region and depth to be segmented (used in 3D images) when segmenting lesion sites in MR sample images.

[0125] Gene mutation prediction is used to predict the type of gene mutation that a user may develop for a particular disease.

[0126] The iterative training of the sub-network and the iterative training of the parent network will be introduced separately below.

[0127] In the iterative training of the sub-network, since the preset goal of the sub-network is to make multiple sample features of the same user converge to the same center, and the sample features of different users are far away from the same center, that is, the sub-network needs to highly represent the user with multiple sample features extracted for the same user, so as to highlight the feature differences of different users, so that the extracted multiple sample features can be more helpful for the execution of downstream tasks.

[0128] In one implementation, to ensure the subnetwork achieves this predetermined goal, a loss function can be constructed during training based on the distances between various sample features. The subnetwork parameters are then updated based on the loss value obtained from this loss function. However, since it's necessary for sample features from different users to be far from the same center, and for multiple sample features from the same user to converge towards the same center, the loss function can be constructed simultaneously based on the distances between different sample features from different users, as well as the distances between different sample features from the same user. The loss value is then calculated, and the subnetwork parameters are updated accordingly.

[0129] In practice, a first loss value can be determined based on the distance between multiple sample features in each iteration of training; and the parameters of the sub-network can be adjusted based on the first loss value.

[0130] The distances between various sample features include the distances between sample features from different users, as well as the distances between sample features from the same user.

[0131] In this embodiment, the distance between multiple sample features of different users and the distance between multiple sample features of the same user can be determined. In one case, the distance can refer to the average distance, for example, the average of the distances between multiple sample features of different users and the average of the distances between multiple sample features of the same user.

[0132] In practice, the first loss value can be determined based on the ratio of the distance between multiple sample features of the same user to the distance between multiple sample features of different users. Based on the first loss value, the parameters of the sub-network are updated using the gradient update index that the multiple sample features of the same user converge to the same center.

[0133] In an optional embodiment, when obtaining the first loss value, for a batch of first sample groups input into the sub-network, feature extraction is performed on the sample data of all attributes included in the first sample group to obtain sample features including all attributes. It should be noted that these sample features may include sample features of different users for the same attribute, and the same user may include sample features of all different attributes.

[0134] For example, the sample data includes MR sample images and clinical sample data. The first sample group input into the subnetwork at one time includes sample data from 10 users. Thus, for MR type sample data, it includes the MR sample features of each of the 10 users, and for clinical type data, it includes the clinical sample features of each of the 10 users. For each user, it includes their own MR sample features and clinical sample features.

[0135] In this optional example, the distance can be calculated for the sample features of each pair of attributes. These sample features can include those of the same user and those of different users. In the case of sample features of different users, one user has sample features of one attribute, while the other user has sample features of the other attribute. Thus, a first loss value can be calculated based on the distances obtained for each pair of attribute sample features.

[0136] In practice, for each pair of attribute sample features, a first distance between the sample features of the two attributes belonging to the same user and a second distance between the sample features of the two attributes belonging to different users can be obtained; and a first loss value can be determined based on the first distance and the second distance corresponding to the sample features of each pair of attributes.

[0137] For example, the sample data includes MR sample images and clinical sample data. The first sample group input into the subnetwork at one time includes sample data from 10 users. When calculating the first loss value, the second distance between the MR sample features and clinical sample features of different users can be calculated, as well as the first distance between the MR sample features and clinical sample features of the same user. Since there are 10 users, there will be 10 first distances, while there are more than 10 second distances.

[0138] To achieve the aforementioned preset goal, the first distance needs to approach infinitesimally small, for example, infinitely close to 0, while the second distance needs to approach infinitely large, for example, infinitely close to 1. The loss function can also be designed according to this standard, thus obtaining the first loss value based on the designed loss function.

[0139] Specifically, the first loss value can be determined based on the ratio of the first distance to the second distance. In one case, this ratio can refer to the ratio of the distance between two types of sample features of the first user to the distance between different types of sample features of the first user and the second user. In practice, this can be understood as comparing the distance between the sample features of user A itself with the distance between the sample features of user A and other users to obtain the difference between the two, so as to more effectively interpret the differences between the features of the same user and the features of different users.

[0140] Updating the parameters of a subnetwork may involve constructing a loss function.

[0141] In an optional embodiment, a designed loss function is provided. Specifically, based on the first distance and the second distance corresponding to the sample features of each pair of attributes, the following loss function (a) is constructed:

[0142]

[0143] The first loss value is determined based on the loss values ​​obtained from the loss function for each pair of attribute sample features.

[0144] Where L(v,g) represents the loss between sample features v and sample features g, and b represents the batch of the first sample group input into the subnetwork;

[0145] A sample feature representing an attribute of the j-th user. Let Zv represent a sample feature that is different from Zv in another attribute of the j-th user. This represents a sample feature of the k-th user that is different from Zv in another attribute.

[0146] τ represents the temperature parameter, and cos represents the cosine distance.

[0147] The temperature coefficient is a hyperparameter, and a small temperature coefficient is generally used for self-supervised contrastive learning. The temperature coefficient adjusts the focus on difficult samples: a smaller temperature coefficient emphasizes separating the sample from the most similar difficult samples. If the temperature coefficient is set too large, the loss function treats all negative samples equally, resulting in a lack of prioritization in model learning; if the temperature coefficient is set too small, the model will overemphasize particularly difficult negative samples, leading to poor convergence or generalization ability.

[0148] As shown in the loss function above, taking an example where the sample data includes MR sample images and clinical sample data, and the first sample group input into the subnetwork contains sample data from 10 users. For example, it could be a sample feature of the MR attribute of the j-th user. For example, it could be a sample feature of the genetic data attributes of the j-th user. For example, it could be a sample feature of the genetic data attributes of the kth user.

[0149] Thus, the formula above shows that the ratio of the first distance based on different sample features of the same user to the sum of the second distances of different sample features of different users determines the corresponding loss value.

[0150] Where cos represents the cosine distance, its determination formula can be: The closer two features are to 1, the more similar they are. v represents one sample feature, and g represents another sample feature.

[0151] Furthermore, the first loss value can be determined based on the ratio of the first distance between each user's different sample features and the sum of the second distances between that user and the different sample features of all other users. For example, obtaining the first distance between user A's different sample features and the sum of the second distances between user A and the different sample features of every other user, the ratio of these two can reflect the difference between a user's sample features and the sample features of all other users. Thus, as the model is trained, it can gradually reflect the sample features that represent the individual characteristics of each user.

[0152] In some embodiments, when the sample data includes two or more types, a loss value can be calculated for each pair of samples according to the loss function (a) described above. Then, due to the way each pair of sample data is combined, multiple combinations can be obtained. For example, if there are three types of sample data, there are six combinations. In this way, multiple loss values ​​can be obtained. In practice, multiple loss values ​​can be weighted and summed to obtain the first loss value.

[0153] In practice, the smaller the numerator (first distance) and the larger the denominator (sum of second distances), the smaller the first loss value will be, according to this formula, which can satisfy the feature extraction target of the sub-network.

[0154] When using this implementation method, designing the loss function as described above can help the subnetwork reach a convergence state quickly, and can highlight the personalized sample features of different users, as well as some common features of different users, so that the subnetwork can extract sample features that effectively reflect user characteristics.

[0155] In some alternative embodiments, when the multiple sample data include MRI sample images, clinical sample data, and gene sample data, in each iteration of training, when determining the first loss value based on the distance between multiple sample features, a loss can be calculated by combining only the MRI sample images and clinical sample data, and a loss can be calculated by combining the MRI sample images and gene sample data. This allows the sample features corresponding to the MRI sample images to be prioritized, ensuring that the features corresponding to the gene sample data and the features corresponding to the clinical sample data are close to the sample features corresponding to the MRI sample images, without requiring the features corresponding to the gene sample data and the features corresponding to the clinical sample data to be close. In this way, gene data and clinical data can be primarily used to assist in the training of the image model, reducing the loss calculation during model training, decreasing the computational load, and enabling the model to converge faster and train the sub-network more quickly.

[0156] In practice, in each iteration of training, a first sub-loss value can be determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the clinical sample data; and a second sub-loss value can be determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the gene sample data; then, a first loss value can be determined based on the first sub-loss value and the second sub-loss value.

[0157] It should be noted that the distance includes the distance between sample features from different users, as well as the distance between sample features from the same user. In other words, the distance between the sample features corresponding to MR sample images and the sample features corresponding to clinical sample data includes both the distance between the aforementioned sample features from the same user and the distance between the aforementioned sample features from different users. Similarly, this applies to the distance between the sample features corresponding to MR sample images and the sample features corresponding to gene sample data.

[0158] The process of determining the first and second sub-loss values ​​can be found in the loss function (I) above, and will not be repeated here. When determining the first loss value based on the first and second sub-loss values, a weight can be set, and the first and second sub-loss values ​​can be weighted and summed to obtain the first loss value.

[0159] Regarding the structure of the sub-network, refer to Figure 2 As shown, a schematic diagram of the structure of a sub-network of this application is illustrated. Figure 2 As shown, a sub-network can include multiple feature extraction units, where different feature extraction units correspond to different sample data. In other words, in the subsequent inference stage, they can correspond to different user data.

[0160] Specifically, the feature extraction unit can be designed with specific network types such as CNN, multilayer perceptron, and long short-term memory (LSTM) to adaptively extract features from sample data of different formats. For example, CNN can be used for feature extraction of image data, while multilayer perceptron and LSM can be used for feature extraction of text data.

[0161] In one implementation, since the sub-network is used to extract features from various sample data to obtain sample features, in practice, since different types of sample data are used to describe user data with different attributes, the format of the sample data can be different. As mentioned above, it can include sample data in image format and user data in text format. Specifically, since different types of sample data correspond to different feature extraction units, the corresponding network can be used to perform targeted feature extraction based on the format, size, etc. of the sample data.

[0162] In practice, when there are at least two types of sample data, including MRI sample images, clinical sample data and gene sample data, the feature extraction unit corresponding to the clinical sample data is a long short-term memory network, and the feature extraction unit corresponding to the gene sample data is a multilayer perceptron.

[0163] In another implementation, the gene sample data may include multiple gene data, and the corresponding feature extraction unit may also include a feature extraction subunit corresponding to each type of gene data. Each feature extraction subunit can be an independent multilayer perceptron.

[0164] In some embodiments, such as Figure 2 As shown, when multiple sample data, including MRI sample images, clinical sample data, and gene sample data, are used, the subnetwork has multiple feature extraction branches, including radiomics feature extraction branch, clinical feature extraction branch, and gene feature extraction branch.

[0165] Among them, such as Figure 2 As shown, the radiomics feature extraction branch includes a convolution module, a residual module, a dimension transformation module, a first self-attention module, and a first perceptron module connected in sequence.

[0166] The clinical feature extraction branch includes a vector transformation module, at least one second self-attention module, a first self-attention module, and a second perceptron module connected in sequence.

[0167] The gene feature extraction branch includes a vector transformation module, a long short-term memory network module, and a third perceptron module connected in sequence.

[0168] The clinical feature extraction branch and the radiomics feature extraction branch can share the first self-attention module, so that the first perceptron module and the second perceptron module are connected after the first self-attention module, respectively.

[0169] This network architecture enables the separate extraction of MRI sample images, clinical sample data, and gene sample data. Since MRI sample images generally describe the structure and morphological features of tissues, convolutional modules, residual modules, and dimension transformation modules, which are advantageous for image feature extraction, are utilized. For clinical sample data and gene sample data, which are text-based data, vector transformation modules and long short-term memory network modules, which are advantageous for text feature extraction, are used for feature extraction, thereby improving the advantage in text-based feature extraction.

[0170] For MR sample images, feature extraction can be performed using residual networks, such as ResNet networks. Figure 3 As shown, a schematic diagram of the steps for feature extraction from MR sample images is presented. Figure 3 As shown, the specific steps may include:

[0171] Step S301: After performing convolution operations at multiple scales on the MR sample image in sequence, the extracted radiomics features are subjected to dimensional transformation to obtain the initial radiomics features.

[0172] In this embodiment, convolution operations at different scales can acquire image features of MR sample images for different receptive fields. That is, through a convolution operation at one scale, spatial features at one scale can be obtained. After convolution operations at multiple scales, it can be understood as gradually expanding the receptive field. Then, the extracted radiomics features can be dimensionally transformed to obtain initial radiomics features. In this way, the initial radiomics features can reflect both local and global features.

[0173] The dimensionality transformation of the extracted radiomics features can refer to increasing the feature dimensions through Linear Projection (linear regression). Specifically, MR sample images can be input into the convolution module, residual module, and dimensionality transformation module to perform the operation in step S301 above.

[0174] Step S302: Perform self-attention mechanism feature extraction on the initial radiomics features to enhance the distinction between lesion features and normal tissue features in the MR sample images, and obtain the sample features corresponding to the MR sample images.

[0175] In this embodiment, a self-attention mechanism can be used to extract features from the initial radiomics features. For example, features can be extracted using a Transformer network to enhance the distinction between lesion features and normal tissue features in the MR sample image. In other words, the difference between the features representing lesion areas and the features representing normal tissue in the initial image features is enhanced, thereby facilitating image segmentation in downstream tasks.

[0176] Of course, in some alternative examples, multilayer perceptual mapping can also be performed on the sample features of the MR after passing through the Transformer network to obtain the sample features corresponding to the final MR sample image.

[0177] This multilayer perceptual mapping can be understood as: classifying the features enhanced by the Transformer network to distinguish the foreground (lesion) and background (normal tissue) of the image.

[0178] Specifically, the initial radiomics features can be input into the first self-attention module and the first perceptron module to perform the operation of step S302 above.

[0179] Since the sample features corresponding to the MR sample image can reflect the imaging characteristics of the lesion site, it can be used as a reference for the appearance and morphology of the lesion site in target prediction, such as medical image segmentation and prognosis prediction.

[0180] When this implementation method is used, because the MR sample images are convolutional at multiple scales, radiomics features reflecting global characteristics can be extracted from a large receptive field, thereby enhancing the feature reflection of the morphology of the lesion site and improving the accuracy of downstream tasks.

[0181] In one embodiment, a schematic diagram is provided on how to extract sample features from gene sample data. In this embodiment, the gene sample data may include various types of gene data, such as single nucleotide diversity data, multi-gene scoring data, and gene association analysis data.

[0182] In practice, the gene sample data can be transformed into a feature vector to obtain the corresponding gene feature vector; the gene feature vector can then be mapped to the sample features corresponding to the gene sample data.

[0183] Since gene sample data is generally text-based, including strings and numerical values, one approach is to use Huffman coding to encode the location information of SNPs and add the mutation type to the last position to obtain the gene feature vector. Then, LSTM (Long Short-Term Memory) blocks are used for feature extraction, followed by MLP projection layers (perceptron layers, i.e.,...) Figure 2 The third perceptron module shown maps the extracted features to sample features corresponding to the gene sample data.

[0184] In this embodiment, single nucleotide polymorphism (SNP) mainly refers to the diversity of DNA sequences caused by variations in a single nucleotide at the genomic level. These SNPs are numerous and exhibit rich polymorphism. Some SNP sites can also affect gene function, leading to changes in biological traits or even disease. SNPs are important evidence for studying genetic variation in human families and animal and plant strains, and are therefore widely used in population genetics research (such as the origin, evolution, and migration of organisms) and the study of disease-related genes. They also play an important role in pharmacogenomics, diagnostics, and biomedical research.

[0185] Specifically, gene sample data can be input into the above... Figure 2 The vector transformation module, long short-term memory network module, and third perceptron module shown are used to perform the above operations.

[0186] In one implementation, when multiple sample data include clinical sample data, feature extraction of the clinical sample data can be performed as follows:

[0187] Each piece of clinical information in the clinical sample data is converted into a clinical feature vector; and a self-attention mechanism is used to extract features from the clinical feature vectors to cross-integrate the various pieces of clinical information in the clinical sample data, thereby obtaining the sample features corresponding to the clinical sample data.

[0188] One approach involves using relevant technologies to transform textual information into feature vectors. This converts various clinical information items in the clinical sample data into clinical feature vectors, which are then used for feature extraction. During feature extraction, a self-attention mechanism can be employed. This mechanism allows for the assessment of the correlation between different clinical information items in the clinical sample data. Based on the correlation between these items, the corresponding feature vectors are fused to obtain the sample features corresponding to the clinical sample data.

[0189] Of course, in some embodiments, the features extracted via the self-attention mechanism can be input into the perceptron to classify the extracted features, thereby extracting features to describe disease categories.

[0190] Specifically, clinical sample data can be input into the aforementioned clinical feature extraction branch to perform the feature extraction process described above. It should be noted that the clinical feature extraction branch may include at least one second self-attention module connected in series. This means that the feature vectors of various clinical information can undergo multiple attention perceptions to continuously enhance the cross-correlation between various clinical information in the clinical sample data, thereby extracting key features used to describe the user's disease.

[0191] In the construction of the parent network, after training the sub-networks, a parent network containing the sub-networks can be constructed based on the sub-networks.

[0192] In one implementation, two ways to construct the parent network are given. One way is to construct it based on the trained sub-network, that is, to directly use the trained sub-network as the feature extraction network of the parent network.

[0193] In practice, the parent network can be obtained by concatenating the network layers corresponding to the target prediction after the trained sub-network. These network layers, based on the target prediction, include at least fully connected layers and activation function layers, and the specific design can be tailored to the actual target prediction needs; details will not be elaborated here.

[0194] This approach allows for the direct addition of network layers to the sub-networks to obtain the parent network, resulting in high construction efficiency. Furthermore, the parent network can be flexibly constructed based on the specific target prediction task. In one application, the trained sub-networks can be published on a platform. When target prediction is needed, the sub-networks can be downloaded from the platform, and the required parent network can then be constructed. Finally, the parent network can be fine-tuned using a labeled second set of samples.

[0195] Another approach is to transfer the parameters of the trained subnetwork, that is, to use the trained parameters of the subnetwork as the initial parameters of the new parent network.

[0196] In practice, an initial network corresponding to the target prediction can be obtained, and the parameters of the trained sub-networks can be transferred to the initial network to obtain the parent network.

[0197] In this implementation, the parameters of the subnetwork can be transferred to the parent network, allowing for fine-tuning of the parent network's parameters using a labeled second sample set. Alternatively, the subnetwork can be published on a platform. When target prediction is needed, the subnetwork's parameters can be downloaded from the platform, transferred to the parent network, and then fine-tuned using a labeled second sample set.

[0198] In the iterative training of the parent network, since the parent network is used to complete the target prediction task, in practice, when there are at least two of the various sample data, including MRI sample images, clinical sample data, and gene sample data, the target prediction can be any one of prognostic prediction, image segmentation, or gene mutation prediction. In this case, the prediction label can be an image segmentation label, i.e., the actual image segmentation result, such as the segmentation result of a tumor image; or it can be a gene mutation label, i.e., the actual type of gene mutation; or it can be a prognostic survival label, i.e., the actual prognostic survival time.

[0199] In this way, when training the second network, the target prediction result output by the parent network can be obtained; and based on the target prediction result and the prediction label corresponding to the second sample group, the second loss value can be determined; then, based on the second loss value, the parameters of the parent network can be adjusted.

[0200] In this embodiment, when determining the second loss value based on the target prediction result and the prediction label corresponding to the second sample group, the second loss value can be a value between 0 and 1, which can be used to characterize the gap between the target prediction result and the prediction label. During training, the parameters of the mother network are adjusted with the goal of minimizing this gap.

[0201] The model training method of this application will be illustrated below with a specific example:

[0202] Reference Figure 5 As shown, an exemplary schematic diagram of a model training method is illustrated, such as... Figure 5 As shown, 501 represents the radiomics feature extraction branch, 502 represents the clinical feature extraction branch, and 503 represents the gene feature extraction branch. The radiomics feature extraction branch 501 includes a convolutional module (initial Block), a residual module (Residual Block), a dimensionality transformation module (Linear projection), a first self-attention module (share Transformer Block), and a first perceptron module (MLP projection).

[0203] The clinical feature extraction branch 502 includes: a vector transformation module Embedding Layer, two second self-attention modules Transformer Blocks, a first self-attention module shared Transformer Block, and a second perceptron module MLP projection.

[0204] The gene feature extraction branch 503 includes a vector transformation module Embedding Layer, a long short-term memory network module LSTM Block, and a third perceptron module MLP projection, which are connected in sequence.

[0205] Taking target prediction as a prognostic prediction as an example, the first sample group includes MRI sample images, clinical sample data, and gene sample data.

[0206] First, using the first sample group pair Figure 2 The subnetwork shown is pre-trained. During pre-training, based on the multiple sample features of the same user and the multiple sample features of different users output by the subnetwork, the distance between multiple sample features of the same user and the distance between multiple sample features of different users are calculated.

[0207] Here, assuming the feature corresponding to the MR sample image is represented by Zv, the sample feature corresponding to the clinical sample data is represented by Zt, and the sample feature corresponding to the gene sample data is represented by Zg, the loss between Zv and Zg, and the loss between Zv and Zt, can be calculated according to the loss function (I) above. Then, the first loss value is obtained according to the following formula (II):

[0208] Next, it can be based on L cont (v,g,t)=λL(v,g)+(1-λ)L(v,t) (Formula 2);

[0209] Where L(v,g) represents the loss between Zv and Zg, L(v,t) represents the loss between Zv and Zt, and λ is an empirical coefficient.

[0210] Next, based on the first loss value, the parameters of the subnetwork are updated until the subnetwork converges to a certain extent. For example... Figure 5 As shown, when updating the subnetwork, the preset model of the subnetwork is that multiple sample features of the same user converge to the same center, that is, the sample features of the same user are clustered into a cluster.

[0211] Then, based on the prognostic prediction, a feature concatenation layer and a fully connected layer are concatenated after the trained sub-network to obtain the parent network.

[0212] The fully connected layer is connected in series after the feature concatenation layer. The feature concatenation layer is used to concatenate the various sample features output by the sub-network and then feed them into the fully connected layer. The fully connected layer is used to output the prognostic prediction obtained after processing the concatenated features, that is, the predicted user survival time.

[0213] Next, a second sample group was obtained from multiple users. The second sample group included MRI sample images, clinical sample data, and genetic sample data, and carried prognostic labels.

[0214] The second sample group is input into the parent network. The second loss value is calculated according to the prognostic prediction value and prognostic label output by the parent network. Based on the second loss value, the parameters of the parent network are updated to obtain the prognostic prediction model.

[0215] Based on the same inventive concept, this application also proposes a prediction method based on multiple user data, referring to... Figure 4 As shown, a flowchart illustrating the steps of this method is presented. Figure 4 As shown, the specific steps may include:

[0216] Step S401: Obtain multiple user data of the user to be predicted, wherein the multiple user data are used to describe user characteristics with different attributes.

[0217] In one application scenario, multiple types of user data may include at least two of image data, clinical data, and genetic data;

[0218] Step S402: Input the various user data into the target model corresponding to the target prediction. The target model is trained using the model training method described in the above embodiments.

[0219] Step S403: Obtain the prediction results output by the target model; wherein, the prediction results include the classification prediction results of the user's category.

[0220] In one application scenario, where multiple user data may include at least two of image data, clinical data, and genetic data, the prediction result may also be at least one of prognostic survival, user category, tumor image segmentation, and gene mutation results.

[0221] In this embodiment, the term "multiple user data" and "multiple sample data" in the first sample group can be understood equivalently. In this application, which can be applied to the medical field, the term "multiple user data" includes at least two of the following: image data, clinical data, and genetic data.

[0222] In this embodiment, the target model can be trained using the model training method described in the embodiment.

[0223] In practice, using the model training method described in the embodiments, a sub-network for sharing can be obtained. Then, based on this sub-network, multiple target models applied to different target predictions can be obtained. Furthermore, for each parent network, it can belong to various user data and obtain the predicted results.

[0224] For example, for a target model of prognostic survival, imaging data, clinical data, and genetic data can be input into the target model to obtain the prognostic survival.

[0225] For the target model of image segmentation, image data and clinical data can be input into the target model to obtain the image segmentation results of the lesion site.

[0226] For the target model of gene mutation results, clinical data and genetic data can be input into the target model to obtain the gene mutation results, depending on the situation. Alternatively, imaging data, clinical data, and genetic data can be input into the target model to obtain the gene mutation classification results.

[0227] Based on the same inventive concept, this application also proposes a model training device, referring to... Figure 6 As shown, a schematic diagram of the device is presented, such as... Figure 6 As shown, the device includes the following modules:

[0228] The first sample acquisition module 601 is used to acquire the first unlabeled sample group corresponding to each of multiple users. The first sample group includes multiple types of sample data, and different types of sample data are used to describe user characteristics with different attributes.

[0229] The pre-training module 602 is used to perform unsupervised iterative pre-training on the sub-network to be trained based on multiple first sample groups and with a preset target as a constraint; wherein, the sub-network is used to extract features for each sample data to obtain sample features corresponding to each sample data, and the preset target is: multiple sample features of the same user converge to the same center.

[0230] The network construction module 603 is used to construct a parent network that includes trained sub-networks; wherein the parent network is used to perform target prediction based on sample features extracted by the sub-networks, and the target prediction includes at least a classification prediction of the user's category;

[0231] The fine-tuning training module 604 is used to perform supervised iterative training on the parent network based on the second sample groups corresponding to multiple users, to obtain the target model; wherein, the second sample group includes multiple sample data and carries prediction labels corresponding to the target prediction.

[0232] Optionally, the multiple sample data includes at least two of the following: nuclear magnetic resonance (MRI) sample images, clinical sample data, and genetic sample data;

[0233] The target prediction further includes at least one of prognostic prediction, image segmentation, and gene mutation prediction; wherein, the image segmentation is used to segment the image region belonging to the target type in the MR sample image.

[0234] Optionally, the pre-training module 602 includes:

[0235] The loss value determination unit is used to determine a first loss value based on the distance between multiple sample features in each iteration of training; wherein, the distance between the multiple sample features includes the distance between sample features of different users and the distance between sample features of the same user;

[0236] An update unit is used to adjust the parameters of the sub-network based on the first loss value.

[0237] Optionally, the loss value determination unit includes:

[0238] The distance determination subunit is used to obtain, for each pair of attribute sample features, a first distance between sample features of the two attributes belonging to the same user, and a second distance between sample features of the two attributes belonging to different users;

[0239] The loss determination subunit is used to determine the first loss value based on the first distance and the second distance corresponding to the sample features of each of the two attributes.

[0240] Optionally, the loss determination subunit is specifically used for:

[0241] Based on the first distance and the second distance corresponding to the sample features of each of the two attributes, the following loss function is constructed:

[0242]

[0243] The first loss value is determined based on the loss value obtained in the loss function for each of the two attributes.

[0244] Where L(v,g) represents the loss between sample features v and sample features g, and b represents the batch of the first sample group input into the subnetwork;

[0245] A sample feature representing an attribute of the j-th user. Let Zv represent a sample feature that is different from Zv in another attribute of the j-th user. This represents a sample feature of the k-th user that is different from Zv in another attribute;

[0246] τ represents the temperature parameter, and cos represents the cosine distance.

[0247] Optionally, the multiple sample data includes MRI sample images, clinical sample data, and gene sample data. The step of determining a first loss value based on the distance between multiple sample features in each training iteration includes:

[0248] In each iteration of training, the first sub-loss value is determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the clinical sample data.

[0249] A second sub-loss value is determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the gene sample data; wherein, the distance includes the distance between sample features of different users and the distance between sample features of the same user;

[0250] The first loss value is determined based on the first sub-loss value and the second sub-loss value.

[0251] Optionally, the network construction module 603 is specifically used to concatenate the network layer corresponding to the target prediction after the trained sub-network to obtain the parent network; wherein the network layer includes at least a fully connected layer according to the target prediction; or, to obtain the initial network corresponding to the target prediction and transfer the parameters of the trained sub-network to the initial network to obtain the parent network.

[0252] Optionally, fine-tuning the training module includes:

[0253] The result acquisition unit is used to acquire the target prediction result output by the parent network each time the parent network is iterated and trained.

[0254] The loss determination unit is used to determine a second loss value based on the target prediction result and the prediction label corresponding to the second sample group;

[0255] The parameter update unit is used to adjust the parameters of the parent network based on the second loss value.

[0256] Optionally, the multiple sample data includes nuclear magnetic resonance (MR) sample images, and the step of extracting features from each type of sample data to obtain the sample features corresponding to each type of sample data includes:

[0257] After performing convolution operations at multiple scales on the MR sample images in sequence, the extracted radiomics features are subjected to dimensionality transformation to obtain initial radiomics features.

[0258] The initial radiomics features are subjected to self-attention mechanism feature extraction to enhance the distinction between lesion features and normal tissue features in the MR sample image, thereby obtaining the sample features corresponding to the MR sample image.

[0259] Optionally, the multiple sample data includes clinical sample data, and the step of extracting features from each type of sample data to obtain the sample features corresponding to each type of sample data includes:

[0260] Convert each piece of clinical information in the clinical sample data into a clinical feature vector;

[0261] The clinical feature vector is subjected to self-attention mechanism feature extraction to cross-integrate various clinical information in the clinical sample data to obtain the sample features corresponding to the clinical sample data.

[0262] Optionally, the sub-network includes: a radiomics feature extraction branch, a clinical feature extraction branch, and a gene feature extraction branch, wherein:

[0263] The image omics feature extraction branch includes a convolution module, a residual module, a dimension transformation module, a first self-attention module, and a first perceptron module connected in sequence.

[0264] The clinical feature extraction branch includes a vector transformation module, at least one second self-attention module, a first self-attention module, and a second perceptron module connected in sequence.

[0265] The gene feature extraction branch includes a vector transformation module, a long short-term memory network module, and a third perceptron module connected in sequence.

[0266] Based on the same inventive concept, this application also proposes a prediction device based on multiple user data, referring to... Figure 7 As shown, a schematic diagram of the device is presented, such as... Figure 7 As shown, the device includes the following modules:

[0267] The data acquisition module 701 is used to acquire various user data of the user to be predicted, wherein the various user data are used to describe user characteristics with different attributes;

[0268] Input module 702 is used to input various user data into a target model corresponding to the target prediction, wherein the target model is trained by the model training method.

[0269] The result acquisition module 703 is used to acquire the prediction results output by the target model; wherein, the prediction results include the classification prediction results of the user's category.

[0270] Reference Figure 8 As shown, a structural block diagram of an electronic device 800 according to an embodiment of the present disclosure is illustrated. Figure 8 As shown in the embodiment of the present invention, an electronic device 800 is provided, which can be used to execute model training methods or prediction methods based on multiple user data.

[0271] Electronic device 800 may include a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being configured to perform a model training method or a prediction method based on a variety of user data.

[0272] like Figure 8 As shown, in one embodiment, the electronic device may include an input device, an output device, and a data acquisition device. When executing the model training method or the prediction method based on multiple user data of the present disclosure embodiment, the data acquisition device can acquire multiple user data. Then, the input device can obtain the multiple user data from the data acquisition device. The multiple user data can be processed by a processor. Specifically, the processing can execute the above-mentioned model training method or the prediction method based on multiple user data. The output device can output a sub-network or output the prediction result output by the target model.

[0273] Of course, in one embodiment, the memory may include volatile memory and non-volatile memory. Volatile memory can be understood as random access memory used to store and retain data. Non-volatile memory refers to computer memory whose stored data will not disappear when the power is turned off. Of course, the computer program of the model training method or prediction method based on multiple user data disclosed herein may be stored in volatile memory and non-volatile memory, or in either one.

[0274] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program that causes a processor to execute the model training method or, when executed, implement the prediction method based on multiple user data.

[0275] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the model training method described above, or, when executed, implements the prediction method based on multiple user data.

[0276] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0277] Finally, it should be noted that, unless otherwise defined, the terms "first," "second," and similar terms used herein do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0278] The above provides a detailed description of a model training method, a prediction method based on multiple user data, an apparatus, and a device provided by this disclosure. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0279] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0280] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0281] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0282] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0283] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This disclosure can be implemented by means of hardware comprising a plurality of different elements and by means of a suitably programmed computer. In a unit claim enumerating a plurality of means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0284] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A model training method, characterized in that, The method includes: Obtain unlabeled first sample groups corresponding to multiple users. The first sample group includes multiple types of sample data, and different types of sample data are used to describe user characteristics with different attributes. The multiple types of sample data include at least two of the following: nuclear magnetic resonance (MRI) sample images, clinical sample data, and genetic sample data. Based on multiple first sample groups, the sub-network to be trained is subjected to unsupervised iterative pre-training with a preset objective as a constraint; wherein, the sub-network is used to extract features for each sample data to obtain sample features corresponding to each sample data, and the preset objective is: multiple sample features of the same user converge to the same center. A parent network is constructed that includes the trained sub-networks; wherein the parent network is used to perform target prediction based on the sample features extracted by the sub-networks, and the target prediction includes at least a classification prediction of the user's category; the target prediction also includes at least one of prognostic prediction, image segmentation, and gene mutation prediction; wherein the image segmentation is used to segment the image regions in the MR sample image that belong to the target type; Based on the second sample groups corresponding to multiple users, the parent network is subjected to supervised iterative training to obtain the target model; wherein, the second sample group includes multiple sample data and carries prediction labels corresponding to the target prediction.

2. The method according to claim 1, characterized in that, Based on multiple first sample groups, unsupervised iterative training is performed on the sub-network to be trained under the constraint of a preset objective, including: In each iteration of training, a first loss value is determined based on the distance between multiple sample features; wherein, the distance between the multiple sample features includes the distance between sample features of different users, and the distance between sample features of the same user; Based on the first loss value, the parameters of the sub-network are adjusted.

3. The method according to claim 2, characterized in that, The determination of the first loss value based on the distance between multiple sample features includes: For each pair of attribute sample features, obtain the first distance between the sample features of the two attributes belonging to the same user, and the second distance between the sample features of the two attributes belonging to different users; The first loss value is determined based on the first distance and the second distance corresponding to the sample features of each of the two attributes.

4. The method according to claim 3, characterized in that, Determining the first loss value based on the first distance and the second distance corresponding to the sample features of each of the two attributes includes: Based on the first distance and the second distance corresponding to the sample features of each of the two attributes, the following loss function is constructed: ； The first loss value is determined based on the loss value obtained in the loss function for each of the two attributes. in, represents the loss between sample features v and sample features g, and b represents the batch size of the first sample group input into the subnetwork; A sample feature representing an attribute of the j-th user. Let Zv represent a sample feature that is different from Zv in another attribute of the j-th user. This represents a sample feature of the k-th user that is different from Zv in another attribute; The temperature parameter is represented by , and cos represents the cosine distance.

5. The method according to claim 2, characterized in that, The various sample data include MRI sample images, clinical sample data, and gene sample data. In each training iteration, a first loss value is determined based on the distance between the various sample features, including: In each iteration of training, the first sub-loss value is determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the clinical sample data. A second sub-loss value is determined based on the distance between the sample features corresponding to the MR sample image and the sample features corresponding to the gene sample data; wherein, the distance includes the distance between sample features of different users and the distance between sample features of the same user; The first loss value is determined based on the first sub-loss value and the second sub-loss value.

6. The method according to claim 1, characterized in that, Constructing a parent network that includes the trained sub-networks includes: The parent network is obtained by splicing the sub-networks obtained through training with the network layers corresponding to the target prediction; wherein the network layers include at least fully connected layers according to the target prediction. Alternatively, an initial network corresponding to the target prediction can be obtained, and the parameters of the trained sub-network can be transferred to the initial network to obtain the parent network.

7. The method according to claim 1, characterized in that, Based on the second sample groups corresponding to multiple users, the parent network undergoes supervised iterative training, including: During each iteration of training the parent network, the target prediction result output by the parent network is obtained; Based on the target prediction result and the prediction label corresponding to the second sample group, a second loss value is determined; The parameters of the parent network are adjusted based on the second loss value.

8. The method according to claim 1, characterized in that, The various sample data include nuclear magnetic resonance (MRI) sample images. The feature extraction for each type of sample data to obtain the corresponding sample features includes: After performing convolution operations at multiple scales on the MR sample images in sequence, the extracted radiomics features are subjected to dimensionality transformation to obtain initial radiomics features. The initial radiomics features are subjected to self-attention mechanism feature extraction to enhance the distinction between lesion features and normal tissue features in the MR sample image, thereby obtaining the sample features corresponding to the MR sample image.

9. The method according to claim 1, characterized in that, The various sample data include clinical sample data. The feature extraction for each type of sample data to obtain the corresponding sample features includes: Convert each piece of clinical information in the clinical sample data into a clinical feature vector; The clinical feature vector is subjected to self-attention mechanism feature extraction to cross-integrate various clinical information in the clinical sample data to obtain the sample features corresponding to the clinical sample data.

10. The method according to claim 1, characterized in that, The sub-network includes: a radiomics feature extraction branch, a clinical feature extraction branch, and a gene feature extraction branch, wherein: The image omics feature extraction branch includes a convolution module, a residual module, a dimension transformation module, a first self-attention module, and a first perceptron module connected in sequence. The clinical feature extraction branch includes a vector transformation module, at least one second self-attention module, a first self-attention module, and a second perceptron module connected in sequence. The gene feature extraction branch includes a vector transformation module, a long short-term memory network module, and a third perceptron module connected in sequence.

11. A prediction method based on multiple user data, characterized in that the method includes: Obtain various user data for the user to be predicted, wherein the various user data are used to describe user characteristics with different attributes; The various user data are input into the target model corresponding to the target prediction, and the target model is trained by the model training method according to any one of claims 1-10; Obtain the prediction results output by the target model; wherein, the prediction results include the classification prediction results of the user's category.

12. A model training device, characterized in that, The device includes: The first sample acquisition module is used to acquire unlabeled first sample groups corresponding to multiple users. The first sample group includes multiple types of sample data, and different types of sample data are used to describe user characteristics with different attributes. The multiple types of sample data include at least two of the following: nuclear magnetic resonance (MRI) sample images, clinical sample data, and gene sample data. The pre-training module is used to perform unsupervised iterative pre-training on the sub-network to be trained based on multiple first sample groups and with a preset target as a constraint; wherein, the sub-network is used to extract features for each sample data to obtain sample features corresponding to each sample data, and the preset target is: multiple sample features of the same user converge to the same center. A network construction module is used to construct a parent network containing trained sub-networks; wherein the parent network is used to perform target prediction based on sample features extracted by the sub-networks, and the target prediction includes at least a classification prediction of the user's category; the target prediction also includes at least one of prognostic prediction, image segmentation, and gene mutation prediction; wherein the image segmentation is used to segment image regions belonging to the target type in the MR sample image; The fine-tuning training module is used to perform supervised iterative training on the parent network based on the second sample groups corresponding to multiple users to obtain the target model; wherein, the second sample group includes multiple sample data and carries prediction labels corresponding to the target prediction.

13. A prediction device based on multiple user data, characterized in that, The device includes: The data acquisition module is used to acquire various user data of the user to be predicted, and the various user data are used to describe user characteristics with different attributes; An input module is used to input various user data into a target model corresponding to the target prediction, wherein the target model is trained by the model training method according to any one of claims 1-10; The result acquisition module is used to acquire the prediction results output by the target model; wherein, the prediction results include the classification prediction results of the user's category.

14. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the model training method as described in any one of claims 1-10, or, when executed, implements the prediction method based on multiple user data as described in claim 11.

Images