Risk prediction method and system, electronic equipment and storage medium

Abstract

The invention discloses a risk prediction method and system, electronic equipment and a storage medium, and relates to the technical field of risk prediction, and the method comprises the steps: carrying out the preprocessing of each preset image, and obtaining a plurality of image blocks; feature extraction is carried out on each image block, a first risk probability prediction value of a first preset event corresponding to each image block is obtained, and then image level features corresponding to each preset image are generated; fusing the image level feature and the interpretive text data corresponding to each preset image to obtain a plurality of samples; based on the plurality of samples, training a second learning model to obtain a trained second learning model; the trained first learning model and the trained second learning model are used for conducting risk prediction on the second preset event, and the first risk probability prediction value of the first preset event corresponding to each image block represents the importance degree of the image block to the second preset event. The risk prediction precision can be effectively improved.

Description

Background Art

[0002] The treatment and prognosis of gastric cancer have always been the focus of medical research. Surgical resection is the main radical treatment for gastric cancer. Although it can significantly improve the survival rate in early patients, postoperative recurrence remains the main challenge affecting the long-term survival of patients. According to statistics, the postoperative recurrence rate of gastric cancer patients is as high as 50% to 70%, most of which occur within two years after surgery, especially in the high-risk recurrence stage within one year after surgery. In addition, the patterns of gastric cancer recurrence are diverse, including local recurrence, peritoneal metastasis and distant metastasis, which increases the complexity of prediction and the difficulty of clinical treatment.

[0003] The mechanism of gastric cancer recurrence is complex and is affected by many factors, such as the patient's tumor stage, histological characteristics, surgical method, adjuvant therapy effect, and individual biological characteristics of the patient (such as gene mutations and molecular expression profiles). In addition, gastric cancer also shows a high degree of heterogeneity. Even for patients with the same tumor stage, their postoperative recurrence risk and prognosis may differ significantly. Therefore, it is difficult to accurately predict the risk of recurrence by relying solely on a single dimension of clinical indicators or pathological data, which poses a huge challenge to clinical decision-making.

[0004] In recent years, artificial intelligence (AI) technology has made breakthrough progress in the application of medicine. For example, deep learning models based on pathological images can automatically extract the microstructural features of tumors, while clinical text models based on natural language processing technology can effectively analyze key information in electronic medical records and laboratory reports. However, most existing studies are still limited to the analysis of single-modal data, making it difficult to fully explore the potential associations between different modal data, thus limiting the performance of the model in predicting gastric cancer recurrence.

[0005] In summary, although the existing technology has made some progress in gastric cancer recurrence prediction, there are still problems such as insufficient prediction accuracy and lack of multimodal data fusion capabilities. How to effectively combine the characteristics of pathological images and clinical text data and build an accurate prediction model through advanced artificial intelligence technology is a key issue that needs to be solved in the current field of gastric cancer recurrence prediction. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and specifically provide a risk prediction method, system, electronic device and storage medium, as follows:

[0007] 1) In the first aspect, the present invention provides a risk prediction method, and the specific technical solution is as follows:

[0008] Acquire a plurality of preset images and explanatory text data of each preset image;

[0009] Preprocess each preset image to obtain multiple blocks;

[0010] Using the trained first learning model, feature extraction is performed on each image block, and a first risk probability prediction value of a first preset event corresponding to each image block is obtained;

[0011] Generate an image-level feature corresponding to each preset image according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event;

[0012] The image-level features and explanatory text data corresponding to each preset image are fused to obtain multiple samples;

[0013] Based on the multiple samples, the second learning model is trained to obtain a trained second learning model;

[0014] Using the trained first learning model and the trained second learning model to perform risk prediction on the second preset event;

[0015] The first risk probability prediction value of the first preset event corresponding to each image block represents the importance of the image block to the second preset event.

[0016] The beneficial effects of a risk prediction method provided by the present invention are as follows:

[0017] The trained first learning model is used to extract features from each image block, and a first risk probability prediction value of the first preset event corresponding to each image block is obtained. The first risk probability prediction value of the first preset event corresponding to each image block represents the importance of the image block to the second preset event, and can selectively focus on the image block that is most meaningful to the second preset time, thereby generating highly expressive image-level features, thereby improving the prediction accuracy of the trained second learning model, which can not only predict the probability of gastric cancer recurrence, but also is suitable for risk prediction of other second preset events (such as the risk of damage to medical equipment or the risk of loss of control of vehicles, etc.).

[0018] Based on the above scheme, a risk prediction method of the present invention can also be improved as follows.

[0019] Further, according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event, an image level feature corresponding to each preset image is generated, including:

[0020] The first risk probability prediction value of the first preset event corresponding to each tile is used as the attention score, and the extracted features of each tile corresponding to each preset image are weighted and merged through the attention mechanism to generate image-level features corresponding to each preset image.

[0021] Further, based on the multiple samples, the second learning model is trained to obtain a trained second learning model, including:

[0022] A cross entropy loss function with a regularization term is adopted, and the second learning model is trained using a gradient descent method to obtain a trained second learning model.

[0023] Furthermore, the trained first learning model is a trained EfficientNetB0 model, and the trained second learning model is a trained support vector machine.

[0024] 2) In the second aspect, the present invention also provides a risk prediction system, and the specific technical solution is as follows:

[0025] It includes data acquisition module, preprocessing module, feature extraction risk prediction module, image level feature generation module, fusion module, model training module and risk prediction module;

[0026] The data acquisition module is used to: acquire multiple preset images and explanatory text data of each preset image;

[0027] The preprocessing module is used to: preprocess each preset image to obtain multiple image blocks;

[0028] The feature extraction risk prediction module is used to: use the trained first learning model to extract features from each image block respectively, and obtain a first risk probability prediction value of a first preset event corresponding to each image block;

[0029] The image level feature generation module is used to: generate an image level feature corresponding to each preset image according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event;

[0030] The fusion module is used to: fuse the image-level features and explanatory text data corresponding to each preset image to obtain multiple samples;

[0031] The model training module is used to: train the second learning model based on multiple samples to obtain a trained second learning model;

[0032] The risk prediction module is used to: use the trained first learning model and the trained second learning model to perform risk prediction on the second preset event;

[0033] The first risk probability prediction value of the first preset event corresponding to each image block represents the importance of the image block to the second preset event.

[0034] Based on the above solution, a risk prediction system of the present invention can also be improved as follows.

[0035] Furthermore, the image-level feature generation module is used to:

[0036] The first risk probability prediction value of the first preset event corresponding to each tile is used as the attention score, and the extracted features of each tile corresponding to each preset image are weighted and merged through the attention mechanism to generate image-level features corresponding to each preset image.

[0037] Furthermore, the model training module is used to:

[0038] A cross entropy loss function with a regularization term is adopted, and the second learning model is trained using a gradient descent method to obtain a trained second learning model.

[0039] Furthermore, the trained first learning model is a trained EfficientNetB0 model, and the trained second learning model is a trained support vector machine.

[0040] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program, and the at least one computer program being loaded and executed by the processor so that the electronic device implements any of the above-mentioned risk prediction methods.

[0041] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the above-mentioned risk prediction methods is implemented.

[0042] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention:

[0044] Figure 1 A schematic diagram of a risk prediction method according to an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the structure of a risk prediction system according to an embodiment of the present invention;

[0046] Figure 3 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0048] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, a risk prediction method according to an embodiment of the present invention comprises the following steps:

[0050] S1, obtaining a plurality of preset images and explanatory text data of each preset image;

[0051] S2, preprocessing each preset image to obtain multiple blocks;

[0052] S3, using the trained first learning model to extract features from each image block, and obtaining a first risk probability prediction value of a first preset event corresponding to each image block;

[0053] S4, generating an image-level feature corresponding to each preset image according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event;

[0054] S5, fusing the image-level features and explanatory text data corresponding to each preset image to obtain multiple samples;

[0055] S6. Based on the multiple samples, the second learning model is trained to obtain a trained second learning model;

[0056] S7, using the trained first learning model and the trained second learning model to perform risk prediction on the second preset event;

[0057] The first risk probability prediction value of the first preset event corresponding to each image block represents the importance of the image block to the second preset event.

[0058] The specific implementation process of using the trained first learning model and the trained second learning model to predict the risk of the second preset event includes:

[0059] Acquire a target image, preprocess the target image to obtain multiple blocks corresponding to the target image, recorded as target blocks, use the trained first learning model to extract features of each target block, and obtain a first risk probability prediction value of a first preset event corresponding to each target block, generate image-level features corresponding to the target image based on the extracted features of each target block and the first risk probability prediction value of the first preset event, input the image-level features corresponding to the target image into the trained second learning model, and obtain a second risk probability prediction value of the second preset event.

[0060] Optionally, in the above technical solution, generating image-level features corresponding to each preset image according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event includes:

[0061] The first risk probability prediction value of the first preset event corresponding to each tile is used as the attention score, and the extracted features of each tile corresponding to each preset image are weighted and merged through the attention mechanism to generate image-level features corresponding to each preset image.

[0062] Optionally, in the above technical solution, the second learning model is trained based on multiple samples to obtain a trained second learning model, including:

[0063] A cross entropy loss function with a regularization term is adopted, and the second learning model is trained using a gradient descent method to obtain a trained second learning model.

[0064] Optionally, in the above technical solution, the trained first learning model is: a trained EfficientNetB0 model, and the trained second learning model is a trained support vector machine. Other learning models can also be selected for training according to actual conditions. For example, convolutional neural networks or long short-term memory artificial neural networks are trained using different samples to obtain a trained first learning model and a trained second learning model.

[0065] The present invention is illustrated by the following examples.

[0066] Embodiment 1:

[0067] In Example 1, the preset image is: a pathological image of a gastrectomy patient, and the explanatory text data of the pathological image of the gastrectomy patient is clinical text data, and the clinical text data includes: age, gender, gastric cancer stage (i.e., the stage of gastric cancer, such as early stage, middle stage, and late stage, etc.) and treatment method of the gastrectomy patient. The first preset event is: suffering from gastric cancer, and the first risk probability prediction value of the first preset event refers to: the probability prediction value of suffering from gastric cancer, the second preset event is: gastric cancer recurrence, and the second risk probability prediction value of the second preset event refers to: the probability prediction value of gastric cancer recurrence, the trained first training model is the trained EfficientNetB0 model, and the trained second training model is the trained support vector machine.

[0068] Embodiment 1 specifically comprises the following steps:

[0069] S101. Collect pathological images and clinical text data of multiple gastrectomy patients (including confirmed gastric cancer patients and non-gastric cancer patients who underwent gastrectomy due to other conditions). The number can be set according to actual conditions, such as 136 cases or 236 cases.

[0070] S102, preprocessing each pathological image obtained in S101 to obtain a plurality of image blocks, specifically:

[0071] S1020. Each pathological image is divided into a plurality of initial image blocks of a first preset size, which are recorded as first initial image blocks. It is determined whether the proportion of the lesion tissue area in each first initial image block is less than a first preset proportion threshold value, and a first judgment result is obtained. The first initial image block with a first judgment result of "no" is determined as a first target initial image block, and a plurality of first target initial image blocks are obtained, wherein the first preset proportion threshold value can be set according to actual conditions. For example, the first preset proportion threshold value can be 30% or 40%, etc. The first initial image block with a first judgment result of "yes" can be deleted, and the first initial image block with a first judgment result of "yes" can also be saved for subsequent backtracking, until each pathological image is processed, and a plurality of first target initial image blocks corresponding to each pathological image are obtained.

[0072] S1021. At a first preset magnification, each first target initial image block is tiled and cropped into an image block of a second preset size, and at a second preset magnification, each first target initial image block is cut into a plurality of image blocks of a third preset size, which are recorded as first sub-image blocks, and then each first sub-image block is scaled to an image block of a second preset size. All image blocks of the second preset size corresponding to each pathological image are image blocks corresponding to the pathological image, which are all recorded as first image blocks. Specifically:

[0073] At a magnification of 20× (first preset magnification), each first target initial tile is tiled and cropped into a tile of 224×224 size (second preset size), and at a magnification of 5× (second preset magnification), each first target initial tile is cut into sub-tiles of 896×896 size (third preset size), recorded as first sub-tiles, and then each first sub-tile is scaled to 224×224 size (second preset size) to obtain multiple tiles, and the tiles obtained here are all recorded as first tiles.

[0074] Among them, the first preset size, the second preset size, the third predicted size, the first preset magnification and the second preset magnification can be set according to actual conditions.

[0075] S103. Use the trained EfficientNetB0 model to perform feature extraction on each first image block respectively to obtain the extracted features of each first image block, and obtain a first risk probability prediction value of gastric cancer corresponding to each first image block, where the value range of the first risk probability prediction value is [0,1].

[0076] Among them, the process of obtaining the trained EfficientNetB0 model includes:

[0077] A plurality of first samples are obtained, and an EfficientNetB0 model is trained based on the plurality of first samples to obtain a trained EfficientNetB0 model.

[0078] The pathological image of each gastric cancer patient and the pathological image of each non-gastric cancer patient are obtained and segmented to obtain a plurality of second sub-blocks, and the gastric cancer lesion tissue in the second sub-block segmented from the pathological image of each gastric cancer patient is marked, and the edge information and texture information of the gastric cancer lesion tissue are marked. Through expert discussion, the gastric cancer lesion tissue in the second sub-block segmented from each pathological image is analyzed to determine the probability of gastric cancer caused by the gastric cancer lesion tissue in the second sub-block segmented from each pathological image. The gastric cancer lesion tissue in the second sub-block segmented from the pathological image of each non-gastric cancer patient can be marked as 0 to indicate that there is no gastric cancer lesion tissue, and the probability of gastric cancer caused by the second sub-block segmented from the pathological image of each gastric cancer patient is marked as 0. Each marked gastric cancer lesion tissue and the second sub-block of the probability of gastric cancer constitute a sample, recorded as the first sample.

[0079] The input of the EfficientNetB0 model is: the second sub-image block, and the output of the EfficientNetB0 model is: the predicted position of gastric cancer lesion tissue and the corresponding edge information and texture information, etc., as well as the predicted value of the probability of suffering from gastric cancer. Based on multiple first samples, the EfficientNetB0 model is trained to obtain a trained EfficientNetB0 model.

[0080] In another implementation, the trained EfficientNetB0 model is obtained by training the EfficientNetB0 model based on the ImageNet data set to obtain a pre-trained EfficientNetB0 model, and using the pre-trained EfficientNetB0 model to predict each first block to fine-tune the trained EfficientNetB0 model to obtain a trained EfficientNetB0 model. In short, the invention uses the trained EfficientNet B0 to extract image features and further obtain the prediction probability of each first block as a new feature and then combines it with the text feature (explanatory text data) to construct a feature set of a sample (i.e., the image-level features and explanatory text data corresponding to each preset image are fused to obtain multiple samples, and the multiple samples constitute the feature set of the sample), and then the trained support vector machine is used to predict the risk of gastric cancer recurrence. The advantage is that when the trained EfficientNet B0 model is used to extract image features, two methods are used (at a first preset magnification, each first target initial tile is tiled and cropped into a tile of a second preset size, and at a second preset magnification, each first target initial tile is divided into multiple tiles of a third preset size, namely, first sub-tiles, and then each first sub-tile is scaled to a tile of a second preset size, and all tiles of the second preset size corresponding to each pathological image are the tiles corresponding to the pathological image) to obtain features at multiple scales, which can improve the prediction accuracy.

[0081] The trained EfficientNetB0 model is used to extract features from each first image block to obtain the extracted features of each first image block, that is, to obtain the predicted position of gastric cancer lesion tissue in each first image block and the corresponding edge information and texture information, etc., and to obtain the first risk probability prediction value of gastric cancer corresponding to each first image block.

[0082] S104, generating image-level features corresponding to each pathological image obtained in S101:

[0083] The first risk probability prediction value of gastric cancer corresponding to each first image block is used as an attention score. The MIL method (i.e., weakly supervised learning method) is used to weight and merge the extracted features of each first image block corresponding to each preset image through the attention mechanism to generate image-level features corresponding to each pathological image, wherein the attention score corresponding to each first image block is the corresponding weight, and the weight can measure the importance of the extracted features to subsequent tasks. The first image block with a larger weight has a greater impact on the prediction of gastric cancer recurrence, and the first image block with a smaller weight has a smaller impact on the prediction of gastric cancer recurrence. Specifically, the predicted position of the gastric cancer lesion tissue and the corresponding edge information and texture information of each first image block corresponding to each pathological image can be weighted and merged by weighted averaging, feature splicing, or feature stacking to generate image-level features corresponding to each preset image. The image-level features corresponding to each pathological image include: the weighted merged extracted features of the pathological image.

[0084] It should be noted that the trained EfficientNetB0 model outputs: the feature vector corresponding to each first image block, each corresponding feature vector can represent the predicted position of gastric cancer lesion tissue and the corresponding edge information and texture information, etc. Therefore, the above-mentioned process of weighted merging the predicted position of gastric cancer lesion tissue and the corresponding edge information and texture information, etc. of each first image block corresponding to each preset image, specifically through the process of weighted merging of the feature vectors corresponding to each first image block, then the image-level features corresponding to each preset image can also be considered as: the weighted merged feature vector of the preset image.

[0085] S105, fusing the image-level features and explanatory text data corresponding to each preset image obtained in S104, the specific fusing process is as follows:

[0086] Semantic information is extracted from the text using word embedding, word vectors, or Transformer models such as BERT and GPT. These models can convert words or sentences in the text into vector representations, and then use feature concatenation to fuse the image-level features and explanatory text data corresponding to each preset image obtained in S104 to obtain multiple samples, which are recorded as second samples.

[0087] S106: Based on the multiple second samples, train the support vector machine to obtain a trained support vector machine.

[0088] During the training process, the cross entropy loss function is used for image-level prediction training, and regularization terms are added to reduce overfitting. The model is trained using the gradient descent method (Adam optimizer) to continuously optimize the patch-level feature extractor and learning rate to improve the accuracy of recurrence prediction.

[0089] During the training process, the plurality of second samples are split into a training set and a test set in an 8:2 form, and the performance of the trained support vector machine is evaluated on the test set.

[0090] S107, preprocessing the target image to obtain a plurality of target blocks corresponding to the target image. The preprocessing process may be:

[0091] Each pathological image is divided into a plurality of initial blocks of a fourth preset size, which are recorded as second initial blocks. At a third preset magnification, each second initial block is tiled and cropped into a block of a fifth preset size. At a fourth preset magnification, each first target initial block is cut into blocks of a sixth preset size, which are recorded as second sub-blocks. Each second sub-block is then scaled to a block of a fifth preset size. All blocks of the fifth preset size are a plurality of target blocks corresponding to the target image, which are recorded as first target blocks. Specifically:

[0092] The target image is divided into multiple initial tiles according to a preset size, recorded as second initial tiles. At a 20× magnification (third preset magnification), each second initial tile is tiled and cropped into a target tile of 224×224 size (fifth preset size), and at a 5× magnification (fourth preset magnification), each second initial tile is cut into sub-tiles of 896×896 size (sixth preset size), recorded as second sub-tiles, and then each second sub-tile is scaled to 224×224 size (fifth preset size) to obtain multiple target tiles, and the target tiles obtained here are all recorded as first target tiles.

[0093] Among them, the fourth preset size, the fifth preset size, the sixth preset size, the third preset magnification and the fourth preset magnification can be set according to actual conditions.

[0094] The trained EfficientNetB0 model is used to perform feature extraction on each first target block to obtain the extracted features of each first target block, and the first risk probability prediction value of gastric cancer corresponding to each first target block is obtained.

[0095] The first risk probability prediction value of gastric cancer corresponding to each first target block is used as the attention score. The MIL method (i.e., weakly supervised learning method) is used to weightedly merge the extracted features of each first target block through the attention mechanism to generate image-level features corresponding to the target image.

[0096] The image-level features corresponding to the target image are input into the trained support vector machine to obtain the second risk probability prediction value of gastric cancer recurrence.

[0097] Among them, the target image can be specified by the doctor, and the trained support vector machine can also output the location of gastric cancer lesion tissue and the corresponding edge information and texture information, etc. The trained EfficientNetB0 model can also output the location of the cancer lesion tissue on each first target block, the probability prediction value of gastric cancer, and the corresponding edge information and texture information, etc. This provides important support for the interpretability of the trained support vector machine output (the second risk probability prediction value of gastric cancer recurrence) and can assist clinical decision-making.

[0098] The beneficial effects of Example 1 are as follows:

[0099] The prediction of postoperative recurrence of gastric cancer is of great significance to the formulation of treatment strategies and survival prognosis of patients. This study processed pathological images based on the Multiple Instance Learning (MIL) method, and further combined clinical text features to construct a prediction model. By utilizing the complementary information of pathological images and clinical text data, the prediction accuracy can be improved and the stability of the model can be enhanced, thereby providing transparent and reliable auxiliary support for clinical applications. This study established a model for predicting patient prognosis based on multimodal data, and the AUROC (area under the ROC curve) was able to reach above 0.8, and further explained the impact of different modal features on patients with gastric cancer recurrence.

[0100] Embodiment 2:

[0101] In Example 2, the preset image is: a welding image of medical equipment, for example, a welding image of components such as a frame, a supporting structure and a circuit board of an X-ray machine, a CT scanner or a magnetic resonance imaging device. The explanatory text data of the welding image includes: welding process parameters and welding quality. The welding process parameters include: welding current, electrode diameter, welding voltage, welding speed, arc length and welding temperature, etc. The first preset event is: judging that the welding quality is unqualified. The first risk probability prediction value of the first preset event refers to: the probability prediction value of judging that the welding quality is unqualified. The second preset event is: an abnormality occurs at the welding position. The second risk probability prediction value of the second preset event refers to: the probability prediction value of the abnormality occurs at the welding position. The trained first training model is the trained EfficientNetB0 model, and the trained second training model is the trained support vector machine.

[0102] Embodiment 2 specifically comprises the following steps:

[0103] S201, collecting welding images and explanatory text data of multiple medical devices, and the number of welding images can be set according to actual conditions.

[0104] S202, preprocessing each welding image obtained in S201 to obtain multiple blocks, specifically:

[0105] S2020. Each welding image is divided into multiple initial blocks of a first preset size, recorded as third initial blocks, and it is determined whether the proportion of the welding area in each third initial block is less than a second preset proportion threshold, to obtain a second judgment result, and the third initial block for which the second judgment result is no is determined as the second target initial block, to obtain multiple second target initial blocks, wherein the second preset proportion threshold may be set according to actual conditions, for example, the second preset proportion threshold may be 30% or 40%, etc., the third initial block for which the second judgment result is yes may be deleted, and the third initial block for which the second judgment result is yes may also be saved for subsequent backtracking until each welding image is processed to obtain the second target initial block corresponding to each welding image.

[0106] S2021, at the first preset magnification, tiling and cropping each second target initial image block into an image block of a second preset size, and at the second preset magnification, cutting each second target initial image block into a plurality of image blocks of a third preset size, recorded as third sub-image blocks, and then scaling each third sub-image block to an image block of the second preset size, and all image blocks of the second preset size corresponding to each welding image are recorded as image blocks corresponding to the welding image, and are all recorded as second image blocks, specifically:

[0107] At a magnification of 20× (the first preset magnification), each second target initial image block is tiled and cropped into an image block of 224×224 size (the second preset size), and at a magnification of 5× (the second preset magnification), each second target initial image block is cut into sub-image blocks of 896×896 size (the third preset size), which are recorded as third sub-image blocks, and then each third sub-image block is scaled to 224×224 size (the second preset size) to obtain multiple image blocks, which are all recorded as second image blocks;

[0108] S203, using the trained EfficientNetB0 model to perform feature extraction on each second image block respectively, to obtain the extracted features of each second image block, and to obtain the first risk probability prediction value for determining that the welding quality is unqualified corresponding to each second image block.

[0109] Among them, the process of obtaining the trained EfficientNetB0 model includes:

[0110] A plurality of third samples are obtained, and an EfficientNetB0 model is trained based on the plurality of third samples to obtain a trained EfficientNetB0 model.

[0111] The welding image of each medical device is obtained and segmented to obtain multiple fourth sub-blocks, and the welding position area in the fourth sub-block segmented from the welding image of each medical device is marked, and the edge information and texture information of the welding position area are marked; the welding position area in the fourth sub-block segmented from the welding image of each medical device is analyzed by expert argumentation to determine the probability of unqualified welding quality caused by the welding position area in the fourth sub-block segmented from the welding image of each medical device; non-welding position areas are marked as 0, and the probability of unqualified welding quality caused by the non-welding position areas is marked as 0; each marked welding position area and the fourth sub-block with the probability of unqualified welding quality constitute a sample, recorded as the third sample.

[0112] Among them, the input of the EfficientNetB0 model is: the fourth sub-block, and the output of the EfficientNetB0 model is: the predicted position of the welding position area and the corresponding edge information and texture information, etc., as well as the predicted value of the probability of determining that the welding quality is unqualified. Based on multiple third samples, the EfficientNetB0 model is trained to obtain a trained EfficientNetB0 model.

[0113] The trained EfficientNetB0 model is used to extract features of each second image block to obtain the extracted features of each second image block, that is, the predicted position of the welding position area in each second image block and the corresponding edge information and texture information, etc., and the first risk probability prediction value for judging the welding quality as unqualified corresponding to each second image block is obtained.

[0114] S204, generating image-level features corresponding to each welding image obtained in S201:

[0115] The first risk probability prediction value for determining unqualified welding quality corresponding to each second image block is used as an attention score. The MIL method (i.e., weakly supervised learning method) is used to weight and merge the extracted features of each second image block corresponding to each welding image through the attention mechanism to generate image-level features corresponding to each welding image, wherein the attention score corresponding to each second image block is the corresponding weight, which can measure the importance of the extracted features to subsequent tasks. The second image block with a larger weight has a greater impact on the prediction of abnormalities at the welding position, and the second image block with a smaller weight has a smaller impact on the prediction of abnormalities at the welding position. Specifically, the predicted position of the welding position area of ​​each second image block corresponding to each welding image and the corresponding edge information and texture information can be weighted and merged by weighted averaging, feature splicing, or feature stacking to generate image-level features corresponding to each welding image. The image-level features corresponding to each welding image include: the weighted merged extracted features of the welding image.

[0116] It should be noted that the trained EfficientNetB0 model outputs: the feature vector corresponding to each second block, each corresponding feature vector can represent the predicted position of the welding position area and the corresponding edge information and texture information, etc. Therefore, the above-mentioned process of weighted merging the predicted position of the welding position area of ​​each second block corresponding to each welding image and the corresponding edge information and texture information, etc., specifically through the process of weighted merging of the feature vectors corresponding to each second block, the image-level features corresponding to each welding image can also be considered as: the weighted merged feature vector of the welding image.

[0117] S205, fusing the image-level features and explanatory text data corresponding to each welding image obtained in S204, and the specific fusing process is as follows:

[0118] Semantic information is extracted from the text using word embedding, word vector or Transformer models such as BERT and GPT. These models can convert words or sentences in the text into vector representations, and then use feature concatenation to fuse the image-level features and explanatory text data corresponding to each welding image obtained in S204 to obtain multiple samples, which are recorded as the fourth samples.

[0119] S206: Based on the plurality of fourth samples, train a support vector machine to obtain a trained support vector machine.

[0120] During the training process, the cross entropy loss function is used for image-level prediction training, and regularization terms are added to reduce overfitting. The model is trained using the gradient descent method (Adam optimizer) to continuously optimize the patch-level feature extractor and learning rate to improve the accuracy of recurrence prediction.

[0121] During the training process, the plurality of fourth samples are split into a training set and a test set in the form of 8:2. The performance of the trained support vector machine is evaluated on the test set.

[0122] S207, preprocessing the target image to obtain a plurality of target blocks corresponding to the target image. The preprocessing process may be:

[0123] The target image is divided into a plurality of initial tiles of a fourth preset size, which are recorded as fourth initial tiles. At a third preset magnification, each fourth initial tile is tiled and cropped into a tile of a fifth preset size. At a fourth preset magnification, each second target initial tile is cut into tiles of a sixth preset size, which are recorded as fourth sub-tiles. Each fourth sub-tile is then scaled to a tile of a fifth preset size. All tiles of the fifth preset size are a plurality of target tiles corresponding to the target image, which are recorded as second target tiles. Specifically:

[0124] The target image is divided into multiple initial tiles according to a preset size, recorded as fourth initial tiles. At a 20× magnification (third preset magnification), each fourth initial tile is tiled and cropped into a target tile of 224×224 size (fifth preset size), and at a 5× magnification, each fourth initial tile is cut into sub-tiles of 896×896 size (sixth preset size), recorded as fourth sub-tiles, and then each fourth sub-tile is scaled to 224×224 size (fifth preset size) to obtain multiple target tiles. The target tiles obtained here are all recorded as second target tiles.

[0125] The trained EfficientNetB0 model is used to extract features of each second target block respectively, to obtain the extracted features of each second target block, and to obtain the first risk probability prediction value for judging the welding quality as unqualified corresponding to each second target block.

[0126] The first risk probability prediction value for judging unqualified welding quality corresponding to each second target block is used as the attention score. The MIL method (i.e., weakly supervised learning method) is used to weightedly merge the extracted features of each second target block through the attention mechanism to generate image-level features corresponding to the target image.

[0127] The image-level features corresponding to the target image are input into the trained support vector machine to obtain the second risk probability prediction value of abnormal welding position.

[0128] Among them, the target image can be specified by the doctor, and the trained support vector machine can also output the position of the welding position area and the corresponding edge information and texture information, etc. The trained EfficientNetB0 model can also output the position of the welding position area on each second target block, the probability prediction value of judging the welding quality as unqualified, and the corresponding edge information and texture information, etc. This provides important support for the interpretability of the trained support vector machine output (the second risk probability prediction value of abnormal welding position), which can assist in the improvement of subsequent welding process.

[0129] The beneficial effects of Example 2 are as follows:

[0130] By utilizing the complementary information of welding images and explanatory text data of welding images, the prediction accuracy of whether an abnormality occurs at the welding position can be improved.

[0131] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0132] like Figure 2 As shown, a risk prediction system 200 according to an embodiment of the present invention includes a data acquisition module 201, a preprocessing module 202, a feature extraction risk prediction module 203, an image level feature generation module 204, a fusion module 205, a model training module 206 and a risk prediction module 207;

[0133] The data acquisition module 201 is used to: acquire a plurality of preset images and explanatory text data of each preset image;

[0134] The preprocessing module 202 is used to: preprocess each preset image to obtain multiple image blocks;

[0135] The feature extraction risk prediction module 203 is used to: use the trained first learning model to extract features from each image block respectively, and obtain a first risk probability prediction value of a first preset event corresponding to each image block;

[0136] The image level feature generation module 204 is used to generate an image level feature corresponding to each preset image according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event;

[0137] The fusion module 205 is used to: fuse the image-level features and explanatory text data corresponding to each preset image to obtain multiple samples;

[0138] The model training module 206 is used to: train the second learning model based on multiple samples to obtain a trained second learning model;

[0139] The risk prediction module 207 is used to: use the trained first learning model and the trained second learning model to perform risk prediction on the second preset event;

[0140] The first risk probability prediction value of the first preset event corresponding to each image block represents the importance of the image block to the second preset event.

[0141] Optionally, in the above technical solution, the image level feature generation module 204 is used to:

[0142] The first risk probability prediction value of the first preset event corresponding to each tile is used as the attention score, and the extracted features of each tile corresponding to each preset image are weighted and merged through the attention mechanism to generate image-level features corresponding to each preset image.

[0143] Optionally, in the above technical solution, the model training module 206 is used to:

[0144] A cross entropy loss function with a regularization term is adopted, and the second learning model is trained using a gradient descent method to obtain a trained second learning model.

[0145] Optionally, in the above technical solution, the trained first learning model is: a trained EfficientNetB0 model, and the trained second learning model is a trained support vector machine.

[0146] It should be noted that the beneficial effects of a risk prediction system 200 provided in the above embodiment are the same as the beneficial effects of the above risk prediction method, which will not be described in detail here. In addition, when the system provided in the above embodiment realizes its functions, it only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described in detail here.

[0147] Among them, the risk prediction system of the present invention can be a computer program (including program code) running in a computer device. For example, the risk prediction system of the present invention is an application software that can be used to execute the corresponding steps in the risk prediction method of the present invention.

[0148] In some embodiments, the risk prediction system of the present invention can be implemented by a combination of software and hardware. As an example, the risk prediction system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the risk prediction method of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0149] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The name of a module does not limit the module itself in some cases.

[0150] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned risk prediction methods is implemented. That is to say, an electronic device according to an embodiment of the present invention may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing a risk prediction method shown in any embodiment of the present invention by calling a computer program.

[0151] In an alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0152] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0153] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 In the figure, only one thick line is used to represent bus 4002, but this does not mean that there is only one bus or one type of bus.

[0154] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0155] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0156] Among them, the electronic device can also be a terminal device, and the terminal device can be any device that can install applications, including at least one of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0157] It should be noted that Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0158] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned risk prediction methods is implemented.

[0159] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0160] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned risk prediction methods.

[0161] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0162] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The computer-readable storage medium provided by the embodiment of the present invention may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or component.

[0164] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0165] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0166] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0167] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0168] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A risk prediction method, characterized in that: include: Obtaining a plurality of preset images and explanatory text data for each preset image; Preprocess each preset image to obtain multiple blocks; Using the trained first learning model, feature extraction is performed on each image block, and a first risk probability prediction value of a first preset event corresponding to each image block is obtained; Generate an image-level feature corresponding to each preset image according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event; The image-level features and explanatory text data corresponding to each preset image are fused to obtain multiple samples; Based on the multiple samples, the second learning model is trained to obtain a trained second learning model; Using the trained first learning model and the trained second learning model to perform risk prediction on a second preset event; The first risk probability prediction value of the first preset event corresponding to each image block represents the importance of the image block to the second preset event.

2. A risk prediction method according to claim 1, characterized in that: According to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event, an image level feature corresponding to each preset image is generated, including: The first risk probability prediction value of the first preset event corresponding to each tile is used as the attention score, and the extracted features of each tile corresponding to each preset image are weighted and merged through the attention mechanism to generate image-level features corresponding to each preset image.

3. A risk prediction method according to claim 1 or 2, characterized in that: Based on multiple samples, the second learning model is trained to obtain a trained second learning model, including: A cross entropy loss function with a regularization term added is adopted, and the second learning model is trained using a gradient descent method to obtain the trained second learning model.

4. A risk prediction method according to claim 1 or 2, characterized in that: The trained first learning model is a trained EfficientNetB0 model, and the trained second learning model is a trained support vector machine.

5. A risk prediction system, characterized in that: It includes data acquisition module, preprocessing module, feature extraction risk prediction module, image level feature generation module, fusion module, model training module and risk prediction module; The data acquisition module is used to: acquire a plurality of preset images and explanatory text data of each preset image; The preprocessing module is used to: preprocess each preset image to obtain multiple image blocks; The feature extraction risk prediction module is used to: use the trained first learning model to extract features from each image block respectively, and obtain a first risk probability prediction value of a first preset event corresponding to each image block; The image-level feature generation module is used to generate an image-level feature corresponding to each preset image according to the extracted features of each image block corresponding to each preset image and the first risk probability prediction value of the first preset event; The fusion module is used to: fuse the image-level features and explanatory text data corresponding to each preset image to obtain multiple samples; The model training module is used to: train the second learning model based on multiple samples to obtain a trained second learning model; The risk prediction module is used to: use the trained first learning model and the trained second learning model to perform risk prediction on a second preset event; The first risk probability prediction value of the first preset event corresponding to each image block represents the importance of the image block to the second preset event.

6. A risk prediction system according to claim 5, characterized in that: The image level feature generation module is used for: The first risk probability prediction value of the first preset event corresponding to each tile is used as the attention score, and the extracted features of each tile corresponding to each preset image are weighted and merged through the attention mechanism to generate image-level features corresponding to each preset image.

7. A risk prediction system according to claim 5 or 6, characterized in that: The model training module is used to: A cross entropy loss function with a regularization term added is adopted, and the second learning model is trained using a gradient descent method to obtain the trained second learning model.

8. A risk prediction system according to claim 5 or 6, characterized in that: The trained first learning model is a trained EfficientNetB0 model, and the trained second learning model is a trained support vector machine.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements a risk prediction method as described in any one of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a risk prediction method according to any one of claims 1 to 4 is implemented.

Images