A training method and device of a prediction model and a processor

Abstract

The application discloses a training method and device of a prediction model and a processor. In the case, a training sample set is obtained; anatomical site entities extracted from medical text data of multi-modal medical data of the training sample are mapped to corresponding spatial anchor points in a medical image coordinate system, a spatial bias matrix is constructed by an initial prediction model according to spatial distances between the spatial anchor points and image blocks of the medical image data; the model performs cross-attention fusion on the medical image data and the medical text data according to the spatial bias matrix to obtain an image-text fusion feature sequence, generates a prediction result according to the feature sequence and structured inspection data; and parameters of the initial prediction model are adjusted based on differences between the prediction result and a real label to obtain the prediction model. The application solves the technical problem of low prediction accuracy of the model caused by insufficient utilization of complementary information between multi-modal data in the prior art, and improves the auxiliary value of the prediction result for drug use and treatment of medical workers.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and processor for training a prediction model. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the breakthroughs in deep learning in medical image analysis, computer-aided diagnostic systems have demonstrated transformative potential in the early screening of various cancers, such as lung cancer. Studies have shown that effective cancer screening and risk identification at an early stage can help improve subsequent treatment outcomes for patients. Therefore, establishing accurate, stable, and real-world clinical auxiliary analysis methods for early lung cancer screening is of great significance.

[0003] Currently, multimodal data fusion has become an important direction for improving the effectiveness of lung cancer risk prediction. Existing technologies typically use image data as the main modality, and introduce auxiliary information such as clinical text and test indicators through feature splicing or fusion to build predictive models for risk prediction. However, this method only uses simple feature splicing or shallow fusion. Due to the heterogeneity of different modal data in structure and semantics (such as the spatiality of images, the narrative nature of text, and the numerical nature of indicators), simple splicing is insufficient to uncover the complex correlations and complementary information between different modal data. As a result, the model cannot effectively capture the weak early signals hidden in the multimodal data. This severely limits the sensitivity and specificity of the model's screening, resulting in low predictive accuracy. The predictive performance is insufficient to meet the requirements of high-confidence clinical decision-making, and therefore cannot provide highly reliable auxiliary evidence for the formulation of subsequent differentiated treatment plans (such as precision medicine, intervention timing selection, and treatment intensity adjustment), and cannot effectively support the optimization of individualized treatment and rehabilitation pathways. Summary of the Invention

[0004] Based on the above problems, this application provides a training method, apparatus and processor for a prediction model, with the aim of solving the technical problem of low prediction accuracy of existing models due to insufficient utilization of complementary information between multimodal data, and improving the auxiliary value of prediction results for medical workers in medication and treatment.

[0005] The embodiments of this application disclose the following technical solutions: In a first aspect, this application provides a method for training a prediction model, the method comprising: Obtain a training sample set for training the initial prediction model; the training sample set includes multiple training samples, each of which includes a set of multimodal medical data and corresponding real labels, the multimodal medical data including medical image data, medical text data and structured test data; Anatomical entities extracted from the medical text data of the multimodal medical data are mapped to corresponding spatial anchor points in the medical image coordinate system. The initial prediction model is then used to construct a spatial bias matrix based on the spatial distance between the spatial anchor points and the image blocks of the medical image data. Each spatial anchor point represents the center coordinate of an anatomical region in the medical image data. The spatial bias matrix is ​​used to quantify the spatial relative positional relationship between the anatomical entities and the image blocks. The initial prediction model performs cross-attention fusion on the medical image data and the medical text data based on the spatial bias matrix to obtain the image-text fusion feature sequence, and generates a prediction result based on the image-text fusion feature sequence and the structured test data. Based on the difference between the prediction results output by the initial prediction model and the true label, the parameters of the initial prediction model are adjusted to obtain the prediction model.

[0006] In an optional implementation, before obtaining the training sample set for training the initial prediction model, the method further includes: The system collects original medical image data, original medical text data, and original structured test data for each of the multiple sample objects. Based on the original medical image data of each sample object, it determines the timestamp, identity identifier, and corresponding real label of the medical image examination for that sample object. The original structured test data includes the name, test value, and test threshold range of each of the multiple test items. For each sample object, the corresponding timestamp of the sample object is used as the time anchor point, and the preset duration is used as the sliding step size to determine the corresponding target time window. Data within the target time window is filtered from the original medical text data and the original structured test data of the sample object to obtain the filtered medical text data and the filtered structured test data. For each sample object, the medical image examination event of the sample object is used as the index event. Based on the identity identifier, the original medical image data, the filtered medical text data, and the filtered structured test data of the sample object are associated to obtain initial multimodal medical data. The initial multimodal medical data is then cleaned to obtain the multimodal medical data. A training sample is constructed based on each of the multimodal medical data and its corresponding real label, and a training sample set is constructed based on all the training samples.

[0007] In an optional implementation, the step of mapping the anatomical site entities extracted from the medical text data of the multimodal medical data to corresponding spatial anchor points in the medical image coordinate system, and constructing a spatial bias matrix using the initial prediction model based on the spatial distance between the spatial anchor points and the image patches of the medical image data, includes: The medical image data is processed into a preset format to obtain processed medical image data. The processed medical image data is then feature-encoded to obtain a medical image feature sequence. The medical image feature sequence consists of the feature vectors of each image block in the medical image data and the spatial position information of the image block in the medical image coordinate system. The medical text data is converted into a string in a preset format to obtain a medical text string, and the medical text string is then feature-encoded to obtain an initial medical text feature sequence. The anatomical location entities in the medical text string are identified to obtain a set of anatomical location entities. The processed medical image data is then divided into multiple anatomical regions, and corresponding anatomical region masks are generated. The anatomical region masks are used to identify the anatomical region to which each pixel in the processed medical image data belongs. Based on the anatomical region mask and the set of anatomical parts, multiple spatial anchor points corresponding to the processed medical image data are determined, and each spatial anchor point is bound to the corresponding token of the initial medical text feature sequence to generate a medical text feature sequence. Using the initial prediction model, the spatial bias matrix is ​​constructed based on the spatial distance between the spatial anchor points in the medical text feature sequence and each of the image blocks.

[0008] In an optional implementation, the step of determining multiple spatial anchor points corresponding to the processed medical image data based on the anatomical region mask and the set of anatomical site entities, and binding each spatial anchor point to the corresponding token of the initial medical text feature sequence to generate a medical text feature sequence, includes: Based on the set of anatomical parts entities and the anatomical region mask, a mapping relationship between anatomical parts entities and anatomical regions is constructed; For each anatomical site entity, the corresponding anatomical region is determined based on the mapping relationship, and the center coordinates of the anatomical region are calculated. The center coordinates of each anatomical region are used as a spatial anchor point, and a set of spatial anchor points is constructed based on all the spatial anchor points; each spatial anchor point in the set of spatial anchor points corresponds to a physical entity of the anatomical part. For each anatomical site entity, the position information of the token corresponding to the anatomical site entity in the initial medical text feature sequence is determined, and the token is bound to the spatial anchor point corresponding to the anatomical site entity based on the position information of the token to obtain the medical text feature sequence.

[0009] In an optional implementation, the step of using the initial prediction model to construct the spatial bias matrix based on the spatial distance between the spatial anchor points in the medical text feature sequence and each of the image patches includes: Using the initial prediction model, for each token in the medical text feature sequence that is bound to the spatial anchor point, the spatial distance between the spatial anchor point corresponding to the token and each of the image blocks is calculated, and a spatial distance matrix is ​​constructed based on all the spatial distances; Using the initial prediction model, a distance decay transformation is performed on each element in the spatial distance matrix to obtain a corresponding bias value; the bias value is used to quantify the spatial proximity between the token and the image patch, and the bias value is negatively correlated with the spatial distance; The spatial bias matrix is ​​constructed using the initial prediction model and based on the bias values ​​corresponding to all the elements.

[0010] In an optional implementation, the step of performing cross-attention fusion on the medical image data and the medical text data based on the spatial bias matrix using the initial prediction model to obtain an image-text fusion feature sequence includes: Using the initial prediction model, a query matrix is ​​constructed based on the initial medical text feature sequence, and a key matrix and a value matrix are constructed based on the medical image feature sequence, respectively. Using the initial prediction model, cross-attention calculation is performed based on the query matrix and the key matrix to obtain an initial attention score matrix; each element in the initial attention score matrix is ​​used to reflect the semantic relevance between the token and the image patch; Using the initial prediction model, the target attention score matrix is ​​determined based on the initial attention score matrix and the spatial bias matrix; Using the initial prediction model, the value matrix is ​​weighted and summed based on the target attention score matrix to obtain the image-text fusion feature sequence.

[0011] In an optional implementation, generating the prediction result based on the image-text fusion feature sequence and the structured test data includes: Based on the test values ​​and threshold ranges of each test item in the structured test data, the gate weight corresponding to each test item is determined, and the name and test value of the test item are weighted according to the gate weight corresponding to each test item to generate a test feature sequence; the gate weight is used to quantify the degree to which the test value deviates from the normal range; Using the multimodal aggregator of the initial prediction model, global feature representations are extracted from the test feature sequence to obtain a first global feature vector, and global feature representations are extracted from the image-text fusion feature sequence to obtain a second global feature vector; The multimodal aggregator concatenates the first global feature vector and the second global feature vector along the token dimension to obtain a multimodal fused feature sequence. Using the joint encoding network of the initial prediction model, deep feature interaction processing is performed on the multimodal fusion feature sequence to obtain a multimodal joint feature representation; The prediction result is generated based on the multimodal joint feature representation using the classification prediction head of the initial prediction model.

[0012] In an optional implementation, the step of determining the gating weight corresponding to each inspection item based on the inspection value and threshold range of each inspection item in the structured inspection data, and then weighting the name and inspection value of the inspection item according to the gating weight corresponding to each inspection item to generate an inspection feature sequence includes: For each of the test items, the abnormal deviation degree corresponding to the test item is calculated based on the test value of the test item and the test threshold range, and the gating weight corresponding to the test item is calculated based on the abnormal deviation degree of the test item using a gating network. Text embedding is performed on the name of each of the test items to convert the name into a dense numerical vector of a preset length, thereby obtaining the item name embedding vector corresponding to each of the test items. Feature engineering is then performed on the test values ​​of each of the test items to obtain the numerical feature vector of the test values ​​corresponding to each of the test items. For each of the test items, the item name embedding vector corresponding to the test item and the test value numerical feature vector are combined to generate a single test feature sequence. The single test feature sequence is then weighted based on the gating weight corresponding to the test item to obtain the threshold-aware test feature. All the threshold-aware test features corresponding to the test items are spliced ​​together according to a preset splicing order to generate the test feature sequence.

[0013] A second aspect of this application provides a training apparatus for a prediction model, the apparatus comprising: The acquisition module is used to acquire a training sample set for training the initial prediction model; the training sample set includes multiple training samples, each of which includes a set of multimodal medical data and corresponding real labels, the multimodal medical data including medical image data, medical text data and structured test data; The construction module is used to map anatomical entities extracted from the medical text data of the multimodal medical data to corresponding spatial anchor points in the medical image coordinate system, and to construct a spatial bias matrix based on the spatial distance between the spatial anchor points and the image blocks of the medical image data using the initial prediction model; each spatial anchor point is the center coordinate of an anatomical region of the medical image data, and the spatial bias matrix is ​​used to quantify the spatial relative positional relationship between the anatomical entities and the image blocks; The prediction module is used to perform cross-attention fusion on the medical image data and the medical text data based on the spatial bias matrix using the initial prediction model to obtain an image-text fusion feature sequence, and to generate a prediction result based on the image-text fusion feature sequence and the structured test data. The parameter adjustment module is used to adjust the parameters of the initial prediction model based on the difference between the prediction result output by the initial prediction model and the true label, so as to obtain the prediction model.

[0014] In a third aspect, this application provides a processor for running a computer program, wherein the computer program executes the training method of the aforementioned prediction model during runtime.

[0015] Compared with the prior art, this application has the following beneficial effects: In this technical solution, a training sample set for training the initial prediction model is first obtained. This training sample set includes multiple training samples, each containing a set of multimodal medical data and corresponding ground truth labels. The multimodal medical data includes medical image data, medical text data, and structured test data, providing a comprehensive, cross-modal association learning data foundation for subsequent model training, thereby improving the model's generalization ability. Then, anatomical entities extracted from the medical text data of the multimodal medical data are mapped to corresponding spatial anchor points in the medical image coordinate system. These spatial anchor points can be the center coordinates of an anatomical region in the medical image data, realizing the transformation of abstract textual information into quantifiable spatial coordinates. This allows the model to accurately locate the anatomical region in the image corresponding to the textual description, thereby achieving... While enhancing the accuracy of feature localization, a bridge is established between text semantics and image space, solving the semantic heterogeneity problem between medical image data and medical text data. Then, based on the spatial distance between spatial anchors and image patches in medical image data, the initial prediction model constructs a spatial bias matrix to quantify the spatial relative positional relationship between anatomical entities and image patches. This spatial bias matrix enables the quantitative encoding of the spatial relative position between images and text, allowing the model to learn the spatial associations of different modal features. This solves the problem that existing technologies struggle to uncover complex associations and complementary information between different modal features, thus providing prior knowledge for subsequent cross-attention fusion. This guides the model to focus more on image regions related to the text description space during fusion, reducing interference from irrelevant features. Then, the initial prediction model performs cross-attention fusion on medical image data and medical text data based on the spatial bias matrix to obtain an image-text fusion feature sequence. This cross-attention mechanism enables bidirectional interaction between image and text features, allowing the model to learn higher-order semantic associations across modalities (such as the intrinsic connection between image lesions and text symptom descriptions). This enhances the model's ability to detect weak early signals. The initial prediction model then generates prediction results based on the image-text fusion feature sequence and structured test data, significantly improving the model's sensitivity and specificity in identifying lung cancer risk. Finally, based on the difference between the prediction results output by the initial prediction model and the true labels, the parameters of the initial prediction model are adjusted to obtain a new prediction model. Supervised optimization allows the model to gradually approximate real clinical scenarios, thereby improving the model's prediction accuracy and robustness. This ensures that the model's prediction performance meets the requirements of high-confidence clinical decision-making, providing highly reliable auxiliary evidence for the formulation of subsequent differentiated treatment plans and effectively supporting the optimization of individualized treatment and rehabilitation pathways. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a training method for a prediction model provided in an embodiment of this application; Figure 2 A flowchart illustrating the construction process of a spatial bias matrix provided in this application embodiment; Figure 3 A flowchart illustrating the generation process of a medical text feature sequence as provided in this application embodiment; Figure 4 A flowchart illustrating the generation process of a test feature sequence, provided for an embodiment of this application; Figure 5 A flowchart illustrating another process for constructing a spatial bias matrix provided in an embodiment of this application; Figure 6 A flowchart illustrating the generation process of an image-text fusion feature sequence provided in this application embodiment; Figure 7 A flowchart of a model prediction process provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a training device for a prediction model provided in an embodiment of this application. Detailed Implementation

[0018] As described earlier, multimodal data fusion has become an important direction for improving the predictive effectiveness of lung cancer risk. Existing technologies typically use image data as the primary modality, incorporating auxiliary information such as clinical text and laboratory indicators through feature splicing or fusion to construct predictive models for risk prediction. However, this method only employs simple feature splicing or shallow fusion. Due to the heterogeneity of different modalities in structure and semantics (e.g., the spatiality of images, the narrative nature of text, and the numerical nature of indicators), simple splicing cannot uncover the complex correlations and complementary information between different modalities, resulting in the model's inability to effectively capture the weak early signals hidden in the multimodal data. This severely limits the sensitivity and specificity of the model's screening, leading to low predictive accuracy and predictive performance that fails to meet the requirements of high-confidence clinical decision-making. Consequently, it cannot provide highly reliable auxiliary evidence for the formulation of subsequent differentiated treatment plans (such as precision medicine, intervention timing selection, and treatment intensity adjustment), and it is difficult to provide effective support for the optimization of individualized treatment and rehabilitation pathways.

[0019] The inventors have proposed a training method, device, and processor for a prediction model. This method first acquires a training sample set for training the initial prediction model. This training sample set includes multiple training samples, each containing a set of multimodal medical data and corresponding ground truth labels. The multimodal medical data includes medical image data, medical text data, and structured test data, providing comprehensive supervision signals for the model and laying the foundation for cross-modal association learning, thereby improving the model's generalization ability. Then, anatomical entities extracted from the medical text data of the multimodal medical data are mapped to corresponding spatial anchor points in the medical image coordinate system. These spatial anchor points can be the center coordinates of an anatomical region in the medical image data, thus transforming abstract textual information into quantifiable spatial coordinates, enabling the model to accurately locate the corresponding textual descriptions in the image. The anatomical region is defined, enhancing the accuracy of feature localization and establishing a bridge between text semantics and image space, thus solving the semantic heterogeneity problem between medical image data and medical text data. Then, using the initial prediction model, a spatial bias matrix is ​​constructed based on the spatial distance between spatial anchor points and image blocks in medical image data to quantify the spatial relative positional relationship between anatomical entities and image blocks. This spatial bias matrix enables the quantitative encoding of the spatial relative position between images and text, allowing the model to explicitly learn the spatial associations of different modal features. This solves the problem that existing technologies struggle to uncover complex associations and complementary information between different modal features, thus providing prior knowledge for subsequent cross-attention fusion. This guides the model to focus more on image regions related to the text description space during fusion, reducing interference from irrelevant features. Then, the initial prediction model performs cross-attention fusion on medical image data and medical text data based on the spatial bias matrix to obtain an image-text fusion feature sequence. This cross-attention mechanism enables bidirectional interaction between image and text features, allowing the model to learn higher-order semantic associations across modalities (such as the intrinsic connection between image lesions and text symptom descriptions). This enhances the model's ability to detect weak early signals. The initial prediction model then generates prediction results based on the image-text fusion feature sequence and structured test data, significantly improving the model's sensitivity and specificity in identifying lung cancer risk. Finally, based on the difference between the prediction results output by the initial prediction model and the true labels, the parameters of the initial prediction model are adjusted to obtain a new prediction model. Supervised optimization allows the model to gradually approximate real clinical scenarios, thereby improving the model's prediction accuracy and robustness. This ensures that the model's prediction performance meets the requirements of high-confidence clinical decision-making, providing highly reliable auxiliary evidence for the formulation of subsequent differentiated treatment plans and effectively supporting the optimization of individualized treatment and rehabilitation pathways.

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0021] Method Implementation Examples

[0022] This application provides an embodiment of a training method for a predictive model. This method can be applied to early screening scenarios for various cancers, such as lung cancer, thyroid cancer, breast cancer, stomach cancer, and liver cancer. The predictive model trained by this method can accurately assess an individual's cancer risk and output early screening results including binary or multi-class classification. Specifically, binary early screening results can provide a "high risk / low risk" judgment for a specific cancer (such as lung cancer) and a corresponding risk score. Multi-class results can simultaneously assess an individual's potential risk of developing multiple cancers, such as lung cancer, liver cancer, and breast cancer, and provide a risk score or risk stratification result for each cancer. To make the purpose, technical solution, and advantages of this application clearer, the training method of the predictive model proposed in this application will be described in detail below, using the specific application scenario of early lung cancer screening (the task is binary classification).

[0023] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the flowchart shows a logical order, in some cases the steps shown or described may be executed in a different order than that shown here.

[0024] See Figure 1 The figure is a flowchart of a training method for a prediction model provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps: Step S101: Obtain the training sample set for training the initial prediction model.

[0025] In one feasible implementation, a training system for a prediction model can serve as the execution entity for the training method of the prediction model in the embodiments of this application. For ease of description, the training system for the prediction model will be referred to simply as the system below.

[0026] In step S101, the training sample set includes multiple training samples. Each training sample includes a set of multimodal medical data and corresponding real labels (e.g., lung cancer risk: high risk / low risk). The multimodal medical data includes medical imaging data (e.g., the patient's chest CT images, and optionally PET images), medical text data (e.g., the patient's chief complaint (i.e., the main reason for the patient's current visit) and present medical history (i.e., a detailed description of the entire process of the patient's current disease from its onset and development to the visit) based on the chief complaint), and structured test data (e.g., blood routine test data, coagulation function test data, tumor marker test data, etc.).

[0027] In this embodiment of the application, the system can obtain a training sample set for training an initial prediction model. Through the multimodal medical data in each training sample of the training sample set, it can provide comprehensive supervision signals for the model, lay the foundation for cross-modal association learning, and thus improve the model's generalization ability.

[0028] The training sample set can be constructed in the following ways: Step S11: Collect the original medical image data, original medical text data, and original structured test data of each of the multiple sample objects, and determine the timestamp, identity identifier, and corresponding real label of the medical image examination of each sample object based on the original medical image data of each sample object.

[0029] In step S11, the original structured test data includes the names, test values, and test threshold ranges of multiple test items.

[0030] In this embodiment, the system can collect the original chest CT images, original medical text data, and original structured test data of each sample object from an imaging system (such as a PACS system, short for Picture Archiving and Communication System), a medical record system (such as a HIS system, short for Hospital Information System), and a testing system (such as a LIS system, short for Laboratory Information System), and determine the timestamp, identity identifier, and corresponding real label of the medical imaging examination of the sample object. Specifically, for sample objects with chest PET image data, the system can also collect their corresponding original chest PET image data.

[0031] Step S12: For each sample object, the corresponding timestamp of the sample object is used as the time anchor point and the preset duration is used as the sliding step size to determine the corresponding target time window. Data within the target time window is selected from the original medical text data and the original structured test data of the sample object to obtain the selected medical text data and the selected structured test data.

[0032] In this embodiment, the system can use the timestamp of the chest CT scan of the sample subject as the time anchor and a preset duration (such as 24 hours, 48 ​​hours, or 36 hours before and after the chest CT scan) as the sliding step to determine the target time window. For example, when the timestamp of the chest CT scan of the sample subject obtained by the system is 10:00:00 on February 2, 2026, and the preset duration is 24 hours before and after the chest CT scan, the system can determine the target time window as from 10:00:00 on February 1, 2026 to 10:00:00 on February 3, 2026. Then, the system can filter out the data within the above-mentioned target time window from the original medical text data and the original structured test data of the sample subject, respectively, to obtain the filtered medical text data and the filtered structured test data.

[0033] Step S13: For each sample object, using the medical image examination event of that sample object as the index event, the original medical image data, the filtered medical text data, and the filtered structured test data of the sample object are associated based on the identity identifier to obtain initial multimodal medical data. The initial multimodal medical data is then cleaned to obtain multimodal medical data.

[0034] In this embodiment of the application, for each sample object, the system can use the sample object's chest CT image examination event as an index event, and associate the sample object's original chest CT image data, filtered medical text data, and filtered structured test data based on the identity identifier to obtain initial multimodal medical data; then the system can perform data cleaning processing on the initial multimodal medical data to obtain multimodal medical data. Specifically, the cleaning processing includes deleting data in the initial multimodal medical data that is missing key modalities (i.e., chest CT images, medical text, and structured test data), has excessive examination time deviation, has unqualified image quality, or has unclear labels.

[0035] Step S14: Based on each multimodal medical data and its corresponding real label, construct a training sample, and construct a training sample set based on all training samples.

[0036] In this embodiment of the application, the system can encapsulate each multimodal medical data and its corresponding real label on a sample object basis, generate corresponding training samples, and construct a training sample set based on all training samples.

[0037] It should be noted that this application first clarifies the source and boundaries of each modality of data by performing strict data source definition; then, using multi-system joint indexing and spatiotemporal alignment technology, it accurately correlates and synchronizes scattered image, text, and test data on a unified time axis, centered on the patient; through standardized data encapsulation, it integrates the cleaned and aligned multimodal data into structured and standardized data units, which can provide high-quality input with spatiotemporal consistency and modality homogeneity for subsequent feature-level coding, effectively breaking down data silos and laying a solid data foundation for building high-precision prediction models.

[0038] It should be noted that steps S11-S14 above are not shown in the figure.

[0039] Step S102 involves mapping the anatomical entities extracted from the medical text data of multimodal medical data to the corresponding spatial anchor points in the medical image coordinate system, and using the initial prediction model to construct a spatial bias matrix based on the spatial distance between the spatial anchor points and the image blocks of the medical image data.

[0040] In step S102, each spatial anchor point is the center coordinate (such as centroid coordinate) of an anatomical region of the medical image data, and the spatial bias matrix is ​​used to quantify the spatial relative positional relationship between the anatomical site entity and the image block.

[0041] In this application, the system can map anatomical entities extracted from medical text data (such as the right upper lobe, left lower lobe, hilar region, subpleural region, etc.) to corresponding spatial anchor points in the medical image coordinate system (such as spatial anchor point P1=(x1,y1,z1) for the right upper lobe and spatial anchor point P2=(x2,y2,z2) for the left lower lobe). These spatial anchor points can be the center coordinates of an anatomical region in the medical image data, realizing the transformation of abstract text information into quantifiable spatial coordinates. This enables the model to accurately locate the anatomical region in the image corresponding to the text description, enhancing the accuracy of feature localization. At the same time, it can establish a bridge between text semantics and image space, solving the problem of semantic heterogeneity between medical image data and medical text data. Then, the system can use the initial prediction model (such as the Transformer model) to construct a spatial bias matrix based on the spatial distance between the spatial anchor points and the image blocks of the chest CT image data. This matrix is ​​used to quantify the spatial relative positional relationship between the anatomical entity and the image block. The spatial bias matrix can be used to quantify and encode the spatial relative position of the image and the text, enabling the model to explicitly learn the spatial associations of different modal features. This solves the problem that existing technologies are unable to mine the complex associations and complementary information between different modal features, thus providing prior knowledge for subsequent cross-attention fusion. This guides the model to pay more attention to the image regions related to the text description space during fusion and reduce the interference of irrelevant features.

[0042] Specifically, see Figure 2 The figure is a flowchart of a spatial bias matrix construction process provided in an embodiment of this application. The process includes the following steps: Step S1021: The format of the medical image data is processed into a preset format to obtain processed medical image data, and the processed medical image data is feature-encoded to obtain a medical image feature sequence.

[0043] In step S1021, the medical image feature sequence consists of the feature vectors of each image block in the medical image data, and the spatial location information of the image block in the medical image coordinate system.

[0044] In this embodiment, the system can perform reading, resampling, fixed-size standardization, window width and level truncation, and normalization on chest CT image data to format it into a preset format. Specifically, the system can first read the corresponding DICOM sequence from the chest CT image data and resample it to a uniform voxel spacing according to the target spacing (e.g., 1.0mm×1.0mm×1.0mm, 0.75mm×0.75mm×1.0mm, 1.0mm×1.0mm×2.0mm) to obtain resampled chest CT image data. Then, the system can adjust the size of the resampled chest CT image data to the target voxel size (e.g., 96×96×96, 128×128×128, 256×256×256) to obtain the size-standardized chest CT image data. The system can fill in the chest CT image data with zero padding for insufficient chest CT image data, and can use center cropping or lung field-based cropping to extract the corresponding area of ​​CT image data for excessively large chest CT image data. Then, the system can truncate the voxel values ​​of the size-standardized chest CT image data according to a preset grayscale window (e.g., [-1000HU, 400HU]), and normalize the truncated voxel values ​​to map them to the [0, 1] interval to obtain the processed chest CT image data.

[0045] Furthermore, the system can divide the processed chest CT image data into multiple image patches (i.e., image blocks), and use a three-dimensional image encoder (such as VISTA3D, which stands for Versatile Imaging Segmentation and Annotation for 3D Computed Tomography) to extract the feature vector corresponding to each patch, and record the physical center position of the patch in the chest CT coordinate system, thereby obtaining a medical image feature sequence composed of the feature vectors of each image patch in the chest CT image data and the spatial position information of the image patch in the chest CT coordinate system.

[0046] It should be noted that the 3D image encoder can also use 3D feature extraction networks such as 3DResNet and 3DSwin Transformer, which can be selected according to actual needs.

[0047] Step S1022: Convert the medical text data into a string in a preset format to obtain a medical text string, and perform feature encoding on the medical text string to obtain an initial medical text feature sequence.

[0048] In this embodiment, the system can concatenate information such as chief complaint, present medical history, gender, and age from medical text data into standardized text according to a preset template. Then, it performs data processing on this standardized text (such as symbol cleaning, terminology normalization, and unit unification) to obtain a medical text string. The preset template can be represented as: "Chief Complaint: {Chief Complaint Content}. Patient Gender: {Gender}; Age: {Age} years. Present Medical History: {Present Medical History Content}."

[0049] Then, the system can perform word segmentation on the above medical text string to obtain multiple tokens. If the number of tokens is greater than a preset value (e.g., 128 tokens), the length of the multiple tokens is truncated to obtain the preset number of tokens. For example, when the medical text string is "Chief complaint: Lung nodule found 3 months ago... Patient gender: Male; Age: 68 years old. Present illness: The patient underwent a chest CT scan 3 months ago and found a ground-glass nodule in the upper lobe of the right lung, approximately 8mm × 7mm in size, without obvious cough, sputum, chest tightness, or shortness of breath..., which was not taken seriously. Now seeking further diagnosis and treatment, the patient came to our hospital and was admitted to the outpatient department with a diagnosis of 'lung nodule'...," after word segmentation, 150 tokens are obtained, including "chief complaint", ":", "found", "lung", "nodule", "3", "months", ".", "patient", and "gender". Then, through length truncation, 128 tokens are obtained. The system can then use a text encoder (e.g., an encoder with an output dimension set to 768 dimensions) to extract features from the above multiple tokens and output the initial medical text feature sequence.

[0050] Step S1023: Identify anatomical entities in the medical text string to obtain a set of anatomical entities, and divide the processed medical image data into multiple anatomical regions to generate corresponding anatomical region masks.

[0051] In step S1023, the anatomical region mask is used to identify the anatomical region to which each pixel in the processed medical image data belongs.

[0052] In this embodiment, the system can utilize a medical named entity recognition model to identify anatomical entities (such as left upper lobe, left lower lobe, right upper lobe, right middle lobe, right lower lobe, hilar region, subpleural region, paramedian region, etc.) in medical text strings. Then, the system can normalize the identified anatomical entities to standardize anatomical names expressed in different ways to a unified standard anatomical label. For example, "right upper lung" and "right upper part" can be uniformly normalized to "right upper lobe," ultimately outputting a standardized set of anatomical entities.

[0053] Optionally, in addition to the medical named entity recognition model, the system can also use dictionary-based matching methods, rule-based template extraction methods, or methods that combine model recognition and rule correction to identify anatomical entities in medical text strings, which can be selected according to actual needs.

[0054] Furthermore, the system can also use a lung lobe segmentation method to divide the processed chest CT image data into multiple anatomical regions such as the left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe, generating corresponding anatomical region masks. Through these anatomical region masks, the anatomical region to which each pixel in the processed chest CT image data belongs can be accurately identified, thus providing an intuitive and quantifiable basis for subsequent matching of text anatomical entities with image anatomy.

[0055] Optionally, in addition to the lung lobe segmentation method, the system can also use methods such as lung region segmentation and standard template mapping to segment the processed chest CT image data to generate corresponding anatomical region masks. Among them, the lung region segmentation method can first segment the left and right lungs in the processed chest CT image data, and then further divide the region based on positional rules. The standard template mapping method can obtain the anatomical region in the processed chest CT image data by mapping using a standard chest anatomy template.

[0056] Step S1024: Based on the anatomical region mask and the set of anatomical parts, determine multiple spatial anchor points corresponding to the processed medical image data, and bind each spatial anchor point to the corresponding token of the initial medical text feature sequence to generate the medical text feature sequence.

[0057] In this embodiment, the system can construct a mapping relationship between anatomical entities and anatomical regions based on anatomical region masks and sets of anatomical entities. Based on this mapping relationship, the system accurately matches anatomical entities to the corresponding anatomical regions in the processed medical image data (e.g., "right upper lobe" matches to the right upper lobe region), and determines the spatial anchor points for each anatomical region (e.g., spatial anchor point P1=(x1,y1,z1) for the right upper lobe). Then, the system binds each spatial anchor point to the corresponding token of the initial medical text feature sequence (e.g., token2 "right upper lobe" is bound to P1=(x1,y1,z1)), generating a medical text feature sequence. This strengthens the deep association between medical text and image data, transforming abstract textual anatomical information into quantifiable and learnable features. This provides high-quality, structured data support for subsequent model training, effectively improving the model's accuracy in recognizing anatomical sites and its ability to correlate textual and image information across modalities. It also helps the model quickly learn the correspondence between the textual representation of anatomical sites and the spatial features of the images.

[0058] Specifically, see Figure 3The figure is a flowchart of a medical text feature sequence generation process provided in an embodiment of this application. The process includes the following steps: Step S10241: Based on the set of anatomical site entities and the anatomical region mask, construct the mapping relationship between anatomical site entities and anatomical regions.

[0059] In this embodiment of the application, the system can match each anatomical entity in the anatomical entity set to the corresponding anatomical region in the processed chest CT image data based on the preset anatomical location name correspondence relationship (for example, matching the right upper lobe to the right upper lobe region mask, matching the subpleural region to the region near the lung boundary, and matching the hilar region to the hilar neighborhood region) to construct the mapping relationship between anatomical entity and anatomical region.

[0060] Step S10242: For each anatomical body entity, determine the corresponding anatomical region based on the mapping relationship, and calculate the center coordinates of the anatomical region.

[0061] In this embodiment of the application, for each anatomical site entity, the system can determine the anatomical region corresponding to the anatomical site entity based on the above mapping relationship, and calculate the representative spatial position of the anatomical region in the processed chest CT image data (such as the centroid coordinates of the anatomical region, the center coordinates of the bounding box of the anatomical region, the coordinates of the selected preset reference point, etc.), and use it as the center coordinates of the anatomical region.

[0062] Step S10243: Use the center coordinates of each anatomical region as a spatial anchor point, and construct a set of spatial anchor points based on all spatial anchor points.

[0063] In step S10243, each spatial anchor point in the set of spatial anchor points corresponds to an anatomical site entity.

[0064] Step S10244: For each anatomical site entity, determine the position information of the token corresponding to the anatomical site entity in the initial medical text feature sequence, and bind the token to the spatial anchor point corresponding to the anatomical site entity based on the position information of the token to obtain the medical text feature sequence.

[0065] In this embodiment of the application, for each anatomical part entity, the system can locate the anatomical part entity to its token position in the initial medical text feature sequence, and bind its corresponding spatial anchor point to the token or token group corresponding to the anatomical part entity; optionally, for tokens that do not contain explicit anatomical part information, they are marked as tokens without anchor points.

[0066] Taking the medical text string "Present medical history: small nodules seen in the upper lobe of the right lung and patchy shadows seen in the lower lobe of the left lung" as an example, the multiple tokens obtained through step S1022 include: token1: present medical history, token2: upper lobe of the right lung, token3: seen, token4: small nodules, token5: lower lobe of the left lung, token6: seen, token7: patchy shadows. Then, the system can identify and standardize two anatomical entities through the above step S1023: Anatomical entity 1: right upper lobe of lung, anatomical entity 2: left lower lobe of lung, and obtain the spatial anchor points of these two anatomical entities in the chest CT coordinate system through steps S10241-S10243: spatial anchor point P1=(x1,y1,z1) for right upper lobe of lung, and spatial anchor point P2=(x2,y2,z2) for left lower lobe of lung; finally, the system can assign the spatial anchor point P1 corresponding to right upper lobe of lung to token2 and the spatial anchor point P2 corresponding to left lower lobe of lung to token5 through step S10244. For token3 "see" and token6 "visible", since they do not represent anatomical locations, they are marked as tokens without anchor points.

[0067] Optionally, since token4 "small nodule" is modified by token2 "right upper lobe", the spatial anchor point P1 can be inherited and assigned to token4; similarly, since token7 "patchy shadow" is modified by token5 "left lower lobe", the spatial anchor point P2 can be inherited and assigned to token7. Thus, the mapping relationship can be obtained as follows: token2 "right upper lobe" maps to P1, token4 "small nodule" maps to P1, token5 "left lower lobe" maps to P2, and token7 "patchy shadow" maps to P2.

[0068] It should be noted that by accurately locating the token positions corresponding to the anatomical entities and binding them to spatial anchors, the model can clearly learn the one-to-one correspondence between the text representation of each anatomical entity (such as token / token group) and the spatial position of the image (such as spatial anchor), which effectively improves the model's text localization accuracy of anatomical entities and its cross-modal association learning ability, and avoids model training bias caused by fuzzy binding of tokens and anchors.

[0069] Furthermore, the system can also perform precise preprocessing and feature enhancement on structured test data through a threshold-aware coding mechanism based on test reference intervals, resulting in test feature sequences. The system uses the clinical reference intervals for each test item as the core benchmark to perform nonlinear mapping and quantitative characterization of the abnormality level of structured test data, strengthening the expression of weak pathological signals near the normal threshold and in the mildly abnormal range. This effectively solves the problem of weak pathological signals being easily masked and difficult to identify in early lung cancer screening, ultimately generating test feature sequences with high discriminative power. This significantly improves the utilization value of structured test data in early lung cancer screening, providing high-quality, highly correlated feature inputs for subsequent construction of high-precision lung cancer prediction models, enabling the models to accurately capture potential pathological clues related to early lung cancer. For details, see [link to relevant documentation]. Figure 4 The figure is a flowchart of a process for generating a test feature sequence according to an embodiment of this application. The process includes the following steps: Step S41: For each inspection item, calculate the abnormal deviation degree corresponding to the inspection item based on the inspection value and inspection threshold range, and use the gating network to calculate the gating weight corresponding to the inspection item based on the abnormal deviation degree corresponding to the inspection item.

[0070] In this embodiment of the application, the system can read the name, test value, reference lower limit and reference upper limit (i.e. test threshold range) of each test item from the structured test data (such as white blood cell count, neutrophil count or proportion, lymphocyte count or proportion, hemoglobin, platelet count, red blood cell distribution width, etc.); then the system can use formula (1) to calculate the abnormal deviation degree corresponding to the test item based on the test value and test threshold range of each test item.

[0071] (1)

[0072] in, a i To examine the degree of abnormal deviation corresponding to the project; x i This is the test value; L i As a reference lower limit, U i This is a reference upper limit; ε To avoid extremely small positive numbers with a denominator of zero.

[0073] Furthermore, the system can input the abnormal deviation degree of each inspection item into the gating network (such as a gating network using a two-layer multilayer perceptron), and use the gating network to calculate the gating weight corresponding to the inspection item based on the abnormal deviation degree of the inspection item.

[0074] Optionally, in addition to calculating the gating weights through a gating network, the system can also calculate the gating weights corresponding to the test items through gating functions (such as the Sigmoid function, ReLU function, and GELU function).

[0075] Step S42: Text embedding is performed on the name of each test item to convert the name into a dense numerical vector of a preset length, thereby obtaining the item name embedding vector corresponding to each test item. Feature engineering is then performed on the test value of each test item to obtain the numerical feature vector of the test value corresponding to each test item.

[0076] In this embodiment, the system can perform text embedding processing on the names of each test item. By using a medical text-specific embedding model (such as Word2Vec or BERT models pre-trained based on a medical dictionary), each test item name is transformed into a dense numerical vector of a preset length (such as 128-dimensional or 256-dimensional), thereby obtaining the item name embedding vector corresponding to each test item, realizing the structured and quantifiable representation of the test item names. Then, the system can perform targeted feature engineering on the test values ​​of each test item. Combining the clinical characteristics of the test items (such as numerical distribution, unit differences, and reference range), through operations such as normalization, standardization, outlier correction, and nonlinear transformation, the system can eliminate the dimensional differences between different test items, strengthen the correlation between test values ​​and early lung cancer pathological characteristics, and obtain the numerical feature vector of the test value corresponding to each test item.

[0077] For example, for structured test data related to early lung cancer screening, if the test item is carcinoembryonic antigen (CEA), the system can convert CEA into a 128-dimensional dense numerical vector as the item name embedding vector through text embedding; then it can standardize its test value (5.2 ng / mL) and the reference range (0-5 ng / mL) to eliminate the differences in testing standards between different laboratories, and then enhance its mild abnormality characteristics through nonlinear transformation to obtain the corresponding test value numerical feature vector.

[0078] Step S43: For each inspection item, combine the item name embedding vector and the inspection value feature vector to generate a single inspection feature sequence, and weight the single inspection feature sequence based on the gating weight corresponding to the inspection item to obtain the threshold-aware inspection feature.

[0079] In this embodiment of the application, for each inspection item, the system can combine the item name embedding vector of the inspection item with the numerical feature vector of the inspection value to generate a single inspection feature sequence, and then use the following formula (2) to weight the single inspection feature sequence based on the gating weight to obtain the threshold-aware inspection feature.

[0080] (2)

[0081] in, z i This represents the threshold-aware test feature; g i Indicates the gating weight; e i The project name embedding vector is represented by vi; the test value numerical feature vector is represented by vi. ( ) represents a linear mapping or a multilayer perceptron.

[0082] It should be noted that by embedding the item name used to accurately identify the category of test items into a vector and combining it with the numerical feature vector of the test value that quantitatively represents and reflects the physiological and pathological state of the examinee, information fusion between the test item name and the test result is achieved, making the feature representation more comprehensive. By assigning gating weights according to the clinical value of the test items (with values ​​from 0 to 1), high-diagnostic-value items can be given high weights and low-correlation items can be given low weights, thereby strengthening the core features and weakening irrelevant interference, making the generated threshold-aware test features more targeted. This can provide standardized and high-quality feature inputs for subsequent model training, thereby improving model training efficiency and recognition accuracy.

[0083] Step S44: All threshold-aware inspection features corresponding to the inspection items are spliced ​​together according to the preset splicing order to generate an inspection feature sequence.

[0084] In this embodiment, the system can uniformly splice the threshold-aware test features of all test items according to a preset splicing order, ultimately generating a complete test feature sequence. The preset splicing order can be set in advance based on the clinical priority, feature importance, or data processing specifications of the test items, ensuring an orderly and standardized splicing process and avoiding feature distortion due to a chaotic splicing order.

[0085] It should be noted that by using a threshold-aware coding mechanism based on the test reference interval, the structured test data can be precisely preprocessed and feature-enhanced. This can effectively strengthen the expression of weak pathological signals near the normal threshold and in the mildly abnormal range, thereby solving the problem that weak pathological signals are easily masked and difficult to identify in early lung cancer screening. This significantly improves the utilization value of structured test data in early lung cancer screening and provides high-quality, highly correlated feature inputs for the subsequent construction of a high-precision lung cancer prediction model, enabling the model to accurately capture potential pathological clues related to early lung cancer.

[0086] Step S1025: Using the initial prediction model, construct a spatial bias matrix based on the spatial distance between the spatial anchor points in the medical text feature sequence and each image block.

[0087] In this embodiment, the system utilizes an initial prediction model to calculate the distance between the spatial anchor point corresponding to each bound spatial anchor point and the physical center of each image patch in the chest CT image. All calculated Euclidean distances are then organized according to their correspondence to form a spatial distance matrix with clear dimensions. In this matrix, rows correspond to the bound spatial anchor points of the tokens, columns correspond to each image patch, and each element represents the distance between the corresponding token anchor point and the physical center of the patch, thus intuitively quantifying the spatial proximity between them. Then, the system further constructs a spatial bias matrix based on the spatial distance matrix, transforming the spatial distance into a learnable spatial bias feature for the model, thereby achieving a quantitative representation of the spatial association between text tokens and image patches.

[0088] Specifically, see Figure 5 The figure is a flowchart of another spatial bias matrix construction process provided in an embodiment of this application, which includes the following steps: Step S10251: Using the initial prediction model, for each token bound to a spatial anchor point in the medical text feature sequence, calculate the spatial distance between the spatial anchor point corresponding to the token and each image block, and construct a spatial distance matrix based on all spatial distances.

[0089] In this embodiment, for each text token with bound spatial anchors, the system can use an initial prediction model to calculate the Euclidean distance between the spatial anchor corresponding to the token and the physical center of each image patch in the medical image. All calculated Euclidean distances are then organized according to their correspondence to form a spatial distance matrix with clear dimensions. For example, taking the medical text string "Present medical history: small nodules seen in the upper lobe of the right lung, patchy shadows seen in the lower lobe of the left lung," the multiple tokens with bound spatial anchors obtained through step S1022 include: (1) token2: upper lobe of the right lung; (2) token4: sub-nodule; (3) token5: lower lobe of left lung; (4) token7: patchy shadow.

[0090] The mapping relationship between tokens and spatial anchors includes: (1) The spatial anchor point P1 corresponding to token2 “right upper lobe” is (x1, y1, z1); (2) token4 “sub-node” inherits the spatial anchor point P1=(x1,y1,z1) corresponding to token2; (3) The spatial anchor point P2 corresponding to token5 “left lower lobe” is (x2, y2, z2); (4) token7 “Spot Shadow” inherits the spatial anchor point P2=(x2,y2,z2) corresponding to token5.

[0091] Then, the physical center coordinates of each image patch are as follows: (1) The center coordinates of Patch1 are Q1=(u1,v1,w1); (2) The center coordinates of Patch2 are Q2=(u2,v2,w2); (3) The center coordinates of Patch3 are Q3=(u3,v3,w3); (4) The center coordinates of Patch4 are Q4=(u4,v4,w4).

[0092] Then, for each text token with bound spatial anchors, the system can use the initial prediction model to calculate the Euclidean distance between the spatial anchor corresponding to the token and the physical center of each image patch in the medical image: (1) The spatial distances between token2 and the four image patches are as follows:

[0093] (2) The spatial distances between token4 and the four image patches are as follows:

[0094] (3) The spatial distances between token5 and the four image patches are as follows:

[0095] (4) The spatial distances between token7 and the four image patches are as follows:

[0096] Then, the system can form the following spatial distance matrix based on the above spatial distances:

[0097] In this matrix, rows correspond to tokens that are bound to spatial anchors, and columns correspond to each image patch. Each element in the matrix is ​​the distance between the corresponding token anchor and the physical center of the patch. If the spatial anchor corresponding to a certain text token is closer to a certain image patch, the distance value at that position is smaller, indicating that the text token and the image patch are closer in anatomical space.

[0098] Optionally, for tokens that are not bound to spatial anchors, the system may process them in any of the following ways: not participate in the spatial distance matrix calculation; assign a default value to their corresponding distance; or assign a zero value to the corresponding position in the subsequent spatial bias matrix.

[0099] Step S10252: Using the initial prediction model, perform distance decay transformation on each element in the spatial distance matrix to obtain the corresponding bias value.

[0100] In step S10252, the bias value is used to quantify the spatial proximity between the token and the image patch, and the bias value is negatively correlated with the spatial distance.

[0101] In this embodiment of the application, the system can use the initial prediction model to perform distance attenuation transformation on the elements in the spatial distance matrix using the following formula (3) to obtain the bias value used to quantify the spatial proximity between the token and the image patch.

[0102] (3)

[0103] in, B m,n Indicates the bias value; d m,n σ represents the element in the spatial distance matrix; σ is the distance attenuation coefficient, which is used to control the range of influence of spatial distance on the offset value. It can be set between 20mm and 60mm, such as 32mm.

[0104] Step S10253: Using the initial prediction model, construct a spatial bias matrix based on the bias values ​​corresponding to all elements.

[0105] In this embodiment of the application, the system can utilize the initial prediction model to construct the following spatial bias matrix based on the bias values ​​corresponding to all elements:

[0106] Wherein, if the spatial anchor point corresponding to any text token is close to the center of any image patch, the corresponding distance element d m,n The smaller the corresponding bias matrix element B m,n The value of d is larger; conversely, if the distance is greater, d m,n If B is larger m,n The value is relatively small, thus the spatial bias matrix can clearly reflect the spatial proximity between the text token and the image patch. For example, when the spatial anchor point P1 corresponding to token2 "right upper lobe of the lung" is at its minimum distance from the center Q1 of patch1, d 2,1 Less than d 2,2 d 2,3 d 2,4Accordingly B 2,1 Greater than B 2,2 B 2,3 B 2,4 This allows token2 "right upper lobe of the lung" to assign a higher spatial attention weight to patch1 in subsequent attention calculations, which helps the model accurately capture the spatial location of the anatomical part corresponding to the text token in the image, weakens the interference of irrelevant image patches, and improves the model's learning accuracy of text-image cross-modal association.

[0107] Step S103: Based on the initial prediction model, cross-attention fusion is performed on medical image data and medical text data according to the spatial bias matrix to obtain the image-text fusion feature sequence, and prediction results are generated based on the image-text fusion feature sequence and structured test data.

[0108] In this embodiment of the application, the system can use the initial prediction model to perform cross-attention fusion on the medical image feature sequence obtained by step S1021 and the initial medical text feature sequence obtained by step S1022 according to the spatial bias matrix using the following formula (4) to obtain the image-text fusion feature sequence. The cross-attention mechanism can realize the bidirectional interaction between image and text features, enabling the model to learn the high-order semantic association across modalities (such as the intrinsic connection between image lesions and text symptom descriptions), enhancing the model's ability to detect early weak signals. Thus, the initial prediction model can generate high-precision prediction results (such as lung cancer risk: low risk) based on the image-text fusion feature sequence and structured test data, significantly improving the model's sensitivity and specificity in identifying lung cancer risk.

[0109] (4)

[0110] Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; B represents the spatial bias matrix; and λ represents the spatial bias weight coefficient, which can be set between 0.1 and 2.0, such as 0.5.

[0111] Specifically, see Figure 6 The figure is a flowchart of the image-text fusion feature sequence generation process provided in an embodiment of this application. The process includes the following steps: Step S1031: Using the initial prediction model, construct a query matrix based on the initial medical text feature sequence, and construct a key matrix and a value matrix based on the medical image feature sequence.

[0112] Taking the medical text string "Present illness: small nodules seen in the upper lobe of the right lung, patchy shadows seen in the lower lobe of the left lung" as an example, the initial medical text feature sequence T can be represented as:

[0113] Wherein, Token2 "Right upper lobe" corresponds to text feature vector T2; Token4 "small nodule" corresponds to text feature vector T4; Token5 "Left lower lobe" corresponds to text feature vector T5; and Token7 "patchy shadow" corresponds to text feature vector T7. The medical image feature sequence can be represented as:

[0114] Wherein, Patch1 corresponds to image feature vector I1; Patch2 corresponds to image feature vector I2; Patch3 corresponds to image feature vector I3; and Patch4 corresponds to image feature vector I4. The query matrix Q constructed by the system from the initial medical text feature sequence can be represented as: Q = TW Q Among them, W Q Represented as a learnable parameter matrix. The key matrix K and value matrix V constructed from the system medical image feature sequences can be expressed as: K = IW K V=IW V W K IW V This is the learnable parameter matrix.

[0115] Step S1032: Using the initial prediction model, cross-attention calculation is performed based on the query matrix and key matrix to obtain the initial attention score matrix.

[0116] In step S1032, each element in the initial attention score matrix is ​​used to reflect the semantic relevance between the token and the image patch.

[0117] Taking the above medical text string as “Present medical history: small nodules in the upper lobe of the right lung and patchy shadows in the lower lobe of the left lung” as an example, the system can use the initial prediction model to perform cross-attention calculation based on the query matrix Q and the key matrix K through formula (5) to obtain the initial attention score matrix A.

[0118] (5)

[0119] Where, d k The dimension of the key vector.

[0120] Step S1033: Using the initial prediction model, determine the target attention score matrix based on the initial attention score matrix and the spatial bias matrix.

[0121] In this embodiment, the system can utilize the initial prediction model to determine the target attention score matrix based on the initial attention score matrix A and the spatial bias matrix B using formula (6). .

[0122] (6)

[0123] Step S1034: Using the initial prediction model, the value matrix is ​​weighted and summed based on the target attention score matrix to obtain the image-text fusion feature sequence.

[0124] In this embodiment of the application, the system can utilize the target attention score matrix. After performing Softmax normalization, the final attention weight matrix is ​​obtained:

[0125] Then, the system can utilize this final attention weight matrix. α The image-text fusion feature sequence F is obtained by weighted summation of the value matrix V: F=αV In F, each line represents the image fusion feature of a text token after the introduction of spatial guidance.

[0126] It should be noted that by transforming the anatomical location information in the text into explicit spatial guidance for the local region of the image through the spatial bias matrix, the text token carrying a clear anatomical orientation can accurately focus on the corresponding image patch region in the cross-attention calculation, enhance the spatial alignment ability between text semantics and local image features, effectively reduce the difficulty of cross-modal feature learning of the model, accelerate the model training convergence speed, and improve the model feature association learning accuracy and generalization performance.

[0127] Further, see Figure 7 The figure is a flowchart of a model prediction process provided in an embodiment of this application, which includes the following steps: Step S1035: Based on the test values ​​and test threshold ranges of each test item in the structured test data, determine the gate weight corresponding to each test item, and perform weighted processing on the name and test value of each test item according to the gate weight corresponding to each test item to generate a test feature sequence.

[0128] In step S1035, the gating weight is used to quantify the degree to which the test value deviates from the normal range.

[0129] In this embodiment of the application, the system can determine the gate weight corresponding to each inspection item based on the inspection value and inspection threshold range of each inspection item in the structured inspection data through the above step S41; then the system can perform weighted processing on the name and inspection value of the inspection item based on the gate weight corresponding to each inspection item through the above steps S42-S44 to generate an inspection feature sequence.

[0130] Step S1036: Using the multimodal aggregator of the initial prediction model, extract the global feature representation in the test feature sequence to obtain the first global feature vector, and extract the global feature representation in the image-text fusion feature sequence to obtain the second global feature vector.

[0131] In this embodiment, the system can use the multimodal aggregator of the initial prediction model to extract a summary representation (i.e., global feature representation) of a preset length from the test feature sequence and the image-text fusion feature sequence, respectively, to obtain the first global feature vector and the second global feature vector.

[0132] Optionally, the multimodal aggregator uses a fixed number of learnable query tokens, such as 64.

[0133] Step S1037: The first global feature vector and the second global feature vector are concatenated along the token dimension using a multimodal aggregator to obtain a multimodal fused feature sequence.

[0134] Step S1038: Using the joint encoding network of the initial prediction model, perform deep feature interaction processing on the multimodal fusion feature sequence to obtain the multimodal joint feature representation.

[0135] In this embodiment, the system can use a 4-layer Transformer coding network to perform deep feature interaction processing on the multimodal fusion feature sequence to obtain a multimodal joint feature representation.

[0136] Step S1039: Using the classification prediction head of the initial prediction model, generate prediction results based on multimodal joint feature representation.

[0137] In this embodiment, the system can utilize the classification prediction head of the initial prediction model to output early lung cancer screening results (such as lung cancer risk: low risk) based on multimodal joint feature representation.

[0138] Step S104: Based on the difference between the prediction results output by the initial prediction model and the true labels, the parameters of the initial prediction model are adjusted to obtain the prediction model.

[0139] In this embodiment, the system can adjust the parameters of the initial prediction model based on the difference between the prediction results output by the initial prediction model and the true label to obtain a prediction model. Through supervised optimization, the model can gradually approach the real clinical scenario, thereby improving the prediction accuracy and robustness of the model. This enables the model's prediction performance to meet the requirements of high-confidence clinical decision-making, and thus provides highly reliable auxiliary evidence for the formulation of subsequent differentiated treatment plans, providing effective support for the optimization of individualized treatment and rehabilitation pathways.

[0140] In one feasible implementation, PET images are introduced during the training phase to build a teacher network-student network knowledge distillation training architecture. This involves constructing a teacher network with complete modal inputs and a student network lacking auxiliary modal inputs. The soft labels and / or intermediate fusion features of the teacher network are used to distill the student network, enabling it to output accurate screening results during the inference phase without the need for auxiliary modalities (i.e., PET images). Specifically, the student network can use the basic joint modeling backbone network constructed in steps S101-S104, taking chest CT image data, medical text data, and structured test data as inputs. It sequentially completes chest CT image preprocessing, text preprocessing, structured test data preprocessing, spatial anchoring perception fusion, and multimodal joint modeling, ultimately outputting early lung cancer screening results. Its network structure is consistent with the inference deployment structure. The teacher network can add a PET modality encoding branch and a corresponding fusion branch to the student network, forming a complete modal network. This complete modal network is used to learn discriminative knowledge across all modalities during training, and this knowledge is transferred to the student network through knowledge distillation.

[0141] Specifically, the system can construct input samples for both the teacher network and the student network separately. Both types of samples are generated based on patient-level multimodal raw samples. The teacher network input samples retain chest CT images, clinical text, and structured test data, and add PET images as an auxiliary modality, thus possessing complete modality information. The student network input samples only contain chest CT images, clinical text, and structured test data, without introducing PET images. The two form a distillation training pairing relationship between the complete modality and the missing auxiliary modality. The two types of input samples are used for the forward computation of the corresponding networks.

[0142] The system then performs dual-network forward computation. It receives its dedicated input samples through the teacher network and preprocesses and encodes the chest CT images, clinical text, and structured test data through steps S101-S102 and S41-S44. Simultaneously, it preprocesses and extracts features from the PET images. Modeling is completed by combining all modalities, outputting the teacher's prediction results and teacher fusion features. The system also receives its three-modal input samples through the student network and performs forward computation based on the basic joint modeling network, without accessing the PET modality branch. It outputs the student's prediction results and student fusion features. All four types of outputs are used to construct the subsequent distillation loss.

[0143] Furthermore, the system can combine teacher prediction results, teacher fusion features, student prediction results, student fusion features, and ground truth labels to construct a joint loss function consisting of task loss, soft label distillation loss, and feature alignment loss. The student network parameters are then iteratively updated based on this joint loss function. This training method allows the full-modal discriminative knowledge of the teacher network to be transferred to the student network, enabling the student network to maintain excellent lung cancer early screening prediction performance even without PET image input. After training, the optimized student network model is output. In actual deployment, only this student network is used, relying solely on chest CT images, clinical text, and structured test data to complete lung cancer early screening inference, without requiring PET images. The joint loss function is as follows:

[0144] Among them, L task L represents the task loss based on the real labels; distil This represents the soft-label distillation loss between the teacher network and the student network; L feat This represents the feature alignment loss between the teacher network and the student network.

[0145] The training method of the prediction model provided in this application transforms abstract textual information into quantifiable spatial coordinates, enabling the model to accurately locate the anatomical region in the image corresponding to the textual description, thus enhancing the accuracy of feature localization. Simultaneously, it establishes a bridge between textual semantics and image space, resolving the semantic heterogeneity issue between medical image data and medical text data. By using a spatial bias matrix to quantify and encode the spatial relative positions of the image and text, the model can explicitly learn the spatial relationships between different modal features. This addresses the difficulty of existing technologies in uncovering complex relationships and complementary information between different modal features, thereby providing prior knowledge for subsequent cross-attention fusion. This guides the model to focus more on image regions spatially related to the textual description during fusion, reducing... It minimizes interference from irrelevant features; through a cross-attention mechanism, it achieves bidirectional interaction between image and text features, enabling the model to learn higher-order semantic associations across modalities (such as the intrinsic connection between image lesions and text symptom descriptions), enhancing the model's ability to detect weak early signals. This allows the initial prediction model to generate prediction results based on image-text fusion feature sequences and structured test data, significantly improving the model's sensitivity and specificity in identifying lung cancer risk. Supervised optimization gradually brings the model closer to real clinical scenarios, improving its prediction accuracy and robustness. This ensures the model's predictive performance meets the requirements of high-confidence clinical decision-making, providing highly reliable auxiliary evidence for the development of subsequent differentiated treatment plans and effectively supporting the optimization of individualized treatment and rehabilitation pathways.

[0146] Device Examples

[0147] This application provides a training apparatus for a prediction model, wherein, Figure 8 This is a schematic diagram of the structure of a training device for a prediction model provided in an embodiment of this application, as shown below. Figure 8 As shown, the device includes: an acquisition module 81, a construction module 82, a prediction module 83, and a parameter adjustment module 84. From Figure 8 You can see the connections between several modules.

[0148] The acquisition module 81 is used to acquire a training sample set for training the initial prediction model. The training sample set includes multiple training samples, each of which includes a set of multimodal medical data and corresponding real labels. The multimodal medical data includes medical image data, medical text data, and structured test data. The construction module 82 is used to map anatomical entities extracted from the medical text data of the multimodal medical data to corresponding spatial anchor points in the medical image coordinate system, and to construct a spatial bias matrix using the initial prediction model based on the spatial distance between the spatial anchor points and the image blocks of the medical image data; each spatial anchor point is the center coordinate of an anatomical region of the medical image data, and the spatial bias matrix is ​​used to quantify the spatial relative positional relationship between the anatomical entities and the image blocks; The prediction module 83 is used to perform cross-attention fusion on the medical image data and the medical text data based on the spatial bias matrix using the initial prediction model to obtain an image-text fusion feature sequence, and to generate a prediction result based on the image-text fusion feature sequence and the structured test data. The parameter adjustment module 84 is used to adjust the parameters of the initial prediction model based on the difference between the prediction result output by the initial prediction model and the true label, so as to obtain the prediction model.

[0149] Optionally, the training apparatus for the prediction model also includes: The acquisition module is used to acquire the original medical image data, original medical text data, and original structured test data of multiple sample objects before acquiring the training sample set for training the initial prediction model. Based on the original medical image data of each sample object, the module determines the timestamp, identity identifier, and corresponding real label of the medical image examination of the sample object. The original structured test data includes the name, test value, and test threshold range of each of the multiple test items. The filtering module is used to determine the corresponding target time window for each sample object, using the timestamp corresponding to the sample object as the time anchor point and the preset duration as the sliding step size, and to filter out the data within the target time window from the original medical text data and the original structured test data of the sample object, respectively, to obtain the filtered medical text data and the filtered structured test data. The processing module is used to, for each of the sample objects, use the medical image examination event of the sample object as the index event, associate the original medical image data, the filtered medical text data and the filtered structured test data of the sample object based on the identity identifier to obtain initial multimodal medical data, and perform data cleaning processing on the initial multimodal medical data to obtain the multimodal medical data. The sample set construction module is used to construct a training sample based on each of the multimodal medical data and the corresponding real label, and to construct the training sample set based on all the training samples.

[0150] Optionally, the building module includes: The first feature encoding unit is used to process the format of the medical image data into a preset format to obtain processed medical image data, and to perform feature encoding on the processed medical image data to obtain a medical image feature sequence; the medical image feature sequence consists of the feature vector of each image block in the medical image data, and the spatial position information of the image block in the medical image coordinate system. The second feature encoding unit is used to convert the medical text data into a string of a preset format to obtain a medical text string, and to perform feature encoding on the medical text string to obtain an initial medical text feature sequence. The recognition unit is used to recognize anatomical entities in the medical text string, obtain a set of anatomical entities, and divide the processed medical image data into multiple anatomical regions to generate corresponding anatomical region masks; the anatomical region masks are used to identify the anatomical region to which each pixel in the processed medical image data belongs. The first generation unit is used to determine multiple spatial anchor points corresponding to the processed medical image data based on the anatomical region mask and the set of anatomical parts, and bind each spatial anchor point to the corresponding token of the initial medical text feature sequence to generate a medical text feature sequence. The first construction unit is used to construct the spatial bias matrix based on the spatial distance between the spatial anchor points in the medical text feature sequence and each of the image blocks using the initial prediction model.

[0151] Optionally, the first generating unit includes: The second construction unit is used to construct the mapping relationship between the anatomical parts entities and the anatomical regions based on the set of anatomical parts entities and the anatomical region mask; The first calculation unit is used to determine the anatomical region corresponding to each anatomical site entity based on the mapping relationship, and to calculate the center coordinates of the anatomical region. The third construction unit is used to take the center coordinates of each anatomical region as a spatial anchor point and construct a set of spatial anchor points based on all the spatial anchor points; each spatial anchor point in the set of spatial anchor points corresponds to an anatomical part entity; The binding unit is used to determine the position information of the token corresponding to the anatomical part entity in the initial medical text feature sequence for each anatomical part entity, and bind the token to the spatial anchor point corresponding to the anatomical part entity based on the position information of the token to obtain the medical text feature sequence.

[0152] Optionally, the first building unit includes: The second calculation unit is used to use the initial prediction model to calculate the spatial distance between the spatial anchor point corresponding to each token bound to the spatial anchor point in the medical text feature sequence and each of the image blocks, and to construct a spatial distance matrix based on all the spatial distances. The transformation unit is used to perform a distance attenuation transformation on each element in the spatial distance matrix using the initial prediction model to obtain a corresponding bias value; the bias value is used to quantify the spatial proximity between the token and the image patch, and the bias value is negatively correlated with the spatial distance; The fourth construction unit is used to construct the spatial bias matrix based on the bias values ​​corresponding to all the elements using the initial prediction model.

[0153] Optionally, the prediction module includes: The fifth construction unit is used to construct a query matrix based on the initial prediction model and the initial medical text feature sequence, and to construct a key matrix and a value matrix based on the medical image feature sequence, respectively. The third calculation unit is used to perform cross-attention calculation based on the query matrix and the key matrix using the initial prediction model to obtain an initial attention score matrix; each element in the initial attention score matrix is ​​used to reflect the semantic correlation between the token and the image patch; The first determining unit is used to determine the target attention score matrix based on the initial attention score matrix and the spatial bias matrix using the initial prediction model. The weighted summation unit is used to use the initial prediction model to perform a weighted summation of the value matrix based on the target attention score matrix to obtain the image-text fusion feature sequence.

[0154] Optionally, the prediction module also includes: The second determining unit is used to determine the gate weight corresponding to each inspection item based on the inspection value and inspection threshold range of each inspection item in the structured inspection data, and to perform weighted processing on the name and inspection value of the inspection item based on the gate weight corresponding to each inspection item to generate an inspection feature sequence; the gate weight is used to quantify the degree to which the inspection value deviates from the normal range; The feature extraction unit is used to extract global feature representations from the test feature sequence using the multimodal aggregator of the initial prediction model to obtain a first global feature vector, and to extract global feature representations from the image-text fusion feature sequence to obtain a second global feature vector; The first concatenation unit is used to concatenate the first global feature vector and the second global feature vector in the token dimension through the multimodal aggregator to obtain a multimodal fused feature sequence. An interactive processing unit is used to perform deep feature interaction processing on the multimodal fusion feature sequence using the joint coding network of the initial prediction model to obtain a multimodal joint feature representation; The second generation unit is used to generate the prediction result based on the multimodal joint feature representation using the classification prediction head of the initial prediction model.

[0155] Optionally, the second determining unit includes: The fourth calculation unit is used to calculate the abnormal deviation degree corresponding to each of the test items based on the test value of the test item and the test threshold range, and to calculate the gate weight corresponding to the test item based on the abnormal deviation degree of the test item using a gating network. The conversion unit is used to embed the text of the name of each of the test items to convert the name into a dense numerical vector of a preset length, thereby obtaining the item name embedding vector corresponding to each of the test items, and to perform feature engineering on the test value of each of the test items to obtain the test value numerical feature vector corresponding to each of the test items. The third generation unit is used to combine the item name embedding vector corresponding to the inspection item and the inspection value numerical feature vector for each inspection item to generate a single inspection feature sequence, and to weight the single inspection feature sequence based on the gating weight corresponding to the inspection item to obtain threshold-aware inspection features. The second splicing unit is used to splice the threshold-aware test features corresponding to all the test items in a preset splicing order to generate the test feature sequence.

[0156] Processor Implementation

[0157] This application provides a processor for running a program, wherein, during program execution, some or all of the steps in the training method for the prediction model described in the foregoing method embodiments are performed.

[0158] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0159] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a prediction model, characterized in that, include: Obtain the training sample set used to train the initial prediction model; The training sample set includes multiple training samples, each of which includes a set of multimodal medical data and corresponding real labels. The multimodal medical data includes medical image data, medical text data, and structured test data. Anatomical entities extracted from the medical text data of the multimodal medical data are mapped to corresponding spatial anchor points in the medical image coordinate system. The initial prediction model is then used to construct a spatial bias matrix based on the spatial distance between the spatial anchor points and the image blocks of the medical image data. Each spatial anchor point represents the center coordinate of an anatomical region in the medical image data. The spatial bias matrix is ​​used to quantify the spatial relative positional relationship between the anatomical entities and the image blocks. The initial prediction model performs cross-attention fusion on the medical image data and the medical text data based on the spatial bias matrix to obtain the image-text fusion feature sequence, and generates a prediction result based on the image-text fusion feature sequence and the structured test data. Based on the difference between the prediction results output by the initial prediction model and the true label, the parameters of the initial prediction model are adjusted to obtain the prediction model.

2. The method according to claim 1, characterized in that, Before obtaining the training sample set for training the initial prediction model, the method further includes: The system collects original medical image data, original medical text data, and original structured test data for each of the multiple sample objects. Based on the original medical image data of each sample object, it determines the timestamp, identity identifier, and corresponding real label of the medical image examination for that sample object. The original structured test data includes the name, test value, and test threshold range of each of the multiple test items. For each sample object, the corresponding timestamp of the sample object is used as the time anchor point, and the preset duration is used as the sliding step size to determine the corresponding target time window. Data within the target time window is filtered from the original medical text data and the original structured test data of the sample object to obtain the filtered medical text data and the filtered structured test data. For each sample object, the medical image examination event of the sample object is used as the index event. Based on the identity identifier, the original medical image data, the filtered medical text data, and the filtered structured test data of the sample object are associated to obtain initial multimodal medical data. The initial multimodal medical data is then cleaned to obtain the multimodal medical data. A training sample is constructed based on each of the multimodal medical data and its corresponding real label, and a training sample set is constructed based on all the training samples.

3. The method according to claim 2, characterized in that, The step involves mapping anatomical entities extracted from the medical text data of the multimodal medical data to corresponding spatial anchor points in the medical image coordinate system, and constructing a spatial bias matrix using the initial prediction model based on the spatial distance between the spatial anchor points and image patches in the medical image data, including: The medical image data is processed into a preset format to obtain processed medical image data. The processed medical image data is then feature-encoded to obtain a medical image feature sequence. The medical image feature sequence consists of the feature vectors of each image block in the medical image data and the spatial position information of the image block in the medical image coordinate system. The medical text data is converted into a string in a preset format to obtain a medical text string, and the medical text string is then feature-encoded to obtain an initial medical text feature sequence. The anatomical location entities in the medical text string are identified to obtain a set of anatomical location entities. The processed medical image data is then divided into multiple anatomical regions, and corresponding anatomical region masks are generated. The anatomical region masks are used to identify the anatomical region to which each pixel in the processed medical image data belongs. Based on the anatomical region mask and the set of anatomical parts, multiple spatial anchor points corresponding to the processed medical image data are determined, and each spatial anchor point is bound to the corresponding token of the initial medical text feature sequence to generate a medical text feature sequence. Using the initial prediction model, the spatial bias matrix is ​​constructed based on the spatial distance between the spatial anchor points in the medical text feature sequence and each of the image blocks.

4. The method according to claim 3, characterized in that, The process involves determining multiple spatial anchor points corresponding to the processed medical image data based on the anatomical region mask and the set of anatomical site entities, and binding each spatial anchor point to the corresponding token of the initial medical text feature sequence to generate a medical text feature sequence, including: Based on the set of anatomical parts entities and the anatomical region mask, a mapping relationship between anatomical parts entities and anatomical regions is constructed; For each anatomical site entity, the corresponding anatomical region is determined based on the mapping relationship, and the center coordinates of the anatomical region are calculated. The center coordinates of each anatomical region are used as a spatial anchor point, and a set of spatial anchor points is constructed based on all the spatial anchor points; each spatial anchor point in the set of spatial anchor points corresponds to a physical entity of the anatomical part. For each anatomical site entity, the position information of the token corresponding to the anatomical site entity in the initial medical text feature sequence is determined, and the token is bound to the spatial anchor point corresponding to the anatomical site entity based on the position information of the token to obtain the medical text feature sequence.

5. The method according to claim 3, characterized in that, The step of constructing the spatial bias matrix using the initial prediction model, based on the spatial distance between the spatial anchor points in the medical text feature sequence and each of the image blocks, includes: Using the initial prediction model, for each token in the medical text feature sequence that is bound to the spatial anchor point, the spatial distance between the spatial anchor point corresponding to the token and each of the image blocks is calculated, and a spatial distance matrix is ​​constructed based on all the spatial distances; Using the initial prediction model, a distance decay transformation is performed on each element in the spatial distance matrix to obtain a corresponding bias value; the bias value is used to quantify the spatial proximity between the token and the image patch, and the bias value is negatively correlated with the spatial distance; The spatial bias matrix is ​​constructed using the initial prediction model and based on the bias values ​​corresponding to all the elements.

6. The method according to claim 4, characterized in that, The process involves using the initial prediction model to perform cross-attention fusion on the medical image data and the medical text data based on the spatial bias matrix to obtain an image-text fusion feature sequence, including: Using the initial prediction model, a query matrix is ​​constructed based on the initial medical text feature sequence, and a key matrix and a value matrix are constructed based on the medical image feature sequence, respectively. Using the initial prediction model, cross-attention calculation is performed based on the query matrix and the key matrix to obtain an initial attention score matrix; each element in the initial attention score matrix is ​​used to reflect the semantic relevance between the token and the image patch. Using the initial prediction model, the target attention score matrix is ​​determined based on the initial attention score matrix and the spatial bias matrix; Using the initial prediction model, the value matrix is ​​weighted and summed based on the target attention score matrix to obtain the image-text fusion feature sequence.

7. The method according to claim 4, characterized in that, The step of generating prediction results based on the image-text fusion feature sequence and the structured test data includes: Based on the test values ​​and threshold ranges of each test item in the structured test data, the gate weight corresponding to each test item is determined, and the name and test value of the test item are weighted according to the gate weight corresponding to each test item to generate a test feature sequence; the gate weight is used to quantify the degree to which the test value deviates from the normal range; Using the multimodal aggregator of the initial prediction model, global feature representations are extracted from the test feature sequence to obtain a first global feature vector, and global feature representations are extracted from the image-text fusion feature sequence to obtain a second global feature vector; The multimodal aggregator concatenates the first global feature vector and the second global feature vector along the token dimension to obtain a multimodal fused feature sequence. Using the joint encoding network of the initial prediction model, deep feature interaction processing is performed on the multimodal fusion feature sequence to obtain a multimodal joint feature representation; The prediction result is generated based on the multimodal joint feature representation using the classification prediction head of the initial prediction model.

8. The method according to claim 7, characterized in that, Based on the test values ​​and threshold ranges of each test item in the structured test data, the gate weight corresponding to each test item is determined, and the name and test value of each test item are weighted according to the gate weight to generate a test feature sequence, including: For each of the test items, the abnormal deviation degree corresponding to the test item is calculated based on the test value of the test item and the test threshold range, and the gating weight corresponding to the test item is calculated based on the abnormal deviation degree of the test item using a gating network. Text embedding is performed on the name of each of the test items to convert the name into a dense numerical vector of a preset length, thereby obtaining the item name embedding vector corresponding to each of the test items. Feature engineering is then performed on the test values ​​of each of the test items to obtain the numerical feature vector of the test values ​​corresponding to each of the test items. For each of the test items, the item name embedding vector corresponding to the test item and the test value numerical feature vector are combined to generate a single test feature sequence. The single test feature sequence is then weighted based on the gating weight corresponding to the test item to obtain the threshold-aware test feature. All the threshold-aware test features corresponding to the test items are spliced ​​together according to a preset splicing order to generate the test feature sequence.

9. A training device for a prediction model, characterized in that, include: The acquisition module is used to acquire the training sample set for training the initial prediction model; The training sample set includes multiple training samples, each of which includes a set of multimodal medical data and corresponding real labels. The multimodal medical data includes medical image data, medical text data, and structured test data. The construction module is used to map anatomical entities extracted from the medical text data of the multimodal medical data to corresponding spatial anchor points in the medical image coordinate system, and to construct a spatial bias matrix based on the spatial distance between the spatial anchor points and the image blocks of the medical image data using the initial prediction model; each spatial anchor point is the center coordinate of an anatomical region of the medical image data, and the spatial bias matrix is ​​used to quantify the spatial relative positional relationship between the anatomical entities and the image blocks; The prediction module is used to perform cross-attention fusion on the medical image data and the medical text data based on the spatial bias matrix using the initial prediction model to obtain an image-text fusion feature sequence, and to generate a prediction result based on the image-text fusion feature sequence and the structured test data. The parameter adjustment module is used to adjust the parameters of the initial prediction model based on the difference between the prediction result output by the initial prediction model and the true label, so as to obtain the prediction model.

10. A processor, characterized in that, Used to run a computer program, which, when running, executes the training method for the prediction model as described in any one of claims 1-8.

Images