Thyroid cancer electronic medical record processing method and system based on multi-modal data

Abstract

The invention relates to the technical field of medical intelligent informatization, and discloses a thyroid cancer electronic medical record processing method and system based on multi-modal data, and the method comprises the steps: obtaining a thyroid image and an associated electronic medical record text containing lesion position description; extracting a text orientation semantic vector representing a relative spatial relationship from the text, and extracting a focus morphology feature vector representing a visual morphology and an image spatial relationship vector representing a spatial position relative to a preset anatomical structure from the image; respectively calculating an orientation semantic consistency score and a morphological neighborhood consistency score, and fusing to obtain a comprehensive consistency score; and finally, comparing the comprehensive consistency score with a preset threshold value to determine whether the text description corresponds to the focus in the image. According to the method, cross-modal accurate matching is realized, the diagnosis and treatment data utilization rate and the judgment accuracy are improved, the high efficiency and accuracy requirements of clinical diagnosis and treatment are met, and meanwhile, the method is adaptive to a medical teaching scene.

Description

Technical Field

[0001] This invention relates to the field of medical intelligent information technology, and more specifically, to a method for processing electronic medical records for thyroid cancer based on multimodal data. Background Technology

[0002] In the clinical diagnosis and treatment of thyroid cancer, the collaborative analysis of multimodal data from electronic medical records and ultrasound images is crucial for lesion assessment and malignancy risk evaluation. Currently, the correlation between thyroid ultrasound images and electronic medical records relies on manual annotation or literal matching, which struggles to address the non-standardized representation of clinical data. The core issue is the inaccurate lesion localization.

[0003] The same lesion often exhibits discrepancies in its location description across different examination reports, such as "nodule in the upper medial segment of the left lobe" versus "nodule near the isthmus of the upper pole of the left lobe." This non-standardization of natural language descriptions prevents the automatic and accurate matching of the spatial location of the lesion between the text and the imaging. Existing technologies often assume standardized textual location descriptions and lack the ability to handle such implicit semantic biases, easily leading to mismatches between textual and image features, inconsistencies in data from multiple reports, and even misclassification as multiple nodules, directly affecting the accuracy of subsequent malignancy risk prediction.

[0004] Existing multimodal data processing methods often rely on general pre-trained models for text semantic extraction, without specific optimization for thyroid-related location descriptions. This makes it difficult to capture the semantic relationships between specialized synonyms such as "upper pole" and "upper segment" or "near" and "adjacent." Image feature extraction focuses on lesion morphology recognition, neglecting to model its relative spatial relationship with anatomical structures such as the thyroid isthmus and capsule. Furthermore, the lack of a cross-modal dual-constraint verification mechanism, relying solely on single-dimensional feature matching to determine the correspondence between images and text, further reduces the accuracy of the association.

[0005] The aforementioned problems make it difficult to fully realize the clinical value of multimodal data, increase the burden on doctors, pose risks of misdiagnosis and missed diagnosis, and make it difficult to directly use this type of clinical data for cultivating multimodal diagnostic and treatment thinking in medical education, thus affecting the standardization and practicality of teaching cases. Summary of the Invention

[0006] In response, the present invention provides a method and system for processing electronic medical records for thyroid cancer based on multimodal data, so as to at least partially solve the above-mentioned technical problems.

[0007] This invention provides a method for processing electronic medical records for thyroid cancer based on multimodal data, comprising the following steps: Obtain thyroid images and associated electronic medical record text descriptions, the text descriptions including location information for the lesions; Based on the text description, a text orientation semantic vector is extracted, which represents the relative spatial relationship of the location description information; Based on the thyroid images, lesion morphology feature vectors and image spatial relationship vectors are extracted. The lesion morphology feature vectors represent the visual morphology of the lesions, and the image spatial relationship vectors represent the spatial position of the lesions relative to the preset anatomical structures in the images. Based on the text orientation semantic vector and the image spatial relationship vector, an orientation semantic consistency score is calculated. The orientation semantic consistency score is used to quantify the degree of matching between the spatial orientation described in the text and the spatial location of the lesion in the image. Based on the text orientation semantic vector and the lesion morphological feature vector, a morphological neighborhood consistency score is calculated. The morphological neighborhood consistency score is used to quantify the degree of matching between the morphological features described in the text and the visual morphology of the lesion in the image. Based on the orientation semantic consistency score and the morphological neighborhood consistency score, a comprehensive consistency score is obtained; Based on the comprehensive consistency score, the text description is compared with a preset threshold to determine whether it corresponds to the lesion in the thyroid ultrasound image.

[0008] In one possible embodiment, extracting the text location semantic vector specifically includes: The text description is preprocessed; The pre-trained language model is used to encode the pre-processed text to obtain an initial text vector representation. The initial text vector representation is mapped through a learnable directional semantic projection layer to obtain a text directional semantic vector with reduced dimensionality and focused on directional semantics.

[0009] In one possible embodiment, text orientation semantic vectors are extracted based on a text orientation semantic parsing module. During the training process, the text orientation semantic parsing module adopts a contrastive learning loss function to make the distance between text orientation semantic vectors corresponding to text segments describing the same spatial region in the vector space smaller than the distance between text orientation semantic vectors corresponding to text segments describing different spatial regions.

[0010] In one possible embodiment, the extraction of lesion morphological feature vectors and image spatial relationship vectors specifically includes: Obtain images of the lesion region or region of interest in the thyroid imaging; The image or region of interest is processed based on the visual Transformer model to extract global visual features; The global visual features are processed through a morphological feature branch and a spatial relationship branch, respectively. The morphological feature branch outputs a morphological feature vector representing the shape, texture and boundary of the lesion itself. The spatial relationship branch combines the coordinate information of the lesion in the image and the position information of the preset anatomical structure detected from the image to output an image spatial relationship vector representing the relative relationship between the lesion and the preset anatomical structure.

[0011] In one possible embodiment, the preset anatomical structure includes at least one of the thyroid isthmus, thyroid capsule, and thyroid lobe margin; the image spatial relationship vector is based on the normalized relative distance and / or directional angle features between the lesion and at least one of the preset anatomical structures.

[0012] In one possible embodiment, the orientation semantic consistency score is calculated, specifically by calculating the cosine similarity between the text orientation semantic vector and the image spatial relationship vector after projection transformation. The morphological neighborhood consistency score is calculated by decoding the expected morphological feature vector from the initial text vector representation output by the pre-trained language model through an independent morphological feature projection layer, then calculating the similarity between the expected morphological feature vector and the lesion morphological feature vector, and combining the matching degree between whether the text contains specific morphological keywords and the morphological labels detected in the image, and performing weighted fusion to obtain the morphological neighborhood consistency score.

[0013] In another aspect, this application also provides an electronic medical record processing system for thyroid cancer based on multimodal data, comprising: The image and text data acquisition module is used to acquire thyroid images and associated electronic medical record text descriptions, wherein the text descriptions include location descriptions of the lesions; The text orientation semantic vector extraction module is used to extract text orientation semantic vectors based on the text description, wherein the text orientation semantic vectors represent the relative spatial relationships of the location description information; The image feature vector extraction module is used to extract lesion morphology feature vectors and image spatial relationship vectors based on the thyroid image. The lesion morphology feature vectors represent the visual morphology of the lesion, and the image spatial relationship vectors represent the spatial position of the lesion relative to a preset anatomical structure in the image. The orientation semantic consistency score calculation module is used to calculate the orientation semantic consistency score based on the text orientation semantic vector and the image spatial relationship vector. The orientation semantic consistency score is used to quantify the degree of matching between the spatial orientation of the text description and the spatial location of the lesion in the image. The morphological neighborhood consistency score calculation module is used to calculate the morphological neighborhood consistency score based on the text orientation semantic vector and the lesion morphological feature vector. The morphological neighborhood consistency score is used to quantify the degree of matching between the morphological features described in the text and the visual morphology of the lesion in the image. The comprehensive consistency score calculation module is used to obtain a comprehensive consistency score based on the orientation semantic consistency score and the morphological neighborhood consistency score. The lesion correspondence determination module is used to determine whether the text description corresponds to the lesion in the thyroid ultrasound image by comparing it with the comprehensive consistency score and a preset threshold.

[0014] In another aspect, this application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the thyroid cancer electronic medical record processing method based on multimodal data as described above.

[0015] In another aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor to implement the thyroid cancer electronic medical record processing method based on multimodal data as described above.

[0016] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the thyroid cancer electronic medical record processing method based on multimodal data as described above.

[0017] This application achieves accurate cross-modal matching under non-standardized expressions through dual constraints of location semantics and image contours. A DeBERTa-v3-large model, fine-tuned with a thyroid medical corpus, collaborates with a location semantic projection layer to accurately extract text location semantic vectors. A Swin Transformer V2-base model simultaneously captures the morphology of lesions and their spatial relationship with preset anatomical structures. Dual consistency scoring and comprehensive judgment effectively avoid the bias of single-dimensional matching, significantly reducing the image-text mismatch rate and the risk of misdiagnosis of multiple nodules. This invention improves the reliability and efficiency of multimodal data collaborative analysis, providing accurate data for subsequent diagnostic and treatment processes such as lesion malignancy risk assessment. Simultaneously, the standardized image-text association results can be directly used as medical teaching cases, contributing to the cultivation of multimodal diagnostic and treatment thinking, and possessing significant clinical application and teaching practice value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of an electronic medical record processing method for thyroid cancer based on multimodal data, provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the text orientation semantic parsing module provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the text orientation semantic vector extraction process provided in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the image feature extraction process provided in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the morphological neighborhood consistency scoring process provided in an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of an electronic medical record processing system for thyroid cancer based on multimodal data, provided in an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this disclosure are information and data authorized by the user or fully authorized by all parties.

[0028] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the thyroid cancer electronic medical record processing method based on multimodal data according to the present invention. It should be noted that these embodiments are merely illustrative of the invention and not intended to limit its scope. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the inventive concept should be included within the scope of protection of the present invention.

[0029] like Figure 1 As shown in the figure, an embodiment of the present invention discloses a schematic diagram of a method for processing electronic medical records for thyroid cancer based on multimodal data, including the following method steps: S1, Obtain thyroid images and associated electronic medical record text descriptions, wherein the text descriptions include location information for the lesions; S2, Based on the text description, extract the text orientation semantic vector, which represents the relative spatial relationship of the location description information; S3. Based on the thyroid images, extract the lesion morphology feature vector and the image spatial relationship vector. The lesion morphology feature vector represents the visual morphology of the lesion, and the image spatial relationship vector represents the spatial position of the lesion relative to the preset anatomical structure in the image. S4. Based on the text orientation semantic vector and the image spatial relationship vector, calculate the orientation semantic consistency score. The orientation semantic consistency score is used to quantify the degree of matching between the spatial orientation of the text description and the spatial location of the lesion in the image. S5. Based on the text orientation semantic vector and the lesion morphological feature vector, calculate the morphological neighborhood consistency score. The morphological neighborhood consistency score is used to quantify the degree of matching between the morphological features described in the text and the visual morphology of the lesion in the image. S6. Based on the orientation semantic consistency score and the morphological neighborhood consistency score, a comprehensive consistency score is obtained. S7. Based on the comprehensive consistency score, determine whether the text description corresponds to the lesion in the thyroid ultrasound image by comparing it with a preset threshold.

[0030] In some embodiments, for step S1, exemplarily, the data may originate from thyroid cancer diagnosis and treatment systems at various levels of medical institutions, including but not limited to hospital ultrasound departments and electronic medical record management systems. The acquired thyroid images may be ultrasound images, meeting clinical-grade data quality requirements. The preferred format is the commonly used DICOM format in the medical field, but general image formats such as PNG can also be selected depending on the actual application scenario. The images must contain complete lesion area information and clearly display anatomical structures; for example, key structures such as the thyroid lobe, isthmus, and capsule should be effectively identifiable. For ultrasound image sequences, complete examination frames or key frames selected by the doctor are acquired to ensure that the image data comprehensively reflects the morphological and spatial location characteristics of the lesion.

[0031] The electronic medical record text description associated with thyroid ultrasound images is extracted from the corresponding patient's electronic medical record report. The text description includes information describing the location of the lesion, such as "nodule in the upper segment of the left lobe, slightly medial" or "occupancy near the isthmus in the upper pole of the right lobe" in natural language. During the data association process, a one-to-one correspondence between ultrasound images and electronic medical record text descriptions can be established using key information such as the patient's unique identifier and examination timestamp, avoiding data confusion.

[0032] In some embodiments, for step S2, the text description is processed by the text orientation semantic parsing module to extract the text orientation semantic vector, which represents the relative spatial relationship implied by the location description information.

[0033] Please see Figure 2 , Figure 2 This is a schematic diagram of the text orientation semantic parsing module provided in an embodiment of the present invention. The semantic parsing module includes a DeBERTa-v3-large model and an orientation semantic projection layer, wherein the orientation semantic projection layer includes a combination of linear transformation and layer normalization. The DeBERTa-v3-large model is used to extract a global semantic vector containing multi-dimensional information such as orientation, morphology, and lesion type from the preprocessed electronic medical record text for thyroid cancer. The orientation semantic projection layer filters and strengthens semantic features related to lesion location from the global semantic vector, while suppressing interference from non-orientational information such as morphology and size, generating an orientation semantic vector focused on relative spatial relationships.

[0034] Specifically, please refer to Figure 3 , Figure 3 This is a schematic diagram of the text location semantic vector extraction process provided in an embodiment of the present invention. According to an embodiment of the present invention, in S301, the text description is first preprocessed. The preprocessing process mainly includes redundant information removal, Chinese word segmentation, and medical terminology standardization.

[0035] Specifically, firstly, for template-like fixed paragraph beginnings that may exist in the electronic medical record text description, such as "The patient's ultrasound examination this time indicated:" and irrelevant modifiers such as "approximately XX mm in size" or "nature to be determined," these are identified and removed using preset keyword matching rules or regular expressions, retaining only the core descriptive fragments directly related to the lesion location. For example, for the text description "The patient's ultrasound examination this time indicated: a solid nodule of approximately 5 mm × 3 mm in size, on the medial side of the upper left lobe, nature to be determined," after removing redundant information, the retained core descriptive fragment is "nodule on the medial side of the upper left lobe."

[0036] Subsequently, a Chinese word segmentation tool was used to segment the core descriptive fragments after removing redundancy, breaking down the continuous natural language text into independent semantic units. Preferably, mature tools such as jieba and HanLP were selected for word segmentation, and a custom dictionary was added using a professional lexicon in the field of thyroid medicine to ensure that specialized terms such as "isthmus," "capsule," and "upper pole" were accurately segmented, avoiding subsequent semantic comprehension deviations due to segmentation errors. For example, after segmenting "left lobe upper segment medial nodule," the semantic unit set {left lobe, upper segment, medial, nodule} was obtained.

[0037] Finally, medical terminology standardization was implemented. Since different doctors may use different terminology to describe the same lesion location—for example, describing "upper pole" as "upper end" and "medial side" as "near midline side"—directly encoding the original descriptions would result in semantic vectors that fail to accurately reflect the same spatial relationships.

[0038] Therefore, this invention constructs a synonym substitution table for thyroid location descriptions, mapping non-standard location descriptions to standard directional terms. Specifically, the synonym substitution table is constructed based on thyroid ultrasound report corpus and clinical expert experience, containing the correspondence between common non-standard terms and standard terms, such as "upper pole = upper end," "medial = near the midline," "lower segment = lower pole," and "near the isthmus = adjacent to the isthmus." During implementation, by traversing the semantic units after word segmentation, the non-standard location terms are replaced with standard directional terms from the synonym substitution table. For example, "left lobe upper end near the midline nodule" is standardized to "left lobe upper pole medial nodule." Thus, through the standardization of medical terminology, the non-standard expressions of different doctors are unified into standard terms.

[0039] In step S302, the initial text vector representation, i.e., the global semantic vector, is obtained. After completing the text description preprocessing, the preprocessed text is encoded using a pre-trained language model to obtain the initial text vector representation. Preferably, the pre-trained language model is the DeBERTa-v3-large model, which is fine-tuned on a thyroid medical corpus to improve the model's semantic understanding of location descriptions in the thyroid domain. The DeBERTa-v3-large model belongs to the pre-trained language model (PLM) category, which is based on an optimized and upgraded Transformer architecture and has excellent semantic representation capabilities in natural language understanding (NLU) tasks.

[0040] Specifically, the fine-tuning process includes: first, collecting textual data such as thyroid ultrasound reports and treatment guidelines, extracting segments containing descriptions of lesion location such as "nodule near the isthmus of the upper pole of the left lobe," and constructing a dataset after cleaning; and then classifying the dataset into "different descriptions of the same location" and "descriptions of different locations" to form positive and negative sample pairs, such as "left lobe upper segment slightly medial" and "left lobe upper pole near the isthmus" as positive sample pairs, and "left lobe upper pole" and "right lobe lower pole" as negative sample pairs. Load the pre-trained DeBERTa-v3-large model and its matching word segmenter, and perform the same word segmentation process as described above. Expand the word segmenter dictionary for thyroid medical terms, such as adding professional terms like "isthmus" and "capsule" to ensure accurate term segmentation. A contrastive learning strategy is employed, with the triplet loss function at its core, to minimize the semantic vector distance between positive sample pairs and maximize the distance between negative sample pairs. A small learning rate, such as 2e-5, and the AdamW optimizer are selected, along with a batch size of 32. An early stopping strategy is used to prevent overfitting. The specific training process is detailed below and will not be repeated here.

[0041] Subsequently, the preprocessed text fragments were input into the DeBERTa-v3-large model, which was fine-tuned using a thyroid medical corpus. The model then performed semantic encoding on the text fragments using a multi-layer bidirectional Transformer encoder.

[0042] During the encoding process, the model uses a self-attention mechanism to capture the dependencies between semantic units, such as the belonging relationship between "left leaf" and "upper pole", and the modification relationship between "biased" and "inner side", and finally outputs a fixed-length initial text vector representation. , among which dimension Typical values ​​are 768 or 1024, which can be adjusted according to the model training effect and subsequent processing requirements.

[0043] Understandably, the DeBERTa-v3-large model possesses stronger semantic understanding capabilities compared to traditional pre-trained language models. Through decoupling the attention mechanism and enhancing the masking strategy, it can more accurately capture deep semantic information in text. Furthermore, by fine-tuning on the thyroid medical corpus, the model's adaptability to descriptions of specific locations within the domain is further improved, ensuring that the output initial text vector representation not only contains general semantic information but also incorporates specialized semantic features specific to the thyroid field.

[0044] In step S303, the text orientation semantic vector is mapped. Specifically, the initial text vector representation is mapped through a learnable orientation semantic projection layer to obtain a text orientation semantic vector with reduced dimensionality and focused on orientation semantics. ,in The typical value for this dimension is 128. This dimension setting effectively reduces the complexity of subsequent calculations while ensuring that semantic information is not lost.

[0045] The structure of the directional semantic projection layer is a learnable linear transformation plus a normalization layer. The parameters of the linear transformation are adaptively adjusted during the model training process, and its function is to transform the initial text vector. The process involves selecting and enhancing features relevant to location semantics while suppressing interference from irrelevant semantic features such as lesion nature and size. Layer normalization is used to standardize the distribution of the projected vectors, accelerating model training convergence and improving the stability of the vector representation. Specifically, the linear transformation process can be represented as:

[0046] in, The weight matrix is ​​a linear transformation matrix. For bias vectors, This is a temporary vector after the linear transformation. Subsequently, for... Perform layer normalization:

[0047] Thus, the final text location semantic vector is obtained. Focusing on characterizing the relative spatial relationships of lesion locations in text descriptions, it can accurately reflect the semantic similarity between descriptions of different locations.

[0048] According to an embodiment of the present invention, in order to ensure that the text orientation semantic vector can accurately cluster similar location descriptions, the text orientation semantic parsing module adopts a contrastive learning loss function during the training process, specifically a triplet loss function or a contrastive loss function, preferably a triplet loss function.

[0049] The training data consists of manually labeled text description pairs, including positive and negative sample pairs. Positive sample pairs are combinations of text fragments describing the same spatial region, confirmed by clinical experts, such as ("left lobe superior pole nodule", "left lobe upper segment near the isthmus nodule"); negative sample pairs are combinations of text fragments describing different spatial regions, such as ("left lobe superior pole nodule", "right lobe inferior pole nodule"). During training, the positive and negative sample pairs are input into the text orientation semantic parsing module to obtain their respective text orientation semantic vectors.

[0050] The goal of the triplet loss function is to ensure that the distance between the text location semantic vectors of positive sample pairs in the vector space is less than the distance between the text location semantic vectors of negative sample pairs, and to ensure that this difference is statistically significant by setting a margin. The mathematical expression of the triplet loss function is:

[0051] in, This is the location semantic vector for the anchor text. This is the directional semantic vector of the positive sample text that describes the same spatial region as the anchor text. This is the directional semantic vector of negative sample text that describes a different spatial region from the anchor text. This is a preset boundary value, typically 1.0.

[0052] By minimizing this loss function, the model can learn more discriminative textual location semantic vectors, which makes the vectors corresponding to text fragments describing the same spatial region cluster in space, and the vectors corresponding to text fragments describing different spatial regions move away from each other in space, thus effectively solving the semantic confusion problem caused by differences in textual expression in traditional methods.

[0053] In some embodiments, for step S3, the thyroid ultrasound image is processed in depth by the image feature extraction module, and the lesion morphology feature vector representing the visual morphology of the lesion itself and the image spatial relationship vector representing the spatial position of the lesion relative to the preset anatomical structure are extracted.

[0054] According to an embodiment of the present invention, please refer to Figure 4 , Figure 4 This is a schematic diagram of the image feature extraction process provided in an embodiment of the present invention.

[0055] Specifically, in S401, the image of the lesion area or region of interest (ROI) is first located and extracted from the thyroid ultrasound image. The acquisition of the ROI can be done using existing techniques, which will not be elaborated here.

[0056] After extracting the region of interest, it is normalized to adjust the size of the region of interest to a preset size, such as 224×224 pixels, and normalize the pixel grayscale value to the range of [0,1] to eliminate the problem of inconsistent size and grayscale value distribution between different images due to differences in shooting parameters and equipment models.

[0057] In step S402, global visual features are acquired. A visual Transformer model is used to process the image of the lesion area or region of interest to extract global visual features. Preferably, the SwinTransformer V2-base model is selected as the visual Transformer model. This model employs a hierarchical window attention mechanism, which can effectively capture local detail features and global structural features of the image, and exhibits excellent performance in medical image processing tasks.

[0058] Specifically, the normalized lesion region image or region of interest is input into the Swin TransformerV2-base model. The model's processing is divided into four stages: In the first stage, the input image is divided into fixed-size image patches, such as 4×4 pixels, through patch partitioning, and each patch is mapped to a one-dimensional vector through linear projection; In the second to fourth stages, window multi-head self-attention (W-MSA) and shift window multi-head self-attention (SW-MSA) are calculated alternately to capture the local and global dependencies between patches, while downsampling operations are used to gradually improve the receptive field and semantic level of the feature map; Finally, global average pooling is used to convert the high-level feature map into a fixed-length global visual feature vector, which integrates multi-dimensional information such as the morphology, texture, boundary, and spatial relationship with surrounding tissues of the lesion.

[0059] Understandably, the Swin Transformer V2-base model, compared to traditional convolutional neural networks (CNNs), is better able to capture long-range dependencies in images and has a stronger ability to represent the spatial relationships between lesions and anatomical structures such as the isthmus and capsule in thyroid ultrasound images.

[0060] In step S403, morphological feature vectors and spatial relationship vectors are generated. The global visual features are processed through morphological feature branches and spatial relationship branches respectively, outputting lesion morphological feature vectors and image spatial relationship vectors. The two branches are independent and run in parallel to ensure the specificity and accuracy of their respective features.

[0061] The morphological feature branch consists of a fully connected layer, a batch normalization layer, and an activation function. Its function is to select and enhance features related to the morphology of the lesion itself from global visual features, and suppress the interference of spatial location-related features.

[0062] Specifically, the global visual features are first input to the first fully connected layer. The weight matrix of this layer is adaptively adjusted through learning, mapping the global visual features to a preset dimension, such as 768 dimensions. Then, the feature distribution is standardized by a batch normalization layer, and a non-linear transformation is introduced through the ReLU activation function to enhance the model's feature representation capability. Finally, the second fully connected layer maps the feature vectors to a dimension consistent with the target dimension and the dimension of the text location semantic vector, i.e. The morphological feature vector of the lesion was obtained. .

[0063] Lesion morphology feature vector It focuses on characterizing the visual morphology of the lesion itself, including the overall shape of the nodule, such as round, oval, or irregular, boundary features, such as clear, blurred, or lobed, internal texture, such as uniform, non-uniform, and distribution of calcification points. These features have strong stability, and the morphological feature vectors of the same lesion under different examination times and different shooting angles have little difference.

[0064] Among them, the spatial relationship branch combines the coordinate information of the lesion in the image and the position information of the preset anatomical structure detected from the image to output an image spatial relationship vector that can characterize the relative relationship between the lesion and the preset anatomical structure, which effectively solves the problem of inaccurate spatial relationship judgment caused by relying solely on text position description in traditional methods.

[0065] Specifically, firstly, the types of preset anatomical structures are determined. According to a preferred embodiment of the present invention, the preset anatomical structures include at least one of the thyroid isthmus, thyroid capsule, and thyroid lobe margin. These anatomical structures have clear anatomical locations and stable morphological characteristics in thyroid ultrasound images and can be used as reference benchmarks for determining the spatial location of lesions.

[0066] Subsequently, the location information of pre-defined anatomical structures is detected from ultrasound images. For the thyroid isthmus, automatic detection can be performed using template matching algorithms or lightweight segmentation networks such as a lightweight version of U-Net. Template matching algorithms pre-construct a standard template for the isthmus and search for the region in the image with the highest similarity to the template to determine the location range of the isthmus. A lightweight segmentation network, trained on ultrasound images labeled with the isthmus region, automatically outputs a segmentation mask for the isthmus, thereby determining its location coordinates. For the thyroid capsule and thyroid lobe margins, edge detection algorithms such as Canny edge detection and the Sobel operator are used to detect tissue edges in the images. Combined with the overall morphological features of the thyroid gland, the location range of the capsule and lobe margins is identified and determined.

[0067] After obtaining the coordinate information of the lesion in the image, namely the center coordinates of the bounding box of the region of interest, or the centroid coordinates of the segmentation mask, and the position information of the preset anatomical structures, namely the bounding box coordinates of the anatomical structures or the position range of the segmentation mask, the relative spatial features between the lesion and each preset anatomical structure are calculated, including normalized relative distance and orientation angle features.

[0068] For the calculation of normalized relative distance, the Euclidean distance between the coordinates of the lesion center and the coordinates of the center of the preset anatomical structure is first calculated. Then divide that distance by the diagonal length of the ultrasound image. The normalized relative distance is obtained. , making This eliminates the influence of different image sizes on distance calculation. For example, if the Euclidean distance between the lesion center and the isthmus center is 10 mm, and the diagonal length of the ultrasound image is 50 mm, then the normalized relative distance is 0.2.

[0069] For the calculation of directional angle features, a two-dimensional rectangular coordinate system is established with the center coordinates of the preset anatomical structure as the origin. The azimuth angle of the lesion center coordinates relative to this origin is the directional angle. , For example, if the lesion is located to the upper right of the center of the isthmus, the directional angle is approximately 45°. To facilitate subsequent vector mapping, the directional angle is converted to radians and mapped to a two-dimensional vector using trigonometric functions (sine and cosine). This vector can accurately represent the directional relationship of the lesion relative to the preset anatomical structure.

[0070] Finally, the normalized relative distance and orientation angle mapping vectors between the lesion and each preset anatomical structure are concatenated to obtain the initial spatial relationship feature vector. Subsequently, this initial feature vector is mapped and standardized through a fully connected layer and a layer normalization layer to output an image spatial relationship vector with the same dimension as the text orientation semantic vector. ( ).

[0071] Therefore, the image spatial relationship vector It can comprehensively and accurately characterize the spatial position of lesions relative to preset anatomical structures in images.

[0072] According to an embodiment of the present invention, in order to ensure the stability and discriminative power of the lesion morphology feature vector and the image spatial relationship vector, the image feature extraction module adopts a phased training strategy, combining self-supervised learning and labeled contrastive learning for model optimization.

[0073] In the first stage, the Swin Transformer V2-base model was pre-trained under self-supervised supervision. The model was initialized using a Masked Image Modeling (MIM) task, which involved randomly masking patches in an image and having the model predict the original pixel values ​​of the masked patches. This improved the model's ability to capture features related to local details and global structure in the image. The pre-training data consisted of a large number of unlabeled thyroid ultrasound images, reducing reliance on labeled data.

[0074] The second stage involves labeled contrastive learning training, with training objectives designed for both the lesion morphological feature vector and the image spatial relationship vector. For the lesion morphological feature vector... The training objective is to minimize the distance between the morphological feature vectors of the same nodule at different angles and magnifications in the vector space. This objective is achieved using a contrastive loss function.

[0075]

[0076] in, The number of different image samples for the same nodule. The cosine similarity function is used. The temperature parameter is typically set to 0.07. Through this loss function, the model can learn morphological feature vectors with affine invariance, ensuring consistency in the morphological features of the same lesion.

[0077] For image spatial relation vectors The training objective is to minimize the distance between the spatial relationship vectors of the same nodule in the vector space under different slicing orders and shooting positions. This objective is achieved using the mean squared error loss function (MSE).

[0078] in, The number of different image samples for the same nodule. This is the reference spatial relationship vector for the nodule, calculated from manually labeled standard location information. By minimizing this loss function, we ensure that the image spatial relationship vector can stably represent the spatial location relationship of the lesion.

[0079] The total loss function is a weighted sum of the morphological feature training loss and the spatial relationship training loss:

[0080] in, and These are weighting coefficients, initially set to 0.5, and can be adjusted based on the model's performance on the validation set.

[0081] Through the above training strategy, the image feature extraction module can output lesion morphology feature vectors and image spatial relationship vectors with high stability and strong discriminative power.

[0082] In some embodiments, for step S4, an orientation semantic consistency score is obtained by calculating the similarity between the text orientation semantic vector and the image spatial relation vector. This score is used to quantify the degree of matching between the spatial orientation of the text description and the spatial location of the lesion in the image.

[0083] According to an embodiment of the present invention, the core of the calculation of the orientation semantic consistency score is the similarity calculation between the text orientation semantic vector and the image spatial relationship vector. However, since the two are derived from two different modalities, text and image, their vector space distributions are different. Therefore, it is necessary to first perform a projection transformation on the image spatial relationship vector so that the two are in the same vector space before performing the similarity calculation.

[0084] First, a linear projection layer is constructed, whose input is the image spatial relation vector. The output is the vector after projection transformation. The parameters of the projection layer are adaptively adjusted during the model training process. The mathematical expression for the linear projection layer is:

[0085] in, Here is the projection layer weight matrix. As a bias vector, the dimensions of the projected vector are consistent with the dimensions of the text location semantic vector. This ensures that the two can perform similarity calculations.

[0086] Subsequently, the semantic vector of text orientation is calculated. Image spatial relationship vector after projection transformation The cosine similarity between the two points is used as the initial location semantic consistency score. The mathematical expression for cosine similarity is:

[0087] in, Represents the vector dot product. This represents the L2 norm of the vectors. The range of values ​​for cosine similarity is... The closer the value is to 1, the higher the degree of matching between the text location description and the image spatial location; the closer the value is to -1, the lower the degree of matching between the two.

[0088] Finally, the initial cosine similarity is normalized and mapped to... The range is used to obtain the final location semantic consistency score. The mathematical expression for normalization is:

[0089] This normalization process facilitates threshold setting and result determination.

[0090] In some embodiments, for step S5, please refer to Figure 5 , Figure 5 This is a schematic diagram of the morphological neighborhood consistency scoring process provided in an embodiment of the present invention. Specifically, by fusing the similarity between the morphological features implied by the text and the actual morphological features of the image, and the matching degree between the text morphological keywords and the image morphological labels, a morphological neighborhood consistency score is obtained, effectively solving the problem of inaccurate morphological matching caused by relying on only a single feature in traditional methods.

[0091] Specifically, in step S501, the expected morphological feature vector of the text is decoded. First, the expected morphological feature vector is decoded from the initial text vector representation output by the pre-trained language model through an independent morphological feature projection layer. Specifically, an additional morphological feature projection layer is added to the text orientation semantic parsing module. This projection layer is set in parallel with the orientation semantic projection layer in step S2, sharing the initial text vector representation output by the pre-trained language model. The morphological feature projection layer consists of a fully connected layer and a layer normalization layer. Its function is to filter and extract semantic information related to the morphology of lesions from the initial text vector, such as morphological features such as "solid", "cystic" and "clear boundary" implied in the text description.

[0092] The mathematical expression for the morphological feature projection layer is:

[0093] in, This is the weight matrix of the morphological feature projection layer. The bias vector is the output expected morphological feature vector. Dimensions and lesion morphology feature vectors Maintaining consistency in dimensions This ensures that the two can perform similarity calculations.

[0094] During model training, the parameters of the morphological feature projection layer are optimized along with other parameters of the text orientation semantic parsing module. By minimizing the mean squared error loss between the expected morphological feature vector and the actual morphological feature vector (which is converted from manually labeled lesion morphology information), the model can accurately decode the expected morphological feature vector related to the actual lesion morphology from the text description. For example, for the text description "irregular boundary nodule in the upper segment of the left lobe," the model can decode the expected morphological feature vector representing the "irregular boundary." .

[0095] In step S502, the similarity of text and image morphological features is calculated. The expected morphological feature vector is then calculated. With lesion morphological feature vector The cosine similarity between the text and the image is used to quantify the degree of match between the morphological features implied by the text and the actual visual morphology of the lesion in the image. The mathematical expression for cosine similarity is:

[0096] Similarly, the range of values ​​for cosine similarity is... The closer the value is to 1, the higher the degree of matching between the morphological features implied by the text and the actual morphological features of the image.

[0097] In step S503, the matching degree between text morphological keywords and image morphological labels is calculated. According to embodiments of the present invention, the text description may contain explicit morphological keywords, such as "solid," "cystic," "well-defined borders," "calcified," and "lobulated," which directly reflect the morphological characteristics of the lesion. Simultaneously, the actual morphological label of the lesion can be detected through the image feature extraction module; therefore, it is necessary to calculate the matching degree between the two. .

[0098] First, a keyword dictionary of thyroid lesion morphology was constructed. The dictionary contains keywords of common lesion morphology in clinical practice and their corresponding standard morphological labels. For example, the keyword "solid" corresponds to the label "solid", the keyword "cystic" corresponds to the label "cystic", the keyword "well-defined borders" corresponds to the label "well-defined borders", and the keyword "calcification" corresponds to the label "calcification", etc.

[0099] Subsequently, keyword matching is performed on the preprocessed text description from step S2 to identify whether it contains keywords from the morphological keyword dictionary. If the text description contains multiple morphological keywords, all relevant keywords are extracted and mapped to standard morphological labels; if the text description does not contain any morphological keywords, it is marked as having no morphological keywords.

[0100] Simultaneously, the actual morphological labels of lesions in the image are detected through the image feature extraction module: for the boundary features of the lesion, the edge detection algorithm calculates the clarity index of the boundary, and the label is determined as "clear boundary" or "blurred boundary" based on the index threshold; for the internal echo features of the lesion, the uniformity index of the internal gray value is calculated, and the label is determined as "solid", "cystic", "mixed" or "calcified" based on the presence of high-brightness calcification points; for the overall shape of the lesion, the label is determined as "regular" such as round or elliptical or "irregular" such as lobed or spiky based on the calculation of morphological parameters such as roundness and aspect ratio.

[0101] Finally, the matching degree between the standard morphological labels corresponding to the text morphological keywords and the actual morphological labels of the images is calculated. The matching degree is calculated using the Jaccard similarity coefficient, whose mathematical expression is:

[0102] in, This is a set of standard morphological tags corresponding to text morphological keywords. This is a set of actual image morphology labels. If the text description does not contain morphology keywords, then... Then set This indicates that the matching degree of this dimension is at a medium level, with no bias towards matching or not matching.

[0103] For example, a text description containing morphological keywords such as "real" and "clearly defined boundaries" corresponds to a set of tags. Image actual form tag set Then the intersection of the two Union Matching degree .

[0104] In S504, weighted fusion yields a morphological neighborhood consistency score. According to an embodiment of the present invention, the cosine similarity between text and image morphological features and the matching degree between text morphological keywords and image morphological labels are weighted and fused, and then mapped to a sigmoid function. The range is used to obtain the final morphological neighborhood consistency score. .

[0105] The mathematical expression for weighted fusion is:

[0106] in, and This is an adjustable coefficient used to balance the weights of the two indicators. Based on experimental verification, the optimal coefficient value is [value to be filled in]. , The sigmoid function maps the weighted fusion result to... The range makes the value of the morphological neighborhood consistency score more in line with the fusion requirements of the subsequent comprehensive score.

[0107] Morphological Neighborhood Consistency Score The closer the value is to 1, the higher the degree of matching between the morphological features implied by the text description and the actual visual morphology of the lesion in the image; the closer the value is to 0, the lower the degree of matching. This score comprehensively considers the implicit morphological semantic features and explicit morphological keywords in the text, as well as the actual morphological features and labels of the image, comprehensively quantifying the morphological matching relationship between the text and the image, and effectively improving the accuracy of morphological matching judgment.

[0108] In some embodiments, for step S6, a comprehensive consistency score is obtained by weighting the orientation semantic consistency score and the morphological neighborhood consistency score.

[0109] According to an embodiment of the present invention, the overall consistency score The calculation uses a weighted sum method, and its mathematical expression is:

[0110] in, The weights for the semantic consistency score of orientation. The weights for the morphological neighborhood consistency score are given, and satisfy the following conditions: .

[0111] The initial weighting uses an equal-weighting strategy, that is... , This setting balances the importance of orientation matching and shape matching in the initial stage.

[0112] However, in practical applications, different medical institutions may have different habits in describing ultrasound reports and different imaging parameters. Therefore, it is necessary to optimize and adjust the weights based on the specific validation set data. Preferably, a grid search method is used to optimize the weights, and the scope of the grid search is... , With a step size of 0.1, the model's accuracy (ACC), precision (P), recall (R), and F1 score on the validation set are calculated under different weight combinations. The weight combination with the best overall performance is selected as the final weight parameters. For example, if on a certain validation set, when... , When the model has the highest F1 score, that weight combination is used to calculate the overall consistency score.

[0113] Understandably, the calculation process for the overall consistency score fully considers the synergistic effect of orientation matching and morphological matching, avoiding judgment bias caused by a single scoring dimension. For example, if the orientation semantic consistency score between a text description and an image is... Morphological neighborhood consistency score The weight combination is , Then the overall consistency score This indicates a high degree of overall matching between the text description and the imaging lesions; if the directional semantic consistency score is high... Morphological neighborhood consistency score Then the overall consistency score This indicates a moderate overall matching degree, requiring further judgment based on threshold values.

[0114] In some embodiments, for step S7, according to the present invention, the preset threshold includes a confirmation threshold. and rejection threshold ,in The setting of thresholds should take into account the accuracy, false negative rate and false negative rate of the method to ensure that the correspondence can be accurately determined in clinical applications, while avoiding medical risks caused by improper threshold settings. This will not be elaborated further here.

[0115] Preferably, the overall consistency score is compared with preset confirmation and rejection thresholds: if the overall consistency score is not lower than the confirmation threshold, the text description is determined to correspond to the lesion in the thyroid ultrasound image; if the overall consistency score is not higher than the rejection threshold, the text description is determined not to correspond to the lesion in the thyroid ultrasound image.

[0116] Specifically, in some embodiments, based on the comparison between the overall consistency score and a preset threshold, the following judgment logic is used to determine whether the text description corresponds to the lesion in the thyroid ultrasound image: If the overall consistency score If the text description corresponds to the lesion in the thyroid ultrasound image, then the spatial orientation of the text description is highly consistent with the spatial location of the lesion in the image, and the morphological features implied by the text description also highly match the actual visual morphology of the lesion in the image. The probability that the two point to the same lesion is extremely high. This determination result can be directly used in subsequent clinical applications such as assessing the malignancy risk of thyroid cancer.

[0117] If the overall consistency score If the text description does not correspond to the lesion in the thyroid ultrasound image, it is determined that the text description does not correspond to the lesion in the image. In this case, the spatial location of the text description differs significantly from the spatial location of the lesion in the image, or the morphological features implied by the text description differ significantly from the actual visual morphology of the lesion in the image. The probability that the two point to the same lesion is extremely low. This determination result can prompt doctors to re-verify the correlation between the text description and the image data to avoid diagnostic errors caused by data mismatch.

[0118] If the overall consistency score satisfies If the text description and the imaging lesion are unclear, the result is "suspected candidate." In this case, the correspondence between the text description and the imaging lesion is ambiguous, and an accurate judgment cannot be made solely by automated algorithms. To ensure the safety and reliability of clinical applications, a manual review process can be triggered in this situation. The text description, ultrasound images, and various consistency score data are submitted to the clinician, who makes the final judgment based on their professional knowledge and clinical experience. Optionally, the manual judgment result can be fed back to the system for subsequent model optimization and training.

[0119] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a thyroid cancer electronic medical record processing system 6 based on multimodal data, provided in an embodiment of this application. Figure 6 As shown, system 600 includes: The image and text data acquisition module 601 is used to acquire thyroid images and associated electronic medical record text descriptions, wherein the text descriptions include location description information for the lesions; The text orientation semantic vector extraction module 602 is used to extract text orientation semantic vectors based on the text description, wherein the text orientation semantic vectors represent the relative spatial relationship of the location description information; The image feature vector extraction module 603 is used to extract lesion morphology feature vectors and image spatial relationship vectors based on the thyroid image. The lesion morphology feature vectors represent the visual morphology of the lesion, and the image spatial relationship vectors represent the spatial position of the lesion relative to a preset anatomical structure in the image. The orientation semantic consistency score calculation module 604 is used to calculate the orientation semantic consistency score based on the text orientation semantic vector and the image spatial relationship vector. The orientation semantic consistency score is used to quantify the degree of matching between the spatial orientation of the text description and the spatial location of the lesion in the image. The morphological neighborhood consistency score calculation module 605 is used to calculate the morphological neighborhood consistency score based on the text orientation semantic vector and the lesion morphological feature vector. The morphological neighborhood consistency score is used to quantify the degree of matching between the morphological features described in the text and the visual morphology of the lesion in the image. The comprehensive consistency score calculation module 606 is used to obtain a comprehensive consistency score based on the orientation semantic consistency score and the morphological neighborhood consistency score. The lesion correspondence determination module 607 is used to determine whether the text description corresponds to the lesion in the thyroid ultrasound image based on the comprehensive consistency score and by comparing it with a preset threshold.

[0120] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0121] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0122] Please see Figure 7 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 7 As shown, the electronic device 700 may include: The system includes at least one processor 701, at least one network interface 704, a user interface 703, a memory 705, and at least one communication bus 702. The communication bus 702 is used to enable connection and communication between the components. The user interface 703 may include buttons, and optionally include a standard wired or wireless interface. The network interface 704 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0123] The processor 701 may include one or more processing cores and connect to various parts within the electronic device 700 via various interfaces and lines. It implements the various functions and data processing of the electronic device 700 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 705, and by accessing data in the memory 705. Optionally, the processor 701 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 701 may also integrate one or more combinations of CPU, GPU, and modem. The CPU is mainly used to process the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem is used for wireless communication. It is understood that the modem may not be integrated into the processor 701, but may be implemented through a separate chip.

[0124] The memory 705 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 705 includes a non-transitory computer-readable medium for storing instructions, programs, code, code sets, or instruction sets. The memory 705 may be divided into a program storage area and a data storage area, wherein the program storage area may be used to store instructions for implementing an operating system, instructions for implementing at least one function such as touch functionality, audio playback functionality, image playback functionality, etc., and instructions for implementing the foregoing method embodiments; the data storage area may be used to store data involved in the relevant method embodiments. The memory 705 may also be at least one storage device located remotely from the processor 701. Figure 7 As shown, the memory 705, which serves as a computer storage medium, may contain an operating system, a network communication module, a user interface module, and program instructions.

[0125] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by processor 701, the functions defined in the methods of this application are performed.

[0126] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0127] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

Claims

1. A method for processing electronic medical records for thyroid cancer based on multimodal data, characterized in that, Includes the following steps: Obtain thyroid images and associated electronic medical record text descriptions, the text descriptions including location information for the lesions; Based on the text description, a text orientation semantic vector is extracted, which represents the relative spatial relationship of the location description information; Based on the thyroid images, lesion morphology feature vectors and image spatial relationship vectors are extracted. The lesion morphology feature vectors represent the visual morphology of the lesions, and the image spatial relationship vectors represent the spatial position of the lesions relative to the preset anatomical structures in the images. Based on the text orientation semantic vector and the image spatial relationship vector, an orientation semantic consistency score is calculated. The orientation semantic consistency score is used to quantify the degree of matching between the spatial orientation described in the text and the spatial location of the lesion in the image. Based on the text orientation semantic vector and the lesion morphological feature vector, a morphological neighborhood consistency score is calculated. The morphological neighborhood consistency score is used to quantify the degree of matching between the morphological features described in the text and the visual morphology of the lesion in the image. Based on the orientation semantic consistency score and the morphological neighborhood consistency score, a comprehensive consistency score is obtained; Based on the comprehensive consistency score, the text description is compared with a preset threshold to determine whether it corresponds to the lesion in the thyroid ultrasound image.

2. The method for processing electronic medical records for thyroid cancer based on multimodal data according to claim 1, characterized in that, Extracting the semantic vector of text location, specifically including: The text description is preprocessed; The pre-trained language model is used to encode the pre-processed text to obtain an initial text vector representation. The initial text vector representation is mapped through a learnable directional semantic projection layer to obtain a text directional semantic vector with reduced dimensionality and focused on directional semantics.

3. The method for processing electronic medical records for thyroid cancer based on multimodal data according to claim 2, characterized in that, The text orientation semantic vector is extracted based on the text orientation semantic parsing module. During the training process, the text orientation semantic parsing module adopts a contrastive learning loss function to make the distance between the text orientation semantic vectors corresponding to text segments describing the same spatial region in the vector space smaller than the distance between the text orientation semantic vectors corresponding to text segments describing different spatial regions.

4. The method for processing electronic medical records for thyroid cancer based on multimodal data according to claim 1, characterized in that, Extracting lesion morphological feature vectors and image spatial relationship vectors, specifically including: Obtain images of the lesion region or region of interest in the thyroid imaging; The image or region of interest is processed based on the visual Transformer model to extract global visual features; The global visual features are processed through a morphological feature branch and a spatial relationship branch, respectively. The morphological feature branch outputs a morphological feature vector representing the shape, texture and boundary of the lesion itself. The spatial relationship branch combines the coordinate information of the lesion in the image and the position information of the preset anatomical structure detected from the image to output an image spatial relationship vector representing the relative relationship between the lesion and the preset anatomical structure.

5. The method for processing electronic medical records for thyroid cancer based on multimodal data according to claim 4, characterized in that, The preset anatomical structures include at least one of the thyroid isthmus, thyroid capsule, and thyroid lobe margin; the image spatial relationship vector is based on the normalized relative distance and / or directional angle features between the lesion and at least one of the preset anatomical structures.

6. The method for processing electronic medical records for thyroid cancer based on multimodal data according to claim 1, characterized in that, The location semantic consistency score is calculated by calculating the cosine similarity between the text location semantic vector and the image spatial relationship vector after projection transformation. The morphological neighborhood consistency score is calculated as follows: First, the expected morphological feature vector is decoded from the initial text vector representation output by the pre-trained language model through an independent morphological feature projection layer. Then, the similarity between the expected morphological feature vector and the lesion morphological feature vector is calculated. Finally, the morphological neighborhood consistency score is obtained by weighted fusion of whether the text contains specific morphological keywords and the matching degree of the morphological labels detected in the image.

7. A multimodal data-based electronic medical record processing system for thyroid cancer, characterized in that, include: The image and text data acquisition module is used to acquire thyroid images and associated electronic medical record text descriptions, wherein the text descriptions include location descriptions of the lesions; The text orientation semantic vector extraction module is used to extract text orientation semantic vectors based on the text description, wherein the text orientation semantic vectors represent the relative spatial relationships of the location description information; The image feature vector extraction module is used to extract lesion morphology feature vectors and image spatial relationship vectors based on the thyroid image. The lesion morphology feature vectors represent the visual morphology of the lesion, and the image spatial relationship vectors represent the spatial position of the lesion relative to a preset anatomical structure in the image. The orientation semantic consistency score calculation module is used to calculate the orientation semantic consistency score based on the text orientation semantic vector and the image spatial relationship vector. The orientation semantic consistency score is used to quantify the degree of matching between the spatial orientation of the text description and the spatial location of the lesion in the image. The morphological neighborhood consistency score calculation module is used to calculate the morphological neighborhood consistency score based on the text orientation semantic vector and the lesion morphological feature vector. The morphological neighborhood consistency score is used to quantify the degree of matching between the morphological features described in the text and the visual morphology of the lesion in the image. The comprehensive consistency score calculation module is used to obtain a comprehensive consistency score based on the orientation semantic consistency score and the morphological neighborhood consistency score. The lesion correspondence determination module is used to determine whether the text description corresponds to the lesion in the thyroid ultrasound image by comparing it with the comprehensive consistency score and a preset threshold.

8. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions can be executed by a processor to implement the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images