Target focus identification method, system and device based on medical image, computer storage medium and computer program product

Abstract

The invention provides a target focus identification method, system and device based on a medical image, a computer storage medium and a computer program product, and the method comprises the steps: generating an MPR feature vector and a DICOM metadata feature vector which are aligned in dimension based on medical image coding, the MPR feature vector comprises the form and spatial position information of an anatomical structure, and the DICOM metadata feature vector comprises the form and spatial position information of the anatomical structure; the DICOM metadata feature vector comprises scanning parameters, equipment information and patient body position information; the MPR feature vector and the DICOM metadata feature vector are fused through a bidirectional cross attention mechanism, and a comprehensive feature vector containing lesion semantics and space details is generated; and processing the comprehensive feature vector by using a neural network model based on a Transform architecture, and outputting an identification result of the target focus. By adopting the scheme provided by the invention, the complementation of semantic information such as anatomical morphological characteristics and scanning parameters is realized, and the recognition precision of the target focus is improved.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to methods, systems, devices, computer storage media, and computer program products for identifying target lesions based on medical images. Background Technology

[0002] Currently, commonly used tomographic scanning methods in clinical practice include computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET). During examination, the target lesion of the patient is typically scanned sequentially to obtain a set of images, usually containing dozens to thousands of consecutive tomographic images. These images are stored in the Digital Imaging and Communications in Medicine (DICOM) standard format, containing pixel value information and metadata.

[0003] When identifying target lesions based on these images, they typically rely solely on pixel values ​​from DICOM, lacking guidance from metadata such as scanning parameters, equipment information, and patient positioning. This leads to the misidentification of noise as tissue during lesion identification and the neglect of prior differences in anatomical structures across different locations during localization. Furthermore, using DICOM metadata alone (such as scan site labels) cannot directly impact pixel-level processing, lacking a spatial correspondence with image content. While existing technologies recognize the need to combine both approaches, the lack of an efficient cross-modal feature fusion mechanism results in current image processing accuracy failing to meet clinical needs in complex scenarios such as small lesion detection and multi-organ segmentation.

[0004] Therefore, how to integrate pixel value information and metadata information in DICOM format files to improve the accuracy of target lesion identification by taking into account both semantic guidance (such as prior knowledge of the scanned location) and spatial details (such as pixel features of lesion edges) is an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, system, device, computer storage medium, and computer program product for target lesion identification based on medical images. The ability to integrate pixel value information and metadata information from DICOM format files helps improve the accuracy of target lesion identification based on medical images.

[0006] In a first aspect, embodiments of the present invention provide a method for identifying target lesions based on medical images, comprising: generating dimensionally aligned MPR feature vectors and DICOM metadata feature vectors based on medical image encoding, wherein the MPR feature vectors contain morphological and spatial location information of anatomical structures, and the DICOM metadata feature vectors contain scanning parameters, device information, and patient positioning information; fusing the MPR feature vectors and the DICOM metadata feature vectors through a bidirectional cross-attention mechanism to generate a comprehensive feature vector containing lesion semantics and spatial details; and processing the comprehensive feature vector using a neural network model based on the Transformer architecture to output the identification result of the target lesion.

[0007] This approach generates dimension-aligned MPR feature vectors and DICOM metadata feature vectors, and fuses them using a bidirectional cross-attention mechanism combined with Transformer architecture processing. This solves the problems of easy misjudgment of noise due to reliance on pixel values ​​alone and lack of spatial correspondence in metadata in existing technologies. It achieves complementarity between anatomical morphological features and semantic information such as scanning parameters, improving the recognition accuracy of target lesions. In particular, it can take into account both spatial details and semantic guidance in small lesion detection and multi-organ segmentation.

[0008] In conjunction with the first aspect, in a first possible implementation of the first aspect, the generation of dimension-aligned MPR feature vectors and DICOM metadata feature vectors based on medical image encoding includes: performing multi-plane reconstruction based on medical images to generate an MPR image queue; the MPR image queue includes: coronal MPR images, sagittal MPR images, and transverse MPR images; inputting the MPR image queue into a deep learning model for processing, the deep learning model including: a backbone feature extraction module including convolutional blocks, an attention enhancement module, and an in-plane multi-scale feature fusion module including a feature pyramid network; and sequentially processing the MPR image queue. The process involves: processing by the backbone feature extraction module, the attention enhancement module, and the in-plane multi-scale feature fusion module to obtain multi-scale feature maps corresponding to each MPR image in the MRP image queue; the deep learning model further includes: a multi-plane interaction module and a feature fusion and encoding module; using the multi-plane interaction module to perform cross-plane correlation fusion on the multi-scale feature maps to obtain a multi-plane fused feature map with a two-dimensional structure; using the feature fusion and encoding module to perform global pooling and fully connected encoding on the multi-plane fused feature map to output a one-dimensional MPR feature vector with N elements, where N is an integer and N > 1.

[0009] The above scheme generates an MPR image queue including coronal MPR images, sagittal MPR images, and transverse MPR images through multi-plane reconstruction. Combined with a backbone feature extraction module, an attention enhancement module, and a feature pyramid network, it achieves multi-scale feature fusion and cross-plane correlation, solving the problem that small lesions are easily missed in single transverse images. It enables the model to simultaneously capture the local details of lesions (such as edge texture) and their relationship with the overall anatomical structure, improving the accuracy of lesion localization and feature expression in three-dimensional space.

[0010] In conjunction with the first aspect, in the second possible implementation of the first aspect, the generation of dimension-aligned MPR feature vectors and DICOM metadata feature vectors based on medical image encoding includes: loading a set of DICOM file sequences corresponding to a sequence of medical images obtained from a tomographic scan, and arranging them according to the scan order to obtain an initial image file queue; reading DICOM files from a specified position in the initial image file queue to form a key image file queue of fixed length; reading sequence-level DICOM metadata and image-level DICOM metadata representing key features from the key image file queue, arranging them according to a preset rule to obtain a sequence DICOM metadata queue of fixed length and fixed information items at each position; the key features include: examination type, scan parameters, scan site, and patient information; inputting the sequence DICOM metadata queue into a deep learning network, the deep learning network including a feature preprocessing and classification layer, a feature encoding layer, a feature fusion layer, a context modeling layer, and an output layer; triggering the deep learning network to transform the sequence DICOM metadata queue into a one-dimensional DICOM metadata feature vector with M elements, where M is a subset of M, and M > 1.

[0011] The above scheme extracts key images and encodes metadata in DICOM file sequences, transforming unstructured metadata such as scan parameters and patient information into structured feature vectors. This provides prior semantic guidance for image features, solves the problem that metadata cannot be directly applied to pixel-level processing, and enables the model to use scan parameters to calibrate image features. It can also combine metadata such as patient medical history to optimize the probability judgment of target lesion identification.

[0012] In conjunction with the first aspect, in the third possible implementation of the first aspect, when M≠N, before generating the dimension-aligned MPR feature vector and DICOM metadata feature vector based on medical image encoding, the target lesion identification method further includes: performing layer normalization, standardizing feature distribution, and projecting the MPR feature vector and DICOM metadata feature vector to the same dimensional space, respectively, to obtain the dimension-aligned MPR feature vector and DICOM metadata feature vector.

[0013] The above scheme achieves dimensional alignment between MPR and DICOM features through layer normalization, standardized feature distribution, and linear projection, eliminating scale differences between features of different modalities, ensuring efficient interaction of features in a unified semantic space, solving the problem of information loss caused by dimensional mismatch during cross-modal fusion, and improving the stability of model training and the generalization ability of multi-center data.

[0014] In conjunction with the first aspect, in the fourth possible implementation of the first aspect, the bidirectional cross-attention mechanism includes: using the MPR feature vector as the query and the DICOM metadata feature vector as the key and value, generating an MPR attention output guided by metadata semantics to suppress noise interference on lesion edges; and using the DICOM metadata feature vector as the query and the MPR feature vector as the key and value, generating a DICOM attention output that integrates anatomical details to locate the spatial position of the lesion.

[0015] The above scheme adopts a bidirectional cross-attention mechanism, using MPR features to query DICOM metadata to achieve noise suppression, and using DICOM metadata to query MPR features to achieve spatial localization, forming a bidirectional guidance between semantics and space. This solves the problem of lack of dynamic interaction in cross-modal fusion in existing technologies. Especially in noisy scenarios such as low-dose CT, it can accurately locate lesions and suppress artifact interference, thereby improving the robustness of recognition.

[0016] In conjunction with the first aspect, in a fifth possible implementation of the first aspect, the fusion includes: Dynamic weights are generated through a feature gating layer to adaptively adjust the contribution of MPR features and DICOM metadata features. Specifically, when identifying lesions smaller than a preset threshold, the spatial detail weight of MPR features is enhanced, and the scanning parameter weight of DICOM metadata is enhanced in noise-sensitive scenarios.

[0017] The above scheme generates dynamic weights through a feature gating layer, and adaptively adjusts the contribution of MPR and DICOM features according to lesion size and noise scene. It strengthens spatial detail features when identifying small lesions and strengthens scanning parameter features in noise-sensitive scenes. This solves the problem that traditional fusion methods cannot adaptively handle different scenes, realizes intelligent optimization of feature fusion, and further improves the lesion detection accuracy under complex working conditions.

[0018] Secondly, embodiments of the present invention provide a target lesion identification system based on medical images, comprising: an encoding module, a fusion module, and an identification module. The encoding module is used to generate dimensionally aligned MPR feature vectors and DICOM metadata feature vectors based on medical image encoding. The MPR feature vectors contain morphological and spatial location information of anatomical structures, and the DICOM metadata feature vectors contain scanning parameters, device information, and patient positioning information. The fusion module is used to fuse the MPR feature vectors and the DICOM metadata feature vectors through a bidirectional cross-attention mechanism to generate a comprehensive feature vector containing lesion semantics and spatial details. The identification module is used to process the comprehensive feature vector using a neural network model based on the Transformer architecture to output the identification result of the target lesion.

[0019] Thirdly, embodiments of the present invention provide a target lesion identification device based on medical images. The device includes: a processor configured to execute computer-executable instructions; and a memory storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by the processor, they implement the various steps of the target lesion identification method based on medical images as described in the first aspect or any possible implementation of the first aspect.

[0020] Fourthly, embodiments of the present invention provide a computer storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the target lesion identification method based on medical images as described in the first aspect or any possible implementation thereof.

[0021] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the target lesion identification method based on medical images as described in the first aspect or any possible implementation of the first aspect.

[0022] In a sixth aspect, embodiments of the present invention provide a chip system applied to an electronic device. The chip system includes one or more processors, which are configured to invoke computer instructions to cause the electronic device to perform a target lesion identification method based on medical images as described in the first aspect or any possible implementation thereof.

[0023] It is understood that the technical effects achieved by the target lesion identification system based on medical images described in the second aspect, the target lesion identification device based on medical images described in the third aspect, the computer storage medium described in the fourth aspect, the computer program product described in the fifth aspect, and the chip system described in the sixth aspect are similar to the technical effects achieved by the corresponding technical means in the target lesion identification method based on medical images described in the first aspect, and will not be elaborated here. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a method for identifying target lesions based on medical images, provided in an embodiment of the present invention. Figure 2A This is a schematic diagram of the process for obtaining MPR feature vectors based on medical images according to an embodiment of the present invention; Figure 2B This is a schematic diagram of the deep learning module architecture when obtaining MRP feature vectors according to an embodiment of the present invention; Figure 3A This is a schematic diagram of the process for obtaining the DICOM metadata queue according to an embodiment of the present invention; Figure 3B This is a schematic diagram of the process for obtaining DICOM metadata feature vectors according to an embodiment of the present invention; Figure 4A This is a schematic diagram of a model architecture for fusing MPR feature vectors and DICOM metadata feature vectors according to an embodiment of the present invention; Figure 4B yes Figure 4A It includes a clear diagram of the areas corresponding to labels 401, 402, and 403; Figure 4C yes Figure 4A It includes a clear diagram of the areas corresponding to labels 404, 405, and 406; Figure 4D yes Figure 4A It includes a clear diagram of the area indicated by the 407 label; Figure 5A This is a schematic diagram of the structure of a neural network model based on the Transformer architecture in one embodiment of the present invention; Figure 5B yes Figure 5A The document includes a clear diagram of the area corresponding to identifier 501; Figure 5C yes Figure 5A A clear diagram showing the areas corresponding to markings 5021 / 5022 / 5023; Figure 5D yes Figure 5AA clear diagram showing the areas corresponding to markings 503, 504, and 505. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0026] It should be understood that the term "multiple" in this invention refers to two or more. In the description of this invention, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply differences.

[0027] With the continuous development of medical imaging technology, CT, MRI, PET and other computed tomographic medical images have been widely used in clinical diagnosis. These imaging examinations generally use DICOM files for data transmission and storage.

[0028] DICOM is an international standard (ISO 12052) in the field of medical imaging, which defines the format, transmission protocol and storage architecture of medical imaging data, and is widely used in the storage, transmission and sharing of medical imaging information.

[0029] According to the DICOM standard, a single patient imaging examination corresponds to a "Study" object, which typically contains one or more Series objects, each of which usually includes multiple Image objects. Conventional sequences of computed tomography (CT), MRI, PET, and other medical imaging techniques are characterized by continuous scanning of specific anatomical regions of the human body, acquiring a set of images with highly consistent spatial location and scanning direction, conforming to specific patterns. Through advanced 3D reconstruction algorithms, these 2D images can be precisely reconstructed into 3D volumetric images, providing more comprehensive and three-dimensional imaging information for precision medical diagnosis.

[0030] Different sequences correspond to different scanning sites or imaging parameters. In the storage, retrieval, processing, and analysis of medical images, when there are multiple sequences, it is crucial to process them one by one and accurately identify the scanning site (such as the head, chest, abdomen, pelvis, etc.) corresponding to each sequence.

[0031] Currently, the identification of scanned areas mainly relies on relevant label information in the DICOM header file, such as study descriptions and series descriptions. However, this descriptive information often lacks standardization due to differences between hospitals, equipment manufacturers, or operators, and may even contain erroneous information, resulting in low accuracy in scanned area identification. Furthermore, relying solely on these text labels for identification cannot fully utilize the rich visual information contained in the images themselves.

[0032] The DICOM file format includes a header information section. This section conforms to the DICOM standard and uses a standard tag list to record and store additional information for each image file, i.e., metadata information, including the examination type, scan parameters, and scanned area. The DICOM header information, or metadata, for different images in a sequence consists of two parts: one part is information common to all images in the sequence, i.e., the parts that are the same for all images in the sequence; the other part is information unique to each image, i.e., each image is different. This differing information is mainly spatial location information, usually arranged and changing according to certain rules, such as an arithmetic progression.

[0033] A sequence of images contains a collection of DICOM header information, or metadata, which includes a wealth of examination-related information such as examination type, scan parameters, scan site, and slice spatial location. If this information can be transformed into feature vectors that neural networks can receive and process, especially fixed-length one-dimensional feature vectors, it will make intelligent processing of images within the sequence (such as disease detection and lesion identification) possible.

[0034] Since a sequence may contain hundreds or even thousands of images, directly selecting the DICOM header information set of all images as the feature vector would result in an excessively long feature vector containing a large amount of redundant information. Furthermore, the variable number of images in a sequence also leads to variable feature vector lengths. Deep learning networks, due to their inherent characteristics, require, on the one hand, input feature vectors to be as concise as possible without redundant information, and on the other hand, to have a fixed length. Therefore, current technology limits the processing of DICOM files by deep learning networks.

[0035] Furthermore, traditional techniques, which analyze and extract features image by image, face significant challenges, including massive computational demands, redundant feature extraction, and a lack of necessary spatial relationships between features. This makes it difficult for existing methods to quickly process sequential data and effectively integrate spatial correlation information between images, thus hindering the efficiency and accuracy of applications such as intelligent medical image analysis and large-scale population AI-based automated screening.

[0036] Therefore, there is an urgent need for a method that can efficiently process the entire sequence and extract a concise method containing pixel value features, spatial relationship features, and DICOM metadata. Based on the above situation, this invention provides a method for processing DICOM file sequences, a medical image analysis method, a system, an apparatus, a computer storage medium, and a computer program product. It aims to provide a technical solution that effectively addresses the aforementioned problems. By removing redundant information and establishing a standardized data processing mechanism to unify feature vector lengths, it achieves the requirement of concise and fixed-dimensional input data to adapt to deep learning networks. This breaks through the bottlenecks of traditional technologies and meets the clinical need for rapid and accurate diagnosis.

[0037] The present invention provides a method, system, device, computer storage medium, and computer program product for target lesion identification based on medical images. It can integrate pixel value information and metadata information from DICOM format files, which helps improve the accuracy of target lesion identification based on medical images.

[0038] Figure 1 This is a flowchart illustrating a target lesion identification method based on medical images according to an embodiment of the present invention, as shown below. Figure 1 As shown, the target lesion identification method may include steps S101 to S103.

[0039] S101. Generate dimension-aligned MPR feature vectors and DICOM metadata feature vectors based on medical image coding. The MPR feature vectors contain morphological and spatial location information of anatomical structures, while the DICOM metadata feature vectors contain scanning parameters, equipment information, and patient positioning information.

[0040] In some possible implementations, the process of generating MRP feature vectors based on medical image encoding is as follows: Figure 2A As shown, the process includes: multiplanar reconstruction based on medical images, first generating an MPR image queue; the MPR image queue includes: coronal MPR images, sagittal MPR images, and transverse MPR images; then the MPR image queue is input into a deep learning model for processing, as shown in the deep learning model... Figure 2BAs shown in Figure 2, the process includes: a backbone feature extraction module with convolutional blocks, an attention enhancement module, and an in-plane multi-scale feature fusion module with a feature pyramid network. After the MPR image queue is processed sequentially through the backbone feature extraction module, the attention enhancement module, and the in-plane multi-scale feature fusion module, multi-scale feature maps corresponding to each MPR image in the MPR image queue are obtained. The deep learning model also includes: a multi-plane interaction module and a feature fusion and encoding module. The multi-plane interaction module performs cross-plane correlation fusion on the multi-scale feature maps to obtain a multi-plane fused feature map with a two-dimensional structure. The feature fusion and encoding module performs global pooling and fully connected encoding on the multi-plane fused feature map, outputting a one-dimensional MPR feature vector with N elements, where N is an integer (N > 1). Specifically, as shown in Figure 2, the process of obtaining MPR feature vectors based on medical images may include the following steps: S201 to S206.

[0041] S201. Acquire raw DICOM data based on computed tomography (CT) technology. S202. Construct three-dimensional volumetric data based on the raw DICOM data. S203. Perform multi-planar reconstruction based on the three-dimensional volumetric data to generate an MPR image queue.

[0042] The MPR image queue includes at least: coronal MPR images, sagittal MPR images, and transverse MPR images; for ease of description, as shown in Figure 2, this application describes the MPR image queue as including: coronal MPR images, sagittal MPR images, and transverse MPR images.

[0043] S204. Input the MPR image queue into the deep learning model for processing. The MPR image queue sequentially passes through the backbone feature extraction module, attention enhancement module, and in-plane multi-scale feature fusion module to obtain the multi-scale feature map corresponding to each MPR image in the MPR image queue. The processing flow is as follows: Figure 2B As shown.

[0044] The deep learning model includes: a backbone feature extraction module with convolutional blocks, an attention enhancement module, and an in-plane multi-scale feature fusion module, a multi-plane interaction module, and a feature fusion and encoding module, which include a feature pyramid network. In some possible implementations, the attention module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule weights channel features using global average pooling and a multilayer perceptron, while the spatial attention submodule enhances spatial features using convolution and a non-linear activation function. The non-linear activation function can be any of the following: Sigmoid, ReLU, or Swish.

[0045] S205. Utilize the multi-plane interaction module to perform cross-plane association and fusion of multi-scale feature maps to obtain a multi-plane fused feature map with a two-dimensional structure.

[0046] In some possible implementations, the multi-plane interaction module calculates the similarity matrix between the multi-scale feature maps corresponding to each MPR image in the MPR image queue, and adjusts the attention weight values ​​of the collaborative diagnostic features based on the similarity matrix and according to the preset attention weight enhancement rules.

[0047] S206. Use the feature fusion and encoding module to perform global pooling and fully connected encoding on the feature map fused by the multi-plane fusion, and output a one-dimensional MPR feature vector with N elements, where N is an integer and N>1.

[0048] In some possible implementations, the process of obtaining DICOM metadata feature vectors based on medical images is as follows: Figure 3A As shown, it may include steps S311 to S315.

[0049] S311. Based on medical images, load the DICOM file sequence corresponding to the image set belonging to a sequence obtained from tomographic scanning, and arrange them according to the scanning order to obtain the initial image file queue.

[0050] In some possible implementations, DICOM files belonging to the same scan sequence can be sorted according to the scan order to form an ordered initial image queue. It should be noted that in some possible implementations, when there are multiple sequences, each sequence can be processed according to the method provided in the embodiments of the present invention.

[0051] S312. Read DICOM files from a specified position in the initial image file queue to form a fixed-length key image file queue.

[0052] In some possible implementations, the specified position can be the first file, a middle file, or the last file in the initial image file queue. In other possible implementations, the specified position can also be the first file or the last file in the initial image file queue. All are feasible, and this invention does not limit this. As an example, this embodiment of the invention describes a scenario where the specified position is the first file, a middle file, or the last file in the initial image file queue, where the key image file is the image file corresponding to the specified position.

[0053] S313. Read the sequence-level DICOM metadata and image-level DICOM metadata used to represent key features from the key image file queue, arrange them according to preset rules, and obtain a sequence DICOM metadata queue with fixed length and fixed information items at each position.

[0054] In some possible implementations, a sequence of DICOM metadata queues with fixed length and fixed information items at each position is obtained, including: filling the sequence of DICOM metadata queues with tag values ​​from the read DICOM file; when a tag value is missing, filling it with "" if the data type corresponding to the tag value is text; filling it with NaN if the data type corresponding to the tag value is numeric; filling it with "" if the data type corresponding to the tag value is enumeration; and normalizing the data type corresponding to the tag value if the data type corresponding to the tag value is a value within a continuous range.

[0055] The process of constructing a sequence DICOM metadata queue can include: constructing a vector V with an initial length of 0. Specifically, the sequence DICOM metadata queue is initialized, constructing a sequence DICOM metadata queue with an initial length of 0. Then, the length of the initial image file queue is written to the starting position of vector V; then, the sequence-level tag of the first image file in the key image file queue (it should be noted that any other image file in the key image queue is also feasible) is read, padded and normalized, and the processed sequence-level DICOM tag value is added to the end of vector V. The number of key image files is P. The sequence-level DICOM tag value of the first image in the key image file queue is read, the tag value is padded and normalized, and added to the end of the sequence DICOM metadata queue. Then, image-level DICOM metadata is added: starting from the first image file in the key image file queue, the image-level DICOM tag value of each image file in the key image file queue is read sequentially, the tag value is padded and normalized, and saved sequentially to the end of the sequence DICOM metadata queue, thus obtaining the sequence DICOM metadata queue.

[0056] S314. Input the sequence DICOM metadata queue into the deep learning network.

[0057] like Figure 3B As shown, the input layer obtains the sequence DICOM metadata queue, and the deep learning network includes: feature preprocessing and classification layer, feature encoding layer, feature fusion layer, context modeling layer and output layer.

[0058] S315. Trigger the deep learning network to transform the sequence DICOM metadata queue into a one-dimensional DICOM metadata feature vector with M elements, where M is an integer and M > 1.

[0059] This step aims to convert the sequence DICOM metadata queue into a high-dimensional feature vector, effectively capturing the complex patterns and relationships in the DICOM metadata through a deep learning network, thereby generating a DICOM metadata feature representation for subsequent analysis.

[0060] The DICOM metadata encoder employs a deep neural network based on the Transformer architecture, specifically designed to handle the heterogeneous characteristics of medical image DICOM metadata. This encoder can simultaneously process categorical, continuous, textual, and missing value features, generating feature vectors that highly represent sequential DICOM metadata information through multi-level feature extraction and self-attention mechanisms.

[0061] The final output DICOM metadata feature vector is a high-dimensional vector (e.g., 256-dimensional) containing rich semantic information of the sequence DICOM metadata. It can effectively represent key characteristics of medical images such as scanning parameters, equipment information, and patient location, laying the foundation for subsequent medical image analysis.

[0062] This feature vector will serve as an important component of multimodal feature fusion, and will be fused with image information from medical images, enabling its use in scenarios such as automatic identification of scanned areas.

[0063] The following is combined with Figure 3B The functions and processing procedures of each layer in a deep learning network are explained one by one.

[0064] (1) Preprocessing and classification layer The input sequence DICOM metadata queue is first classified and processed according to data type. Categorical features (such as scan parameters, device type, etc.) are transformed through one-hot encoding or embedding technology; continuous features (such as pixel values, scan spacing, etc.) are processed using standardization or Min-Max normalization; textual features (such as inspection description, sequence name, etc.) are processed using sequence encoding methods; missing values ​​are processed using special masking to preserve the semantic meaning of missing information.

[0065] (2) Feature coding layer Categorical features are converted into dense vector representations through an embedding layer (nn.Embedding). Continuous features are processed by linear projection and batch normalization layers to adjust the dimensionality and numerical distribution. Textual features are extracted using a dedicated Transformer encoder to extract semantic information. Masked features are encoded with positional markers to ensure that the model can identify the location and pattern of missing values.

[0066] (3) Feature fusion layer The encoded features of each type are concatenated to form a complete metadata representation. A multi-head attention mechanism is applied to enhance the interaction between features. The attention weights are dynamically adjusted to adjust the importance of each metadata field, highlighting key information related to the identification of the scanned part.

[0067] It should be noted that the preprocessing steps in the preceding steps are used to transform the data into "data in a uniform format." For example, DICOM metadata contains different types of data, among which... Continuous data (e.g., patient age 35 years, scan slice thickness 5 mm) are normalized to the 0-1 range.

[0068] Categorical data (such as gender "male" or equipment type "CT") is encoded using one-hot encoding into numeric vectors such as [1,0] or [0,1].

[0069] Textual data (such as the sequence description "head CT scan") is transformed into a digital vector containing core semantics using the Transformer model.

[0070] Missing values ​​(such as unrecorded contrast agent doses) are indicated to the model with a special label (such as [NaN]) to tell the model "no data here".

[0071] Concatenation can be used to combine processed vectors such as age, gender, device type, and text description in sequence to form a long "feature mosaic". For example: [0.6 (normalized age), 1, 0 (male gender), 0, 1, 0 (CT device), 0.3, 0.7, ... (text semantic vector)] Multi-head self-attention: allowing "multiple experts" to analyze the importance of data. For example, if there are 3 experts, expert 1 focuses on the relationship between "age" and "scanned area": ​​for example, there are more head scans of children, and the weight of the "age" feature will be higher in samples of younger children.

[0072] Expert 2 focuses on the correlation between "equipment type" and "location": CT is often used for the chest, while MRI is often used for the brain, so the characteristics of "equipment CT" have higher weight in chest scans.

[0073] Expert 3 focuses on keywords in the "text description": if the text contains a "header", the weight of "header" related features is increased.

[0074] After each expert conducts an independent analysis, they assign an "importance score" (weight) to each feature. Finally, the opinions of all experts are combined to obtain the final weight distribution.

[0075] The following description uses feature weight adjustment for chest CT scans as an example to illustrate the aforementioned steps. Assume that the DICOM metadata contains the following fields: Age: 50 years old (normalized to 0.5) Gender: Male (One-hot encoding [1,0]) Equipment type: CT (One-hot encoding [0,1,0], assuming there are three types: CT / MRI / PET) Sequence description: "Chest CT Scan" (after processing by Transformer, a semantic vector is obtained, such as [0.8,0.2,0.1,...], where 0.8 represents the semantic meaning of "chest") Layer thickness: 5mm (0.3 after normalization).

[0076] After concatenation, the resulting feature vector is: [0.5,1,0,0,1,0,0.8,0.2,0.1,0.3] (10 dimensions in total). The "expert analysis" process for bullish self-attention includes: Expert 1 (Focusing on age + device): It was found that the probability of chest scans was high among people aged 50 and above using CT equipment, so the weights of "age 0.5" and "CT equipment" were each increased by 0.2.

[0077] Expert 2 (Focusing on text keywords): The keyword "Chest" is directly associated with the chest, so the weight of "0.8 in the text semantic vector" is increased by 0.3.

[0078] Expert 3 (Focusing on layer thickness and location): Chest CT scans commonly use a 5mm slice thickness, so we assign a weight of +0.1 to "slice thickness 0.3".

[0079] Final weight distribution: Age: 0.5 (original value) × 1.2 (weight) = 0.6 CT equipment: 1 (1 in One-hot) × 1.2 = 1.2 The semantic value of the text "chest": 0.8 × 1.3 = 1.04 Other feature weights remain unchanged (such as gender, layer thickness, etc.).

[0080] The significance of multiple heads is understandable: different "heads" can capture correlations across different dimensions (e.g., age-location, text-location, device-location), avoiding information omissions from a single perspective. The model learns through training which feature combinations are most useful for identifying the chest / head / abdomen, and dynamic weights automatically enhance the influence of key features while suppressing irrelevant features (e.g., gender has little impact on chest scans, so its weight can remain unchanged). Compared to traditional methods, which typically rely on manual settings to prioritize text descriptions, the self-attention mechanism allows the model to discover that "sometimes device type is more reliable than text" (e.g., even with an incorrect text description, the CT scanner can still correctly point to the chest).

[0081] (4) Context modeling layer The first feedforward network layer uses linear transformation and GELU activation function to extract high-level feature patterns. Layer normalization maintains training stability and accelerates convergence. The second feedforward network layer further extracts abstract features and applies layer normalization again to ensure stable feature distribution.

[0082] Linear transformation is used to convert data of different units / types into a common unit. For example, if the input data includes "age 50 years old" (numerical), "equipment type CT" (categorical, one-hot encoded as [0,1,0]), and "slice thickness 5mm" (numerical), linear transformation will use mathematical formulas to convert these data into vectors of the same dimension (e.g., all converted to numbers in the range of 0-10). For instance, it converts "50 years old," "CT," and "5mm" into a format suitable for subsequent processing (e.g., age → 0.5, CT → [0,1,0] → 2, slice thickness → 0.5).

[0083] The GELU activation function is used to automatically identify and enhance key data features. Through the formula \(GELU(x)=x\cdot\Phi(x)\) (where \(\Phi(x)\) is the cumulative distribution function of a normal distribution), large numerical features are given higher weights, while small numerical features are weakened or even "filtered out". For example, if the converted value of "equipment type CT" is 2 (relatively large), GELU will make it 2.5 (enhanced); "age 0.5" is relatively small and can remain at 0.5 or be slightly reduced; irrelevant features (such as gender) may be weakened to 0.1.

[0084] The second feedforward network layer is used for abstract feature extraction. It performs "secondary processing" on the data processed in the first stage to extract more complex correlation features. For example, if the first stage obtains "age 0.5", "CT equipment 2.5", and "slice thickness 0.5", the second feedforward network will calculate the combined feature of "CT equipment + slice thickness 5mm" (e.g., 2.5 + 0.5 = 3), find that this is a typical parameter of chest CT, and then generate the abstract feature of "probability of chest CT".

[0085] Layer normalization, used for "standardized calibration" of data, has two main components. The first layer normalization is applied after the first feedforward network to ensure consistent data distribution across different samples within the same batch. For example, if a batch contains 100 samples, each with features such as age, device, and slice thickness, layer normalization calculates the mean and variance of each feature, adjusting the data to a range where the mean is 0 and the variance is 1. The second layer normalization is applied after the second feedforward network to ensure consistent data distribution across different batches, avoiding training fluctuations. For instance, if today's samples are primarily chest CT scans (mean age 45), and tomorrow's samples are primarily head MRI scans (mean age 30), layer normalization will adjust the data for both days to the same distribution range, preventing the model from "learning skewed" due to changes in data distribution.

[0086] The corresponding context modeling layer will be illustrated using the processing of chest CT data as an example. The input data includes: Age: 50 years old (original value) → 0.5 after linear transformation → 0.5 after GELU activation (normal feature, no enhancement); Equipment type: CT (One-hot encoding [0,1,0]) → 2 after linear transformation → 2.5 after GELU activation (key features, enhancement); Slice thickness: 5mm → 0.5 after linear transformation → 0.5 after GELU activation (common features); First feedforward network output: [0.5, 2.5, 0.5] → after layer normalization → [0, 1, 0] (assuming the mean of this batch is 1 and the variance is 1); Second feedforward network processing: calculate the combined feature of "equipment CT + slice thickness 5mm": 1 + 0 = 1 → generate abstract feature "chest CT possible" = 0.8; Second layer normalization: If the mean of this feature in the next batch is 0.6 and the variance is 0.2, then adjust it to: ((0.8-0.6)0.2=1) to ensure stable distribution.

[0087] (5) Output layer Global feature pooling summarizes the entire sequence information to generate a fixed-dimensional representation. A fully connected layer adjusts the feature dimension to the required size, batch normalization ensures stable feature distribution, and a Dropout layer (p=0.1) prevents overfitting. Finally, the DICOM metadata feature vector is output with a preset dimension (e.g., 256 or 512).

[0088] The output layer acts as a "packaging stage in the feature processing plant," with pooling used to refine overall features; a fully connected layer standardizes the packaging; batch normalization ensures stable quality; and Dropout enhances anti-interference capabilities. This process accurately identifies medical image scanning areas from different hospitals and using different equipment.

[0089] Global feature pooling is like taking a "family photo" of features. It compresses all features of an entire sequence (such as age, device type, text description, etc.) into a "comprehensive feature package," much like taking a group photo of a group of people, retaining only the overall impression rather than the details of each individual. The input features are 100-dimensional vectors. For example, given 100 features such as "age 50 years old," "device CT," and "slice thickness 5mm," global pooling calculates the average or maximum value of these features, generating a 1-dimensional "comprehensive feature." An analogy: If a class has 50 students, pooling is equivalent to calculating the average height of the entire class, using a single number to represent the overall height feature of the class.

[0090] Fully connected layers are similar to "boxing" features uniformly. Their function includes converting all input features, regardless of their dimensionality, into a fixed dimension (e.g., 256 dimensions), facilitating subsequent processing. For example, if pooling yields 10-dimensional features, a fully connected layer uses mathematical formulas to "stretch" them into 256 dimensions. For instance, 10-dimensional data such as "age + device + slice thickness" can be converted into 256 values, each representing an abstract feature (e.g., "probability of chest scan" or "correlation with CT equipment"). This is analogous to using a cookie cutter; no matter the size of the dough, the resulting cookie will always be a fixed shape.

[0091] Batch normalization is similar to "standardizing" the flavor of features. Its functions include ensuring that the feature distribution of each batch of data is consistent, preventing the model from "learning skewed" due to data fluctuations. For example, if the first batch of data mainly consists of chest CT scans of middle-aged people (mean age 50) and the second batch mainly consists of head MRI scans of young people (mean age 30), batch normalization will adjust the age features to a range with a mean of 0 and a variance of 1. For example, it calculates the mean (e.g., 40 years old) and variance of the "age" feature in a batch of data; it converts each age value into "how far it is from the mean" (e.g., 50 years old → (50-40) / standard deviation, 30 years old → (30-40) / standard deviation); the adjusted age feature distribution is like being "standardized" into the same bowl of soup, with a consistent flavor.

[0092] Dropout layers are like playing "hide-and-seek" with features. Their function includes randomly hiding 10% of the features (p=0.1) when p=0.1, preventing the model from over-relying on certain features. For example, during training, the feature "CT scan equipment" can be intentionally hidden, forcing the model to use other features such as "slice thickness" and "text description" to identify the body part, preventing the model from assuming that "any CT scan indicates a chest area." For instance, one implementation might be: assuming the input has 256 features, during each training iteration, 26 features (256 × 0.1 ≈ 26) are randomly selected and set to 0, while the remaining features are amplified by a factor of 1 / 0.9 ≈ 1.11. For example, the feature value of "CT scan equipment" is 0.9, which might be randomly set to 0, while the feature value of "slice thickness" is 0.7, which is amplified to 0.7 × 1.11 ≈ 0.78.

[0093] The following describes the processing flow of the output layer for a chest CT scan, using the processing steps in the output layer as an example. Input features (before pooling): [0.8 (text "chest"), 0.9 (CT equipment), 0.5 (age 50 years), 0.7 (slice thickness 5mm), ... (total 100 dimensions)]; Global average pooling: Calculate the average value: (0.8 + 0.9 + 0.5 + 0.7 + ...) / 100 = 0.65 (generating a 1-dimensional comprehensive feature); Then, fully connected layer (converted to 256): Through matrix calculation, the 1-dimensional feature is transformed into 256 dimensions, for example: [0.2 (chest probability), 0.7 (CT correlation), 0.1 (year)]. [Age influence),...,0.5 (slice thickness weight)]; Batch normalization: If the mean of the batch data is 0.4 and the variance is 0.2, then "chest probability 0.2" is adjusted to (0.2-0.4) / 0.2=-1, and "CT correlation 0.7" is adjusted to (0.7-0.4) / 0.2=1.5; Dropout (p=0.1): Randomly select 26 features and set them to 0, such as "age influence 0.1" being set to 0, and "slice thickness weight 0.5" being amplified to 0.5×1.11≈0.55; Output result: Finally, a 256-dimensional DICOM metadata feature vector is obtained, which can be used for subsequent fusion with image features to identify the body part.

[0094] In some possible implementations, it can be achieved by Figure 4A The input layer obtains a one-dimensional MPR feature vector with N elements and a one-dimensional DICOM metadata feature vector with M elements. When M≠N, it can utilize... Figure 4AThe preprocessing layer in the process involves layer normalization, standardization of feature distribution, and projection onto the same dimensional space for both the MPR feature vector and the DICOM metadata feature vector, resulting in dimension-aligned MPR feature vectors and DICOM metadata feature vectors. A simplified example illustrates the processing flow: assume we have a one-dimensional MPR feature vector with 5 elements [1.2, 2.5, 3.1, 0.8, 4.0], and a one-dimensional DICOM metadata feature vector with 3 elements [5.0, 2.0, 3.5]. First, layer normalization is performed. For the MPR feature vector, its mean is calculated as (1.2+2.5+3.1+0.8+4.0) / 5=2.32, and its variance is [(1.2-2.32)^2+(2.5-2.32)^2+(3.1-2.32)^2+(0.8-2.32)^2+(4.0-2.32)^2] / 5≈1.42. After normalization, we get [(1.2-2.32) / sqrt(1.42),(2.5-2.32) / sqrt(1.42),(3.1-2.32) / sqrt(1.42),(0.8-2.32) / sqrt(1.42),(4.0 ... The variance is approximately [(5.0-2.32) / sqrt(1.42)]≈[-0.93,0.15,0.65,-1.26,1.40]. For the DICOM metadata feature vector, the mean is (5.0+2.0+3.5) / 3=3.5, and the variance is [(5.0-3.5)^2+(2.0-3.5)^2+(3.5-3.5)^2] / 3=1.5. After normalization, we get [(5.0-3.5) / sqrt(1.5),(2.0-3.5) / sqrt(1.5),(3.5-3.5) / sqrt(1.5)]≈[1.22,-1.22,0], completing the standardized feature distribution. Suppose we want to project these two vectors into a two-dimensional space. Principal component analysis or linear transformation can be used to project the normalized MPR feature vector and DICOM metadata feature vector. Projecting the MPR feature vector [-0.93, 0.15, 0.65, -1.26, 1.40] into two dimensions may yield [-0.3, 0.7], and projecting the DICOM metadata feature vector [1.22, -1.22, 0] may yield [0.5, -0.8]. This results in dimension-aligned MPR and DICOM metadata feature vectors, which are then used for further processing.

[0095] Understandably, the input layer, as the starting point for system data inflow, is responsible for collecting raw input data, receiving MPR feature vectors (including lesion morphology, structure, and other information extracted from multi-plane reconstructed images) and DICOM metadata feature vectors (patient age, scan parameters, medical history, and other clinical semantic information), providing basic input for subsequent multimodal feature collaborative processing.

[0096] For example, in the scenario of lung disease diagnosis, the MPR feature vector received by the input layer may include dimensions such as "nodule edge curvature (value 0.8, representing irregular edge)" and "mean nodule density (value -500HU, indicating soft tissue density)"; the DICOM metadata feature vector includes information such as "patient age (e.g., value 65 years)" and "number of years of smoking history (value: 30 years)". Both types of features serve as the initial material for subsequent analysis.

[0097] The feature preprocessing layer 402 performs standardization and dimension alignment operations on the MPR and DICOM feature vectors, respectively. When the MPR and DICOM feature vectors have different numbers of elements and are dimensionally misaligned, layer normalization, standardization of feature distribution, and projection onto the same dimensional space are performed on the MPR and DICOM metadata feature vectors, respectively, to obtain dimension-aligned MPR and DICOM metadata feature vectors. Specifically, layer normalization (LayerNorm) eliminates the distribution differences between feature dimensions within a single sample (such as differences in numerical range, mean, and variance of different dimensions). Standardization adjusts the mean and variance of each feature dimension in the entire dataset, eliminating global distribution differences between feature dimensions. Then, a linear projection layer maps the two types of features with different dimensions to the same dimensional space (e.g., unified to 256 dimensions), enabling multimodal features to have the basic conditions for subsequent cross-modal interaction. For example, suppose the original MPR feature vector is 128-dimensional, and the values ​​of each dimension are scattered due to differences in image reconstruction algorithms (e.g., "nodule diameter" is 0-5cm, "edge smoothness" is 0-1); the DICOM metadata is 64-dimensional, with "age" ranging from 0-100 and "radiation dose" ranging from 0-500mGy. After layer normalization, the mean of each dimension of the MPR feature approaches 0 and the variance approaches 1, and the same applies to the DICOM feature; during linear projection, MPR is converted into a 256-dimensional vector through a 128×256 weight matrix, and DICOM is converted through a 64×256 weight matrix.

[0098] It should be noted that, Figure 4B , Figure 4C and Figure 4D They are Figure 4A Part of it, because Figure 4A There's a lot of content, and when displayed on the same page, the images aren't clear enough. Figure 4B , Figure 4C and Figure 4D For reference Figure 4A The specific details.

[0099] S102. By fusing MPR feature vectors and DICOM metadata feature vectors through a bidirectional cross-attention mechanism, a comprehensive feature vector containing lesion semantics and spatial details is generated.

[0100] The cross-attention layer 403 constructs a bidirectional cross-attention mechanism, enabling MPR and DICOM features to engage in bidirectional "query-association-fusion". In the first stage, MPR features are used as the query and DICOM features as the key and value. Scaled Dot-Product Attention is used to calculate feature association weights, uncovering the focus of MPR spatial features on DICOM semantic information. In the second stage, DICOM features are used as the query and MPR features as the key and value, enabling DICOM semantics to guide MPR spatial features, completing the bidirectional flow and initial fusion of cross-modal information. For example, if the MPR feature "irregular nodule margin (value 0.8)" is used as a query, and its dot product is calculated with DICOM features such as "age > 60 years (value 0.7)" and "smoking history ≥ 20 years (value 0.9)", then after scaling (dividing by the square root of 256 to obtain a value of 16) and using Softmax to obtain weights (e.g., "age" weight 0.4, "smoking history" weight 0.6), the DICOM values ​​are weighted and summed to allow the MPR feature to incorporate the clinical semantics of "advanced age + long-term smoking". Similarly, if the DICOM feature "family history of lung cancer (value 0.8)" is used as a query, and associated with the MPR features "nodule lobulation sign (value 0.7)" and "spiculation sign (value 0.6)", after weighted fusion, the DICOM feature incorporates spatial morphological information.

[0101] In some possible implementations, the bidirectional cross-attention mechanism includes: using the MPR feature vector as the query and the DICOM metadata feature vector as the key and value, generating an MPR attention output guided by metadata semantics to suppress noise interference on lesion edges; and using the DICOM metadata feature vector as the query and the MPR feature vector as the key and value, generating a DICOM attention output that integrates anatomical details to locate the spatial position of lesions. For example, in lung CT image analysis, if an image is blurred due to equipment parameters, and the value of "nodule edge sharpness" in the MPR feature vector is only 0.3, it indicates that the nodule edge looks very blurry. In this case, the MPR feature vector is treated as the "questioner," and the DICOM metadata feature vector is treated as the "responder." The "responder" records information such as scan slice thickness and filtering algorithm. After calculating the correlation between the two, it is found that the scan slice thickness information is particularly important, and it is given a high score (weight) of 0.7. Combined with the information it provides that "thin slice scanning helps to see the edges clearly," the nodule edge sharpness value is adjusted to 0.5, making the edge information more accurate and reducing noise interference. Conversely, if the location of the patient's lesion recorded in the DICOM metadata is unclear, with a value of 0.2, the DICOM metadata feature vector is treated as the "questioner," and the MPR feature vector as the "responder." The MPR feature vector records information such as the specific location of the nodule in the lung lobe and its distance from blood vessels. After calculation, the information on "the location of the nodule in the lung lobe" is found to be the most useful, and it is given a high score (weight) of 0.8. With reference to this information, the clarity value of the lesion location is increased to 0.7 in order to more accurately locate the lesion.

[0102] Feature enhancement layer 404 is used to deepen feature representation through a combination of "multi-head mechanism + residual connection + feedforward network + layer normalization". Multi-head attention splits the cross-attention output into multiple subspaces for parallel processing, capturing multi-dimensional feature associations; residual connection directly transmits the original features, avoiding gradient vanishing and maintaining information integrity; feedforward network (FFN) extracts high-order feature patterns through nonlinear transformation (such as ReLU); layer normalization stabilizes the data distribution. The multi-stage collaboration enhances feature discriminative power, providing more representative features for subsequent classification. For example, the 256-dimensional features output by cross-attention are split into eight 32-dimensional subspaces by the multi-head mechanism. Subspace 1 focuses on the correlation between "nodule edge and age", while subspace 2 explores the relationship between "nodule density and smoking history". Residual connections add the original MPR / DICOM features (such as "nodule diameter" value of 0.5) to attention-enhanced features (value of 0.6) to preserve basic information. The feedforward network first maps the features to 512 dimensions (linear transformation + ReLU) and then compresses them back to 256 dimensions to enhance the expression of "malignant feature combinations (such as irregular edges + advanced age + long-term smoking)". The final layer normalization ensures that the feature distribution of each subspace is stable and outputs enhanced features that are easier to distinguish between benign and malignant.

[0103] Feature gating layer 405 is used to construct an adaptive gating mechanism to dynamically adjust the contribution ratio of MPR and DICOM enhanced features. For MPR and DICOM enhanced features, gating weights are generated through Sigmoid activation (with a value range constrained to [0,1]). Weight generation is driven by the feature's own attributes (such as the edge sharpness of MPR and the age value of DICOM) and the current diagnostic scenario (such as low-dose CT scan, conventional CT scan, or different disease types), automatically adapting to the differences in the importance of modal information to diagnosis under different scenarios. For example, in low-dose CT, MPR spatial details are easily interfered with by noise, so gating will reduce their weight; in conventional CT, MPR details are reliable, so the weight will be increased. This solves the problem of unbalanced contributions from multimodal features, guiding the model to focus on information with greater diagnostic value. For example, in the scenario of pulmonary disease diagnosis, if the input is MPR features from low-dose CT (due to high noise, the "nodule edge smoothness" dimension value is 0.3, indicating low reliability) and DICOM metadata ("patient age 70 years old" dimension value is 0.9, indicating clear information), the gating mechanism will be more effective. During gating, the gating weights are calculated for MPR enhanced features: after sigmoid activation, the weight output for "nodule edge features" is 0.3; for DICOM enhanced features, the weight for "age > 60 years" is 0.7 after sigmoid. This reduces the influence of unreliable spatial features and enhances clinical semantic information. If a conventional CT input is used, the MPR feature "nodule lobulation sign" has a dimension value of 0.8 (clear details), and the gating weight is increased to 0.8; the DICOM feature weight is adjusted to 0.2, flexibly adapting to the diagnostic needs of different scanning scenarios.

[0104] In some possible implementations, the fusion includes: generating dynamic weights through a feature gating layer to adaptively adjust the contribution of MPR features and DICOM metadata features. Specifically, when identifying lesions smaller than a preset threshold, the spatial detail weight of the MPR features is enhanced; and in noise-sensitive scenarios, the scanning parameter weight of the DICOM metadata is enhanced. For example, the preset threshold could be a lesion diameter of 3mm, etc., as a criterion. When the minimum circumscribed circle diameter of the lesion is less than 3mm, the spatial detail feature vector of the MPR features can be assigned a weight coefficient of no less than 0.7; when the noise index in the DICOM metadata exceeds 15%, the weight ratio of the scanning parameter feature vector of the DICOM metadata is increased to above 0.6, such as 0.65.

[0105] The feature integration layer 406 generates a comprehensive feature vector through a dual-path fusion strategy. The concatenation path directly connects the gated MPR and DICOM features, reducing dimensionality and preserving the original associations of the multimodal features. The weighted path calculates modality weights through attention, highlighting key information. The fusion of the two path results in a comprehensive feature vector that simultaneously contains lesion spatial details and semantic information, providing a foundation for the standardization process of the output layer. For example, if the gated MPR features (containing "edge irregularity 0.8") and the DICOM features (containing "age 0.7") are concatenated and dimensionality reduced, resulting in a fused expression with a certain dimension value of "0.8 (spatial) + 0.7 (semantic)," this fused expression is ultimately standardized into a unit vector by the output layer for use in downstream tasks.

[0106] Output layer 407 is designed to perform final processing on the results of the feature integration layer, generating a high-quality comprehensive feature vector. First, a feature transformation layer uses a linear transformation to adjust the dimensionality of the feature vector, making it suitable for the dimensionality requirements of subsequent tasks. Then, L2 normalization is used to scale the feature vector to a unit length, eliminating differences in feature scale between samples and ensuring numerical consistency and stability. After these two steps, a comprehensive feature vector containing lesion semantics and spatial details is finally output, providing a standardized feature representation for subsequent analysis and judgment tasks. For example, in the diagnosis of lung diseases, the feature integration layer outputs a 256-dimensional feature vector, which integrates spatial details of nodules from the MPR image (such as nodule edge irregularity, lobulation, density, etc.) and lesion semantic information from DICOM metadata (such as patient age, smoking history, family history, etc.). If a subsequent task requires a 64-dimensional feature vector, the output layer uses a linear transformation layer with a 256×64 weight matrix to convert the 256-dimensional feature vector into a 64-dimensional vector. The resulting 64-dimensional feature vector has different numerical ranges for each dimension, such as [2.5, 0.3, 1.8, ...]. To avoid the impact of numerical scale differences on subsequent applications, the output layer performs L2 normalization on it. First, the L2 norm of the vector is calculated, i.e. A specific numerical value is obtained; then, each element in the vector is divided by this L2 norm to make the vector length unit length, resulting in a standardized vector such as [2.5 / calculation result, 0.3 / calculation result, 1.8 / calculation result, ...]. The final output, a 64-dimensional unit-length comprehensive feature vector, preserves the spatial details of lung lesions in the MPR image while incorporating semantic information from DICOM metadata, serving as input for further analysis, research, or other processing tasks.

[0107] S103. The comprehensive feature vector is processed using a neural network model based on the Transformer architecture to output the identification result of the target lesion.

[0108] This step, based on a neural network model with a Transformer architecture, processes the comprehensive feature vector and outputs the identification results of the scanned areas in the medical images. See the Transformer architecture for details. Figure 5A It should be noted that, Figure 5B , Figure 5C and Figure 5D They are Figure 5A Part of it, because Figure 5A There's a lot of content, and when displayed on the same page, the images aren't clear enough. Figure 5B , Figure 5C and Figure 5D For reference Figure 5A The specific details.

[0109] The processing flow of a neural network model based on the Transformer architecture is described below.

[0110] The input processing layer 501, acting as a "pre-normalization center" for multimodal feature processing and the first step in target lesion identification, specifically normalizes multimodal features containing target lesion information (such as lesion-related scanning parameters in DICOM metadata and spatial features of lesions in MPR images). Through the feature embedding layer, semantic features such as "lesion layer thickness" and "contrast agent injection parameters" in DICOM and spatial features such as "lesion edge contour" and "density distribution" in MPR images are uniformly mapped to a dimension that the model can efficiently compute (e.g., 64-dimensional). The positional encoding layer injects sequence identifiers into the feature sequences, enabling the model to recognize the logical relationship of "scan parameter description preceding lesion image features" (e.g., the sequential dependency between "high-resolution scan parameters" and "lesion detail features"). The input normalization layer eliminates differences in feature dimensions (e.g., the numerical range of lesion diameter "5mm" differs from that of edge smoothness "0.8"), ensuring that subsequent networks can fairly process all features related to the target lesion, thus laying a solid data foundation for accurate identification.

[0111] For example, in a lung nodule identification scenario, the input multimodal features include: DICOM metadata: scan slice thickness 1mm (related to the display of small lesions), contrast agent type (affecting lesion enhancement features); MPR image features: lobulation sign of the nodule's edge in the coronal plane (value 0.7), and axial density value (40HU). The input processing layer upscales the scan slice thickness scalar to 64 dimensions, embeds the contrast agent type as a 64-dimensional vector, and maintains the nodule edge and density features at 64 dimensions, uniformly outputting a standardized feature sequence with positional encoding (slice thickness → contrast agent → edge → density), enabling the model to focus on feature combinations related to lung nodules (target lesions), such as "thin slice scan + lobulation sign". In another example, for brain tumor identification, the input features include: DICOM metadata: scan sequence (T1 enhancement, related to tumor enhancement), patient age (affecting the probability of tumor type); MPR image features: the degree of ring enhancement of the tumor on the T1 enhanced image (value 0.9), and its boundary with surrounding tissue (value 0.6). After passing through input processing layer 501, the T1 enhanced sequence is encoded into a 64-dimensional vector. After age standardization, it is projected to 64 dimensions. The enhancement level and boundary features are kept in 64 dimensions, generating a feature sequence with position encoding (scan sequence → age → enhancement → boundary), ensuring that key features of the target lesion (brain tumor) such as "T1 enhancement + ring enhancement" are processed preferentially.

[0112] like Figure 5A As shown, the Transformer encoder 502 consists of three cascaded Transformer blocks (5021, 5022, 5023). This layer, through multiple Transformer blocks (5021 / 5022 / 5023), is used to uncover hidden correlations between features related to the target lesion. Each Transformer block uses a multi-head self-attention mechanism to dynamically interact the "scan parameters" in the DICOM metadata with the "lesion morphology" features in the MPR image (e.g., automatically discovering the negative correlation between "low-dose scan parameters" and "blurred lesion edge features"). Residual connections and layer normalization ensure that the basic features of the target lesion are not lost, and the feedforward network extracts the deep logic of "scan parameters → image quality → lesion discernibility". After multiple iterations, the output feature sequence can accurately represent the "semantic-spatial" association pattern of the target lesion (e.g., "high-resolution scan + lesion lobulation sign = malignant tumor tendency").

[0113] For example, in one instance, when identifying liver tumors, the input feature sequence is processed by Transformer encoder 1 5021: multi-head self-attention calculates the correlation between "DICOM contrast agent injection rate" and "MPR tumor enhancement peak," finding that the enhancement peak is significant when the injection rate is high (weight 0.8), establishing a correlation between "contrast agent parameters → tumor blood supply characteristics"; Transformer encoder 2 5022 further refines the feature pattern of "fast in and fast out enhancement + high injection rate = liver cancer," preserving the original tumor edge features through residual connections (value 0.7); in the feature sequence output by Transformer encoder 3 5023, the abstract expression of "liver cancer feature combination" is enhanced (value 0.9), providing deep interactive features for subsequent identification of target lesions (liver cancer). In another example, for a ground-glass nodule in the lung (the target lesion), the Transformer encoder 502 works as follows: Transformer encoder 1 calculates the correlation between "DICOM scan tube voltage 120kV" and "MPR nodule density -600HU", finding that density measurement is more accurate under high voltage (weight 0.7); Transformer encoder 2 5022 abstracts the combination of "tube voltage + density + nodule diameter 5mm" into a "pure ground-glass nodule" feature through a feedforward network (value 0.8); In the feature sequence output by Transformer encoder 3 5023, the "pure ground-glass nodule" related features are given high weight, suppressing normal lung tissue features (weight 0.3), forming a specific feature representation of the target lesion.

[0114] The feature extraction layer 503 is a crucial step in mining the features of the target lesion. Its role is to filter out information closely related to the target lesion from the multimodal feature sequence output by the Transformer encoding layer. Through a global attention pooling mechanism, the correlation between each feature location and the target lesion is automatically calculated, giving high weight to lesion region features and weakening background and irrelevant tissue features. Then, fully connected layers, GELU activation function, and Dropout layer are used to further compress feature dimensions, enhance feature expressive power, and prevent overfitting. Finally, a global summary feature vector containing key information such as the core morphology, density, and location of the target lesion is output, laying the foundation for accurate lesion identification.

[0115] For example, in a lung cancer screening scenario, the target lesion is a lung nodule. The input multimodal feature sequence includes features such as the edge morphology and density distribution of the nodule in the MPR image, as well as information such as scan parameters and patient history from DICOM metadata. The feature extraction layer uses global attention pooling to increase the weight of features such as spiculation and lobulation of the nodule edge to 0.8, while the weight of texture features of normal lung tissue is only 0.2. After a fully connected layer and GELU activation, the feature combination pattern of "irregular edge + uneven density = suspected malignant nodule" is strengthened, and the final output is a 512-dimensional vector containing key features of the nodule, which helps to determine the benign or malignant nature of the nodule. In another example, for the identification of brain hemorrhage lesions, the target lesion is the intracranial hemorrhage area. The input features include the signal intensity and location information of the hemorrhage lesion in the MRI image, as well as patient age and underlying diseases from DICOM metadata. The feature extraction layer uses global attention pooling to increase the weight of high signal features and hematoma morphology features of the hemorrhage lesion on T2-weighted images to 0.7, while the weight of ordinary signal features of the surrounding brain tissue is set to 0.3. After dimensionality reduction by the fully connected layer and GELU activation, the correlation feature of "sudden high signal + elderly hypertensive patient = cerebral hemorrhage" is highlighted, and the output feature vector can accurately characterize the hemorrhage lesion, assisting doctors in quickly locating and assessing the hemorrhage.

[0116] The classification head 504 is used to transform the globally summarized features output by the feature extraction layer into probability scores that can be directly used to determine the type of lesion. It first compresses the feature dimensions progressively through multiple classification mapping layers (e.g., 512-dimensional → 256-dimensional → 128-dimensional), filtering out redundant information irrelevant to target lesion identification. Simultaneously, it enhances the non-linear expressive power of the features using the GELU activation function (e.g., strengthening the combined association of "lesion enhancement pattern + patient age"), and uses a Dropout layer to randomly discard some features to improve the model's robustness against interference. Finally, the output layer maps the features to a dimension consistent with the number of target lesion categories (e.g., outputting 4 dimensions if 4 types of lesions are identified), and uses batch normalization to unify the feature distribution, outputting a matching score for each category, providing a quantitative basis for clinically determining the specific type of the target lesion.

[0117] For example, in the identification of benign and malignant lung nodules, the 512-dimensional feature vector output by the feature extraction layer includes information such as "nodule lobulation intensity 0.7" and "patient's 30-year smoking history". After compression by the 512→256-dimensional mapping layer, the GELU activation highlights the feature combination of "lobulation + long-term smoking = malignant tendency"; after purification by the 256→128-dimensional mapping layer, the output layer is mapped to 2 dimensions (benign / malignant), and after batch normalization, the output matching score [0.3, 0.7] is given, indicating that the target lesion (lung nodule) is more likely to be malignant. In another example, for the identification of target lesions in the brain (such as tumor / hemorrhage / infarction), the classification head receives a 512-dimensional feature vector (including "T1-weighted image low signal intensity 0.8" and "patient's history of hypertension"). After being compressed to 128 dimensions through multi-layer mapping, GELU activation strengthens the association feature of "low signal + high blood pressure = cerebral hemorrhage"; the output layer is mapped to 3 dimensions, and after batch normalization, the output score is [0.1, 0.8, 0.1], which shows that the target lesion (abnormal area in the brain) has the highest matching degree for cerebral hemorrhage.

[0118] The output layer 505 generates the identification results of the target lesion. It transforms the lesion category matching score output by the classification head 504 into a probabilistic identification result with clinical guidance significance. Using the Softmax activation function, the score is normalized to a probability distribution between 0 and 1, clearly showing the probability that the target lesion belongs to each category. Combined with temperature scaling technology, the concentration of the probability distribution can be flexibly adjusted to make the results more consistent with actual diagnostic needs. Finally, the probability value of each target lesion category is output, presenting the identification results in an intuitive and quantitative way, providing a reliable basis for doctors' diagnostic and treatment decisions.

[0119] For example, in the diagnosis of thyroid nodules, the target lesion is a nodule within the thyroid gland, which may be benign, malignant, or cystic. The 3D matching score output by the classification head is [0.2, 0.6, 0.2]. After Softmax activation, it is transformed into a probability distribution [0.18, 0.65, 0.17], meaning the probability of the thyroid nodule being benign is 18%, malignant is 65%, and cystic is 17%. If temperature scaling (temperature set to 0.5) is used to further enhance the probability difference, the output becomes [0.1, 0.85, 0.05], clearly indicating that the nodule is highly likely to be malignant, and further fine-needle aspiration biopsy is recommended. In another example, for the identification of target lesions in the skeletal system, such as bone tumors, fractures, and osteomyelitis, the classification head outputs a 4D matching score [0.1, 0.3, 0.5, 0.1] (corresponding to different types of skeletal lesions). The probability obtained after Softmax activation is [0.09, 0.27, 0.54, 0.1], indicating that the target lesion is most likely osteomyelitis; the probability is fine-tuned to [0.08, 0.25, 0.6, 0.07] by temperature scaling (temperature set to 0.8), further highlighting the diagnostic tendency of osteomyelitis.

[0120] This embodiment offers significant advantages in target lesion identification. The input processing layer ensures effective input of lesion-related DICOM semantics and MPR spatial features by unifying the dimensions and distribution of multimodal features. The Transformer encoding layer utilizes a self-attention mechanism to deeply mine hidden correlations between lesion features, strengthening cross-modal interaction between "scanning parameters and lesion morphology." The feature extraction layer focuses on core lesion features through global attention, suppressing background interference. The classification head and output layer accurately map features to probabilistic results of lesion categories, improving identification reliability. For example, in lung nodule identification, it can automatically associate "thin-slice scanning parameters" with "nodule lobulation sign," quickly determining benign or malignant. This scheme avoids human intervention bias through an adaptive weighting mechanism and captures complex pathological features using a hierarchical structure, effectively improving the efficiency and accuracy of target lesion identification in medical images, providing quantitative support for precise clinical diagnosis.

[0121] It should be noted that in some embodiments of the present invention, the target lesion is not pre-inputted or manually specified, but is gradually determined through the automated analysis of medical images by the model. For example, the MPR feature vector contains morphological and spatial location information of anatomical structures (such as organ contours and tissue density differences), while the DICOM metadata feature vector records scanning parameters and patient information (such as high-resolution scan slice thickness and contrast agent injection parameters). These data implicitly contain potential clues about the lesion (such as abnormal density areas and morphological distortions). The bidirectional cross-attention mechanism fuses the two types of features through dynamic weights. For example, in lung CT, MPR features capture the lobulation sign of nodules, while DICOM metadata is associated with thin-slice scanning parameters (improving the display accuracy of small lesions). The interaction between the two can enhance the ability to locate "abnormal structures". The neural network model based on the Transformer architecture extracts the semantic information (such as "ground-glass nodule" and "enhancing tumor") and spatial location (such as "right upper lobe") of the lesion from the comprehensive feature vector through multi-layer feature interaction and abstraction. The final output recognition result is the target lesion and its type automatically detected by the model, without the need for manual pre-input of the lesion location or range.

[0122] This solution achieves end-to-end detection of target lesions from raw medical image data through fully automated processing of "feature encoding - cross-modal fusion - deep recognition", avoiding the subjectivity and limitations of manual annotation.

[0123] This invention also provides a target lesion recognition system based on medical images, comprising: an encoding module, a fusion module, and a recognition module. The encoding module generates dimension-aligned MPR feature vectors and DICOM metadata feature vectors based on medical image encoding. The MPR feature vectors contain morphological and spatial location information of anatomical structures, while the DICOM metadata feature vectors contain scanning parameters, device information, and patient positioning information. The fusion module fuses the MPR feature vectors and DICOM metadata feature vectors using a bidirectional cross-attention mechanism to generate a comprehensive feature vector containing lesion semantics and spatial details. The recognition module processes the comprehensive feature vector using a Transformer-based neural network model and outputs the recognition result of the target lesion.

[0124] An embodiment of the present invention provides a target lesion identification device based on medical images. The target lesion identification device includes: a memory and a processor. The processor is configured to execute computer-executable instructions. The memory stores one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement each step of any of the target lesion identification methods based on medical images described in the preceding method embodiments.

[0125] This invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements any of the embodiments of the method for identifying target lesions based on medical images described above.

[0126] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the embodiments of the method for identifying target lesions based on medical images described above.

[0127] This invention also provides a chip system for use in an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to execute any of the methods in the preceding embodiments of the target lesion identification method based on medical images.

[0128] It is understood that the beneficial effects that the above-mentioned target lesion identification system based on medical images, target lesion identification device based on medical images, computer storage medium, computer program product, and chip system can achieve can be referred to the beneficial effects described in the method embodiments, and will not be repeated here.

[0129] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0130] The above-described embodiments are optional embodiments provided by this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the technical scope disclosed in this application should be included within the protection scope of this application.

Claims

1. A method for identifying target lesions based on medical images, characterized in that, include: Based on medical image coding, dimension-aligned MPR feature vectors and DICOM metadata feature vectors are generated. The MPR feature vectors contain morphological and spatial location information of anatomical structures, and the DICOM metadata feature vectors contain scanning parameters, equipment information, and patient positioning information. The MPR feature vectors and DICOM metadata feature vectors are fused through a bidirectional cross-attention mechanism to generate a comprehensive feature vector containing lesion semantics and spatial details. The comprehensive feature vector is processed using a neural network model based on the Transformer architecture to output the identification result of the target lesion.

2. The target lesion identification method according to claim 1, characterized in that, The MPR feature vector and DICOM metadata feature vector generated based on medical image encoding with dimension alignment include: Multiplanar reconstruction is performed based on medical images to generate an MPR image queue; the MPR image queue includes: coronal MPR images, sagittal MPR images, and transverse MPR images. The MPR image queue is input into a deep learning model for processing. The deep learning model includes: a backbone feature extraction module including convolutional blocks, an attention enhancement module, and an in-plane multi-scale feature fusion module including a feature pyramid network. After the MPR image queue is processed sequentially by the backbone feature extraction module, the attention enhancement module, and the in-plane multi-scale feature fusion module, a multi-scale feature map corresponding to each MPR image in the MPR image queue is obtained. The deep learning model also includes: a multi-plane interaction module and a feature fusion and encoding module. The multi-plane interaction module is used to perform cross-plane correlation fusion on the multi-scale feature map to obtain a multi-plane fused feature map with a two-dimensional structure; the feature fusion and encoding module is used to perform global pooling and fully connected encoding on the multi-plane fused feature map to output a one-dimensional MPR feature vector with N elements, where N is an integer and N>1.

3. The target lesion identification method according to claim 2, characterized in that, The MPR feature vector and DICOM metadata feature vector generated based on medical image encoding with dimension alignment include: Load the DICOM file sequence corresponding to the set of medical images belonging to a sequence obtained from the tomographic scan, and arrange them in the scan order to obtain the initial image file queue; Read DICOM files from a specified position in the initial image file queue to form a fixed-length key image file queue; Sequence-level DICOM metadata and image-level DICOM metadata representing key features are read from the key image file queue, and arranged according to a preset rule to obtain a sequence DICOM metadata queue with fixed length and fixed information items at each position; the key features include: examination type, scan parameters, scan site and patient information; The sequence DICOM metadata queue is input into a deep learning network, which includes a feature preprocessing and classification layer, a feature encoding layer, a feature fusion layer, a context modeling layer, and an output layer. The deep learning network is triggered to transform the sequence DICOM metadata queue into a one-dimensional DICOM metadata feature vector with M elements, where M is a number and M > 1.

4. The target lesion identification method according to claim 3, characterized in that, When M≠N, before generating the dimension-aligned MPR feature vector and DICOM metadata feature vector based on medical image encoding, the target lesion identification method further includes: The MPR feature vector and DICOM metadata feature vector are processed to obtain dimension-aligned MPR feature vector and DICOM metadata feature vector.

5. The method according to claim 1, characterized in that, The bidirectional cross-attention mechanism includes: Using the MPR feature vector as the query and the DICOM metadata feature vector as the key and value, an MPR attention output guided by metadata semantics is generated to suppress noise interference on lesion edges; and using the DICOM metadata feature vector as the query and the MPR feature vector as the key and value, a DICOM attention output incorporating anatomical details is generated to locate the spatial position of the lesion.

6. The method according to any one of claims 1 to 5, characterized in that, The fusion includes: Dynamic weights are generated through a feature gating layer to adaptively adjust the contribution of MPR features and DICOM metadata features. Specifically, when identifying lesions smaller than a preset threshold, the spatial detail weight of MPR features is enhanced, and the scanning parameter weight of DICOM metadata is enhanced in noise-sensitive scenarios.

7. A target lesion identification system based on medical images, characterized in that, include: The module consists of an encoding module, a fusion module, and a recognition module. The encoding module is used to generate dimension-aligned MPR feature vectors and DICOM metadata feature vectors based on medical image encoding. The MPR feature vectors contain morphological and spatial location information of anatomical structures, and the DICOM metadata feature vectors contain scanning parameters, equipment information, and patient positioning information. The fusion module is used to fuse the MPR feature vector and the DICOM metadata feature vector through a bidirectional cross-attention mechanism to generate a comprehensive feature vector containing lesion semantics and spatial details. The identification module is used to process the comprehensive feature vector using a neural network model based on the Transformer architecture and output the identification result of the target lesion.

8. A target lesion identification device based on medical images, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the target lesion identification method based on medical images as described in any one of claims 1-6.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target lesion identification method based on medical images as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the target lesion identification method based on medical images as described in any one of claims 1-6.

Images