Tumor image information collection method based on feature fusion

Abstract

The invention relates to the technical field of medical image processing and medical big data management, and discloses a tumor image information collection method based on feature fusion, and the method comprises the steps: obtaining a multi-source heterogeneous tumor image and report, and carrying out the image standardization and label structuring processing; constructing a double-flow network containing shallow visual and deep semantic branches, and generating multi-scale fusion features; an adversarial field adaptive mechanism is introduced, and feature distribution differences caused by different imaging devices are eliminated; performing weighted re-calibration on the features by using a channel and a space attention mechanism, and eliminating redundant features to output key feature vectors; and establishing unique ID association between the feature vector and the diagnosis tag, and respectively mapping the feature vector and the diagnosis tag to a vector database and a relational database. According to the method, the problems of high heterogeneity, insufficient feature expression and high redundancy of multi-center image data are effectively solved, and standardized and structured efficient collection and retrieval of tumor image information are realized.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and medical big data management technology, specifically a method for collecting tumor image information based on feature fusion. Background Technology

[0002] Currently, with the rapid advancement of medical digitalization, the diagnosis and treatment of tumors increasingly rely on medical imaging data such as computed tomography (CT) scans and magnetic resonance imaging (MRI). Medical institutions accumulate massive amounts of imaging data daily, containing rich pathological information ranging from microscopic texture details to macroscopic morphological structures. Effectively collecting and managing this diverse and heterogeneous imaging data from various sources has become a crucial data foundation for building large-scale medical artificial intelligence models and assisting in precise clinical decision-making.

[0003] For the management and application of the aforementioned image data, existing medical image archiving and communication systems or disease-specific research databases typically employ standardized extraction, transformation, and loading processes. The conventional operating model involves using traditional radiomics methods to calculate manual features such as the gray-level co-occurrence matrix, or directly using general convolutional neural network models to map two-dimensional or three-dimensional images into fixed-length numerical vectors. The system then performs simple association and storage of these feature data with clinical text reports to support basic data retrieval, comparison, and statistical analysis, serving subsequent retrospective medical record studies or the development of auxiliary diagnostic systems.

[0004] However, existing technologies still have many limitations in addressing the complex and ever-changing needs of tumor data collection. Conventional feature extraction methods often sever the intrinsic connection between low-level visual texture and high-level semantic location in images. This single-scale feature representation struggles to capture the strong correlation between lesion microenvironment details and overall topological morphology, easily leading to the omission of crucial diagnostic information. Furthermore, because image data originates from different medical institutions, imaging equipment from different manufacturers, and various scanning protocols, there are significant feature distribution offsets between data. Existing collection processes lack effective domain distribution alignment and correction mechanisms, making extracted features overly sensitive to equipment parameters and hindering the universal recognition and analysis of cross-center data. In addition, raw medical images contain a large amount of background tissue and noise unrelated to lesions. Existing methods lack adaptive filtering and fine-tuning of feature dimensions. Storing a large amount of uncleaned redundant features not only wastes storage resources but also introduces interference noise, dragging down the convergence efficiency and predictive performance of subsequent AI model training.

[0005] Therefore, this invention provides a tumor image information collection method based on feature fusion to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a tumor image information collection method based on feature fusion, which solves the problems of strong feature heterogeneity, insufficient single feature expression ability, and high data redundancy in the existing multi-center medical image data collection process.

[0007] To achieve the above objectives, the present invention provides a method for collecting tumor image information based on feature fusion, comprising the following steps: S1. Obtain tumor medical image data from different medical institutions and corresponding clinical diagnosis reports. Perform spatial resampling and grayscale normalization processing on the tumor medical image data, and extract fixed-size three-dimensional image blocks containing tumor lesions and surrounding tissues based on the tumor center coordinates. At the same time, perform text parsing on the clinical diagnosis reports to generate structured diagnostic labels. S2. Input the three-dimensional image block into a deep neural network model containing a shallow feature extraction branch and a deep semantic extraction branch set in parallel. Fuse the low-level visual features output by the shallow feature extraction branch with the high-level semantic features output by the deep semantic extraction branch to generate an initial fused feature vector. S3. Calculate the adversarial loss between the feature distribution of the source domain data and the feature distribution of the target domain data, and adjust the parameters of the deep neural network model based on the domain adaptation mechanism to correct the initial fused feature vector; S4. Using the channel attention unit and spatial attention unit embedded in the deep neural network model, the corrected initial fusion feature vector is weighted and calibrated, and then filtered according to a preset numerical threshold to output the key feature vector. S5. Generate a unique index ID, and associate the key feature vector with the structured diagnostic label. Map the key feature vector to a vector database and the structured diagnostic label to a relational database through the unique index ID to achieve bidirectional indexing.

[0008] By adopting the above technical solutions, this invention solves the problems of strong feature heterogeneity, insufficient single feature expression ability, and high data redundancy in existing multi-center medical imaging data. First, through a dual-branch network architecture, it can simultaneously capture the microscopic texture details and macroscopic semantic structure of tumors, avoiding information loss caused by single-scale feature extraction. Second, by introducing an adversarial domain adaptation mechanism, the feature extraction network learns common domain-invariant features across data sources from different devices or medical institutions, effectively eliminating feature distribution shifts caused by differences in the physical parameters of imaging devices. Third, by combining an attention mechanism and a threshold screening strategy, adaptive recalibration and dimensionality reduction of features are achieved, preserving high-diagnostic-value feature components while suppressing background noise. Fourth, a bidirectional index storage mode of feature vectors and structured labels is constructed, balancing the efficiency of similarity retrieval of high-dimensional features with the precise query requirements of clinical text, achieving structured and efficient collection of tumor image information.

[0009] Preferably, the acquisition and preprocessing steps further include: after performing spatial resampling and grayscale normalization, locating and extracting a fixed-size three-dimensional image block containing the tumor lesion and surrounding microenvironment tissue from the tumor medical image data based on the center coordinates marked by the physician or a pre-generated coarse segmentation mask, for inputting into the deep neural network model.

[0010] By adopting the above technical solution, focusing on the region of interest eliminates the interference of a large number of irrelevant background tissues on feature extraction, while unifying the geometric dimension of the input data, reducing the computational load of the subsequent neural network, and improving the model's attention to the lesion area.

[0011] Preferably, the deep neural network model is constructed and fused as follows: the shallow feature extraction branch consists of stacked shallow convolutional neural network layers, used to generate shallow feature maps containing low-level visual features; the deep semantic extraction branch uses a deep residual network or a densely connected network as the backbone architecture, used to generate deep feature maps containing high-level semantic features; the deep feature maps are upsampled to make their spatial dimensions consistent with those of the shallow feature maps; the upsampled deep feature maps are concatenated with the shallow feature maps in the channel dimension and input into a convolutional layer for dimensionality reduction to generate the initial fused feature vector.

[0012] By adopting the above technical solution, shallow networks are used to retain high-resolution texture and edge information, deep networks are used to extract abstract category and spatial relationship information, and cross-scale information alignment and complementarity are achieved through upsampling and channel splicing, thereby generating multimodal fusion features with stronger expressive power.

[0013] Preferably, the specific implementation of the domain adaptation mechanism is as follows: A dataset with a standard distribution is set as the source domain, and new data to be aggregated is set as the target domain; a domain discriminator network connected to the deep neural network model is constructed, which is used to predict whether the input feature vector comes from the source domain or the target domain; a gradient reversal layer is introduced between the deep neural network model and the domain discriminator network, which multiplies the gradient returned by the domain discriminator network by a negative constant during the backpropagation phase and then passes it to the deep neural network model; the parameters of the deep neural network model and the domain discriminator network are updated by minimizing a total loss function weighted by task classification loss and domain adversarial loss, forcing the deep neural network model to learn domain-invariant features.

[0014] By employing the above technical solution, an adversarial training environment was constructed using a gradient inversion layer. During training, the domain discriminator strives to distinguish feature sources, while the feature extraction network strives to generate indistinguishable features. This game-theoretic process forces feature distributions to align between the source and target domains. The optimization process of the total loss function essentially maximizes the classification error of the domain discriminator while ensuring the accuracy of the diagnostic task, thereby obtaining a robust feature representation that is insensitive to the device source.

[0015] Preferably, the task classification loss is calculated using the cross-entropy loss function, which is used to constrain the features generated by the deep neural network model to accurately predict the clinical diagnostic labels of the source domain data; the domain adversarial loss is calculated using the binary cross-entropy loss function, which is used to measure the ability of the domain discriminator network to distinguish between the source domain and the target domain.

[0016] By adopting the above technical solution, the composition of the optimized objective function is clarified, ensuring that the model does not lose its ability to distinguish tumor pathological types while eliminating domain differences.

[0017] Preferably, the specific implementation of feature filtering and calibration is as follows: inputting the calibrated features into the channel attention unit to generate a channel attention weight vector; using the channel attention weight vector to perform channel-dimensional multiplication weighting on the calibrated features to obtain an intermediate feature map after channel weighting; inputting the intermediate feature map into the spatial attention unit to generate a spatial attention weight map; using the spatial attention weight map to perform spatial-dimensional multiplication weighting on the intermediate feature map to obtain a refined feature map; performing global average pooling on the refined feature map and flattening it into a one-dimensional feature vector, and according to the preset numerical threshold, removing feature elements with values ​​lower than the numerical threshold, and outputting the key feature vector.

[0018] By employing the above technical solution, a visual attention mechanism was simulated, explicitly modeling the importance of feature channels and spatial locations. Through cascaded weighting operations, feature responses highly correlated with tumor pathology were enhanced, while invalid features were suppressed. Subsequent threshold removal further achieved feature sparsity, removing redundant information and generating compact feature vectors with high semantic density.

[0019] Preferably, the channel attention unit operates as follows: global average pooling and global max pooling are performed on the input features in the spatial dimension to generate two spatial context description vectors; the two spatial context description vectors are input into a shared multilayer perceptron network; the two feature vectors output by the multilayer perceptron network are added element-wise and processed by the Sigmoid activation function to generate the channel attention weight vector.

[0020] By adopting the above technical solutions, and by comprehensively utilizing average pooling and max pooling to aggregate spatial information, we can more comprehensively capture the interdependencies between feature channels, thereby accurately determining which feature channels are more critical for classification tasks.

[0021] Preferably, the spatial attention unit operates as follows: average pooling and max pooling operations are performed on the input features along the channel dimension to generate two spatial feature description maps; the two spatial feature description maps are concatenated along the channel dimension; a convolutional layer is used to perform a convolution operation on the concatenated features and compress the number of channels to a certain range, and then the features are processed by the Sigmoid activation function to generate the spatial attention weight map.

[0022] By adopting the above technical solution, information is aggregated in the channel dimension to highlight the salience of spatial location features. The generated weight map can accurately locate the lesion area, so that the network focus is on the tumor body and its edge, rather than irrelevant background.

[0023] Preferably, the specific implementation of the bidirectional index and storage is as follows: the unique index ID is a globally unique identifier containing timestamp information and data source organization code; the key feature vector together with the globally unique identifier is written as a metadata tag into the vector database configured with an inverted file index or a hierarchical small world graph algorithm; the structured diagnostic tag is written into the data table of the relational database, and the globally unique identifier is used as the primary key of the data table.

[0024] By adopting the above technical solution, the logical unification of heterogeneous data storage systems is achieved using globally unique identifiers. Vector databases support fast nearest neighbor search for high-dimensional features, while relational databases support complex structured queries. The combination of the two meets the diverse needs of medical data application scenarios.

[0025] Preferably, the method further includes a step of retrieving data based on aggregated data: receiving a new tumor image to be diagnosed, and calling the aforementioned processing flow to generate a query feature vector corresponding to the new tumor image; calculating the Euclidean distance or cosine similarity between the query feature vector and the historical feature vectors stored in the vector database, filtering out the most similar historical feature vector and extracting its associated unique index ID; using the extracted unique index ID, performing a query operation in the relational database to retrieve detailed diagnostic reports, treatment plans, and prognostic results of historical confirmed cases corresponding to the unique index ID.

[0026] By adopting the above technical solution, a content-based image retrieval application was realized. The system can quickly match historical confirmed cases based on the similarity of image features, providing clinicians with evidence-based diagnostic references and treatment suggestions.

[0027] This invention provides a method for collecting tumor image information based on feature fusion. It has the following beneficial effects: 1. This invention achieves multi-scale fusion of macroscopic semantics and microscopic texture in tumor images by constructing a parallel network architecture that includes shallow visual branches and deep semantic branches. Combined with channel and spatial attention mechanisms for feature weighting and recalibration, it effectively enhances the feature response of lesion regions and suppresses background noise. This design solves the problem of insufficient representational power of single features, improving the accuracy and completeness of feature vector representation of tumor pathological information.

[0028] 2. This invention introduces an adversarial domain adaptation mechanism, utilizing a gradient inversion layer to construct adversarial training between the feature extractor and the domain discriminator, forcing the network to learn common domain-invariant features between the source and target domains. This mechanism effectively eliminates data distribution shifts caused by differences in different medical institutions, imaging equipment, and scanning parameters, overcomes the strong heterogeneity problem of multi-center medical imaging data, and ensures the robustness and versatility of the model in cross-center data collection and analysis.

[0029] 3. This invention employs an attention-weight-based threshold filtering strategy to eliminate redundant features, generating compact key feature vectors, and establishing a unique ID-linked index between the feature vector database and the clinical label relational database. This approach not only reduces the storage footprint and computational complexity of high-dimensional data but also enables rapid, bidirectional, and accurate retrieval from image features to clinical diagnostic information, providing efficient and structured data support for clinical auxiliary diagnosis. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the tumor image information collection method based on feature fusion according to the present invention; Figure 2 This is a schematic diagram of the multi-source heterogeneous data acquisition and preprocessing process of the present invention; Figure 3 This is a schematic diagram of the multi-scale feature fusion network architecture of the present invention; Figure 4 This is a schematic diagram of the adversarial domain adaptive feature correction module of the present invention; Figure 5 This is a schematic diagram of the key feature filtering module based on the attention mechanism of the present invention; Figure 6 This is a schematic diagram of the feature and tag linkage collection and storage process of the present invention. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] See attached document Figure 1 This invention provides a tumor image information aggregation method based on feature fusion. This method primarily addresses the problems of strong feature heterogeneity, insufficient single-feature representation ability, and high data redundancy in existing multi-center medical imaging data. The method constructs a complete processing pipeline including feature extraction, fusion, correction, and filtering to transform raw medical images from different sources into standardized, high-semantic-density feature vectors and establish strong correlations with clinical diagnostic information. The method includes the following steps: steps S1 to S5.

[0033] Step S1: Acquire and preprocess multi-source heterogeneous data. This step first receives tumor medical image data from different medical institutions and imaging equipment manufacturers, along with corresponding clinical diagnostic reports. Since images from different sources differ in resolution, slice thickness, and grayscale range, this step performs standardization processing on the tumor medical images. This includes spatial resampling to unify voxel spacing and grayscale normalization to map pixel values ​​to a unified numerical range. Simultaneously, this step parses the unstructured clinical diagnostic reports, extracts key diagnostic information, and transforms it into structured diagnostic tags, providing a data foundation for subsequent associated storage.

[0034] Step S2: Construct a multi-scale feature fusion network. This step constructs a deep neural network model containing parallel shallow feature extraction branches and deep semantic extraction branches. The shallow feature extraction branch processes the tumor medical image to extract low-level visual features such as texture, edges, and shape; the deep semantic extraction branch processes the tumor medical image to extract high-level semantic features such as lesion category and spatial location relationship. Subsequently, this step aligns and fuses the low-level visual features and the high-level semantic features in the channel dimension or spatial dimension through feature pyramid or cascading operations, generating an initial fused feature vector containing multi-scale information.

[0035] Step S3: Feature Correction Based on Domain Adaptation. To eliminate the feature distribution differences introduced by different imaging devices, this step introduces a domain adaptation mechanism. This step sets the dataset with a standard distribution as the source domain and the new data to be aggregated as the target domain. By calculating the statistical difference or adversarial loss between the feature distributions of the source domain data and the target domain data, the parameters of the feature extraction network in Step S2 are adjusted, forcing the network to learn domain-invariant features. After this step, the distribution of tumor medical images from different centers in the feature space will tend to be consistent, thereby eliminating interference caused by device specificity.

[0036] Step S4: Selecting Key Features Based on Attention Mechanism. This step embeds an attention module into the feature extraction network, including channel attention units and spatial attention units. The channel attention unit is used to calculate the importance weight of each channel in the initial fused feature vector, and the spatial attention unit is used to calculate the importance weight of each pixel region in the feature map. This step uses the calculated weights to weight and calibrate the initial fused feature vector, enhancing the feature responses highly correlated with tumor lesions while suppressing background noise and feature responses from irrelevant tissues. Subsequently, based on a preset numerical threshold, feature dimensions with weights below the threshold are removed, and the dimensionality-reduced key feature vector is output.

[0037] Step S5: Perform linked collection and storage of features and labels. This step establishes a bidirectional index between medical image features and clinical diagnostic information. This step generates a unique index ID, maps the key feature vectors output in Step S4 to a vector database, and maps the structured diagnostic labels generated in Step S1 to a relational database. Through the unique index ID, the system can achieve rapid lookup from feature vectors to diagnostic labels, and reverse retrieval from diagnostic labels to feature vectors, completing the structured collection of tumor image information.

[0038] See attached document Figure 2This invention provides a tumor image information collection method based on feature fusion, wherein step S1 specifically includes the acquisition, cleaning, image standardization processing, region of interest extraction, and structured mapping of diagnostic labels of multi-source heterogeneous data.

[0039] The system first establishes a communication connection with the hospital's image archiving and communication system or radiology information system to obtain the original medical digital imaging and communication format files. These files originate from different medical institutions and imaging equipment manufactured by different equipment manufacturers, including but not limited to computed tomography (CT) scanners or magnetic resonance imaging (MRI) scanners. Because the raw data contains patients' private information, the system performs a data cleaning procedure. This procedure locates private data tag groups in the header of the medical digital imaging and communication files, identifies sensitive fields including patient names and patient identifiers, and performs anonymization. The system generates unique hash codes or pseudo-random serial numbers to replace the original sensitive field values, ensuring that the identity of a specific patient cannot be traced back through data during subsequent processing and storage, thus meeting the compliance requirements for medical data privacy protection.

[0040] After acquiring the anonymized data, the system performs spatial resampling on the image data. Since images from different devices or scanning protocols have varying layer thicknesses and pixel pitches, directly inputting them into the neural network would lead to scale inconsistencies. The system reads the pixel pitch and layer thickness attributes from the file header and uses cubic spline interpolation or trilinear interpolation to resample all 3D image data to a fixed isotropic resolution, i.e., unifying the physical size of each voxel to 1.0 mm in the X, Y, and Z axes. This operation eliminates geometric distortions caused by differences in physical resolution, ensuring that the feature extraction network has a consistent spatial reference frame when processing images from different sources.

[0041] Subsequently, the system performs grayscale normalization on the resampled images. For computed tomography (CT) images, the system first converts the original stored pixel values ​​to Hausfield units. Considering that differences in scanning parameters such as tube voltage and tube current can affect the grayscale distribution of the image, the system sets specific window width and window level ranges. For pixel values ​​exceeding the upper limit of the set range, the system truncates them to the upper limit value; for pixel values ​​below the lower limit of the set range, the system truncates them to the lower limit value. The truncated values ​​are linearly mapped to the floating-point range of 0 to 1. This step eliminates interference from abnormally bright or dark areas and constrains the data distribution of all input images within a numerical range where the neural network model can easily converge.

[0042] The system then performs region of interest localization and extraction. Since the original full-size medical image contains a large amount of background tissue irrelevant to tumor diagnosis, such as air, the examination bed, or bones far from the lesion, the system determines the geometric center of the tumor lesion based on the physician-annotated center coordinates or a pre-generated coarse segmentation mask. Using this geometric center as a reference, the system extracts a fixed-size 3D image patch. This 3D image patch not only completely contains the tumor lesion itself but also includes the surrounding microenvironment tissue within a certain range. This local extraction strategy preserves edge information used to analyze tumor invasion while reducing the amount of data required for subsequent computation and eliminating interference from irrelevant background noise in feature extraction.

[0043] Simultaneously, the system performs structured mapping processing on clinical diagnostic reports associated with medical images. Original diagnostic reports typically exist in unstructured natural language text form, making them difficult to directly use for computer retrieval or model training. The system utilizes regular expression matching or natural language processing algorithms to scan key fields in the text reports. The system extracts core diagnostic elements, including pathological type, tumor stage, gene mutation status, and metastasis, and maps these elements to a predefined standard data dictionary. For example, it converts the specific cancer type described in the text into a standard classification code. The extracted information is encapsulated into a structured key-value pair data structure, serving as the ground truth label for the image data, ensuring an accurate correspondence between image features and clinical semantics.

[0044] See attached document Figure 3 This invention provides a method for tumor image information aggregation based on feature fusion. Step S2 specifically includes constructing and executing a deep neural network with a two-stream architecture. This network consists of a shallow visual feature extraction branch, a deep semantic feature extraction branch, and a multi-scale feature fusion module. The system uses the preprocessed 3D image patches from step S1 as input data to the network.

[0045] The system first transmits the input 3D image patches in parallel to a shallow visual feature extraction branch and a deep semantic feature extraction branch. The shallow visual feature extraction branch is configured to focus on capturing high-frequency details of the image, including the texture roughness of tumor tissue, edge sharpness, and changes in local gray-level gradients. This branch consists of stacked shallow convolutional neural network layers, specifically three to five consecutive 3D convolutional layers. Each convolutional layer uses a small-sized convolutional kernel, such as a 3x3x3 voxel kernel, with a stride of 1, and either no pooling layers or only max-pooling layers with a stride of 1, to prevent the spatial resolution of the feature map from being oversampled. Through this design, the shallow branch outputs a shallow feature map with high spatial resolution but fewer channels, which preserves the subtle morphological structural information in the original image.

[0046] Meanwhile, the deep semantic feature extraction branch processes the same input image patch, aiming to extract abstract pathological semantic features, such as the overall shape category of the tumor, internal heterogeneity patterns, and its topological relationship with surrounding blood vessels. This branch uses a deep residual network ResNet-50 or a densely connected network DenseNet-121 as its backbone architecture. The backbone network contains multiple cascaded residual blocks or dense blocks, with downsampling layers or pooling layers with a stride of 2 inserted between levels. As the number of network layers increases, the spatial resolution of the feature map decreases progressively, while the channel dimension increases progressively, and the receptive field expands until it covers the entire input image patch. This branch ultimately outputs a deep feature map with low spatial resolution but containing highly abstract semantic information, typically with 1024 or 2048 channels.

[0047] After obtaining the shallow and deep feature maps, the system performs spatial alignment. Since the deep feature map has undergone multiple downsampling operations, its spatial size is smaller than that of the shallow feature map. The system uses trilinear interpolation or transposed convolution to upsample the deep feature map, enlarging its spatial dimension to be completely consistent with that of the shallow feature map. This step ensures pixel-level correspondence between the two different scales of features in spatial location, establishing the geometric foundation for subsequent fusion operations.

[0048] The system then performs multi-scale feature fusion. It concatenates spatially aligned deep and shallow feature maps along the channel dimension, forming a hybrid feature tensor with the number of channels equal to the sum of the two. To fuse information from different levels and reduce computational complexity, this hybrid feature tensor is input into a convolutional layer with a kernel size of 1x1x1. This convolutional layer performs linear combination and dimensionality reduction between channels, compressing the number of channels in the hybrid feature tensor to a preset dimension, such as 512. After activation function processing, the system outputs an initial fused feature vector. This vector simultaneously encodes the macroscopic semantic structure and microscopic texture details of the tumor in a single feature representation, addressing the problem of insufficient single-scale feature representation.

[0049] See attached document Figure 4 This invention provides a tumor image information collection method based on feature fusion, wherein step S3 specifically includes defining the source domain and the target domain, constructing an adversarial domain adaptive module, and performing model optimization based on a composite loss function.

[0050] The system first defines source and target domains in the dataset. The source domain contains labeled tumor medical image data whose image features conform to a baseline distribution, such as historical archived data from a specific leading hospital or a specific model of imaging equipment. The target domain contains newly added image data to be aggregated or heterogeneous image data from other medical centers with different imaging parameter settings. The system calculates the statistical distribution differences between the source and target domain data in the feature space. Due to differences in imaging physics and equipment parameters, tumors of the same pathological type exhibit significant shifts in grayscale histograms and texture features in the source and target domain images; this shift is called domain shift. This step aims to eliminate this domain shift through algorithms, making the features extracted by the feature extraction network insensitive to the equipment origin.

[0051] The system constructs a domain discriminator network, which is connected to the multi-scale feature fusion network in step S2. The domain discriminator network consists of three fully connected layers and a non-linear activation function layer. Its input receives the initial fused feature vector output by the feature fusion network. The output of the domain discriminator network is a binary classification node used to predict whether the input feature vector comes from the source domain or the target domain. During training, the system introduces a gradient reversal layer, located between the feature extraction network and the domain discriminator network. In the forward propagation phase, the gradient reversal layer identically transmits the feature vector; in the backpropagation phase, the gradient reversal layer multiplies the gradient returned by the domain discriminator by a negative constant before transmitting it to the feature extraction network. This structure forms an adversarial training mechanism: the domain discriminator strives to accurately distinguish the source domain of the features, while the feature extraction network strives to generate common features that cannot be distinguished by the domain discriminator.

[0052] The system updates network parameters by minimizing a total loss function, which is a weighted average of task classification loss and domain adversarial loss. The task classification loss is used to constrain the features generated by the feature extraction network to accurately predict the clinical diagnostic labels of tumors (calculated only for labeled source domain data); the domain adversarial loss is used to measure the ability of the domain discriminator to distinguish between the source and target domains.

[0053] The system uses the following formula to calculate the total loss function. ; in, Hyperparameters representing the strength of adaptation in the control domain; This represents the task classification loss of the source domain data; This indicates the extent of the domain's defensive losses.

[0054] Specifically, task classification loss The cross-entropy loss function is used for calculation, and the formula is as follows: ; in, Indicates the number of samples in the source domain; This indicates the total number of categories for the classification task; It is the first The source domain sample belongs to the first The actual label of the class (0 or 1); It is the first The input image of each source domain sample; The network predicts that the sample belongs to the first... The probability of a class.

[0055] Domain confrontation loss The binary cross-entropy loss function is used to calculate the loss, covering all samples in both the source and target domains, as shown in the following formula: ; in, Indicates the number of samples in the target domain; It is the first Domain labels for each sample (e.g., source domain labeled as 0, target domain labeled as 1); This indicates that the feature extraction network is effective for samples. The generated feature vector, This represents the output probability of the domain discriminator.

[0056] By optimizing the above objective function, the system maximizes the classification error of the domain discriminator while ensuring the accuracy of tumor diagnosis, forcing the feature distribution generated by the feature extraction network to align between the source and target domains, thereby obtaining domain-invariant features with cross-center robustness.

[0057] See attached document Figure 5 This invention provides a tumor image information collection method based on feature fusion, wherein step S4 specifically includes embedding channel attention units and spatial attention units in the feature extraction network, performing feature weighting and recalibration, and performing redundant feature removal according to the weight distribution strategy.

[0058] The system first receives the domain-adaptive corrected feature map from step S3 as input data. To accurately identify the feature channels crucial for tumor diagnosis from high-dimensional features, the system inputs the feature map into a channel attention unit. This unit is designed to explicitly model the dependencies between feature channels. The system performs global average pooling and global max pooling operations on the input feature map in the spatial dimension, generating two distinct spatial context description vectors. These two vectors aggregate the spatial information of the feature map; the former reflects the average response of the background, and the latter reflects the peak response of salient features. Subsequently, the system inputs these two description vectors into a shared multilayer perceptron network. This multilayer perceptron contains two fully connected layers, with an intermediate layer used to reduce dimensionality and thus the number of parameters. The system element-wise adds the two feature vectors output by the multilayer perceptron and maps the values ​​to a closed interval of 0 to 1 using a sigmoid activation function, generating a channel attention weight vector. Each scalar value in this weight vector represents the importance of the corresponding feature channel; a larger value indicates that the pathological information contained in that channel is more critical.

[0059] After acquiring the channel weights, the system performs channel recalibration of the feature map and inputs the results into the spatial attention unit. The spatial attention unit aims to identify the spatial importance of the tumor lesion region and its surrounding background region in the image. The system performs average pooling and max pooling operations on the channel-recalibrated feature map in the channel dimension, generating two two-dimensional or three-dimensional spatial feature description maps. The system concatenates these two spatial feature description maps in the channel dimension to form a dual-channel feature map. Subsequently, the system uses a convolutional layer with a kernel size of 7x7 (for two-dimensional) or 7x7x7 (for three-dimensional) to perform a convolution operation on the concatenated feature map, compressing the number of channels to 1. After processing with the sigmoid activation function, the system generates a spatial attention weight map. This weight map has the same spatial size as the original input feature map, and the value of each pixel or voxel in the map represents the contribution of that spatial location to the classification task, thereby enabling the network to focus on the tumor core region and infiltration edge, and suppress irrelevant background noise.

[0060] The system performs cascaded weighting and recalibration operations on the original feature map based on the generated channel attention weights and spatial attention weights. First, the input feature map is multiplied and weighted according to the channel dimension using the channel attention weights; second, the result is multiplied and weighted according to the spatial dimension using the spatial attention weights.

[0061] The weighted recalibration process is calculated using the following formula: ; ; in, Represents the original feature map of the input; This represents the channel weight vector generated by the channel attention unit; This represents the element-wise multiplication operation; This represents the intermediate feature map after channel weighting; This represents the spatial weight map generated by the spatial attention unit; This represents the final refined feature map after double calibration. Calculated using this formula, values ​​carrying key pathological information in the feature map are amplified, while values ​​carrying redundant information or noise are suppressed to near zero.

[0062] Finally, the system executes a redundant feature removal strategy to generate a compact feature vector. The system refines the feature map... Global average pooling is performed to flatten the feature vector into a one-dimensional feature vector. A preset feature selection threshold is also applied. (For example, 0.01). The system iterates through each element in the one-dimensional feature vector and determines its value. For values ​​less than the threshold... The system classifies any feature elements as redundant or invalid and removes or zeros them from the vector. For features with values ​​greater than or equal to the threshold, the system... The system retains the feature elements. The remaining non-zero elements after filtering form the final key feature vector. This key feature vector has the characteristics of low dimensionality and high semantic density, retaining only the representational information that is highly relevant to tumor diagnosis, thereby significantly reducing the subsequent storage space occupation and the complexity of retrieval calculations.

[0063] See attached document Figure 6 This invention provides a tumor image information collection method based on feature fusion, wherein step S5 specifically includes constructing a linked data model, performing heterogeneous database collaborative storage, and retrieval application based on the collected data.

[0064] The system first establishes a linked data model for associating medical image features with clinical diagnostic labels. The system generates a globally unique identifier for each set of processed tumor image data and its corresponding diagnostic information. This globally unique identifier, acting as a non-repeatable index key, is simultaneously assigned to the key feature vector output in step S4 and the structured diagnostic label generated in step S1. This identifier not only identifies the specific patient examination order but also implicitly contains the timestamp information of data generation and the code of the data source institution, ensuring data uniqueness in a multi-center aggregation environment. Through this globally unique identifier, the system logically binds high-dimensional numerical feature data and textual diagnostic data into the same entity, enabling them to maintain a one-to-one logical relationship even with separate physical storage.

[0065] After generating a globally unique identifier, the system performs collaborative storage operations on heterogeneous databases. The system deploys a vector database management system to store high-dimensional feature data. The system serializes the compact key feature vector output in step S4 and writes it into the index structure of the vector database. During the writing process, the system attaches the globally unique identifier as a metadata tag to the corresponding feature vector. The vector database is configured with specialized indexing algorithms, such as inverted file indexing or hierarchical small-world graph algorithms, to support fast nearest neighbor search for high-dimensional vectors.

[0066] Concurrently, a relational database management system is deployed to store structured clinical text data. The system establishes a data table containing fields for basic patient information, pathological diagnosis, gene testing results, and prognostic follow-up. The system writes the key-value pairs obtained through structured mapping in step S1 into the corresponding columns of the aforementioned data table. During the writing process, the system stores the generated globally unique identifier as the primary key in the relational database table. At this point, feature vectors are stored in the vector database, while clinical semantic information is stored in the relational database; both are linked across databases using the same globally unique identifier. This heterogeneous storage scheme ensures the computational performance of feature vector retrieval while maintaining the integrity and query flexibility of the clinical text data.

[0067] Based on the aforementioned collected and stored data, the system performs image-based retrieval and application. When the system receives a new tumor image to be diagnosed, it first calls the processing modules in steps S1 to S4 to generate a query feature vector corresponding to the new image. The system inputs this query feature vector into the vector database and calculates its Euclidean distance or cosine similarity with historical feature vectors already stored in the database. The system sorts the similarity values ​​from high to low, selects a preset number of the most similar historical feature vectors, and extracts the globally unique identifiers attached to these feature vectors.

[0068] Subsequently, the system uses the extracted globally unique identifiers as query conditions to perform a joint query operation in the relational database, retrieving detailed diagnostic reports, treatment plans, and prognostic results of historical confirmed cases corresponding to these identifiers. The system aggregates and displays this historical clinical data, providing an objective reference for the diagnosis of current cases based on similar past cases. Furthermore, the system supports reverse association analysis, which involves filtering a sample set based on specific diagnostic labels in the relational database and batch retrieving the corresponding feature vectors to statistically analyze the spatial distribution patterns of features for specific pathological types.

Claims

1. A method for collecting tumor image information based on feature fusion, characterized in that, Includes the following steps: S1. Obtain tumor medical image data from different medical institutions and corresponding clinical diagnosis reports. Perform spatial resampling and grayscale normalization processing on the tumor medical image data, and extract fixed-size three-dimensional image blocks containing tumor lesions and surrounding tissues based on the tumor center coordinates. At the same time, perform text parsing on the clinical diagnosis reports to generate structured diagnostic labels. S2. Input the three-dimensional image block into a deep neural network model containing a shallow feature extraction branch and a deep semantic extraction branch set in parallel. Fuse the low-level visual features output by the shallow feature extraction branch with the high-level semantic features output by the deep semantic extraction branch to generate an initial fused feature vector. S3. Calculate the adversarial loss between the feature distribution of the source domain data and the feature distribution of the target domain data, and adjust the parameters of the deep neural network model based on the domain adaptation mechanism to correct the initial fused feature vector; S4. Using the channel attention unit and spatial attention unit embedded in the deep neural network model, the corrected initial fusion feature vector is weighted and calibrated, and then filtered according to a preset numerical threshold to output the key feature vector. S5. Generate a unique index ID, and associate the key feature vector with the structured diagnostic label. Map the key feature vector to a vector database and the structured diagnostic label to a relational database through the unique index ID to achieve bidirectional indexing.

2. The tumor image information collection method based on feature fusion according to claim 1, characterized in that, Step S1 further includes: After performing spatial resampling and grayscale normalization, based on the center coordinates annotated by the physician or a pre-generated coarse segmentation mask, a fixed-size three-dimensional image block containing the tumor lesion and surrounding microenvironment tissue is located and extracted from the tumor medical image data for input into the deep neural network model.

3. The tumor image information collection method based on feature fusion according to claim 1, characterized in that, The specific implementation method of step S2 is as follows: The shallow feature extraction branch consists of stacked shallow convolutional neural network layers, used to generate shallow feature maps containing low-level visual features. The deep semantic extraction branch uses a deep residual network or a densely connected network as the backbone architecture to generate a deep feature map containing high-level semantic features. An upsampling operation is performed on the deep feature map to make the spatial dimension of the deep feature map consistent with the spatial dimension of the shallow feature map; The deep feature map, which has undergone upsampling, is concatenated with the shallow feature map along the channel dimension and then input into a convolutional layer for dimensionality reduction to generate the initial fused feature vector.

4. The tumor image information collection method based on feature fusion according to claim 1, characterized in that, The specific implementation method of step S3 is as follows: Define the dataset with a standard distribution as the source domain and the new data to be collected as the target domain; Construct a domain discriminator network connected to the deep neural network model, the domain discriminator network being used to predict whether the input feature vector comes from the source domain or the target domain; A gradient inversion layer is introduced between the deep neural network model and the domain discriminator network. During the backpropagation phase, the gradient inversion layer multiplies the gradient returned by the domain discriminator network by a negative constant and then passes it to the deep neural network model. By minimizing a total loss function weighted by task classification loss and domain adversarial loss, the parameters of the deep neural network model and the domain discriminator network are updated, forcing the deep neural network model to learn domain-invariant features.

5. The tumor image information collection method based on feature fusion according to claim 4, characterized in that, The task classification loss is calculated using the cross-entropy loss function, which is used to constrain the features generated by the deep neural network model to accurately predict the clinical diagnostic labels of the source domain data; the domain adversarial loss is calculated using the binary cross-entropy loss function, which is used to measure the ability of the domain discriminator network to distinguish between the source domain and the target domain.

6. The tumor image information collection method based on feature fusion according to claim 1, characterized in that, The specific implementation method of step S4 is as follows: The corrected features are input into the channel attention unit to generate a channel attention weight vector; The corrected features are multiplied and weighted by the channel dimension using the channel attention weight vector to obtain an intermediate feature map after channel weighting. The intermediate feature map is input into the spatial attention unit to generate a spatial attention weight map; The intermediate feature map is multiplied and weighted by spatial dimension using the spatial attention weight map to obtain a refined feature map; The refined feature map is subjected to global average pooling and flattened into a one-dimensional feature vector. Based on the preset numerical threshold, feature elements with values ​​lower than the numerical threshold are removed, and the key feature vector is output.

7. The tumor image information collection method based on feature fusion according to claim 6, characterized in that, The channel attention unit operates as follows: Global average pooling and global max pooling operations are performed on the input features in the spatial dimension to generate two spatial context description vectors. The two spatial context description vectors are input into a shared multilayer perceptron network; The two feature vectors output by the multilayer perceptron network are added element-wise and then processed by the sigmoid activation function to generate the channel attention weight vector.

8. The tumor image information collection method based on feature fusion according to claim 6, characterized in that, The spatial attention unit operates as follows: The input features are subjected to average pooling and max pooling operations along the channel dimension to generate two spatial feature description maps. The two spatial feature description maps are stitched together along the channel dimension; A convolutional layer is used to perform a convolution operation on the concatenated features and compress the number of channels to a range of 1 to a preset value. Then, the spatial attention weight map is generated by processing it with the Sigmoid activation function.

9. The tumor image information collection method based on feature fusion according to claim 1, characterized in that, The specific implementation method of step S5 is as follows: The unique index ID is a globally unique identifier that includes timestamp information and the code of the data source organization; The key feature vector, together with the globally unique identifier, is written as a metadata tag into the vector database configured with an inverted file index or a hierarchical small world graph algorithm. The structured diagnostic labels are written into the data table of the relational database, and the globally unique identifier is used as the primary key of the data table.

10. The tumor image information collection method based on feature fusion according to claim 1, characterized in that, The tumor image information collection method based on feature fusion also includes a retrieval step based on the collected data: Receive a new tumor image to be diagnosed, and call the processing flow from step S1 to step S4 to generate a query feature vector corresponding to the new tumor image; Calculate the Euclidean distance or cosine similarity between the query feature vector and the historical feature vectors stored in the vector database, filter out the most similar historical feature vectors, and extract the unique index ID attached to the most similar historical feature vectors; Using the extracted unique index ID, a query operation is performed in the relational database to retrieve detailed diagnostic reports, treatment plans, and prognostic results of historical confirmed cases corresponding to the unique index ID.

Images