Animal X-ray medical image data processing method and computer device

Abstract

The present invention relates to the field of animal X-ray medical image data processing, and specifically to an animal X-ray medical image data processing method and computer device, which improves the ability to understand image data and the accuracy of data processing results. The method includes: constructing a joint multi-task animal X-ray multimodal data set, establishing an X-ray image data set specifically for animals, including multimodal information such as structural anatomy, lesion annotation and diagnostic text, covering multiple lesion classifications such as fractures, foreign bodies, tumors, etc., to provide high-quality training and verification data for deep learning models; generating multimodal data set labels; performing data preprocessing on multimodal data sets; constructing a multi-task learning model; implementing feature sharing and feature fusion strategies of the multi-task learning model; training the multimodal multi-task learning model; and evaluating the performance of the multimodal multi-task learning model. The present invention is suitable for animal X-ray medical image data processing.

Description

Technical Field

[0001] The present invention relates to the field of animal X-ray medical image data processing, and in particular to an animal X-ray medical image data processing method and a computer device. Background Art

[0002] The processing of animal medical imaging data faces the challenges of limited data size, difficult labeling, and complex and diverse lesions, which further affect the construction and optimization of deep learning models. At the same time, there are significant differences between animal X-ray images and human medical images in terms of lesion distribution and tissue structure, making it difficult for existing models that are mostly trained based on human images to be directly applied to animal medical images. In addition, the diverse manifestations of different diseases and the uneven distribution of imaging features also require the model to have higher flexibility and adaptability in dealing with lesion differences.

[0003] On the other hand, multimodal joint analysis methods for animals such as cats and dogs are still relatively lacking. There are few applications in animal graphic data processing. Most existing methods are based on human image training, and there are few applications in graphic diagnosis of animal medical images. Most existing models lack specific cross-modal optimization strategies.

[0004] In summary, the current veterinary medical imaging data processing solutions have problems such as low data processing efficiency, low ability to understand data, and inaccurate data processing results. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an animal X-ray medical image data processing method and computer device to improve the efficiency of data processing, the ability to understand image data and the accuracy of data processing results.

[0006] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. In a first aspect, the present invention provides an animal X-ray medical image data processing method, comprising:

[0007] S1. Construct a joint multi-task animal X-ray multimodal dataset;

[0008] S2, generate multimodal dataset labels;

[0009] S3, preprocessing the multimodal dataset;

[0010] S4. Build a multi-task learning model;

[0011] S5. Implement feature sharing and feature fusion strategies for multi-task learning models;

[0012] S6. Training the multimodal multi-task learning model;

[0013] S7. Evaluate the performance of the multimodal multi-task learning model.

[0014] Furthermore, step S1 specifically includes:

[0015] X-ray imaging equipment is used to collect X-ray imaging data of animals, which includes bone injury, soft tissue injury, foreign matter, tumor and lesion imaging data, and the collected X-ray imaging data is desensitized.

[0016] Furthermore, step S2 specifically includes:

[0017] Generate segmentation task labels: Use Python and OpenCV to perform data preprocessing and label format conversion to generate pixel-level mask labels for segmentation tasks. The mask labels are annotated using the annotation tool Labelbox. Each pixel is assigned a unique integer value based on the anatomical or lesion area in which it is located. The generated mask is saved in a single-channel image format and maintains the same size as the original image file.

[0018] Generate detection task labels: Use the LabelImg tool to annotate the lesion area in the image with a bounding box, record the coordinate information of the bounding box, manually annotate the category of each bounding box and the optional confidence score, and store the label data as a JSON file in COCO format;

[0019] Generate classification task labels: Use the Pandas library to organize classification labels and record the classification information of the detected lesion area, including whether the fracture type is closed or open and whether the tumor is benign or malignant. Encode the label data in one-hot encoding format and store it in a CSV file. The classification label file and the detection label file maintain the same lesion area index to ensure label alignment of the branch tasks.

[0020] Furthermore, step S2 specifically includes:

[0021] Label and image batch processing: Design a custom data loader through PyTorch's DataLoader module to support batch processing and load images and multi-task labels synchronously. Use the Albumentations data augmentation library to perform augmentation operations on X-ray images, ensuring that segmentation masks, detection bounding boxes, and classification labels are automatically transformed synchronously during the augmentation process.

[0022] Generate a diagnostic report and give the corresponding diagnostic report text for each X-ray image.

[0023] Furthermore, step S3 specifically includes:

[0024] X-ray image data preprocessing:

[0025] De-noising the X-ray image data, using the median filter algorithm to reduce random noise interference. The median filter algorithm replaces the value of each pixel with the median of the pixel values ​​in the neighborhood to eliminate salt noise and pepper noise in the image;

[0026] Histogram equalization technology is used to optimize the brightness distribution of X-ray image data;

[0027] After histogram equalization, contrast stretching is performed on the X-ray image data;

[0028] Finally, a normalization process is performed to standardize the pixel values ​​of the X-ray image data to a set range;

[0029] Text preprocessing:

[0030] Clean the original diagnostic report text to remove extra spaces, line breaks, and character noise. For text containing non-standard medical terms and redundant information, remove them through keyword matching;

[0031] Medical terms in the diagnostic report text are standardized through the medical terminology database, and synonymous or near-synonymous medical terms are expressed in a unified manner. This process is performed using Python language and PyMedTerm library.

[0032] The diagnostic report text is segmented into sentences and punctuation is corrected using the NLTK library, which separates and reconstructs the diagnostic points in the text through syntactic analysis;

[0033] Extract all the words in the diagnosis report text to get a dictionary describing the content of the X-ray image data, use the jieba tool to segment each diagnosis report text, and use spaces to replace the characters in the diagnosis report text. Add the start and end before and after each manually annotated diagnosis report text, fill in unknown words, and fill all the diagnosis report texts to the maximum length, count and encode each segmented word;

[0034] All extracted words are encoded into numerical labels, and the feature value of each medical entity is mapped to a unique numerical code and stored in a CSV file using one-hot encoding;

[0035] Finally, the words of the diagnostic report text are expanded using synonyms to generate more expressive text.

[0036] Furthermore, step S4 specifically includes:

[0037] Build the backbone network:

[0038] Select the EfficientNet network as the backbone network, combine the features of different levels in the EfficientNet network with the feature pyramid, and provide them to the segmentation and detection branches respectively. The low-level features are provided to the segmentation branch, focusing on local edges and detail information, and the high-level features are provided to the detection branch, focusing on global semantic information. Multi-level feature maps are generated through the EfficientNet network;

[0039] Build the split branch:

[0040] Improved based on U-Net architecture;

[0041] Encoder part: The features generated by the EfficientNet network are combined with the feature vector after the multi-task feature sharing and decoupling module as the encoder input of the segmentation branch to reduce computational redundancy;

[0042] Skip connection: The features of the backbone network are passed to the decoder through skip connections;

[0043] Decoder part: gradually restores the image spatial resolution and outputs a segmentation mask consistent with the input size for annotating anatomical structures and lesion areas;

[0044] Build the detection branch:

[0045] The Fast R-CNN framework is used to locate and classify the lesion area in the X-ray image. Fast R-CNN realizes the location and category recognition of the lesion area by combining the features extracted by the EfficientNet network with the features after multi-task feature sharing and decoupling modules. The process is as follows:

[0046] Region Proposal Network: Generates candidate boxes through the features generated by the EfficientNet network and feature pyramid, providing input for the positioning of the lesion area;

[0047] ROI pooling: Perform ROI pooling operation on each candidate box area to standardize the feature size;

[0048] Bounding box regression and classification: perform bounding box regression and classification on the candidate box, and output the coordinates, category label and confidence of the lesion area;

[0049] Build the classification branch:

[0050] Feature extraction: Obtain feature vectors of the lesion area from the backbone network and detection branches;

[0051] Fully connected layer classification: The feature vector is mapped to the set category label through the fully connected layer, and the detailed attributes of the lesion are output;

[0052] Construct feature fusion layer:

[0053] A feature fusion layer is set after the segmentation branch, detection branch, classification branch and EfficientNet network trunk. The original image features extracted by EfficientNet are integrated with the multi-task features through the feature fusion layer to provide complete multimodal information for the report generation module.

[0054] Build the report generation module:

[0055] The report generation module uses GPT-2 and utilizes fused feature vectors to generate diagnostic text that meets medical standards.

[0056] Furthermore, step S5 specifically includes:

[0057] A multi-task feature sharing and decoupling module is added after the backbone network. The multi-task feature sharing and decoupling module is specifically used to:

[0058] Share low-level features: Let segmentation, detection, and classification tasks share the low-level features of the backbone network;

[0059] Feature Pyramid Network: Utilizes the pyramid structure of multi-scale feature maps to generate feature resolutions suitable for different tasks. Segmentation tasks utilize low-level features, while detection and classification tasks use high-level features.

[0060] Task-specific convolutional layer: In the decoupling module, based on the feature pyramid network, a dedicated convolutional layer is set for each task. By applying the task-specific convolutional layer on the feature map output by the feature pyramid network, the segmentation, detection and classification tasks obtain independent feature spaces based on shared features, reducing feature competition and interference between tasks. The dedicated convolutional layer of each task is set according to its feature requirements;

[0061] Set up a dynamic task weight adjustment mechanism to dynamically adjust the weight of each task according to the training progress;

[0062] Total loss function for:

[0063] ;

[0064] in, is the loss function for the segmentation task, is the loss function for the detection task, is the classification task loss function, is the weight parameter of the segmentation task, is the weight parameter of the detection task, is the weight parameter of the classification task;

[0065] After the segmentation, detection, and classification branches complete their tasks, the fused features are input into the report generation module, which performs contextual analysis on the fused features through a hierarchical self-attention mechanism to generate a coherent diagnostic report text.

[0066] Furthermore, step S6 specifically includes:

[0067] Backbone network initialization: initialized by loading weights pre-trained on ImageNet;

[0068] Task branch initialization: For segmentation, detection and classification tasks, the parameters of each task branch are initialized respectively. Kaiming initialization is used to randomly assign weight values ​​to each branch.

[0069] Initial weight assignment: Assign initial task weights and set weight values ​​for segmentation, detection, and classification tasks respectively to control the influence of different tasks in the multi-task loss function;

[0070] Dynamic weight adjustment mechanism: Design a dynamic weight adjustment strategy based on the task loss convergence situation. The multi-task learning model adjusts the weight value in real time according to the loss decline rate of each task;

[0071] Optimizer selection: Adopt Adam optimizer, which improves the efficiency and stability of training by adaptively adjusting the learning rate of each parameter;

[0072] Learning rate strategy: Set the initial learning rate and adopt a warm-up learning rate strategy to gradually increase the learning rate in the initial stage, and then apply cosine annealing learning rate decay in the stable stage to gradually guide the convergence of model parameters;

[0073] Pre-training and integration of report generation module:

[0074] Perform causal language modeling pre-training on the report generation module based on GPT-2;

[0075] The loss function is defined as follows:

[0076] ;

[0077] N represents the total length of the sequence, represents all tokens before the i-th token, is the target token;

[0078] Multi-task feature fusion and text generation integrated training:

[0079] After the backbone network completes image feature extraction and each branch task, the feature fusion layer integrates the image feature vectors extracted by the segmentation, detection and classification branches and the original backbone network into a unified multi-scale semantic feature vector and inputs it into the report generation module.

[0080] Furthermore, step S7 specifically includes:

[0081] The performance of the multimodal multi-task learning model is evaluated. Based on the evaluation results, the hyperparameters are optimized and adjusted to seek the best multimodal multi-task learning model configuration, including learning rate, batch size, and optimizer type. For the feature decoupling and sharing strategies in the multi-task learning model, the feature map exchange and fusion mechanism is continuously optimized based on performance evaluation feedback.

[0082] In a second aspect, the present invention provides a computer device, including a memory, wherein the memory stores program instructions, and when the program instructions are executed, the above-mentioned animal X-ray medical image data processing method is executed.

[0083] The beneficial effects of the present invention are:

[0084] The present invention constructs a joint multi-task animal X-ray multimodal dataset, which contains multimodal information such as structural anatomy, lesion annotation and diagnostic text, covering multiple lesion classifications such as fractures, foreign bodies, tumors, etc., providing high-quality training and verification data for deep learning models.

[0085] The present invention uses EfficientNet as the backbone feature extraction network, integrates the improved U-Net segmentation branch and Faster R-CNN positioning branch, respectively realizes anatomical structure segmentation and lesion detection data processing, and ensures efficient collaboration of segmentation, detection and text generation tasks through shared features and feature fusion strategies, thereby improving data processing efficiency.

[0086] The present invention designs a dynamic feature sharing and task weight control strategy, dynamically adjusts the loss weight based on the training progress of each task, and ensures the balance of different tasks in training. Through multi-task feature fusion and feature map exchange mechanism, it promotes knowledge sharing between tasks and improves the overall performance of the model.

[0087] The present invention can realize efficient and coordinated processing of multiple tasks such as image segmentation, detection, classification and diagnostic text generation, enhance the model's ability to recognize complex lesions and the accuracy of data processing, significantly improve the automation level of image analysis and diagnostic consistency, and provide technical support for the automation and intelligent data processing of animal medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 The present invention provides a flowchart of an animal X-ray medical imaging data processing method. DETAILED DESCRIPTION

[0089] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0090] The present invention provides a method for processing animal X-ray medical imaging data, such as Figure 1 As shown, specifically including:

[0091] S1. Construct a joint multi-task animal X-ray multimodal dataset;

[0092] X-ray imaging equipment is used to collect X-ray imaging data of animals, which includes bone injury, soft tissue injury, foreign matter, tumor and lesion imaging data, and the collected X-ray imaging data is desensitized.

[0093] S2, generate multimodal dataset labels;

[0094] The present invention saves the original X-ray image data in high-resolution JPG image format to maintain the richness and clarity of image details. The labeling of the data set of the present invention is implemented on the PyTorch deep learning framework, and the operating environment is a high-performance computing server that supports CUDA acceleration for processing medical image data, specifically including the following steps:

[0095] Segmentation task label generation: Use Python and OpenCV for data preprocessing and label format conversion to generate pixel-level mask labels for segmentation tasks. The mask is manually annotated by animal X-ray medical imaging experts using the annotation tool Labelbox to ensure the accuracy of the lesion area and anatomical structure. Each pixel is assigned a unique integer value (such as bone = 1, soft tissue = 2, lesion = 3) according to its anatomical or lesion area. The generated mask is saved in a single-channel image format (PNG) and keeps the same size as the original image file for efficient loading and processing in the deep learning framework.

[0096] Detection task label generation: Use the LabelImg tool to annotate the lesion area in the image with a bounding box and record the coordinate information of the bounding box in the format of [xmin, ymin, xmax, ymax]. In addition, annotate the category of each bounding box (such as fracture, mass, inflammation) and an optional confidence score (a value between 0 and 1). These label data are stored as a JSON file in COCO format so that they can be read by the object detection task in the deep learning framework. The format of the label file meets the input requirements of the FasterR-CNN object detection model.

[0097] Classification task label generation: Use the Pandas library to organize classification labels and record refined classification information of detected lesion areas, such as fracture type (closed or open), tumor nature (benign or malignant), etc. Label data can be further encoded into one-hot encoding format and stored in a CSV file for subsequent model input reading and processing. The classification label file maintains the same lesion area index as the detection label file to ensure label alignment of branch tasks.

[0098] Label and image batch processing: Design a custom data loader through PyTorch's DataLoader module to support batch processing and load images and multi-task labels synchronously. Use the Albumentations data augmentation library to perform augmentation operations on images (such as rotation, scaling, etc.), ensuring that segmentation masks, detection bounding boxes, and classification labels are automatically transformed synchronously during the augmentation process to maintain the correctness of task labels.

[0099] For the diagnostic report generation task, a corresponding diagnostic report text needs to be given for each X-ray image. This task can be performed by multiple professional veterinary imaging experts to give the corresponding diagnostic text report. The report text is saved in a CSV file, which contains the diagnostic text corpus of 112 conditions such as fractures, rectal dilatation, pneumonia, gallstones, emphysema, etc., for use in the diagnostic report generation task.

[0100] S3, preprocessing the multimodal dataset;

[0101] Image preprocessing:

[0102] In the steps of the present invention, a series of data preprocessing operations are performed on the constructed X-ray image data set to optimize data quality and improve image clarity, thereby improving the recognition accuracy of the subsequent deep learning model. This step includes denoising, histogram equalization, contrast stretching and normalization, as follows:

[0103] The present invention first denoises the X-ray image data and uses a median filter algorithm to reduce random noise interference while retaining image edges and key details. The median filter replaces the value of each pixel with the median of the pixel value in the neighborhood, eliminating salt noise and pepper noise in the image, improving image clarity and pathological feature recognition, and ensuring that the model accurately locates the lesion area.

[0104] Then, histogram equalization technology is applied to optimize the image brightness distribution in order to enhance the contrast. Histogram equalization makes the pathological features and anatomical structures in the image clearer by adjusting the distribution of pixel gray values, especially enhancing the details in low-contrast areas, providing clearer basic image data for subsequent feature extraction.

[0105] After histogram equalization, the present invention also performs contrast stretching on the image data. By expanding the grayscale range of the image, the contrast between the lesion area and normal tissue is effectively improved, which helps to clearly present key features during the deep learning model training and reasoning process, thereby improving the accuracy and stability of the data processing results.

[0106] Finally, normalization is performed to standardize the image pixel values ​​to the set range [0,1] to ensure the scale consistency of different image input features. Normalization can effectively reduce the impact of grayscale range differences of different images on model training, thereby accelerating model convergence, improving training stability, and ensuring data processing consistency at the model input stage.

[0107] Text preprocessing:

[0108] In the steps of the present invention, a systematic text preprocessing process is formulated to improve the quality and consistency of X-ray image diagnosis text generation. The process aims to clean, standardize and structure the original diagnosis text to ensure that the input text data of the subsequent model meets the expression requirements of medical language. The main steps include the following:

[0109] First, the original diagnosis text is cleaned to remove extra spaces, line breaks, special characters and other noise to ensure the neatness of the text data. Texts containing non-standard medical terms and redundant information (such as diagnosis date, medical record number, etc.) are removed through keyword matching to ensure the simplicity and coherence of the diagnosis content.

[0110] Medical terminology is standardized in the text through the medical terminology database, and synonymous or near-synonymous medical terms are expressed in a unified manner. This process is performed using Python language and PyMedTerm library to ensure the accuracy and consistency of medical terminology.

[0111] The diagnostic text is segmented into sentences and punctuation is corrected to ensure that each sentence is complete. Sentence segmentation and punctuation correction ensure that the generated diagnostic report is clear and easy to read. This step can be implemented using the NLTK library, which separates and reconstructs the diagnostic points in the text through syntactic analysis.

[0112] Extract all the words in the report to get a dictionary describing the content of the X-ray image. Specifically, use the jieba tool to segment each report in the dataset and use spaces to replace the characters in the report. In order to facilitate the model to better recognize the report sentences, add the beginning " <start>"Peace and Ending" <end>" indicates the beginning and end of a sentence, and unknown words are represented by " <unk>"fill and use all report lengths" <end>" is padded to the maximum report length, and each word is counted and encoded.

[0113] All extracted words are encoded as numerical labels to facilitate the use of multi-task learning models. The feature value of each medical entity is mapped to a unique numerical code and stored in a CSV file using one-hot encoding to ensure compatibility and consistency of model training.

[0114] In order to increase the robustness of the model and the diversity of text samples, synonym expansion is used to generate more natural expressions. For example, the expression of the term "fracture" can be supplemented with commonly used descriptors such as "break" or "injury". This diversified expression is achieved through the SynonymNet synonym expansion tool, which effectively enhances the model's understanding and generalization capabilities of different natural expressions.

[0115] S4. Build a multi-task learning model;

[0116] In the multi-task learning framework design of the present invention, the efficient and accurate EfficientNet is used as the backbone network to provide powerful feature extraction capabilities. With the help of the deep feature extraction module of EfficientNet, the present invention can obtain rich multi-level features from the input animal X-ray images, providing a solid foundation for subsequent segmentation, detection, classification and report generation tasks. The specific design is as follows:

[0117] Backbone network design:

[0118] The present invention selects EfficientNet as the backbone network. Based on its excellent composite scaling strategy, it can balance the network depth, width and resolution, thereby maximizing the efficiency and effect of feature extraction under limited computing resources. The features at different levels in the EfficientNet network are combined with the feature pyramid and provided to the segmentation and detection branches respectively. Low-level features are provided to the segmentation branch, focusing on local edges and detail information; high-level features are provided to the detection branch, focusing on global semantic information. Through the multi-level feature maps generated by EfficientNet, the backbone network provides a shared feature basis for subsequent segmentation, detection, classification and report generation branches, ensuring efficient capture of anatomical structures and pathological features in X-ray images.

[0119] Split Branch Design:

[0120] In order to achieve accurate segmentation of anatomical structures and lesion areas in X-ray images, the present invention has made an improved design based on the U-Net architecture. The modified U-Net combines the feature map of EfficientNet with the encoding-decoding structure of U-Net, so that the model has excellent segmentation detail restoration capabilities.

[0121] The specific design of the split branch is as follows:

[0122] Encoder part: The features generated by the EfficientNet network are combined with the feature vector after the multi-task feature sharing and decoupling module as the encoder input of the segmentation branch to reduce computational redundancy.

[0123] Skip connections: The multi-scale features of the backbone network are passed to the decoder through skip connections, ensuring that image details of different scales are preserved during the resolution restoration process.

[0124] Decoder part: gradually restores the image spatial resolution and outputs a segmentation mask consistent with the input size for annotating anatomical structures and lesion areas.

[0125] Detection branch design:

[0126] In the detection task, the present invention uses the Fast R-CNN framework to locate and classify the lesion area in the X-ray image. Fast R-CNN realizes the location and category recognition of key lesion areas by combining the features extracted by the EfficientNet network with the features after multi-task feature sharing and decoupling modules. The process is as follows:

[0127] Region Proposal Network (RPN): Generates candidate boxes through the features generated by the EfficientNet network and feature pyramid, providing input for the positioning of the lesion area.

[0128] ROI pooling: Perform ROI pooling operation on each candidate box area to standardize the feature size.

[0129] Bounding box regression and classification: Perform bounding box regression and classification on the candidate boxes, and output the coordinates, category label and confidence of the lesion area.

[0130] Classification branch design:

[0131] In order to refine the attribute classification of the lesion area, the present invention designs a simple fully connected layer to further classify the detected lesion area. The design is as follows:

[0132] Feature extraction: Obtain the feature vector of the lesion area from the backbone network and the detection branch.

[0133] Fully connected layer classification: The feature vector is mapped to a specific category label through the fully connected layer, and the detailed attributes of the lesion (such as fracture type or tumor nature) are output to provide detailed information for diagnosis.

[0134] Feature fusion layer:

[0135] A feature fusion layer is set after the segmentation branch, detection branch, classification branch and EfficientNet network backbone. The feature fusion layer integrates the original image features extracted by EfficientNet with the multi-task features to provide complete multimodal information for the report generation module.

[0136] Report generation module:

[0137] GPT-2 uses autoregressive generation capabilities to generate diagnostic text that meets medical standards using a fused feature vector that combines multi-task and original image features, capturing the interrelationships of multi-task features and integrating them into a coherent natural language description, thereby ensuring the accuracy and consistency of the diagnostic text, covering all key information identified in the image;

[0138] S5. Implement feature sharing and feature fusion strategies for multi-task learning models;

[0139] The multi-task learning framework of the present invention realizes the collaborative optimization of segmentation, detection, classification and report generation tasks through strategies such as sharing and decoupling features, dynamic task weight adjustment, and multi-scale feature fusion. The specific design is as follows:

[0140] Multi-task feature sharing and decoupling module:

[0141] In order to avoid feature conflicts between tasks and meet the different requirements of segmentation, detection, and classification tasks, the present invention designs a multi-task feature sharing and decoupling module after the backbone network, which includes the following functions:

[0142] Shared low-level features: Allow segmentation, detection, and classification tasks to share low-level features of the backbone network (such as edges and textures), thereby providing basic visual understanding for each task and reducing computational redundancy.

[0143] Feature Pyramid Network: Utilizes the pyramid structure of multi-scale feature maps to generate feature resolutions suitable for different tasks. Segmentation tasks use low-level (high-resolution) features, while detection and classification tasks use high-level (low-resolution) features to achieve detailed expression at different scales.

[0144] Task-specific convolutional layers: Based on the feature pyramid network in the decoupling module, dedicated convolutional layers are designed for each task to further enhance the feature focusing capability of specific tasks. By applying these task-specific convolutional layers on the feature graph output by the feature pyramid network, segmentation, detection, and classification tasks can obtain independent feature spaces based on shared features, reducing feature competition and interference between tasks. The dedicated convolutional layer of each task can be refined according to its feature requirements to better meet the specific feature requirements of each task while maintaining shared low-level features. For the feature graph output by the feature pyramid network , the dedicated convolutional features for each task are:

[0145]

[0146] Through task-specific convolution, we ensure that each task can focus on specific feature requirements and reduce the interference caused by feature competition.

[0147] Dynamic task weight adjustment:

[0148] In multi-task learning, different tasks have different convergence speeds and difficulties, which may cause the loss of some tasks to dominate the optimization process. To avoid this phenomenon, the present invention designs a dynamic task weight adjustment mechanism to dynamically adjust the weight of each task according to the training progress. The total loss function is:

[0149]

[0150] in, , , is a dynamically adjusted weight parameter. During the training process, the model optimizes these weights to make the training focus on tasks that are difficult to converge, thereby improving the overall performance of multiple tasks. is the weight parameter of the segmentation task, is the weight parameter of the detection task, is the weight parameter for the classification task.

[0151] Attention Mechanism:

[0152] In order to further optimize the selectivity of features, the present invention adds a multi-task attention mechanism to the task-specific feature stream, improves the key feature response of each task through attention weights, and reduces redundant features. The specific design is as follows:

[0153] Multi-task attention module: Adaptive attention mechanism is added to the dedicated feature map of each task, so that the model can focus on specific important areas during segmentation, detection and classification. For example, for segmentation tasks, the attention mechanism strengthens the capture of anatomical structures; for detection tasks, the attention mechanism enhances the edge features of the lesion area. The attention-weighted feature is calculated as:

[0154]

[0155]

[0156]

[0157] In each task (such as segmentation, detection, and classification), a specific attention mechanism weights the features to increase the model's focus on key areas, thereby optimizing the performance of specific tasks. This independent attention design increases the focus of each task on key information and ensures the model's feature utilization efficiency. Represents the input features or feature maps of the task (such as segmentation, detection, classification), which contain information extracted from the original image. is the attention weight of the task, representing , as well as , which is a weight calculated by the attention mechanism, reflecting the area or feature that the model needs to focus on in this task. These weights can be regarded as selective amplification or suppression of input features. is the feature map after applying the attention mechanism, representing , as well as , the feature map after attention weighting will strengthen important feature information while weakening irrelevant or minor information.

[0158] Multi-source information fusion and report generation module docking:

[0159] After the segmentation, detection, and classification branches complete their tasks, the image features pass through the multi-source information fusion layer, integrate multi-scale information, and pass these features as input to the report generation module. The report generation module uses a decoder model based on GPT-2. GPT-2 processes fused features through its multi-layer self-attention mechanism, deeply understands the contextual information of these features, and generates a coherent and well-structured data processing result report text. This design ensures efficient integration of segmentation, detection, and classification information, and provides deep semantic support for report generation, improving the comprehensiveness and accuracy of the report.

[0160] Through the above strategy design, the present invention achieves efficient collaborative optimization of segmentation, detection, classification and report generation tasks, while improving the accuracy of a single task and ensuring the overall performance of the multi-task framework.

[0161] S6. Training the multimodal multi-task learning model;

[0162] In the initial training stage of the animal X-ray imaging diagnosis task, in order to ensure the effective startup of the multi-task model and the coordinated optimization between the tasks, the present invention performs detailed initialization settings on the model, specifically including the following steps:

[0163] Parameter initialization of the backbone network and task branches:

[0164] Backbone network initialization: EfficientNet is used as the backbone network, and the weights pre-trained on ImageNet are loaded to provide excellent feature extraction capabilities. Through the transfer learning method, the model can directly use rich image features, which helps to reduce training time and improve diagnostic accuracy.

[0165] Task branch initialization: For segmentation, detection, and classification tasks, each task branch is initialized with dedicated parameters. Kaiming initialization is used to randomly assign weight values ​​to each branch to ensure stability in the early stages of training and prevent gradient anomalies.

[0166] Dynamic task weight initialization:

[0167] Initial weight allocation: To ensure that the model pays balanced attention to each task in the early stage of multi-task training, initial task weights are allocated, and weight values ​​are set for segmentation, detection, and classification tasks respectively. =0.34, =0.33 and = 0.33, which is used to control the influence of different tasks in the multi-task loss function.

[0168] Dynamic weight adjustment mechanism: A dynamic weight adjustment strategy based on task loss convergence is designed. The model adjusts the weight value in real time according to the loss decline rate of each task. The weight value is gradually increased for tasks with slower loss decline, and the optimization effect of more difficult tasks is prioritized to reduce the training imbalance problem between multiple tasks.

[0169] Optimizer and learning rate schedule:

[0170] Optimizer selection: Choose the Adam optimizer, which is particularly suitable for multi-task learning frameworks due to its adaptability and can effectively deal with the gradient differences of different task branches. The automatic adjustment function of the Adam optimizer ensures stability during training.

[0171] Learning rate strategy: The initial learning rate is set to 0.001, and a warm-up learning rate strategy is used to prevent gradient instability in the early stages of training. The learning rate is gradually increased in the early stages, and then cosine annealing learning rate decay is applied in the stable stage to gradually guide the convergence of model parameters.

[0172] Pre-training and deep fine-tuning of the report generation module:

[0173] On a large-scale medical text dataset, the report generation module based on GPT-2 is pre-trained with an autoregressive language model (CLM). The CLM pre-training strategy requires the model to predict the next word given the context of the previous text, aiming to lay the foundation for the report generation module to learn medical terminology, language expression, and contextual relevance. Through this pre-training method, GPT-2 can better understand and generate text related to the medical field when generating medical reports, ensuring the accuracy and professionalism of the reports.

[0174] The loss function is defined as follows:

[0175] ;

[0176] N represents the total length of the sequence, represents all tokens before the i-th token, is the target token;

[0177] Through CLM pre-training, the model can gradually master the characteristic expressions of medical language, laying a semantic foundation for generating high-quality diagnostic texts.

[0178] For deep fine-tuning, in the first stage of training, the GPT-2 model does not participate in the overall training process of the task. It only trains the backbone network and each branch to ensure that the feature fusion layer can generate high-quality and reliable multi-scale semantic features.

[0179] In the second stage of training, the backbone network and its branches that have been trained in the first stage are frozen, and then the pre-trained GPT-2 model is deeply fine-tuned to enable it to better adapt to the task requirements and generate accurate and coherent report texts.

[0180] Multi-task feature fusion and text generation deep fine-tuning training:

[0181] After the backbone network completes image feature extraction and each branch task, the feature fusion layer integrates the features extracted by these branches with the original image features extracted by EfficientNet into a unified multi-scale semantic feature vector. This feature vector is mapped to the text space through a multi-layer fully connected layer MLP to obtain the prompt vector Prefix embeddings and input it into the report generation module. This fused feature helps the model to comprehensively utilize image features when generating diagnostic reports, thereby generating high-quality and medically meaningful coherent diagnostic text.

[0182] During the deep fine-tuning training phase of text generation, the pre-trained GPT-2 model is deeply fine-tuned by introducing Prefixembeddings as training input. Through this process, GPT-2 learns to generate medical reports that are highly relevant to the image content based on the fused multi-scale semantic features. The fine-tuning process not only optimizes the model's language generation capabilities, but also ensures the accuracy of the generated text in medical terms and context, improving the professionalism and consistency of the report.

[0183] Text generation task loss function:

[0184] Cross-Entropy Loss

[0185] Cross entropy loss is used to calculate the gap between the generated text and the real text. By minimizing the cross entropy loss, the model can learn how to generate content similar to the real report text.

[0186] Assume that the probability distribution generated by the model is , that is, given the context Predict the current word , and the true label is , then the cross entropy loss is defined as:

[0187] ;

[0188] T is the length of the report text, Is the model given context Time prediction word The probability of represents the true label of the t-th word.

[0189] S7. Evaluate the performance of the multimodal multi-task learning model.

[0190] In step 7 of the present invention, the performance of the multi-task learning model on the validation set and test set is carefully analyzed to gain a deeper understanding of the model's generalization ability and clinical application potential. The validation set is used to evaluate model performance and adjust hyperparameters during training, while the test set is used to simulate model performance in a real application environment.

[0191] Based on the evaluation results, we will optimize the hyperparameters to find the best model configuration, including learning rate, batch size, optimizer type, etc. The adjustment of hyperparameters aims to ensure that the model achieves the best performance on the validation set while avoiding overfitting, thereby ensuring the model's good generalization ability for unseen data.

[0192] In addition, for the feature decoupling and sharing strategy in the multi-task learning framework, the present invention continuously optimizes the exchange and fusion mechanism of feature maps based on performance evaluation feedback to ensure that knowledge sharing between tasks can maximize overall performance. This process helps the model achieve higher accuracy and consistency in data processing tasks, further improving its clinical applicability.

[0193] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.< / end> < / unk> < / end> < / start>

Claims

1. A method for processing animal X-ray medical image data, characterized in that: include: S1. Construct a joint multi-task animal X-ray multimodal dataset; S2, generate multimodal dataset labels; S3, preprocessing the multimodal dataset; S4. Build a multi-task learning model; S5. Implement feature sharing and feature fusion strategies for multi-task learning models; S6. Training the multimodal multi-task learning model; S7. Evaluate the performance of multimodal multi-task learning models; Step S2 specifically includes: Generate segmentation task labels: Use Python and OpenCV to perform data preprocessing and label format conversion to generate pixel-level mask labels for segmentation tasks. The mask labels are annotated using the annotation tool Labelbox. Each pixel is assigned a unique integer value based on the anatomical or lesion area in which it is located. The generated mask is saved in a single-channel image format and maintains the same size as the original image file. Generate detection task labels: Use the LabelImg tool to annotate the lesion area in the image with a bounding box, record the coordinate information of the bounding box, manually annotate the category of each bounding box and the optional confidence score, and store the label data as a JSON file in COCO format; Generate classification task labels: Use the Pandas library to organize classification labels and record the classification information of the detected lesion area, including whether the fracture type is closed or open and whether the tumor is benign or malignant. Encode the label data in one-hot encoding format and store it in a CSV file. The classification label file and the detection label file keep the same lesion area index to ensure label alignment of branch tasks. Label and image batch processing: Design a custom data loader through PyTorch's DataLoader module to support batch processing and load images and multi-task labels synchronously. Use the Albumentations data augmentation library to perform augmentation operations on X-ray images, ensuring that segmentation masks, detection bounding boxes, and classification labels are automatically transformed synchronously during the augmentation process. Generate a diagnostic report and give the corresponding diagnostic report text for each X-ray image; Step S3 specifically includes: X-ray image data preprocessing: De-noising the X-ray image data, using the median filter algorithm to reduce random noise interference. The median filter algorithm replaces the value of each pixel with the median of the pixel values ​​in the neighborhood to eliminate salt noise and pepper noise in the image; Histogram equalization technology is used to optimize the brightness distribution of X-ray image data; After histogram equalization, contrast stretching is performed on the X-ray image data; Finally, a normalization process is performed to standardize the pixel values ​​of the X-ray image data to a set range; Text preprocessing: Clean the original diagnostic report text to remove extra spaces, line breaks, and character noise. For text containing non-standard medical terms and redundant information, remove them through keyword matching; Medical terms in the diagnostic report text are standardized through the medical terminology database, and synonymous or near-synonymous medical terms are expressed in a unified manner. This process is performed using Python language and PyMedTerm library. The diagnostic report text is segmented into sentences and punctuation is corrected using the NLTK library, which separates and reconstructs the diagnostic points in the text through syntactic analysis; Extract all the words in the diagnosis report text to get a dictionary describing the content of the X-ray image data, use the jieba tool to segment each diagnosis report text, and use spaces to replace the characters in the diagnosis report text. Add the start and end before and after each manually annotated diagnosis report text, fill in unknown words, and fill all the diagnosis report texts to the maximum length, count and encode each segmented word; All extracted words are encoded into numerical labels, and the feature value of each medical entity is mapped to a unique numerical code and stored in a CSV file using one-hot encoding; The words of the final diagnosis report text are expanded using synonyms to generate more expressive text; Step S4 specifically includes: Build the backbone network: Select the EfficientNet network as the backbone network, combine the features of different levels in the EfficientNet network with the feature pyramid, and provide them to the segmentation and detection branches respectively. The low-level features are provided to the segmentation branch, focusing on local edges and detail information, and the high-level features are provided to the detection branch, focusing on global semantic information. Multi-level feature maps are generated through the EfficientNet network; Build the split branch: Improved based on U-Net architecture; Encoder part: Use the features generated by the EfficientNet network combined with the feature vector after the multi-task feature sharing and decoupling module as the encoder input of the segmentation branch to reduce computational redundancy; Skip connection: The features of the backbone network are passed to the decoder through skip connections; Decoder part: gradually restores the image spatial resolution and outputs a segmentation mask consistent with the input size for annotating anatomical structures and lesion areas; Build the detection branch: The Fast R-CNN framework is used to locate and classify the lesion area in the X-ray image. Fast R-CNN realizes the location and category recognition of the lesion area by combining the features extracted by the EfficientNet network with the features after multi-task feature sharing and decoupling modules. The process is as follows: Region Proposal Network: Generates candidate boxes through the features generated by the EfficientNet network and feature pyramid, providing input for the positioning of the lesion area; ROI pooling: Perform ROI pooling operation on each candidate box area to standardize the feature size; Bounding box regression and classification: perform bounding box regression and classification on the candidate boxes, and output the coordinates, category labels and confidence of the lesion area; Build the classification branch: Feature extraction: Obtain feature vectors of the lesion area from the backbone network and detection branches; Fully connected layer classification: The feature vector is mapped to the set category label through the fully connected layer, and the detailed attributes of the lesion are output; Construct feature fusion layer: A feature fusion layer is set after the segmentation branch, detection branch, classification branch and EfficientNet network trunk. The original image features extracted by EfficientNet are integrated with the multi-task features through the feature fusion layer to provide complete multimodal information for the report generation module. Build the report generation module: The report generation module uses GPT-2 and uses the fused feature vectors to generate diagnostic text that meets medical standards; Step S5 specifically includes: A multi-task feature sharing and decoupling module is added after the backbone network. The multi-task feature sharing and decoupling module is specifically used to: Share low-level features: Let segmentation, detection, and classification tasks share the low-level features of the backbone network; Feature Pyramid Network: Utilizes the pyramid structure of multi-scale feature maps to generate feature resolutions suitable for different tasks. Segmentation tasks utilize low-level features, while detection and classification tasks use high-level features. Task-specific convolutional layer: In the decoupling module, based on the feature pyramid network, a dedicated convolutional layer is set for each task. By applying the task-specific convolutional layer on the feature map output by the feature pyramid network, the segmentation, detection and classification tasks obtain independent feature spaces based on shared features, reducing feature competition and interference between tasks. The dedicated convolutional layer of each task is set according to its feature requirements; Set up a dynamic task weight adjustment mechanism to dynamically adjust the weight of each task according to the training progress; Total loss function for: ； in, is the segmentation task loss function, is the loss function for the detection task, is the classification task loss function, is the weight parameter of the segmentation task, is the weight parameter of the detection task, is the weight parameter of the classification task; After the segmentation, detection, and classification branches complete their tasks, the fused features are input into the report generation module, which performs contextual analysis on the fused features through a hierarchical self-attention mechanism to generate a coherent diagnostic report text. Step S6 specifically includes: Backbone network initialization: initialized by loading weights pre-trained on ImageNet; Task branch initialization: For segmentation, detection and classification tasks, the parameters of each task branch are initialized respectively. Kaiming initialization is used to randomly assign weight values ​​to each branch. Initial weight assignment: Assign initial task weights and set weight values ​​for segmentation, detection, and classification tasks respectively to control the influence of different tasks in the multi-task loss function; Dynamic weight adjustment mechanism: Design a dynamic weight adjustment strategy based on the task loss convergence situation. The multi-task learning model adjusts the weight value in real time according to the loss decline rate of each task; Optimizer selection: Adopt Adam optimizer, which improves the efficiency and stability of training by adaptively adjusting the learning rate of each parameter; Learning rate strategy: Set the initial learning rate and adopt a warm-up learning rate strategy to gradually increase the learning rate in the initial stage, and then apply cosine annealing learning rate decay in the stable stage to gradually guide the convergence of model parameters; Pre-training and integration of report generation module: Perform causal language modeling pre-training on the report generation module based on GPT-2; Loss Function The definition is as follows: ； N represents the total length of the sequence, represents all tokens before the i-th token, is the target token; Multi-task feature fusion and text generation integrated training: After the backbone network completes image feature extraction and each branch task, the image feature vectors extracted by the segmentation, detection and classification branches and the original backbone network are integrated into a unified multi-scale semantic feature vector through the feature fusion layer and input into the report generation module; Step S7 specifically includes: The performance of the multimodal multi-task learning model is evaluated. Based on the evaluation results, the hyperparameters are optimized and adjusted to seek the best multimodal multi-task learning model configuration, including learning rate, batch size, and optimizer type. For the feature decoupling and sharing strategies in the multi-task learning model, the feature map exchange and fusion mechanism is continuously optimized based on performance evaluation feedback.

2. The animal X-ray medical image data processing method according to claim 1, characterized in that: Step S1 specifically includes: X-ray imaging equipment is used to collect X-ray imaging data of animals, which includes bone injury, soft tissue injury, foreign matter, tumor and lesion imaging data, and the collected X-ray imaging data is desensitized.

3. A computer device, comprising a memory, wherein the memory stores program instructions, characterized in that: When the program instructions are executed, the animal X-ray medical image data processing method as described in any one of claims 1-2 is executed.

Images