X-ray image segmentation model training method and device and computer equipment
By training and optimizing the X-ray image segmentation model, using the differences and prompt encoding of the image encoder and decoder, the problem of insufficient flexibility and generalization ability of the existing model in complex image processing is solved, and a stronger image is achieved Segmentation performance and applicability.
Patent Information
- Application Number
- CN202411047270.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-06
AI Technical Summary
Existing industrial X-ray detection segmentation models lack flexibility and generalization capabilities, making it difficult to process complex or new product images, resulting in a degradation of segmentation performance.
By obtaining sample X-ray industrial image data and unlabeled data with actual segmentation results, model training is performed using the difference between image encoder and decoder, combining prompt encoding and loss function optimization to achieve lighter and generalized X-ray with X-ray capability Ray image segmentation model.
It improves the generalization ability and image processing ability of the model, can deal with various types of image data more flexibly, reduces the dependence and labeling costs on specific tasks, and is suitable for low-end device deployment.
Smart Images

Figure CN119091246A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of industrial X-ray detection, and in particular to a training method, apparatus, computer equipment, computer-readable storage medium and computer program product for an X-ray image segmentation model. Background Art
[0002] Industrial X-ray inspection plays a key role in ensuring product quality and safety, especially in detecting internal structures and defects. Traditional inspection methods rely heavily on manual labor, which is not only inefficient but also prone to errors. In recent years, image segmentation technology, especially deep learning-based methods, has begun to be applied in this field, aiming to automate and improve the accuracy of inspection. However, these technologies still have some inherent challenges when facing industrial X-ray images.
[0003] Many existing industrial X-ray inspection segmentation models are designed for specific types of defects or materials and lack sufficient flexibility and generalization capabilities. Once new or complex product results are encountered, the segmentation performance will deteriorate. Summary of the invention
[0004] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for an X-ray image segmentation model with better generalization ability in order to solve the above technical problems.
[0005] In a first aspect, the present application provides a training method for an X-ray image segmentation model, comprising:
[0006] Acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data does not have a corresponding actual segmentation result;
[0007] Inputting the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and inputting the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain a second predicted image code, and training the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model;
[0008] The first sample X-ray industrial image data is input into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and according to the difference between the predicted segmentation result and the actual segmentation result, the image encoder and the output decoder of the initial X-ray image segmentation model are trained to obtain a trained X-ray image segmentation model.
[0009] In one embodiment, according to the difference between the first predicted image code and the second predicted image code, training the image encoder of the X-ray image segmentation model to be trained to obtain the initial X-ray image segmentation model includes:
[0010] Obtaining a mean square error loss function preset for an image encoder of the to-be-trained X-ray image segmentation model, and obtaining a first weight preset for the image encoder of the to-be-trained X-ray image segmentation model;
[0011] According to the difference and mean square error loss functions, the corresponding image coding loss value is obtained;
[0012] The first weight is adjusted accordingly based on the image coding loss value until the mean square error loss function converges to obtain an initial X-ray image segmentation model.
[0013] In one of the embodiments, the initial X-ray image segmentation model further includes a hint encoder;
[0014] Inputting the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data includes:
[0015] Inputting the first sample X-ray industrial image data into an image encoder of an initial X-ray image segmentation model to obtain a first sample image code corresponding to the first sample X-ray industrial image data;
[0016] Obtaining a preset prompt code from a prompt encoder; the prompt code is obtained based on user prompt information preset in the prompt encoder;
[0017] The first sample image code and the prompt code are input into the output decoder of the initial X-ray image segmentation model to obtain the predicted segmentation result corresponding to the first sample X-ray industrial image data.
[0018] In an exemplary embodiment, the image encoder includes a stem module, a light-weight module, and a base-weight module;
[0019] Inputting the first sample X-ray industrial image data into the image encoder of the initial X-ray image segmentation model to obtain the first sample image encoding corresponding to the first sample X-ray industrial image data includes:
[0020] Inputting the first sample X-ray industrial image data into the stem module to obtain the first sample X-ray industrial image data with the image size reduced by a preset multiple;
[0021] Inputting the first sample X-ray industrial image data whose image size is reduced by a preset multiple into the lightweight module to obtain initial spatial features associated with the first sample X-ray industrial image data;
[0022] Input the initial spatial features into the basic quantity module to obtain the spatial features corresponding to the initial spatial features;
[0023] A first sample image code corresponding to the first sample X-ray industrial image data is acquired according to the spatial feature.
[0024] In one embodiment, the hint coding includes sparse hint coding and dense hint coding; the output decoder is pre-set with learnable parameters;
[0025] Inputting the first sample image code and the hint code into the output decoder of the initial X-ray image segmentation model, obtaining a predicted segmentation result corresponding to the first sample X-ray industrial image data, including:
[0026] Generate a corresponding position code according to the first sample image code;
[0027] concatenating the learnable parameters and the sparse cue code to obtain an output unit, and adding the dense cue code and the first sample image code to obtain a fused image code;
[0028] Based on the position coding, the output unit and the fused image coding, a predicted segmentation result corresponding to the first sample X-ray industrial image data is obtained.
[0029] In one of the embodiments, the output decoder is pre-set with a second weight;
[0030] According to the difference between the predicted segmentation result and the actual segmentation result, the image encoder and output decoder of the initial X-ray image segmentation model are trained to obtain the trained X-ray image segmentation model, including:
[0031] According to the difference between the predicted segmentation result and the actual segmentation result, and the focus loss function and the sieve loss function pre-set for the initial X-ray image segmentation model, a corresponding model loss value is obtained;
[0032] The first weight and the second weight are adjusted accordingly based on the model loss value until the focus loss function and the sieve loss function converge to obtain a trained X-ray image segmentation model.
[0033] In an exemplary embodiment, acquiring first sample X-ray industrial image data and second sample X-ray industrial image data includes:
[0034] Acquire original first X-ray industrial image data and original second X-ray industrial image data;
[0035] Adjusting the image sizes of the original first X-ray industrial image data and the original second X-ray industrial image data, and performing random cropping to obtain cropped first X-ray industrial image data and second X-ray industrial image data;
[0036] horizontally flipping the cropped first X-ray industrial image data and the second X-ray industrial image data according to a preset probability to obtain flipped first X-ray industrial image data and second X-ray industrial image data;
[0037] The flipped first X-ray industrial image data and the second X-ray industrial image data are converted into a tensor format and normalized to obtain first sample X-ray industrial image data and second sample X-ray industrial image data.
[0038] In a second aspect, the present application also provides an X-ray image segmentation method, comprising:
[0039] Acquire X-ray industrial image data of the target to be segmented;
[0040] The target X-ray industrial image data is input into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, wherein the X-ray image segmentation model is trained by the training method of the X-ray image segmentation model according to any one of claims 1 to 7.
[0041] In one embodiment, the target X-ray industrial image data is input into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, including:
[0042] Inputting the target X-ray industrial image data into an image encoder of the X-ray image segmentation model to obtain a target image code associated with the target X-ray industrial image data, and obtaining a hint code from a hint encoder of the X-ray image segmentation model;
[0043] The target image code and the prompt code are input into the output decoder of the X-ray image segmentation model to obtain the predicted segmentation result corresponding to the target X-ray industrial image data.
[0044] In a third aspect, the present application also provides a training device for an X-ray image segmentation model, comprising:
[0045] A data acquisition module, used to acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data does not have a corresponding actual segmentation result;
[0046] a primary training module, configured to input the second sample X-ray industrial image data into an image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and input the second sample X-ray industrial image data into an image encoder of a pre-built reference image segmentation model to obtain a second predicted image code, and train the image encoder of the X-ray image segmentation model to be trained according to a difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model;
[0047] The secondary training module is used to input the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain the predicted segmentation result corresponding to the first sample X-ray industrial image data, and train the image encoder and output decoder of the initial X-ray image segmentation model according to the difference between the predicted segmentation result and the actual segmentation result to obtain a trained X-ray image segmentation model.
[0048] In a fourth aspect, the present application further provides an X-ray image segmentation device, comprising:
[0049] An acquisition module, used for acquiring target X-ray industrial image data to be segmented;
[0050] An input module is used to input the target X-ray industrial image data into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, wherein the X-ray image segmentation model is trained by the training method of the X-ray image segmentation model according to any one of claims 1 to 7.
[0051] In a fifth aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0052] Acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data does not have a corresponding actual segmentation result;
[0053] Inputting the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and inputting the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain a second predicted image code, and training the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model;
[0054] The first sample X-ray industrial image data is input into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and according to the difference between the predicted segmentation result and the actual segmentation result, the image encoder and the output decoder of the initial X-ray image segmentation model are trained to obtain a trained X-ray image segmentation model.
[0055] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:
[0056] Acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data does not have a corresponding actual segmentation result;
[0057] Inputting the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and inputting the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain a second predicted image code, and training the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model;
[0058] The first sample X-ray industrial image data is input into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and according to the difference between the predicted segmentation result and the actual segmentation result, the image encoder and the output decoder of the initial X-ray image segmentation model are trained to obtain a trained X-ray image segmentation model.
[0059] In a seventh aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0060] Acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data does not have a corresponding actual segmentation result;
[0061] Inputting the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and inputting the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain a second predicted image code, and training the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model;
[0062] The first sample X-ray industrial image data is input into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and according to the difference between the predicted segmentation result and the actual segmentation result, the image encoder and the output decoder of the initial X-ray image segmentation model are trained to obtain a trained X-ray image segmentation model.
[0063] The training method, apparatus, computer device, computer-readable storage medium and computer program product of the above-mentioned X-ray image segmentation model obtain first sample X-ray industrial image data with corresponding actual segmentation results and second sample X-ray industrial image data without corresponding actual segmentation results, input the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and input the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain a second predicted image code, according to the difference between the first predicted image code and the second predicted image code, train the image encoder of the X-ray image segmentation model to be trained to obtain an initial X-ray image segmentation model, then input the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and according to the difference between the predicted segmentation result and the actual segmentation result, train the image encoder and output decoder of the initial X-ray image segmentation model to obtain a trained X-ray image segmentation model. The X-ray image segmentation model structure obtained after the secondary training is more lightweight, and the generalization ability of the model and the ability to process images are greatly enhanced. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0065] Figure 1 A diagram showing an application environment of an X-ray image segmentation model training method in one embodiment;
[0066] Figure 2 A schematic diagram of a flow chart of an X-ray image segmentation model training method in one embodiment;
[0067] Figure 3 A schematic diagram of a flow chart of an X-ray image segmentation method in another embodiment;
[0068] Figure 4is a schematic diagram of a first sample X-ray industrial image data in an embodiment;
[0069] Figure 5 is a structural schematic diagram of an X-ray image segmentation model in another embodiment;
[0070] Figure 6 is a schematic diagram of the structure of an image encoder in an X-ray image segmentation model in one embodiment;
[0071] Figure 7 is a schematic diagram of the structure of an output decoder in an X-ray image segmentation model in another embodiment;
[0072] Figure 8 A schematic diagram of a process for training an initial X-ray image segmentation model in one embodiment;
[0073] Fig. 9 A schematic diagram of a process for training an X-ray image segmentation model in another embodiment;
[0074] Fig.10 A schematic diagram of an interface for interactive operations between a user and a model in one embodiment;
[0075] Fig.11 is a structural block diagram of a training device for an X-ray image segmentation model in one embodiment;
[0076] Fig.12 is a structural block diagram of an X-ray image segmentation device in another embodiment;
[0077] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0078] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0079] The training method of the X-ray image segmentation model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The data storage system pre-stores the first sample X-ray industrial image data obtained from the terminal 102, the second sample X-ray industrial image data, and the X-ray image segmentation model to be trained. The server 104 obtains the first sample X-ray industrial image data and the second sample X-ray industrial image data from the data storage system, the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data does not correspond to an actual segmentation result, then inputs the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and inputs the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain a second predicted image code, and trains the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image model; then inputs the first sample X-ray industrial image data into the initial X-ray image model to obtain a corresponding predicted segmentation result, and trains the image encoder and output decoder of the initial X-ray image segmentation model according to the difference between the predicted segmentation result and the actual segmentation result to obtain a trained X-ray image segmentation model. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0080] In an exemplary embodiment, Figure 2 As shown, a training method for an X-ray image segmentation model is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S201 to S203. Among them:
[0081] Step S201, obtaining first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data has no corresponding actual segmentation result.
[0082] Step S202: input the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and input the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain a second predicted image code, and train the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model.
[0083] Among them, the first sample X-ray industrial image data can be understood as industrial X-ray images of various products, and there are corresponding actual segmentation results; the second sample X-ray industrial image data can be understood as image data obtained from a public X-ray image dataset, which does not correspond to actual segmentation results; the pre-built benchmark image segmentation model can be understood as an image segmentation model with a large model architecture and a large construction cost; the image encoder can be understood as a neural network model, which is usually complex in structure and is used to convert the input image data into a higher-dimensional representation for the processing and analysis of subsequent tasks. The image encoder is usually composed of multiple convolutional layers, pooling layers, and fully connected layers, etc., which learns the features of each part of the image and encodes it into a more compact and abstract representation.
[0084] For example, the server 104 obtains the first sample X-ray industrial image data and the second sample X-ray industrial image data from the data storage system, and then inputs the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain the first predicted image code, and inputs the second sample X-ray industrial image data into the image encoder of the pre-built reference image segmentation model to obtain the second predicted image code, and trains the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain the initial X-ray image segmentation model. The image encoder of the X-ray image segmentation model to be trained is trained using the image encoder of the preset reference X-ray image segmentation model, thereby reducing the model architecture complexity of the X-ray image segmentation model to be trained, improving the lightweight design of the X-ray image segmentation model, and thus achieving normal operation on low-end devices, greatly improving the applicability of the model and reducing the model installation cost, and laying a data foundation for subsequent secondary model training.
[0085] Step S203: input the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain the predicted segmentation result corresponding to the first sample X-ray industrial image data, and train the image encoder and output decoder of the initial X-ray image segmentation model according to the difference between the predicted segmentation result and the actual segmentation result to obtain a trained X-ray image segmentation model.
[0086] The output decoder can be understood as a neural network model component, which usually corresponds to the encoder and is used to restore the feature representation extracted by the encoder to the original data form. In image processing tasks, the output decoder can receive the feature vector or feature map generated by the encoder, and then restore these features to the original image data through reverse operations.
[0087] Optionally, the server 104 inputs the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and then trains the image encoder and output decoder of the initial X-ray image segmentation model according to the difference between the predicted segmentation result and the actual segmentation result to obtain a trained X-ray image segmentation model. By using the first sample X-ray industrial image data to train the initial X-ray image segmentation model, the trained model has stronger generalization ability and can more flexibly handle various types of image data.
[0088] In the training method of the above-mentioned X-ray image segmentation model, a first sample X-ray industrial image data with a corresponding actual segmentation result and a second sample X-ray industrial image data without a corresponding actual segmentation result are obtained, the second sample X-ray industrial image data is input into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and the second sample X-ray industrial image data is input into the image encoder of the SAM model to obtain a second predicted image code, according to the difference between the first predicted image code and the second predicted image code, the image encoder of the X-ray image segmentation model to be trained is trained to obtain an initial X-ray image segmentation model, and then the first sample X-ray industrial image data is input into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and according to the difference between the predicted segmentation result and the actual segmentation result, the image encoder and output decoder of the initial X-ray image segmentation model are trained to obtain a trained X-ray image segmentation model. The X-ray image segmentation model obtained after the second training has a lighter structure, and the generalization ability of the model and the ability to process images are greatly enhanced.
[0089] In one embodiment, according to the difference between the first predicted image code and the second predicted image code, the image encoder of the X-ray image segmentation model to be trained is trained to obtain an initial X-ray image segmentation model, including: obtaining a mean square error loss function pre-set for the image encoder of the X-ray image segmentation model to be trained, and obtaining a first weight pre-set for the image encoder of the X-ray image segmentation model to be trained; according to the difference and the mean square error loss function, obtaining a corresponding image coding loss value; and adjusting the first weight accordingly based on the image coding loss value until the mean square error loss function converges to obtain the initial X-ray image segmentation model.
[0090] Among them, the mean square error loss function is usually used to measure the difference between the model output and the true value or label, and is one of the commonly used loss functions in regression tasks. This loss function calculates the average of the square of the difference between the predicted value and the true value, and optimizes the model parameters by minimizing this average variance so that the model's prediction result is closer to the true value; the first weight can be understood as the weight parameter currently corresponding to the image encoder of the X-ray image segmentation model to be trained. It is known that this weight parameter can be set by itself, but it should be smaller than the preset benchmark image segmentation model.
[0091] In an exemplary embodiment, the server 104 obtains the mean square error loss function and the first weight pre-set for the image encoder of the X-ray image segmentation model to be trained, obtains the corresponding image coding loss value according to the difference between the first predicted image coding and the second predicted image coding and the mean square error loss function, and adjusts the first weight accordingly according to the image coding loss value until the mean square error loss function converges, that is, reaches the minimum value, and obtains the initial X-ray image segmentation model. The most suitable first weight is adjusted by the above means, thereby achieving the lightweight of the initial X-ray image segmentation model, laying a data foundation for the secondary training of the model.
[0092] In one embodiment, the initial X-ray image segmentation model also includes a prompt encoder; inputting the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, including: inputting the first sample X-ray industrial image data into the image encoder of the initial X-ray image segmentation model to obtain a first sample image code corresponding to the first sample X-ray industrial image data; obtaining a preset prompt code from the prompt encoder; the prompt code is obtained based on user prompt information preset in the prompt encoder; inputting the first sample image code and the prompt code into the output decoder of the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data.
[0093] Among them, the prompt encoder is responsible for converting different types of prompt information provided by the user into a vector representation that the model can understand, so that the model can generate segmentation masks of objects in the image based on this.
[0094] For example, the server 104 inputs the first sample X-ray industrial image data into the image encoder of the initial X-ray image segmentation model to obtain the first sample image code corresponding to the first sample X-ray industrial image data, then obtains the preset prompt code from the prompt encoder of the initial X-ray image segmentation model, and then inputs the first sample image code and the prompt code into the output decoder of the initial X-ray image segmentation model to obtain the predicted segmentation result corresponding to the first sample X-ray industrial image data. The predicted segmentation result is not only associated with the image itself but also associated with the user prompt information, which increases the data source that affects the output result and improves the reliability of the output result.
[0095] In an exemplary embodiment, an image encoder includes a stem module, a lightweight module and a basic quantity module; the first sample X-ray industrial image data is input into the image encoder of the initial X-ray image segmentation model to obtain the first sample image code corresponding to the first sample X-ray industrial image data, including: inputting the first sample X-ray industrial image data into the stem module to obtain the first sample X-ray industrial image data with the image size reduced by a preset multiple; inputting the first sample X-ray industrial image data with the image size reduced by a preset multiple into the lightweight module to obtain the initial spatial features associated with the first sample X-ray industrial image data; inputting the initial spatial features into the basic quantity module to obtain the spatial features corresponding to the initial spatial features; and obtaining the first sample image code corresponding to the first sample X-ray industrial image data according to the spatial features.
[0096] Among them, the stem module can be understood as a module for image resizing of image data; the lightweight module can be understood as a module for converting image data into initial spatial features; the basic quantity module can be understood as a module for further decomposing the initial spatial features to obtain spatial features.
[0097] Optionally, the server 104 inputs the first sample X-ray industrial image data into the stem module to obtain the first sample X-ray industrial image data with the image size reduced by a preset multiple, where the preset multiple can be 4 times or other numbers, and then inputs the first sample X-ray industrial image data with the image size reduced by the preset multiple into the lightweight module to obtain the initial spatial features associated with the first sample X-ray industrial image data, and then inputs the initial spatial features into the basic quantity module to obtain the spatial features corresponding to the initial spatial features, and finally obtains the first sample image code corresponding to the first sample X-ray industrial image data according to the spatial features. By obtaining data features with more representative meanings through the above-mentioned processing means for image data, the availability of image data is improved.
[0098] In one embodiment, the hint coding includes sparse hint coding and dense hint coding; the output decoder is pre-set with learnable parameters; the first sample image coding and the hint coding are input into the output decoder of the initial X-ray image segmentation model to obtain the predicted segmentation result corresponding to the first sample X-ray industrial image data, including: generating the corresponding position coding according to the first sample image coding; splicing the learnable parameters and the sparse hint coding to obtain the output unit, and adding the dense hint coding and the first sample image coding to obtain the fused image coding; based on the position coding, the output unit and the fused image coding, the predicted segmentation result corresponding to the first sample X-ray industrial image data is obtained.
[0099] Among them, position encoding can be understood as a technique used in neural networks to represent the position information of each element in the input sequence. It is usually added to the input data in the form of vectors or matrices, so that each element is given a unique position information, in order for the neural network to understand the order and position relationship of the elements in the input data, so as to better capture the structure and context information in the sequence; sparse hint encoding includes point hint vectors and box hint vectors; dense hint encoding can be understood as the vector hint of the object mask in the image data;
[0100] Learnable parameters can be understood as parameters in a neural network that need to be learned and adjusted through model training. These parameters can control the behavior and decision-making process of the neural network.
[0101] For example, the server 104 first generates a corresponding position code according to the first sample image code, splices the learnable parameters and the sparse hint code to obtain an output unit, and adds the dense hint code and the position code to obtain a fused image code, and finally obtains the predicted segmentation result corresponding to the first sample X-ray industrial image data according to the position code, the output unit and the fused image code. Through the multi-step processing of the image code and the hint code, the prediction effect of the model and the image segmentation result can be improved.
[0102] In one of the embodiments, the output decoder is pre-set with a second weight; based on the difference between the predicted segmentation result and the actual segmentation result, the image encoder and the output decoder of the initial X-ray image segmentation model are trained to obtain a trained X-ray image segmentation model, including: based on the difference between the predicted segmentation result and the actual segmentation result, and the focus loss function and the sieve loss function pre-set for the initial X-ray image segmentation model, the corresponding model loss value is obtained; based on the model loss value, the first weight and the second weight are correspondingly adjusted until the focus loss function and the sieve loss function converge, thereby obtaining the trained X-ray image segmentation model.
[0103] Among them, the focal loss function can be understood as a loss function designed for the problem of class imbalance. By reducing the contribution of easy-to-classify samples to the overall loss, the model can focus more on the learning of difficult samples; the sieve loss function can be understood as a loss function commonly used in image segmentation tasks. It measures the similarity between the segmentation results output by the model and the true label based on the Dice coefficient; the second weight can be understood as the weight parameter corresponding to the output decoder, which shares the weight parameter with the output decoder of the pre-set benchmark image segmentation model.
[0104] Optionally, the server 104 obtains the corresponding model loss value according to the difference between the predicted segmentation result and the actual segmentation result, and the focal loss function and sieve loss function pre-set for the initial X-ray image segmentation model, and then adjusts the first weight corresponding to the image encoder and the second weight corresponding to the output decoder based on the model loss value, until the focal loss function and the sieve loss function converge, that is, when the two have the lowest value, the first weight and the second weight at this time are used as the weight parameters corresponding to the final image encoder and output decoder, and the X-ray image segmentation model training is completed. By adjusting the weight parameters, the predicted segmentation result of the model is made closer to the actual segmentation result, thereby improving the prediction effect of the model.
[0105] In an exemplary embodiment, obtaining first sample X-ray industrial image data and second sample X-ray industrial image data includes: obtaining original first X-ray industrial image data and original second X-ray industrial image data; adjusting the image size of the original first X-ray industrial image data and the original second X-ray industrial image data, and performing random cropping to obtain cropped first X-ray industrial image data and second X-ray industrial image data; horizontally flipping the cropped first X-ray industrial image data and second X-ray industrial image data according to a preset probability to obtain flipped first X-ray industrial image data and second X-ray industrial image data; converting the flipped first X-ray industrial image data and second X-ray industrial image data into tensor format and performing normalization processing to obtain first sample X-ray industrial image data and second sample X-ray industrial image data.
[0106] Among them, the original first X-ray industrial image data and the original second X-ray industrial image data can be understood as unprocessed image data containing many defects and low validity; the tensor format can be understood as a multidimensional array structure, which is the basic representation form of data in a neural network, and is usually used to represent input data, model parameters, and intermediate layer outputs.
[0107] For example, the server 104 obtains the original first X-ray industrial image data and the original second X-ray industrial image data from the data storage system, first adjusts the image size of the original first X-ray industrial image data and the original second sample X-ray industrial image data, and randomly crops them to obtain the cropped first X-ray industrial image data and the second X-ray industrial image data, then horizontally flips the cropped first X-ray industrial image data and the second X-ray industrial image data according to a preset probability to obtain the flipped first X-ray industrial image data and the second X-ray industrial image data, and finally converts the flipped first X-ray industrial image data and the second X-ray industrial image data into a tensor format and normalizes them to obtain the first sample X-ray industrial image data and the second sample X-ray industrial image data used for model training. The above data processing means improves the availability of image data, thereby improving the training effect of model training and improving the generalization ability of the model.
[0108] In an exemplary embodiment, Figure 3 As shown, a training method for an X-ray image segmentation model is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps S301 and S302. Among them:
[0109] Step S301, obtaining target X-ray industrial image data to be segmented.
[0110] Step S302, input the target X-ray industrial image data into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, wherein the X-ray image segmentation model is trained by the training method of the X-ray image segmentation model according to any one of claims 1 to 7.
[0111] Optionally, the server 104 preloads an X-ray image segmentation model trained by the training method of the X-ray image segmentation model according to any one of claims 1 to 7, and then the server 104 obtains the target X-ray industrial image data to be segmented from the terminal 102, and then inputs the target X-ray image data into the pre-trained X-ray image segmentation model to obtain the predicted segmentation result of the target X-ray industrial image data. Using the pre-loaded X-ray image segmentation model to perform image segmentation improves the efficiency of image segmentation, which is conducive to improving the user's service experience.
[0112] In one embodiment, target X-ray industrial image data is input into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, including: inputting the target X-ray industrial image data into an image encoder of the X-ray image segmentation model to obtain a target image code associated with the target X-ray industrial image data, and obtaining a hint code from a hint encoder of the X-ray image segmentation model; inputting the target image code and the hint code into an output decoder of the X-ray image segmentation model to obtain a predicted segmentation result corresponding to the target X-ray industrial image data.
[0113] For example, the server 104 inputs the target X-ray industrial image data into the image encoder of the X-ray image segmentation model to obtain the target image code associated with the target X-ray industrial image data, and obtains the hint code from the output decoder of the X-ray image segmentation model, and then inputs the target image code and the hint code into the output decoder of the X-ray image segmentation model to obtain the predicted segmentation result corresponding to the target X-ray industrial image data.
[0114] When the reference image segmentation model used is the SAM model (elective Attention Model), the present method may also be that the server 104 also obtains user prompt information from the terminal 102, and inputs the user prompt information into the prompt encoder to obtain the corresponding prompt encoder, and then similarly inputs the target image code and the prompt code into the output decoder, and finally obtains the predicted segmentation result corresponding to the target X-ray industrial image data.
[0115] Through the above method, the image data feature representation is optimized and information fusion is achieved, thereby achieving accurate prediction of the model and flexible application of the model.
[0116] In an exemplary embodiment, the present application provides a detailed embodiment of the training and application of an X-ray image segmentation model, see Figure 4-Figure 10 , where Efficientformer is a lightweight visual transformer, stem1 is the stem 1 module, stem2 is the stem 2 module, PE is the position encoding module, LB layer is a lightweight module, LGB layer is a basic module; mask is a binary image, FOCAL LOSS is the focus loss function, DICE LOSS is the sieve loss function; MSE is the MeanSquared Error loss function, mean square error loss function; including:
[0117] Step 1: Collect industrial X-ray images of various products, annotate them, and obtain the initial image dataset 1, such as Figure 4 As shown in Figure 2, a public X-ray image dataset is collected without annotating it, and dataset 2 is obtained.
[0118] Step 2: Divide data set 1 into a training set and a test set in a ratio of 4:1;
[0119] Step 3: Preprocessing operation. For datasets 1 and 2, first change the input image size to A×A, then randomly crop the image to obtain an image of size B×B, and then randomly flip it horizontally with a probability of 0.5. After converting the image to Tensor format, standardize and normalize it.
[0120] Step 4: Establish a general X-ray image segmentation model. Referring to the SAM model, the general X-ray image segmentation model consists of three parts: image encoder, prompt encoder and output decoder. Figure 5 shown.
[0121] Step 5: Import the pre-trained weight parameters into the image encoder, initialize the prompt encoder and output decoder of the general X-ray image segmentation model, import the pre-trained weights, and then train the network to finally obtain the general X-ray image segmentation model.
[0122] Step 6: Input the test set into the trained XRAY image super-resolution network, design the relevant software interface, and obtain the X-ray image segmentation result according to the prompt points or bounding boxes set by the user.
[0123] Specifically, in step 3, with the help of the Python language environment, the torchvision.Resize() method is used to adjust the image size. Subsequently, the torchvision.RandomCrop() method is applied to randomly crop the image, and then the torchvision.RandomHorizontalFlip() method is used to perform a horizontal random flip operation on the image with a probability of 50%. Next, the processed image is converted to Tensor format through the transforms.ToTensor() method, and preliminary normalization processing is performed in the process. Finally, the transforms.Normalize() function is used to complete the standardization processing step of the image data to ensure that the data meets the predefined normalization and standardization requirements. The specific algorithm details of the specification and normalization include:
[0124]
[0125] in, is the mean of the images, is the image tensor, is the standard deviation, is the maximum value of the image tensor, is the minimum value of the image tensor, Represents the normalized image tensor, Represents the normalized image tensor.
[0126] Specifically, in step 4, in order to improve its operating efficiency on edge and low-end devices, the image encoder uses an improved EfficientFormer, which introduces global context enhancement and promotes cross-scale information interaction. Figure 6As shown in the figure, the improved Efficientformer consists of five modules: Stem1, Stem2, PE, LB, and LGB. Stem1 consists of a residual connection of 5×5 convolution and 3×3 convolution in the training stage. In the inference stage, reparameterization can be equivalent to a 5×5 convolution. The image is reduced by two times after Stem1. Stem2 consists of three 1×1 convolutions, one 5×5 and one 3×3 separable convolution. The first 1×1 convolution is used for dimensionality increase, and then the residual connection of 5×5 and 3×3 separable convolution is used to extract features. The 1×1 convolution is used for dimensionality reduction, and finally the 1×1 convolution is used. The result of the convolution is residually connected, and the image is reduced by two times after Stem2. After that, it passes through two layers of LB layers. The LB layer consists of 2 1×1 convolutions and CBAM. The input features are connected to the input residual through two layers of 1×1 convolutions, and finally the output features are obtained through the CBAM module. The spatial features after the LB layer will be encoded into PATH features by large-scale separable convolution as the input of the LGB layer. The LGB layer is a Transformer structure improved by referring to Efficientformer. In the LGB layer, the input features are mapped into three spatial features Q, K, and V through three independent 1×1 convolutions. In order to reduce the amount of calculation, V and K are further encoded through large-scale convolution kernels. Q is downsampled through dual-path attention and pooling is used as static local downsampling. 3×3DWCONV is used as a learnable local downsampling, which utilizes both local relations and global dependencies. The processed QKV is subjected to the Attention operation, and the output result is passed through a layer of 1×1 convolution, and then a layer of separable 3×3 convolution to provide zero-filled position encoding. After that, the feature passes through a layer of LB layer that removes the CBAM attention mechanism, and finally undergoes the same operation again to obtain the final encoding feature. According to the operation of the SAM model, to obtain the image encoding, it is necessary to pass through another layer of 1×1 convolution and 3×3 convolution to obtain the final image encoding.
[0127] Specifically, in step 4, the prompt encoder retains the structure in the SAM model, and the prompt encoder is responsible for converting different types of prompt information provided by the user into vector representations that the model can understand, so that the model can generate segmentation masks of objects in the image accordingly. Two types of prompts can be processed: sparse prompts and dense prompts.
[0128] Sparse Tips:
[0129] Point hint: Each point is represented by the coordinate encoding of its position in the image, which is added to the learned embedding vectors of the foreground and background areas. Assuming a point , the position code is , if the point is in the foreground, add a learned embedding vector , if it is background, add , expressed using the formula:
[0130]
[0131] In the above formula, represents the point coordinates, It means calculating point embedding according to point coordinates. The calculation steps are as follows: the horizontal and vertical coordinates are normalized first, the point matrix is multiplied by a random Gaussian matrix to get the matrix coord, and finally the sin and cos of the coord are calculated respectively, and the concatenation matrix is used to get the final point embedding (Nxax256). Represents the embedding vectors of the learnable foreground and background.
[0132] Box hint: A box is represented by its two corner embeddings. Each corner is obtained by adding the position encoding to the specific corner embedding, which is expressed as:
[0133]
[0134] In the above formula, represents a vertex, Indicates the bottom point, and Learnable embedding vectors representing lows and highs.
[0135] Dense mask processing:
[0136] Dense hints directly correspond to object masks in the image. They are first input at a preset multiple lower resolution than the input image, and then further downsampled by a preset multiple through two 2×2 convolutional layers with a stride of 2, with the number of output channels being 4 and 16 respectively. Finally, a 1×1 convolutional layer converts the number of channels to 256, and a GELU activation function and layer normalization operation are used after each layer. Ensure that the mask hint is spatially aligned with the image and can be effectively fused with the image encoding. The formula is expressed as:
[0137]
[0138] The hint encoder converts all types of hints into fixed-dimensional vector representations through the above method, which are then used to interact with the image encoding to jointly produce segmentation masks in the lightweight mask decoder, making it flexible to respond to different types of user input.
[0139] Specifically, in step 4, the output decoder structure is as follows Figure 7As shown in the figure, the output decoder connects the results of the image encoding and the hint encoding to generate the segmentation mask. The output decoder is designed to be lightweight and can efficiently map the image encoding and a set of hint encodings to the output mask while maintaining real-time performance. In the output decoder, the learnable parameters mask token and iou tokens are initialized, the positional encoding of the input image encoding is generated, the sparse hint encoding is concatenated with the mask token and iou tokens into an output token, the dense hint encoding is added to the image encoding, and the positional encoding, output token and fused image encoding are input into the Transformer network. First, after two layers of TwoWayAttentionBlock, the keys and querise are output. The keys are connected to the fused image encoding residual, and the querise is connected to the positional encoding residual. The two sets of fused vectors and keys are used as the output of the last layer of Attention.
[0140] Specifically, in step 5, SAM-ViT-H is used to perform distillation training on the improved EfficientFormer on dataset 2, such as Figure 8 As shown. Knowledge is distilled directly from the large image encoder SAM-ViT-H to the small image encoder without relying on the combined decoder. The mean square error (MSE) loss function is used to guide the distillation process of image embedding. MSE loss directly measures the difference between the image embedding generated by the student model (small image encoder) and the embedding generated by the teacher model (large image encoder), which simplifies the optimization process and may reduce the training complexity. The formula is described as follows:
[0141]
[0142] in, and They represent the encoding outputs of the student model (lightweight image encoder) and the teacher model (ViT-H) for the input image respectively; N is the number of samples.
[0143] Specifically, in step 5, the image encoder loads the distillation pre-trained weights, and the prompt encoder and output decoder directly load the pre-trained weights in SAM-ViT-H because the original model structure is retained, such as Fig. 9 As shown, on dataset 1, the parameters of the prompt encoder are frozen, and different learning rates are used to train the image encoder and fine-tune the output decoder. The training is iterated 60,000 times, and the loss is calculated using FOCAL LOSS and DICE LOSS. The formula is as follows:
[0144]
[0145] Among them, α is a hyperparameter used to adjust the weights of positive and negative samples; p is the predicted probability; γ is a hyperparameter used to adjust the weights of positive and negative samples.
[0146]
[0147] Represents the label value, represents the predicted value, Represents a hyperparameter used to adjust the weights of positive and negative samples.
[0148] When training a neural network, these loss functions are calculated simultaneously, and the model parameters are adjusted according to the values of the loss functions to make the model's predictions closer to the true labels.
[0149] Specifically, in step 6, the trained model will be loaded into the corresponding software, such as Fig.10 As shown, the software interface supports user interaction with the model. An implementation method is described below:
[0150] The implementation method is point prompt operation. After loading the model in the software interface, load an X-ray image and click the image encoding operation. The X-ray image will be encoded into high-level features through the image encoder. Click the point prompt button to support interactive operations with the image in the image display window. Click the left mouse button on the position on the image where the target needs to be segmented to generate a foreground point prompt, and the corresponding target is covered by the segmentation mask.
[0151] Compared with the prior art, this application has the following advantages:
[0152] 1. A lighter model structure, using lightweight network structure design and knowledge distillation training methods, makes it possible to deploy the model on low-end devices.
[0153] 2. Powerful generalization capability. The improved SAM model design is different from previous deep learning models for specific tasks. It can demonstrate good generalization performance on unseen industrial parts and diverse defect types without the need for retraining for each new situation, greatly improving practicality and efficiency.
[0154] 3. Reduce annotation costs: With the help of the prompt learning mechanism, a small amount of prompt information provided by users can be effectively used for model training and segmentation, which significantly reduces the dependence on large-scale annotated data sets and reduces the time and economic cost of data preparation.
[0155] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0156] Based on the same inventive concept, the embodiment of the present application also provides a training device for an X-ray image segmentation model for implementing the training method for the X-ray image segmentation model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more X-ray image segmentation models provided below can refer to the limitations of the training method for the X-ray image segmentation model above, and will not be repeated here.
[0157] In an exemplary embodiment, Fig.11 As shown, a training device for an X-ray image segmentation model is provided, comprising: a data acquisition module 1101, a primary training module 1102 and a secondary training module 1103, wherein:
[0158] The data acquisition module 1101 is used to acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has corresponding actual segmentation results, and the second sample X-ray industrial image data has no corresponding actual segmentation results.
[0159] A primary training module 1102 is used to input the second sample X-ray industrial image data into the image encoder of the X-ray image segmentation model to be trained to obtain a first predicted image code, and to input the second sample X-ray industrial image data into the image encoder of a pre-built reference image segmentation model to obtain a second predicted image code, and to train the image encoder of the X-ray image segmentation model to be trained according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model.
[0160] The secondary training module 1103 is used to input the first sample X-ray industrial image data into the initial X-ray image segmentation model, obtain the predicted segmentation result corresponding to the first sample X-ray industrial image data, and train the image encoder and output decoder of the initial X-ray image segmentation model according to the difference between the predicted segmentation result and the actual segmentation result to obtain a trained X-ray image segmentation model.
[0161] In one embodiment, the one-time training module 1102 is also used to obtain a mean square error loss function pre-set for the image encoder of the X-ray image segmentation model to be trained, and to obtain a first weight pre-set for the image encoder of the X-ray image segmentation model to be trained; obtain the corresponding image coding loss value according to the difference and mean square error loss function; and adjust the first weight accordingly based on the image coding loss value until the mean square error loss function converges to obtain the initial X-ray image segmentation model.
[0162] In one embodiment, the initial X-ray image segmentation model further includes a prompt encoder, and the secondary training module 1103 further includes:
[0163] The image code acquisition submodule is used to input the first sample X-ray industrial image data into the image encoder of the initial X-ray image segmentation model to obtain the first sample image code corresponding to the first sample X-ray industrial image data.
[0164] The prompt code acquisition submodule is used to acquire a preset prompt code from the prompt encoder; the prompt code is obtained based on the user prompt information preset in the prompt code;
[0165] The segmentation result acquisition submodule is used to input the first sample image code and the prompt code into the output decoder of the initial X-ray image segmentation model to obtain the predicted segmentation result corresponding to the first sample X-ray industrial image data.
[0166] In an exemplary embodiment, the image encoder includes a stem module, a lightweight module and a basic quantity module, and the image encoding acquisition submodule is further used to input the first sample X-ray industrial image data into the stem module to obtain the first sample X-ray industrial image data with the image size reduced by a preset multiple; input the first sample X-ray industrial image data with the image size reduced by a preset multiple into the lightweight module to obtain the initial spatial features associated with the first sample X-ray industrial image data; input the initial spatial features into the basic quantity module to obtain the spatial features corresponding to the initial spatial features; and obtain the first sample image code corresponding to the first sample X-ray industrial image data according to the spatial features.
[0167] In one embodiment, the hint coding includes sparse hint coding and dense hint coding; the output decoder is pre-set with learnable parameters, and the segmentation result acquisition submodule is further used to generate corresponding position coding according to the first sample image coding; the learnable parameters and the sparse hint coding are spliced to obtain an output unit, and the dense hint coding and the first sample image coding are added to obtain a fused image coding; based on the position coding, the output unit and the fused image coding, the predicted segmentation result corresponding to the first sample X-ray industrial image data is obtained.
[0168] In one embodiment, the output decoder is pre-set with a second weight, and the one-time training module 1102 is further used to obtain the corresponding model loss value based on the difference between the predicted segmentation result and the actual segmentation result, and the focus loss function and the sieve loss function pre-set for the initial X-ray image segmentation model; based on the model loss value, the first weight and the second weight are correspondingly adjusted until the focus loss function and the sieve loss function converge, thereby obtaining a trained X-ray image segmentation model.
[0169] In an exemplary embodiment, the data acquisition module 1101 is also used to acquire original first X-ray industrial image data and original second X-ray industrial image data; adjust the image size of the original first X-ray industrial image data and the original second X-ray industrial image data, and perform random cropping to obtain cropped first X-ray industrial image data and second X-ray industrial image data; horizontally flip the cropped first X-ray industrial image data and second X-ray industrial image data according to a preset probability to obtain flipped first X-ray industrial image data and second X-ray industrial image data; convert the flipped first X-ray industrial image data and second X-ray industrial image data into tensor format and perform normalization processing to obtain first sample X-ray industrial image data and second sample X-ray industrial image data.
[0170] In one embodiment, Fig.12 As shown, an X-ray image segmentation device is provided, including: an acquisition module 1201 and an input module 1202, wherein:
[0171] The acquisition module 1201 is used to acquire target X-ray industrial image data to be segmented.
[0172] The input module 1202 is used to input the target X-ray industrial image data into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, wherein the X-ray image segmentation model is trained by the training method of the X-ray image segmentation model according to any one of claims 1 to 7.
[0173] In one embodiment, the input module 1202 is further used to input the target X-ray industrial image data into the image encoder of the X-ray image segmentation model to obtain the target image code associated with the target X-ray industrial image data, and obtain the hint code from the hint encoder of the X-ray image segmentation model; input the target image code and the hint code into the output decoder of the X-ray image segmentation model to obtain the predicted segmentation result corresponding to the target X-ray industrial image data.
[0174] The training device of the X-ray image segmentation model and each module in the X-ray image segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0175] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.13 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the first sample X-ray industrial image data and the second sample X-ray industrial image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for an X-ray image segmentation model is implemented.
[0176] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0177] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the training method of the X-ray image segmentation model and / or the X-ray image segmentation method of the above-mentioned embodiment is implemented.
[0178] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the training method of the X-ray image segmentation model and / or the X-ray image segmentation method of the above-mentioned embodiment is implemented.
[0179] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the training method of the X-ray image segmentation model and / or the X-ray image segmentation method of the above-mentioned embodiment.
[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0181] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0182] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0183] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A training method for an X-ray image segmentation model, It is characterized in that The method comprises: Acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data has no corresponding actual segmentation result; Inputting the second sample X-ray industrial image data into an image encoder of an X-ray image segmentation model to be trained to obtain a first predicted image code, and inputting the second sample X-ray industrial image data into an image encoder of a pre-built reference image segmentation model to obtain a second predicted image code, and training the image encoder of the X-ray image segmentation model to be trained according to a difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model; The first sample X-ray industrial image data is input into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and based on the difference between the predicted segmentation result and the actual segmentation result, the image encoder and the output decoder of the initial X-ray image segmentation model are trained to obtain a trained X-ray image segmentation model.
2. The method according to claim 1, It is characterized in that The step of training the image encoder of the to-be-trained X-ray image segmentation model according to the difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model comprises: Acquire a mean square error loss function preset by an image encoder for the to-be-trained X-ray image segmentation model, and acquire a first weight preset by an image encoder for the to-be-trained X-ray image segmentation model; According to the difference and the mean square error loss function, obtaining a corresponding image coding loss value; The first weight is adjusted accordingly based on the image coding loss value until the mean square error loss function converges, thereby obtaining the initial X-ray image segmentation model.
3. The method according to claim 1, It is characterized in that The initial X-ray image segmentation model also includes a prompt encoder; The step of inputting the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data includes: Inputting the first sample X-ray industrial image data into the image encoder of the initial X-ray image segmentation model to obtain a first sample image code corresponding to the first sample X-ray industrial image data; Acquire a preset prompt code from the prompt encoder; the prompt code is obtained based on user prompt information preset in the prompt encoder; The first sample image code and the hint code are input into the output decoder of the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data.
4. The method according to claim 3, It is characterized in that The image encoder includes a stem module, a lightweight module and a basic module; The step of inputting the first sample X-ray industrial image data into the image encoder of the initial X-ray image segmentation model to obtain a first sample image code corresponding to the first sample X-ray industrial image data includes: Inputting the first sample X-ray industrial image data into the stem module to obtain the first sample X-ray industrial image data with image size reduced by a preset multiple; Inputting the first sample X-ray industrial image data whose image size is reduced by a preset multiple into the lightweight module to obtain initial spatial features associated with the first sample X-ray industrial image data; Inputting the initial spatial feature into the basic quantity module to obtain a spatial feature corresponding to the initial spatial feature; A first sample image code corresponding to the first sample X-ray industrial image data is acquired according to the spatial feature.
5. The method according to claim 3, It is characterized in that The hint coding includes sparse hint coding and dense hint coding; the output decoder is pre-set with learnable parameters; The step of inputting the first sample image code and the hint code into the output decoder of the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data comprises: Generate a corresponding position code according to the first sample image code; The learnable parameter and the sparse hint code are concatenated to obtain an output unit, and the dense hint code and the first sample image code are added to obtain a fused image code; Based on the position code, the output unit and the fused image code, a predicted segmentation result corresponding to the first sample X-ray industrial image data is obtained.
6. The method according to claim 2, It is characterized in that The output decoder is pre-set with a second weight; The step of training the image encoder and the output decoder of the initial X-ray image segmentation model according to the difference between the predicted segmentation result and the actual segmentation result to obtain the trained X-ray image segmentation model comprises: According to the difference between the predicted segmentation result and the actual segmentation result, and the focus loss function and the sieve loss function pre-set for the initial X-ray image segmentation model, obtaining a corresponding model loss value; The first weight and the second weight are adjusted accordingly based on the model loss value until the focus loss function and the sieve loss function converge, thereby obtaining a trained X-ray image segmentation model.
7. The method according to claim 1, It is characterized in that The obtaining of the first sample X-ray industrial image data and the second sample X-ray industrial image data comprises: Acquiring original first X-ray industrial image data and original second X-ray industrial image data; Adjusting the image sizes of the original first X-ray industrial image data and the original second X-ray industrial image data, and performing random cropping to obtain cropped first X-ray industrial image data and second X-ray industrial image data; horizontally flipping the cropped first X-ray industrial image data and the second X-ray industrial image data according to a preset probability to obtain flipped first X-ray industrial image data and second X-ray industrial image data; The flipped first X-ray industrial image data and the second X-ray industrial image data are converted into a tensor format and normalized to obtain the first sample X-ray industrial image data and the second sample X-ray industrial image data.
8. A method for segmenting an X-ray image, It is characterized in that The method comprises: Acquire X-ray industrial image data of the target to be segmented; The target X-ray industrial image data is input into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, wherein the X-ray image segmentation model is trained by the training method of the X-ray image segmentation model according to any one of claims 1 to 7.
9. The method according to claim 8, It is characterized in that The step of inputting the target X-ray industrial image data into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data includes: Inputting the target X-ray industrial image data into an image encoder of the X-ray image segmentation model to obtain a target image code associated with the target X-ray industrial image data, and obtaining a hint code from a hint encoder of the X-ray image segmentation model; The target image code and the prompt code are input into the output decoder of the X-ray image segmentation model to obtain the predicted segmentation result corresponding to the target X-ray industrial image data.
10. A training device for an X-ray image segmentation model, It is characterized in that The device comprises: A data acquisition module, used to acquire first sample X-ray industrial image data and second sample X-ray industrial image data; the first sample X-ray industrial image data has a corresponding actual segmentation result, and the second sample X-ray industrial image data does not have a corresponding actual segmentation result; a primary training module, configured to input the second sample X-ray industrial image data into an image encoder of an X-ray image segmentation model to be trained to obtain a first predicted image code, and input the second sample X-ray industrial image data into an image encoder of a pre-built reference image segmentation model to obtain a second predicted image code, and train the image encoder of the X-ray image segmentation model to be trained according to a difference between the first predicted image code and the second predicted image code to obtain an initial X-ray image segmentation model; A secondary training module is used to input the first sample X-ray industrial image data into the initial X-ray image segmentation model to obtain a predicted segmentation result corresponding to the first sample X-ray industrial image data, and train the image encoder and output decoder of the initial X-ray image segmentation model according to the difference between the predicted segmentation result and the actual segmentation result to obtain a trained X-ray image segmentation model.
11. An X-ray image segmentation device, It is characterized in that The device comprises: An acquisition module, used for acquiring target X-ray industrial image data to be segmented; An input module is used to input the target X-ray industrial image data into a pre-trained X-ray image segmentation model to obtain a predicted segmentation result of the target X-ray industrial image data, wherein the X-ray image segmentation model is trained by the training method of the X-ray image segmentation model according to any one of claims 1 to 7.
12. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.
13. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
14. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
