Model training method and lesion image segmentation method and device

Abstract

The invention provides a model training method and a lesion image segmentation method and device, and belongs to the technical field of medical image analysis. The model training method comprises the following steps: constructing a U-shaped image segmentation network, the U-shaped image segmentation network comprising an encoder and a decoder; wherein the encoder is used for encoding the lesion image to obtain an encoding feature; the decoder is used for decoding the coding features to obtain a decoded lesion image; a boundary key point of a lesion area in the real lesion image is extracted through a differential evolution boundary enhancement module, and a boundary area of the real lesion image is obtained; and based on the loss of the boundary region of the decoded lesion image and the real lesion image and the loss between the decoded lesion image and the real lesion image, optimizing the parameters of the U-shaped image segmentation network, the parameter-optimized U-shaped image segmentation network being used for lesion image segmentation. The model obtained by training based on the method can realize image segmentation with low parameter quantity, low calculation load and high precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image analysis, and in particular to a model training method, a lesion image segmentation method, a model training device, a lesion image segmentation device, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] Medical image segmentation is a key task in clinical diagnosis and treatment, aiming to accurately outline the contours of organs or tumors. Deep learning has made significant progress in the medical field, especially in the automatic diagnosis and segmentation of skin lesions, which has become an important research hotspot.

[0003] However, existing image lesion segmentation methods are mainly based on complex deep learning techniques. Deep learning techniques require a large number of parameters and computing requirements to meet image segmentation accuracy, which greatly limits their application scenarios. Therefore, developing a lightweight and high-precision image segmentation method is still an urgent and unsolved challenge. SUMMARY

[0004] The present application aims to at least one of the above technical problems or at least provide a useful commercial option. In view of this, the present application provides a model training method and lesion image segmentation method and device, which can realize low parameter quantity, low computing load, and high precision lesion image segmentation.

[0005] In a first aspect of the present application, a model training method is provided, which comprises: constructing a U-shaped image segmentation network, the U-shaped image segmentation network comprising an encoder and a decoder; wherein the encoder is used to encode a lesion image to obtain encoded features; the decoder is used to decode the encoded features to obtain a decoded lesion image; extracting boundary key points of a lesion area in a real lesion image through a differential evolution boundary enhancement module to obtain a real lesion image boundary area; based on the loss of the decoded lesion image and the real lesion image boundary area, and the loss between the decoded lesion image and the real lesion image, optimizing the parameters of the U-shaped image segmentation network, and the U-shaped image segmentation network after parameter optimization is used for lesion image segmentation.

[0006] In some implementable manners, the boundary key points of the lesion area in the real lesion image are extracted by the differential evolution boundary enhancement module to obtain a boundary area of the real lesion image, including: initializing a population, each individual being a two-dimensional coordinate set, representing a plurality of key points in a set boundary range, used to construct a polygon approximating the lesion boundary; performing a mutation operation to generate a mutation vector; combining the original individual and the mutation vector through a crossover operation to generate a test vector; evaluating the fitness according to the intersection-over-union between the polygon formed by the test vector and the real mask image, and replacing the current vector with the test vector; under the constraint of a maximum number of iterations or a convergence condition, obtaining an individual with the highest fitness as the boundary key point set of the real lesion image to obtain the boundary area of the real lesion image.

[0007] In some implementable manners, the U-shaped image segmentation network parameters are optimized based on the loss between the decoded lesion image and the real lesion image, and the loss of the decoded lesion image and the boundary area of the real lesion image, including: based on a deep supervision mechanism, supervising the mask image and the boundary image generated at different scales; based on the intersection-over-union loss, minimizing the difference between the decoded lesion image and the boundary area of the real lesion image; based on the cross-entropy loss and the Dice loss, minimizing the difference between the decoded lesion image and the real lesion image; based on the difference, optimizing the U-shaped image segmentation network parameters.

[0008] In some implementable manners, the encoder is used to encode the lesion image to obtain an encoded feature, including: embedding a channel attention mixed separable convolution module at a first predetermined stage of the encoder, extracting a first feature of the lesion image through a depth separable convolution and a channel attention mechanism; embedding a grouping Hadamard product channel shuffling module at a second predetermined stage of the encoder, respectively, enhancing the first feature through a grouping Hadamard product and a channel scattering operation to obtain a second feature; obtaining the encoded feature according to the first feature and the second feature.

[0009] In some implementable manners, the number of channels of the encoder is {8, 12, 16, 32, 48, 64} in turn.

[0010] In some implementable manners, the channel attention mixed separable convolution module is embedded at the first predetermined stage of the encoder to extract the first feature of the lesion image through a depth separable convolution and a channel attention mechanism, including: performing channel expansion through 1x1 point convolution; performing feature extraction through 3x3 depth separable convolution; performing weighting operation through channel attention; and improving non-linear expression ability through an activation function.

[0011] In some implementations, the embedding a grouped Hadamard product channel shuffle module at the second predetermined stage of the encoder respectively enhances the first feature by a grouped Hadamard product and channel scatter operation to obtain a second feature, including: dividing the first feature into multiple groups along the channel dimension; applying a Hadamard product and convolution operation to each group of features respectively to extract local features of different dimensions; and splicing each group of features and performing a channel shuffle operation to enhance cross-group information flow.

[0012] In a second aspect of the present application, a lesion image segmentation method is provided, including: obtaining a lesion image to be segmented; inputting the lesion image to be segmented into a U-shaped image segmentation network to obtain a lesion image segmentation result; wherein the U-shaped image segmentation network is trained by the method of the first aspect.

[0013] In a third aspect of the present application, a model training device is provided, including: a network construction module for constructing a U-shaped image segmentation network, the U-shaped image segmentation network including an encoder and a decoder; wherein the encoder is used to encode a lesion image to obtain an encoded feature; the decoder is used to decode the encoded feature to obtain a decoded lesion image; an image boundary region determination module for extracting boundary key points of a lesion region in a real lesion image by a differential evolution boundary enhancement module to obtain a real lesion image boundary region; a training module for optimizing parameters of the U-shaped image segmentation network based on a loss of the decoded lesion image and the real lesion image boundary region, and a loss between the decoded lesion image and the real lesion image, and the U-shaped image segmentation network after parameter optimization is used for lesion image segmentation.

[0014] In a fourth aspect of the present application, a lesion image segmentation device is provided, including: a lesion image acquisition module for acquiring a lesion image to be segmented; a prediction module for inputting the lesion image to be segmented into a U-shaped image segmentation network to obtain a lesion image segmentation result; wherein the U-shaped image segmentation network is trained by the method of the first aspect.

[0015] In a fifth aspect of the present application, an electronic device is provided, including: a processor and a memory; the aforementioned memory is used to store a computer program; the aforementioned processor is used to execute the aforementioned computer program to implement the model training method of the first aspect or the lesion image segmentation method of the second aspect.

[0016] In a sixth aspect of the present application, a computer readable storage medium is provided, which stores computer instructions or programs, when the computer instructions or programs are executed on a computer, the model training method of the first aspect or the lesion image segmentation method of the second aspect is executed.

[0017] In a seventh aspect of the present application, a computer program product is provided, which includes computer instructions, when part or all of the computer instructions are executed on a computer, the model training method of the first aspect or the lesion image segmentation method of the second aspect is executed.

[0018] Compared with the prior art, the scheme according to the embodiments of the present application can achieve at least one of the following technical advantages:

[0019] The existing skin lesion segmentation method has large parameter quantity and high computing load, and is not suitable for the challenge of the medical environment with limited computing resources in reality. The model training method provided in the present application combines a differential evolution boundary enhancement module. In the training process, the key points of the boundary of the real lesion area extracted based on evolutionary optimization are introduced. By guiding the network to focus on the prediction error of the boundary area, the whole image loss and the boundary area loss are considered in the optimization process, thereby improving the recognition ability of the model to the edge of the lesion area, and effectively improving the segmentation precision of the boundary of the lesion area.

[0020] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0022] Figure 1 The system architecture diagram according to an embodiment of the present application is shown in the figure;

[0023] Figure 2 The flowchart of the model training method according to an embodiment of the present application is shown in the figure;

[0024] Figure 3 The U-shaped image segmentation network structure diagram according to an embodiment of the present application is shown in the figure;

[0025] Figure 4 The flowchart of the lesion image segmentation method according to another embodiment of the present application is shown in the figure;

[0026] Figure 5 a schematic diagram of a model training apparatus according to an embodiment of the present application;

[0027] Figure 6 a schematic diagram of a lesion image segmentation apparatus according to an embodiment of the present application;

[0028] Figure 7 a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0030] It should be noted that the terms "first", "second", and the like in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server including a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0031] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0032] Before introducing the technical solutions of the present application, the related knowledge of the present application will be introduced as follows:

[0033] In the embodiments of the present application, the "U-shaped image segmentation network" is a deep neural network based on an encoder-decoder structure, which generally includes a downsampling path (encoder) and an upsampling path (decoder), and its structure presents a "U" shape. The encoder extracts high-dimensional semantic features of the input image, and the decoder gradually restores the spatial resolution of the image and combines the shallow features of the encoder to achieve accurate image segmentation.

[0034] In the embodiments of the present application, the "encoder" is a neural network module that converts the original input data into a high-dimensional feature representation with semantic information, which is used to extract key information in the input and compress redundant content.

[0035] In the embodiments of the present application, the "decoder" is a neural network module that restores or converts the high-dimensional feature representation generated by the encoder into the target form output (such as an image, a sequence or a label map), which is used to generate the final result.

[0036] In the embodiments of the present application, the "deep supervision mechanism" is a training strategy that introduces auxiliary loss functions at the intermediate layers of the neural network to jointly supervise multi-scale or multi-layer outputs, so as to accelerate the convergence speed and improve the model performance. In the traditional neural network training, the loss is usually calculated only at the output layer and then backpropagated. However, the deep supervision mechanism introduces auxiliary loss functions (auxiliary loss) at the intermediate layers or multiple output branches of the network to supervise the intermediate features, guiding the network to learn more multi-level semantic information during the training process.

[0037] In the embodiments of the present application, the "Intersection over Union (IoU)" is an index for measuring the degree of overlap between two image regions, which is used to evaluate the similarity between the constructed polygon boundary and the real mask in the boundary point optimization. The higher the value, the better the boundary coincidence.

[0038] In the embodiments of the present application, the "Dice loss" is a function for measuring the degree of overlap between the predicted region and the real region, which is particularly suitable for medical image segmentation scenarios where the foreground and background categories are imbalanced.

[0039] In the embodiments of the present application, the "cross-entropy loss" is a function for measuring the difference between the model prediction result and the real label.

[0040] In the embodiments of the present application, the "training set" is the data used to train the model. It contains a series of input samples (independent variables) and corresponding known outputs (dependent variables). In the training phase, the deep learning model uses the samples in the training set to learn and adjust the parameters (weights) of the model to minimize the error between the predicted results and the true labels. By constantly trying different parameters and algorithms, the model gradually learns the rules and characteristics between samples, and determines more accurate prediction ability.

[0041] In the embodiments of the present application, the "test set" is a data set used to evaluate the performance of the deep learning model. It also contains input samples (independent variables) and corresponding true outputs (dependent variables), but the samples in the test set have not been used in the training phase of the model, so that the data in the test set is unknown to the model. After training is completed, the test set is used to evaluate the performance of the model, that is, the model predicts the samples in the test set and compares them with the true labels. By comparing the predicted results and the true labels, the performance of the model on unknown data is evaluated, and the generalization ability and prediction accuracy are judged.

[0042] The technical problems, inventive concepts and system architecture to be solved by the embodiments of the present application will be described below:

[0043] As described above, at present, the related skin lesion segmentation method is mainly based on complex deep learning technology, which has high requirements for resource allocation, resulting in great limitation of its application scene.

[0044] To solve the technical problems of the related prior art, the present application proposes a model training method. This method realizes high-precision image segmentation results under the conditions of low parameter quantity and low computational complexity by constructing a lightweight U-shaped image segmentation network combined with a differential evolution enhancement module.

[0045] In some embodiments, the system architecture of the embodiments of the present application is as shown in Figure 1 .

[0046] Figure 1 For a system architecture schematic diagram related to the embodiments of the present application, the system architecture relates to a user device 101, a data acquisition device 102, a training device 103, an execution device 104, a database 105 and a content library 106.

[0047] The data acquisition device 102 is configured to read the training samples from the content library 106 and store the read training samples in the database 105. The training samples related to the embodiments of the present application can come from biological related literature or public databases.

[0048] The training device 103 trains the U-shaped image segmentation network based on the training samples maintained in the database 105, so that the trained U-shaped image segmentation network can perform image segmentation with high precision.

[0049] In addition, with reference to Figure 2 The execution device 104 is configured with an I / O interface 107 to interact with external devices. For example, the I / O interface receives the input instruction sent by the user device 101. The input instruction includes a lesion image to be segmented. The computing module 108 in the execution device 104 processes the input instruction using the target model to determine the output instruction. The output instruction includes a lesion image segmentation result, and the output instruction is sent to the user device 101 through the I / O interface.

[0050] The user device 101 can include a mobile phone, a tablet computer, a notebook computer, a palm computer, a mobile internet device (MID), a desktop computer, or other terminal devices with a browser function.

[0051] The execution device 104 can be a server.

[0052] For example, the server can be a rack server, a blade server, a tower server, or a cabinet server, etc. The server can be a standalone test server, or a test server cluster composed of multiple test servers.

[0053] In this embodiment, the execution device 104 is connected with the user device 101 through a network. The network can be an intranet, the Internet, a global system for mobile communication (GSM), a wideband code division multiple access (WCDMA), a fourth generation (4G) network, a fifth generation (5G) network, Bluetooth, wireless fidelity (Wi-Fi), a call network, or other wireless or wired networks.

[0054] It should be noted that, Figure 1The system architecture provided by the embodiments of the present application is only a schematic diagram, and the positional relationship between the devices, elements, modules and the like shown in the diagram does not constitute any limitation. In some embodiments, the data acquisition device 102, the user device 101, the training device 103 and the execution device 104 can be the same device. The database 105 can be distributed on one server or multiple servers, and the content library 106 can be distributed on one server or multiple servers.

[0055] The technical solutions of the present application will be described in detail below.

[0056] Figure 2 A flowchart of a model training method according to an embodiment of the present application. The method can be executed by a training device, for example, the training device 103 shown in FIG. 1, but is not limited thereto. For example, as shown in FIG. 2, the method can include the following steps. Figure 1 Figure 2

[0057] S210, constructing a U-shaped image segmentation network, the U-shaped image segmentation network comprising an encoder and a decoder; wherein the encoder is configured to encode a lesion image to obtain an encoded feature; and the decoder is configured to decode the encoded feature to obtain a decoded lesion image.

[0058] In some implementations, the U-shaped image segmentation network can be a U-net network.

[0059] It can be understood that the present application does not specifically limit the type of lesion image, for example, it can be a skin lesion image.

[0060] In some implementations, the encoder is configured to encode a lesion image to obtain an encoded feature, comprising: embedding a channel attention mixed separable convolution module at a first predetermined stage of the encoder, extracting a first feature of the lesion image through a combination of depth separable convolution and channel attention mechanism; embedding a group Hadamard product channel shuffling module at a second predetermined stage of the encoder, respectively, and enhancing the first feature through group Hadamard product and channel scattering operations to obtain a second feature; and obtaining an encoded feature according to the first feature and the second feature.

[0061] In this embodiment, the encoder of the U-shaped image segmentation network is configured to perform feature extraction and encoding processing on the input lesion image. The encoder adopts a multi-stage down-sampling structure to gradually extract image semantic information at different levels.

[0062] ​​Specifically, in the first predetermined stage of the encoder, a channel attention mixed separable convolution module is embedded. This module combines a depth separable convolution with a channel attention mechanism, first expands the channels through a 1x1 point convolution, then extracts the spatial features of the image through a 3x3 depth separable convolution, and finally uses the channel attention mechanism to weight the importance of different channels and introduces an activation function to enhance the non-linear expression ability. The features generated by this module are the first features, which are shallow features or low-order features of the encoder, mainly representing the local details and edge texture information of the image.

[0063] In the second predetermined stage of the encoder, a grouped Hadamard product channel shuffling module is embedded. This module divides the first features into multiple subgroups along the channel dimension, and each group performs Hadamard product and convolution operations to obtain deep features with more abstract expression ability; then the features of each group are spliced and subjected to channel shuffling operation to improve the information interaction efficiency between different channels. The features generated by this module are the second features, which are deep features or high-order features of the encoder, mainly representing the semantic structure and global relationship of the lesion area in the image.

[0064] Finally, the output of the encoder, the encoded features, is obtained, which provides multi-scale and semantically rich feature representation support for the fine segmentation in the decoder stage.

[0065] It can be understood that the aforementioned encoder adopts a multi-stage down-sampling structure, which can be six stages, eight stages, etc. If the down-sampling structure adopted is six stages, the first predetermined stage is the first three stages, and the second predetermined stage is the last three stages. The same applies to the others, which will not be repeated here.

[0066] In some implementations, the encoder of the U-shaped image segmentation network adopts a six-stage step-by-step down-sampling structure, and the convolution modules of each stage are used to extract image features at different scales. Preferably, the channel numbers of each stage of the encoder are set to {8, 12, 16, 32, 48, 64} in turn. This channel configuration scheme realizes a significant reduction in network parameter quantity and saving of computing resources on the premise of ensuring that the network has multi-scale expression ability. A smaller channel number (e.g. 8 and 12) is used in the initial stage to capture low-level features, and the channel number is gradually increased to 64 in the subsequent stages, which helps to fully extract deep semantic information. At the same time, by nonlinearly combining features at different scales, it helps to improve the expression ability of the lesion area and the segmentation accuracy.

[0067] S220, extracting boundary key points of the lesion area in the real lesion image through the differential evolution boundary enhancement module to obtain a boundary area of the real lesion image;

[0068] In the automatic segmentation of lesion images, traditional segmentation networks usually rely on pixel-level loss functions (such as cross-entropy loss) to optimize model performance. However, relying solely on pixel-level supervision signals often overlooks the fine structure of lesion region boundaries, especially in cases where lesion boundaries are ambiguous, shapes are complex, or areas are small. The network has difficulty accurately capturing edge information, leading to inaccurate boundary predictions and fuzzy contours, which affects the overall segmentation effect and the accuracy of clinical diagnosis.

[0069] To solve the above problems, the present embodiment proposes a boundary enhancement module based on differential evolution algorithm, which can automatically extract boundary key points in real lesion images, construct more accurate boundary regions, and provide detailed boundary supervision information to guide the segmentation network to better learn boundary features.

[0070] Implementation steps of differential evolution boundary enhancement module

[0071] Through the optimization process of the evolutionary algorithm, the module finds the best set of boundary key points, with the following steps:

[0072] Step 1: Initialize population

[0073] Around the boundary of the real lesion mask image, a boundary range is pre-set.

[0074] Within this range, several individuals (i.e. candidate solutions) are initialized, each consisting of multiple two-dimensional coordinate points. These coordinate points collectively form a polygon that approximates the boundary of the lesion region.

[0075] For example, assume each polygon is composed of 20 key points, and the coordinates (x, y) of each point are randomly generated within the pre-set boundary box in the image.

[0076] Step 2: Mutation operation

[0077] Mutate the coordinate points in each individual, i.e. generate a new set of candidate points through the mutation strategy defined in the differential evolution algorithm.

[0078] This operation aims to explore better boundary point configurations and escape local optima.

[0079] Step 3: Crossover operation

[0080] Cross the mutated individuals with the original individuals to generate new trial individuals.

[0081] By combining the coordinate points of different individuals, the diversity of the population is enhanced, and the search efficiency is improved.

[0082] Step 4: Fitness evaluation and selection

[0083] The intersection over union (IoU) between the polygon corresponding to the trial individual and the real mask image is calculated as a fitness indicator.

[0084] If the IoU of the trial individual is better than the current individual, the current individual is replaced by the trial individual.

[0085] Step 5: Iteration termination condition

[0086] The mutation, crossover and selection operations are continuously iterated until the maximum number of iterations or fitness convergence is reached.

[0087] The individual with the highest fitness is finally selected, and the corresponding two-dimensional coordinate point is the boundary key point set used to construct the boundary region of the real lesion image.

[0088] For example, taking an actual lesion image as an example, the real mask shows that the lesion region edge is relatively fuzzy and irregular in shape. The initial polygon key points are randomly distributed near the boundary, and the initial IoU is about 0.65.

[0089] After 50 generations of differential evolution algorithm iteration, the boundary key points are gradually adjusted, and the final polygon contour closely follows the real lesion boundary, with an IoU of 0.92, which significantly approximates the real boundary shape.

[0090] The optimized boundary serves as a boundary supervision signal during network training, effectively guiding the model to distinguish the subtle edge differences between the lesion region and the background, and improving the accuracy of boundary recognition.

[0091] The embodiment introduces a differential evolution boundary enhancement module, which can automatically and efficiently extract the boundary key points of the real lesion image, construct an accurate boundary region, effectively improve the recognition ability and segmentation accuracy of the segmentation network for the lesion edge, and solve the problem of poor segmentation effect of traditional methods in the case of fuzzy boundary and complex shape, thereby improving the overall accuracy of lesion image segmentation.

[0092] S230, based on the loss of the decoded lesion image and the boundary region of the real lesion image, and the loss between the decoded lesion image and the real lesion image, the U-shaped image segmentation network parameters are optimized, and the U-shaped image segmentation network after parameter optimization is used for lesion image segmentation.

[0093] In the embodiment, for the problem of lesion image segmentation network parameter optimization, a joint optimization strategy based on multiple loss functions is adopted to improve the segmentation accuracy and boundary recognition ability.

[0094] Based on the deep supervision mechanism, mask maps and boundary maps are generated at different scales in the network, and supervision signals are applied to them. Through the supervision of multi-scale features, the network can more effectively capture the detailed information and boundary features of the lesion area, solving the problem of inaccurate segmentation and fuzzy edges caused by traditional single-scale supervision.

[0095] The Intersection over Union (IoU) loss function is used to minimize the difference between the decoded lesion image and the real lesion image boundary region. This loss function enhances the network's attention to lesion boundaries, helping to improve the accuracy of boundary extraction and solving common problems such as fuzzy or broken boundaries.

[0096] Combining the cross-entropy loss and Dice loss, the difference between the decoded lesion image and the real lesion image is minimized. Cross-entropy loss promotes the accuracy of pixel classification, while Dice loss enhances the sensitivity to small lesion areas. The two work together to effectively improve the overall segmentation quality.

[0097] Through the joint optimization of the above multi-loss function, the segmentation effect of the U-shaped image segmentation network in the lesion area and the boundary area is significantly improved, and the generalization ability and robustness of the model are improved.

[0098] For the sake of detail, the following is combined Figure 3 The scheme of the present application is described exemplarily:

[0099] Step 1: Construct a lightweight U-shaped segmentation network, which includes an encoder and a decoder. The encoder contains six stages, and the channel numbers are {8, 12, 16, 32, 48, 64} in turn.

[0100] Step 2: Embed a channel attention mixed separable convolution module (SCGS module) in the first 3 stages of the encoder, extract shallow features through depth separable convolution and channel attention mechanism;

[0101] Step 2.1: First, perform channel expansion through 1x1 point convolution;

[0102] Step 2.2: Then perform 3x3 depth separable convolution to extract features;

[0103] Step 2.3: Then apply the squeeze and excitation module for channel attention weighting;

[0104] Step 2.4: Finally, use the GELU activation function to improve the non-linear expression ability;

[0105] The output feature calculation process is as follows:

[0106] Output=BN2(Conv1×1(SE(GELU(BN1(Conv3×3^DW(GELU(Conv1×1(x))))))))).

[0107] Step 3: A grouped Hadamard product channel shuffling module (HGAS module) is embedded in the last three stages of the encoder to enhance high-order feature expression through grouped Hadamard product and channel shuffling operations;

[0108] Step 3.1: Divide the input features into four groups along the channel dimension;

[0109] Assume the input feature map is Divide it into four groups [x1,x2,x3,x4];

[0110] Step 3.2: Apply Hadamard product and convolution operations to each set of features to extract local features of different dimensions;

[0111] y1=x1⊙Conv xγ (resize(P xy ,H,W));

[0112] y2=permute(x2⊙Conv_zx(resize(P zx ,H,W)),(0,2,3,1));

[0113] y3=permute(x3⊙Conv_zy(resize(P zy ,H,W)),(0,2,1,3));

[0114] y4 = DWConv(x4);

[0115] Step 3.3: Concatenate the features of each group and perform channel shuffling to enhance the flow of information across groups;

[0116] Output: x out =ChannelShuffle(concat(y1,y2,y3,y4])).

[0117] Step 4: A feature fusion module (fusion layer) is used in the decoder to perform weighted fusion of the skip connection, the output of the previous decoding stage, and the region and boundary prediction map of the previous layer to generate new decoding features.

[0118] Step 5: Extract boundary key points of the skin lesion area in the image through the differential evolution boundary enhancement module, and optimize the key point set to maximize the intersection-over-union ratio with the true mask.

[0119] Step 5.1: Initialize a population consisting of several two-dimensional coordinates.

[0120] Step 5.2: Perform mutation operation, and generate mutation vector by adopting DE / rand / 1 strategy.

[0121] Step 5.3: Perform crossover operation to generate trial vector.

[0122] Step 5.4: Calculate the intersection over union loss between the polygon formed by the trial vector and the real mask as fitness.

[0123] Step 5.5: Update the population individuals according to the fitness until the maximum number of iterations or the convergence condition is reached.

[0124] The optimization process of each generation individual xi is as follows:

[0125] v i =x r1 +F·(x r2 -x r3 );

[0126] u i =Crossover(x i ,v i ,CR);

[0127] f(u i )=IoU(polygon(u i ),b);

[0128] If f(u i )>f(x i ), then x i ←u i ;

[0129] Finally, the optimal solution x best is output.

[0130] Step 6: Utilize deep supervision mechanism to impose supervision on the mask map and boundary map generated at different scales, and jointly optimize network parameters by Dice loss, IoU loss and cross-entropy loss to obtain the optimal skin lesion segmentation result.

[0131] In some implementable manners, the training set and test set of the model training method of the present application adopt a public skin disease dataset, and the performance better than the existing model can be realized under the condition of low parameter quantity and low calculation complexity.

[0132] Figure 4 A flowchart of a lesion image segmentation method of an embodiment of the present application, which can be executed by an execution device, for example, can be executed by the execution device 104 shown in Figure 1 , but is not limited thereto, as shown in Figure 4 , the method can include:

[0133] S410, obtaining an image of the lesion to be segmented;

[0134] S420, inputting the lesion image to be segmented into a U-shaped image segmentation network to obtain a lesion image segmentation result; wherein, the U-shaped image segmentation network is trained by the aforementioned model training method.

[0135] This method can efficiently and accurately segment lesion images by using a low-parameter U-shaped image segmentation network constructed using the above-mentioned model training method.

[0136] It is understandable that the platform for the implementation of the above method can be based on the Python environment and implemented in combination with a deep learning framework (such as PyTorch or TensorFlow). The basic operations involved, such as data reading and writing, neural network training and reasoning, are well known in the art and will not be described in detail here. During specific implementation, the method of the present application can realize an automated operation process through software technology. It should be noted that the embodiments described in this specification are only exemplary and not restrictive, and the undescribed parts belong to the prior art.

[0137] The above describes the method provided in the embodiment of the present application, and the following describes the device provided in the embodiment of the present application.

[0138] Figure 5 A schematic diagram of a model training device 500 according to an embodiment of the present application is shown as follows: Figure 5 As shown, the apparatus 500 includes: a network construction module 510, an image boundary region determination module 520 and a training module 530, wherein:

[0139] The network construction module 510 is used to construct a U-shaped image segmentation network, which includes an encoder and a decoder; wherein the encoder is used to encode the lesion image to obtain encoded features; the decoder is used to decode the encoded features to obtain a decoded lesion image; the image boundary area determination module 520 is used to extract the boundary key points of the lesion area in the real lesion image through the differential evolution boundary enhancement module to obtain the boundary area of ​​the real lesion image; the training module 530 is used to optimize the parameters of the U-shaped image segmentation network based on the loss of the boundary area between the decoded lesion image and the real lesion image, and the loss between the decoded lesion image and the real lesion image. The U-shaped image segmentation network after parameter optimization is used for lesion image segmentation.

[0140] In some implementable manners, the boundary key points of the lesion area in the real lesion image are extracted by the differential evolution boundary enhancement module to obtain a boundary area of the real lesion image, including: initializing a population, each individual being a set of two-dimensional coordinates, representing a plurality of key points within a set boundary range, used to construct a polygon approximating the lesion boundary; performing a mutation operation to generate a mutation vector; combining the original individual and the mutation vector through a crossover operation to generate a trial vector; evaluating the fitness according to the intersection-over-union between the polygon formed by the trial vector and the real mask image, and replacing the current vector with the trial vector; under the constraint of a maximum number of iterations or a convergence condition, obtaining an individual with the highest fitness as the boundary key point set of the real lesion image, and obtaining the boundary area of the real lesion image.

[0141] In some implementable manners, the U-shaped image segmentation network parameters are optimized based on the loss between the decoded lesion image and the real lesion image, and the loss of the decoded lesion image and the boundary area of the real lesion image, including: based on a deep supervision mechanism, supervising the mask images and boundary images generated at different scales; based on the intersection-over-union loss, minimizing the difference between the decoded lesion image and the boundary area of the real lesion image; based on the cross-entropy loss and the Dice loss, minimizing the difference between the decoded lesion image and the real lesion image; based on the difference, optimizing the U-shaped image segmentation network parameters.

[0142] In some implementable manners, the encoder is used to encode the lesion image to obtain an encoded feature, including: embedding a channel attention mixed separable convolution module at a first predetermined stage of the encoder, extracting a first feature of the lesion image through a depth separable convolution and a channel attention mechanism; embedding a grouping Hadamard product channel shuffling module at a second predetermined stage of the encoder, respectively, enhancing the first feature through a grouping Hadamard product and a channel scattering operation to obtain a second feature; obtaining an encoded feature according to the first feature and the second feature.

[0143] In some implementable manners, the number of channels of the encoder is {8, 12, 16, 32, 48, 64} in turn.

[0144] In some implementable manners, the channel attention mixed separable convolution module is embedded at the first predetermined stage of the encoder to extract the first feature of the lesion image through a depth separable convolution and a channel attention mechanism, including: performing channel expansion through 1x1 point convolution; performing feature extraction through 3x3 depth separable convolution; performing weighting operation through channel attention; and improving non-linear expression ability through an activation function.

[0145] In some implementable manners, the embedding a grouped Hadamard product channel shuffle module at the second predetermined stage of the encoder respectively, enhancing the first feature through grouped Hadamard product and channel scatter operations to obtain a second feature, comprises: dividing the first feature into multiple groups along the channel dimension; applying Hadamard product and convolution operations to each group of features respectively to extract local features of different dimensions; and splicing each group of features and performing a channel shuffle operation to enhance cross-group information flow.

[0146] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, no longer described here. Specifically, Figure 5 The device shown can perform Figure 2 The corresponding method embodiments, and the aforementioned and other operations and / or functions of each module in the device are respectively to realize Figure 2 The corresponding flow in each method in the device, and to be brief, no longer described here.

[0147] The device of the embodiments of the present application is described above in conjunction with the drawings from the perspective of functional modules. It should be understood that the functional modules can be realized by hardware, or by instructions in the form of software, or by a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits and / or software instructions in the hardware of the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processor execution completion, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps in the above method embodiments.

[0148] Figure 6 A schematic diagram of a lesion image segmentation device 600 according to an embodiment of the present application is shown in Figure 6 The device 600 comprises a lesion image acquisition module 610 and a prediction module 620, wherein

[0149] The lesion image acquisition module 610 is configured to acquire a lesion image to be segmented; the prediction module 620 is configured to input the lesion image to be segmented into a U-shaped image segmentation network to obtain a lesion image segmentation result; wherein the U-shaped image segmentation network is obtained by training the aforementioned model training method.

[0150] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, no longer described here. Specifically,Figure 6 The apparatus 600 shown can perform Figure 4 corresponding method embodiments, and the foregoing and other operations and / or functions of the various modules in the apparatus 600 are respectively for implementing Figure 4 corresponding flows in the various methods in the For brevity, they will not be repeated here.

[0151] The device of the embodiments of the present application is described above in conjunction with the accompanying drawings from the perspective of functional modules. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor and / or instructions in the form of software, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing for execution by the processor, or executed by a combination of hardware and software modules in the code processing. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps in the above method embodiments.

[0152] Figure 7 is a schematic block diagram of an electronic device 700 provided by the embodiments of the present application. The electronic device 700 can be the training device or the execution device described above, but is not limited thereto. As Figure 7 indicated, the electronic device 700 can include:

[0153] The memory 710 is used to store the computer program 730 and transmit the computer program 730 to the processor 720. In other words, the processor 720 can call and run the computer program 730 from the memory 710 to implement the method in the embodiments of the present application.

[0154] For example, the processor 720 can be used to perform the steps in the above method according to the instructions in the computer program 730.

[0155] In some embodiments of the present application, the processor 720 can include but is not limited to:

[0156] A general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

[0157] In some embodiments of the present application, the memory 710 includes, but is not limited to:

[0158] volatile memory and / or non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0159] In some embodiments of the present application, the computer program 730 can be divided into one or more modules, which are stored in the memory 710 and executed by the processor 720 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 730 in the electronic device.

[0160] As shown in Figure 7 The electronic device 700 can further include:

[0161] a transceiver 740, which can be connected to the processor 720 or the memory 710.

[0162] The processor 720 can control the transceiver 740 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 740 can include a transmitter and a receiver. The transceiver 740 can further include an antenna, and the number of antennas can be one or more.

[0163] It should be understood that various components in the electronic device 700 are connected through a bus system, wherein the bus system includes, in addition to a data bus, a power supply bus, a control bus, and a status signal bus.

[0164] According to an aspect of the present application, a computer readable storage medium is provided, and computer instructions or programs are stored on the computer readable storage medium. When the computer instructions or programs are executed by a computer, the computer is enabled to perform the method of the above method embodiments. Alternatively, the embodiments of the present application also provide a computer program product containing instructions, and the instructions are executed by a computer to perform the method of the above method embodiments.

[0165] According to another aspect of the present application, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method of the above method embodiments.

[0166] In other words, when implemented using software, the functions can be implemented in whole or in part using a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the whole or part of the flow or function according to the embodiments of the present application is generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0167] Those skilled in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0168] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interface, device or module, and can be electrical, mechanical or other forms.

[0169] The modules illustrated as separate components may or may not be physically separate, and the components illustrated as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected to implement the embodiments of the present application according to actual needs. For example, the functional modules in various embodiments of the present application can be integrated into one processing module, or each module can be physically present separately, or two or more modules can be integrated into one module.

[0170] The above is merely specific embodiments of the present application, and the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Constructing a U-shaped image segmentation network, the U-shaped image segmentation network includes an encoder and a decoder; wherein the encoder is used to encode the lesion image to obtain encoded features; and the decoder is used to decode the encoded features to obtain a decoded lesion image; The boundary key points of the lesion area in the real lesion image are extracted through the differential evolution boundary enhancement module to obtain the boundary area of ​​the real lesion image; Based on the loss of the boundary area between the decoded lesion image and the true lesion image, and the loss between the decoded lesion image and the true lesion image, the parameters of the U-shaped image segmentation network are optimized, and the U-shaped image segmentation network after parameter optimization is used for lesion image segmentation.

2. The method according to claim 1, characterized in that The step of extracting boundary key points of the lesion area in the real lesion image by using the differential evolution boundary enhancement module to obtain the boundary area of ​​the real lesion image includes: Initialize the population, where each individual is a set of two-dimensional coordinates representing several key points within the set boundary, which are used to construct a polygon approximating the lesion boundary; Perform mutation operation to generate mutation vector; Combining the original individual with the mutation vector through a crossover operation to generate a test vector; According to the intersection-and-union ratio between the polygon formed by the test vector and the true mask image, the fitness is evaluated, and the current vector is replaced by the test vector; Under the constraint of the maximum number of iterations or convergence conditions, the individual with the highest fitness is obtained as the boundary key point set of the real lesion image, and the boundary area of ​​the real lesion image is obtained.

3. The method according to claim 2, characterized in that Optimizing the parameters of the U-shaped image segmentation network based on the loss between the decoded lesion image and the true lesion image, and the loss of the boundary area between the decoded lesion image and the true lesion image, comprises: Based on the deep supervision mechanism, supervision is imposed on mask maps and boundary maps generated at different scales; Minimizing the difference between the decoded lesion image and the boundary area of ​​the true lesion image based on the intersection-over-union loss; Minimizing the difference between the decoded lesion image and the true lesion image based on cross entropy loss and Dice loss; Based on the difference, the parameters of the U-shaped image segmentation network are optimized.

4. The method according to any one of claims 1 to 3, characterized in that The encoder is used to encode the lesion image to obtain encoding features, including: embedding a channel-attention hybrid separable convolution module in a first predetermined stage of the encoder to jointly extract a first feature of the lesion image through a depthwise separable convolution and a channel-attention mechanism; In the second predetermined stage of the encoder, a grouped Hadamard product channel shuffling module is respectively embedded to enhance the first feature by grouped Hadamard product and channel shuffling operations to obtain a second feature; Obtaining a coding feature according to the first feature and the second feature; Preferably, the number of encoder channels is {8, 12, 16, 32, 48, 64} in sequence.

5. The method according to claim 4, characterized in that The embedding of a channel-attention hybrid separable convolution module in the first predetermined stage of the encoder to jointly extract the first feature of the lesion image through depthwise separable convolution and channel attention mechanism includes: Channel expansion through 1×1 point convolution; 3×3 depth-wise separable convolution is used for feature extraction; Use channel attention for weighted operation; Activation functions are used to improve nonlinear expression capabilities.

6. The method according to claim 5, characterized in that The method further comprises: embedding a grouped Hadamard product channel shuffling module in the second predetermined stage of the encoder, performing enhancement processing on the first feature by grouped Hadamard product and channel shuffling operations to obtain a second feature, including: Dividing the first feature into multiple groups along the channel dimension; Apply Hadamard product and convolution operations to each set of features to extract local features of different dimensions; The features of each group are concatenated and channel shuffled to enhance the flow of information across groups.

7. A lesion image segmentation method, characterized in that: include: Acquire the lesion image to be segmented; Inputting the lesion image to be segmented into a U-shaped image segmentation network to obtain a lesion image segmentation result; Wherein, the U-shaped image segmentation network is obtained by training using the method described in any one of claims 1-6.

8. A model training device, characterized in that: include: A network construction module is used to construct a U-shaped image segmentation network, wherein the U-shaped image segmentation network includes an encoder and a decoder; wherein the encoder is used to encode the lesion image to obtain encoded features; and the decoder is used to decode the encoded features to obtain a decoded lesion image; An image boundary region determination module is used to extract boundary key points of the lesion region in the real lesion image through a differential evolution boundary enhancement module to obtain the boundary region of the real lesion image; A training module is used to optimize the parameters of the U-shaped image segmentation network based on the loss of the boundary area between the decoded lesion image and the true lesion image, and the loss between the decoded lesion image and the true lesion image. The U-shaped image segmentation network after parameter optimization is used for lesion image segmentation.

9. A lesion image segmentation device, characterized in that: include: A lesion image acquisition module is used to acquire the lesion image to be segmented; A prediction module is used to input the lesion image to be segmented into a U-shaped image segmentation network to obtain a lesion image segmentation result; Wherein, the U-shaped image segmentation network is obtained by training using the method described in any one of claims 1-6.

10. An electronic device, characterized in that: include: processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the model training method according to any one of claims 1 to 6 or the lesion image segmentation method according to claim 7.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions or programs, which, when executed on a computer, enable the model training method according to any one of claims 1 to 6 or the lesion image segmentation method according to claim 7 to be executed.

12. A computer program product, characterized in that The computer program product includes computer instructions, and when part or all of the computer instructions are run on a computer, the model training method according to any one of claims 1 to 6 or the lesion image segmentation method according to claim 7 is executed.

Images