Training method of lesion detection model and lesion detection method, device, equipment and storage medium

By adjusting the parameters of the lesion detection model, using the output results of the decoder layer and the prediction feedforward network, the problem of ignoring the output of the decoder layer in the detection transformer model training is solved, and the accuracy of the lesion prediction of the model is improved.

CN118505683BActive Publication Date: 2024-11-29GUANGZHOU SIDE MEDICAL TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410949037.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-29
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

During the training process of detecting transformer model, the query output of each transformer layer of the decoder is ignored, affecting the accuracy of the model output results.

Method used

The encoder of the first lesion detection model obtains the feature vector of the lesion training image and inputs it into the decoder to obtain the output results of each decoder layer. Based on the output results and the predicted feedforward network, the model parameters are adjusted to improve the input data and output results of the decoder layer.

Benefits of technology

By selectively recovering the output results of the decoder layer in front of the decoder, the input data of the current decoder layer is improved, thereby improving the accuracy of the prediction results of the lesion detection model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118505683B_ABST
    Figure CN118505683B_ABST
Patent Text Reader

Abstract

The present application relates to the field of image processing technology, and provides a training method for a lesion detection model, and a lesion detection method, device, equipment, and storage medium. The method comprises: obtaining a feature vector of a lesion training image through an encoder of a first lesion detection model and inputting it into a decoder of the first lesion detection model to obtain output results of each decoder layer; obtaining several detection results of each decoder layer according to the output results and a predictive feedforward network; when the decoder layer belongs to the third layer or is greater than the third layer, the input data is: the splicing result between the target detection results in several detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer; adjusting the parameters of the first lesion detection model according to the difference between several detection results of each decoder layer and the label data to obtain a second lesion detection model. The use of this method improves the accuracy of the lesion prediction results of the second lesion detection model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for a lesion detection model, a training device for a lesion detection model, a lesion detection method for endoscopic images, a lesion detection device for endoscopic images, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the gradual digitization of imaging equipment and the development of computer communication technology, digital image transmission and electronic film have emerged. Many traditional hospitals have transformed into digital hospitals, increasing the amount of medical imaging data that doctors need to review. Therefore, it is necessary to reduce the workload of doctors through intelligent diagnosis and assistance of medical imaging.

[0003] At present, lesion detection in medical images is mainly based on the detection transformer model.

[0004] However, during the training of the detection transformer model, all decoding layers roughly use a unified Hungarian matching and loss function, ignoring the query output by each transformer layer of the decoder, which affects the accuracy of the output results of the detection transformer model. Summary of the invention

[0005] Based on this, it is necessary to provide a training method for a lesion detection model, a training device for a lesion detection model, a lesion detection method for endoscopic images, a lesion detection device for endoscopic images, a computer device, a computer-readable storage medium and a computer program product to address the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a method for training a lesion detection model, comprising:

[0007] Obtaining a feature vector of a lesion training image through an encoder of a first lesion detection model;

[0008] Inputting the feature vector into a decoder of the first lesion detection model to obtain output results of each decoder layer;

[0009] According to the output results of each decoder layer and the prediction feedforward network, several detection results of each decoder layer are obtained; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results in several detection results of the first two decoder layers of the decoder layer and the output result of the first decoder layer of the decoder layer;

[0010] According to the differences between the detection results of each decoder layer and the label data, the parameters of the first lesion detection model are adjusted to obtain the second lesion detection model.

[0011] In one embodiment, the method further comprises:

[0012] Comparing a number of detection results of the first two decoder layers of the decoder layer with the label data to obtain a number of error results corresponding to the number of detection results;

[0013] The error result that is smaller than the error threshold is taken as the target error result;

[0014] A target detection result is obtained according to the detection result corresponding to the target error result.

[0015] In one embodiment, obtaining a target detection result according to a detection result corresponding to the target error result includes:

[0016] Obtaining confidence levels corresponding to the plurality of detection results;

[0017] The confidence level greater than the confidence threshold is taken as the target confidence level;

[0018] When the error result of the detection result is the target error result and the confidence is the target confidence, the detection result is used as the target detection result.

[0019] In one embodiment, the first lesion detection model includes a training main branch and a training auxiliary branch, the decoder and the prediction feedforward network are present on the training main branch, and the auxiliary decoder and the prediction feedforward network are present on the training auxiliary branch;

[0020] The auxiliary decoder layer of the auxiliary decoder is obtained by removing the multi-head self-attention module from the decoder layer of the decoder.

[0021] In one embodiment, obtaining the feature vector of the lesion training image through the encoder of the first lesion detection model includes:

[0022] Obtaining a multi-layer feature map of the lesion training image through a swing transformer module of a first lesion detection model;

[0023] Obtaining a splicing vector of the lesion training image through the multi-layer feature map and a feature mapping module of the first lesion detection model;

[0024] A feature vector of a lesion training image is obtained according to the splicing vector and the encoder of the first lesion detection model.

[0025] In a second aspect, the present application also provides a method for detecting lesions in an endoscopic image, comprising:

[0026] Acquiring an endoscopic image to be inspected;

[0027] The endoscopic image is input into a second lesion detection model to obtain a lesion prediction result of the endoscopic image; the second lesion detection model is a lesion detection model obtained according to the above-mentioned lesion detection model training method embodiment.

[0028] In a third aspect, the present application also provides a training device for a lesion detection model, comprising:

[0029] A feature vector acquisition module, used to obtain a feature vector of a lesion training image through an encoder of a first lesion detection model;

[0030] an output result acquisition module, used for inputting the feature vector into the decoder of the first lesion detection model to obtain the output results of each decoder layer; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results in the detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer;

[0031] A detection result acquisition module is used to obtain the detection results of each decoder layer according to the output results of each decoder layer and the prediction feedforward network;

[0032] The model acquisition module is used to adjust the parameters of the first lesion detection model according to the difference between the detection results of each decoder layer and the label data to obtain the second lesion detection model.

[0033] In a fourth aspect, the present application also provides a lesion detection device for endoscopic images, comprising:

[0034] An endoscopic image acquisition module, used for acquiring an endoscopic image to be detected;

[0035] A lesion prediction result acquisition module is used to input the endoscopic image into a second lesion detection model to obtain a lesion prediction result of the endoscopic image; the second lesion detection model is a lesion detection model obtained according to the above-mentioned lesion detection model training method embodiment.

[0036] In a fifth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.

[0037] In a sixth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the above method.

[0038] In a seventh aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to execute the above method.

[0039] The present application obtains a feature vector of a lesion training image through an encoder of a first lesion detection model; inputs the feature vector into a decoder of the first lesion detection model to obtain output results of each decoder layer; obtains several detection results of each decoder layer according to the output results of each decoder layer and a prediction feedforward network; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the concatenation result between the target detection results in several detection results of the first two decoder layers of the decoder layer and the output results of the previous decoder layer of the decoder layer; according to the difference between several detection results of each decoder layer and the label data, adjust the parameters of the first lesion detection model to obtain a second lesion detection model. The present application obtains input data of the current decoder layer by selectively recycling the output results of the decoder layer before the decoder of the first lesion detection model, thereby improving the output results of the current decoder layer and improving the accuracy of the lesion prediction results of the second lesion detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A diagram of an application environment of a training method for a lesion detection model in one embodiment;

[0042] Figure 2 A schematic diagram of a flow chart of a method for training a lesion detection model in one embodiment;

[0043] Figure 3 A schematic diagram of selectively recycling the output result of a decoder layer preceding a decoder in one embodiment;

[0044] Figure 4 A schematic diagram of the structure of a decoder and an auxiliary decoder in one embodiment;

[0045] Figure 5 A schematic diagram of a process of obtaining a supplementary positive query in one embodiment;

[0046] Figure 6 A flowchart of a positive query generator in one embodiment;

[0047] Figure 7 A diagram showing an application environment of a method for detecting lesions in an endoscopic image in an embodiment;

[0048] Figure 8 is a schematic flow chart of a method for detecting lesions in endoscopic images in one embodiment;

[0049] Fig. 9 is a flow chart of a training method for a lesion detection model in another embodiment;

[0050] Fig.10 is a schematic structural diagram of a second lesion detection method in one embodiment;

[0051] Fig.11 is a structural block diagram of a training device for a lesion detection model in one embodiment;

[0052] Fig.12 is a structural block diagram of a lesion detection device for endoscopic images in one embodiment;

[0053] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] The present application embodiment provides a method for training a lesion detection model, which can be executed by a computer device, such as Figure 1 As shown, the computer device can collect lesion training images to obtain a second lesion detection model. It is understandable that the computer device can be implemented by a server, a terminal, or an interactive system between a terminal and a server. In this embodiment, the method includes Figure 2 The steps shown are:

[0056] Step S201, obtaining a feature vector of a lesion training image through an encoder of a first lesion detection model.

[0057] The structure of the first lesion detection model may be determined with reference to a detection transformer (DETR) model. The first lesion detection model may include an encoder, a decoder, and a prediction feedforward network.

[0058] The encoder can consist of 6 encoder layers, each of which has a standard architecture consisting of a multi-head self-attention module and a feed-forward network (FFN).

[0059] Images with lesions can be collected, and the lesion areas of the images with lesions can be framed and labeled with corresponding lesion category labels, and the labeled images with lesions can be used as lesion training images.

[0060] The lesion training image is input into the encoder of the first lesion detection model to obtain an encoding result of the lesion training image, and the encoding result of the lesion training image is used as a feature vector of the lesion training image.

[0061] Step S202, input the feature vector into the decoder of the first lesion detection model to obtain the output results of each decoder layer.

[0062] Step S203, obtaining several detection results of each decoder layer according to the output results of each decoder layer and the prediction feedforward network; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results in several detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer.

[0063] The decoder can be composed of 6 decoder layers, each of which consists of a multi-head self-attention module, a multi-head cross-attention module and a feed-forward network. The prediction feed-forward network consists of a classification branch and a regression branch, where the classification branch and the regression branch each consist of a linear layer.

[0064] Among them, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results in several detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer, and the corresponding processing mechanism is called a query regression mechanism.

[0065] If the decoder includes N decoder layers, such as Figure 3 As shown, when the decoder layer belongs to the i+2th layer (1≤i≤N-2), the input data of the i+2th decoder layer is: the concatenation result between the target detection results among several detection results of the i-th decoder layer and the output result of the i+1th decoder layer; the target detection result is the detection result close to the label data.

[0066] The decoder includes 6 decoder layers as an example: after the feature vector is input into the decoder of the first lesion detection model, the feature vector is processed by the first decoder layer of the decoder to obtain the output result of the first decoder layer; the output result of the first decoder layer is used as the input data of the second decoder layer, and the input data of the second decoder layer is processed by the second decoder layer to obtain the output result of the second decoder layer; the output result of the first decoder layer is input into the prediction feedforward network to obtain several detection results of the first decoder layer, the target detection result is determined based on the several detection results of the first decoder layer, and the output result of the second decoder layer is spliced, and the spliced ​​result is used as the input data of the third decoder layer, and the input data of the third decoder layer is processed by the third decoder layer of the decoder to obtain the output result of the third decoder layer; the output result of the second decoder layer is input into the prediction feedforward network to obtain several detection results of the second decoder layer, and the target detection result is determined based on the several detection results of the second decoder layer, and the output result of the third decoder layer is spliced, and the spliced ​​result is used as the input data of the fourth decoder layer, and the fourth decoder layer of the decoder is used. The input data of the fourth decoder layer is processed to obtain the output result of the fourth decoder layer; the output result of the third decoder layer is input into the prediction feedforward network to obtain several detection results of the third decoder layer, the target detection result is determined based on the several detection results of the third decoder layer, and the result is spliced ​​with the output result of the fourth decoder layer, and the spliced ​​result is used as the input data of the fifth decoder layer, and the input data of the fifth decoder layer is processed by the fifth decoder layer of the decoder to obtain the output result of the fifth decoder layer; the output result of the fourth decoder layer is input into the prediction feedforward network to obtain several detection results of the fourth decoder layer, the target detection result is determined based on the several detection results of the fourth decoder layer, and the result is spliced ​​with the output result of the fifth decoder layer, and the spliced ​​result is used as the input data of the sixth decoder layer, and the input data of the sixth decoder layer is processed by the sixth decoder layer of the decoder to obtain the output result of the sixth decoder layer; the output result of the fifth decoder layer is input into the prediction feedforward network to obtain several detection results of the fifth decoder layer, and the output result of the sixth decoder layer is input into the prediction feedforward network to obtain several detection results of the sixth decoder layer.

[0067] Among them, the detection results may include a lesion detection frame and a lesion category corresponding to the lesion detection frame, wherein each lesion detection frame can be determined by four coordinate parameters, namely, the horizontal coordinate of the upper left corner vertex of the lesion detection frame, the vertical coordinate of the upper left corner vertex of the lesion detection frame, the height of the lesion detection frame and the width of the lesion detection frame. The lesion categories may include polyps, ulcers, erosions and xanthomas.

[0068] Step S204, adjusting the parameters of the first lesion detection model according to the differences between the detection results of each decoder layer and the label data to obtain a second lesion detection model.

[0069] When obtaining a lesion training image, that is, when the lesion area of ​​the image with the lesion is framed and the corresponding lesion category label is marked, the image lesion area of ​​the lesion can be obtained and framed to obtain a real lesion detection frame. Each real lesion detection frame carries 4 coordinate parameters, namely the horizontal coordinate of the upper left corner vertex of the real lesion detection frame, the vertical coordinate of the upper left corner vertex of the real lesion detection frame, the height of the real lesion detection frame and the width of the real lesion detection frame; and the marked lesion category label is used as the real lesion category corresponding to the real lesion detection frame, and the real lesion detection frame and the real lesion category corresponding to the real lesion detection frame are used as label data.

[0070] Through Hungarian matching, the lesion detection frame output by each decoder layer is matched with the real lesion detection frame. The loss value is obtained based on the difference between the lesion detection frame output by each decoder layer and the real lesion detection frame category, as well as the difference between the lesion category corresponding to the lesion detection frame of each decoder layer and the real lesion category corresponding to the real lesion detection frame. According to the loss value, the parameters of the first lesion detection model are adjusted until the loss value is less than the loss threshold. According to the first lesion detection model at this time, the second lesion detection model is obtained.

[0071] You can first use a public endoscopic lesion detection dataset, such as HyperKvasir, to pre-train the model to obtain a first lesion detection model, and then use your own endoscopic lesion detection dataset to train the first lesion detection model to obtain a second lesion detection model.

[0072] In the training method of the above-mentioned lesion detection model, the output results of the decoder layer before the decoder of the first lesion detection model are selectively recovered to obtain the input data of the current decoder layer, thereby improving the output results of the current decoder layer and improving the accuracy of the lesion prediction results of the second lesion detection model.

[0073] In one of the embodiments, the method provided by the present application also includes: comparing several detection results of the first two decoder layers of the decoder layer with the label data to obtain several error results corresponding to the several detection results; taking the error results less than the error threshold as the target error results; and obtaining the target detection results based on the detection results corresponding to the target error results.

[0074] Each test result is compared with the corresponding label data to obtain an error result corresponding to each test result. The test result may include a lesion detection frame and a lesion category corresponding to the lesion detection frame, and the label data may include a real lesion detection frame and a real lesion category corresponding to the real lesion detection frame. Each lesion detection frame is compared with the corresponding real lesion detection frame category, and the lesion category corresponding to each lesion detection frame is compared with the real lesion category corresponding to the corresponding real lesion detection frame to obtain a corresponding error result.

[0075] The error threshold can be set according to the actual situation. The error result less than the error threshold is used as the target error result; and the target detection result is obtained according to the detection result corresponding to the target error result.

[0076] In this embodiment, several detection results of the first two decoder layers of the decoder layer are compared with the label data to obtain error results, and screening is performed based on the error results to determine target detection results with higher accuracy.

[0077] In one of the embodiments, a target detection result is obtained according to a detection result corresponding to a target error result, and the specific steps are as follows: obtaining confidences corresponding to a number of detection results; taking a confidence greater than a confidence threshold as a target confidence; and taking the detection result as a target detection result when the error result of the detection result is a target error result and the confidence is a target confidence.

[0078] When the prediction feedforward network outputs the test results, it will output the confidence level corresponding to each test result, which can be called a score. The confidence level corresponding to the test result is a measure of the accuracy of the test result. The confidence level can be set between 0 and 1. The closer to 1, the higher the accuracy of the test result, and the closer to 0, the lower the accuracy of the test result.

[0079] The confidence threshold can be set according to the actual situation. The confidence greater than the confidence threshold is used as the target confidence. When the error result of the detection result is the target error result and the confidence is the target confidence, the accuracy of the detection result is high, and the detection result is used as the target detection result.

[0080] In this embodiment, screening is performed based on the error results and confidence levels to determine target detection results with higher accuracy.

[0081] In one embodiment, the first lesion detection model includes a training main branch and a training auxiliary branch, the training main branch has a decoder and a prediction feedforward network, the training auxiliary branch has an auxiliary decoder and a prediction feedforward network; the auxiliary decoder layer of the auxiliary decoder is obtained by removing the multi-head self-attention module from the decoder layer of the decoder.

[0082] The first lesion detection model may include a training main branch and a training auxiliary branch. The training main branch has a decoder and a prediction feedforward network, and the training auxiliary branch has an auxiliary decoder and a prediction feedforward network. The auxiliary decoder layer of the auxiliary decoder is obtained by removing the multi-head self-attention module from the decoder layer of the decoder. The structures of the decoder and the auxiliary decoder are as follows: Figure 4 As shown, the decoder can be composed of 6 decoder layers, each of which is composed of a multi-head self-attention module, a multi-head cross-attention module and a feed-forward network; the auxiliary decoder can be composed of 6 auxiliary decoder layers, each of which is composed of a multi-head cross-attention module and a feed-forward network, and the auxiliary decoding and decoder share weight parameters.

[0083] An auxiliary detection head and a positive query generator may be set before training the auxiliary decoder on the auxiliary branch, wherein the positive query generator is composed of two parallel linear layers.

[0084] The training auxiliary branch can be two branches, wherein the auxiliary detection head of the first training auxiliary branch can be a fast region convolutional neural network (Faster Region-based Convolutional Neural Network, Faster RCNN) detection head, including a region proposal network (Region Proposal Networks, RPN) and a target detection network (ROIAlign), and the auxiliary detection head of the second training auxiliary branch can be an adaptive sample training selection (Adaptive TrainingSample Selection, ATSS) detection head.

[0085] For the first training auxiliary branch, the feature vector e of the lesion training image can be re-divided according to the corresponding position to obtain feature maps e1, e2, ..., eJ, and the feature maps e1, e2, ..., eJ are converted back to their original sizes to obtain encoded feature maps F1, F2, ..., FJ. The encoded feature maps are respectively input into the Faster RCNN detection head, and are screened according to the real lesion detection box coordinates and the lesion detection box coordinates output by the Faster RCNN detection head to obtain the lesion detection box set B1, and the feature vector e* corresponding to B1 is obtained. B1 and e* are input into the first positive query generator to generate the first supplementary positive query Q1, as shown in Figure 5 shown.

[0086] The first supplementary positive query Q1 is input into the first auxiliary decoder, and the output result of the auxiliary decoder layer before the first auxiliary decoder is not recovered. Specifically, the first supplementary positive query Q1 is input into the first layer of the auxiliary decoder of the auxiliary decoder, and the output result of the first layer of the auxiliary decoder is used as the input data of the second layer of the auxiliary decoder. Recursively, the output result of the previous layer of the auxiliary decoder is used as the input data of the next layer of the auxiliary decoder to obtain the output results of each auxiliary decoder layer. The output results of each auxiliary decoder layer are input into the prediction feedforward network to obtain several detection results of each auxiliary decoder layer.

[0087] For the second training auxiliary branch, the second forward query Q2 generated by the adaptive sample training selection detection head can be obtained in the same way. The second supplementary positive query Q2 is input into the second auxiliary decoder, and the output result of the auxiliary decoder layer before the second auxiliary decoder is not recovered. Specifically, the second supplementary positive query Q2 is input into the first layer of the auxiliary decoder of the second auxiliary decoder, and the output result of the first layer of the auxiliary decoder is used as the input data of the second layer of the auxiliary decoder. Recursively, the output result of the previous layer of the auxiliary decoder is used as the input data of the next layer of the auxiliary decoder to obtain the output results of each auxiliary decoder layer. The output results of each auxiliary decoder layer are input into the prediction feedforward network to obtain several detection results of each auxiliary decoder layer.

[0088] The workflow of the query builder is as follows Figure 6 As shown, a vector B* consisting of K detection box coordinates is obtained from the auxiliary detection head, and the corresponding encoder outputs a vector e* consisting of K features. B* is encoded by sine and cosine positions and becomes K*2C. After passing through linear layer 1, its size becomes K*C. The size of e* remains unchanged after linear layer 2. The two are added to obtain the supplementary positive query Q*.

[0089] In this embodiment, the auxiliary decoder layer of the auxiliary decoder is obtained by removing the multi-head self-attention module from the decoder layer of the decoder, thereby improving the one-to-many label matching result of the auxiliary decoder and further improving the accuracy of the lesion prediction result of the second lesion detection model.

[0090] In one of the embodiments, a feature vector of a lesion training image is obtained through an encoder of a first lesion detection model, and the specific steps are as follows: a multi-layer feature map of the lesion training image is obtained through a swing transformer module of the first lesion detection model; a splicing vector of the lesion training image is obtained through the multi-layer feature map and a feature mapping module of the first lesion detection model; a feature vector of the lesion training image is obtained based on the splicing vector and the encoder of the first lesion detection model.

[0091] A large number of endoscopic digestive tract images can be collected, with a number of more than 5 million, including images with lesions and images without lesions, and no labeling is required. A self-supervised pre-training method can be used to train a swin transformer module based on a large number of endoscopic images. The swin transformer module can be called a backbone module.

[0092] This embodiment uses a large number of endoscopic digestive tract images with very high correlation to perform self-supervised pre-training on the swing transformer module, so that the swing transformer module has good feature extraction capabilities for lesion training images.

[0093] The lesion training image is input into the swing transformer module of the first lesion detection model to obtain a multi-layer feature map of the lesion training image. The resolution and number of channels of the feature map are different, and the resolutions are H1*W1, H2*W2, H3*W3, ..., and the number of channels are C1, C2, C3, ..., respectively.

[0094] The feature mapping module includes multiple 1*1 convolutional layers, which can adjust the number of channels of the feature map to C, C can be set to 256, and flatten the feature map.

[0095] The multi-layer feature map is input into the feature mapping module of the first lesion detection model. The feature map passes through multiple 1*1 convolutional layers to unify the channels into C. After the feature map is flattened, the size of the feature map changes from C*H1*W1, C*H2*W2, C*H3*W3, ... to C*H1W1, C*H2W2, C*H3W3, .... The two-dimensional vector is represented as z1, z2, z3, ..., zJ, where J is the number of feature maps. The vectors z1, z2, z3, ..., zJ are concatenated to obtain a vector Z. The size of the concatenated vector Z is C*(H1W1+H2W2+H3W3+...).

[0096] The concatenated vector Z is input into the encoder of the first lesion detection model to obtain the feature vector of the lesion training image.

[0097] In this embodiment, according to the swing transformer module, the feature mapping module and the encoder of the first lesion detection model, the features of the lesion training image can be extracted to obtain the feature vector of the lesion training image.

[0098] The present application also provides a method for detecting lesions in an endoscopic image. ... method can be performed by a computer device. Figure 7As shown, the computer device can obtain the endoscopic image to be detected, and then obtain the lesion prediction result of the endoscopic image. It can be understood that the computer device can be implemented by a server, a terminal, or an interactive system between a terminal and a server. In this embodiment, the method includes Figure 8 The steps shown are:

[0099] Step S801, obtaining an endoscopic image to be detected.

[0100] The endoscopic image of the lesion location and lesion category to be detected is called the endoscopic image to be detected.

[0101] Step S802, input the endoscopic image into the second lesion detection model to obtain the lesion prediction result of the endoscopic image; the second lesion detection model is a lesion detection model obtained according to the embodiment of the training method of the above-mentioned lesion detection model.

[0102] The second lesion detection model is a lesion detection model obtained according to the training method of the lesion detection model of the present application, and the second lesion detection model can detect the lesion location and lesion category in the endoscopic image and output the corresponding lesion prediction result. The lesion prediction result includes a predicted lesion detection frame and a predicted lesion category corresponding to the predicted lesion detection frame.

[0103] In this embodiment, the second lesion detection model obtained according to the training method of the lesion detection model of the present application outputs lesion prediction results with higher accuracy. Therefore, by inputting the endoscopic image into the second lesion detection model, a lesion prediction result of the endoscopic image with higher accuracy can be obtained.

[0104] In order to better understand the above method, an application example of the training method of the lesion detection model of the present application is described in detail below. Fig. 9 shown.

[0105] By inputting the lesion training image into the swing transformer module of the first lesion detection model, a multi-layer feature map of the lesion training image can be obtained; by inputting the multi-layer feature map into the feature mapping module of the first lesion detection model, a splicing vector can be obtained; by inputting the splicing vector into the encoder of the first lesion detection model, a feature vector of the lesion training image can be obtained.

[0106] Then it is divided into three training branches, where the main training branch includes an auxiliary decoder and a prediction feedforward network, the first training auxiliary branch includes a first auxiliary detection head, a first positive query generator, a first auxiliary decoder and a prediction feedforward network, and the second training auxiliary branch includes a second auxiliary detection head, a second positive query generator, a second auxiliary decoder and a prediction feedforward network. The auxiliary decoder layers of the first auxiliary decoder and the second auxiliary decoder are obtained by removing the multi-head self-attention module from the decoder layer of the decoder.

[0107] For training the main branch, the feature vector is input into the decoder of the first lesion detection model to obtain the output results of each decoder layer; according to the output results of each decoder layer and the prediction feedforward network, several detection results of each decoder layer are obtained, and the several detection results of each decoder layer are called the main branch detection results; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results in the several detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer.

[0108] For the first training auxiliary branch, a coding feature map can be obtained according to the feature vector, and the coding feature map can be input into the first auxiliary detection head to obtain the first lesion detection frame set, and the feature vector corresponding to the first lesion detection frame set can be obtained. The first lesion detection frame set and the feature vector corresponding to the first lesion detection frame set are input into the first positive query generator to obtain the first supplementary positive query; the first supplementary positive query is input into the first auxiliary decoder, and the output result of the auxiliary decoder layer before the first auxiliary decoder is not recovered. Specifically, the first supplementary positive query is input into the first auxiliary decoder of the first auxiliary decoder, and the output result of the first auxiliary decoder is used as the input data of the second auxiliary decoder. In this way, the output result of the upper auxiliary decoder is used as the input data of the next auxiliary decoder to obtain the output results of each auxiliary decoder layer. The output results of each auxiliary decoder layer are input into the prediction feedforward network to obtain several detection results of each auxiliary decoder layer. The several detection results of each auxiliary decoder layer are called the first auxiliary branch detection results.

[0109] For the second training auxiliary branch, the encoding feature map can also be obtained according to the feature vector, and the encoding feature map can be input into the second auxiliary detection head to obtain the second lesion detection frame set, and the feature vector corresponding to the second lesion detection frame set can be obtained. The second lesion detection frame set and the feature vector corresponding to the lesion detection frame set are input into the second positive query generator to obtain the second supplementary positive query; the second supplementary positive query is input into the auxiliary decoder, and the output result of the auxiliary decoder layer before the auxiliary decoder is not recovered. Specifically, the second supplementary positive query is input into the first auxiliary decoder of the second auxiliary decoder, and the output result of the first auxiliary decoder is used as the input data of the second auxiliary decoder. In this way, the output result of the upper auxiliary decoder is used as the input data of the next auxiliary decoder to obtain the output results of each auxiliary decoder layer. The output results of each auxiliary decoder layer are input into the prediction feedforward network to obtain several detection results of each auxiliary decoder layer. The several detection results of each auxiliary decoder layer are called the second auxiliary branch detection results.

[0110] The detection result may include a lesion detection frame and a lesion category corresponding to the lesion detection frame, the label data may include a real lesion detection frame and a real lesion category corresponding to the real lesion detection frame, and the lesion detection frame may be matched with the real lesion detection frame through Hungarian matching. The total loss function of the entire training process may be obtained based on the difference between the main branch detection result, the first auxiliary branch detection result, the second auxiliary branch detection result and the label data, and the parameters of the first lesion detection model may be adjusted based on the total loss function to obtain the second lesion detection model.

[0111] The specific steps to obtain the total loss function of the entire training process are as follows:

[0112] For the encoder loss function, the loss from the i-th auxiliary detection head is shown in Equation (1).

[0113] (1)

[0114] in, represents the loss of the i-th auxiliary detection head, i can be 1 or 2, is the loss function used for training the i-th auxiliary detection head itself, P is the lesion detection frame coordinates, lesion category and score (confidence) obtained by the auxiliary detection head, and G is the true lesion detection frame and the true lesion category corresponding to the true lesion detection frame.

[0115] The overall encoder loss function is shown in Equation (2).

[0116] (2)

[0117] in, represents the total loss of the encoder, represents the loss of the first auxiliary detection head, Indicates the loss of the second auxiliary detection head.

[0118] For the decoder, the loss function of the lth decoder layer is shown in Equation (3).

[0119] (3)

[0120] in, Represents the loss of the lth decoder layer, which is the weighted sum of regression loss and classification loss. The regression loss is the absolute value error loss (L1 loss) and the generalized intersection-over-union loss (GIOUloss). The classification loss is the quality focal loss (Quality Focal Loss). is the lesion detection frame obtained by the lth decoder layer, the lesion category corresponding to the lesion detection frame, and the score (confidence) corresponding to the lesion detection frame, and G is the true lesion detection frame and the true lesion category corresponding to the true lesion detection frame.

[0121] For the i-th auxiliary decoder, the loss function of the l-th auxiliary decoder layer is shown in Equation (4).

[0122] (4)

[0123] in, represents the loss of the lth auxiliary decoder layer of the i-th auxiliary decoder, i can be 1 or 2, is the lesion detection frame, the lesion category corresponding to the lesion detection frame, and the score (confidence) corresponding to the lesion detection frame obtained by the lth decoder layer of the i-th auxiliary decoder, and G is the true lesion detection frame and the true lesion category corresponding to the true lesion detection frame.

[0124] The total loss function of the entire training process is shown in formula (5).

[0125] (5)

[0126] in represents the total loss of the training process, represents the loss of the l-th decoder layer, represents the loss of the lth auxiliary decoder layer of the i-th auxiliary decoder, Represents the total loss of the encoder, α and β are empirical coefficients, α is 0.1, and β is 1.

[0127] After the first lesion detection model is trained, the following processing can be performed: the training auxiliary branch is removed, and the decoder does not start the query regression mechanism, does not use Hungarian matching, and sets a threshold to filter the predicted lesion detection box and the predicted lesion category corresponding to the predicted lesion detection box. The processed first lesion detection model is used as the second lesion detection model, such as Fig.10 The second lesion detection model can be used in the inference stage to achieve lesion detection.

[0128] In this embodiment, the output results of the decoder layer in front of the decoder of the first lesion detection model are selectively recovered to obtain the input data of the current decoder layer, thereby improving the output results of the current decoder layer and improving the accuracy of the lesion prediction results of the second lesion detection model; and the auxiliary decoder layers of the first auxiliary decoder and the second auxiliary decoder are obtained by removing the multi-head self-attention modules from the decoder layers of the decoders, thereby improving the one-to-many label matching results of the first auxiliary decoder and the second auxiliary decoder, and further improving the accuracy of the lesion prediction results of the second lesion detection model; in addition, two auxiliary detection heads are used to assist in training, improve the problem of too sparse supervision of the encoder, improve the learning efficiency of the cross-attention mechanism in the decoder, greatly enhance the learning of the identification features of the encoder and decoder, and further improve the accuracy of the lesion prediction results of the second lesion detection model.

[0129] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0130] Based on the same inventive concept, the embodiment of the present application also provides a training device for a lesion detection model for implementing the training method for the lesion detection model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more lesion detection models provided below can refer to the limitations of the training method for the lesion detection model above, and will not be repeated here.

[0131] In an exemplary embodiment, Fig.11 As shown, a training device for a lesion detection model is provided, wherein:

[0132] A feature vector acquisition module 1101 is used to obtain a feature vector of a lesion training image through an encoder of a first lesion detection model;

[0133] The output result acquisition module 1102 is used to input the feature vector into the decoder of the first lesion detection model to obtain the output results of each decoder layer; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results in the detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer;

[0134] A detection result acquisition module 1103 is used to obtain the detection results of each decoder layer according to the output results of each decoder layer and the prediction feedforward network;

[0135] The model acquisition module 1104 is used to adjust the parameters of the first lesion detection model according to the difference between the detection results of each decoder layer and the label data to obtain the second lesion detection model.

[0136] In one of the embodiments, the device also includes a target detection result acquisition module, which is used to: compare several detection results of the first two decoder layers of the decoder layer with the label data to obtain several error results corresponding to the several detection results; use the error results less than the error threshold as the target error results; and obtain the target detection results based on the detection results corresponding to the target error results.

[0137] In one of the embodiments, the target detection result acquisition module is also used to: obtain the confidence corresponding to the plurality of detection results; use the confidence greater than the confidence threshold as the target confidence; when the error result of the detection result is the target error result and the confidence is the target confidence, use the detection result as the target detection result.

[0138] In one of the embodiments, the first lesion detection model includes a training main branch and a training auxiliary branch, the decoder and the prediction feedforward network are present on the training main branch, and the auxiliary decoder and the prediction feedforward network are present on the training auxiliary branch; the auxiliary decoder layer of the auxiliary decoder is obtained by removing the multi-head self-attention module from the decoder layer of the decoder.

[0139] In one embodiment, the feature vector acquisition module 1101 is also used to: obtain a multi-layer feature map of the lesion training image through the swing transformer module of the first lesion detection model; obtain a splicing vector of the lesion training image through the multi-layer feature map and the feature mapping module of the first lesion detection model; and obtain a feature vector of the lesion training image based on the splicing vector and the encoder of the first lesion detection model.

[0140] Each module in the training device of the above-mentioned lesion detection model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0141] Based on the same inventive concept, the embodiment of the present application also provides an endoscopic image lesion detection device for implementing the above-mentioned endoscopic image lesion detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more endoscopic image lesion detection device embodiments provided below can refer to the limitations of the endoscopic image lesion detection method above, and will not be repeated here.

[0142] In an exemplary embodiment, Fig.12 As shown, a lesion detection device for endoscopic images is provided, wherein:

[0143] An endoscopic image acquisition module 1201 is used to acquire an endoscopic image to be detected;

[0144] The lesion prediction result acquisition module 1202 is used to input the endoscopic image into a second lesion detection model to obtain a lesion prediction result of the endoscopic image; the second lesion detection model is a lesion detection model obtained according to an embodiment of the above-mentioned lesion detection model training method.

[0145] Each module in the above-mentioned endoscopic image lesion detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. In an exemplary embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as follows: Fig.13As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of a training method for a lesion detection model and a lesion detection method for endoscopic images. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a lesion detection model and a lesion detection method for endoscopic images are implemented.

[0146] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0149] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0151] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a lesion detection model, characterized in that: The method comprises: Obtaining a feature vector of a lesion training image through an encoder of a first lesion detection model; Inputting the feature vector into a decoder of the first lesion detection model to obtain output results of each decoder layer; According to the output results of each decoder layer and the prediction feedforward network, several detection results of each decoder layer are obtained; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results of several detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer; the target detection result is a detection result close to the label data; According to the differences between the detection results of each decoder layer and the label data, the parameters of the first lesion detection model are adjusted to obtain the second lesion detection model.

2. The method according to claim 1, characterized in that The method further comprises: Comparing a number of detection results of the first two decoder layers of the decoder layer with the label data to obtain a number of error results corresponding to the number of detection results; The error result that is smaller than the error threshold is taken as the target error result; A target detection result is obtained according to the detection result corresponding to the target error result.

3. The method according to claim 2, characterized in that The step of obtaining a target detection result according to a detection result corresponding to the target error result includes: Obtaining confidence levels corresponding to the plurality of detection results; The confidence level greater than the confidence threshold is taken as the target confidence level; When the error result of the detection result is the target error result and the confidence is the target confidence, the detection result is used as the target detection result.

4. The method according to claim 1, characterized in that: The first lesion detection model includes a training main branch and a training auxiliary branch, the decoder and the prediction feedforward network are present on the training main branch, and the auxiliary decoder and the prediction feedforward network are present on the training auxiliary branch; The auxiliary decoder layer of the auxiliary decoder is obtained by removing the multi-head self-attention module from the decoder layer of the decoder.

5. The method according to claim 1, characterized in that The step of obtaining a feature vector of a lesion training image through an encoder of the first lesion detection model comprises: Obtaining a multi-layer feature map of the lesion training image through a swing transformer module of a first lesion detection model; Obtaining a splicing vector of the lesion training image through the multi-layer feature map and a feature mapping module of the first lesion detection model; A feature vector of a lesion training image is obtained according to the splicing vector and the encoder of the first lesion detection model.

6. A method for detecting lesions in endoscopic images, characterized in that: The method comprises: Acquiring an endoscopic image to be inspected; The endoscopic image is input into a second lesion detection model to obtain a lesion prediction result of the endoscopic image; the second lesion detection model is a lesion detection model obtained according to the method according to any one of claims 1 to 5.

7. A training device for a lesion detection model, characterized in that: The device comprises: A feature vector acquisition module, used to obtain a feature vector of a lesion training image through an encoder of a first lesion detection model; An output result acquisition module, used for inputting the feature vector into the decoder of the first lesion detection model to obtain output results of each decoder layer; A detection result acquisition module is used to obtain the detection results of each decoder layer according to the output results of each decoder layer and the prediction feedforward network; wherein, when the decoder layer belongs to the third layer or is greater than the third layer, the input data of the decoder layer is: the splicing result between the target detection results of several detection results of the first two decoder layers of the decoder layer and the output result of the previous decoder layer of the decoder layer; the target detection result is a detection result close to the label data; The model acquisition module is used to adjust the parameters of the first lesion detection model according to the difference between the detection results of each decoder layer and the label data to obtain the second lesion detection model.

8. A lesion detection device for endoscopic images, characterized in that: The device comprises: An endoscopic image acquisition module, used for acquiring an endoscopic image to be detected; A lesion prediction result acquisition module is used to input the endoscopic image into a second lesion detection model to obtain a lesion prediction result of the endoscopic image; the second lesion detection model is a lesion detection model obtained according to the method according to any one of claims 1 to 5.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Target detection model training method, target detection method and device

    CN114580548A

  • Target detection method, model training method and related device

    CN118097341A

  • Method and device for image conversion, electronic equipment, and storage medium

    WO2020114087A1

  • Lesion detection model training method, and method for recognizing lesion in image

    WO2023015743A1