Lesion Detection Device, System and Storage Medium

Abstract

The present invention belongs to the technical field of image detection, and discloses a lesion detection device, system and storage medium. The present invention divides a to-be-detected image into grids to obtain a grid image; performs lesion detection on the grid image through a preset lesion detection model to obtain lesion grids and the corresponding lesion types of the lesion grids; and marks the grid image according to the lesion grids and the lesion types. By dividing the to-be-detected image into grids, performing lesion detection on the grid image by using a pre-trained preset lesion detection model, and finally marking the grid image according to the lesion detection result, the manual participation process is reduced, the detection speed is greatly improved, the doctor can be assisted to quickly determine the lesion position, and the doctor's diagnosis efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to a lesion detection device, system and storage medium. Background Art

[0002] Wireless Capsule Endoscopy (WCE) is a wireless miniature camera that can be swallowed by patients and is designed for gastrointestinal examinations. Its purpose is to visualize the small intestinal mucosa to assist in detecting abnormalities in the small intestine.

[0003] Generally, a single wireless capsule endoscopy examination takes more than 8 hours and can take more comprehensive pictures of the entire digestive tract of the patient, which is beneficial to helping doctors obtain a full range of diagnoses and improve the detection rate of digestive tract diseases. However, tens of thousands of pictures are generated for each WCE examination of a case. Analyzing such a large amount of data is undoubtedly very difficult for endoscopy doctors. Even for very experienced experts, it takes at least 2-3 hours, and there may be missed diagnoses and misdiagnoses due to the tiny size of lesion features or subjective factors of doctors, causing patients to miss the best early treatment time.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of the present invention is to provide a lesion detection device, system and storage medium, aiming to solve the technical problem that the prior art cannot quickly locate lesions in the images taken by wireless capsule endoscopy.

[0006] To achieve the above purpose, the present invention provides a lesion detection device, which includes: a processor, a memory, and a lesion detection program stored on the memory and executable on the processor. The lesion detection program is configured to implement the following steps:

[0007] Perform grid division on the image to be detected to obtain a grid image;

[0008] Perform lesion detection on the grid image through a preset lesion detection model to obtain a lesion grid and the lesion type corresponding to the lesion grid;

[0009] Mark the grid image according to the lesion grid and the lesion type.

[0010] Optionally, before the step of performing grid division on the image to be detected to obtain a grid image, it further includes:

[0011] Obtain a preset sample image set and the lesion marking frames corresponding to each preset sample image in the preset sample image set;

[0012] Perform grid division on the preset sample images in the preset sample image set to obtain a grid image set;

[0013] Generate image lesion markers corresponding to each grid sample image in the grid image set based on the lesion bounding box;

[0014] Construct a model sample set according to the image lesion markers and the grid image set;

[0015] Train an initial lesion detection model according to the model sample set to obtain a preset lesion detection model.

[0016] Optionally, the step of generating image lesion markers corresponding to each grid sample image in the grid image set based on the lesion bounding box includes:

[0017] Traverse the grid image set, and use the traversed grid sample image as the target grid image;

[0018] Obtain the lesion bounding box corresponding to the target grid image;

[0019] Determine the target lesion grid in the target grid image according to the lesion bounding box;

[0020] Determine the grid intersection over union corresponding to each target lesion grid according to the lesion bounding box;

[0021] Set corresponding lesion labels for each target lesion grid based on the grid intersection over union;

[0022] Generate image lesion markers corresponding to the target sample image according to the lesion labels and the target lesion grid;

[0023] At the end of the traversal, obtain image lesion markers corresponding to each grid sample image in the grid image set.

[0024] Optionally, the step of training an initial lesion detection model according to the model sample set to obtain a preset lesion detection model includes:

[0025] Traverse the model sample set, and use the traversed model training sample as the target model sample;

[0026] Perform lesion detection on the target model sample through the initial lesion detection model to obtain predicted lesion information;

[0027] Determine the model loss value according to the image lesion markers corresponding to the target model sample and the predicted lesion information;

[0028] Adjust the parameters of the initial lesion detection model according to the model loss value;

[0029] At the end of the traversal, the initial lesion detection model with adjusted parameters is used as the preset lesion detection model.

[0030] Optionally, the step of determining the model loss value according to the image lesion label corresponding to the target model sample and the predicted lesion information includes:

[0031] Determine the true lesion location, true lesion category, and true grid confidence according to the image lesion label;

[0032] Determine the predicted lesion location, predicted lesion category, and predicted grid confidence according to the predicted lesion information;

[0033] Determine the model loss value according to the true lesion location, true lesion category, true grid confidence, predicted lesion location, predicted lesion category, and predicted grid confidence.

[0034] Optionally, the step of determining the model loss value according to the true lesion location, true lesion category, true grid confidence, predicted lesion location, predicted lesion category, and predicted grid confidence includes:

[0035] Calculate the model loss value according to the true lesion location, true lesion category, true grid confidence, predicted lesion location, predicted lesion category, and predicted grid confidence through a preset loss formula;

[0036] The preset loss formula is:

[0037]

[0038] In the formula, λ lcoord is the preset location weight, ω ln oobj is the preset confidence weight for non-target objects, λ is the preset confidence weight for target objects, ω is the preset classification weight, N is the number of grids in each row and each column divided according to the image size, i is the row number of the grid, j is the column number of the grid, x ij is the predicted lesion location, C ij is the predicted grid confidence, p ij (c) is the predicted lesion category, is the true lesion location, is the true grid confidence, is the true lesion category.

[0039] The step of performing lesion detection on the target model sample through the initial lesion detection model to obtain predicted lesion information includes:

[0040] Extract grid sample images from the target model sample;

[0041] The multi-level feature extraction is performed on the grid sample image by the feature extraction module to obtain multi-dimensional image features;

[0042] The lesion detection is performed on the multi-dimensional image features by the feature analysis module to obtain the lesion detection results corresponding to each grid in the grid sample image;

[0043] The lesion detection results corresponding to each grid are aggregated to obtain the predicted lesion information.

[0044] Optionally, the step of marking the grid image according to the lesion grid and the lesion type includes:

[0045] Search for the marked background color corresponding to the lesion grid in the preset type color mapping table according to the lesion type;

[0046] The background color of the grid image is adjusted according to the marked background color and the lesion grid to mark the grid image.

[0047] In addition, to achieve the above object, the present invention also provides a lesion detection system, and the lesion detection system includes:

[0048] An image processing module, configured to perform grid division on the image to be detected to obtain a grid image;

[0049] A lesion detection module, configured to perform lesion detection on the grid image through a preset lesion detection model to obtain a lesion grid and the lesion type corresponding to the lesion grid;

[0050] An image marking module, configured to mark the grid image according to the lesion grid and the lesion type.

[0051] In addition, to achieve the above object, the present invention also provides a computer-readable storage medium, on which a lesion detection program is stored, and when the lesion detection program is executed, the following steps are implemented:

[0052] Perform grid division on the image to be detected to obtain a grid image;

[0053] Perform lesion detection on the grid image through a preset lesion detection model to obtain a lesion grid and the lesion type corresponding to the lesion grid;

[0054] Mark the grid image according to the lesion grid and the lesion type.

[0055] The present invention divides the image to be detected into grids to obtain a grid image, detects lesions in the grid image through a preset lesion detection model to obtain lesion grids and the corresponding lesion types of the lesion grids, and marks the grid image according to the lesion grids and the lesion types. By dividing the image to be detected into grids, using a pre-trained preset lesion detection model to detect lesions in the grid image, and finally marking the grid image according to the lesion detection results, the manual participation process is reduced, the detection speed is greatly improved, the doctor can be assisted to quickly determine the lesion location, and the doctor's diagnosis efficiency is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic structural diagram of an electronic device in the hardware operating environment related to the solution of the embodiment of the present invention;

[0057] Figure 2 is a schematic flowchart of the first embodiment of the lesion detection device of the present invention;

[0058] Figure 3 is a schematic flowchart of the second embodiment of the lesion detection device of the present invention;

[0059] Figure 4 is a schematic diagram of the data processing flow direction of an embodiment of the lesion detection device of the present invention;

[0060] Figure 5 is a structural block diagram of the first embodiment of the lesion detection system of the present invention.

[0061] The implementation, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] Refer to Figure 1 , Figure 1 is a schematic structural diagram of a lesion detection device in the hardware operating environment related to the solution of the embodiment of the present invention.

[0064] As Figure 1As shown in the figure, the electronic device may include: a processor 1001, such as a Central Processing Unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wireless-Fidelity (WI-FI) interface). The memory 1005 may be a high-speed Random Access Memory (RAM) or a stable Non-Volatile Memory (NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0065] Those skilled in the art can understand that Figure 1 the structure shown in does not constitute a limitation on the electronic device, and it may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0066] As Figure 1 shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and a lesion detection program.

[0067] In Figure 1 the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention may be provided in a lesion detection device, and the electronic device calls the lesion detection program stored in the memory 1005 through the processor 1001.

[0068] An embodiment of the present invention provides a lesion detection device. Referring to Figure 2 , Figure 2 is a schematic flowchart of a first embodiment of a lesion detection device of the present invention.

[0069] This embodiment provides a lesion detection device, which includes: a processor, a memory, and a lesion detection program stored on the memory and executable on the processor. The lesion detection program is configured to implement the following steps:

[0070] Step S10: Perform grid division on the image to be detected to obtain a grid image.

[0071] It should be noted that the image to be detected can be an image of a patient's digestive tract collected by a wireless capsule endoscope. Dividing the image to be detected into grids to obtain a grid image can be dividing the image to be detected into N*N grids according to a preset image division rule. Therefore, according to the pixel division, each grid will obtain corresponding position coordinates, and then the divided image to be detected is used as the grid image.

[0072] In actual use, if the size of the image to be detected is 240*240, dividing the image to be detected into N*N grids can be with 24 pixels as the side length of a single grid, and dividing the image to be detected into 10*10 square grids. Of course, the specific side length of the grid can be adjusted according to actual needs. For example, the side length of the grid can be adjusted to 8 pixels or more.

[0073] Step S20: Detect lesions in the grid image through a preset lesion detection model to obtain lesion grids and the corresponding lesion types of the lesion grids.

[0074] It should be noted that the preset lesion detection model can be a neural network model pre-trained for lesion detection. The preset lesion detection model can detect lesions in the input grid image and output grids with and without lesions, as well as the corresponding lesion types of the objects in the grids with objects when the analysis is completed. Among them, the lesion types can include: occupancy, bleeding, ulcer, bubble, normal, impurity, etc.

[0075] Step S30: Mark the grid image according to the lesion grids and the lesion types.

[0076] It should be noted that marking the grid image according to the lesion grids and the lesion types can be marking or setting corresponding labels for the lesion grids in the grid image according to different lesion types. Among them, different lesion types can correspond to different marking methods or different labels. For example: the label corresponding to occupancy is 1, the label corresponding to bleeding is 5, and the label corresponding to ulcer is 6.

[0077] Further, in order to improve the recognizability of the lesion area in the grid image and facilitate the doctor's diagnosis, step S30 in this embodiment may include:

[0078] Search for the marked background color corresponding to the lesion grid in a preset type color mapping table according to the lesion type;

[0079] Adjust the background color of the grid image according to the marked background color and the lesion grid to achieve marking of the grid image.

[0080] It should be noted that the preset type color mapping table can be a data table containing the mapping relationship between the lesion types and the marked background colors. This mapping relationship can be set in advance by the management personnel of the lesion detection device according to actual needs. For example, the mapping relationship is set in advance so that the preset type color mapping table contains the corresponding relationships: bleeding corresponds to blue, bubbles correspond to green, normal corresponds to orange, occupancy corresponds to pink, impurities correspond to purple, and ulcers correspond to yellow. Of course, in actual use, there will inevitably be parts where there are doubts in the identified area. That is, when the model performs lesion detection on the object in the grid, the possibility that the object belongs to multiple lesion types is very similar, and it is impossible to determine the specific lesion type corresponding to the grid. At this time, it can be marked as gray, indicating that the area cannot be determined.

[0081] In actual use, the background color of the grid image is adjusted according to the marked background color and the lesion type in the grid to mark the grid image. It can be to modify the background color of the area corresponding to the lesion grid in the grid image to the marked background color to improve the recognizability of the lesion grid in the grid image.

[0082] In this embodiment, the image to be detected is divided into grids to obtain a grid image; a preset lesion detection model is used to perform lesion detection on the grid image to obtain the lesion grid and the lesion type corresponding to the lesion grid; the grid image is marked according to the lesion grid and the lesion type. By dividing the image to be detected into grids, using a pre-trained preset lesion detection model to perform lesion detection on the grid image, and finally marking the grid image according to the lesion detection result, the manual participation process is reduced, the detection speed is greatly improved, the doctor can be assisted to quickly determine the lesion location, and the doctor's diagnosis efficiency is greatly improved.

[0083] Reference Figure 3 , Figure 3 is a schematic flowchart of the second embodiment of a lesion detection device of the present invention.

[0084] Based on the above first embodiment, before the step S10 of this embodiment of the lesion detection device, it further includes:

[0085] Step S01: Obtain a preset sample image set and the lesion marking frames corresponding to each preset sample image in the preset sample image set.

[0086] It should be noted that the preset sample image set can be a set aggregated by a large number of sample images. The preset sample images can be images of the patient's digestive tract collected by a wireless capsule endoscope and determined to contain lesions. Obtaining the preset sample image set can be to read a preset number of sample images from a database storing sample images and aggregate the sample images to obtain the preset sample image set.

[0087] In actual use, the sample images are all images that have been manually marked, and there is a corresponding lesion marking file for each sample image. Obtaining the lesion marking frames corresponding to the preset sample images in the preset sample image set can be to read the lesion marking files corresponding to the preset sample images in the preset sample image set from the database storing the sample images, read the coordinate data of the marking frames from the lesion marking files, and then generate lesion marking frames in the preset sample images according to the read coordinate data.

[0088] Step S02: Divide the preset sample images in the preset sample image set into grids to obtain a grid image set.

[0089] It should be noted that dividing the preset sample images in the preset sample image set into grids to obtain a grid image set can be to divide the preset sample images in the preset sample image set into grids respectively, divide the preset sample images into N*N grids, so as to obtain multiple grid sample images, and then aggregate all the obtained grid sample images to obtain a grid image set.

[0090] Step S03: Generate image lesion markings corresponding to each grid sample image in the grid image set based on the lesion marking frames.

[0091] It should be noted that the image lesion markings corresponding to the grid sample images can include the lesion presence markings corresponding to each grid in the grid sample images, where the lesion presence markings are used to mark whether there is a lesion object in the grid.

[0092] In a specific implementation, in order to accurately determine whether there is a lesion object in each grid of the grid sample image and ensure the accuracy of the generated image lesion markings, step S03 in this embodiment may include:

[0093] Traverse the grid image set and use the traversed grid sample image as the target grid image;

[0094] Obtain the lesion marking frame included in the target grid image;

[0095] Determine the target lesion grids in the target grid image according to the lesion marking frame;

[0096] Determine the grid intersection over union of each target lesion grid relative to the corresponding lesion marking frame according to the lesion marking frame;

[0097] Set corresponding lesion labels for each target lesion grid based on the grid intersection over union;

[0098] Generate the image lesion markings corresponding to the target sample image according to the lesion labels and the target lesion grids;

[0099] At the end of the traversal, obtain the image lesion markers corresponding to each grid sample image in the grid image set.

[0100] It should be noted that traversing the grid image set can be traversing the grid image set according to the set index from small to large. Of course, other traversal methods can also be used, and this embodiment does not limit this. Determining the target lesion grid in the target grid image according to the lesion marker box can be taking the grid covered by the lesion marker box in the target grid image as the target lesion grid.

[0101] Determining the grid intersection over union corresponding to the target lesion grid according to the lesion marker box can be obtaining the area (P) of the target lesion grid and the area (G) of the lesion marker box, then obtaining the intersection area (P∩G) and the union area (P∪G) of the target lesion grid and the lesion marker box, then calculating the ratio of the intersection area to the union area ((P∩G) / (P∪G)), and taking the obtained ratio as the grid intersection over union corresponding to the target lesion grid.

[0102] In actual use, setting the corresponding lesion label for each target lesion grid based on the grid intersection over union can be comparing the grid intersection over union corresponding to the target lesion grid with a preset intersection over union threshold. If the grid intersection over union is greater than the preset intersection over union threshold, set the lesion label corresponding to the target lesion grid to having a lesion object; if the grid intersection over union is less than or equal to the preset intersection over union threshold, set the lesion label corresponding to the target lesion grid to not having a lesion object. Among them, since setting the threshold too large is likely to lose target information and too small will affect the accuracy of target positioning, the intersection over union threshold can be set according to the proportion of the area of a single grid in the area of the entire image. For example: dividing an image of 240*240 pixel size into 10*10 grids, then its intersection over union threshold can be set to 0.01. Of course, according to the different actual numbers of divided grids, the setting of the intersection over union threshold also varies, and this embodiment does not limit this.

[0103] In specific implementation, generating the image lesion marker corresponding to the target sample image according to the lesion label and the target lesion grid can be aggregating the lesion labels corresponding to each target lesion grid, and taking the aggregated data as the image lesion marker corresponding to the target sample image.

[0104] Step S04: Construct a model sample set according to the image lesion marker and the grid image set.

[0105] It should be noted that constructing a model sample set according to the image lesion marker and the grid image set can be combining each grid sample image in the grid image set with its corresponding image lesion marker into a model training sample, and then aggregating the obtained multiple model training samples into a model sample set.

[0106] Step S05: Train the initial lesion detection model according to the model sample set to obtain a preset lesion detection model.

[0107] It should be noted that training the initial lesion detection model according to the model sample set to obtain a preset lesion detection model can be to input the model sample set into the initial lesion detection model for training, and continuously adjust the parameters of the initial lesion detection model during the training until the model converges, and then use the converged initial lesion detection model as the preset lesion detection model.

[0108] In a specific implementation, the model sample set can be divided into a training set and a test set according to a preset ratio, and then the initial lesion detection model is continuously trained through the training set, and after the training converges, the accuracy of the converged model is detected through the test set. When the accuracy reaches a preset index, the converged model is used as the preset lesion detection model.

[0109] In a specific implementation, in order to reasonably adjust the parameters during the training process and improve the model training efficiency, step S05 in this embodiment may include:

[0110] Traverse the model sample set, and use the traversed model training sample as the target model sample;

[0111] Perform lesion detection on the target model sample through the initial lesion detection model to obtain predicted lesion information;

[0112] Determine the model loss value according to the image lesion label corresponding to the target model sample and the predicted lesion information;

[0113] Adjust the parameters of the initial lesion detection model according to the model loss value;

[0114] At the end of the traversal, use the initial lesion detection model with adjusted parameters as the preset lesion detection model.

[0115] It should be noted that traversing the model sample set can be to traverse the model sample set in ascending order according to the set index. Of course, other traversal methods can also be used, such as: random traversal or reverse traversal. This embodiment does not limit this.

[0116] It should be noted that when using the initial lesion detection model to detect lesions in the target model samples and obtain the predicted lesion information, it can be to input the target model samples into the initial lesion detection model, so that the initial lesion detection model performs lesion detection on them and outputs the corresponding predicted lesion information. Among them, the predicted lesion information can include the lesion type corresponding to each lesion grid and the confidence level corresponding to each lesion grid (the confidence level is used to represent the credibility of the lesion grid marking, and the value range can be [0, 1]) and other information. Adjusting the parameters of the initial lesion detection model according to the model loss value can be to adjust parameters such as the learning rate in the initial lesion detection model, so that the model loss value tends to become smaller.

[0117] Among them, because a dataset containing 6 different lesion categories (occupation, hemorrhage, ulcer, bubble, normal, impurity) may be used when training the model, so these 6 values represent the probability that any one object exists in the grid, which can be denoted as P(C 1 |object), P(C 2 |object),.....P(C 6 |object), indicating that if there is an object object in the grid, then the probability that it is C i is P(C i |object).

[0118] Since the position of the grid is fixed, the confidence level of each grid represents the size of the possibility that the grid contains a certain object, that is, Confidence = Pr(object), where Pr(object) is the probability that an object exists in the grid, and the confidence level ranges from 0 to 1.

[0119] After obtaining the confidence level output of the network, calculate the confidence level scores of each grid containing different lesions, that is, P(C i |object) * Pr(object) = Pr(C i ), this product predicts the specific probability that the grid containing the object belongs to various lesions. When the confidence level score is the largest, it indicates that this category is the category to which the grid object belongs.

[0120] In a specific implementation, the initial lesion detection model may include a feature extraction module and a feature analysis module. At this time, the step of using the initial lesion detection model to detect lesions in the target model samples and obtain the predicted lesion information may include:

[0121] Extract the grid sample image from the target model sample;

[0122] Perform multi-level feature extraction on the grid sample image through the feature extraction module to obtain multi-dimensional image features;

[0123] The lesion detection is performed on the multi-dimensional image features by the feature analysis module to obtain the lesion detection results corresponding to each grid in the grid sample image;

[0124] The lesion detection results corresponding to each grid are aggregated to obtain the predicted lesion information.

[0125] It should be noted that the feature extraction module can be composed of multiple network structures. Multiple network structures cooperate to perform multi-level feature extraction on the input grid sample image and finally output a multi-dimensional image feature map.

[0126] In actual use, the feature extraction module can be composed of ten groups of network structures, and the structures of each group can be as follows:

[0127] The first group can adopt the traditional filling method for the images used for training, that is, scale them to 240*240 to meet the detection network structure. For the images used for detection, when adjusting the image size to 240*240, scale them proportionally while maintaining the aspect ratio of the original image, and fill the remaining part with gray to reduce the amount of calculation and improve the model inference speed; then use the Mosaic data augment technology to splice four images into one image in a random scaling, random cropping, and random arrangement manner as the training data, increasing the number of targets and enriching the background information of the detection objects;

[0128] The second group: The image adjusted to 240*240*3 after the data augmentation operation can be sent into the first convolutional layer conv1 (with a specification of 7*7*64). After passing through the BN layer and the max pooling layer Maxpool, it is downsampled and output as a feature map of 120*120*64;

[0129] The third group: The feature map output by the second group is processed by the convolutional layer conv2 (with a specification of 3*3*192) and the max pooling layer Maxpool, and then output as a feature map of 60*60*192;

[0130] The fourth group and the fifth group are both composed of convolutional layers Conv3 and Con4 stacked by 1*1 and 3*3 and a max pooling layer Maxpool. Channel integration and dimensionality reduction to reduce parameters are achieved by alternating the 3*3 and 1*1 convolutional layers, and feature maps of 30*30*512 and 15*15*1024 are output respectively;

[0131] The sixth group: It is composed of the convolutional layer Conv5, keeping the feature map size unchanged, using stacked 1*1 and 3*3 convolutional kernels to reduce the number of channels, and finally outputting a feature map of 15*15*512.

[0132] The feature analysis module can be composed of four groups of network structures. To avoid confusion with the above-mentioned network structures, here they are referred to as the seventh to tenth groups. The network structures of each group can be as follows:

[0133] The seventh group: The SPP module is adopted to enrich the expression ability of the feature map. After the feature map output by the sixth group is processed by four Maxpool branches, it can be merged into a feature map of 15*15*2048, and then connected to a 1*1 convolutional layer (conv6+BN+LeakyReLu) to compress the channels, and a feature map of 15*15*512 is output;

[0134] The eighth group: The SE module is added. Through the Squeeze operation, global average pooling and compression are performed on multiple feature maps to endow them with a global receptive field, and then the Excitation operation is used to comprehensively capture channel dependencies, thereby enhancing the sensitivity of the model to channel features;

[0135] The ninth group: It consists of a fully connected layer FC1, which can flatten the output of the eighth group into a one-dimensional vector, that is, 1x115200, and then input it into 4096 neurons to output a 1x4096-dimensional vector;

[0136] The tenth group: It consists of a fully connected layer FC2. The 1x4096-dimensional vector output by the ninth group can be output after passing through the LeakyReLu function and the BN layer. After passing through N*N(C+1) neurons to obtain a 1xN*N*(C+1)-dimensional vector, the Sigmoid linear activation function is adopted in the fully connected layer FC2, and it is reshaped into a tensor of N*N*(1+C) (C represents the lesion category) as the final lesion detection result, which is used to represent the grids with and without lesions and the lesion categories corresponding to the grids with lesions.

[0137] For the sake of easy understanding, reference is made to Figure 4 for illustration, but it does not limit the present solution. Figure 4 This is the schematic diagram of the data processing flow of this embodiment. Among them, according to the flow in Figure 4 , after the data is processed by the feature extraction module and the feature analysis module at multiple levels, a tensor of N*N*(1+C) can be finally obtained. According to this tensor, the grids with and without lesions and the lesion categories corresponding to the grids with lesions can be determined. Finally, based on the lesion categories corresponding to each lesion grid, the lesion grids in the grid image are marked, and the grid image marked with lesions can be obtained, and then it is output, so that experts can quickly determine the lesion location and lesion category for medical diagnosis.

[0138] It should be noted that the use of BN layer, Leaky ReLu non-linear activation function, and data augmentation (such as flipping, rotation, translation, etc.) operations can accelerate the convergence of the model and prevent overfitting. The BN algorithm (Batch Normalization) has the characteristic of improving the generalization ability of the network and can solve the problem of the change of the data distribution in the middle layer during training.

[0139] In the specific implementation, after obtaining the predicted lesion information, it is necessary to determine the difference between the predicted result and the actual result. At this time, the loss value can be calculated. When calculating the position loss of the lesion site, the position loss can be expressed as:

[0140]

[0141] In the formula, represents the true position of the lesion in the grid image, x ij represents the predicted lesion position in the grid image, N is the number of grids in each row and each column divided according to the image size, i is the row number of the grid, and j is the column number of the grid.

[0142] Among them, the network model predicts the lesion position according to the learned training data, and the predicted result of the network is displayed in the output image. At this time, there are generally the following three situations between the true value image and the predicted output image:

[0143] 1. It means that there is a missed detection in the predicted result;

[0144] 2. It means that the predicted result is consistent with the true value;

[0145] 3. It means that there is a false detection in the predicted result.

[0146] In the actual calculation of the loss, the loss can be composed of three parts: position loss, category loss, and confidence loss. At this time, in order to ensure the reasonable calculation of the model loss value, the steps of determining the model loss value according to the image lesion label corresponding to the target model sample and the predicted lesion information in this embodiment may include:

[0147] Determine the true lesion position, true lesion category, and true grid confidence according to the image lesion label;

[0148] Determine the predicted lesion position, predicted lesion category, and predicted grid confidence according to the predicted lesion information;

[0149] Determine the model loss value according to the true lesion position, the true lesion category, the true grid confidence, the predicted lesion position, the predicted lesion category, and the predicted grid confidence.

[0150] It should be noted that the real lesion position can be the position coordinates of each lesion grid included in the image lesion annotation, the real lesion category can be the lesion category corresponding to each lesion grid included in the image lesion annotation, and the real grid confidence can be the confidence corresponding to each lesion grid included in the image lesion annotation. The predicted lesion position can be the position coordinates of each lesion grid included in the predicted lesion information, the predicted lesion category can be the lesion category corresponding to each lesion grid included in the predicted lesion information, and the predicted grid confidence can be the confidence corresponding to each lesion grid included in the predicted lesion information.

[0151] In actual use, the model loss value can be calculated according to the real lesion position, the real lesion category, the real grid confidence, the predicted lesion position, the predicted lesion category, and the predicted grid confidence through a preset loss formula. The preset loss formula can be:

[0152]

[0153] In the formula, λ lcoord is the preset position weight, ω ln oobj is the preset confidence weight for grids without target objects, λ is the preset confidence weight for grids with target objects, ω is the preset classification weight, N is the number of grids in each row and each column divided according to the image size, i is the row number of the grid, j is the column number of the grid, x ij is the predicted lesion position, C ij is the predicted grid confidence, p ij (c) is the predicted lesion category, is the real lesion position, is the real grid confidence, is the real lesion category.

[0154] In this embodiment, a preset sample image set and the lesion annotation boxes corresponding to each preset sample image in the preset sample image set are obtained; the preset sample images in the preset sample image set are divided into grids to obtain a grid image set; image lesion annotations corresponding to each grid sample image in the grid image set are generated based on the lesion annotation boxes; a model sample set is constructed according to the image lesion annotations and the grid image set; and an initial lesion detection model is trained according to the model sample set to obtain a preset lesion detection model. Since data will be collected in advance to train the initial lesion detection model, it can be ensured that the model has been trained before lesion detection, providing an implementation basis for rapid lesion detection. Moreover, by determining the image lesion annotations corresponding to each grid sample image according to the lesion annotation boxes, the operation of screening candidate boxes in the traditional object detection process is avoided, and the feature information of the lesion-containing grids learned in the network can be directly mapped to the image, effectively improving the detection efficiency of lesions in capsule endoscopy images.

[0155] In addition, an embodiment of the present invention further provides a storage medium, on which a lesion detection program is stored. When the lesion detection program is executed by a processor, the following steps are implemented:

[0156] Perform grid division on the image to be detected to obtain a grid image;

[0157] Perform lesion detection on the grid image through a preset lesion detection model to obtain a lesion grid and the corresponding lesion type of the lesion grid;

[0158] Mark the grid image according to the lesion grid and the lesion type.

[0159] Refer to Figure 5 , Figure 5 which is the structural block diagram of the first embodiment of the lesion detection system of the present invention.

[0160] As Figure 5 shown, the lesion detection system proposed by the embodiment of the present invention includes:

[0161] An image processing module 10, configured to perform grid division on the image to be detected to obtain a grid image;

[0162] A lesion detection module 20, configured to perform lesion detection on the grid image through a preset lesion detection model to obtain a lesion grid and the corresponding lesion type of the lesion grid;

[0163] An image marking module 30, configured to mark the grid image according to the lesion grid and the lesion type.

[0164] In this embodiment, grid division is performed on the image to be detected to obtain a grid image; lesion detection is performed on the grid image through a preset lesion detection model to obtain a lesion grid and the corresponding lesion type of the lesion grid; the grid image is marked according to the lesion grid and the lesion type. By performing grid division on the image to be detected, using a pre-trained preset lesion detection model to perform lesion detection on the grid image, and finally marking the grid image according to the lesion detection result, the manual participation process is reduced, the detection speed is greatly improved, the doctor can be assisted to quickly determine the lesion position, and the doctor's diagnosis efficiency is greatly improved.

[0165] Further, the image processing module 10 is further configured to obtain a preset sample image set and lesion bounding boxes corresponding to each preset sample image in the preset sample image set; perform grid division on the preset sample images in the preset sample image set to obtain a grid image set; generate image lesion markers corresponding to each grid sample image in the grid image set based on the lesion bounding boxes; construct a model sample set according to the image lesion markers and the grid image set; and train an initial lesion detection model according to the model sample set to obtain a preset lesion detection model.

[0166] Further, the image processing module 10 is further configured to traverse the grid image set, and use the traversed grid sample image as a target grid image; obtain the lesion bounding box corresponding to the target grid image; determine a target lesion grid in the target grid image according to the lesion bounding box; determine the grid intersection over union corresponding to each target lesion grid according to the lesion bounding box; set corresponding lesion labels for each target lesion grid based on the grid intersection over union; generate an image lesion marker corresponding to the target sample image according to the lesion label and the target lesion grid; and obtain image lesion markers corresponding to each grid sample image in the grid image set when the traversal ends.

[0167] Further, the image processing module 10 is further configured to traverse the model sample set, and use the traversed model training sample as a target model sample; perform lesion detection on the target model sample through the initial lesion detection model to obtain predicted lesion information; determine a model loss value according to the image lesion marker corresponding to the target model sample and the predicted lesion information; adjust the parameters of the initial lesion detection model according to the model loss value; and use the initial lesion detection model with adjusted parameters as the preset lesion detection model when the traversal ends.

[0168] Further, the image processing module 10 is further configured to determine the real lesion position, real lesion category, and real grid confidence according to the image lesion marker; determine the predicted lesion position, predicted lesion category, and predicted grid confidence according to the predicted lesion information; and determine a model loss value according to the real lesion position, the real lesion category, the real grid confidence, the predicted lesion position, the predicted lesion category, and the predicted grid confidence.

[0169] Further, the image processing module 10 is further configured to calculate a model loss value according to the real lesion position, the real lesion category, the real grid confidence, the predicted lesion position, the predicted lesion category, and the predicted grid confidence through a preset loss formula;

[0170] The preset loss formula is:

[0171]

[0172] Wherein, λ lcoord is the preset position weight, ω ln oobj is the confidence weight of the preset object without the target object, λ is the confidence weight of the preset object with the target object, ω is the preset classification weight, N is the number of grids in each row and each column divided according to the image size, i is the row number of the grid, j is the column number of the grid, x ij is the predicted lesion position, C ij is the predicted grid confidence, p ij (c) is the predicted lesion category, is the true lesion position, is the true grid confidence, is the true lesion category.

[0173] Furthermore, the initial lesion detection model includes a feature extraction module and a feature analysis module;

[0174] The image processing module 10 is further configured to extract a grid sample image from the target model sample; perform multi-level feature extraction on the grid sample image through the feature extraction module to obtain multi-dimensional image features; perform lesion detection on the multi-dimensional image features through the feature analysis module to obtain the lesion detection results corresponding to each grid in the grid sample image; aggregate the lesion detection results corresponding to each grid to obtain predicted lesion information.

[0175] Furthermore, the image marking module 30 is further configured to find the marked background color corresponding to the lesion grid in the preset type color mapping table according to the lesion type; adjust the background color of the grid image according to the marked background color and the lesion grid to mark the grid image.

[0176] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present invention. In specific applications, those skilled in the art can set according to needs, and the present invention does not limit this.

[0177] It should be noted that the above-described work process is only illustrative and does not limit the protection scope of the present invention. In actual applications, those skilled in the art can select some or all of them according to actual needs to achieve the purpose of the solution of this embodiment, and there is no limitation here.

[0178] In addition, it should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article or system comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or system. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or system comprising that element.

[0179] The serial numbers of the above embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0180] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a Read Only Memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0181] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A lesion detection device, characterized in that, the lesion detection device includes: a processor, a memory, and a lesion detection program stored on the memory and executable on the processor, and the lesion detection program is configured to implement the following steps: Perform grid division on the image to be detected to obtain a grid image; Perform lesion detection on the grid image through a preset lesion detection model to obtain lesion grids and the corresponding lesion types of the lesion grids; Mark the grid image according to the lesion grids and the lesion types; Wherein, before the step of performing grid division on the image to be detected to obtain a grid image, it further includes: Obtain a preset sample image set and the corresponding lesion marking frames of each preset sample image in the preset sample image set; Perform grid division on the preset sample images in the preset sample image set to obtain a grid image set; Traverse the grid image set, and use the traversed grid sample image as the target grid image; Obtain the lesion marking frame corresponding to the target grid image; Determine the target lesion grids in the target grid image according to the lesion marking frame; Determine the grid intersection over union corresponding to each target lesion grid according to the lesion marking frame; Set corresponding lesion labels for each target lesion grid based on the grid intersection over union; Generate an image lesion marking corresponding to the target sample image according to the lesion label and the target lesion grid; At the end of the traversal, obtain the image lesion markings corresponding to each grid sample image in the grid image set; Construct a model sample set according to the image lesion markings and the grid image set; Train an initial lesion detection model according to the model sample set to obtain a preset lesion detection model; Wherein, the step of training the initial lesion detection model according to the model sample set to obtain a preset lesion detection model includes: Traverse the model sample set, and use the traversed model training sample as the target model sample; Perform lesion detection on the target model sample through the initial lesion detection model to obtain predicted lesion information; Determine the true lesion position, true lesion category, and true grid confidence according to the image lesion marking; Determine the predicted lesion position, predicted lesion category, and predicted grid confidence according to the predicted lesion information; Calculate the model loss value according to the true lesion position, the true lesion category, the true grid confidence, the predicted lesion position, the predicted lesion category, and the predicted grid confidence through a preset loss formula; Adjust the parameters of the initial lesion detection model according to the model loss value; At the end of the traversal, use the initial lesion detection model with adjusted parameters as the preset lesion detection model.

2. The lesion detection device according to claim 1, characterized in that, the preset loss formula is: Where λ lcoord is the preset position weight, ω lnoobj is the confidence weight for the preset without the target object, λ is the confidence weight for the preset with the target object, ω is the preset classification weight, N is the number of grids per row and per column divided according to the image size, i is the row number of the grid, j is the column number of the grid, x ij is the predicted lesion position, C ij is the predicted grid confidence, p ij (c) is the predicted lesion category, is the true lesion position, is the true grid confidence, is the true lesion category.

3. The lesion detection device according to claim 1, characterized in that, the initial lesion detection model includes a feature extraction module and a feature analysis module; The step of performing lesion detection on the target model sample through the initial lesion detection model to obtain predicted lesion information includes: Extract grid sample images from the target model samples; Perform multi-level feature extraction on the grid sample images through the feature extraction module to obtain multi-dimensional image features; Perform lesion detection on the multi-dimensional image features through the feature analysis module to obtain the lesion detection results corresponding to each grid in the grid sample images; Aggregate the lesion detection results corresponding to each grid to obtain predicted lesion information.

4. The lesion detection device according to claim 1, wherein, the step of marking the grid image according to the lesion grid and the lesion type includes: searching for the marked background color corresponding to the lesion grid in a preset type color mapping table according to the lesion type; adjusting the background color of the grid image according to the marked background color and the lesion grid to mark the grid image.

5. A lesion detection system, wherein, the lesion detection system includes: an image processing module for dividing a to-be-detected image into grids to obtain grid images; a lesion detection module for performing lesion detection on the grid images through a preset lesion detection model to obtain lesion grids and the lesion types corresponding to the lesion grids; an image marking module for marking the grid images according to the lesion grids and the lesion types; wherein, the image processing module is further configured to obtain a preset sample image set and the lesion marking frames corresponding to each preset sample image in the preset sample image set; divide the preset sample images in the preset sample image set into grids to obtain a grid image set; traverse the grid image set, and use the traversed grid sample image as a target grid image; obtain the lesion marking frame corresponding to the target grid image; determine the target lesion grids in the target grid image according to the lesion marking frame; determine the grid intersection over union corresponding to each target lesion grid according to the lesion marking frame; set corresponding lesion labels for each target lesion grid based on the grid intersection over union; generate an image lesion marking corresponding to the target sample image according to the lesion label and the target lesion grid; at the end of the traversal, obtain the image lesion markings corresponding to each grid sample image in the grid image set; construct a model sample set according to the image lesion markings and the grid image set; train an initial lesion detection model according to the model sample set to obtain a preset lesion detection model; The image processing module is further configured to traverse the model sample set, and use the traversed model training samples as target model samples; perform lesion detection on the target model samples through the initial lesion detection model to obtain predicted lesion information; determine the true lesion position, true lesion category, and true grid confidence according to the image lesion markings; determine the predicted lesion position, predicted lesion category, and predicted grid confidence according to the predicted lesion information; calculate the model loss value according to the true lesion position, the true lesion category, the true grid confidence, the predicted lesion position, the predicted lesion category, and the predicted grid confidence through a preset loss formula; adjust the parameters of the initial lesion detection model according to the model loss value; at the end of the traversal, use the initial lesion detection model with adjusted parameters as the preset lesion detection model.

6. A computer-readable storage medium, characterized in that a lesion detection program is stored on the computer-readable storage medium, and when the lesion detection program is executed, the following steps are implemented: Perform grid division on the image to be detected to obtain a grid image; Perform lesion detection on the grid image through a preset lesion detection model to obtain lesion grids and the lesion types corresponding to the lesion grids; Mark the grid image according to the lesion grids and the lesion types; Wherein, before the step of performing grid division on the image to be detected to obtain a grid image, the following steps are further included: Obtain a preset sample image set and the lesion marking frames corresponding to each preset sample image in the preset sample image set; Perform grid division on the preset sample images in the preset sample image set to obtain a grid image set; Traverse the grid image set, and use the traversed grid sample image as the target grid image; Obtain the lesion marking frame corresponding to the target grid image; Determine the target lesion grids in the target grid image according to the lesion marking frame; Determine the grid intersection over union corresponding to each target lesion grid according to the lesion marking frame; Set corresponding lesion labels for each target lesion grid based on the grid intersection over union; Generate image lesion markings corresponding to the target sample image according to the lesion labels and the target lesion grids; At the end of the traversal, obtain the image lesion markings corresponding to each grid sample image in the grid image set; Construct a model sample set according to the image lesion markings and the grid image set; Train an initial lesion detection model according to the model sample set to obtain a preset lesion detection model; Wherein, the step of training an initial lesion detection model according to the model sample set to obtain a preset lesion detection model includes: Traverse the model sample set, and use the traversed model training samples as target model samples; Perform lesion detection on the target model samples through the initial lesion detection model to obtain predicted lesion information; Determine the true lesion position, true lesion category, and true grid confidence according to the image lesion markings; Determine the predicted lesion position, predicted lesion category, and predicted grid confidence according to the predicted lesion information; Calculate the model loss value according to the true lesion location, the true lesion category, the true grid confidence, the predicted lesion location, the predicted lesion category, and the predicted grid confidence through a preset loss formula; Adjust the parameters of the initial lesion detection model according to the model loss value; At the end of the traversal, use the initial lesion detection model with adjusted parameters as the preset lesion detection model.

Images