Focus detection based on deep learning and device thereof

Abstract

The invention discloses a focus detection method and device based on deep learning, and relates to the technical field of deep learning, and the method comprises the steps: obtaining a capsule endoscopy image sequence, and carrying out the video frame extraction of the capsule endoscopy image sequence, so as to obtain a gall bladder endoscopy image; constructing a focus detection neural network for detecting the focus of the gallbladder endoscopic image, and performing multi-level feature extraction and multi-level feature fusion on the focus of the gallbladder endoscopic image based on the focus detection neural network to obtain a preliminary detection result; and constructing a deep calculation neural network for deeply extracting focus features and calculating the number of the focus, performing deep fusion feature extraction on the gallbladder endoscopic image according to the preliminary detection result based on the deep calculation neural network, refining repeated judgment of the focus, and generating a final detection result of the focus. And counting and analyzing the lesions according to the final detection result to obtain the number of the lesions and lesion information of each lesion. According to the invention, focus detection speed and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a deep learning-based lesion detection and a device thereof. Background Art

[0002] Crohn's disease (CD) is a chronic, recurrent disease that can affect the entire digestive tract from the mouth to the anus, with 70% of Crohn's disease patients having small intestinal involvement. Currently, the diagnosis and evaluation of small intestinal lesions mainly rely on small intestinal CT imaging, small intestinal MR imaging, small intestinal endoscopy, and capsule endoscopy (CE). However, CT small intestinal imaging and MR small intestinal imaging have their advantages in diagnosing mucosal thickening and extraluminal lesions, but are not sensitive to early mucosal lesions; small intestinal endoscopy has high risks and poor patient tolerance, making it difficult to complete the entire small intestinal examination. CE is painless, visible, and sensitive to mucosal lesions, so patients are more likely to accept CE as a means of diagnosis and evaluation of small intestinal lesions.

[0003] CE mainly moves toward the distal end of the small intestine through the movement of the gastrointestinal tract and passively moves to the distal end of the small intestine with the help of its own gravity, and takes pictures at a rate of about 2 frames per second. It takes about 6-8 hours to complete a small intestinal CE examination, and some patients may need more than 10 hours. Therefore, a small intestinal CE examination can generate tens of thousands of pictures, or even more than 100,000 pictures. Clinicians often spend a lot of time and energy reading and diagnosing CE image data.

[0004] At present, artificial intelligence (AI) has been developed in assisting traditional endoscopy in diagnosing gastrointestinal lesions. Its application in CE with a larger amount of data should be more advantageous and valuable. Among AI algorithms, the convolutional neural network (CNN) model is currently more popular for endoscopic image recognition. There have been some studies on the use of CNN for the recognition of small intestinal lesions, such as small intestinal bleeding, tumors, ulcers, parasites, and celiac disease. CNN has good performance in terms of discrimination accuracy. However, due to the complexity of CE images, there are still some challenges to be solved in this field. CE images are affected by factors such as light, shooting angle, and lens quality. There may be problems such as food residues, bubbles, and bright spots. These factors will interfere with image feature extraction, and challenges in image quality need to be addressed. Different lesion types and locations may appear in CE images, and there is also diversity in the shape, size, and color of the lesions. Therefore, the algorithm needs to have the ability to distinguish and adapt to different lesions and image features to meet the challenges of image diversity. In addition, computer deep learning on CE mostly uses images as data sets to train models, and there are fewer studies using videos as data sets. In clinical practice, the diagnosis and assessment of the activity of small intestinal CD cannot be determined by a single frame of capsule endoscopy, but must be determined in combination with the length, range, and stenosis of the lesion. Therefore, the application of deep learning in videos is more practical in clinical practice.

[0005] The use of artificial intelligence methods for automatic detection is the current development trend. It can not only improve complex detection equipment, but also improve the efficiency and accuracy of CE in diagnosing and evaluating small intestinal CD, reduce the limitations of naked eye observation and reduce the workload of clinicians. It is also expected to promote the application and development of deep learning in the medical field and promote the development of medical intelligence. Currently, a technology that can combine the diagnosis of small intestinal CD with artificial intelligence is needed.

[0006] The above information disclosed in the above Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0007] The present invention provides a deep learning-based lesion detection and device thereof to solve the above-mentioned technical problems.

[0008] The first aspect of the present application provides a deep learning-based lesion detection method, which includes the following steps: Acquiring a capsule endoscopy image sequence, and performing video frame extraction on the capsule endoscopy image sequence to obtain a gallbladder endoscopy image; Constructing a lesion detection neural network for detecting lesions in gallbladder endoscopy images, performing multi-level feature extraction and multi-level feature fusion on the gallbladder endoscopy images based on the lesion detection neural network to obtain preliminary lesion fusion features, and detecting, locating, confidence scoring and classifying the lesion fusion features to obtain preliminary lesion detection results; A deep computing neural network for deeply extracting lesion features and calculating the number of lesions is constructed, and based on the deep computing neural network, deep fusion feature extraction is performed on the gallbladder endoscopic image according to the preliminary detection results, and repeated judgments of lesions are refined to generate final detection results of lesions. Lesions are counted according to the final detection results to obtain the number of lesions, and the lesions are analyzed according to the final detection results to obtain lesion information of each lesion, wherein the lesion information includes lesion size and duration.

[0009] The second aspect of the present application provides a deep learning-based lesion detection device, which includes: An image acquisition module acquires a capsule endoscopy image sequence and performs video frame extraction on the capsule endoscopy image sequence to obtain a gallbladder endoscopy image; A preliminary lesion detection module, constructing a lesion detection neural network for detecting lesions in gallbladder endoscopy images, performing multi-level feature extraction and multi-level feature fusion on the gallbladder endoscopy images based on the lesion detection neural network to obtain preliminary lesion fusion features, and performing detection, positioning, confidence scoring and classification on the lesion fusion features to obtain preliminary lesion detection results; The analysis and counting module constructs a deep computing neural network for deeply extracting lesion features and calculating the number of lesions. Based on the deep computing neural network, the gallbladder endoscopy image is deeply fused and feature extracted according to the preliminary detection results, the repeated judgment of the lesions is refined, and the final detection results of the lesions are generated. The lesions are counted according to the final detection results to obtain the number of lesions: Beneficial effects of the present invention: The present application utilizes deep learning neural network technology to detect the gallbladder endoscopy image sequence frame by frame to extract the lesion features of each gallbladder endoscopy image frame, perform repeated lesion judgment on the lesion features of each gallbladder endoscopy image frame, record the lesion size and duration of each lesion, and count all lesions (excluding repeated lesions) to obtain the number of lesions; the method has fast lesion detection speed and high accuracy, thereby improving the efficiency and accuracy of CE in diagnosing and evaluating small intestinal CD, reducing the limitations of naked eye observation and alleviating the workload of clinicians, and is also expected to promote the application and development of deep learning in the medical field and promote the development of medical intelligence.

[0010] This application uses lesion detection neural network and deep computing neural network to perform two-stage detection of CD lesions and obtain CD lesion feature detection results. First, the pre-built lesion detection neural network is used to extract multi-level features of lesions and fuse multi-level features of lesions in gallbladder endoscopy images to obtain preliminary detection results; then the deep computing neural network is used to perform deep detection on the lesion feature map after the preliminary detection results to extract the lesion features of each frame of gallbladder endoscopy images, and analyze and calculate the lesions of each frame of gallbladder endoscopy images. Through this multi-level, multi-feature analysis method, not only the speed and accuracy of lesion detection are improved, but also the efficient and accurate counting and analysis of lesions in the video are achieved.

[0011] The above invention content is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the drawings to represent the same parts.

[0013] Figure 1 A flowchart of a lesion detection method based on deep learning provided in Example 1 of the present invention; Figure 2 A lesion detection flow chart of a lesion detection neural network provided in Embodiment 1 of the present invention; Figure 3 A flow chart of lesion detection using a deep computing neural network provided in Example 1 of the present invention; Figure 4 A schematic diagram of the structure of a lesion detection device based on deep learning provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0014] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0015] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0016] In the description of the present invention, it is necessary to understand that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0017] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0018] The following describes a method for scoring knee osteoarthritis functions according to an embodiment of the present invention in conjunction with the accompanying drawings. Example

[0019] Figure 1 FIG. 1 is a flow chart of an embodiment of a method for detecting lesions based on deep learning according to the present invention. Figure 1 As shown, the lesion detection method includes the following steps: Step S100, acquiring a capsule endoscopy image sequence, and performing video frame extraction on the capsule endoscopy image sequence to obtain a gallbladder endoscopy image; Optionally, the OpenCV library may be used to extract the capsule endoscopy image sequence frame by frame.

[0020] It should be noted that the capsule endoscopy image sequence is a continuous image sequence automatically captured in the digestive tract by the capsule endoscopy imaging system. This imaging system can adjust the shooting speed as needed. The usual setting is two frames per second, but it can be adjusted according to specific diagnosis and treatment needs. These images are stored in a predetermined database in the order of the time they were taken.

[0021] The video frame extraction of the capsule endoscopy image sequence is to traverse the images in the capsule endoscopy image sequence and obtain the current image traversed. In such an image sequence, any frame of image can be designated as the "current image", and the image immediately following the "current image" is designated as the "next frame of image". After the lesion recognition of the "current image" is completed, the "next frame of image" is used as the "current image" for lesion recognition. For example, if there are two images A and B, and according to the shooting time, image A is before image B, then in the traversal process of the image sequence, image A is the "current image", and the image B immediately following it is the "next frame of image".

[0022] Step S200, constructing a lesion detection neural network for detecting lesions in gallbladder endoscopy images, performing multi-level feature extraction and multi-level feature fusion on the gallbladder endoscopy images based on the lesion detection neural network to obtain preliminary lesion fusion features, and performing detection, positioning, confidence scoring and classification on the lesion fusion features to obtain preliminary lesion detection results; Figure 2 This is a lesion detection flow chart of the lesion detection neural network provided in Example 1 of the present invention; Figure 2 As shown, step S200 includes: Step S201, preprocessing the gallbladder endoscopic image to obtain a preprocessed gallbladder endoscopic image; Step S202, dynamically optimizing the sparse feature extraction of the CNN network, optimizing the structure of the FPN network, extracting lesion features at different levels according to the optimized CNN network and FPN network, fusing the extracted features of different scales to obtain a multi-scale lesion feature map of the processed gallbladder endoscopy image and a corresponding multi-scale lesion detection frame; In step S202, the CNN network is a Sparse R-CNN network. The Sparse R-CNN network is a new paradigm of target detection, which is different from the traditional one-stage and two-stage target detection methods. Sparse R-CNN usually uses sparse feature extraction: by performing convolution operations on the input image, sparse feature maps are extracted to reduce feature redundancy. Sparse RoI Transformer: The feature map is pooled using a sparse RoI Transformer to obtain a feature representation of each. The Sparse R-CNN network performs dynamic sparse optimization on sparse feature extraction, that is, it adopts a dynamic sparse sampling strategy to select only key points that contribute to target detection, thereby reducing the amount of calculation and improving system efficiency.

[0023] In multi-scale feature fusion, the design of the traditional FPN network usually extracts features of different levels from the backbone network for fusion. The present disclosure optimizes the structure of the FPN network based on the ResNet architecture, so that the FPN network integrates more complex top-down and lateral connection structures, extracts lesion features at different levels of the gallbladder endoscopy image through the optimized FPN network, and inputs the extracted features of different scales into a more efficient feature fusion module for fusion processing, so as to obtain a multi-scale lesion detection frame and a corresponding multi-scale feature map of the processed gallbladder endoscopy image; Among them, the FPN network in the present disclosure extracts features {C2, C3, C4, C5} from different layers of the backbone network, and these features correspond to different scales. Each layer contains different levels of semantic information, the bottom feature map has a higher resolution but less semantic information, and the top feature map has a lower resolution but contains more semantic information.

[0024] The horizontal connection structure realizes the fusion between feature maps of different scales. Through the above fusion method, features of different resolutions can be integrated to generate more accurate feature expressions.

[0025] Among them, the top-down connection structure realizes the gradual transfer of high-level semantic information to low-level feature maps, so that the lower-level feature maps can be enhanced by combining the rich semantic information in the high-level feature maps.

[0026] Exemplarily, the input features of the FPN network are first transformed through a 1x1 convolution layer to adjust the number of channels and enhance the flexibility of subsequent processing. Subsequently, the transformed features are divided into two branches: one branch is further subdivided into four sub-branches, each of which is subjected to spatial feature extraction through depthwise separable convolutional layers of different sizes (3x3, 5x5, 7x7, 9x9) to capture information at different scales; the other branch is directly passed to the end of the module. The outputs of these four depthwise convolutions are merged and feature fused through another 1x1 convolution layer. The fused result is added to the direct branch and output after layer normalization and ReLU activation to achieve effective information integration and nonlinear enhancement. The optimized FPN network not only improves the adaptability and expression ability of multi-scale features, but also strengthens the training stability of the FPN network through the residual connection strategy.

[0027] Exemplarily, the feature fusion module is a fuse fusion module, and the extracted features of different scales are input into the fuse fusion module for more efficient fusion. Each fusion module fuses the features of different scales to generate the final feature map. These fused feature maps retain the original resolution while incorporating semantic information from other scales, so that the detector can better handle targets of different scales. This design combines global semantic information while maintaining high resolution, thereby improving the accuracy and robustness when processing lesion targets of various sizes, thereby significantly improving the performance of target detection tasks.

[0028] Exemplarily, the input data in the feature fusion module is first processed by four parallel subpaths, each of which includes a specific type of pooling operation (max pooling or average pooling) followed by a 1x1 convolutional layer for feature transformation. These transformed feature maps are processed by a set of normalization layers (GroupNorm) and activation functions (sigmoid) to adjust and standardize the range of feature values. Next, the four processed feature maps are combined by weighted summation to find the channel attention of the feature map, and then multiplied back to the original feature map. After the channel attention layer, a cross-spatial attention mechanism is introduced to enhance the correlation between features and further improve the expressiveness of information. It can be understood that the cross-spatial attention is achieved by performing information interaction operations on the outputs e and f of the two main branches, where e represents the feature map after activation by channel attention, and f is the multi-scale feature extracted from the multi-scale feature extraction module. Specifically, the original feature map first passes through a "multi-scale feature extraction module", which is used to extract and process input features from different scales, further enhancing the FPN model's ability to process multi-scale information. After each dimension is transformed, a cross multiplication operation is performed to realize the interaction of different information, and finally the information is added and activated by a sigmoid function, and then element-wise multiplication is performed with the feature map after dimensionality reduction to achieve feature enhancement in the cross space. In this way, the feature fusion module can dynamically adjust the importance of features to ensure that key information is highlighted.

[0029] Step S203, performing convolution processing on the multi-scale lesion feature map according to the lesion detection frame to obtain an accurate feature map corresponding to the lesion detection frame; In step S203, convolution processing is performed on the multi-scale lesion feature map output by the CNN network and the FPN network according to the lesion detection frame, so as to accurately extract the feature area corresponding to each lesion detection frame from the multi-scale feature map, thereby reducing the quantization error in ROI Pooling (Region of interest pooling).

[0030] Step S204, updating the position of the lesion detection frame, and classifying and regressing the lesion features of the precise feature map to obtain a classification regression feature map; In step S204, a dynamic instance interaction network is used to perform a series of interactive layer feature updates on the lesion detection frame, mainly including position update of the lesion detection frame, and feature optimization of classification and regression tasks; Interaction layers are used to further update and optimize the features of each lesion detection frame, that is, to update and optimize the lesion features of the precise feature map, which usually includes several rounds of iterations to improve the accuracy of the prediction.

[0031] Step S205, performing a classification operation on the lesions in the classification regression feature map to preliminarily identify the type of the lesions in the precise feature map; determining the bounding box and position of the lesions in the classification regression feature map; In step 205, the classification head (ClassificationHead) in the prediction head (Prediction Heads) is used to predict the category of the target in each lesion detection frame, so as to implement the classification operation of the lesion in the classification regression feature map; The precise position and size of the lesion detection frame are determined by using the regression head in the prediction head to obtain the bounding box and position of the lesion in the classification regression feature map, and the position may be the bounding box coordinates.

[0032] Step S206, determining the lesion features of the gallbladder endoscopy image according to the bounding box, position and type of the lesion, and outputting the bounding box, position and type of the lesion as a preliminary detection result.

[0033] Step S300, constructing a deep computing neural network for deeply extracting lesion features and calculating the number of lesions, performing deep fusion feature extraction on the gallbladder endoscopy image based on the preliminary detection results based on the deep computing neural network, refining repeated judgments of lesions, generating final detection results of lesions, counting lesions according to the final detection results to obtain the number of lesions, analyzing the lesions according to the final detection results to obtain lesion information of each lesion, wherein the lesion information includes lesion size and duration; Figure 3 The flowchart of the deep computing neural network for lesion detection provided in the first embodiment of the present invention is as follows; Figure 3 As shown, step S300 includes: Step S301, obtaining the preliminary detection result and the frame number of the current gallbladder endoscopy image; Step S302, extracting the lesion area image at the corresponding position of each frame of gallbladder endoscopy image according to the position information of the lesion feature in the preliminary detection result; Step S303, optimizing the resnet50 network, using the optimized resnet50 network to perform deep fusion feature extraction on the lesion area image to output the deep fusion feature, and extracting the HSV color histogram of the deep fusion feature through the function in the opencv library; In step S303, the optimized resnet50 network is used to perform deep fusion feature extraction on the lesion area image to extract multi-dimensional features such as color and depth of areas of different sizes around the lesion, and high-level feature extraction is performed to capture the complex details of the lesion.

[0034] Step S304, comparing the deep fusion features of the lesions in the gallbladder endoscopy image of the current frame with the deep fusion features of the lesions in the gallbladder endoscopy image of the historical frame, and determining whether the cosine similarity and the HSV color histogram similarity match. If they match, the matched multiple lesions are repeated lesions; In step S304, the gallbladder endoscopy image of the historical frame refers to a capsule endoscopy image sequence in which lesions have been identified in the capsule endoscopy image sequence, and by comparing the deep fusion features of the lesions in the gallbladder endoscopy image of the current frame with the deep fusion features of the lesions in the gallbladder endoscopy images of all historical frames, it is determined whether the lesion in the current frame is a repeated lesion.

[0035] In order to ensure consistent recognition of the same lesion in different frames, the cosine similarity and HSV color histogram similarity are used to match the features of the lesions just extracted to determine whether the two lesions are the same lesion.

[0036] Step S305, recording the number of repetitions of repeated lesions, and recording the non-repetitive lesions and the deep fusion features corresponding to the non-repetitive lesions; Step S306, counting the number of lesions to obtain the number of lesions; Step S307, analyzing the deep fusion features corresponding to the non-repeated lesions to obtain lesion information of the non-repeated lesions, wherein the lesion information includes lesion size and duration.

[0037] Optionally, in an embodiment of the present application, After step S304, the method may further include: performing auxiliary repeated lesion feature judgment on the deep lesion fusion features obtained by performing deep fusion feature extraction according to the time series analysis information and the intersection-over-union ratio information.

[0038] Optionally, in an embodiment of the present application,Before step S307, the method may also include: obtaining the current sequence of capsule endoscopy images and the total sequence of capsule endoscopy image sequences, determining whether the capsule endoscopy image of the current sequence is the last frame of capsule endoscopy image according to the current sequence and the total sequence, and if so, terminating the detection; if not, continuing to input the capsule endoscopy image of the next sequence as the capsule endoscopy image of the current sequence into the lesion detection neural network and the deep computing neural network in sequence.

[0039] In step S307, the lesion duration is determined according to the number of consecutive frames of the capsule endoscopy image where the repeated lesion is located.

[0040] The present application utilizes deep learning neural network technology to detect the gallbladder endoscopy image sequence frame by frame to extract the lesion features of each gallbladder endoscopy image frame, perform repeated lesion judgment on the lesion features of each gallbladder endoscopy image frame, record the lesion size and duration of each lesion, and count all lesions (excluding repeated lesions) to obtain the number of lesions; the method has fast lesion detection speed and high accuracy, thereby improving the efficiency and accuracy of CE in diagnosing and evaluating small intestinal CD, reducing the limitations of naked eye observation and alleviating the workload of clinicians, and is also expected to promote the application and development of deep learning in the medical field and promote the development of medical intelligence.

[0041] This application uses lesion detection neural network and deep computing neural network to perform two-stage detection of CD lesions and obtain CD lesion feature detection results. First, the pre-built lesion detection neural network is used to extract multi-level features of lesions and fuse multi-level features of lesions in gallbladder endoscopy images to obtain preliminary detection results; then the deep computing neural network is used to perform deep detection on the lesion feature map after the preliminary detection results to extract the lesion features of each frame of gallbladder endoscopy images, and analyze and calculate the lesions of each frame of gallbladder endoscopy images. Through this multi-level, multi-feature analysis method, not only the speed and accuracy of lesion detection are improved, but also the efficient and accurate counting and analysis of lesions in the video are achieved.

[0042] Lesion detection methods also include: Step S400, adopting a secondary confirmation mechanism, obtains the doctor's review results of the capsule endoscopy image sequence, performs standardization correction on the capsule endoscopy image sequence according to the review results, and retrains the lesion detection neural network and the deep computing neural network according to the corrected capsule endoscopy image sequence. Example

[0043] Figure 4 FIG. 1 is a schematic diagram of the structure of a lesion detection device based on deep learning according to an embodiment of the present invention. Figure 4 As shown, including: An image acquisition module acquires a capsule endoscopy image sequence and performs video frame extraction on the capsule endoscopy image sequence to obtain a gallbladder endoscopy image; A preliminary lesion detection module, constructing a lesion detection neural network for detecting lesions in gallbladder endoscopy images, performing multi-level feature extraction and multi-level feature fusion on the gallbladder endoscopy images based on the lesion detection neural network to obtain preliminary lesion fusion features, and performing detection, positioning, confidence scoring and classification on the lesion fusion features to obtain preliminary lesion detection results; The analysis and counting module constructs a deep computing neural network for deeply extracting lesion features and calculating the number of lesions, performs deep fusion feature extraction on the gallbladder endoscopic image based on the preliminary detection results based on the deep computing neural network, refines repeated judgments on lesions, generates final detection results of lesions, counts lesions according to the final detection results to obtain the number of lesions, and analyzes the lesions according to the final detection results to obtain lesion information of each lesion, wherein the lesion information includes lesion size and duration.

[0044] The present application utilizes deep learning neural network technology to detect the gallbladder endoscopy image sequence frame by frame to extract the lesion features of each gallbladder endoscopy image frame, perform repeated lesion judgment on the lesion features of each gallbladder endoscopy image frame, record the lesion size and duration of each lesion, and count all lesions (excluding repeated lesions) to obtain the number of lesions; the method has fast lesion detection speed and high accuracy, thereby improving the efficiency and accuracy of CE in diagnosing and evaluating small intestinal CD, reducing the limitations of naked eye observation and alleviating the workload of clinicians, and is also expected to promote the application and development of deep learning in the medical field and promote the development of medical intelligence.

[0045] This application uses lesion detection neural network and deep computing neural network to perform two-stage detection of CD lesions and obtain CD lesion feature detection results. First, the pre-built lesion detection neural network is used to extract multi-level features of lesions and fuse multi-level features of lesions in gallbladder endoscopy images to obtain preliminary detection results; then the deep computing neural network is used to perform deep detection on the lesion feature map after the preliminary detection results to extract the lesion features of each frame of gallbladder endoscopy images, and analyze and calculate the lesions of each frame of gallbladder endoscopy images. Through this multi-level, multi-feature analysis method, not only the speed and accuracy of lesion detection are improved, but also the efficient and accurate counting and analysis of lesions in the video are achieved.

[0046] It should be noted that the specific implementation of the lesion detection device based on deep learning is consistent with the lesion detection method based on deep learning, which will not be repeated here. Example

[0047] The electronic device provided in the embodiment of the present application may include: Including memory and processor; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the deep learning-based lesion detection method of embodiment 1. Example

[0048] In order to implement the above embodiments, the present disclosure further proposes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first embodiment is implemented.

[0049] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0050] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0051] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A lesion detection method based on deep learning, characterized in that: The following steps are involved: Acquiring a capsule endoscopy image sequence, and performing video frame extraction on the capsule endoscopy image sequence to obtain a gallbladder endoscopy image; Constructing a lesion detection neural network for detecting lesions in gallbladder endoscopy images, performing multi-level feature extraction and multi-level feature fusion on the gallbladder endoscopy images based on the lesion detection neural network to obtain preliminary lesion fusion features, and detecting, locating, confidence scoring and classifying the lesion fusion features to obtain preliminary lesion detection results; A deep computing neural network for deeply extracting lesion features and calculating the number of lesions is constructed, and based on the deep computing neural network, deep fusion feature extraction is performed on the gallbladder endoscopic image according to the preliminary detection results, and repeated judgments of lesions are refined to generate final detection results of lesions. Lesions are counted according to the final detection results to obtain the number of lesions, and the lesions are analyzed according to the final detection results to obtain lesion information of each lesion, wherein the lesion information includes lesion size and duration.

2. The deep learning-based lesion detection method according to claim 1, characterized in that: The method of constructing a lesion detection neural network for detecting lesions in gallbladder endoscopy images, extracting multi-level features and fusing multi-level features of lesions in the gallbladder endoscopy images based on the lesion detection neural network to obtain preliminary lesion fusion features, and detecting, locating, confidence scoring and classifying the preliminary lesion fusion features to obtain preliminary lesion detection results specifically includes: Preprocessing the gallbladder endoscopic image to obtain a preprocessed gallbladder endoscopic image; Dynamically sparsely optimize the sparse feature extraction of the CNN network, perform structural optimization on the FPN network, extract lesion features at different levels according to the optimized CNN network and FPN network, and fuse the extracted features of different scales to obtain a multi-scale lesion feature map of the processed gallbladder endoscopy image and a corresponding multi-scale lesion detection frame; Perform convolution processing on the multi-scale lesion feature map according to the lesion detection frame to obtain the accurate feature map corresponding to the lesion detection frame; The position of the lesion detection frame is updated, and the lesion features of the precise feature map are classified and regressed to obtain a classification regression feature map; the lesions in the classification regression feature map are classified to preliminarily identify the type of the lesions in the precise feature map; Determining a bounding box and a position of a lesion in the classification regression feature map; The lesion features of the gallbladder endoscopic image are determined according to the bounding box, position and type of the lesion, and the bounding box, position and type of the lesion are output as preliminary detection results.

3. The deep learning-based lesion detection method according to claim 1, characterized in that: The method of constructing a deep computing neural network for deeply extracting lesion features and calculating the number of lesions, performing deep fusion feature extraction on the gallbladder endoscopy image based on the deep computing neural network according to the preliminary detection results, refining repeated judgments of lesions, generating final detection results of lesions, and counting lesions according to the final detection results to obtain the number of lesions specifically includes: Obtaining preliminary test results and the frame number of the current gallbladder endoscopy image; Extracting the lesion area image at the corresponding position of each frame of gallbladder endoscopy image according to the position information of the lesion feature in the preliminary detection result; The resnet50 network is optimized, and the deep fusion feature extraction of the lesion area image is performed using the optimized resnet50 network to output the deep fusion feature, and the HSV color histogram of the deep fusion feature is extracted through the function in the opencv library; Compare the deep fusion features of the lesions in the gallbladder endoscopy image of the current frame with the deep fusion features of the lesions in the gallbladder endoscopy image of the historical frame, and determine whether the cosine similarity and the similarity of the HSV color histogram match. If they match, the matched multiple lesions are repeated lesions; The number of repetitions of repeated lesions is recorded, and the non-repetitive lesions and the corresponding deep fusion features of non-repetitive lesions are recorded; And count the number of lesions to obtain the number of lesions; The deep fusion features corresponding to the non-repetitive lesions are analyzed to obtain lesion information of the non-repetitive lesions, wherein the lesion information includes lesion size and duration.

4. The deep learning-based lesion detection method according to claim 3, characterized in that: The repeated determination of the refined lesions also includes: The deep lesion fusion features obtained by deep fusion feature extraction based on time series analysis information and intersection-over-union information are used to assist in the judgment of repeated lesion features.

5. The deep learning-based lesion detection method according to claim 3, characterized in that: The lesion duration was determined based on the number of consecutive frames of capsule endoscopy images where the repeated lesions were located.

6. The deep learning-based lesion detection method according to claim 1, characterized in that: Before generating the final detection result of the lesion, the method also includes: obtaining the current sequence of capsule endoscopy images and the total sequence of capsule endoscopy image sequences, determining whether the capsule endoscopy image of the current sequence is the last frame of capsule endoscopy image according to the current sequence and the total sequence, and if so, terminating the detection; if not, continuing to input the capsule endoscopy image of the next sequence as the capsule endoscopy image of the current sequence into the lesion detection neural network and the deep computing neural network in sequence.

7. A lesion detection device based on deep learning, characterized in that: include: An image acquisition module acquires a capsule endoscopy image sequence and performs video frame extraction on the capsule endoscopy image sequence to obtain a gallbladder endoscopy image; A preliminary lesion detection module, constructing a lesion detection neural network for detecting lesions in gallbladder endoscopy images, performing multi-level feature extraction and multi-level feature fusion on the gallbladder endoscopy images based on the lesion detection neural network to obtain preliminary lesion fusion features, and performing detection, positioning, confidence scoring and classification on the lesion fusion features to obtain preliminary lesion detection results; The analysis and counting module constructs a deep computing neural network for deeply extracting lesion features and calculating the number of lesions, performs deep fusion feature extraction on the gallbladder endoscopic image based on the preliminary detection results based on the deep computing neural network, refines repeated judgments on lesions, generates final detection results of lesions, counts lesions according to the final detection results to obtain the number of lesions, and analyzes the lesions according to the final detection results to obtain lesion information of each lesion, wherein the lesion information includes lesion size and duration.

8. An electronic device, characterized in that: Including memory and processor; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the deep learning-based lesion detection method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the deep learning-based lesion detection method as described in any one of claims 1 to 6 is implemented.

Images