Endoscopic image recognition system and method based on deep learning

Abstract

The present application provides an endoscopic image recognition system and method based on deep learning, which relates to the field of intelligent diagnosis and treatment. The system obtains an endoscopic detection image of a suspected lesion to be identified, extracts a normal state image of the suspected lesion to be identified from a database, and uses image recognition and analysis technology based on deep learning to extract features and optimize the significance of the endoscopic detection image and the normal state image, thereby automatically judging whether a lesion occurs in the suspected lesion to be identified based on the semantic difference coefficient between the optimized detection state and the normal state. In this way, subtle features that are difficult to detect with the naked eye can be captured, thereby more accurately identifying tiny lesions, reducing the possibility of missed diagnosis and misdiagnosis, and reducing dependence on manual operations, thereby improving the automation of endoscopic image recognition.

Description

Technical Field

[0001] The present application relates to the field of intelligent diagnosis and treatment, and more specifically, to an endoscopic image recognition system and method based on deep learning. Background Art

[0002] In the field of medical diagnosis, especially in the screening and diagnosis of digestive tract diseases, endoscopy has become an indispensable means due to its non-invasiveness and intuitiveness. Endoscopy can directly observe the internal conditions of the digestive tract, including the esophagus, stomach, duodenum and even the colon, helping doctors to find signs of early lesions, such as inflammation, polyps, ulcers or tumors.

[0003] However, traditional endoscopic image recognition methods mainly rely on doctors or technicians to observe images with the naked eye to perform manual feature extraction. Specifically, some manually designed visual features, such as texture, color, and shape, although these features can reflect the basic information of the lesion to a certain extent, their recognition accuracy is limited when faced with complex and changeable endoscopic images. In other words, the selection and extraction process of manual features is highly subjective and it is easy to miss important subtle features. In addition, in complex endoscopic images, due to the interference of background texture, the small size of the lesion area, or the similarity with surrounding tissues, tiny lesions are often difficult to detect with the traditional naked eye.

[0004] Therefore, an optimized endoscopic image recognition solution is needed. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, there is provided an endoscope image recognition system based on deep learning, which comprises:

[0007] An endoscopic detection image acquisition module for suspected lesion sites, used to obtain endoscopic detection images of suspected lesion sites to be identified;

[0008] A normal state image acquisition module for suspected lesion parts, used for extracting the normal state image of the suspected lesion parts to be identified from the database;

[0009] A part state feature extraction module, used for passing the endoscope detection image and the normal state image through a part state semantic feature extractor to obtain an endoscope detection part state feature map and a part normal state feature map;

[0010] A part state feature optimization module, used for respectively passing the endoscope detection part state feature map and the part normal state feature map through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map;

[0011] A part semantic difference calculation module, used for calculating a part semantic difference coefficient between the optimized endoscope detection part state feature map and the optimized part normal state feature map;

[0012] A recognition result generation module, used for obtaining a recognition result based on the semantic difference coefficient of the part;

[0013] The part state feature optimization module comprises: an endoscope detection part state feature gradient amplitude value calculation unit, which is used to calculate the multi-directional gradient value distribution of each position in the endoscope detection part state feature map, and determine the gradient amplitude value of each position in the endoscope detection part state feature map based on the multi-directional gradient value distribution of each position to obtain the endoscope detection part state feature gradient amplitude distribution feature map; an endoscope detection part state feature gradient amplitude value local description operator calculation unit, which is used to calculate the gradient amplitude value local description operator of each position in the endoscope detection part state feature gradient amplitude distribution feature map to obtain the endoscope detection part state feature gradient amplitude value local description operator. A local significant distribution feature map of the gradient amplitude of the state characteristic of the endoscope detection part; a gated local significant feature mask unit for the state of the endoscope detection part, which is used to input the local significant distribution feature map of the gradient amplitude of the state characteristic of the endoscope detection part into a gated masker based on the GELU function to obtain a local significant gated mask feature map of the gradient amplitude of the state characteristic of the endoscope detection part; an optimized endoscope detection part state feature generation unit, which is used to calculate the position point multiplication between the local significant gated mask feature map of the gradient amplitude of the state characteristic of the endoscope detection part and the endoscope detection part state feature map to obtain the optimized endoscope detection part state feature map.

[0014] In the above-mentioned deep learning-based endoscopic image recognition system, the part state feature extraction module is used to: pass the endoscopic detection image and the normal state image through a part state semantic feature extractor based on the AlexNet model to obtain the endoscopic detection part state feature map and the part normal state feature map.

[0015] In the above-mentioned endoscope image recognition system based on deep learning, the endoscope detection part state characteristic gradient amplitude value calculation unit is used to: calculate the gradient value of each characteristic value in the endoscope detection part state characteristic map along the width direction to obtain the endoscope detection part state width direction characteristic gradient value characteristic map; calculate the gradient value of each characteristic value in the endoscope detection part state characteristic map along the height direction to obtain the endoscope detection part state height direction characteristic gradient value characteristic map; calculate the gradient value of each characteristic value in the endoscope detection part state characteristic map along the channel direction to obtain the endoscope detection part state channel direction characteristic gradient value characteristic map; calculate the square root of the sum of the squares of the characteristic values ​​at corresponding positions in the endoscope detection part state width direction characteristic gradient value characteristic map, the endoscope detection part state height direction characteristic gradient value characteristic map and the endoscope detection part state channel direction characteristic gradient value characteristic map to obtain the endoscope detection part state characteristic gradient amplitude distribution characteristic map.

[0016] In the above-mentioned endoscope image recognition system based on deep learning, the endoscope detection part state feature gradient amplitude local description operator calculation unit is used to: determine the gradient amplitude local neighborhood range, and calculate the sequence of endoscope detection part state feature gradient amplitude neighborhood distribution feature maps based on the gradient amplitude local neighborhood range and the endoscope detection part state feature gradient amplitude distribution feature map; count the number of feature maps in the sequence of endoscope detection part state feature gradient amplitude neighborhood distribution feature maps to obtain the endoscope detection part state feature gradient amplitude neighborhood number statistics; calculate the endoscope detection part state feature gradient amplitude distribution feature map and the endoscope detection part state feature gradient amplitude distribution feature map. The position differences between each endoscope detection part state feature gradient amplitude neighborhood distribution feature map in the sequence of degree neighborhood distribution feature maps are divided by the endoscope detection part state feature gradient amplitude distribution feature map by position point to obtain a sequence of endoscope detection part state feature gradient relative amplitude distribution neighborhood feature maps; the position point addition of each endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map in the sequence of endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map is divided by the position point by the statistical value of the endoscope detection part state feature gradient amplitude neighborhood to obtain the endoscope detection part state feature gradient amplitude local significant distribution feature map.

[0017] In the above-mentioned endoscope image recognition system based on deep learning, the part semantic difference calculation module is used to: obtain the part semantic difference coefficient by passing the optimized endoscope detection part state feature map and the optimized part normal state feature map through a feature difference calculator.

[0018] In the above-mentioned endoscope image recognition system based on deep learning, the part semantic difference calculation module is used to: calculate the point multiplication of the optimized endoscope detection part state feature map and the optimized part normal state feature map by position, and then calculate the square root of the feature value of each position in the endoscope detection part-normal state feature map to obtain the endoscope detection part-normal state modulation feature map;

[0019] The characteristic values ​​of each position in the endoscope detection part-normal state modulation feature map are added to obtain the endoscope detection part-normal state association value; the global means of the optimized endoscope detection part state feature map and the optimized part normal state feature map are calculated respectively to obtain the optimized endoscope detection part state feature mean and the optimized part normal state feature mean; the optimized endoscope detection part state feature mean, the optimized part normal state feature mean and the square of the number of characteristic values ​​in the feature map are multiplied to obtain a product value, and the square root of the product value is calculated to obtain the endoscope detection part-normal state average value; the ratio of the endoscope detection part-normal state association value to the endoscope detection part-normal state average value is calculated, and then the square root of the difference between constant one and the ratio is calculated to obtain the part semantic difference coefficient.

[0020] In the above-mentioned deep learning-based endoscopic image recognition system, the recognition result generation module is used to determine the recognition result of the suspected lesion site to be identified based on the site semantic difference coefficient and a predetermined threshold, and the recognition result is used to indicate whether the suspected lesion site to be identified has a lesion.

[0021] According to another aspect of the present application, there is provided an endoscope image recognition method based on deep learning, comprising:

[0022] Acquire an endoscopic detection image of a suspected lesion to be identified;

[0023] Extracting the normal state image of the suspected lesion site to be identified from the database;

[0024] The endoscope detection image and the normal state image are passed through a part state semantic feature extractor to obtain an endoscope detection part state feature map and a part normal state feature map;

[0025] The endoscope detection part state feature map and the part normal state feature map are respectively passed through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map;

[0026] Calculating the location semantic difference coefficient between the optimized endoscope detection location state feature map and the optimized location normal state feature map;

[0027] Obtaining a recognition result based on the semantic difference coefficient of the part;

[0028] The endoscope detection part state feature map and the part normal state feature map are respectively passed through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map, including:

[0029] Calculating the multi-directional gradient value distribution of each position in the endoscope detection part state characteristic map, and determining the gradient amplitude value of each position in the endoscope detection part state characteristic map based on the multi-directional gradient value distribution of each position to obtain an endoscope detection part state characteristic gradient amplitude distribution characteristic map;

[0030] Calculating the local description operator of the gradient amplitude of each position in the endoscope detection part state feature gradient amplitude distribution feature map to obtain the local significant distribution feature map of the endoscope detection part state feature gradient amplitude;

[0031] Inputting the local significant distribution feature map of the endoscope detection part state characteristic gradient amplitude into a gated masker based on a GELU function to obtain a local significant gated mask feature map of the endoscope detection part state characteristic gradient amplitude;

[0032] The optimized endoscope detection part state feature map is obtained by calculating the position point multiplication between the local significant gated mask feature map of the endoscope detection part state feature gradient amplitude and the endoscope detection part state feature map.

[0033] In the above-mentioned endoscopic image recognition method based on deep learning, the endoscopic detection image and the normal state image are passed through a part state semantic feature extractor to obtain an endoscopic detection part state feature map and a part normal state feature map, including: passing the endoscopic detection image and the normal state image through a part state semantic feature extractor based on an AlexNet model to obtain the endoscopic detection part state feature map and the part normal state feature map.

[0034] Compared with the prior art, the deep learning-based endoscopic image recognition system and method provided by the present application obtains an endoscopic detection image of a suspected lesion site to be identified, extracts a normal state image of the suspected lesion site to be identified from a database, and uses deep learning-based image recognition and analysis technology to extract features and optimize the significance of the endoscopic detection image and the normal state image, thereby automatically determining whether a lesion occurs in the suspected lesion site to be identified based on the semantic difference coefficient between the optimized detection state and the normal state. In this way, subtle features that are difficult to detect with the naked eye can be captured, thereby more accurately identifying microlesions, reducing the possibility of missed diagnosis and misdiagnosis, and at the same time reducing dependence on manual operations, thereby improving the degree of automation of endoscopic image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0036] Figure 1 4 is a system block diagram of an endoscopic image recognition system based on deep learning according to an embodiment of the present application.

[0037] Figure 2 Schematic diagram of data flow of an endoscopic image recognition system based on deep learning according to an embodiment of the present application.

[0038] Figure 3 This is a block diagram of a part state feature optimization module in an endoscopic image recognition system based on deep learning according to an embodiment of the present application.

[0039] Figure 4 Flow chart of an endoscopic image recognition method based on deep learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0041] In the field of medical diagnosis, especially in the early detection and diagnosis of digestive tract diseases, endoscopy technology plays a vital role due to its non-invasive and intuitive nature. Through endoscopy, doctors can directly observe the details of the digestive tract, such as the esophagus, stomach, duodenum, and colon, so as to promptly detect potential early lesions, such as inflammation, polyps, ulcers, or tumors.

[0042] Nevertheless, traditional endoscopic image analysis methods mainly rely on doctors or technicians to manually extract features by observing the images with the naked eye. This method usually involves some manually designed visual features, such as texture, color, and shape, which can provide basic information about the lesions, but their accuracy is often limited when dealing with complex and changeable endoscopic images. This is because the selection and extraction process of manual features is highly subjective and it is easy to overlook important detail features. In addition, in complex endoscopic images, traditional naked eye detection methods often have difficulty identifying tiny lesions due to the interference of background textures, the tiny size of the lesion area, or the similarity with surrounding tissues.

[0043] Based on this, the present application proposes an endoscopic image recognition solution based on deep learning, which obtains an endoscopic detection image of the suspected lesion site to be identified, extracts the normal state image of the suspected lesion site to be identified from the database, and uses image recognition and analysis technology based on deep learning to extract features and optimize the significance of the endoscopic detection image and the normal state image, so as to automatically judge whether the suspected lesion site to be identified has a lesion based on the semantic difference coefficient between the optimized detection state and the normal state. In this way, subtle features that are difficult to detect with the naked eye can be captured, thereby more accurately identifying tiny lesions, reducing the possibility of missed diagnosis and misdiagnosis, and at the same time reducing dependence on manual operation and improving the automation of endoscopic image recognition.

[0044] Figure 1 4 is a system block diagram of an endoscopic image recognition system based on deep learning according to an embodiment of the present application. Figure 2 FIG. 1 is a data flow diagram of an endoscope image recognition system based on deep learning according to an embodiment of the present application. Figure 1 and Figure 2As shown, in the deep learning-based endoscopic image recognition system 100, it includes: a suspected lesion site endoscopic detection image acquisition module 110, which is used to obtain an endoscopic detection image of the suspected lesion site to be identified; a suspected lesion site normal state image acquisition module 120, which is used to extract the normal state image of the suspected lesion site to be identified from the database; a part state feature extraction module 130, which is used to pass the endoscopic detection image and the normal state image through a part state semantic feature extractor to obtain an endoscopic detection part state feature map and a part normal state feature map; a part state feature optimization module 140, which is used to pass the endoscopic detection part state feature map and the part normal state feature map through a feature distribution gradient mask significance optimization module respectively to obtain an optimized endoscopic detection part state feature map and an optimized part normal state feature map; a part semantic difference calculation module 150, which is used to calculate the part semantic difference coefficient between the optimized endoscopic detection part state feature map and the optimized part normal state feature map; a recognition result generation module 160, which is used to obtain a recognition result based on the part semantic difference coefficient.

[0045] In the embodiment of the present application, the endoscopic detection image acquisition module 110 of the suspected lesion site and the normal state image acquisition module 120 of the suspected lesion site are respectively used to obtain the endoscopic detection image of the suspected lesion site to be identified and to extract the normal state image of the suspected lesion site to be identified from the database. It should be understood that the endoscopic detection image to be identified contains morphological information, color information, texture information, etc. of the internal tissue of the digestive tract. Among them, color information can be used as an important basis for judging whether there is a lesion, such as normal tissues are mostly light pink, while the lesion area may be red; texture information is also an important basis for judgment, and the surface texture of the lesion tissue is often significantly different from that of normal tissue. The normal state image of the suspected lesion site to be identified usually contains the morphological information, color information and texture information of the normal digestive tract tissue, which is a reference standard for evaluating whether the digestive tract tissue has a lesion. In general, the endoscopic detection image and the normal state image of the suspected lesion site to be identified contain rich color, texture and shape information, which form a basis for judging the lesion through comparative analysis and feature extraction of the deep learning model, thereby helping doctors make more accurate diagnoses in endoscopic examinations.

[0046] In an embodiment of the present application, the part state feature extraction module 130 is used to pass the endoscopic detection image and the normal state image through a part state semantic feature extractor to obtain an endoscopic detection part state feature map and a part normal state feature map. Specifically, in an embodiment of the present application, the part state feature extraction module 130 is used to: pass the endoscopic detection image and the normal state image through a part state semantic feature extractor based on the AlexNet model to obtain the endoscopic detection part state feature map and the part normal state feature map. It should be understood that considering that both the endoscopic detection image and the normal state image contain key feature information, including color, texture, shape and other information, these features are very critical for identifying the lesion site in the endoscopic image. AlexNet is a deep convolutional neural network (CNN), which performs well in the field of image recognition and can automatically learn rich feature representations from images. Based on this, in the technical solution of the present application, the endoscopic detection image and the normal state image are passed through a part state semantic feature extractor based on the AlexNet model to capture and extract the key features of the part state, thereby obtaining an endoscopic detection part state feature map and a part normal state feature map.

[0047] In an embodiment of the present application, the part state feature optimization module 140 is used to pass the endoscope detection part state feature map and the part normal state feature map through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map. Accordingly, considering that there is a large amount of interference information and details in the endoscope detection part state feature map and the part normal state feature map, and in order to highlight the difference in key visual features related to normal tissue and lesion sites, and improve the model's recognition ability of lesion sites, in the technical solution of the present application, the endoscope detection part state feature map and the part normal state feature map are passed through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map. In particular, the feature distribution gradient mask significance optimization module introduces a feature gradient amplitude metric and a gated mask mechanism to highlight and strengthen key lesion areas that may exist in the input features.

[0048] Here, the endoscope detection part state feature map is specifically described. In detail, first, the gradient amplitude value of each position in the endoscope detection part state feature map is calculated to generate the endoscope detection part state feature gradient amplitude distribution feature map. That is, the gradient amplitude value is usually higher at the edge of the image and where the texture changes significantly. These places are often the characteristics of the lesion area. By calculating the gradient amplitude of the detection part state, these local key areas where lesions may exist can be highlighted, which is convenient for subsequent lesion detection and analysis. Then, the local description operator of each position in the endoscope detection part state feature gradient amplitude distribution feature map is calculated to highlight the local area with a more significant gradient amplitude to obtain the endoscope detection part state feature gradient amplitude local significant distribution feature map. Furthermore, the endoscope detection part state feature gradient amplitude local significant distribution feature map is input into the gated masker based on the GELU function for nonlinear transformation and feature dynamic selection. It should be understood that the GELU function, as an activation function, can introduce nonlinear transformations to help the model capture more complex feature relationships and enhance the model's expressive power. The gated masker can dynamically select features based on the output of the GELU function, suppress unimportant information, and highlight features that are more meaningful for lesion detection. Finally, the masked features are point-by-point multiplied with the endoscope detection part state feature map to generate an optimized endoscope detection part state feature map containing the original detection state features and gradient significance information, so as to highlight key areas in the image, such as lesions, making these areas more prominent in subsequent processing.

[0049] Specifically, Figure 3 FIG. 1 is a block diagram of a part state feature optimization module in an endoscope image recognition system based on deep learning according to an embodiment of the present application. Figure 3As shown, the part state feature optimization module 140 includes: an endoscope detection part state feature gradient amplitude value calculation unit 141, which is used to calculate the multi-directional gradient value distribution of each position in the endoscope detection part state feature map, and determine the gradient amplitude value of each position in the endoscope detection part state feature map based on the multi-directional gradient value distribution of each position to obtain the endoscope detection part state feature gradient amplitude distribution feature map; an endoscope detection part state feature gradient amplitude value local description operator calculation unit 142, which is used to calculate the gradient amplitude value local description operator of each position in the endoscope detection part state feature gradient amplitude distribution feature map to obtain An endoscope detection part state characteristic gradient amplitude local significant distribution feature map; an endoscope detection part state gated local significant feature mask unit 143, used to input the endoscope detection part state characteristic gradient amplitude local significant distribution feature map into a gated masker based on a GELU function to obtain an endoscope detection part state characteristic gradient amplitude local significant gated mask feature map; an optimized endoscope detection part state feature generation unit 144, used to calculate the position point multiplication between the endoscope detection part state characteristic gradient amplitude local significant gated mask feature map and the endoscope detection part state characteristic map to obtain the optimized endoscope detection part state characteristic map.

[0050] More specifically, in an embodiment of the present application, the endoscope detection part state characteristic gradient amplitude value calculation unit 141 is used to: calculate the gradient value of each characteristic value in the endoscope detection part state characteristic map along the width direction to obtain the endoscope detection part state width direction characteristic gradient value characteristic map; calculate the gradient value of each characteristic value in the endoscope detection part state characteristic map along the height direction to obtain the endoscope detection part state height direction characteristic gradient value characteristic map; calculate the gradient value of each characteristic value in the endoscope detection part state characteristic map along the channel direction to obtain the endoscope detection part state channel direction characteristic gradient value characteristic map; calculate the square root of the sum of the squares of the characteristic values ​​at corresponding positions in the endoscope detection part state width direction characteristic gradient value characteristic map, the endoscope detection part state height direction characteristic gradient value characteristic map and the endoscope detection part state channel direction characteristic gradient value characteristic map to obtain the endoscope detection part state characteristic gradient amplitude distribution characteristic map.

[0051] More specifically, in the embodiment of the present application, the endoscope detection part state feature gradient amplitude local description operator calculation unit 142 is used to: determine the local neighborhood range of the gradient amplitude, and calculate the sequence of the endoscope detection part state feature gradient amplitude neighborhood distribution feature map based on the local neighborhood range of the gradient amplitude and the endoscope detection part state feature gradient amplitude distribution feature map; count the number of feature maps in the sequence of the endoscope detection part state feature gradient amplitude neighborhood distribution feature map to obtain the statistical value of the number of endoscope detection part state feature gradient amplitude neighborhoods; calculate the local neighborhood range of the gradient amplitude distribution feature map and the endoscope detection part state feature gradient amplitude neighborhood distribution feature map. The position differences between each endoscope detection part state feature gradient amplitude neighborhood distribution feature map in the sequence of domain distribution feature maps are divided by the endoscope detection part state feature gradient amplitude distribution feature map by position point to obtain a sequence of endoscope detection part state feature gradient relative amplitude distribution neighborhood feature maps; the position point addition of each endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map in the sequence of endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map is divided by the position point by the statistical value of the endoscope detection part state feature gradient amplitude neighborhood to obtain the endoscope detection part state feature gradient amplitude local significant distribution feature map.

[0052] In the embodiment of the present application, specifically, the part state feature optimization module 140 is used to: process the endoscope detection part state feature map through the feature distribution gradient mask significance optimization module with the following gradient mask significance formula to obtain the optimized endoscope detection part state feature map; wherein the gradient mask significance formula is:

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] in, , , , , and They are respectively the first , , , , and The eigenvalues ​​at the positions, is the first The gradient amplitude value at each position, is the first The gradient amplitude value at each position, is the first The gradient amplitude value at each position, is the first The eigenvalues ​​at the positions, is the first The eigenvalues ​​at the positions, The range is Neighborhood of express The number of elements in , The local significant distribution feature map of the characteristic gradient amplitude of the endoscope detection part state is represented by The eigenvalues ​​at the positions, represents the Gaussian error linear gating function, The local significant gated mask feature map of the state feature gradient amplitude of the endoscope detection part is represented by The eigenvalues ​​at the positions, is a local significant gated mask feature map of the endoscope detection part state characteristic gradient amplitude, is the endoscope detection part state characteristic diagram, It means point multiplication by position. is the state characteristic diagram of the optimized endoscope detection part. Similarly, the encoding method of the normal state characteristic diagram of the part is also as shown in the above formula.

[0061] In the embodiment of the present application, the part semantic difference calculation module 150 is used to calculate the part semantic difference coefficient between the optimized endoscope detection part state feature map and the optimized part normal state feature map. Specifically, in the embodiment of the present application, the part semantic difference calculation module 150 is used to: pass the optimized endoscope detection part state feature map and the optimized part normal state feature map through a feature difference calculator to obtain the part semantic difference coefficient. More specifically, in the embodiment of the present application, the part semantic difference calculation module 150 is used to: calculate the square root of the characteristic value of each position in the endoscope detection part-normal state feature map after calculating the position point multiplication of the optimized endoscope detection part state feature map and the optimized part normal state feature map to obtain the endoscope detection part-normal state modulation feature map; add the characteristic values ​​of each position in the endoscope detection part-normal state modulation feature map to obtain the endoscope detection part-normal state association value; respectively calculate the optimized endoscope detection part state feature map and the optimized part normal state feature map. The global mean of the normal state feature map is calculated to obtain the optimized endoscope detection part state feature mean and the optimized part normal state feature mean; the optimized endoscope detection part state feature mean, the optimized part normal state feature mean and the square of the number of feature values ​​in the feature map are multiplied to obtain a product value, and the square root of the product value is calculated to obtain the endoscope detection part-normal state average value; the ratio of the endoscope detection part-normal state association value to the endoscope detection part-normal state average value is calculated, and then the square root of the difference between a constant one and the ratio is calculated to obtain the part semantic difference coefficient.

[0062] It should be understood that in order to intuitively reflect the degree of difference in visual features between the detected part and the normal part, so as to more accurately identify the potential lesion area, in the technical solution of the present application, the optimized endoscope detection part state feature map and the optimized part normal state feature map are passed through a feature difference calculator to obtain the part semantic difference coefficient. In particular, the feature difference calculator calculates in detail the geometric mean of the product of the eigenvalues ​​of each position in the optimized endoscope detection part state feature map and the optimized part normal state feature map, and performs global normalization processing, so as to accurately quantify the semantic difference between the lesion part and the normal part in the endoscopic detection image, provide a numerical measurement for lesion detection, and thus improve the accuracy of lesion detection.

[0063] In the embodiment of the present application, specifically, the part semantic difference calculation module 150 is used to: process the optimized endoscope detection part state feature map and the optimized part normal state feature map through a feature difference calculator, using the following feature difference calculation formula to obtain the part semantic difference coefficient; wherein the feature difference calculation formula is:

[0064] ;

[0065] in, and The optimized endoscope detection part state characteristic diagram and the optimized part normal state characteristic diagram are respectively The eigenvalues ​​at the positions, , and are the width, height, and number of channels of each feature matrix along the channel dimension in the feature map, respectively. and are the global means of the state characteristic graph of the optimized endoscope detection part and the normal state characteristic graph of the optimized part, respectively. is the number of eigenvalues ​​in the feature graph, is the semantic difference coefficient of the part.

[0066] In the embodiment of the present application, the recognition result generation module 160 is used to obtain the recognition result based on the semantic difference coefficient of the part. Specifically, in the embodiment of the present application, the recognition result generation module 160 is used to: determine the recognition result of the suspected lesion part to be identified based on the semantic difference coefficient of the part and a predetermined threshold, and the recognition result is used to indicate whether the suspected lesion part to be identified has a lesion. It should be understood that in order to intuitively reflect the degree of difference in visual features between the detection part and the normal part, so as to more accurately identify the potential lesion area, in the technical solution of the present application, the optimized endoscope detection part state feature map and the optimized part normal state feature map are passed through a feature difference calculator to obtain the part semantic difference coefficient. In particular, the feature difference calculator calculates the geometric mean of the product of the eigenvalues ​​of each position in the optimized endoscope detection part state feature map and the optimized part normal state feature map in detail, and performs global normalization processing, so as to accurately quantify the semantic difference between the lesion part and the normal part in the endoscopic detection image, and provide a numerical measurement for lesion detection, thereby improving the accuracy of lesion detection.

[0067] Preferably, considering that the optimized endoscopic detection part state feature map and the optimized part normal state feature map respectively represent the image semantic features of the endoscopic detection image and the normal state image, which are optimized for significance based on the image semantic feature space distribution gradient, when calculating the part semantic difference coefficient therebetween to map them to the common image semantic feature difference metric space, there will also be an image semantic feature aggregation mapping offset caused by the inconsistent significance of the image semantic feature space distribution gradient, so as to further improve the comprehensibility of the common mapping of the image semantic features of the optimized endoscopic detection part state feature map and the optimized part normal state feature map, thereby improving the calculation accuracy of the part semantic difference coefficient.

[0068] Therefore, when the optimized endoscope detection part state feature map and the optimized part normal state feature map are passed through the feature difference calculator to obtain the part semantic difference coefficient, the present application jointly optimizes the optimized endoscope detection part state feature map and the optimized part normal state feature map, including the steps of: expanding the optimized endoscope detection part state feature map and the optimized part normal state feature map into an optimized endoscope detection part state feature vector and an optimized part normal state feature vector, and further cascading them into an optimized endoscope detection part state-normal state cascade feature vector; subtracting the length of the optimized endoscope detection part state-normal state cascade feature vector from the The 0 norm of the optimized endoscope detection part state-normal state cascade feature vector is obtained to obtain the optimized endoscope detection part state-normal state cascade isolated representation value; the logarithm with base 2 of the sum of the square of the optimized endoscope detection part state-normal state cascade isolated representation value and the optimized endoscope detection part state-normal state cascade isolated representation value is calculated to obtain the optimized endoscope detection part state-normal state cascade information order value; the power function is calculated with each eigenvalue of the optimized endoscope detection part state-normal state cascade feature vector as the base, and the difference of the optimized endoscope detection part state-normal state cascade isolated representation value minus one is the exponential value, and Perform a point multiplication with the optimized endoscope detection part state-normal state cascade information order value to obtain an optimized endoscope detection part state-normal state cascade leading vector; perform a point multiplication on the optimized endoscope detection part state-normal state cascade feature vector and the difference between the optimized endoscope detection part state-normal state cascade isolated representation value minus one, and then perform a point multiplication with the reciprocal of the optimized endoscope detection part state-normal state cascade isolated representation value to obtain an optimized endoscope detection part state-normal state cascade field constraint vector; calculate each eigenvalue of the optimized endoscope detection part state-normal state cascade field constraint vector with the natural constant as the base as an exponent An exponential function is used to obtain an optimized endoscope detection part state-normal state cascade field bias vector; the optimized endoscope detection part state-normal state cascade guide vector is dot-added with the optimized endoscope detection part state-normal state cascade field bias vector to obtain an optimized optimized endoscope detection part state-normal state cascade feature vector; the optimized optimized endoscope detection part state-normal state cascade feature vector is split into an optimized optimized endoscope detection part state feature vector and an optimized optimized part normal state feature vector, and restored to an optimized optimized endoscope detection part state feature graph and an optimized optimized part normal state feature graph.

[0069] The optimization process of optimizing the cascade feature vector of the endoscope detection part state-normal state is expressed as:

[0070] ;

[0071] ;

[0072] in, is the optimized endoscope detection part state-normal state cascade feature vector, represents the length of the optimized endoscope detection part state-normal state cascade feature vector, represents the zero norm of the eigenvector, represents the optimized endoscope detection site state-normal state cascade isolated representation value, To calculate the power function with each eigenvalue of the optimized endoscope detection part state-normal state cascade eigenvector as the base, the difference between the optimized endoscope detection part state-normal state cascade isolated representation value minus one is the exponent, It means point multiplication by position. Indicates adding by position point. represents the natural exponential function, Represents the optimized endoscope detection part state-normal state cascade feature vector after optimization.

[0073] Therefore, the high-dimensional feature manifold of the optimized endoscope detection part state feature map and the optimized part normal state feature map is represented by the eigenvalue of the feature set as the vector field of the aggregation dimension, and the overlapping value of the vector field of the optimized endoscope detection part state-normal state cascade feature vector at the isolated zero position is used as the order information to fix the local position of the eigenvalue of its feature set, and a bias for the reversibility of the feature regression distribution field of the optimized endoscope detection part state-normal state cascade feature vector is added as a reward to achieve the mapping target tracking of the regression distribution of the optimized endoscope detection part state-normal state cascade feature vector for the eigenvalue position, so that the feature sets of the optimized endoscope detection part state feature map and the optimized part normal state feature map perceive the mapping migration to the aggregate distribution, thereby improving the calculation accuracy of the part semantic difference coefficient by improving the comprehensibility of the common mapping of the image semantic features of the optimized endoscope detection part state feature map and the optimized part normal state feature map. In this way, subtle features that are difficult to detect with the naked eye can be captured, thereby more accurately identifying tiny lesions and reducing the possibility of missed diagnosis and misdiagnosis. At the same time, it reduces dependence on manual operations and improves the degree of automation of endoscopic image recognition.

[0074] In summary, the deep learning-based endoscopic image recognition system 100 according to the embodiment of the present application is explained, which obtains an endoscopic detection image of a suspected lesion site to be identified, extracts a normal state image of the suspected lesion site to be identified from a database, and uses deep learning-based image recognition and analysis technology to extract features and optimize the significance of the endoscopic detection image and the normal state image, thereby automatically judging whether a lesion occurs in the suspected lesion site to be identified based on the semantic difference coefficient between the optimized detection state and the normal state. In this way, subtle features that are difficult to detect with the naked eye can be captured, thereby more accurately identifying tiny lesions, reducing the possibility of missed diagnosis and misdiagnosis, and at the same time reducing dependence on manual operations, thereby improving the degree of automation of endoscopic image recognition.

[0075] As described above, the deep learning-based endoscopic image recognition system 100 according to the embodiment of the present application can be implemented in various terminal devices, such as a server for deep learning-based endoscopic image recognition. In one example, the deep learning-based endoscopic image recognition system 100 according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the deep learning-based endoscopic image recognition system 100 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the deep learning-based endoscopic image recognition system 100 can also be one of the many hardware modules of the terminal device.

[0076] Alternatively, in another example, the deep learning-based endoscopic image recognition system 100 and the terminal device may also be separate devices, and the deep learning-based endoscopic image recognition system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0077] Figure 4 FIG. 1 is a flow chart of an endoscope image recognition method based on deep learning according to an embodiment of the present application. Figure 4As shown, in the endoscopic image recognition method based on deep learning, it includes: S110, obtaining an endoscopic detection image of a suspected lesion part to be identified; S120, extracting a normal state image of the suspected lesion part to be identified from a database; S130, passing the endoscopic detection image and the normal state image through a part state semantic feature extractor to obtain an endoscopic detection part state feature map and a part normal state feature map; S140, passing the endoscopic detection part state feature map and the part normal state feature map through a feature distribution gradient mask significance optimization module respectively to obtain an optimized endoscopic detection part state feature map and an optimized part normal state feature map; S150, calculating the part semantic difference coefficient between the optimized endoscopic detection part state feature map and the optimized part normal state feature map; S160, obtaining a recognition result based on the part semantic difference coefficient.

[0078] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned endoscope image recognition method based on deep learning have been referred to above. Figures 1 to 3 The description of the deep learning-based endoscopic image recognition system has been introduced in detail, and therefore, its repeated description will be omitted.

[0079] In summary, the deep learning-based endoscopic image recognition method based on the embodiment of the present application is explained, which obtains the endoscopic detection image of the suspected lesion site to be identified, and extracts the normal state image of the suspected lesion site to be identified from the database, and uses the image recognition and analysis technology based on deep learning to extract features and optimize the significance of the endoscopic detection image and the normal state image, so as to automatically judge whether the suspected lesion site to be identified has a lesion based on the semantic difference coefficient between the optimized detection state and the normal state. In this way, subtle features that are difficult to detect with the naked eye can be captured, thereby more accurately identifying small lesions, reducing the possibility of missed diagnosis and misdiagnosis, and at the same time reducing the dependence on manual operation and improving the automation of endoscopic image recognition.

[0080] The above description is only a preferred embodiment of the present application and does not constitute any form of limitation to the present application. Although the present application has been disclosed in the form of a preferred embodiment, it is not intended to limit the present application. Any technician familiar with the profession can make some changes or modifications to the equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present application. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. An endoscope image recognition system based on deep learning, It is characterized in that include: An endoscopic detection image acquisition module for suspected lesion sites, used to obtain endoscopic detection images of suspected lesion sites to be identified; A normal state image acquisition module for suspected lesion parts, used for extracting the normal state image of the suspected lesion parts to be identified from the database; A part state feature extraction module, used for passing the endoscope detection image and the normal state image through a part state semantic feature extractor to obtain an endoscope detection part state feature map and a part normal state feature map; A part state feature optimization module, used for respectively passing the endoscope detection part state feature map and the part normal state feature map through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map; A part semantic difference calculation module, used for calculating a part semantic difference coefficient between the optimized endoscope detection part state feature map and the optimized part normal state feature map; A recognition result generation module, used for obtaining a recognition result based on the semantic difference coefficient of the part; Wherein, the part state feature optimization module includes: an endoscope detection part state characteristic gradient amplitude value calculation unit, used to calculate the multi-directional gradient value distribution of each position in the endoscope detection part state characteristic map, and determine the gradient amplitude value of each position in the endoscope detection part state characteristic map based on the multi-directional gradient value distribution of each position to obtain an endoscope detection part state characteristic gradient amplitude distribution characteristic map; An endoscope detection part state feature gradient amplitude local description operator calculation unit, used to calculate the gradient amplitude local description operator of each position in the endoscope detection part state feature gradient amplitude distribution feature map to obtain the endoscope detection part state feature gradient amplitude local significant distribution feature map; An endoscope detection part state gated local significant feature mask unit, used for inputting the endoscope detection part state feature gradient amplitude local significant distribution feature map into a gated masker based on a GELU function to obtain an endoscope detection part state feature gradient amplitude local significant gated mask feature map; An optimized endoscope detection part state feature generation unit is used to calculate the position point multiplication between the endoscope detection part state feature gradient amplitude local significant gated mask feature map and the endoscope detection part state feature map to obtain the optimized endoscope detection part state feature map; Wherein, the endoscope detection part state characteristic gradient amplitude value calculation unit is used for: Calculating the gradient value of each characteristic value in the endoscope detection part state characteristic map along the width direction to obtain the endoscope detection part state width direction characteristic gradient value characteristic map; Calculating the gradient value of each characteristic value in the endoscope detection part state characteristic map along the height direction to obtain the endoscope detection part state height direction characteristic gradient value characteristic map; Calculating the gradient value of each characteristic value in the endoscope detection part state characteristic map along the channel direction to obtain the endoscope detection part state channel direction characteristic gradient value characteristic map; Calculate the square root of the sum of squares of characteristic values ​​at corresponding positions in the endoscope detection part state width direction characteristic gradient value characteristic map, the endoscope detection part state height direction characteristic gradient value characteristic map, and the endoscope detection part state channel direction characteristic gradient value characteristic map to obtain the endoscope detection part state characteristic gradient amplitude distribution characteristic map; Wherein, the endoscope detection part state characteristic gradient amplitude local description operator calculation unit is used for: Determine a local neighborhood range of the gradient amplitude, and calculate a sequence of endoscope detection part state feature gradient amplitude neighborhood distribution feature maps based on the local neighborhood range of the gradient amplitude and the endoscope detection part state feature gradient amplitude distribution feature map; Counting the number of feature graphs in the sequence of the endoscope detection part state feature gradient amplitude neighborhood distribution feature graphs to obtain a statistical value of the number of endoscope detection part state feature gradient amplitude neighborhoods; Calculating the position difference between each endoscope detection part state feature gradient amplitude neighborhood distribution feature map in the sequence of the endoscope detection part state feature gradient amplitude distribution feature map and the endoscope detection part state feature gradient amplitude neighborhood distribution feature map, and dividing the difference by the position point with the endoscope detection part state feature gradient amplitude distribution feature map to obtain a sequence of endoscope detection part state feature gradient relative amplitude distribution neighborhood feature maps; Each endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map in the sequence of the endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map is added point by point and then divided by the statistical value of the endoscope detection part state feature gradient amplitude neighborhood number by point to obtain the endoscope detection part state feature gradient amplitude local significant distribution feature map.

2. The deep learning-based endoscope image recognition system according to claim 1, It is characterized in that The part state feature extraction module is used to: pass the endoscope detection image and the normal state image through a part state semantic feature extractor based on the AlexNet model to obtain the endoscope detection part state feature map and the part normal state feature map.

3. The deep learning-based endoscope image recognition system according to claim 2, It is characterized in that The part semantic difference calculation module is used to: obtain the part semantic difference coefficient by passing the optimized endoscope detection part state feature map and the optimized part normal state feature map through a feature difference calculator.

4. The deep learning-based endoscope image recognition system according to claim 3, It is characterized in that The part semantic difference calculation module is used for: After calculating the point-by-point multiplication of the optimized endoscope detection part state characteristic map and the optimized part normal state characteristic map, the square root of the characteristic value of each position in the endoscope detection part-normal state characteristic map is calculated to obtain the endoscope detection part-normal state modulation characteristic map; Adding the characteristic values ​​of each position in the endoscope detection part-normal state modulation characteristic map to obtain an endoscope detection part-normal state correlation value; Respectively calculating the global mean of the optimized endoscope detection part state characteristic graph and the optimized part normal state characteristic graph to obtain the optimized endoscope detection part state characteristic mean and the optimized part normal state characteristic mean; Multiplying the optimized endoscope detection part state feature mean, the optimized part normal state feature mean and the square of the number of feature values ​​in the feature map to obtain a product value, and calculating the square root of the product value to obtain the endoscope detection part-normal state average value; The ratio of the endoscope detection part-normal state association value to the endoscope detection part-normal state average value is calculated, and then the square root of the difference between a constant one and the ratio is calculated to obtain the part semantic difference coefficient.

5. The deep learning-based endoscope image recognition system according to claim 4, It is characterized in that The recognition result generating module is used to determine the recognition result of the suspected lesion part to be identified based on the part semantic difference coefficient and a predetermined threshold, and the recognition result is used to indicate whether the suspected lesion part to be identified has a lesion.

6. An endoscopic image recognition method based on deep learning, It is characterized in that include: Acquire an endoscopic detection image of a suspected lesion to be identified; Extracting the normal state image of the suspected lesion site to be identified from the database; The endoscope detection image and the normal state image are passed through a part state semantic feature extractor to obtain an endoscope detection part state feature map and a part normal state feature map; The endoscope detection part state feature map and the part normal state feature map are respectively passed through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map; Calculating the location semantic difference coefficient between the optimized endoscope detection location state feature map and the optimized location normal state feature map; Obtaining a recognition result based on the semantic difference coefficient of the part; The endoscope detection part state feature map and the part normal state feature map are respectively passed through a feature distribution gradient mask significance optimization module to obtain an optimized endoscope detection part state feature map and an optimized part normal state feature map, including: Calculating the multi-directional gradient value distribution of each position in the endoscope detection part state characteristic map, and determining the gradient amplitude value of each position in the endoscope detection part state characteristic map based on the multi-directional gradient value distribution of each position to obtain an endoscope detection part state characteristic gradient amplitude distribution characteristic map; Calculating the local description operator of the gradient amplitude at each position in the endoscope detection part state feature gradient amplitude distribution feature map to obtain the local significant distribution feature map of the endoscope detection part state feature gradient amplitude; Inputting the local significant distribution feature map of the endoscope detection part state characteristic gradient amplitude into a gated masker based on a GELU function to obtain a local significant gated mask feature map of the endoscope detection part state characteristic gradient amplitude; Calculate the position point multiplication between the local significant gated mask feature map of the endoscope detection part state characteristic gradient amplitude and the endoscope detection part state feature map to obtain the optimized endoscope detection part state feature map; The method of calculating the multidirectional gradient value distribution of each position in the endoscope detection part state characteristic map and determining the gradient amplitude value of each position in the endoscope detection part state characteristic map based on the multidirectional gradient value distribution of each position to obtain the endoscope detection part state characteristic gradient amplitude distribution characteristic map includes: Calculating the gradient value of each characteristic value in the endoscope detection part state characteristic map along the width direction to obtain the endoscope detection part state width direction characteristic gradient value characteristic map; Calculating the gradient value of each characteristic value in the endoscope detection part state characteristic map along the height direction to obtain the endoscope detection part state height direction characteristic gradient value characteristic map; Calculating the gradient value of each characteristic value in the endoscope detection part state characteristic map along the channel direction to obtain the endoscope detection part state channel direction characteristic gradient value characteristic map; Calculate the square root of the sum of squares of characteristic values ​​at corresponding positions in the endoscope detection part state width direction characteristic gradient value characteristic map, the endoscope detection part state height direction characteristic gradient value characteristic map, and the endoscope detection part state channel direction characteristic gradient value characteristic map to obtain the endoscope detection part state characteristic gradient amplitude distribution characteristic map; The step of calculating the local description operator of the gradient amplitude at each position in the endoscope detection part state feature gradient amplitude distribution feature map to obtain the local significant distribution feature map of the endoscope detection part state feature gradient amplitude includes: Determine a local neighborhood range of the gradient amplitude, and calculate a sequence of endoscope detection part state feature gradient amplitude neighborhood distribution feature maps based on the local neighborhood range of the gradient amplitude and the endoscope detection part state feature gradient amplitude distribution feature map; Counting the number of feature graphs in the sequence of the endoscope detection part state feature gradient amplitude neighborhood distribution feature graphs to obtain a statistical value of the number of endoscope detection part state feature gradient amplitude neighborhoods; Calculating the position difference between each endoscope detection part state feature gradient amplitude neighborhood distribution feature map in the sequence of the endoscope detection part state feature gradient amplitude distribution feature map and the endoscope detection part state feature gradient amplitude neighborhood distribution feature map, and dividing the difference by the position point with the endoscope detection part state feature gradient amplitude distribution feature map to obtain a sequence of endoscope detection part state feature gradient relative amplitude distribution neighborhood feature maps; Each endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map in the sequence of the endoscope detection part state feature gradient relative amplitude distribution neighborhood feature map is added point by point and then divided by the statistical value of the endoscope detection part state feature gradient amplitude neighborhood number by point to obtain the endoscope detection part state feature gradient amplitude local significant distribution feature map.

7. The endoscope image recognition method based on deep learning according to claim 6, It is characterized in that The endoscope detection image and the normal state image are passed through a part state semantic feature extractor to obtain an endoscope detection part state feature map and a part normal state feature map, including: the endoscope detection image and the normal state image are passed through a part state semantic feature extractor based on an AlexNet model to obtain the endoscope detection part state feature map and the part normal state feature map.

Images