Out-of-detection prompting method and device based on convolutional neural network, equipment and medium

Abstract

The invention relates to the technical field of computers, and discloses a convolutional neural network-based leak detection prompting method, device and equipment and a medium, and the method comprises the steps: obtaining image data of a current detection region shot by a camera device in an endoscope detection process, and carrying out the definition detection of the image data based on a LeNet convolutional neural network, and obtaining a definition value; measuring a movement acceleration value of a lens body of the camera device by using a fiber grating acceleration sensor; according to the definition value of the image data and the moving acceleration value of the mirror body, whether the current detection area is a leak detection area is judged; according to the method, definition detection is carried out on the image data through the LeNet convolutional neural network, the movement acceleration value of the lens body of the camera device is measured, whether the image is clearly displayed, whether the lens body moves too fast and whether missing detection is carried out are judged, and the missing detection prompt information is generated when missing detection is determined. A doctor is reminded to reinspect the leak detection area, and the lesion detection rate is increased.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a missed detection prompting method, device, equipment and medium based on a convolutional neural network. Background Art

[0002] Endoscopes are medical devices that can observe the inside of the human body and are widely used in the fields of inspection and treatment. Endoscope inspection devices usually consist of an insertion unit, a handheld unit, an image processing system, and a display system.

[0003] Endoscopic inspection in related technologies mainly relies on the doctor's operating skills and experience to insert the endoscope into the patient for inspection. Due to the complex shape of the human body, higher requirements are placed on the doctor's operating skills, which increases the difficulty of the doctor's operation. In addition, during the inspection process, it is easy to miss the inspection due to too fast insertion speed and improper insertion skills. Therefore, how to timely provide missed inspection prompts during the inspection with an endoscope has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present invention provides a missed detection prompt method, device, equipment and medium based on a convolutional neural network to solve the problem of how to generate missed detection prompt information and provide missed detection prompts to doctors during the endoscopic insertion inspection process.

[0005] In a first aspect, the present invention provides a missed detection prompting method based on a convolutional neural network, the method comprising:

[0006] Acquire image data of the current detection area captured by the camera device during the endoscope detection process, and perform clarity detection on the image data based on the LeNet convolutional neural network to obtain a clarity value of the image data;

[0007] Using a fiber grating acceleration sensor to measure the moving acceleration value of the camera lens;

[0008] According to the clarity value of the image data and the moving acceleration value of the mirror body, it is determined whether the current detection area is a missed detection area;

[0009] If the current detection area is a missed detection area, a missed detection prompt message is generated.

[0010] The present invention performs clarity detection on image data based on the LeNet convolutional neural network, and uses a fiber grating acceleration sensor to measure the moving acceleration value of the camera lens. It determines whether the image is displayed clearly and whether the lens moves too fast according to the clarity value and the moving acceleration value, so as to determine whether the current detection area is a missed detection area. When it is determined to be a missed detection area, a missed detection prompt message is generated to remind the doctor to re-examine the missed detection area, thereby improving the detection rate of lesions.

[0011] In an optional implementation, the clarity detection of the image data is performed based on the LeNet convolutional neural network to obtain the clarity value of the image data, including:

[0012] Collecting target sample image data, and establishing a data set based on the collected target sample image data;

[0013] Normalize the image pixel values ​​in the data set;

[0014] The normalized data set is input into the pre-built LeNet convolutional neural network for training, and an image clarity detection model based on the LeNet convolutional neural network is obtained;

[0015] The image data is input into the image clarity detection model based on the LeNet convolutional neural network to obtain the clarity value corresponding to the image data.

[0016] The present invention normalizes the established target sample image data set to speed up the training speed and improve the accuracy of the training model, and inputs the normalized data set into a pre-built LeNet convolutional neural network for training, so as to use the trained image clarity detection model based on the LeNet convolutional neural network to perform clarity detection on the image data. The LeNet convolutional neural network can perform clarity detection without interference under complex image backgrounds.

[0017] In an optional implementation, the normalized data set is input into a pre-built LeNet convolutional neural network for training to obtain an image clarity detection model based on the LeNet convolutional neural network, including:

[0018] Pre-build the LeNet convolutional neural network. The structure of the LeNet convolutional neural network model includes an input layer, a first layer, a second layer, a third layer, a fourth layer, a fifth layer, a sixth layer, and an output layer. The first layer and the third layer are convolutional layers, the second layer and the fourth layer are pooling layers, and the fifth layer and the sixth layer are fully connected layers.

[0019] Input the normalized data set into the LeNet convolutional neural network for training;

[0020] The cross entropy loss function is used to calculate the output layer error in the LeNet convolutional neural network, and the weight parameters of the convolution kernel and the fully connected layer are updated through the back propagation algorithm until the training of the LeNet convolutional neural network is stopped when the preset conditions are met. Finally, an image clarity detection model based on the LeNet convolutional neural network is obtained.

[0021] The present invention adopts the classic LeNet convolutional neural network. The convolution layer and pooling layer in the LeNet convolutional neural network can enable the LeNet convolutional neural network to recognize and abstract image structure patterns at different levels, which is crucial for image clarity detection. The cross entropy loss function is used to calculate the output layer error in the LeNet convolutional neural network, and the weight parameters of the convolution kernel and the fully connected layer are updated through the back propagation algorithm to optimize the performance of the LeNet convolutional neural network, so that the trained LeNet convolutional neural network can have high accuracy and strong generalization ability.

[0022] In an optional implementation, the image data is input into an image clarity detection model based on a LeNet convolutional neural network to obtain a clarity value corresponding to the image data, including:

[0023] Grayscale the image data and select the Sobel operator to calculate the gradient of the image data in the horizontal and vertical directions;

[0024] Use the selected Sobel operator to perform convolution operation on the grayscale processed image data, and calculate the grayscale gradient of each pixel in the horizontal and vertical directions respectively;

[0025] The whole image is traversed to calculate the sum of squares of grayscale gradients within the threshold range, and the calculated sum of squares is determined as the clarity value corresponding to the image data.

[0026] The present invention performs grayscale processing on image data, simplifies data processing, reduces the amount of data, and uses the Sobel operator to calculate grayscale gradients in the horizontal and vertical directions to capture edge information in different directions of the image data. The present invention is suitable for performing clarity detection on images with complex structures, where the clarity difference may be caused by edge changes. The square sum of the grayscale gradients of the entire image within a threshold range is calculated, and the Sobel operator is used to suppress the influence of noise, so as to intuitively reflect the clarity value of the entire image.

[0027] In an optional implementation, judging whether the current detection area is a missed detection area according to the clarity value of the image data and the movement acceleration value of the scope includes:

[0028] If the clarity value of the image data is less than a preset clarity threshold, and / or the movement acceleration value of the scope is greater than a preset acceleration value, then the current detection area is determined to be a missed detection area;

[0029] If the clarity value of the image data is greater than a preset clarity threshold, or the movement acceleration value of the mirror body is less than a preset acceleration value, the current detection area is determined to be a non-missed detection area.

[0030] The present invention comprehensively judges whether the current detection area is a missed detection area according to the size relationship between the clarity value of the image data and the preset clarity threshold, and the size relationship between the movement acceleration value of the mirror body and the preset acceleration value, thereby improving the accuracy of the judgment result.

[0031] In an optional implementation, if the current detection area is a missed detection area, missed detection prompt information is generated, including:

[0032] If the current detection area is a missed detection area, the depth information of the current scope is determined, and the depth information of the current scope is marked in the display screen.

[0033] The present invention determines the depth information of the current scope when determining that the current detection area is a missed detection area, so as to determine the depth of the scope's penetration, mark the depth information of the current scope in the display screen, prompt the doctor of missed detection, and remind the doctor of the depth, so that the doctor can re-examine.

[0034] In an optional implementation, if the current detection area is a missed detection area, after generating missed detection prompt information, the method further includes:

[0035] When re-inspection of missed inspection areas is detected, the missed inspection prompt information is deleted.

[0036] The present invention deletes the missed inspection prompt information when it is detected that the missed inspection area needs to be re-inspected, so as to avoid the waste of resources caused by the doctor's second re-inspection.

[0037] In a second aspect, the present invention provides a missed detection prompting device based on a convolutional neural network, the device comprising:

[0038] A clarity detection module is used to obtain image data of the current detection area captured by the camera device during the endoscope detection process, and perform clarity detection on the image data based on the LeNet convolutional neural network to obtain a clarity value of the image data;

[0039] An acceleration calculation module is used to measure the moving acceleration value of the camera lens using a fiber grating acceleration sensor;

[0040] A judgment module, used to judge whether the current detection area is a missed detection area according to the clarity value of the image data and the moving acceleration value of the mirror body;

[0041] The prompt module is used to provide missed detection prompt information if the current detection area is a missed detection area.

[0042] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the missed detection prompt method based on a convolutional neural network of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the missed detection prompt method based on a convolutional neural network according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a missed detection prompting method based on a convolutional neural network according to an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of an application of a missed detection prompting method based on a convolutional neural network according to an embodiment of the present invention;

[0047] Figure 3 is a structural block diagram of a missed detection prompting device based on a convolutional neural network according to an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] According to an embodiment of the present invention, an embodiment of a missed detection prompting method based on a convolutional neural network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] In this embodiment, a missed detection prompt method based on a convolutional neural network is provided, which can be used in a mobile terminal. Figure 1 is a flow chart of a missed detection prompting method based on a convolutional neural network according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0052] Step S101, obtaining image data of the current detection area captured by the camera device during the endoscope detection process, and performing clarity detection on the image data based on the LeNet convolutional neural network to obtain a clarity value of the image data.

[0053] In the embodiment of the present invention, during the detection process using the endoscope, the camera device collects image signals and obtains image data of the current detection area captured by the camera device. The image data is detected for clarity based on the LeNet convolutional neural network. The convolutional neural network used in this embodiment replaces the activation function sigmoid in the traditional LeNet convolutional neural network model with the activation function ELU to avoid the problems of gradient disappearance and excessive calculation. At the same time, the activation function ELU is used to make the network output distribution zero-mean, and its training is faster. Here, the depth of the two convolutional layers is changed to 16 and 32 to increase the number of feature parameters and further improve the accuracy and generalization ability of the model.

[0054] Step S102, using a fiber grating acceleration sensor to measure the moving acceleration value of the camera lens.

[0055] In the embodiment of the present invention, the moving acceleration value of the camera lens is measured by using a fiber Bragg grating acceleration sensor. The fiber Bragg grating acceleration sensor is an inertial sensor. The principle is that when external vibration occurs, the acceleration excitation signal of the vibration will act on the inertial mass block, causing it to vibrate accordingly, thereby causing the fiber Bragg grating acceleration sensor to expand or bend, causing the center wavelength of the fiber Bragg grating acceleration sensor to drift. The fiber Bragg grating demodulator can be used to demodulate the real-time center wavelength of the fiber Bragg grating acceleration sensor, and then calculate the acceleration value of the external vibration. Ignoring the environmental interference during the actual inspection process, the external vibration acceleration is approximately equal to the moving acceleration value of the lens.

[0056] Step S103, judging whether the current detection area is a missed detection area according to the clarity value of the image data and the moving acceleration value of the scope.

[0057] In the embodiment of the present invention, the clarity value of the image data and the moving acceleration value of the scope are used to determine whether the current detection area is a missed detection area, that is, the image display is not clear and the scope moves too fast, resulting in missed detection of the detection area. Therefore, the clarity value of the image data and the moving acceleration value of the scope can effectively determine whether the detection area is missed.

[0058] Step S104: if the current detection area is a missed detection area, then generating missed detection prompt information.

[0059] In an embodiment of the present invention, if the current detection area is determined to be a missed detection area, missed detection prompt information is generated and displayed on a display device to prompt the doctor to check the missed detection area, thereby improving the detection rate of lesions.

[0060] The missed detection prompt method based on convolutional neural network provided in this embodiment performs clarity detection on image data based on LeNet convolutional neural network, and uses fiber grating accelerometer to measure the moving acceleration value of the camera lens, and judges whether the image is displayed clearly and whether the lens moves too fast according to the clarity value and the moving acceleration value, so as to judge whether the current detection area is a missed detection area, and generates missed detection prompt information when it is determined to be a missed detection area, reminding the doctor to re-examine the missed detection area, thereby improving the detection rate of lesions.

[0061] In this embodiment, a missed detection prompt method based on a convolutional neural network is provided, which can be used in the above-mentioned mobile terminal. The process includes the following steps:

[0062] Step S201, obtaining image data of the current detection area captured by the camera device during the endoscope detection process, and performing clarity detection on the image data based on the LeNet convolutional neural network to obtain a clarity value of the image data.

[0063] Specifically, in the above step S201, the clarity value of the image data obtained by performing clarity detection on the image data based on the LeNet convolutional neural network includes:

[0064] Step S2011, collecting target sample image data, and establishing a data set according to the collected target sample image data.

[0065] Step S2012, normalizing the image pixel values ​​in the data set.

[0066] Step S2013, inputting the normalized data set into a pre-built LeNet convolutional neural network for training, thereby obtaining an image clarity detection model based on the LeNet convolutional neural network.

[0067] Step S2014: input the image data into an image clarity detection model based on the LeNet convolutional neural network to obtain a clarity value corresponding to the image data.

[0068] In an embodiment of the present invention, human target sample image data is collected, and intestinal sample images are taken as an example, and the intestinal sample image data is classified and labeled according to its position information in the intestine to form a data set. It can be understood that the target sample image data can be intestinal sample image data or other sample image data, which is only an example and not a limitation.

[0069] Normalize the image pixel values ​​in the dataset to speed up training and improve the accuracy of the model.

[0070] Specifically, the calculation formula for pixel normalization is as follows:

[0071] Normalized pixel value = (original pixel value - minimum pixel value) / (maximum pixel value - minimum pixel value)

[0072] The normalized data set is input into the pre-built LeNet convolutional neural network for training to obtain an image clarity detection model based on the LeNet convolutional neural network. Taking intestinal detection as an example, when performing endoscopic intestinal detection on patients, the intestinal image data of the patients is input into the image clarity detection model based on the LeNet convolutional neural network to obtain the image clarity detection result, that is, the clarity value corresponding to the image data.

[0073] The established sample image data set is normalized to speed up the training speed and improve the accuracy of the training model. The normalized data set is input into the pre-built LeNet convolutional neural network for training, so that the image data can be analyzed for clarity using the trained image clarity detection model based on the LeNet convolutional neural network. The LeNet convolutional neural network can perform clarity detection without interference under complex image backgrounds.

[0074] Specifically, the above step S2013 includes:

[0075] Step S20131, pre-build the LeNet convolutional neural network.

[0076] Step S20132, input the normalized data set into the LeNet convolutional neural network for training.

[0077] Step S20133, using the cross entropy loss function to calculate the output layer error in the LeNet convolutional neural network, updating the weight parameters of the convolution kernel and the fully connected layer through the back propagation algorithm until the preset conditions are met, stopping the training of the LeNet convolutional neural network, and obtaining an image clarity detection model based on the LeNet convolutional neural network.

[0078] In an embodiment of the present invention, the structure of the LeNet convolutional neural network model includes an input layer, a first layer, a second layer, a third layer, a fourth layer, a fifth layer, a sixth layer and an output layer, the first layer and the third layer are convolutional layers, the second layer and the fourth layer are pooling layers, and the fifth layer and the sixth layer are fully connected layers. Specifically, the structure of the LeNet convolutional neural network model is as follows: the input layer is a 1920×1080 grayscale image; the first convolution layer uses a 21×21 convolution kernel, a step size of 1, and an activation function of ELU, and the image size changes from 1920×1080×1 to 1900×1060×16; the second pooling layer uses a median pooling method, the pooling kernel size is 4*4, and the step size is 4; the third convolution layer uses a 11*11 convolution kernel, a step size of 1, and a convolution depth of The number of neurons is 32, the activation function is ELU, and the image size changes from 475*265*16 to 465*255*32; the fourth pooling layer uses the maximum pooling method, the pooling kernel size is 5*5, the output image size is 93*51*32, and the step size is 5; the fifth fully connected layer outputs 120 neurons, and the activation function is ELU; the sixth fully connected layer outputs 84 neurons, and the activation function is ELU; the output layer has 10 neurons, corresponding to 10 different types of results. During the training process, the cross entropy loss function is used to calculate the output layer error, and the weight parameters of the convolution kernel and the fully connected layer are updated through the back propagation algorithm. Iterative training is performed until the preset number of iterations is reached, or the model performance is no longer improved, or the loss value is lower than the preset threshold, the training of the LeNet convolutional neural network is stopped, and the image clarity detection model based on the LeNet convolutional neural network is obtained.

[0079] The classic LeNet convolutional neural network is adopted. The convolutional layers and pooling layers in the LeNet convolutional neural network enable the LeNet convolutional neural network to recognize and abstract image structure patterns at different levels, which is crucial for image clarity detection. The cross entropy loss function is used to calculate the output layer error in the LeNet convolutional neural network. The weight parameters of the convolution kernel and the fully connected layer are updated through the back propagation algorithm to optimize the performance of the LeNet convolutional neural network, which can make the trained LeNet convolutional neural network have high accuracy and strong generalization ability.

[0080] Specifically, the above step S2014 includes:

[0081] Step S20141, grayscale the image data, and select the Sobel operator to calculate the gradient of the image data in the horizontal and vertical directions.

[0082] Step S20142, using the selected Sobel operator to perform a convolution operation on the grayscale processed image data, and calculate the grayscale gradient of each pixel in the horizontal and vertical directions respectively.

[0083] Step S20143, traverse the entire image to calculate the sum of squares of grayscale gradients within the threshold range, and determine the calculated sum of squares as the clarity value corresponding to the image data.

[0084] In the embodiment of the present invention, the Tenengrad gradient method is mainly used to calculate the clarity of the image. First, the image data is grayed. In the second step, the Sobel operator is selected to calculate the gradient of the image data in the horizontal and vertical directions. In the third step, the convolution operation is performed on the grayed image data using the selected Sobel operator to calculate the grayscale gradient of each pixel in the horizontal and vertical directions respectively. These gradient values ​​represent the strength and direction of the edge in the image.

[0085] The grayscale gradient calculation method is:

[0086] 1. Convolve the horizontal Sobel operator with the image to obtain the horizontal gradient value of each pixel (G x ).

[0087] 2. Convolve the vertical Sobel operator with the image to obtain the vertical gradient value of each pixel (G y ).

[0088] 3. The gray gradient value calculation formula is:

[0089]

[0090] 4. Traverse the entire image and calculate the sum of the squares of the grayscale gradient within the threshold range T as the clarity evaluation value F tenengrad :

[0091]

[0092] By graying the image data, simplifying data processing and reducing the amount of data, the Sobel operator is used to calculate the grayscale gradient in the horizontal and vertical directions to capture the edge information of the image data in different directions. It is suitable for clarity detection of images with complex structures, where the clarity difference may be caused by edge changes. The sum of the squares of the grayscale gradients of the entire image within the threshold range is calculated, and the Sobel operator is used to suppress the influence of noise to intuitively reflect the clarity value of the entire image.

[0093] Step S202: Using a fiber grating acceleration sensor to measure the moving acceleration value of the camera lens.

[0094] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0095] Step S203, judging whether the current detection area is a missed detection area according to the clarity value of the image data and the moving acceleration value of the scope.

[0096] Specifically, the above step S203 includes:

[0097] Step S2031: If the clarity value of the image data is less than a preset clarity threshold, and / or the movement acceleration value of the scope is greater than a preset acceleration value, it is determined that the current detection area is a missed detection area.

[0098] Step S2032: If the clarity value of the image data is greater than a preset clarity threshold, or the movement acceleration value of the scope is less than a preset acceleration value, then the current detection area is determined to be a non-missed detection area.

[0099] In the embodiment of the present invention, the clarity threshold and the scope movement acceleration value are preset, that is, when the clarity value of the image data is less than the preset clarity threshold, the image display is not clear, and when the scope movement acceleration value is less than or greater than the preset acceleration value, the scope moves too fast, and both are missed detections. Therefore, when the clarity value of the image data is less than the preset clarity threshold, or the scope movement acceleration value is greater than the preset acceleration value, or the clarity value of the image data is less than the preset clarity threshold, and the scope movement acceleration value is greater than the preset acceleration value, the current detection area is determined to be a missed detection area.

[0100] If the clarity value of the image data is greater than the preset clarity threshold, or the movement acceleration value of the mirror is less than the preset acceleration value, it means that the image display is clear and the movement speed of the mirror is moderate, then the current detection area is determined to be a non-missed detection area.

[0101] By comprehensively judging whether the current detection area is a missed detection area based on the size relationship between the clarity value of the image data and the preset clarity threshold, and the size relationship between the movement acceleration value of the mirror and the preset acceleration value, the accuracy of the judgment result is improved.

[0102] Step S204: if the current detection area is a missed detection area, then generating missed detection prompt information.

[0103] Specifically, the above step S204 includes:

[0104] Step S2041: If the current detection area is a missed detection area, the depth information of the current scope is determined, and the depth information of the current scope is marked in the display screen.

[0105] In an embodiment of the present invention, after determining that the current detection area is a missed detection area by combining the image clarity value and the scope movement acceleration value, the depth information of the current scope, that is, the insertion amount of the scope, is calculated through depth perception, and the depth information of the current scope is marked on the display screen to prompt the doctor.

[0106] When the current detection area is determined to be a missed detection area, the depth information of the current scope is determined to determine the depth of the scope's penetration, and the depth information of the current scope is marked on the display screen to remind the doctor of missed detection and the depth, so that the doctor can re-examine.

[0107] Step S205, when it is detected that the missed inspection area is re-inspected, the missed inspection prompt information is deleted.

[0108] In the embodiment of the present invention, when it is detected that the doctor re-examines the missed inspection area, the missed inspection prompt message is deleted and the missed inspection prompt information disappears.

[0109] The missed detection prompt method based on convolutional neural network provided in this embodiment deletes the missed detection prompt information when it is detected that the missed detection area is re-examined, so as to avoid the waste of resources caused by the doctor's second re-examination.

[0110] like Figure 2 As shown, the missed detection prompting method based on convolutional neural network includes the following steps:

[0111] Step S1: Obtain a camera signal from a camera device and calculate the current image clarity.

[0112] Step S2: Calculate the current movement amount, movement speed, rotation angle, and rotation speed according to the acceleration sensor.

[0113] Step S3: judging whether the current area is a missed detection area according to the acceleration value and the clarity value of the current picture.

[0114] Step S4: After confirming that the current area is a missed detection area, mark the current depth information.

[0115] Step S5: The missed inspection depth information is displayed on the screen to remind the doctor to re-examine the missed inspection area.

[0116] In this embodiment, a missed detection prompting device based on a convolutional neural network is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0117] This embodiment provides a missed detection prompting device based on a convolutional neural network, such as Figure 3 As shown, including:

[0118] The clarity detection module 301 is used to obtain image data of the current detection area captured by the camera device during the endoscope detection process, and perform clarity detection on the image data based on the LeNet convolutional neural network to obtain a clarity value of the image data.

[0119] The acceleration calculation module 302 is used to measure the moving acceleration value of the camera lens using a fiber grating acceleration sensor.

[0120] The judgment module 303 is used to judge whether the current detection area is a missed detection area according to the clarity value of the image data and the movement acceleration value of the scope.

[0121] The prompt module 304 is used to generate missed detection prompt information if the current detection area is a missed detection area.

[0122] In some optional implementations, the clarity detection module 301 includes:

[0123] The data collection unit is used to collect target sample image data and establish a data set according to the collected target sample image data.

[0124] The normalization processing unit is used to perform normalization processing on the image pixel values ​​in the data set.

[0125] The module training unit is used to input the normalized data set into the pre-built LeNet convolutional neural network for training, so as to obtain an image clarity detection model based on the LeNet convolutional neural network.

[0126] The clarity detection unit is used to input the image data into the image clarity detection model based on the LeNet convolutional neural network to obtain the clarity value corresponding to the image data.

[0127] In some optional embodiments, the module training unit includes:

[0128] The construction subunit is used to pre-construct the LeNet convolutional neural network. The structure of the LeNet convolutional neural network model includes an input layer, a first layer, a second layer, a third layer, a fourth layer, a fifth layer, a sixth layer and an output layer, wherein the first layer and the third layer are convolutional layers, the second layer and the fourth layer are pooling layers, and the fifth layer and the sixth layer are fully connected layers.

[0129] The training subunit is used to input the normalized data set into the LeNet convolutional neural network for training.

[0130] The updating subunit is used to calculate the output layer error in the LeNet convolutional neural network using the cross entropy loss function, and to update the weight parameters of the convolution kernel and the fully connected layer through the back propagation algorithm until the training of the LeNet convolutional neural network is stopped when the preset conditions are met, thereby obtaining an image clarity detection model based on the LeNet convolutional neural network.

[0131] In some optional embodiments, the clarity detection unit includes:

[0132] The grayscale processing subunit is used to perform grayscale processing on the image data and select the Sobel operator to calculate the gradient of the image data in the horizontal and vertical directions.

[0133] The gradient calculation subunit is used to perform a convolution operation on the grayscale processed image data using the selected Sobel operator, and calculate the grayscale gradient of each pixel in the horizontal and vertical directions respectively.

[0134] The clarity calculation subunit is used to traverse the entire image to calculate the sum of squares of grayscale gradients within a threshold range, and determine the calculated sum of squares as a clarity value corresponding to the image data.

[0135] In some optional implementations, the determination module 303 includes:

[0136] The first determination unit is used to determine that the current detection area is a missed detection area if the clarity value of the image data is less than a preset clarity threshold, and / or the movement acceleration value of the mirror body is greater than a preset acceleration value.

[0137] The second determination unit is used to determine that the current detection area is a non-missed detection area if the clarity value of the image data is greater than a preset clarity threshold, or the movement acceleration value of the mirror body is less than a preset acceleration value.

[0138] In some optional implementations, the prompt module 304 includes:

[0139] The prompt unit is used to determine the depth information of the current scope if the current detection area is a missed detection area, and mark the depth information of the current scope in the display screen.

[0140] In some optional embodiments, the device further comprises:

[0141] The deletion module is used to delete the missed detection prompt information when it is detected that the missed detection area is re-inspected.

[0142] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0143] The missed detection prompting device based on convolutional neural network in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0144] The embodiment of the present invention also provides a computer device having the above Figure 3 The missed detection prompt device based on convolutional neural network is shown.

[0145] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0146] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0147] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0148] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0150] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0151] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, etc. The output device 40 can include a display device, etc.

[0152] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0153] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope of the present application.

Claims

1. A missed detection prompting method based on convolutional neural network, characterized in that: The method comprises: Acquire image data of the current detection area captured by the camera device during the endoscope detection process, and perform clarity detection on the image data based on the LeNet convolutional neural network to obtain a clarity value of the image data; Using a fiber grating acceleration sensor to measure the moving acceleration value of the camera lens; According to the clarity value of the image data and the moving acceleration value of the mirror body, determining whether the current detection area is a missed detection area; If the current detection area is a missed detection area, a missed detection prompt message is generated.

2. The missed detection prompting method based on convolutional neural network according to claim 1 is characterized in that: The performing definition detection on the image data based on the LeNet convolutional neural network to obtain a definition value of the image data includes: Collecting target sample image data, and establishing a data set based on the collected target sample image data; Normalizing the image pixel values ​​in the data set; Inputting the normalized data set into a pre-built LeNet convolutional neural network for training to obtain an image clarity detection model based on the LeNet convolutional neural network; The image data is input into the image clarity detection model based on the LeNet convolutional neural network to obtain a clarity value corresponding to the image data.

3. The missed detection prompting method based on convolutional neural network according to claim 2 is characterized in that: The normalized data set is input into a pre-built LeNet convolutional neural network for training to obtain an image clarity detection model based on the LeNet convolutional neural network, including: Pre-constructing a LeNet convolutional neural network, the structure of the LeNet convolutional neural network model includes an input layer, a first layer, a second layer, a third layer, a fourth layer, a fifth layer, a sixth layer and an output layer, wherein the first layer and the third layer are convolutional layers, the second layer and the fourth layer are pooling layers, and the fifth layer and the sixth layer are fully connected layers; Inputting the normalized data set into the LeNet convolutional neural network for training; The cross entropy loss function is used to calculate the output layer error in the LeNet convolutional neural network, and the weight parameters of the convolution kernel and the fully connected layer are updated through the back propagation algorithm. The training of the LeNet convolutional neural network is stopped until the preset conditions are met, and an image clarity detection model based on the LeNet convolutional neural network is obtained.

4. The missed detection prompting method based on convolutional neural network according to claim 2, characterized in that: The step of inputting the image data into the image clarity detection model based on the LeNet convolutional neural network to obtain a clarity value corresponding to the image data includes: Gray-scale the image data, and use a Sobel operator to calculate the gradient of the image data in the horizontal and vertical directions; Use the selected Sobel operator to perform convolution operation on the grayscale processed image data, and calculate the grayscale gradient of each pixel in the horizontal and vertical directions respectively; The whole image is traversed to calculate the sum of squares of grayscale gradients within the threshold range, and the calculated sum of squares is determined as the clarity value corresponding to the image data.

5. The missed detection prompting method based on convolutional neural network according to claim 1, characterized in that: The step of judging whether the current detection area is a missed detection area according to the clarity value of the image data and the movement acceleration value of the scope includes: If the clarity value of the image data is less than a preset clarity threshold, and / or the movement acceleration value of the scope is greater than a preset acceleration value, then the current detection area is determined to be a missed detection area; If the clarity value of the image data is greater than a preset clarity threshold, or the movement acceleration value of the mirror body is less than a preset acceleration value, it is determined that the current detection area is a non-missed detection area.

6. The missed detection prompting method based on convolutional neural network according to claim 1, characterized in that: If the current detection area is a missed detection area, generating missed detection prompt information includes: If the current detection area is a missed detection area, the depth information of the current scope is determined, and the depth information of the current scope is marked in the display screen.

7. The missed detection prompting method based on convolutional neural network according to claim 1, characterized in that: If the current detection area is a missed detection area, after generating missed detection prompt information, the method further includes: When it is detected that the missed inspection area is re-inspected, the missed inspection prompt information is deleted.

8. A missed detection prompting device based on a convolutional neural network, characterized in that: The device comprises: A clarity detection module is used to obtain image data of the current detection area captured by the camera device during the endoscope detection process, and perform clarity detection on the image data based on the LeNet convolutional neural network to obtain a clarity value of the image data; An acceleration calculation module is used to measure the moving acceleration value of the camera lens using a fiber grating acceleration sensor; A judgment module, used for judging whether the current detection area is a missed detection area according to the clarity value of the image data and the moving acceleration value of the mirror body; The prompt module is used to generate missed detection prompt information if the current detection area is a missed detection area.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the missed detection prompting method based on a convolutional neural network as described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the missed detection prompt method based on a convolutional neural network as described in any one of claims 1 to 7.

Images