Endoscopic preoperative detection method based on neural network model, endoscope, medium

Abstract

The present application discloses an endoscopic preoperative detection method, an endoscope, and a medium based on a neural network model. The method includes: before the operation, using a target endoscope to collect a target image corresponding to a reference part; identifying a first target object in the target image and identifying a second target object in the reference image; wherein the reference image is obtained by collecting the reference part using a reference endoscope; determining the missing area of ​​the first target object relative to the second target object; and adjusting the target endoscope according to the missing area. Through the above method, the image acquisition quality in the subsequent operation can be improved and the number of adjustments made by the user in the subsequent operation can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of endoscopes, and in particular to an endoscope preoperative detection method, an endoscope, and a medium based on a neural network model. Background Art

[0002] During the use of the endoscope, as the number of uses increases, the hardware structure and corresponding circuits in the endoscope will wear out. In addition, the use of the endoscope actually requires the operator to have strong theoretical knowledge. During the use, the endoscope must be operated many times before the required images can be collected using the endoscope.

[0003] Therefore, it is necessary to test and adjust the endoscope before surgery. Summary of the invention

[0004] The endoscopic preoperative detection method, endoscope, and medium based on a neural network model provided in this application can improve the quality of image acquisition during subsequent operations.

[0005] A technical solution adopted in the present application is to provide an endoscopic preoperative detection method based on a neural network model, the method comprising: before the operation, using a target endoscope to collect a target image corresponding to a reference part; identifying a first target object in the target image and identifying a second target object in the reference image; wherein the reference image is obtained by collecting the reference part using a reference endoscope; determining the missing area of ​​the first target object relative to the second target object; and adjusting the target endoscope according to the missing area.

[0006] Among them, identifying the first target object in the target image and identifying the second target object in the reference image include: identifying the first outline of the first target object in the target image and identifying the second outline of the second target object in the reference image; determining the missing area of ​​the first target object relative to the second target object includes: determining the missing area of ​​the first outline relative to the second outline.

[0007] Wherein, determining the missing area of ​​the first contour relative to the second contour includes: moving the first contour to the second contour until the first contour is completely within the area of ​​the second contour, thereby determining the first missing area of ​​the first contour relative to the second contour; and / or inputting the first target object of the first contour and the second target object of the second contour into the neural network model, so that the neural network model performs pixel completion on the first target object of the first contour to obtain the first target object of the third contour; in response to the difference between the third contour and the second contour being less than a threshold, using the completed pixel area as the second missing area; adjusting the target endoscope according to the missing area includes: adjusting the target endoscope according to the first missing area and / or the second missing area.

[0008] Among them, adjusting the target endoscope according to the first missing area or the second missing area includes: dividing the first missing area or the second missing area into a plurality of missing sub-areas based on the reference image; counting the missing area corresponding to each missing sub-area; performing weighted processing on each missing area to obtain adjustment parameters of the target endoscope; and adjusting the target endoscope according to the adjustment parameters.

[0009] Wherein, each missing area is weighted to obtain the adjustment parameters of the target endoscope, including: sorting all missing areas from large to small, and assigning weights from large to small according to the sorting order.

[0010] Wherein, adjusting the target endoscope according to the first missing area and the second missing area includes: obtaining a first adjustment parameter of the target endoscope according to the first missing area; obtaining a second adjustment parameter of the target endoscope according to the second missing area; obtaining a target adjustment parameter according to the first adjustment parameter and the second adjustment parameter; and adjusting the target endoscope using the target adjustment parameter.

[0011] The neural network model includes a coding layer, a feature fusion layer and a decoding layer. The feature fusion layer is used to enhance and fuse the features of each level extracted by the coding layer and output them to the decoding layer. The coding layer is divided into a first type of coding layer and a second type of coding layer. The first target object of the first contour and the second target object of the second contour are input into the neural network model so that the neural network model performs pixel completion on the first target object of the first contour to obtain the first target object of the third contour, including: using the first type of coding layer to extract features of the first target object of the first contour to obtain the first feature map corresponding to each level, and inputting the first feature map into the attention mechanism layer; using the second type of coding layer to extract features of the second target object of the second contour to obtain the second feature map corresponding to each level, and inputting the second feature map into the attention mechanism layer; using the attention mechanism layer to extract features of the first feature map and the second feature map according to the corresponding level to obtain the third feature map corresponding to each level; inputting the third feature map corresponding to each level into the feature fusion layer for enhanced fusion to obtain the fourth feature map corresponding to each level; inputting the fourth feature map corresponding to each level into the decoding layer for pixel completion to obtain the first target object of the third contour.

[0012] Among them, the attention mechanism layer is used to extract features from the first feature map and the second feature map according to the corresponding levels to obtain the third feature map corresponding to each level, including: using the attention mechanism layer to perform a convolution operation on the first feature map to obtain a first target feature map, and inputting the first target feature map to the next encoding layer of the first type of encoding layer; using the attention mechanism layer to perform a convolution operation on the second feature map to obtain a second target feature map, and inputting the second target feature map to the next encoding layer of the second type of encoding layer; performing feature fusion on the first target feature map and the second target feature map to obtain the third feature map.

[0013] Another technical solution adopted in the present application is to provide an endoscope, which includes a host end, a photoelectric conversion module and an insertion structure; the host end, the photoelectric conversion module and the insertion structure cooperate to implement the method provided by the above technical solution.

[0014] Another technical solution adopted by the present application is to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the method provided by the above technical solution.

[0015] The beneficial effects of the present application are as follows: different from the prior art, the neural network model-based endoscopic preoperative detection method, endoscope, and medium of the present application use a target endoscope to collect a target image corresponding to a reference part before surgery; identify a first target object in the target image and identify a second target object in the reference image; wherein the reference image is obtained by collecting a reference part using a reference endoscope; determine the missing area of ​​the first target object relative to the second target object; and adjust the target endoscope according to the missing area so that the target endoscope reaches the optimal image acquisition state, thereby improving the image acquisition quality during subsequent surgery, and reducing the adjustment of the target endoscope by subsequent users during surgery, thereby improving image acquisition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1 It is a structural schematic diagram of an embodiment of an endoscope provided by the present application;

[0018] Figure 2 It is a structural schematic diagram of an embodiment of an optoelectronic link of an endoscope provided by the present application;

[0019] Figure 3It is a flowchart of an embodiment of an endoscopic preoperative detection method based on a neural network model provided by the present application;

[0020] Figure 4 It is a flowchart of another embodiment of the endoscopic preoperative detection method based on the neural network model provided by the present application;

[0021] Figure 5 It is a schematic diagram of the internal processing flow of the neural network model provided by this application;

[0022] Figure 6 It is a flowchart of another embodiment of the endoscopic preoperative detection method based on the neural network model provided by the present application;

[0023] Figure 7 It is a flowchart of another embodiment of the endoscopic preoperative detection method based on the neural network model provided by the present application;

[0024] Figure 8 is a schematic diagram of comparison between a reference image and a target image provided by the present application;

[0025] Fig. 9 It is a flowchart of another embodiment of the endoscopic preoperative detection method based on the neural network model provided by the present application;

[0026] Fig.10 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0028] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] During the use of the endoscope, as the number of uses increases, the hardware structure and corresponding circuits in the endoscope will wear out. In addition, the use of the endoscope actually requires the operator to have strong theoretical knowledge. During the use, the endoscope must be operated many times before the required images can be collected using the endoscope.

[0030] Therefore, it is necessary to test and adjust the endoscope before surgery.

[0031] Based on this, the present application proposes to use a target endoscope to collect a target image corresponding to a reference part before surgery; identify a first target object in the target image and identify a second target object in the reference image; wherein the reference image is obtained by collecting the reference part using a reference endoscope; determine the missing area of ​​the first target object relative to the second target object; and adjust the target endoscope according to the missing area so that the target endoscope reaches the optimal image acquisition state, thereby improving the image acquisition quality during subsequent surgery, and reducing the adjustment of the target endoscope by subsequent users during surgery, thereby improving image acquisition efficiency.

[0032] See also Figure 1 , Figure 1 1 is a schematic diagram of the structure of an embodiment of an endoscope provided by the present application. The endoscope comprises: a host end, a photoelectric conversion module and an insertion structure. The host end, the photoelectric conversion module and the insertion structure cooperate to realize the technical solution formed by any one of the following embodiments or any combination of the embodiments.

[0033] The host side can be provided with an operation screen, a USB interface, a scanning terminal, a medical caster, a switch button, a camera handle bracket, a camera handle interface, and an LED indicator. Among them, the operation screen is used to display functions and provide an operation interface. The USB interface is used for external USB. The scanning terminal is used for scanning code recognition. The medical caster is used to move the host side. The switch button is used to turn the host side on and off. The camera handle bracket is used to hang the camera handle. The camera handle interface is used to connect the camera handle. The LED indicator is used to indicate the working status of the host side. If the indicator is on, the trolley is working; if the indicator is off, the trolley is not working.

[0034] Specifically, the entire system of the endoscope is as follows:

[0035] The camera module uses the FPGA board as the central control center to connect the barcode scanner, camera handle (CMOS-ISP board), camera serial port screen, video recording board and light source driver board to complete the barcode analysis, imaging parameter control, camera serial port screen control, video recording control and interaction with the light source driver board. Users can use the camera serial port screen to take photos, record, set the system time, set the language and restore factory settings. The light source module uses the light source driver board as the central control center to connect the light source serial port screen, temperature sensor, light source, FPGA board and start switch to adjust the light source brightness, monitor the light source temperature and start the light source driver board to power each module. The light generated by the light source module provides a light guide connected to the camera handle for clear imaging.

[0036] The corresponding software functions are as follows:

[0037] The functions included in the light source software are: power on, bulb switch function, brightness adjustment function, temperature display function, and running time function.

[0038] Light bulb switch function: The software can turn the light bulb on or off.

[0039] Brightness adjustment function: The software can adjust the brightness of the bulb from 10 to 100, with each level changing by 10.

[0040] Temperature display function: The software interface displays the current temperature of the bulb.

[0041] Running time function: The software interface displays the actual cumulative running time of the light source. When the lamp runs for more than 20,000 hours, a prompt box will appear.

[0042] The functions included in the camera system software are: power on, user access control, lock screen function, system setting function (including system time setting, language setting, and restoring factory settings), video recording function, photo taking function, and one-key enhancement function.

[0043] User access control: When the device is turned on, the boot UI is displayed and a progress bar is loaded. After the progress bar is loaded, the display screen jumps to the code scanning UI. Scan the QR code on the sheath to display the image.

[0044] Lock screen function: The camera system host can lock the operation interface to prevent accidental touches.

[0045] Product system setting function:

[0046] 1 System time setting: You can set the system time in 24-hour format.

[0047] 2 Language setting: You can set the language, supporting Chinese and English.

[0048] 3 Restore factory settings: Restore factory settings.

[0049] Video recording function: insert the USB storage medium, the product has the video recording function, and can be stored in real time through USB, the storage format is .ts.

[0050] Photo function: insert the USB storage medium, the product has the photo function, which can be stored via USB, and the storage format is JPG.

[0051] One-key enhancement function: After pressing the button, the image is adjusted to the preset value. For the brightness of the picture, select one-key enhancement to adjust the overall brightness of the picture.

[0052] Camera system software, including power-on, user access control, screen lock, photo taking, video recording and one-key enhancement functions.

[0053] The entire image processing link includes: sensor board, cable, sensor adapter board, FPGA (Field Programmable Gate Array) mainboard and display.

[0054] Sensor board: uses a chip to convert the light signal irradiated on the sensor into an electrical signal, and outputs the electrical signal through the rear-end cable.

[0055] Cable: Transmits electrical signals and provides a power path for the Sensor board.

[0056] Sensor adapter board: The core of the sensor adapter board is an FPGA chip. Its main function is to sample and convert the electrical signals transmitted by the sensor board into serial signals, and convert the serialized data into optical signals through the optoelectronic conversion module, and transmit the signals to the host through optical fiber.

[0057] FPGA main board: Receives the optical signal transmitted from the sensor adapter board and converts it into an electrical signal through the photoelectric conversion module. It then processes the original image into a target image through a series of image ISP algorithms and reconstructs the image to output in a specified format.

[0058] Display: Display the processed image.

[0059] like Figure 2As shown, the camera handle includes a photoelectric conversion module. The light source exciter in the host end provides an incident light source, which is connected to the inside of the handle from the handle tail end 60 through the light source optical fiber on the host end, and then extends along the optical fiber 40 inside the handle to the light source incident end 50, and vertically incident on the photoelectric conversion module 20 (handle fluorescence module) upward, and is incident on the surface of the living tissue at the right after a 90° total reflection through the dichroic mirror set at a 45° angle inside the handle fluorescence module. After the fluorescence excitation on the surface of the living tissue, the reflected light is formed, and the reflected light returns to the dichroic mirror along the original path. At this time, due to the excitation change in the band, the reflected light is all transmitted through the dichroic mirror and then transmitted to the rear-end photoelectric conversion module. After being received by the photoelectric conversion module, the optical signal is converted into an electrical signal. In order to prevent electromagnetic interference, the handle will perform electro-optical conversion again, and the electrical signal will be converted into an optical signal again. The optical signal is transmitted to the image signal processor in the machine through the return optical fiber signal line, and then the photoelectric conversion is performed for the second time to convert the returned optical signal into an electrical signal. Then, through the processing of the electrical signal, a microscopic image magnified more than 1000 times is output on the display.

[0060] See also Figure 3 , Figure 3 1 is a flow chart of an embodiment of an endoscopic preoperative detection method based on a neural network model provided by the present application. The method comprises:

[0061] Step 31: Before the operation, a target image corresponding to the reference part is collected using a target endoscope.

[0062] During the use of the endoscope, as the number of uses increases, the hardware structure and corresponding circuits in the endoscope will wear out. In addition, the use of the endoscope actually requires the operator to have strong theoretical knowledge. During the use, the endoscope must be operated many times before the required images can be collected using the endoscope.

[0063] Based on this, before the operation, the endoscope can be tested and adjusted to ensure that the endoscope is normal and adjusted to the best state for easy use by the user. After the endoscope is adjusted to the best state, the user can reduce the adjustment of the endoscope and improve the subsequent use efficiency.

[0064] In some embodiments, a real endoscopic environment may be simulated, and in the simulated endoscopic environment, a target image corresponding to a reference part in the endoscopic environment may be acquired using a target endoscope.

[0065] In some embodiments, the endoscopic environment can be simulated according to the actual surgical scenario. For example, if a colonoscopy is required, the endoscopic environment of the intestine is simulated. For example, if a gastroscopy is required, the endoscopic environment of the stomach is simulated.

[0066] The reference part can be one part in the endoscopic environment or multiple parts in the endoscopic environment. That is, when the reference part is one part in the endoscopic environment, one target image is acquired. When the reference part is multiple parts in the endoscopic environment, a corresponding target image is acquired for each reference part using the target endoscope.

[0067] Step 32: Identify a first target object in the target image and identify a second target object in the reference image.

[0068] The reference image is obtained by collecting the reference part using a reference endoscope.

[0069] The reference endoscope is a standard endoscope. That is, the reference endoscope can be understood as an endoscope without any wear and tear and without any abnormality. In other words, the reference image collected by the reference endoscope should be a standard image.

[0070] In some embodiments, a first target object in a target image and a second target object in a reference image may be identified by a target detection model.

[0071] Step 33: Determine the missing area of ​​the first target object relative to the second target object.

[0072] After the first target object and the second target object are identified, the contours of the corresponding first target object and the second target object, that is, the missing area of ​​the first target object relative to the second target object can be determined through the contours.

[0073] Step 34: Adjust the target endoscope according to the missing area.

[0074] In some embodiments, the missing difference between the target object in the target image and the reference image may be caused by different parameters between endoscopes, such as light intensity, light emission angle, light collection angle, photoelectric conversion efficiency and / or light reflection angle.

[0075] Based on this, parameters related to light intensity, light emission angle, light collection angle, photoelectric conversion efficiency and / or light reflection angle, as well as hardware position in the target endoscope can be adjusted according to the missing area.

[0076] That is, the target endoscope is adjusted so that the target image captured by the target endoscope is infinitely close to the reference image.

[0077] In some embodiments, the parameters of the reference endoscope can be obtained and used directly in the target endoscope. Then, the target endoscope with the changed parameters is used to collect the target image of the reference part again, and the target image is used again to compare with the reference image to obtain the missing area of ​​the target object between the images. By directly using the parameters of the reference endoscope in the target endoscope, the target endoscope can be made close to the reference endoscope to a large extent. Then, when the images are compared again, the missing area of ​​the missing area of ​​the target object will be greatly reduced, so the target endoscope can be adjusted more quickly, thereby reducing the adjustment of the target endoscope and improving the adjustment efficiency.

[0078] In some embodiments, when the reference parts are multiple parts in the endoscopic environment, a corresponding target image is collected for each reference part using the target endoscope. Therefore, each reference part corresponds to a corresponding reference image. Further, by comparing the target image corresponding to each reference part with the reference image, the missing area between the target objects in the image is obtained, and then multiple missing areas can be obtained. Then the target endoscope is adjusted based on the multiple missing areas. For example, each missing area is quantified to obtain a corresponding quantization value. Then the median of these quantization values ​​is selected, and the target endoscope is adjusted according to the median. In some other embodiments, the missing area corresponding to each reference part can be weighted according to the positional relationship of the reference parts, and then the weighted comparison result is obtained. Then the target endoscope is adjusted according to the weighted missing area.

[0079] In some other embodiments, an average of the quantified values ​​may be calculated and the target endoscope may be adjusted based on the average.

[0080] In this embodiment, a target endoscope is used to capture a target image corresponding to a reference part before surgery; a first target object in the target image is identified and a second target object in the reference image is identified; wherein the reference image is obtained by capturing the reference part using a reference endoscope; a missing area of ​​the first target object relative to the second target object is determined; and based on the missing area, the target endoscope is adjusted so that the target endoscope reaches an optimal image capture state, thereby improving the image capture quality during subsequent surgery, reducing the adjustment of the target endoscope by subsequent users during surgery, and improving image capture efficiency.

[0081] See also Figure 4 , Figure 4 1 is a flow chart of another embodiment of the endoscopic preoperative detection method based on a neural network model provided by the present application. The method comprises:

[0082] Step 41: Before the operation, a target image corresponding to the reference part is collected using a target endoscope.

[0083] Step 42: Identify a first contour of a first target object in the target image, and identify a second contour of a second target object in the reference image.

[0084] Step 43: Determine the missing area of ​​the first contour relative to the second contour.

[0085] In some embodiments, the first contour is moved to the second contour until the first contour is completely within the region of the second contour, thereby determining a first missing region of the first contour relative to the second contour.

[0086] In some embodiments, a first target object of a first contour and a second target object of a second contour are input into a neural network model so that the neural network model performs pixel completion on the first target object of the first contour to obtain the first target object of a third contour; in response to a difference between the third contour and the second contour being less than a threshold, the completed pixel area is used as the second missing area.

[0087] The neural network model can be trained in advance using reference images.

[0088] Step 44: Adjust the target endoscope according to the missing area.

[0089] Take the first missing region mentioned above as an example to illustrate:

[0090] Based on the reference image, the first missing region is divided into a plurality of missing sub-regions; the missing area corresponding to each missing sub-region is counted; each missing area is weighted to obtain the adjustment parameters of the target endoscope; and the target endoscope is adjusted according to the adjustment parameters.

[0091] In some embodiments, the central point of the image may be used as a reference to establish horizontal and vertical coordinates, so as to divide the first missing region into four missing sub-regions, each of which corresponds to a quadrant.

[0092] Specifically, the weighting method can be as follows: sort all missing areas from large to small, and assign weights from large to small according to the sorting order. A large missing area means that the endoscope has not collected the area corresponding to the missing area, so a larger weight needs to be assigned so as to adjust the endoscope to the area as much as possible.

[0093] Take the second missing region mentioned above as an example for explanation:

[0094] Based on the reference image, the second missing region is divided into a plurality of missing sub-regions; the missing area corresponding to each missing sub-region is counted; each missing area is weighted to obtain the adjustment parameters of the target endoscope; and the target endoscope is adjusted according to the adjustment parameters.

[0095] Specifically, the weighting method may be as follows: sort all missing areas from large to small, and assign weights from large to small according to the sorting order.

[0096] In one application scenario, a first contour is moved to a second contour until the first contour is completely within the area of ​​the second contour, thereby determining a first missing area of ​​the first contour relative to the second contour; and a first target object of the first contour and a second target object of the second contour are input into a neural network model, so that the neural network model performs pixel completion on the first target object of the first contour to obtain the first target object of the third contour; in response to the difference between the third contour and the second contour being less than a threshold, the completed pixel area is used as the second missing area.

[0097] According to the first missing area, a first adjustment parameter of the target endoscope is obtained; according to the second missing area, a second adjustment parameter of the target endoscope is obtained; according to the first adjustment parameter and the second adjustment parameter, a target adjustment parameter is obtained; and the target endoscope is adjusted using the target adjustment parameter.

[0098] In this application scenario, the first adjustment parameter is used to assist in adjusting the second adjustment parameter, so as to correct the relevant errors of the neural network model and obtain more accurate adjustment parameters.

[0099] Combination Figure 5 , to illustrate the pixel completion of the neural network model:

[0100] Among them, the neural network model includes a coding layer, a feature fusion layer and a decoding layer. The feature fusion layer is used to enhance and fuse the features of each level extracted by the coding layer and output them to the decoding layer; the coding layer is divided into the first type of coding layer and the second type of coding layer.

[0101] Step 51: Use the first type of encoding layer to extract features of the first target object of the first contour, obtain the first feature map corresponding to each level, and input the first feature map into the attention mechanism layer.

[0102] Specifically, the first type of coding layer includes multiple coding layers, the first feature map output by each coding layer is used as the input of the attention mechanism layer, and the output of the attention mechanism layer is used as the input of the next coding layer in the first type of coding layer. The first type of coding layer extracts features from the key frame, and the extracted first feature map has the appearance information of the first target object.

[0103] In other embodiments, the first encoding layer in the first type of encoding layer and the second type of encoding layer is not connected to the attention mechanism layer, but is only connected to the next encoding layer.

[0104] Since the scale of each coding layer is different, first feature maps of different scales will be formed accordingly, which are represented by different levels here.

[0105] Step 52: Use the second type of encoding layer to extract features of the second target object of the second contour, obtain the second feature map corresponding to each level, and input the second feature map into the attention mechanism layer.

[0106] Specifically, the second type of coding layer includes multiple coding layers, the second feature map output by each coding layer is used as the input of the attention mechanism layer, and the output of the attention mechanism layer is used as the input of the next coding layer in the second type of coding layer. The second type of coding layer extracts features from the key frame, and the extracted second feature map has the appearance information of the second target object.

[0107] Since the scale of each coding layer is different, a second feature map of different scales will be formed accordingly, which is represented by different levels here.

[0108] In some embodiments, the number of encoding layers of the first type of encoding layer and the second type of encoding layer is the same, and the scales of the first type of encoding layer and the second type of encoding layer at the corresponding levels are the same, then the scales of the first feature map and the second feature map input to the attention mechanism layer are the same, which facilitates processing by the attention mechanism layer.

[0109] Step 53: Use the attention mechanism layer to extract features from the first feature map and the second feature map according to the corresponding levels to obtain the third feature map corresponding to each level.

[0110] In some embodiments, the number of attention mechanism layers is the same as the number of encoding layers of the first type of encoding layer, the input end of each attention mechanism layer is connected to an encoding layer of the first type of encoding layer and an encoding layer of the second type of encoding layer, and the output end of each attention mechanism layer is connected to an encoding layer of the first type of encoding layer, an encoding layer of the second type of encoding layer, and a feature fusion layer. The output end of the last attention mechanism layer is only connected to the feature fusion layer.

[0111] In some embodiments, the first feature map may be convolved using the attention mechanism layer to obtain a first target feature map, and the first target feature map may be input into the next encoding layer of the first type of encoding layer. The second feature map may be convolved using the attention mechanism layer to obtain a second target feature map, and the second target feature map may be input into the next encoding layer of the second type of encoding layer. The first target feature map and the second target feature map may then be feature fused to obtain a third feature map.

[0112] Step 54: Input the third feature maps corresponding to each level into the feature fusion layer for enhanced fusion to obtain the fourth feature maps corresponding to each level.

[0113] For example, the third feature map of the low level may be enhanced and fused with the third feature map of the high level, and the third feature map of the high level may be enhanced and fused with the third feature map of the low level.

[0114] In this way, the corresponding fourth feature map will have more feature information.

[0115] Step 55: Input the fourth feature map corresponding to each level into the decoding layer for pixel completion to obtain the first target object of the third contour.

[0116] Among them, this embodiment includes multiple decoding layers, each decoding layer is connected in sequence, and the output of each decoding layer is used as the input of the next decoding layer, and the fourth feature map corresponding to each level is used as the input of each decoding layer, that is, each decoding layer has two inputs, and the output of the last decoding layer is used as the first target object of the third contour.

[0117] In some embodiments, a convolution operation is performed on the fourth feature map corresponding to each level to obtain the first fourth reference feature map after the operation. The Nth fourth reference feature map and the fourth feature map are upsampled using the Nth decoding layer to obtain the N+1th fourth reference feature map. The N+1th fourth reference feature map is used as the first target object of the third contour obtained after pixel completion.

[0118] A 3×3 convolution is set before the decoding layer, and a convolution operation is performed on the fourth feature maps corresponding to each level to further integrate the fourth feature maps corresponding to each level to obtain the first fourth reference feature map after the operation.

[0119] See also Figure 6 , Figure 6 1 is a flow chart of another embodiment of the endoscopic preoperative detection method based on a neural network model provided by the present application. The method comprises:

[0120] Step 61: Before the operation, a target image corresponding to the reference part is collected using a target endoscope.

[0121] Step 62: Detect the image quality of the target image.

[0122] In some embodiments, the target image acquired by the target endoscope may not be comparable to the reference image due to some factors, such as poor imaging quality of the target image.

[0123] Therefore, it is necessary to give priority to solving the image quality problem so that the imaging problem of the actual reference part can be considered under the premise of guaranteed image quality.

[0124] Step 63: In response to the image quality being lower than a preset value, adjusting the target endoscope.

[0125] In some embodiments, low image quality may be due to a problem with the transmission link in the target endoscope, and the target endoscope can be adjusted according to the connectivity and signal strength of the transmission link to eliminate external factors of the target endoscope.

[0126] In some embodiments, the user may also be prompted to re-plug and re-unplug the hardware interfaces between corresponding components in the target endoscope to eliminate the phenomenon of low image quality caused by the previous plug-in problem and ensure the connectivity of the transmission link.

[0127] Step 64: After adjustment, the target endoscope is reused to capture the target image corresponding to the reference part, and the image quality of the target image is tested again until the image quality is higher than a preset value.

[0128] By executing step 64 , the target image acquired by the target endoscope is made higher than a preset value, so that the target image and the reference image are comparable, and then step 65 is executed.

[0129] Step 65: Identify the first target object in the target image and identify the second target object in the reference image.

[0130] Step 66: Determine the missing area of ​​the first target object relative to the second target object.

[0131] Step 67: Adjust the target endoscope according to the missing area.

[0132] The technical solutions of steps 65 to 67 are the same or similar to those of any embodiment of the present application. For details, please refer to other embodiments and will not be repeated here.

[0133] In this embodiment, a target endoscope is used to capture a target image corresponding to a reference part before surgery; a first target object in the target image is identified and a second target object in the reference image is identified; wherein the reference image is obtained by capturing the reference part using a reference endoscope; a missing area of ​​the first target object relative to the second target object is determined; and based on the missing area, the target endoscope is adjusted so that the target endoscope reaches an optimal image capture state, thereby improving the image capture quality during subsequent surgery, reducing the adjustment of the target endoscope by subsequent users during surgery, and improving image capture efficiency.

[0134] Furthermore, before image recognition is performed, the image quality of the target image is evaluated, and then the image quality of the target image is improved to facilitate subsequent image recognition, thereby reducing invalid image comparisons and improving image comparison efficiency.

[0135] See also Figure 7 , Figure 71 is a flow chart of another embodiment of the endoscopic preoperative detection method based on a neural network model provided by the present application. The method comprises:

[0136] Step 71: Before the operation, a target image corresponding to the reference part is collected using a target endoscope.

[0137] Step 72: Acquire a first position of the target object in the target image and acquire a second position of the target object in the reference image.

[0138] In some embodiments, target detection technology may be used to detect the positions of target objects in the target image and the reference image, such as obtaining a first position of the target object in the target image and obtaining a second position of the target object in the reference image.

[0139] Step 73: Compare the first position and the second position to obtain a comparison result.

[0140] Step 74: Adjust the target endoscope according to the comparison result.

[0141] In response to the comparison result representing the difference between the first position and the second position, the light emission angle and / or the light collection angle of the target endoscope for capturing the reference site are adjusted.

[0142] In some embodiments, in combination Figure 8 To explain:

[0143] exist Figure 8 In the figure, the target image is on the left and the reference image is on the right. Both the target image and the reference image are divided into 9 grids. For example, the target object in the target image is in the 4th grid. The target object in the reference image is in the 5th grid. Therefore, if there is a position difference between the target object in the target image and the target object in the reference image, the light emitting angle and / or the light receiving angle of the target endoscope for collecting the reference part are automatically adjusted. For example, the light emitting angle of the target endoscope is automatically adjusted to the left and / or the light emitting angle of the target endoscope is automatically adjusted to the left. Specifically, a linear relationship can be constructed according to the relative position between the first position and the second position, and the adjustment angle of the light emitting angle and / or the light receiving angle can be obtained according to the linear relationship. For example, the target object in the target image is in the 1st grid. The target object in the reference image is in the 5th grid, then the light emitting angle of the target endoscope is automatically adjusted diagonally upward and / or the light emitting angle of the target endoscope is automatically adjusted diagonally upward, and the specific adjustment angle is obtained according to the linear relationship.

[0144] In this embodiment, before the operation, a target endoscope is used to capture a target image corresponding to a reference part; the target image and the reference image are compared to obtain a comparison result; wherein the reference image is obtained by capturing the reference part using a reference endoscope; according to the comparison result, the light emitting angle and / or the light receiving angle of the reference part captured by the target endoscope are adjusted. On the one hand, the target endoscope is adjusted before the operation so that the target endoscope reaches an optimal image capture state, thereby improving the image capture quality during subsequent operations. On the other hand, the target endoscope is adjusted before the operation so that the target endoscope reaches an optimal image capture state, thereby reducing the adjustment of the target endoscope by subsequent users during the operation, thereby improving the image capture efficiency.

[0145] See also Fig. 9 , Fig. 9 1 is a flow chart of an embodiment of an endoscopic preoperative detection method based on a neural network model provided by the present application. The method comprises:

[0146] Step 91: Before the operation, a target image corresponding to the reference part is collected using a target endoscope.

[0147] Step 92: Acquire a first illumination intensity of the target object in the target image and acquire a first illumination intensity of the location of the target object in the reference image.

[0148] Step 93: Compare the first light intensity and the second light intensity to obtain a comparison result.

[0149] Step 94: Adjust the target endoscope according to the comparison result.

[0150] In some embodiments, in response to the comparison result characterizing a difference between the first illumination intensity and the second illumination intensity, the illumination intensity of the light source of the target endoscope is adjusted.

[0151] In some embodiments, in response to the comparison result indicating that the first illumination intensity is greater than the second illumination intensity, the illumination intensity of the light source of the target endoscope is reduced.

[0152] In some embodiments, in response to the comparison result indicating that the first illumination intensity is less than the second illumination intensity, the illumination intensity of the light source of the target endoscope is increased.

[0153] In this embodiment, before the operation, the target endoscope is used to capture the target image corresponding to the reference part; the target image and the reference image are compared to obtain a comparison result; wherein the reference image is obtained by capturing the reference part using a reference endoscope; according to the comparison result, the light intensity of the light source of the target endoscope is adjusted. On the one hand, the target endoscope is adjusted before the operation so that the target endoscope reaches the best image capture state, thereby improving the image capture quality during subsequent operations; on the other hand, the target endoscope is adjusted before the operation so that the target endoscope reaches the best image capture state, which can reduce the adjustment of the target endoscope by subsequent users during the operation and improve the image capture efficiency.

[0154] In some embodiments, the endoscope includes a photoelectric conversion module, and in response to the comparison result representing the difference between the first position and the second position, the angle of the dichroic mirror in the photoelectric conversion module and / or the conversion parameter of the photoelectric conversion module are adjusted. In some embodiments, the photoelectric conversion module is provided with an adjustment mechanism, which is used to adjust the angle of the dichroic mirror in the photoelectric conversion module.

[0155] In one application scenario, the target endoscope includes a host, a photoelectric conversion module, and an insertion structure. In response to the comparison result indicating that the target image is worse than the reference image, a prompt is provided to reconnect the connection position between the host, the photoelectric conversion module, and the insertion structure.

[0156] In one application scenario, after adjusting the target endoscope, parameter information involved in adjusting the target endoscope is stored so that when the target endoscope is used again, the parameter information can be obtained to adjust the target endoscope.

[0157] In one application scenario, when performing preoperative inspection on a target endoscope, after comparing the target image with the reference image to obtain the comparison result, the big data network model can be used to predict the content that needs to be adjusted in the target endoscope based on the comparison result. The content that needs to be adjusted is sorted from large to small according to the degree of influence, and then the corresponding content is adjusted in sequence according to the sorted sequence number. After each adjustment, the target image is collected again using the target endoscope, and the target image is compared with the reference image to obtain the comparison result. When the comparison result indicates that there is no difference between the target image and the reference image or the difference is less than the preset value, the adjustment of the target endoscope is stopped. The parameter information involved in adjusting the target endoscope is stored so that when the target endoscope is used again, the parameter information can be obtained to adjust the target endoscope.

[0158] See also Fig.10 , Fig.101 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 110 is used to store a computer program 111. When the computer program 111 is executed by a processor, it is used to implement the following method steps:

[0159] Before the operation, a target endoscope is used to capture a target image corresponding to a reference part; a first target object in the target image is identified and a second target object in the reference image is identified; wherein the reference image is obtained by capturing the reference part using a reference endoscope; a missing area of ​​the first target object relative to the second target object is determined; and the target endoscope is adjusted according to the missing area.

[0160] It can be understood that when the computer program 111 is executed by the processor, it is also used to implement the method of any of the above embodiments.

[0161] In summary, the neural network model-based endoscopic preoperative detection method, endoscope, and medium of the present application use a target endoscope to collect a target image corresponding to a reference part before surgery; identify a first target object in the target image and identify a second target object in the reference image; wherein the reference image is obtained by collecting the reference part using a reference endoscope; determine the missing area of ​​the first target object relative to the second target object; and adjust the target endoscope according to the missing area so that the target endoscope reaches the optimal image acquisition state, thereby improving the image acquisition quality during subsequent surgery, and reducing the adjustment of the target endoscope by subsequent users during surgery, thereby improving image acquisition efficiency.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0165] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program codes.

[0166] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A preoperative detection method for endoscopy based on a neural network model, characterized in that: The method comprises: Before surgery, a target endoscope is used to collect target images corresponding to the reference site; Identifying a first target object in the target image and identifying a second target object in a reference image; wherein the reference image is obtained by capturing a reference part using a reference endoscope; determining a missing area of ​​the first target object relative to the second target object; According to the missing area, adjusting the target endoscope; the identifying the first target object in the target image and the identifying the second target object in the reference image include: identifying a first contour of a first target object in the target image, and identifying a second contour of a second target object in the reference image; The determining the missing area of ​​the first target object relative to the second target object comprises: Determining a missing area of ​​the first contour relative to the second contour; the determining the missing area of ​​the first contour relative to the second contour comprises: Moving the first contour to the second contour until the first contour is completely within the area of ​​the second contour, thereby determining a first missing area of ​​the first contour relative to the second contour; Inputting the first target object of the first contour and the second target object of the second contour into a neural network model, so that the neural network model performs pixel completion on the first target object of the first contour to obtain the first target object of the third contour; in response to the difference between the third contour and the second contour being less than a threshold, taking the completed pixel area as the second missing area; The step of adjusting the target endoscope according to the missing area includes: The target endoscope is adjusted according to the first missing area and / or the second missing area.

2. The method according to claim 1, characterized in that The step of adjusting the target endoscope according to the first missing area or the second missing area includes: Taking the reference image as a reference, dividing the first missing region or the second missing region into a plurality of missing sub-regions; Count the missing area corresponding to each missing sub-region; Performing weighted processing on each missing area to obtain adjustment parameters of the target endoscope; The target endoscope is adjusted according to the adjustment parameters.

3. The method according to claim 2, characterized in that The step of performing weighted processing on each missing area to obtain adjustment parameters of the target endoscope includes: All missing areas are sorted from large to small, and weights are assigned from large to small according to the sorting order.

4. The method according to claim 1, characterized in that: The step of adjusting the target endoscope according to the first missing area and the second missing area includes: Obtaining a first adjustment parameter of the target endoscope according to the first missing area; Obtaining a second adjustment parameter of the target endoscope according to the second missing area; Obtaining a target adjustment parameter according to the first adjustment parameter and the second adjustment parameter; The target endoscope is adjusted using the target adjustment parameters.

5. The method according to claim 1, characterized in that The neural network model includes a coding layer, a feature fusion layer and a decoding layer, wherein the feature fusion layer is used to enhance and fuse the features of each level extracted by the coding layer, and output the features to the decoding layer; the coding layer is divided into a first type of coding layer and a second type of coding layer, and the first target object of the first contour and the second target object of the second contour are input into the neural network model so that the neural network model performs pixel completion on the first target object of the first contour to obtain the first target object of the third contour, including: Using the first type of encoding layer to extract features of the first target object of the first contour, obtain first feature maps corresponding to each level, and input the first feature maps into the attention mechanism layer; Using the second type of encoding layer to extract features of the second target object of the second contour, obtain second feature maps corresponding to each level, and input the second feature maps into the attention mechanism layer; Using the attention mechanism layer, extracting features from the first feature map and the second feature map according to corresponding levels to obtain third feature maps corresponding to each level; Inputting the third feature maps corresponding to each level into the feature fusion layer for enhanced fusion to obtain the fourth feature maps corresponding to each level; The fourth feature map corresponding to each level is input into the decoding layer for pixel completion to obtain the first target object of the third contour.

6. The method according to claim 5, characterized in that The using the attention mechanism layer to extract features from the first feature map and the second feature map according to corresponding levels to obtain third feature maps corresponding to each level includes: Performing a convolution operation on the first feature map using an attention mechanism layer to obtain a first target feature map, and inputting the first target feature map into a next encoding layer of the first type of encoding layer; Performing a convolution operation on the second feature map using an attention mechanism layer to obtain a second target feature map, and inputting the second target feature map into a next encoding layer of the second type of encoding layer; The first target feature map and the second target feature map are feature-fused to obtain the third feature map.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the computer program is used to implement the method according to any one of claims 1 to 6.

Images