Handheld endoscope imaging method based on image restoration, endoscope and medium

Abstract

The invention discloses a handheld endoscope imaging method based on image restoration, an endoscope and a medium. The method comprises the following steps: in response to a first image collected by the handheld endoscope under a preset magnification factor, automatically controlling the handheld endoscope to collect a second image under an original magnification factor; magnifying the second image according to a preset magnification factor to obtain a third image; respectively acquiring image quality of the first image and the third image; in response to the condition that the image quality of the third image is greater than that of the first image, performing image restoration on the first image to obtain a fourth image; and displaying the fourth image. In this way, the imaging stability of the amplified image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of endoscopes, and in particular to a handheld endoscope imaging method, an endoscope and a medium based on image restoration. Background Art

[0002] As a medical device, the imaging quality of the endoscope indirectly affects the safety of diagnosis and treatment. When collecting images, the operator can collect images of corresponding magnification by operating the magnification of the endoscope, and then display them.

[0003] In this process, how to ensure the imaging stability of the enlarged image is an urgent problem that needs to be solved. Summary of the invention

[0004] The handheld endoscope imaging method, endoscope and medium based on image restoration provided in the present application can improve the imaging stability of the magnified image.

[0005] A technical solution adopted in the present application is to provide a handheld endoscope imaging method based on image restoration, the method comprising: in response to the handheld endoscope capturing a first image at a preset magnification, automatically controlling the handheld endoscope to capture a second image at the original magnification; amplifying the second image according to the preset magnification to obtain a third image; respectively acquiring the image qualities of the first image and the third image; in response to the image quality of the third image being greater than the image quality of the first image, performing image restoration on the first image to obtain a fourth image; and displaying the fourth image.

[0006] The method of amplifying the second image according to a preset magnification factor to obtain a third image includes: inputting the preset magnification factor and the second image into an image processing network, so that the image processing network upsamples and amplifies the second image according to the preset magnification factor to obtain the third image.

[0007] Among them, obtaining the image quality of the first image and the third image includes: identifying a first area corresponding to the target in the first image, and identifying a second area corresponding to the target in the third image; using the first area as the image quality of the first image, and using the second area as the image quality of the third image; in response to the image quality of the third image being greater than the image quality of the first image, performing image restoration on the first image to obtain a fourth image, including: in response to the second area of ​​the third image being greater than the first area of ​​the first image, performing image restoration on the first image to obtain the fourth image.

[0008] Among them, performing image restoration on the first image to obtain the fourth image includes: determining the actual magnification corresponding to the first image according to the image quality of the third image and the image quality of the first image; and performing image restoration on the first image according to the preset magnification and the actual magnification to obtain the fourth image.

[0009] Among them, according to the preset magnification and the actual magnification, the first image is restored to obtain the fourth image, including: determining the magnification difference according to the preset magnification and the actual magnification; inputting the magnification difference and the first image into the image processing network, so that the image processing network upsamples and amplifies the first image according to the magnification difference to obtain the fourth image.

[0010] The image processing network includes a first processing branch and a second processing branch, and the multiple difference and the first image are input into the image processing network so that the image processing network upsamples and amplifies the first image according to the multiple difference to obtain a fourth image, including: inputting the multiple difference and the first image into the first processing branch for upsampling and amplification to obtain a first feature map; inputting a preset magnification factor and the second image into the second processing branch for upsampling and amplification to obtain a second feature map; and fusing the first feature map and the second feature map to obtain the fourth image.

[0011] Among them, the first processing branch includes sequentially connected upsampling layers, and the second processing branch includes sequentially connected upsampling layers; the output end of the current level upsampling layer of the second processing branch is also connected to the input end of the next level upsampling layer of the first processing branch; or, the output end of the current level upsampling layer of the second processing branch is also connected to the input end of the next level upsampling layer of the first processing branch, and the output end of the current level upsampling layer of the first processing branch is also connected to the input end of the next level upsampling layer of the second processing branch.

[0012] The fusing of the first feature map and the second feature map to obtain the fourth image includes: obtaining a third area belonging to the target in the second feature map; and fusing the third area into the first feature map to obtain the fourth image.

[0013] Another technical solution adopted in the present application is to provide an endoscope, which includes a host end, a photoelectric conversion module and an insertion structure; the host end includes a display, and the host end, the photoelectric conversion module, the display and the insertion structure cooperate to implement the method provided by the above technical solution.

[0014] Another technical solution adopted by the present application is to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the method provided by the above technical solution.

[0015] The beneficial effects of the present application are as follows: different from the prior art, the handheld endoscope imaging method, endoscope and medium based on image restoration of the present application, when the handheld endoscope captures a first image at a preset magnification, automatically controls the handheld endoscope to capture a second image at the original magnification; amplifies the second image according to the preset magnification to obtain a third image; uses the third image to determine whether the first image meets the requirements of the preset magnification, and when the image quality of the third image is greater than that of the first image, performs image restoration on the first image to obtain a fourth image; and displays the fourth image, thereby ensuring that the displayed fourth image is infinitely close to the corresponding preset magnification, thereby improving the imaging stability of the enlarged image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a structural schematic diagram of an embodiment of an endoscope provided by the present application; Figure 2 It is a structural schematic diagram of an embodiment of an optoelectronic link of an endoscope provided by the present application; Figure 3 It is a flowchart of an embodiment of a handheld endoscope imaging method based on image restoration provided by the present application; Figure 4 It is a flowchart of another embodiment of the handheld endoscope imaging method based on image restoration provided by the present application; Figure 5 It is a flowchart of an embodiment of step 44 provided by the present application; Figure 6 This is a schematic diagram of an application scenario of a handheld endoscope imaging method based on image restoration provided by the present application; Figure 7 This is a schematic diagram of another application scenario of the handheld endoscope imaging method based on image restoration provided by the present application; Figure 8 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0018] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] As a medical device, the imaging quality of the endoscope indirectly affects the safety of diagnosis and treatment. When collecting images, the operator can collect images of corresponding magnification by operating the magnification of the endoscope, and then display them.

[0020] In this process, how to ensure the imaging stability of the magnified image is an urgent problem to be solved. For example, if the actual magnification of the image collected according to the preset magnification is less than the preset magnification, the displayed image does not correspond to the actual image to be displayed, which will affect the subsequent judgment of the target in the image.

[0021] Based on this, the present application proposes that when a handheld endoscope captures a first image at a preset magnification, the handheld endoscope is automatically controlled to capture a second image at the original magnification; the second image is amplified according to the preset magnification to obtain a third image; the third image is used to determine whether the first image meets the requirements of the preset magnification, and when the image quality of the third image is greater than that of the first image, the first image is repaired to obtain a fourth image; the fourth image is displayed to ensure that the displayed fourth image is infinitely close to the corresponding preset magnification, thereby improving the imaging stability of the enlarged image.

[0022] See also Figure 1 , Figure 1 1 is a schematic diagram of the structure of an embodiment of an endoscope provided by the present application. The endoscope comprises: a host end, a photoelectric conversion module and an insertion structure. The host end, the photoelectric conversion module and the insertion structure cooperate to realize the technical solution formed by any one of the following embodiments or any combination of the embodiments.

[0023] The host side can be provided with an operation screen, a USB interface, a scanning terminal, a medical caster, a switch button, a camera handle bracket, a camera handle interface, and an LED indicator. Among them, the operation screen is used to display functions and provide an operation interface. The USB interface is used for external USB. The scanning terminal is used for scanning code recognition. The medical caster is used to move the host side. The switch button is used to turn the host side on and off. The camera handle bracket is used to hang the camera handle. The camera handle interface is used to connect the camera handle. The LED indicator is used to indicate the working status of the host side. If the indicator is on, the trolley is working; if the indicator is off, the trolley is not working.

[0024] Specifically, the entire system of the endoscope is as follows: The camera module uses the FPGA board as the central control center to connect the barcode scanner, camera handle (CMOS-ISP board), camera serial port screen, video recording board and light source driver board to complete the barcode analysis, imaging parameter control, camera serial port screen control, video recording control and interaction with the light source driver board. Users can use the camera serial port screen to take photos, record, set the system time, set the language and restore factory settings. The light source module uses the light source driver board as the central control center to connect the light source serial port screen, temperature sensor, light source, FPGA board and start switch to adjust the light source brightness, monitor the light source temperature and start the light source driver board to power each module. The light generated by the light source module provides a light guide connected to the camera handle for clear imaging.

[0025] The corresponding software functions are as follows: The functions included in the light source software are: power on, bulb switch function, brightness adjustment function, temperature display function, and running time function.

[0026] Light bulb switch function: The software can turn the light bulb on or off.

[0027] Brightness adjustment function: The software can adjust the brightness of the bulb from 10 to 100, with each level changing by 10.

[0028] Temperature display function: The software interface displays the current temperature of the bulb.

[0029] Running time function: The software interface displays the actual cumulative running time of the light source. When the lamp runs for more than 20,000 hours, a prompt box will appear.

[0030] The functions included in the camera system software are: power on, user access control, lock screen function, system setting function (including system time setting, language setting, and restoring factory settings), video recording function, photo taking function, and one-key enhancement function.

[0031] User access control: When the device is turned on, the boot UI is displayed and a progress bar is loaded. After the progress bar is loaded, the display screen jumps to the code scanning UI. Scan the QR code on the sheath to display the image.

[0032] Lock screen function: The camera system host can lock the operation interface to prevent accidental touches.

[0033] Product system setting function: 1 System time setting: You can set the system time in 24-hour format.

[0034] 2 Language setting: You can set the language, supporting Chinese and English.

[0035] 3 Restore factory settings: Restore factory settings.

[0036] Video recording function: insert the USB storage medium, the product has the video recording function, and can be stored in real time through USB, the storage format is .ts.

[0037] Photo function: insert the USB storage medium, the product has the photo function, which can be stored via USB, and the storage format is JPG.

[0038] One-key enhancement function: After pressing the button, the image is adjusted to the preset value. For the brightness of the picture, select one-key enhancement to adjust the overall brightness of the picture.

[0039] Camera system software, including power-on, user access control, screen lock, photo taking, video recording and one-key enhancement functions.

[0040] The entire image processing link includes: sensor board, cable, sensor adapter board, FPGA (Field Programmable Gate Array) mainboard and display.

[0041] Sensor board: uses a chip to convert the light signal irradiated on the sensor into an electrical signal, and outputs the electrical signal through the rear-end cable.

[0042] Cable: Transmits electrical signals and provides a power path for the Sensor board.

[0043] Sensor adapter board: The core of the sensor adapter board is an FPGA chip. Its main function is to sample and convert the electrical signals transmitted by the sensor board into serial signals, and convert the serialized data into optical signals through the optoelectronic conversion module, and transmit the signals to the host through optical fiber.

[0044] FPGA main board: Receives the optical signal transmitted from the sensor adapter board and converts it into an electrical signal through the photoelectric conversion module. It then processes the original image into a target image through a series of image ISP algorithms and reconstructs the image to output in a specified format.

[0045] Display: Display the processed image.

[0046] like Figure 2 As shown, the camera handle includes a photoelectric conversion module. The light source exciter in the host end provides an incident light source, which is connected to the inside of the handle from the handle tail end 60 through the light source optical fiber on the host end, and then extends along the optical fiber 40 inside the handle to the light source incident end 50, and vertically incident on the photoelectric conversion module 20 (handle fluorescence module) upward, and is incident on the surface of the living tissue at the right after a 90° total reflection through the dichroic mirror set at a 45° angle inside the handle fluorescence module. After the fluorescence excitation on the surface of the living tissue, the reflected light is formed, and the reflected light returns to the dichroic mirror along the original path. At this time, due to the excitation change in the band, the reflected light is all transmitted through the dichroic mirror and then transmitted to the rear-end photoelectric conversion module. After being received by the photoelectric conversion module, the optical signal is converted into an electrical signal. In order to prevent electromagnetic interference, the handle will perform electro-optical conversion again, and the electrical signal will be converted into an optical signal again. The optical signal is transmitted to the image signal processor in the machine through the return optical fiber signal line, and then the photoelectric conversion is performed for the second time to convert the returned optical signal into an electrical signal. Then, through the processing of the electrical signal, a microscopic image magnified more than 1000 times is output on the display.

[0047] In some embodiments, the endoscope described above may be a handheld endoscope.

[0048] See also Figure 3 , Figure 3 1 is a flow chart of an embodiment of a handheld endoscope imaging method based on image restoration provided by the present application. The method comprises: Step 31: In response to the handheld endoscope capturing a first image at a preset magnification, automatically controlling the handheld endoscope to capture a second image at an original magnification.

[0049] In some embodiments, the magnification of the handheld endoscope can be set, and the first image can be acquired by using the handheld endoscope at a preset magnification. For example, the preset magnification can be 10 times, 20 times, 30 times, 40 times, 50 times, 100 times, 200 times, 300 times, 400 times, or 500 times.

[0050] After the handheld endoscope acquires the first image at the preset magnification, the handheld endoscope is automatically controlled to acquire the second image at the original magnification. The original magnification indicates no magnification. If expressed in numerical values, the original magnification is 0.

[0051] It can be understood that the first image and the second image are acquired with respect to the same target.

[0052] Step 32: Enlarge the second image according to a preset magnification factor to obtain a third image.

[0053] In some embodiments, step 32 may be to amplify the second image by software to obtain the third image.

[0054] In some embodiments, the preset magnification and the second image are input into the image processing network, so that the image processing network upsamples and amplifies the second image according to the preset magnification to obtain a third image. In some embodiments, the image processing network can use the preset magnification as an upsampling parameter, and adjust the image processing network according to the upsampling parameter, so that the image processing network upsamples and amplifies the second image according to the preset magnification to obtain a third image.

[0055] Step 33: Obtain the image quality of the first image and the third image respectively.

[0056] It can be understood that the first image is directly acquired at a preset magnification by using the image acquisition component in the handheld endoscope. The third image is obtained by software processing. If there is no abnormality in the acquisition process of the first image, the image quality of the first image should be greater than the image quality of the third image. If there is an abnormality in the acquisition process of the first image, the image quality of the first image may be less than the image quality of the third image.

[0057] Step 34: In response to the image quality of the third image being greater than the image quality of the first image, performing image restoration on the first image to obtain a fourth image.

[0058] If the image quality of the third image is greater than that of the first image, it means that an abnormality occurred during the acquisition of the first image, and the first image acquired by the image acquisition component in the handheld endoscope does not correspond to the preset magnification. In other words, the actual magnification corresponding to the first image at this time is less than the preset magnification.

[0059] At this time, it is necessary to perform image restoration on the first image to obtain a fourth image that meets the preset magnification, so that the displayed image can correspond to the preset magnification.

[0060] In some embodiments, in response to the image quality of the third image being lower than that of the first image, target recognition is performed on the first image, and the target is marked in the image, and the image with the target marked is used as the fourth image.

[0061] In other embodiments, in response to the image quality of the third image being greater than the image quality of the first image, a compensation magnification may be calculated based on the difference between the third image and the first image. A target image is acquired at the compensation magnification using a handheld endoscope, and the target image is used as the fourth image. Specifically, a difference percentage between the third image and the first image is calculated. An incremental magnification is obtained by multiplying a preset magnification by the difference percentage. A compensation magnification is obtained by adding the preset magnification to the incremental magnification. The image quality of the target image acquired using the compensation magnification is greater than the image quality of the third image.

[0062] Step 35: Display the fourth image.

[0063] In this embodiment, when the handheld endoscope captures a first image at a preset magnification, the handheld endoscope is automatically controlled to capture a second image at the original magnification; the second image is amplified according to the preset magnification to obtain a third image; the third image is used to determine whether the first image meets the requirements of the preset magnification, and when the image quality of the third image is greater than that of the first image, the first image is repaired to obtain a fourth image; the fourth image is displayed to ensure that the displayed fourth image is infinitely close to the corresponding preset magnification, thereby improving the imaging stability of the enlarged image.

[0064] See also Figure 4 , Figure 4 1 is a flow chart of another embodiment of a handheld endoscope imaging method based on image restoration provided by the present application. The method comprises: Step 41: In response to the handheld endoscope capturing a first image at a preset magnification, automatically controlling the handheld endoscope to capture a second image at an original magnification.

[0065] Step 42: Enlarge the second image according to a preset magnification factor to obtain a third image.

[0066] In some embodiments, the preset magnification and the second image are input into an image processing network, so that the image processing network upsamples and amplifies the second image according to the preset magnification to obtain a third image.

[0067] Step 43: Identify a first region corresponding to the target in the first image, and identify a second region corresponding to the target in the third image.

[0068] In some embodiments, the target detection network can be used to identify the first image and the third image respectively, thereby identifying a first area corresponding to the target in the first image and identifying a second area corresponding to the target in the third image.

[0069] It can be understood that the first image is directly acquired at a preset magnification using the image acquisition component in the handheld endoscope. The third image is obtained by software processing. If there is no abnormality in the acquisition process of the first image, the first area corresponding to the target in the first image should be larger than the second area corresponding to the target in the third image. If there is an abnormality in the acquisition process of the first image, the first area corresponding to the target in the first image may be smaller than the second area corresponding to the target in the third image.

[0070] Step 44: In response to the second area of ​​the third image being larger than the first area of ​​the first image, image restoration is performed on the first image to obtain a fourth image.

[0071] In some embodiments, in response to the second area of ​​the third image being smaller than the first area of ​​the first image, the target is marked in the first image, and the image with the marked target is used as the fourth image.

[0072] In other embodiments, in response to the second area of ​​the third image being larger than the first area of ​​the first image, a compensation magnification can be calculated according to the difference between the second area and the first area. A target image is acquired at the compensation magnification using a handheld endoscope, and the target image is used as the fourth image. Specifically, a difference percentage between the second area and the first area is calculated. An incremental magnification is obtained by multiplying a preset magnification by the difference percentage. A compensation magnification is obtained by adding the preset magnification to the incremental magnification. The image quality of the target image acquired using the compensation magnification is greater than the image quality of the third image.

[0073] In some embodiments, see Figure 5 , step 44 may be the following process: Step 441: Determine the actual magnification corresponding to the first image according to the image quality of the third image and the image quality of the first image.

[0074] In some embodiments, the actual magnification corresponding to the first image is determined by comparing the area size of the second area of ​​the third image with the area size of the first area of ​​the first image, such as calculating the ratio between the first area and the second area, and calculating the actual magnification using the ratio and the preset magnification.

[0075] Step 442: Perform image restoration on the first image according to the preset magnification and the actual magnification to obtain a fourth image.

[0076] In some embodiments, a magnification difference can be determined based on a preset magnification and an actual magnification; the magnification difference and the first image are input into an image processing network so that the image processing network upsamples and amplifies the first image according to the magnification difference to obtain a fourth image.

[0077] It can be understood that although the third image also corresponds to the preset magnification, the third image is obtained by software. Compared with the first image being magnified again, the magnification difference of the third image is larger, so the fourth image obtained by upsampling and amplifying the first image is more consistent with the image at the preset magnification.

[0078] In some embodiments, the image processing network may use the multiple difference as an upsampling parameter, and adjust the image processing network according to the upsampling parameter, so that the image processing network upsamples and amplifies the first image according to the multiple difference to obtain a fourth image.

[0079] In an application scenario, the image processing network includes a first processing branch and a second processing branch. Based on this, the magnification difference and the first image can be input into the first processing branch for upsampling and amplification to obtain a first feature map; the preset magnification and the second image can be input into the second processing branch for upsampling and amplification to obtain a second feature map; and the first feature map and the second feature map can be fused to obtain a fourth image.

[0080] In order to improve the imaging accuracy of the fourth image, the second feature map corresponding to the second image is fused to the first feature map in the above manner, thereby obtaining a more accurate fourth image.

[0081] In some embodiments, the fourth image can be obtained by fusing the first feature map and the second feature map in the following manner: obtaining the third area belonging to the target in the second feature map; fusing the third area into the first feature map to obtain the fourth image. By obtaining the third area belonging to the target in the second feature map and fusing the third area into the first feature map to obtain the fourth image, it is not necessary to fuse all the contents of the second feature map into the first feature map, which not only improves the fusion efficiency, but also improves the accuracy of the target in the fourth image and the imaging stability by targeting the feature area corresponding to the fusion target.

[0082] In some embodiments, the first processing branch includes sequentially connected upsampling layers, and the second processing branch includes sequentially connected upsampling layers; the output end of the current upsampling layer of the second processing branch is also connected to the input end of the next upsampling layer of the first processing branch. Figure 6 For explanation: the first processing branch includes a first upsampling layer, a second upsampling layer and a third upsampling layer connected in sequence, and the second processing branch includes a first upsampling layer, a second upsampling layer and a third upsampling layer connected in sequence.

[0083] The multiple difference and the first image are input to a first upsampling layer in a first processing branch, and the first upsampling layer outputs a first upsampling feature.

[0084] The preset magnification and the second image are input to the first upsampling layer in the second processing branch, and the first upsampling layer outputs the second upsampling features.

[0085] The first up-sampled features and the second up-sampled features are input to the second up-sampled layer in the first processing branch, and the second up-sampled layer outputs the third up-sampled features.

[0086] The second up-sampled features are input to the second up-sampled layer in the second processing branch, and the second up-sampled layer outputs the fourth up-sampled features.

[0087] The third up-sampled features and the fourth up-sampled features are input to the third up-sampling layer in the first processing branch, and the third up-sampling layer outputs a first feature map.

[0088] The fourth up-sampled features are input to the third up-sampling layer in the second processing branch, and the third up-sampling layer outputs a second feature map.

[0089] The first feature map and the second feature map are fused at the fusion layer to obtain a fourth image.

[0090] Through the above method, in the process of upsampling the first image, the features corresponding to the upsampling of the second image are fused into the corresponding upsampling layer in the first processing branch, so that the first feature map finally output by the first processing branch can be more accurate, and then fused with the second feature map to produce an accurate fourth image.

[0091] In some embodiments, the first processing branch includes serially connected upsampling layers, and the second processing branch includes serially connected upsampling layers; the output end of the current level upsampling layer of the second processing branch is also connected to the input end of the next level upsampling layer of the first processing branch, and the output end of the current level upsampling layer of the first processing branch is also connected to the input end of the next level upsampling layer of the second processing branch.

[0092] Combination Figure 7 For explanation: the first processing branch includes a first upsampling layer, a second upsampling layer and a third upsampling layer connected in sequence, and the second processing branch includes a first upsampling layer, a second upsampling layer and a third upsampling layer connected in sequence.

[0093] The multiple difference and the first image are input to a first upsampling layer in a first processing branch, and the first upsampling layer outputs a first upsampling feature.

[0094] The preset magnification and the second image are input to the first upsampling layer in the second processing branch, and the first upsampling layer outputs the second upsampling features.

[0095] The first up-sampled features and the second up-sampled features are input to the second up-sampled layer in the first processing branch, and the second up-sampled layer outputs the third up-sampled features.

[0096] The first up-sampled features and the second up-sampled features are input to the second up-sampled layer in the second processing branch, and the second up-sampled layer outputs the fourth up-sampled features.

[0097] The third up-sampled features and the fourth up-sampled features are input to the third up-sampling layer in the first processing branch, and the third up-sampling layer outputs a first feature map.

[0098] The third up-sampled features and the fourth up-sampled features are input to a third up-sampling layer in the second processing branch, and the third up-sampling layer outputs a second feature map.

[0099] The first feature map and the second feature map are fused at the fusion layer to obtain a fourth image.

[0100] Through the above method, in the process of upsampling the first image, the features corresponding to the upsampling of the second image are fused to the corresponding upsampling layer in the first processing branch, and in the process of upsampling the second image, the features corresponding to the upsampling of the first image are fused to the corresponding upsampling layer in the second processing branch, so that the first feature map finally output by the first processing branch and the second feature map finally output by the second processing branch can be more accurate, and then an accurate fourth image is fused.

[0101] Step 45: Display the fourth image.

[0102] In this embodiment, when the handheld endoscope captures a first image at a preset magnification, the handheld endoscope is automatically controlled to capture a second image at the original magnification; the second image is amplified according to the preset magnification to obtain a third image; the third image is used to determine whether the first image meets the requirements of the preset magnification, and when the image quality of the third image is greater than that of the first image, the first image is repaired to obtain a fourth image; the fourth image is displayed to ensure that the displayed fourth image is infinitely close to the corresponding preset magnification, thereby improving the imaging stability of the enlarged image.

[0103] See also Figure 8 , Figure 8 1 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 110 is used to store a computer program 111. When the computer program 111 is executed by a processor, it is used to implement the following method steps: In response to the handheld endoscope capturing a first image at a preset magnification, the handheld endoscope is automatically controlled to capture a second image at the original magnification; the second image is amplified according to the preset magnification to obtain a third image; the image qualities of the first image and the third image are respectively obtained; in response to the image quality of the third image being greater than the image quality of the first image, the first image is image repaired to obtain a fourth image; and the fourth image is displayed.

[0104] It can be understood that when the computer program 111 is executed by the processor, it is also used to implement the method of any of the above embodiments.

[0105] In summary, the handheld endoscope imaging method, endoscope and medium based on image restoration of the present application, when the handheld endoscope captures a first image at a preset magnification, automatically controls the handheld endoscope to capture a second image at the original magnification; amplifies the second image according to the preset magnification to obtain a third image; uses the third image to determine whether the first image meets the preset magnification requirement, and when the image quality of the third image is greater than that of the first image, performs image restoration on the first image to obtain a fourth image; displays the fourth image, thereby ensuring that the displayed fourth image is infinitely close to the corresponding preset magnification, thereby improving the imaging stability of the enlarged image.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0110] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A handheld endoscope imaging method based on image restoration, characterized in that: The method comprises: In response to the handheld endoscope capturing a first image at a preset magnification, automatically controlling the handheld endoscope to capture a second image at an original magnification; Amplify the second image according to the preset magnification to obtain a third image; respectively acquiring image qualities of the first image and the third image; In response to the image quality of the third image being greater than the image quality of the first image, performing image restoration on the first image to obtain a fourth image; The fourth image is displayed.

2. The method according to claim 1, characterized in that The step of magnifying the second image according to the preset magnification factor to obtain a third image includes: The preset magnification factor and the second image are input into an image processing network, so that the image processing network upsamples and amplifies the second image according to the preset magnification factor to obtain a third image.

3. The method according to claim 1, characterized in that The acquiring the image qualities of the first image and the third image includes: Identifying a first area corresponding to the target in the first image, and identifying a second area corresponding to the target in the third image; Taking the first area as the image quality of the first image, and taking the second area as the image quality of the third image; In response to the image quality of the third image being greater than the image quality of the first image, performing image restoration on the first image to obtain a fourth image includes: In response to the second area of ​​the third image being larger than the first area of ​​the first image, image restoration is performed on the first image to obtain a fourth image.

4. The method according to claim 1, characterized in that: The performing image restoration on the first image to obtain a fourth image includes: determining an actual magnification corresponding to the first image according to the image quality of the third image and the image quality of the first image; Image restoration is performed on the first image according to the preset magnification and the actual magnification to obtain the fourth image.

5. The method according to claim 4, characterized in that The performing image restoration on the first image according to the preset magnification and the actual magnification to obtain the fourth image includes: Determine a magnification difference according to the preset magnification and the actual magnification; The multiple difference and the first image are input into an image processing network, so that the image processing network upsamples and amplifies the first image according to the multiple difference to obtain the fourth image.

6. The method according to claim 5, characterized in that The image processing network includes a first processing branch and a second processing branch, and the multiple difference and the first image are input into the image processing network so that the image processing network upsamples and amplifies the first image according to the multiple difference to obtain the fourth image, including: Inputting the multiple difference and the first image into the first processing branch for upsampling and amplification to obtain a first feature map; Inputting the preset magnification factor and the second image into a second processing branch for upsampling and amplification to obtain a second feature map; The first feature map and the second feature map are fused to obtain the fourth image.

7. The method according to claim 6, characterized in that The first processing branch includes sequentially connected upsampling layers, and the second processing branch includes sequentially connected upsampling layers; the output end of the current upsampling layer of the second processing branch is also connected to the input end of the next upsampling layer of the first processing branch; Or, the output end of the current level upsampling layer of the second processing branch is also connected to the input end of the next level upsampling layer of the first processing branch, and the output end of the current level upsampling layer of the first processing branch is also connected to the input end of the next level upsampling layer of the second processing branch.

8. The method according to claim 6, characterized in that The fusing the first feature map and the second feature map to obtain the fourth image includes: Acquire a third region belonging to the target in the second feature map; The third region is fused into the first feature map to obtain the fourth image.

9. An endoscope, characterized in that: The endoscope includes a host end, a photoelectric conversion module and an insertion structure; wherein the host end includes a display, and the host end, the photoelectric conversion module, the display and the insertion structure cooperate to implement the method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the computer program is used to implement the method according to any one of claims 1 to 8.

Images