Endoscope, anti-jitter method and medium for restoring images based on jitter data

Abstract

The present application discloses an endoscope, an anti-jitter method and a medium for restoring images based on jitter data. The method includes: in response to an external image acquisition instruction, obtaining the jitter data of the endoscope; wherein the external image acquisition instruction is generated by external control; predicting the generation time of the internal image acquisition instruction according to the jitter data and the instruction transmission time; wherein the instruction transmission time indicates the time required for the internal image acquisition instruction to be sent to the image acquisition component; generating the internal image acquisition instruction according to the generation time, and using the internal image acquisition instruction to control the image acquisition component to perform image acquisition; wherein the endoscope is relatively still during image acquisition. Through the above method, anti-jitter image acquisition is achieved and image acquisition efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of endoscopes, and in particular to an endoscope for restoring images based on jitter data, an anti-jitter method and a medium. Background Art

[0002] As a medical device, the optical properties of the endoscope affect its imaging quality, which indirectly affects the safety of diagnosis and treatment. When collecting images, the operator's body will inevitably shake due to the need to press the camera button, which can easily cause the captured image to be blurred.

[0003] In the related art, the image obtained by taking pictures is processed and / or corrected to reduce the image blur defect caused by shaking at the time of shooting. Summary of the invention

[0004] The endoscope, anti-jitter method and medium for restoring images based on jitter data provided in the present application can realize anti-jitter image acquisition and improve image acquisition efficiency.

[0005] A technical solution adopted in the present application is to provide an endoscope anti-shake method for restoring images based on jitter data, the method comprising: obtaining jitter data of the endoscope in response to an external image acquisition instruction; wherein the external image acquisition instruction is generated by external control; predicting the generation time of the internal image acquisition instruction based on the jitter data and the instruction transmission time; wherein the instruction transmission time represents the time required for the internal image acquisition instruction to be sent to the image acquisition component; generating the internal image acquisition instruction based on the generation time, and using the internal image acquisition instruction to control the image acquisition component to perform image acquisition; wherein the endoscope is relatively stationary during image acquisition.

[0006] Wherein, a jitter data sensor is arranged in the insertion structure of the endoscope; and in response to an external image acquisition instruction, the jitter data of the endoscope is acquired, including: in response to the external image acquisition instruction, the jitter data acquired by the jitter data sensor is acquired.

[0007] Among them, predicting the generation time of the internal image acquisition instruction based on the jitter data and the instruction transmission time includes: predicting the first time required for the endoscope to change from a current state to a relatively still state based on the jitter data; predicting the generation time of the internal image acquisition instruction based on the first time and the instruction transmission time.

[0008] Among them, predicting the generation time of the internal image acquisition instruction based on the first time length and the instruction transmission time includes: determining the target time when the endoscope is relatively still based on the first time length and the current time; and predicting the generation time of the internal image acquisition instruction using the target time and the instruction transmission time.

[0009] Among them, determining the target time when the endoscope is relatively still based on the first duration and the current time includes: obtaining the transmission duration of the jitter data; determining the target time when the endoscope is relatively still based on the first duration, the transmission duration and the current time.

[0010] Among them, predicting the first time duration required for the endoscope to change from the current state to a relatively still state based on the jitter data includes: inputting the jitter data into a time prediction model to obtain the first time duration required for the endoscope to change from the current state to a relatively still state predicted by the time prediction model.

[0011] Among them, the duration prediction model includes a state prediction network, a duration prediction network and a comprehensive prediction network; the jitter data is input into the duration prediction model to obtain the first duration required for the endoscope to change from the current state to a relative still state predicted by the duration prediction model, including: inputting the jitter data and the transmission duration of the jitter data into the state prediction network to obtain the current state of the endoscope output by the state prediction network; inputting the jitter data and the transmission duration of the jitter data into the duration prediction network to obtain the first target duration required for the endoscope to be relatively still output by the duration prediction network; inputting the current state of the endoscope and the first target duration into the comprehensive prediction network to obtain the first duration required for the endoscope to be relatively still output by the comprehensive prediction network.

[0012] Among them, the current state of the endoscope and the first target duration are input into the comprehensive prediction network to obtain the first duration required for the endoscope to be relatively still output by the comprehensive prediction network, including: using the comprehensive prediction network to predict the second target duration required for the endoscope to be relatively still according to the current state of the endoscope; using the comprehensive prediction network to output the first duration required for the endoscope to be relatively still according to the first target duration and the second target duration.

[0013] Another technical solution adopted in the present application is to provide an endoscope for restoring images based on jitter data, the endoscope comprising a host end, a photoelectric conversion module and an insertion structure; the host end, the photoelectric conversion module and the insertion structure cooperate to implement the method provided by the above technical solution.

[0014] Another technical solution adopted by the present application is to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the method provided by the above technical solution.

[0015] The beneficial effects of the present application are as follows: different from the prior art, the endoscope, anti-jitter method and medium for restoring images based on jitter data of the present application synchronously obtain the jitter data of the endoscope when receiving an external image acquisition instruction; and then predict the generation time of the internal image acquisition instruction according to the jitter data and the instruction transmission time; wherein the instruction transmission time indicates the time required for the internal image acquisition instruction to be sent to the image acquisition component; so as to generate the internal image acquisition instruction according to the generation time, and use the internal image acquisition instruction to control the image acquisition component to perform image acquisition when the endoscope is relatively still, and acquire an image that meets the image sampling requirements. In this process, the image acquisition instruction is The instructions are divided into external image acquisition instructions and internal image acquisition instructions, and the generation time of the internal image acquisition instructions is predicted, so that when the internal image acquisition instructions are sent to the image acquisition component, the endoscope is synchronously in a relatively static state, and images that meet the image sampling requirements are acquired, effectively reducing the problem of poor image quality caused by jitter. Furthermore, the present application does not require additional mechanical structure, and utilizes the relatively static characteristics to predict the moment when the endoscope is in a relatively static state to complete image acquisition. It can be applied to any jitter scenario of the endoscope, thereby realizing anti-jitter image acquisition, and there is no need to post-process and / or correct the images obtained by taking pictures as in the related art, thereby improving image acquisition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1 It is a structural schematic diagram of an embodiment of an endoscope for restoring an image based on jitter data provided by the present application;

[0018] Figure 2 It is a structural schematic diagram of an embodiment of an optoelectronic link of an endoscope for restoring an image based on jitter data provided by the present application;

[0019] Figure 3 It is a flow chart of an embodiment of an endoscope anti-shake method for restoring an image based on shake data provided by the present application;

[0020] Figure 4 It is a flow chart of another embodiment of the endoscope anti-shake method based on jitter data restoration image provided by the present application;

[0021] Figure 5 It is a flowchart of an embodiment of step 43 provided by the present application;

[0022] Figure 6 It is a flowchart of an embodiment of step 431 provided by the present application;

[0023] Figure 7 This is a schematic diagram of an application scenario of step 431 provided in this application;

[0024] Figure 8 It is a flowchart of an embodiment of step 42 provided by the present application;

[0025] Fig. 9 is a flowchart of an embodiment of step 83 provided by the present application;

[0026] Fig.10 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0028] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] As a medical device, the optical properties of the endoscope affect its imaging quality, which indirectly affects the safety of diagnosis and treatment. When collecting images, the operator's body will inevitably shake due to the need to press the camera button, which can easily cause the captured image to be blurred.

[0030] In the related art, the image obtained by taking pictures is processed and / or corrected to reduce the image blur defect caused by shaking at the time of shooting.

[0031] Based on this, the present application proposes to synchronously obtain the jitter data of the endoscope when receiving an external image acquisition instruction; and then predict the generation time of the internal image acquisition instruction based on the jitter data and the instruction transmission time; wherein the instruction transmission time represents the time required for the internal image acquisition instruction to be sent to the image acquisition component; so as to generate the internal image acquisition instruction according to the generation time, and use the internal image acquisition instruction to control the image acquisition component to perform image acquisition when the endoscope is in a relatively static state, and acquire an image that meets the image sampling requirements. In this process, the image acquisition instruction is divided into external image acquisition instructions and internal image acquisition instructions, and the generation time of the internal image acquisition instruction is predicted, so that the endoscope is synchronously in a relatively static state when the internal image acquisition instruction is sent to the image acquisition component, and an image that meets the image sampling requirements is acquired, effectively reducing the problem of poor image quality caused by jitter. Furthermore, the present application does not require additional mechanical structure, and uses the relatively static characteristics to predict the moment when the endoscope is in a relatively static state to complete image acquisition. It can be applied to any jitter scenario of the endoscope, thereby realizing anti-jitter image acquisition and improving image acquisition efficiency.

[0032] See also Figure 1 , Figure 1 The schematic diagram of the structure of an embodiment of an endoscope for restoring an image based on jitter data provided by the present application. The endoscope comprises: a host end, a photoelectric conversion module and an insertion structure. The host end, the photoelectric conversion module and the insertion structure cooperate to realize the technical solution formed by any one of the following embodiments or any combination of the embodiments.

[0033] The host side can be provided with an operation screen, a USB interface, a scanning terminal, a medical caster, a switch button, a camera handle bracket, a camera handle interface, and an LED indicator. Among them, the operation screen is used to display functions and provide an operation interface. The USB interface is used for external USB. The scanning terminal is used for scanning code recognition. The medical caster is used to move the host side. The switch button is used to turn the host side on and off. The camera handle bracket is used to hang the camera handle. The camera handle interface is used to connect the camera handle. The LED indicator is used to indicate the working status of the host side. If the indicator is on, the trolley is working; if the indicator is off, the trolley is not working.

[0034] Specifically, the entire system of the endoscope based on jitter data image restoration is as follows:

[0035] The camera module uses the FPGA board as the central control center to connect the barcode scanner, camera handle (CMOS-ISP board), camera serial port screen, video recording board and light source driver board to complete the barcode analysis, imaging parameter control, camera serial port screen control, video recording control and interaction with the light source driver board. Users can use the camera serial port screen to take photos, record, set the system time, set the language and restore factory settings. The light source module uses the light source driver board as the central control center to connect the light source serial port screen, temperature sensor, light source, FPGA board and start switch to adjust the light source brightness, monitor the light source temperature and start the light source driver board to power each module. The light generated by the light source module provides a light guide connected to the camera handle for clear imaging.

[0036] The corresponding software functions are as follows:

[0037] The functions included in the light source software are: power on, bulb switch function, brightness adjustment function, temperature display function, and running time function.

[0038] Light bulb switch function: The software can turn the light bulb on or off.

[0039] Brightness adjustment function: The software can adjust the brightness of the bulb from 10 to 100, with each level changing by 10.

[0040] Temperature display function: The software interface displays the current temperature of the bulb.

[0041] Running time function: The software interface displays the actual cumulative running time of the light source. When the lamp runs for more than 20,000 hours, a prompt box will appear.

[0042] The functions included in the camera system software are: power on, user access control, lock screen function, system setting function (including system time setting, language setting, and restoring factory settings), video recording function, photo taking function, and one-key enhancement function.

[0043] User access control: When the device is turned on, the boot UI is displayed and a progress bar is loaded. After the progress bar is loaded, the display screen jumps to the code scanning UI. Scan the QR code on the sheath to display the image.

[0044] Lock screen function: The camera system host can lock the operation interface to prevent accidental touches.

[0045] Product system setting function:

[0046] 1 System time setting: You can set the system time in 24-hour format.

[0047] 2 Language setting: You can set the language, supporting Chinese and English.

[0048] 3 Restore factory settings: Restore factory settings.

[0049] Video recording function: insert the USB storage medium, the product has the video recording function, and can be stored in real time through USB, the storage format is .ts.

[0050] Photo function: insert the USB storage medium, the product has the photo function, which can be stored via USB, and the storage format is JPG.

[0051] One-key enhancement function: After pressing the button, the image is adjusted to the preset value. For the brightness of the picture, select one-key enhancement to adjust the overall brightness of the picture.

[0052] Camera system software, including power-on, user access control, screen lock, photo taking, video recording and one-key enhancement functions.

[0053] The entire image processing link includes: sensor board, cable, sensor adapter board, FPGA (Field Programmable Gate Array) mainboard and display.

[0054] Sensor board: uses a chip to convert the light signal irradiated on the sensor into an electrical signal, and outputs the electrical signal through the rear-end cable.

[0055] Cable: Transmits electrical signals and provides a power path for the Sensor board.

[0056] Sensor adapter board: The core of the sensor adapter board is an FPGA chip. Its main function is to sample and convert the electrical signals transmitted by the sensor board into serial signals, and convert the serialized data into optical signals through the optoelectronic conversion module, and transmit the signals to the host through optical fiber.

[0057] FPGA main board: Receives the optical signal transmitted from the sensor adapter board and converts it into an electrical signal through the photoelectric conversion module. It then processes the original image into a target image through a series of image ISP algorithms and reconstructs the image to output in a specified format.

[0058] Display: Display the processed image.

[0059] like Figure 2As shown, the camera handle includes a photoelectric conversion module. The light source exciter in the host end provides an incident light source, which is connected to the inside of the handle from the handle tail end 60 through the light source optical fiber on the host end, and then extends along the optical fiber 40 inside the handle to the light source incident end 50, and vertically incident on the photoelectric conversion module 20 (handle fluorescence module) upward, and is incident on the surface of the living tissue at the right after a 90° total reflection through the dichroic mirror set at a 45° angle inside the handle fluorescence module. After the fluorescence excitation on the surface of the living tissue, the reflected light is formed, and the reflected light returns to the dichroic mirror along the original path. At this time, due to the excitation change in the band, the reflected light is all transmitted through the dichroic mirror and then transmitted to the rear-end photoelectric conversion module. After being received by the photoelectric conversion module, the optical signal is converted into an electrical signal. In order to prevent electromagnetic interference, the handle will perform electro-optical conversion again, and the electrical signal will be converted into an optical signal again. The optical signal is transmitted to the image signal processor in the machine through the return optical fiber signal line, and then the photoelectric conversion is performed for the second time to convert the returned optical signal into an electrical signal. Then, through the processing of the electrical signal, a microscopic image magnified more than 1000 times is output on the display.

[0060] See also Figure 3 , Figure 3 1 is a flow chart of an embodiment of an endoscope anti-shake method for restoring an image based on shake data provided by the present application. The method comprises:

[0061] Step 31: In response to an external image acquisition instruction, obtain the jitter data of the endoscope; wherein the external image acquisition instruction is generated by external control.

[0062] In some embodiments, a jitter data sensor is provided in the insertion structure of the endoscope. The jitter data sensor can be used to collect corresponding jitter data. Therefore, in response to the external image acquisition instruction, the jitter data collected by the jitter data sensor is obtained. In some embodiments, the jitter data sensor can be provided at the front end of the insertion structure. If it is provided near the entrance pupil end of the insertion structure, more accurate jitter data can be collected.

[0063] In some embodiments, the external image acquisition instruction may be generated by manual triggering, such as setting a trigger button on the outside of the endoscope, and generating the external image acquisition instruction by pressing the trigger button.

[0064] Step 32: Predict the generation time of the internal image acquisition instruction according to the jitter data and the instruction transmission time.

[0065] The instruction transmission time indicates the time required for the internal image acquisition instruction to be sent to the image acquisition component.

[0066] It is understandable that the control terminal in the endoscope is composed of corresponding electronic components. Therefore, the transmission of instructions between electronic components also takes time. From the generation of external image acquisition instructions to the end of image acquisition, this takes time.

[0067] The shaking of the endoscope also requires a corresponding shaking time from the beginning to the end of the shaking.

[0068] Based on this, the generation time of the internal image acquisition instruction can be predicted according to the jitter data and the instruction transmission time.

[0069] In some embodiments, the time obtained by adding the generation time of the internal image acquisition instruction to the instruction transmission time is exactly the time when the door mirror shaking ends.

[0070] Step 33: Generate an internal image acquisition instruction according to the generation time, and use the internal image acquisition instruction to control the image acquisition component to perform image acquisition.

[0071] The endoscope is relatively still during image acquisition, and the images acquired according to the internal image acquisition instructions meet the sampling requirements.

[0072] In this embodiment, when an external image acquisition instruction is received, the jitter data of the endoscope is synchronously acquired; and then the generation time of the internal image acquisition instruction is predicted based on the jitter data and the instruction transmission time; wherein the instruction transmission time indicates the time required for the internal image acquisition instruction to be sent to the image acquisition component; so as to generate the internal image acquisition instruction according to the generation time, and use the internal image acquisition instruction to control the image acquisition component to perform image acquisition when the endoscope is relatively still, and acquire an image that meets the image sampling requirements. In this process, the image acquisition instruction is divided into external image acquisition instructions and internal image acquisition instructions, and the generation time of the internal image acquisition instruction is predicted, so that when the internal image acquisition instruction is sent to the image acquisition component, the endoscope is synchronously in a relatively still state, and an image that meets the image sampling requirements is acquired, effectively reducing the problem of poor image quality caused by jitter. Furthermore, the present application does not require additional mechanical structure, and uses the relatively still characteristic to predict the moment when the endoscope is relatively still to complete image acquisition, and can be applied to any jitter scenario of the endoscope, thereby achieving anti-jitter image acquisition and improving image acquisition efficiency.

[0073] See also Figure 4 , Figure 4 1 is a flow chart of another embodiment of an endoscope anti-shake method for restoring an image based on shake data provided by the present application. The method comprises:

[0074] Step 41: In response to an external image acquisition instruction, obtain the jitter data of the endoscope; wherein the external image acquisition instruction is generated by external control.

[0075] In some embodiments, a jitter data sensor is provided in the insertion structure of the endoscope. The jitter data sensor can be used to collect corresponding jitter data. Therefore, in response to the external image acquisition instruction, the jitter data collected by the jitter data sensor is obtained. In some embodiments, the jitter data sensor can be provided at the front end of the insertion structure. If it is provided near the entrance pupil end of the insertion structure, more accurate jitter data can be collected.

[0076] Step 42: Predict a first time duration required for the endoscope to be in a relatively static state from a current state according to the jitter data.

[0077] In some embodiments, shaking of the endoscope may be caused by external manipulation.

[0078] In some embodiments, the shaking of the endoscope may be caused by the environment in which the endoscope is located. For example, when an endoscope is used for gastroscopy, the shaking of the endoscope may be caused by gastric activity.

[0079] Regardless of the factors causing the endoscope's shaking, the endoscope and the environment it is in are in a relatively static state. When the endoscope is in a relatively static state, the endoscope can actually be used to collect images, and the images collected at this time are not affected by the shaking.

[0080] Based on this, a corresponding duration prediction model can be used to predict the first duration required for the endoscope to change from a current state to a relatively still state according to the jitter data.

[0081] Step 43: predicting the generation time of the internal image acquisition instruction according to the first duration and the instruction transmission time.

[0082] The instruction transmission time indicates the time required for the internal image acquisition instruction to be sent to the image acquisition component.

[0083] After the first duration is determined, the generation time of the internal image acquisition instruction can be predicted based on the first duration and the instruction transmission time.

[0084] In some embodiments, see Figure 5 , step 43 may be the following process:

[0085] Step 431: Determine a target time when the endoscope is relatively still based on the first duration and the current time.

[0086] In some embodiments, the current time is added to the first time duration to obtain the target time when the endoscope changes from a shaking state to a relatively still state.

[0087] In some embodiments, see Figure 6 , step 431 may be the following process:

[0088] Step 61: Obtain the transmission duration of the jitter data.

[0089] In some embodiments, the jitter data is also collected and transmitted by the sensor, so the transmission duration of the jitter data needs to be included in the calculation of the target time.

[0090] Step 62: Determine the target time when the endoscope is relatively still according to the first time duration, the transmission time duration and the current time.

[0091] Among them, the first duration is longer than the transmission duration. This is because it takes time from the sensor collecting jitter data to transmission prediction. However, the prediction is made using jitter data, and in fact, the actual jitter data has changed at this time. Therefore, the transmission duration needs to be included in the calculation of the target time.

[0092] Combination Figure 7 To explain:

[0093] For example, the jitter data is collected at time A1, and the jitter data is transmitted to the prediction module at time B1. The prediction module predicts the time C1 when the endoscope is in a relatively static state from the jitter state of the jitter data based on the jitter data. However, in fact, the time C1 has lagged behind, and the time when the endoscope is in a relatively static state from the jitter state of the jitter data is D1. The time length between D1 and C1 is the transmission time length of the jitter data.

[0094] That is, step 62 can be expressed by the following formula:

[0095] T1=T0+T2-T3. T1 represents the target time, T0 represents the current time, T2 represents the first duration, and T3 represents the transmission duration. For example, the target time is the 10th millisecond. The instruction transmission time is 1 millisecond. Then the generation time of the internal image acquisition instruction is the 9th millisecond.

[0096] Step 432: Use the target time and the instruction transmission time to predict the generation time of the internal image acquisition instruction.

[0097] Since the instruction transmission takes time, the generation time of the internal image acquisition instruction can be obtained by subtracting the instruction transmission time from the target time. The target time is the 10th millisecond. The instruction transmission time is 1 millisecond. Then the generation time of the internal image acquisition instruction is the 9th millisecond.

[0098] Step 44: Generate an internal image acquisition instruction according to the generation time, and use the internal image acquisition instruction to control the image acquisition component to perform image acquisition.

[0099] The endoscope is relatively still during image acquisition, and the images acquired according to the internal image acquisition instructions meet the sampling requirements.

[0100] In this embodiment, when an external image acquisition instruction is received, the jitter data of the endoscope is synchronously acquired; and then the generation time of the internal image acquisition instruction is predicted based on the jitter data and the instruction transmission time; wherein the instruction transmission time indicates the time required for the internal image acquisition instruction to be sent to the image acquisition component; so as to generate the internal image acquisition instruction according to the generation time, and use the internal image acquisition instruction to control the image acquisition component to perform image acquisition when the endoscope is relatively still, and acquire an image that meets the image sampling requirements. In this process, the image acquisition instruction is divided into external image acquisition instructions and internal image acquisition instructions, and the generation time of the internal image acquisition instruction is predicted, so that when the internal image acquisition instruction is sent to the image acquisition component, the endoscope is synchronously in a relatively still state, and an image that meets the image sampling requirements is acquired, effectively reducing the problem of poor image quality caused by jitter. Furthermore, the present application does not require additional mechanical structure, and uses the relatively still characteristic to predict the moment when the endoscope is relatively still to complete image acquisition, and can be applied to any jitter scenario of the endoscope, thereby achieving anti-jitter image acquisition and improving image acquisition efficiency.

[0101] Furthermore, when determining the moment of relative stillness, the signal transmission time and data transmission time of the endoscope are fully taken into consideration. The same part of the endoscope also changes from jitter to relative stillness. Therefore, the signal transmission time and data transmission time need to be removed to accurately determine the moment when the endoscope is in relative stillness, and then when the endoscope is relatively still, the internal image acquisition command is sent to the image acquisition component to complete image acquisition without the influence of jitter.

[0102] In some embodiments, the above step 42 may be performed in the following manner: input the jitter data into a duration prediction model to obtain a first duration predicted by the duration prediction model for the endoscope to change from a current state to a relatively still state.

[0103] In some embodiments, the duration prediction model can be obtained by collecting corresponding jitter data and the first duration as samples for training in advance. The duration prediction model can be constructed based on a neural network.

[0104] In some embodiments, the duration prediction model includes a state prediction network, a duration prediction network, and a comprehensive prediction network. Figure 8 To explain:

[0105] Step 81: Input the jitter data and the transmission duration of the jitter data into the state prediction network to obtain the current state of the endoscope output by the state prediction network.

[0106] In some embodiments, the current state of the endoscope represents the state at the time the dither data is input to the state prediction network.

[0107] In some embodiments, the current state of the endoscope represents the state of the endoscope at the time when the state prediction network outputs the result.

[0108] Step 82: Input the jitter data and the transmission duration of the jitter data into the duration prediction network, and obtain the first target duration required for the endoscope to remain relatively still, which is output by the duration prediction network.

[0109] Step 83: Input the current state of the endoscope and the first target duration into the comprehensive prediction network, and obtain the first duration required for the endoscope to remain relatively still, which is output by the comprehensive prediction network.

[0110] The comprehensive prediction network can make a prediction based on the current state and the first target duration, and output the first duration required for the endoscope to remain relatively still.

[0111] In some embodiments, see Fig. 9 , step 83 may be the following process:

[0112] Step 831: Utilize the comprehensive prediction network to predict the second target duration required for the endoscope to remain relatively still according to the current state of the endoscope.

[0113] The comprehensive prediction network is used to predict the second target duration required for the endoscope to be relatively still according to the current state of the endoscope, which can assist in verifying the accuracy of the state prediction network and the accuracy of the duration prediction network.

[0114] Step 832: Utilize the comprehensive prediction network to output the first time duration required for the endoscope to remain relatively still based on the first target time duration and the second target time duration.

[0115] In some embodiments, the first target duration and the second target duration may be weighted in combination with the accuracy of the state prediction network and the accuracy of the duration prediction network to obtain the first duration required for the endoscope to remain relatively still.

[0116] In this embodiment, the state prediction network, the duration prediction network and the comprehensive prediction network are used to comprehensively predict the first duration required for the endoscope to be relatively still, which can improve the prediction accuracy and thus improve the clarity of the subsequently collected images.

[0117] See also Fig.10 , Fig.10 1 is a schematic diagram of a computer readable storage medium according to an embodiment of the present application. The computer readable storage medium 110 is used to store a computer program 111. When the computer program 111 is executed by a processor, it is used to implement the following method steps:

[0118] In response to an external image acquisition instruction, jitter data of the endoscope is acquired; wherein the external image acquisition instruction is generated by external control; the generation time of the internal image acquisition instruction is predicted according to the jitter data and the instruction transmission time; the internal image acquisition instruction is generated according to the generation time, and the image acquisition component is controlled by the internal image acquisition instruction to perform image acquisition; wherein the image acquired according to the internal image acquisition instruction meets the sampling requirements.

[0119] It can be understood that when the computer program 111 is executed by the processor, it is also used to implement the method of any of the above embodiments.

[0120] In summary, the endoscope, anti-jitter method and medium for restoring images based on jitter data of the present application, in this embodiment, when receiving an external image acquisition instruction, synchronously obtain the jitter data of the endoscope; and then predict the generation time of the internal image acquisition instruction according to the jitter data and the instruction transmission time; wherein the instruction transmission time represents the time required for the internal image acquisition instruction to be sent to the image acquisition component; so as to generate the internal image acquisition instruction according to the generation time, and use the internal image acquisition instruction to control the image acquisition component to perform image acquisition when the endoscope is relatively still, and collect images that meet the image sampling requirements. In this process, the image acquisition instruction is divided into external image acquisition instructions and internal image acquisition instructions, and the generation time of the internal image acquisition instruction is predicted, so that when the internal image acquisition instruction is sent to the image acquisition component, the endoscope is synchronously in a relatively still state, and an image that meets the image sampling requirements is collected, effectively reducing the problem of poor image quality caused by jitter. Furthermore, the present application does not need to add an additional mechanical structure, and uses the relatively still characteristics to predict the moment when the endoscope is relatively still to complete image acquisition, which can be applied to any jitter scenario of the endoscope, thereby realizing anti-jitter image acquisition and improving image acquisition efficiency.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program codes.

[0125] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An endoscope anti-shake method for restoring an image based on shake data, characterized in that: The method comprises: In response to an external image acquisition instruction, acquiring jitter data of the endoscope; wherein the external image acquisition instruction is generated by external control; Inputting the jitter data into a duration prediction model to obtain a first duration required for the endoscope to change from a current state to a relatively still state predicted by the duration prediction model; Predicting the generation time of the internal image acquisition instruction according to the first duration and the instruction transmission time; wherein the instruction transmission time indicates the duration required for the internal image acquisition instruction to be sent to the image acquisition component; generating the internal image acquisition instruction according to the generation time, and controlling the image acquisition component to perform image acquisition using the internal image acquisition instruction; wherein the endoscope is relatively still during image acquisition; Wherein, the duration prediction model includes a state prediction network, a duration prediction network and a comprehensive prediction network; the jitter data is input into the duration prediction model to obtain the first duration required for the endoscope to change from the current state to relative stillness predicted by the duration prediction model, including: inputting the jitter data and the transmission duration of the jitter data into the state prediction network to obtain the current state of the endoscope output by the state prediction network; inputting the jitter data and the transmission duration of the jitter data into the duration prediction network to obtain the first target duration required for the endoscope to be relatively still output by the duration prediction network; using the comprehensive prediction network to predict the second target duration required for the endoscope to be relatively still according to the current state of the endoscope; using the comprehensive prediction network to output the first duration required for the endoscope to be relatively still according to the first target duration and the second target duration.

2. The method according to claim 1, characterized in that The insertion structure of the endoscope is provided with a jitter data sensor; the step of obtaining the jitter data of the endoscope in response to an external image acquisition instruction comprises: In response to an external image acquisition instruction, the jitter data acquired by the jitter data sensor is acquired.

3. The method according to claim 1, characterized in that The predicting the generation time of the internal image acquisition instruction according to the first duration and the instruction transmission time includes: Determine a target time when the endoscope is relatively still according to the first time length and the current time; The target time and the instruction transmission time are used to predict the generation time of the internal image acquisition instruction.

4. The method according to claim 3, characterized in that The step of determining the target time when the endoscope is relatively still according to the first time duration and the current time includes: Obtaining the transmission duration of the jitter data; A target time when the endoscope is relatively still is determined according to the first time duration, the transmission time duration and the current time.

5. An endoscope for restoring an image based on jitter data, characterized in that: The endoscope includes a host end, a photoelectric conversion module and an insertion structure; the host end, the photoelectric conversion module and the insertion structure cooperate to implement the method described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 4.

Images