Endoscopic image abnormality detection and processing method and system based on FPGA

The endoscopic image abnormalities are detected and processed through the FPGA control component, and resetting them for different abnormal situations, solving the problem that the endoscopic image processing system is unavailable in the prior art, and achieving rapid recovery and image of the system The stability and continuity of the output support hot plugging of different input sources.

CN115098338BActive Publication Date: 2024-11-22SUZHOU JINGGUAN MEDICAL TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210870011.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-22
Publication Date
2024-11-22
Estimated Expiration
2042-07-22

AI Technical Summary

Technical Problem

The existing endoscopic image abnormality processing methods will cause the system to be unavailable for a short time when the system is abnormal during the operation, and the watchdog mechanism cannot detect all abnormalities, resulting in unstable image processing.

Method used

Using the endoscopic image abnormality detection and processing method based on FPGA, the FPGA control component continuously detects the connection signal, data flow format, image decoding processing and output image random noise and brightness of the endoscopic mirror body, and performs different abnormal situations. Targeted processing and reset to realize the rapid recovery and image output of the system.

Benefits of technology

Improve the stability and anti-interference ability of the endoscopic image processing system, realize the hot plug function of different input sources, avoid the problem of the system being unavailable for a short time under abnormal conditions, and ensure the continuity of image output and quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115098338B_ABST
    Figure CN115098338B_ABST
Patent Text Reader

Abstract

The present invention provides an FPGA-based endoscope image abnormality detection and processing method and system, the method includes step 1: connecting the endoscope body to the FPGA control component; step 2: reading the connection signal / body information identification of the endoscope body, if it can be done, proceed to step 3, if it cannot be done, proceed to the hot plug processing flow; step 3: detecting the input data stream, if it is normal, proceed to step 4, if it is not or it is wrong, proceed to the input data stream abnormal processing flow; step 4: decoding / processing the image data, if it is correct, proceed to step 5, if it is wrong, proceed to the image decoding / processing abnormal processing flow; step 5: detecting the random noise and overall brightness of the output image, if it is correct, proceed to step 6, if it is wrong, proceed to the output image abnormal processing flow; step 6: output the image. The present invention can ensure that the system returns to normal state in a short time while not affecting the output display of the normal output image, and can perform hot plugging at the same time through targeted detection and processing of abnormal situations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an endoscopic image abnormality detection and processing method and system based on FPGA, belonging to the field of image processing. Background Art

[0002] Endoscope is a detection instrument that integrates traditional optics, ergonomics, precision machinery, modern electronics, mathematics, software, etc. It can enter the stomach through the mouth or other natural channels into the body. The endoscope can be used to see images of organs inside the human body and show lesions that X-rays cannot show, so it is very useful for doctors.

[0003] Medical endoscopes are often used in surgery, so there are high requirements for the stability and quality of the transmitted images. It is necessary to ensure that abnormalities in endoscopic images are detected and processed in a timely manner during the working process, so as to obtain stable and effective images. The existing endoscope abnormality processing method usually adopts a watchdog mechanism to restart the entire system after detecting the abnormality. However, if the system has an abnormality during the operation and simply restarts the system, the endoscopic image processing system will be unavailable for a short period of time, unable to operate / process the emergency, and may cause serious consequences. In addition, the watchdog mechanism only monitors the operating status and cannot detect all abnormalities.

[0004] In view of this, it is necessary to propose a new FPGA-based endoscopic image abnormality detection and processing method to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide an endoscopic image abnormality detection and processing method and system based on FPGA, so as to improve the stability and anti-interference ability of the endoscopic image processing system and realize the hot-swap function of different input sources.

[0006] To achieve the above object, the present invention provides an FPGA-based endoscope image abnormality detection and processing method, comprising:

[0007] Step 1: Connect the endoscope body to the FPGA control component;

[0008] Step 2: The FPGA control component continuously reads the connection signal / scope identification information of the endoscope, and if it can be read, it proceeds to step 3; if it cannot be read, it enters the hot plug processing flow;

[0009] Step 3: Continuously detect the data stream input by the endoscope body. If the format of the input data stream is normal, proceed to step 4. If there is no input data stream or the format of the input data stream is wrong, proceed to the input data stream exception handling process.

[0010] Step 4: Continue to decode / process the image data of the data stream input by the endoscope body. If the image data decoding / processing is correct, proceed to step 5. If the image data decoding / processing is wrong, proceed to the image decoding / processing exception handling process.

[0011] Step 5: Continuously detect the random noise and overall brightness of the output image. If the random noise and overall brightness of the output image are normal, proceed to step 6. If the random noise / overall brightness of the output image is abnormal, proceed to the output image abnormality processing flow.

[0012] Step 6: The FPGA controls the component to output images to the display.

[0013] As a further improvement of the present invention, the FPGA control component includes an IIC configuration module, and the endoscope body includes a sensor. In step 2, the FPGA control component identifies the type of the inserted endoscope body through the IIC configuration module, and configures the register corresponding to the sensor according to the identified type of the endoscope body to initialize the sensor.

[0014] As a further improvement of the present invention, the hot plug processing flow in step 2 includes:

[0015] Step 21: Reset the data decoding module and the ISP algorithm module in the FPGA control component;

[0016] Step 22: Stop outputting images, and prompt and wait for the user to replug and replug the endoscope body.

[0017] As a further improvement of the present invention, the FPGA control component includes a data decoding module, an ISP algorithm module, a video cache module and an image output module connected in sequence, and the input data stream exception processing flow in step 3 includes:

[0018] Step 31: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module;

[0019] Step 32: The data decoding module and the ISP algorithm module are reset;

[0020] Step 33: The sensor is powered off and reset;

[0021] Step 34: The sensor is powered on, reset is released, and registers of the sensor are reconfigured through the IIC configuration module;

[0022] Step 35: The data decoding module and the ISP algorithm module are reset and output the image again.

[0023] As a further improvement of the present invention, the FPGA control component includes a data decoding module, an ISP algorithm module, a video cache module and an image output module connected in sequence, and the image decoding / processing exception handling process in step 4 includes:

[0024] Step 41: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module;

[0025] Step 42: The data decoding module and the ISP algorithm module are reset;

[0026] Step 43: The data decoding module and the ISP algorithm module end resetting and re-output the image.

[0027] As a further improvement of the present invention, the FPGA control component includes a data decoding module, an ISP algorithm module, a video cache module and an image output module connected in sequence, and the output image abnormality processing flow in step 5 includes:

[0028] Step 51: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module;

[0029] Step 52: The data decoding module and the ISP algorithm module are reset;

[0030] Step 53: The sensor is powered off and reset;

[0031] Step 54: The sensor is powered on, reset is released, and registers of the sensor are reconfigured through the IIC configuration module;

[0032] Step 55: The data decoding module and the ISP algorithm module are reset and output the image again.

[0033] As a further improvement of the present invention, the method for determining whether the overall brightness of the output image is abnormal in step 5 includes:

[0034] Step 501: Check the overall brightness average value of the output image and set a brightness threshold of the overall brightness average value. If the overall brightness average value of the output image within time T1 is lower than the brightness threshold, it is determined that the image brightness is abnormal.

[0035] As a further improvement of the present invention, the method for determining the random noise of the output image in step 5 includes:

[0036] Step 502: Use a sliding window of a preset size to check the pixels of the output image, check the proportion of random noise in the output image, set a threshold for the proportion of pure color blocks in the output image, and if within time T2, the overall proportion of the pure color blocks in the output image exceeds the threshold, the image data is considered abnormal.

[0037] As a further improvement of the present invention, the FPGA control component includes an IIC configuration module, a data decoding module, an ISP algorithm module, a video cache module and an image output module. The FPGA-based endoscope image abnormality detection and processing method also includes step 7: if the step 6 cannot output the image to the display / the image displayed on the display is abnormal, the image output module keeps outputting the last frame of normal output image obtained from the video cache module unchanged, all modules of the FPGA control component except the image output module and the video cache module are reset, and the endoscope body is initialized to re-output the image.

[0038] The present invention also proposes an FPGA-based endoscope image anomaly detection and processing system, which is suitable for the above-mentioned FPGA-based endoscope image anomaly detection and processing method, including an FPGA control component, an endoscope body and a display connected to the FPGA control component, the FPGA control component including an IIC configuration module, a data decoding module, an ISP algorithm module, a video cache module and an image output module, the IIC configuration module and the data decoding module are electrically and signal-connected to the endoscope body, the data decoding module is signal-connected to the ISP algorithm module, the ISP algorithm module is signal-connected to the video cache module, the video cache module is signal-connected to the image output module, and the image output module is signal-connected to the display.

[0039] The beneficial effects of the present invention are as follows: the FPGA-based endoscope image anomaly detection and processing method and system of the present invention performs targeted detection and processing of various abnormal situations during the endoscopic image imaging process, resets some modules with problems, ensures that the system can return to normal in a short time after an abnormality / interference occurs, and does not affect the output display of normal output images. At the same time, it ensures that different input sources can be hot-swapped in any state, thereby solving the problem that the existing endoscopic image anomaly detection and processing methods have a single mode for detecting abnormal situations and a rough processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a step diagram of the FPGA-based endoscope image abnormality detection and processing method of the present invention.

[0041] Figure 2It is a flow chart of anomaly detection of the present invention.

[0042] Figure 3 It is a flowchart of the exception handling of the present invention.

[0043] Figure 4 It is a structural schematic diagram of the FPGA-based endoscope image abnormality detection and processing system of the present invention.

[0044] Figure 5 It is a flow chart of the watchdog mechanism of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 1 to Figure 3 As shown, the present invention discloses an FPGA-based endoscope image abnormality detection and processing method, which uses an FPGA-based FPGA control component to detect and process abnormal conditions of transmitted images during the operation of an endoscope device, including:

[0047] Step 1: Connect the endoscope body to the FPGA control component;

[0048] Step 2: The FPGA control component continuously reads the connection signal / scope identification information of the endoscope body, and if it can be read, it proceeds to step 3; if it cannot be read, it enters the hot plug processing flow;

[0049] Step 3: Continuously detect the data stream input by the endoscope body. If the format of the input data stream is normal, proceed to step 4. If there is no input data stream or the format of the input data stream is wrong, proceed to the input data stream exception handling process.

[0050] Step 4: Continue to decode / process the image data of the data stream input by the endoscope body. If the image data decoding / processing is correct, proceed to step 5. If the image data decoding / processing is wrong, proceed to the image decoding / processing exception handling process;

[0051] Step 5: Continue to detect the random noise and overall brightness of the output image. If the random noise and overall brightness of the output image are normal, proceed to step 6. If the random noise / overall brightness of the output image is abnormal, proceed to the output image abnormality processing flow.

[0052] Step 6: The FPGA controls the component to output images to the display.

[0053] Step 1 is the beginning stage of the operation of the entire endoscopic imaging device. When preparing to use the endoscopic imaging device for work, the endoscope body is connected to the FPGA control component, and the image is collected through the endoscope body, and the acquired image data is transmitted to the FPGA control component. When installing or replacing a conventional endoscope body, it is necessary to turn off the power. However, due to the use of the FPGA-based endoscope image anomaly detection and processing method of the present application, endoscope bodies of different specifications can be directly plugged in and out during the operation of the entire FPGA-based endoscope image anomaly detection and processing system, realizing the hot-swap function of different input sources, improving the convenience of endoscope body installation, and saving time.

[0054] Then enter step 2, the FPGA control component includes an IIC configuration module, and the endoscope body includes a sensor. The FPGA control component needs to first continuously read the connection signal / body identification information of the endoscope body. When the model of the endoscope body and the continuously transmitted signal can be identified, the IIC configuration module in the FPGA control component configures the register of the sensor in the endoscope body according to the type of the inserted endoscope body identified, and initializes the sensor in the endoscope body. The sensor in the endoscope body is an image sensor. Image sensors have a variety of specifications and models, which need to be identified and judged, so that the IIC configuration module configures different registers. Configuring different registers is the initialization configuration method for most sensors, and reconfiguring the register of the sensor using the IIC configuration module is equivalent to reinitializing the sensor.

[0055] If the connection signal / scope identification information of the endoscope cannot be continuously read, it means that the endoscope is not installed correctly or the endoscope model is not suitable for the entire endoscope system. At this time, it is judged that the endoscope and the FPGA control component are disconnected and enter the hot plug processing process, which is as follows:

[0056] Step 21: Reset the data decoding module and ISP algorithm module in the FPGA control component;

[0057] Step 22: Stop outputting images, and prompt and wait for the user to replug and plug in the endoscope body.

[0058] The hot-swap processing flow allows the user to re-plug the endoscope body to ensure the correct connection between it and the FPGA control component, and reset the data decoding module and ISP algorithm module so that the data transmitted by the endoscope body can be re-processed to generate an image.

[0059] After confirming that the endoscope body is correctly connected to the FPGA control component in step 2, step 3 is entered to continuously detect the data stream input by the endoscope body. If the input data stream is abnormal, that is, there is no data stream input of the image data returned by the sensor of the endoscope body or the format of the input data stream is wrong, so that the data decoding module and the ISP algorithm module cannot decode and generate an image, indicating that the sensor of the endoscope body used to input image data is abnormal at this time. At this time, the input data stream abnormality processing process is entered, which specifically includes:

[0060] Step 31: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module unchanged;

[0061] Step 32: Reset the data decoding module and ISP algorithm module;

[0062] Step 33: The sensor is powered off and reset;

[0063] Step 34: Power on the sensor, release the reset, and reconfigure the sensor registers through the IIC configuration module;

[0064] Step 35: The data decoding module and the ISP algorithm module are reset and the image is output again.

[0065] The input data stream exception handling process implements the sensor reset operation by cutting off the power to the endoscope lens and resetting the sensor in the endoscope body, resetting the sensor status and information, and resetting the data decoding module and ISP algorithm module that process the image data input by the sensor, to ensure that the data stream input by the sensor in the endoscope body remains valid and normal, and that the reset of the data decoding module and ISP algorithm module can work normally, and parse the data stream input by the sensor into a correct image.

[0066] When the data stream input by the endoscope is ensured to be normal through step 3, step 4 is entered to continuously decode / process the image data of the data stream input by the endoscope. When the image data decoding / processing error occurs, it is mainly manifested in that the data decoding module cannot decode the data stream input by the endoscope and / or the ISP algorithm module cannot process the decoded data stream to generate an image, thereby failing to output a correct image. This situation indicates that one or more of the data decoding module for decoding the data stream transmitted by the sensor and the ISP algorithm module for synthesizing the image after decoding the data stream of the sensor are abnormal, which will lead to the failure to generate a correct image. Therefore, when the abnormal situation is detected, the image decoding / processing abnormality handling process is entered, which is specifically:

[0067] Step 41: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module unchanged;

[0068] Step 42: Reset the data decoding module and the ISP algorithm module;

[0069] Step 43: The data decoding module and the ISP algorithm module finish resetting and re-output the image.

[0070] By resetting the data decoding module and the ISP algorithm module, the functions of both are restored to ensure that both functions are correct and can correctly process the data stream transmitted by the sensor.

[0071] After passing the abnormality detection of steps 2 to 4, the generated image is usually output through the ISP algorithm module, so that the image can be output on the display screen, but the correctness of the output image cannot be guaranteed. In actual use testing, one of the abnormal conditions is the continuous output of erroneous image data with correct format but incorrect data, that is, the output image is generated at this time, but the output image has errors. For abnormality detection in this case, it is necessary not only to check the correctness of the data stream format of the image data, but also to check the rationality of the image data as a whole.

[0072] In normal use scenarios, the output image will not be pure black, and the overall brightness of the output image will not remain close to 0; normal output images have random noise, and large areas of pure color blocks are not possible. Therefore, in normal use scenarios, the image random noise and overall image brightness are continuously monitored. If the image random noise is abnormal for a long time / the overall image brightness is abnormal, it means that the sensor has been sending a data stream of image data with correct format but wrong data for a long time, that is, at least one of the sensor and / or data decoding module and / or ISP algorithm module reset is abnormal. At this time, enter step 5 of the output image abnormality processing flow, which is specifically:

[0073] Step 51: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module;

[0074] Step 52: Reset the data decoding module and ISP algorithm module;

[0075] Step 53: The sensor is powered off and reset;

[0076] Step 54: Power on the sensor, release the reset, and reconfigure the sensor registers through the IIC configuration module;

[0077] Step 55: The data decoding module and the ISP algorithm module are reset and output the image again.

[0078] Among them, the abnormality of the average value of the overall brightness of the output image in the abnormality detection stage of step 5 is specifically:

[0079] Step 501: Check the overall brightness average of the output image and set a brightness threshold of the overall brightness average. If the overall brightness average of the output image within time T1 is lower than the brightness threshold, it is determined that the image brightness is abnormal.

[0080] In the embodiment of the present application, the T1 time is the time for inputting 20 frames of images, and the brightness threshold is the gray value average value lower than 1. After entering step 5, through the abnormal detection of the overall brightness average value of the output image, it is determined whether the overall brightness average value of the output image within the T1 time is lower than the brightness threshold. When it is lower than the brightness threshold, the output image continues to be black, and the patient's condition information cannot be obtained based on the output image. At this time, it is considered that there is an abnormality in the overall brightness of the image, and the output image abnormality processing flow is entered. After processing, the overall brightness of the output image is restored to normal to ensure the effectiveness of the output image.

[0081] The specific abnormality of long-term random noise in the output image in the abnormality detection stage of step 5 is:

[0082] Step 502: Use a sliding window of a preset size to check the pixels of the output image, check the proportion of random noise in the output image, set a threshold for the proportion of pure color blocks in the output image, and if the overall proportion of pure color blocks in the output image exceeds the threshold within T2, the image data is considered abnormal.

[0083] In the embodiment of the present application, the sliding window of the preset size is specifically a window of 9*9 pixels. The pure color block refers to a color block in which the RGB three-channel values ​​of all pixels in the window are the same and less than 200. The threshold value of the pure color block is that the pure color block pixel ratio exceeds 50% of the output image pixels. The T2 time is the time for inputting 20 frames of images. In the image display, the full range of the RGB three-channel value is 0-255. The closer the value is to 255, the closer the color is to white. The closer the value is to 0, the closer the color is to black. The threshold limit of the RGB three-channel of the pure color block is added to avoid misjudgment in normal use scenarios, especially before the actual use of the endoscope, there is often a manual white balance operation. In this scenario, the image appears to be full-screen close to pure white, that is, the RGB three-channel values ​​are close to 255. In the abnormal detection process, it is necessary to avoid misjudgment due to the detection of a large number of pure white blocks during the manual white balance process. The abnormality of the image with a large number of pure color blocks in the output image is usually manifested as pure black blocks / pure gray blocks / non-pure white blocks of other colors with the same RGB three-channel values. At this time, the overall detail loss of the output image is serious / no details, and no abnormality of pure white blocks (that is, RGB three-channel values ​​close to 255) has been found. Therefore, if the upper limit of the threshold for abnormal pure color blocks is set reasonably, such abnormalities can be detected. By sliding a sliding window of 9*9 pixels on the output image, the proportion threshold of pure color blocks in the output image is counted to realize the abnormal detection of long-term random noise in the output image. When the proportion threshold exceeds 50% of the pixels of the output image, there is a large area of ​​pure color blocks in the output image, and the details are seriously lost. Doctors cannot obtain effective information from the output image, which is easy to cause misjudgment.

[0084] Step 5 continuously detects the random noise and overall brightness of the output image converted from the data stream input by the endoscope body, enters the output image abnormality processing flow after an abnormality occurs, cuts off the power to the endoscope lens and resets the sensor in the endoscope body to implement the sensor reset operation, reset the sensor status and information, and reset the data decoding module and ISP algorithm module that process the image data input by the sensor at the same time, to ensure that the output image generated by the data stream input by the sensor in the endoscope body is correct and has no abnormalities.

[0085] After completing the abnormal situation detection in steps 2 to 5, the correct output image can be output, and the image is output to the display by the FPGA control component through step 6, thereby completing the image transmission of the endoscope.

[0086] The FPGA-based endoscope image anomaly detection and processing method of the present application also includes step 7: if step 6 fails to output an image to the display / the image displayed on the display is abnormal, the image output module keeps outputting the last frame of normal output image obtained from the video cache module unchanged, all modules of the FPGA control component except the image output module and the video cache module are reset, and the endoscope body is initialized, so as to ensure that each module works correctly and the output image is correct and reliable.

[0087] In actual work, there may be abnormal situations different from steps 2 to 5. For these abnormal situations, all modules of the FPGA control component except the image output module and the video cache module are reset through step 7, and the endoscope body is initialized. The image output module and the video cache module are not reset to ensure that the last frame of normal output image can still be output unchanged. All other modules are reset or initialized to ensure that each module works correctly and the output image is correct and reliable.

[0088] In the present application, step 31, step 41, and step 51 are the same, which are that the image output module keeps outputting the last frame of normal output image obtained from the video cache module unchanged. These steps are used to ensure that even if an abnormal situation is found, the last frame of normal output image is always output during the process of handling the abnormal situation, so that the wrong image will not affect the doctor's judgment in the exploration, and the image output will not be lost due to the failure of some modules to work during the module reset process after entering the abnormal handling process, which will cause the doctor to be at a loss for a while and cause accidents. The output of the last frame of normal output image ensures that the doctor has a certain basis for judgment and the reliability of the work. In actual performance, the time required for the abnormal handling process is relatively short, and the output image is actually only a short frame drop on the display, which does not affect the doctor's work.

[0089] At the same time, in this application, the input data stream abnormality processing flow of step 3 and the output image abnormality processing flow of step 5 are the same in detail. The main problems are caused by the sensor, but the judgment basis of the two is different. Step 3 is mainly determined by the data decoding module whether there is an input data stream from the sensor or whether the format of the input data stream is correct. At this time, the sensor has not worked normally and the input data stream is input. The abnormal situation is reflected by the inability of the data decoding module to work. The problem exists. Step 5 is to judge the output image generated by the ISP algorithm module, and continuously detect the random noise and overall brightness of the output image. At this time, although the sensor is in working state, there is an error, and the wrong data stream is transmitted, resulting in incorrect output images. The abnormal situation is reflected by the error of the output image. The two abnormal situations of step 3 and step 5 are different in the process of the endoscope system working. However, since the data decoding module and the ISP algorithm module in the sensor and FPGA control component are highly related, these three are processed at the same time during the abnormality handling process.

[0090] See also Figure 4 As shown, the present invention also provides an FPGA-based endoscope image abnormality processing system, which is suitable for an FPGA-based endoscope image abnormality detection and processing method, including an FPGA control component, an endoscope body and a display connected to the FPGA control component, and the FPGA control component at least includes an IIC configuration module, a data decoding module, an ISP algorithm module, a video cache module and an image output module.

[0091] FPGA control components are control components based on field programmable gate arrays (FPGAs). They are a product of further development based on programmable devices such as PAL (programmable array logic) and GAL (general array logic). It appears as a semi-custom circuit in the field of application-specific integrated circuits (ASICs), which not only solves the shortcomings of custom circuits, but also overcomes the shortcomings of the limited number of gate circuits of the original programmable devices. FPGA control components can realize programmable processing of abnormal detection and processing of endoscopic images.

[0092] The IIC configuration module is used to configure the registers of the sensors inside the endoscope and initialize the sensors according to the type of the inserted endoscope body identified.

[0093] The data decoding module is used to decode the data stream of the original image data input by the sensor to facilitate subsequent processing.

[0094] The ISP algorithm module contains a variety of image processing algorithms, which are used to process the data stream decoded by the data decoding module, optimize the image quality, and regenerate the output image.

[0095] The video cache module is used to cache the image data of the output image from the ISP algorithm module to avoid image tearing.

[0096] The image output module is used to encode the cached output image from the video cache module and output it to the display through multiple interfaces (HDMI / DVI / SDI, etc.).

[0097] The IIC configuration module and the data decoding module are electrically connected to the endoscope body and the signals are connected, the data decoding module and the ISP algorithm module are signal connected, the ISP algorithm module and the video cache module are signal connected, the video cache module and the image output module are signal connected, and the image output module and the display are signal connected.

[0098] The data stream output from the sensor is decoded by the data decoding module. The ISP algorithm module optimizes the decoded data, regenerates the image, and caches it in the video cache module. The image output module then obtains the cached image from the video cache module and outputs it.

[0099] The present application designs a new FPGA-based endoscope image anomaly detection and processing method and system. It analyzes and detects the types of anomalies that may be encountered during the operation of the FPGA image processing system, and resets and restores the abnormal modules in a targeted manner. During the recovery process, the main program is guaranteed to run normally, and the system outputs the previous frame of normal output image; after the recovery is completed and the image data flow is normal, it will enter the normal working state. The above recovery process is completed in a very short time. During the recovery process, the processing system functions normally, with little impact on the video image output. The image output only has frame loss, and the overall image output is not stuck. Please refer to Figure 5 As shown in the figure, compared with the overall system reset of a simple watchdog mechanism, this method can ensure that the image processing system can also see normal images during the recovery process. Relying on this method, not only can the stability and anti-interference ability of the FPGA-based endoscope image abnormality detection and processing system be improved, but also the hot-swap function of different input sources can be realized.

[0100] To summarize, the FPGA-based endoscopic image anomaly detection and processing method and system of the present invention performs targeted detection and processing of various abnormal situations during the endoscopic image imaging process, resets some modules with problems, and ensures that the system can return to normal in a short time after an abnormality / interference occurs, while not affecting the output display of normal output images. At the same time, it ensures that different input sources can be hot-swapped in any state, thereby solving the problems of a single mode of detecting abnormal situations and a rough processing method in existing endoscopic image anomaly detection and processing methods.

[0101] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for detecting and processing abnormalities in endoscope images based on FPGA, characterized in that: include: Step 1: Connect the endoscope body to the FPGA control component; Step 2: The FPGA control component continuously reads the connection signal / scope identification information of the endoscope body, and if it can be read, it proceeds to step 3; if it cannot be read, it enters the hot plug processing flow; Step 3: Continuously detect the data stream input by the endoscope body. If the format of the input data stream is normal, proceed to step 4. If there is no input data stream or the format of the input data stream is wrong, proceed to the input data stream exception handling process. Step 4: Continue to decode / process the image data of the data stream input by the endoscope body. If the image data decoding / processing is correct, proceed to step 5. If the image data decoding / processing is wrong, proceed to the image decoding / processing exception handling process. Step 5: Continuously detect the random noise and overall brightness of the output image. If the random noise and overall brightness of the output image are normal, proceed to step 6. If the random noise / overall brightness of the output image is abnormal, proceed to the output image abnormality processing flow. Step 6: The FPGA controls the component to output images to the display.

2. The FPGA-based endoscope image abnormality detection and processing method according to claim 1, characterized in that: The FPGA control component includes an IIC configuration module, and the endoscope body includes a sensor. In step 2, the FPGA control component identifies the type of the inserted endoscope body through the IIC configuration module, and configures the register corresponding to the sensor according to the identified type of the endoscope body to initialize the sensor.

3. The FPGA-based endoscope image abnormality detection and processing method according to claim 1, characterized in that: The hot swap process described in step 2 includes: Step 21: Reset the data decoding module and the ISP algorithm module in the FPGA control component; Step 22: Stop outputting images, and prompt and wait for the user to replug and replug the endoscope body.

4. The FPGA-based endoscope image abnormality detection and processing method according to claim 2 is characterized in that: The FPGA control component includes a data decoding module, an ISP algorithm module, a video cache module and an image output module connected in sequence. The input data stream exception processing process in step 3 includes: Step 31: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module; Step 32: The data decoding module and the ISP algorithm module are reset; Step 33: The sensor is powered off and reset; Step 34: The sensor is powered on, reset is released, and registers of the sensor are reconfigured through the IIC configuration module; Step 35: The data decoding module and the ISP algorithm module are reset and output the image again.

5. The FPGA-based endoscope image abnormality detection and processing method according to claim 1, characterized in that: The FPGA control component includes a data decoding module, an ISP algorithm module, a video cache module and an image output module connected in sequence. The image decoding / processing exception handling process in step 4 includes: Step 41: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module; Step 42: The data decoding module and the ISP algorithm module are reset; Step 43: The data decoding module and the ISP algorithm module end resetting and re-output the image.

6. The FPGA-based endoscope image abnormality detection and processing method according to claim 2, characterized in that: The FPGA control component includes a data decoding module, an ISP algorithm module, a video cache module and an image output module connected in sequence. The output image abnormality processing flow in step 5 includes: Step 51: the image output module keeps outputting the last frame of normal output image obtained from the video buffer module; Step 52: The data decoding module and the ISP algorithm module are reset; Step 53: The sensor is powered off and reset; Step 54: The sensor is powered on, reset is released, and registers of the sensor are reconfigured through the IIC configuration module; Step 55: The data decoding module and the ISP algorithm module are reset and output the image again.

7. The FPGA-based endoscope image abnormality detection and processing method according to claim 6, characterized in that: The method for determining whether the overall brightness of the output image is abnormal in step 5 includes: Step 501: Check the overall brightness average value of the output image and set a brightness threshold of the overall brightness average value. If the overall brightness average value of the output image within time T1 is lower than the brightness threshold, it is determined that the image brightness is abnormal.

8. The FPGA-based endoscope image abnormality detection and processing method according to claim 6, characterized in that: The method for determining whether the random noise of the output image is abnormal in step 5 includes: Step 502: Use a sliding window of a preset size to check the pixels of the output image, check the proportion of random noise in the output image, set a threshold for the proportion of pure color blocks in the output image, and if within time T2, the overall proportion of the pure color blocks in the output image exceeds the threshold, the image data is considered abnormal.

9. The FPGA-based endoscope image abnormality detection and processing method according to claim 1, characterized in that: The FPGA control component includes an IIC configuration module, a data decoding module, an ISP algorithm module, a video cache module and an image output module. The FPGA-based endoscope image abnormality detection and processing method also includes step 7: if step 6 cannot output the image to the display / the image displayed on the display is abnormal, the image output module keeps outputting the last frame of normal output image obtained from the video cache module unchanged, all modules of the FPGA control component except the image output module and the video cache module are reset, and the endoscope body is initialized to re-output the image.

10. An FPGA-based endoscope image processing system, applicable to the FPGA-based endoscope image abnormality detection and processing method as claimed in any one of claims 1 to 9, characterized in that: The invention comprises an FPGA control component, an endoscope body connected to the FPGA control component, and a display. The FPGA control component comprises an IIC configuration module, a data decoding module, an ISP algorithm module, a video cache module, and an image output module. The IIC configuration module and the data decoding module are electrically and signal-connected to the endoscope body, the data decoding module is signal-connected to the ISP algorithm module, the ISP algorithm module is signal-connected to the video cache module, the video cache module is signal-connected to the image output module, and the image output module is signal-connected to the display.

Citation Information

Patent Citations

  • Ultra-fine flexible electronic endoscope image processing method

    CN108124087A

  • Denoising method and device, endoscope, surgical robot and readable storage medium

    CN113907695A

  • Endoscopy procedure for improved display of a video signal and associated endoscopy system and computer program product

    US20210120149A1