Display optimization method, system and device of endoscope image and storage medium

Abstract

The invention discloses an endoscope image display optimization method, system and device and a storage medium. The method comprises the following steps: splitting an original endoscope image acquired in real time into a first image signal and a second image signal which are synchronously transmitted in real time; receiving the first image signal in real time by using the FPGA controller, and processing the first image signal in real time to obtain a first processed image; synchronously utilizing the ARM processor to receive a second image signal in real time, and performing real-time processing on the second image signal to obtain a second processed image; and receiving the second processed image in real time by using the FPGA controller, fusing the first processed image and the second processed image in real time, generating a fused image, and displaying the fused image. According to the invention, the accuracy and reliability of image fusion are ensured, the delay time of image display is effectively controlled, the image display effect is optimized, and a clearer, more accurate and more real-time operation view can be provided for doctors.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and specifically to a method, system, apparatus, and storage medium for optimizing the display of endoscopic images. Background Technology

[0002] As a core device for minimally invasive diagnosis and treatment, the real-time performance, clarity, and stability of image display in medical endoscopic systems directly affect the precision and safety of surgical procedures. Currently, medical endoscopic systems primarily employ the following two technical solutions for image display: The first type is based on the SOC (System-on-a-Chip) solution, which integrates image acquisition, image processing and transmission interface into one unit, such as the Hisilicon 3519AV100 or 3559Av100 series SOC chips. Although these chips are highly integrated, their functions are relatively fixed and their customization is low. There is still a delay of about 80~100ms from the sensor to the image output.

[0003] The second image processing architecture is based on an acquisition card using a programmable processing chip such as an FPGA as its core. In this architecture, the acquisition card first acquires the image signal from the endoscopic device. Then, the image data is buffered, undergoes real-time image processing (including noise reduction, enhancement, and color restoration), AI recognition, and encoding / decoding. Finally, the processing results (such as lesion marking, tissue segmentation, and other auxiliary information) are sent to the display unit along with the original image for fusion display. This solution improves the flexibility of image processing through its programmability, adapting to diverse clinical needs, but it still has significant technical limitations: On the one hand, there is a significant display latency issue. The latency of this capture card solution depends entirely on the hardware processing performance of the capture card, typically with a latency of at least 3 frames (more than 33ms in a 30fps system). This latency severely affects the doctor's hand-eye coordination, potentially causing missed opportunities for optimal operation and posing potential risks in delicate surgeries.

[0004] On the other hand, there is an irreconcilable contradiction between image quality and latency. In the existing acquisition card architecture, reducing the complexity of image processing algorithms or omitting some optimization steps to reduce latency will directly lead to insufficient image noise reduction, color distortion, and blurred details, thus affecting the doctor's accurate identification of lesions. Conversely, if complex optimization algorithms and AI analysis models are used to ensure image quality, it will significantly increase data processing time and further prolong system latency.

[0005] Therefore, current endoscopic image display technology struggles to achieve low-latency transmission while meeting clinical image quality requirements, making it difficult to achieve an ideal balance between image quality and latency. Summary of the Invention

[0006] In view of this, the present invention provides a method, system, device and storage medium for optimizing the display of endoscopic images, in order to solve the problem that existing endoscopic image display technologies are unable to achieve low-latency transmission while meeting clinical image quality requirements, and are unable to achieve an ideal balance between image quality and latency.

[0007] This invention provides a method for optimizing the display of endoscopic images, the method comprising: The raw endoscopic images are acquired in real time and then split into a first image signal and a second image signal that are transmitted synchronously in real time. The first image signal is received in real time using an FPGA controller, and the first image signal is processed in real time to obtain a first processed image; simultaneously, the second image signal is received in real time using an ARM processor, and the second image signal is processed in real time to obtain a second processed image. The FPGA controller receives the second processed image in real time and fuses the first processed image and the second processed image in real time to generate a fused image and display it.

[0008] Optionally, the FPGA controller is configured with an internal buffer; Receiving the first image signal using an FPGA controller includes: Using the FPGA controller, the first image signal is received in a pre-receive mode, such that when the internal buffer starts reading data from the first pixel of the first image signal in the current frame according to the pre-receive time, the last pixel of the first image signal in the current frame has already arrived in the FPGA controller. Let the first image signal of the current frame be the first image signal of frame t. Then the time point at which the internal buffer reads the first pixel of the first image signal of frame t satisfies: T_end_pixel(t)<T_read_start(t)+△Tpre; Where T_end_pixel(t) is the time point when the last pixel of the first image signal in frame t arrives at the FPGA controller, T_read_start(t) is the time point when the internal buffer reads the first pixel of the first image signal in frame t, and △Tpre is the preset advance reception time.

[0009] Optionally, before receiving the first image signal in real time using the FPGA controller, the method further includes: The pre-set receiving time is set in advance.

[0010] Optionally, the total delay time from acquiring the original endoscopic image in frame t to generating and displaying the fused image satisfies: Ttotal(t)=Tcmos(t)+Ts1(t)+Tf1(t)-△Tpre+Thdmi(t); Wherein, Ttotal(t) is the total delay time from the acquisition of the original endoscopic image in frame t to the generation and display of the fused image, Tcmos(t) is the sampling time of the original endoscopic image in frame t, Ts1(t) is the transmission time of the original endoscopic image in frame t, Tf1(t) is the internal image transmission and reception time of the FPGA controller, and Thdmi(t) is the external image transmission time of the FPGA controller.

[0011] Optionally, the FPGA controller is further configured with a pipelined image processing unit; The first image signal is processed in real time to obtain a first processed image, including: The first image signal is processed using the pipeline image processing unit to obtain the first processed image. The pipeline image processing includes at least one of the following: noise reduction, enhancement, color correction, and color restoration.

[0012] Optionally, for the original endoscopic image in frame t, when t≥2, the fused image obtained is specifically the image obtained by fusing the first processed image in frame t and the second processed image in frame (t-1).

[0013] Optionally, for the original endoscopic image in frame t, the first processed image in frame t and the second processed image in frame (t-1) are fused in real time to generate a fused image, including: Compare the pixel value of each pixel in the second processed image of frame t-1 with the preset mirror area identifier value; Based on the comparison results, the pixel values ​​of the corresponding pixels in the first processed image in frame t or the second processed image in frame (t-1) are respectively used as the pixel values ​​of the fused image and output to complete the image fusion.

[0014] Optionally, based on the comparison result, the pixel values ​​of corresponding pixels in the first processed image of frame t or the second processed image of frame t-1 are respectively used as the pixel values ​​of the fused image and output, including: Select the pixel (x) in the second processed image of frame t-1. 2,t-1 ,y 2,t-1 ), when pixel (x 2,t-1 ,y 2,t-1When the pixel value of (x) matches the preset mirror area identifier value, the selected pixel point (x) in the first processed image of frame t is selected. 2,t-1 ,y 2,t-1 The corresponding pixel (x) 1,t ,y 1,t The pixel values ​​of the fused image are used as the pixel values ​​of the fused image and output. When pixel (x) 2,t-1 ,y 2,t-1 When the pixel value of (x) is inconsistent with the preset mirror area identifier value, the pixel point (x) in the second processed image of frame t-1 is... 2,t-1 ,y 2,t-1 The pixel values ​​of the fused image are used as the output. Iterate through each pixel in the second processed image of frame t-1, and obtain the pixel value of each pixel in the fused image in the same way and output it.

[0015] Optionally, the second image signal is processed in real time to obtain a second processed image, including: Using the ARM processor, the second image signal is subjected to AI recognition to obtain the AI ​​recognition result; User interface information is generated based on the AI ​​recognition results; The second processed image is generated based on the AI ​​recognition results and the user interface information.

[0016] Optionally, the method further includes: Based on the AI ​​recognition results, the image processing parameters are adjusted during the real-time processing of the first image signal to optimize the real-time processing of the first image signal by the FPGA controller, resulting in an optimized first processed image.

[0017] Furthermore, the present invention also provides a display optimization system for endoscopic images, applied in the aforementioned method for display optimization of endoscopic images, comprising: An endoscope image acquisition module is used to acquire raw endoscope images in real time and split the raw endoscope images into a first image signal and a second image signal that are transmitted synchronously in real time. The image processing and analysis module is used to receive the first image signal in real time using an FPGA controller and process the first image signal in real time to obtain a first processed image; simultaneously, it uses an ARM processor to receive the second image signal in real time and processes the second image signal in real time to obtain a second processed image; it is also used to receive the second processed image in real time using the FPGA controller and fuse the first processed image and the second processed image in real time to generate a fused image. An image display output module is used to output the fused image.

[0018] Furthermore, the present invention also provides an endoscope image display optimization device, the device comprising: An image sensor is used to acquire raw endoscopic images in real time and to split the raw endoscopic images into a first image signal and a second image signal that are transmitted synchronously in real time. An FPGA controller, communicatively connected to the image sensor, is used to receive the first image signal transmitted by the image sensor in real time and process the first image signal in real time to obtain a first processed image. An ARM processor is communicatively connected to both the image sensor and the FPGA controller. It is used to receive the second image signal transmitted by the image sensor in real time and process the second image signal in real time to obtain a second processed image. The FPGA controller is also used to receive the second processed image transmitted by the ARM processor, and to fuse the first processed image and the second processed image in real time to generate a fused image; The display is communicatively connected to the FPGA controller and is used to receive and display the fused image.

[0019] Furthermore, the present invention also provides a computer storage medium comprising: at least one instruction that, when executed by a computer, implements the method steps of the aforementioned method for optimizing the display of endoscopic images.

[0020] The beneficial effects of this invention are as follows: The raw endoscopic image acquired in real time is split into two parts. One part (the first image signal) is directly transmitted to the FPGA controller, and the FPGA controller directly controls the image display, achieving a direct FPGA connection without involving image data encryption / decryption, minimizing unnecessary processing steps, and achieving near real-time image display. The FPGA controller processes the first image signal in real time, and the other part (the second image signal), split in half by the ARM processor, is also processed in real time. Finally, the FPGA controller fuses the first and second processed images in real time. On the one hand, the entire architecture uses the FPGA controller as the processing core, maximizing efficiency and efficiency. By controlling the fixed display latency (e.g., reducing the display latency to below 60ms, which can reduce latency by more than 30% compared to traditional solutions), stable image quality and performance can be maintained during long-term continuous surgery, overcoming the problem of image quality degradation in existing domestic endoscope products during long-term surgery. On the other hand, it can utilize a variety of intelligent image processing algorithms integrated into the FPGA controller to perform pipelined image processing, effectively improving image quality and image processing efficiency while achieving extremely low image transmission latency. It can achieve an ideal balance between image quality and image latency, ensuring a clear surgical field of view in extreme surgical scenarios, and providing high-definition images with 4K resolution, ultra-wide color gamut, and high fluorescence sensitivity, thereby helping to improve surgical efficiency and safety. Attached Figure Description

[0021] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings: Figure 1 The architecture and signal transmission flowchart of traditional endoscopic image display technology are shown; Figure 2 The overall delay timing diagram of conventional endoscopic image display technology is shown; Figure 3 A flowchart of a method for optimizing the display of endoscopic images according to Embodiment 1 of the present invention is shown; Figure 4 The diagram illustrates the architecture and signal transmission flowchart of Embodiment 1 of the present invention; Figure 5 The overall delay timing diagram in Embodiment 1 of the present invention is shown; Figure 6 A structural diagram of an endoscope image display optimization system according to Embodiment 2 of the present invention is shown; Figure 7 The diagram shows a structural diagram of an endoscope image display optimization device according to Embodiment 3 of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] In this embodiment of the invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0026] Example 1 In traditional endoscopic image display solutions, the architecture and signal transmission process are as follows: Figure 1 As shown. In Figure 1 In this process, the CMOS signal is transmitted from the CMOS sensor to the FPGA controller, processed and buffered by the FPGA controller, and then transmitted to the ARM processor. The ARM processor identifies the signal and buffers it again, and finally the HDMI driver unit outputs the signal for display.

[0027] Depend on Figure 1 The signal transmission flowchart shown can be used to derive the overall delay timing diagram of the signal in the processing link, such as... Figure 2 As shown, based on its working topology, the delay times of each module are decomposed into a linear superposition relationship, and thus the total delay time of a single frame image from image acquisition to image display is: Ttotal = Tcmos + Ts1 + Tf1 + Ta1+Thdmi; Ttotal: The total latency from image acquisition to image display for a single frame; Tcmos: CMOS image exposure and sampling time; Ts1: CMOS single-frame image transmission time; Tf1: FPGA single-frame image reception and forwarding time; Ta1: ARM single-frame image reception and processing time; Thdmi: Time it takes for a single frame of image to be sent from the ARM processor to the display; Typically, CMOS image capture operates at 30fps, while HDMI output displays at 60fps. Figure 2 The timing diagram, combined with actual measurements of the hardware and software system, shows that the total delay from image acquisition to image display for a single frame is between 100 and 120 ms. This delay significantly affects the doctor's hand-eye coordination, which may lead to missing the best operating opportunity and pose potential risks in delicate surgeries.

[0028] Therefore, this embodiment provides the following method for optimizing the display of endoscopic images, such as... Figure 3 As shown, the method includes: S1: Acquire raw endoscopic images in real time, and split the raw endoscopic images into a first image signal and a second image signal that are transmitted synchronously in real time; S2: The FPGA controller receives the first image signal in real time and processes it in real time to obtain a first processed image; simultaneously, the ARM processor receives the second image signal in real time and processes it in real time to obtain a second processed image. S3: The FPGA controller receives the second processed image in real time, and fuses the first processed image and the second processed image in real time to generate a fused image and display it.

[0029] In this embodiment, the raw endoscopic image acquired in real time is split into two parts. One part (i.e., the first image signal) is directly transmitted to the FPGA controller, and the FPGA controller directly controls the image display. This achieves a direct FPGA connection, eliminating the need for image data encryption and decryption, minimizing unnecessary processing steps, and achieving near real-time image display. The FPGA controller processes the first image signal in real time, and the ARM processor processes the other part (i.e., the second image signal) in real time. Finally, the FPGA controller fuses the first and second processed images in real time. The entire architecture uses the FPGA controller as the processing core. The system can control fixed display latency as much as possible (e.g., reducing display latency to below 60ms, which can reduce latency by more than 30% compared to traditional solutions), maintaining stable image quality and performance during long, continuous surgeries, and overcoming the problem of image quality degradation in existing domestic endoscope products during long surgeries. On the other hand, it can utilize multiple intelligent image processing algorithms integrated into the FPGA controller to perform pipelined image processing, achieving extremely low image transmission latency while effectively improving image quality and image processing efficiency. It ensures a clear surgical field of view in extreme surgical scenarios, providing high-definition images with 4K resolution, ultra-wide color gamut, and high fluorescence sensitivity, thereby helping to improve surgical efficiency and safety.

[0030] The following provides a detailed description of each step in the endoscopic image display optimization method of this embodiment.

[0031] In step S1, the raw endoscopic image can be acquired in real time by an image sensor (such as a CMOS sensor). In step S1, the raw endoscopic image is split into two, which can be achieved by a splitter integrated into the image sensor or by an image sensor supporting dual-output. The first and second image signals from the split image are identical, facilitating a correct temporal and spatial correspondence between the second processed image and the first processed image, thereby ensuring the reliability of the image fusion display.

[0032] An FPGA controller (referring to a controller that integrates FPGA devices; FPGA stands for Field-Programmable Gate Array) allows users to customize digital logic circuits using hardware description languages ​​(such as Verilog and VHDL). Its architecture typically consists of programmable logic blocks (CLBs), configurable interconnects, and I / O blocks, supporting true parallel processing.

[0033] The FPGA controller is equipped with an internal buffer.

[0034] The internal buffer configured by the FPGA controller facilitates the temporary storage of image signals, preventing signal loss and ensuring the continuity of signal processing. Furthermore, it allows the FPGA controller to utilize its integrated image processing algorithms to process the initial image signal in real time, improving image quality and ensuring that doctors can see more realistic tissue structures, thus aiding in the diagnosis.

[0035] In step S2, the first image signal is received using the FPGA controller, including: Using the FPGA controller, the first image signal is received in a pre-receive mode, such that when the internal buffer starts reading data from the first pixel of the first image signal in the current frame according to the pre-receive time, the last pixel of the first image signal in the current frame has already arrived in the FPGA controller. Let the first image signal of the current frame be the first image signal of frame t. Then the time point at which the internal buffer reads the first pixel of the first image signal of frame t satisfies: T_end_pixel(t)<T_read_start(t)+△Tpre; Where T_end_pixel(t) is the time point when the last pixel of the first image signal in frame t arrives at the FPGA controller, T_read_start(t) is the time point when the internal buffer reads the first pixel of the first image signal in frame t, and △Tpre is the preset advance reception time.

[0036] By employing the aforementioned pre-reception mode, the FPGA controller can prepare for processing the first image signal in advance, fully utilizing the storage capacity of the internal buffer and avoiding waiting time during data transmission. This enables an overall advancement in the timing of image acquisition and display. Simultaneously with signal reception, the FPGA controller can perform preliminary processing on the first image signal in the internal buffer, such as adjusting basic parameters like brightness and contrast, to meet the requirements of subsequent fusion processing. This effectively reduces latency and improves signal processing efficiency while ensuring image quality.

[0037] The above-mentioned relationship, which is satisfied by the time point when the internal buffer reads the first pixel of the first image signal of the first frame of the t-th frame, can ensure that when the internal buffer needs to read the data of a certain frame, the end pixel of that frame has arrived in the buffer in advance by △Tpre time, thus avoiding the risk of "reading empty".

[0038] Furthermore, before receiving the first image signal in real time using the FPGA controller, the method further includes: The pre-set receiving time is set in advance.

[0039] By pre-setting the pre-reception time, it is possible to ensure that the overall timing from image acquisition to display is advanced.

[0040] Specifically, the advance reception time can be dynamically calculated by combining three factors: buffer bandwidth, image timing parameters, and control logic delay. This can be achieved using conventional methods. For example, the theoretical minimum and safety margin of the advance reception time can be calculated using buffer bandwidth and image timing parameters respectively. Then, the advance reception time can be calculated based on the sum of the theoretical minimum and safety margin. Further details will not be elaborated here.

[0041] In step S2, the FPGA controller is further configured with a pipelined image processing unit; The first image signal is processed in real time to obtain a first processed image, including: S21: The first image signal is processed using the pipeline image processing unit to obtain the first processed image; The pipeline image processing includes at least one of the following: noise reduction, enhancement, color correction, and color restoration.

[0042] The pipelined image processing of the Image Signal Processing (ISP) unit effectively improves the quality of the initial image signal, enhances image processing efficiency, and further optimizes latency. Specifically, noise reduction removes random noise from the image, making it clearer and smoother, reducing interference, and facilitating more accurate capture of tissue details. Enhancement highlights key information in the image, such as increasing the contrast of blood vessels, making their outlines more prominent, and facilitating more accurate capture of their distribution and condition. Color correction and color restoration ensure that the image colors more realistically and accurately reflect the actual tissue colors, aiding in subsequent identification of lesions based on tissue color changes.

[0043] In step S2, the second image signal is processed in real time to obtain a second processed image, including: S22: Using the ARM processor, perform AI recognition on the second image signal to obtain the AI ​​recognition result; S23: Generate user interface information based on the AI ​​recognition results; S24: Generate the second processed image based on the AI ​​recognition result and the user interface information.

[0044] During AI (Artificial Intelligence) recognition, the ARM processor leverages its powerful computing capabilities and advanced algorithms to accurately analyze various tissue features and lesion information contained in the second image signal. For example, it can identify features such as the size, shape, and boundaries of lesions; these features constitute the AI ​​recognition results. The second-processed image generated based on these AI recognition results is used for subsequent image fusion, providing doctors with crucial diagnostic data. User interface information (UI information) can include marking of lesion areas, quantitative data display of tissue features, and diagnostic aids. Generating a second-processed image based on this UI information and using it for subsequent image fusion also provides valuable data references for doctors' subsequent decision-making.

[0045] ARM (Advanced RISC Machine) is a processor based on the Reduced Instruction Set Computing (RISC) architecture, which can perform complex control processing tasks at low cost.

[0046] It should be understood that in this embodiment, the processing of the first image signal by the pipelined image processing unit in the FPGA controller (i.e., step S21) is performed synchronously with the processing of the second image signal by the ARM processor (i.e., steps S22~S24).

[0047] Furthermore, the method also includes: Based on the AI ​​recognition results, the image processing parameters are adjusted during the real-time processing of the first image signal to optimize the real-time processing of the first image signal by the FPGA controller, resulting in an optimized first processed image.

[0048] Specifically, by adjusting the image processing parameters during the real-time processing of the first image signal based on the AI ​​recognition results, the FPGA controller can optimize the processing of the first image signal, thereby ensuring that the final fused image can more accurately reflect the lesion and tissue characteristics. For example, when the AI ​​recognition results show that the lesion area is dark in color and has low contrast, the contrast parameter in the first image signal processing can be appropriately increased to make the lesion area appear clearer.

[0049] When adjusting image processing parameters, an adaptive adjustment strategy can be adopted. This means dynamically adjusting the parameters of each processing stage in the pipeline image processing unit based on different AI recognition results. For example, if the AI ​​identifies significant noise interference in the image, affecting the judgment of lesion features, the noise reduction parameters can be increased to enhance the noise reduction effect.

[0050] Meanwhile, to ensure the timeliness and accuracy of adjustments, a feedback mechanism can be established. Once the ARM processor completes AI recognition and obtains the results, it immediately feeds the relevant information back to the FPGA controller. The FPGA controller then quickly adjusts the parameters of the first image signal processing based on the feedback. This allows for further optimization of the quality of the first processed image without affecting overall processing efficiency or reducing latency, achieving a balance between improving image quality and reducing latency, resulting in higher-quality image display.

[0051] In this embodiment, the image processing parameters may be local contrast enhancement parameters, dynamic noise reduction level, local sharpening weight, etc.

[0052] Specifically, the AI ​​recognition results used for adjusting image processing parameters can be historical frame data. That is, when the current frame is frame t, the FPGA controller processes the first image signal of frame t in real time, while the AI ​​recognition result of frame t-1 is fed back to the FPGA controller. Historical frame data can be used to predict the current frame image and processing parameters. For example, if the AI ​​recognition result of frame t-1 indicates that the lesion area is continuously expanding and the boundary is gradually blurring, then when the FPGA controller processes the first image signal of frame t, the contrast and sharpness parameters can be appropriately enhanced to make the lesion area more prominent.

[0053] Of course, machine learning algorithms can also be used to assist in parameter tuning. For example, by collecting a large amount of historical image data, along with corresponding AI recognition results and image processing parameter adjustment records, a machine learning model can be trained. This model can learn the mapping relationship between different AI recognition results and optimal image processing parameters. When a new AI recognition result is generated, the model can quickly provide appropriate image processing parameter adjustment suggestions. The FPGA controller then adjusts the parameters according to these suggestions, thereby achieving more accurate and efficient image processing.

[0054] In step S3, for the original endoscopic image in frame t, when t≥2, the fused image obtained is specifically the image obtained by fusing the first processed image in frame t and the second processed image in frame t-1.

[0055] By fusing the first processed image in frame t and the second processed image in frame (t-1), the existing processing results can be utilized to the maximum extent while ensuring real-time image processing. During image fusion, the FPGA controller overlays and adjusts the first and second processed images to ensure that the fused image contains both the optimized image quality and organizational details of the first processed image and the AI ​​recognition results and user interface information from the second processed image.

[0056] For example, during the fusion process, the details and colors of the tissue structures clearly presented in the first processed image will be fully preserved; while the marked lesion areas and displayed quantitative data in the second processed image will be overlaid on the first processed image with appropriate transparency and color, so that doctors can intuitively obtain important diagnostic information when observing the images.

[0057] It should be noted that, in practical applications, the image recognition and processing flow of the ARM processor only begins after at least one frame has been received. That is, the ARM processor does not process the first frame of the original endoscopic image, but instead begins real-time processing of the second image signal starting from the second frame of the original endoscopic image. Therefore, for t=1, i.e., the first frame of the original endoscopic image, the FPGA controller directly outputs the first processed image after processing the first image signal, and outputs and displays the first processed image of the first frame as the fused image.

[0058] Specifically, for the original endoscopic image in frame t, the first processed image in frame t and the second processed image in frame (t-1) are fused in real time to generate a fused image, including: S31: Compare the pixel value of each pixel in the second processed image of frame t-1 with the preset mirror area identifier value; S32: Based on the comparison results, the pixel values ​​of the corresponding pixels in the first processed image in frame t or the second processed image in frame t-1 are respectively used as the pixel values ​​of the fused image and output to complete the image fusion.

[0059] In step S31, when the ARM generates the AI ​​analysis results and UI interface (i.e., the "second processing image"), it does not generate a complete image that covers the entire screen. Instead, it deliberately reserves an area where the real-time endoscope image should be displayed, namely the "mirror area display area". This area is represented by a special color area or a specific brightness / chromaticity code. This special color area or specific brightness / chromaticity code is called the preset mirror area identifier value. Based on this identifier value, it can tell the FPGA controller that this area is not normal UI information and needs to be replaced during subsequent fusion.

[0060] By comparing the pixel value of each pixel in the second processed image of frame t-1 with the preset mirror area identifier value, the fusion strategy adopted by each pixel during fusion can be determined, which facilitates the subsequent use of pixel-by-pixel comparison and replacement to achieve accurate image fusion display.

[0061] In step S32, pixel values ​​are selected based on the comparison result of step S31. If the pixel value of a certain pixel in the (t-1)th frame of the second processed image is greater than the preset mirror area identifier value, it indicates that the pixel in the current second processed image may contain important AI recognition results and user interface information, which is the image information that the ARM wants to display, such as lesion area markings, quantitative data, etc. Retaining it in the fused image can provide doctors with key diagnostic basis. In this case, the pixel value of the pixel in the (t-1)th frame of the second processed image is output as the pixel value of the corresponding position in the fused image. Conversely, if the pixel value of the pixel is less than the preset mirror area identifier value, it indicates that the information of the pixel in the current second processed image is not the image information that the ARM wants to display, and the pixel value of the corresponding pixel in the tth frame of the first processed image is output as the pixel value of the fused image.

[0062] In this embodiment, the preset endoscope zone label value can be set according to the actual needs and experience of the endoscope in different surgical scenarios, and there is no limitation here.

[0063] Further, step S32 includes: Select the pixel (x) in the second processed image of frame t-1. 2,t-1 ,y 2,t-1 ), when pixel (x 2,t-1 ,y 2,t-1 When the pixel value of (x) matches the preset mirror area identifier value, the selected pixel point (x) in the first processed image of frame t is selected. 2,t-1 ,y 2,t-1 The corresponding pixel (x) 1,t ,y 1,t The pixel values ​​of the fused image are used as the pixel values ​​of the fused image and output. When pixel (x) 2,t-1 ,y 2,t-1 When the pixel value of (x) is inconsistent with the preset mirror area identifier value, the pixel point (x) in the second processed image of frame t-1 is... 2,t-1 ,y 2,t-1 The pixel values ​​of the fused image are used as the output. Iterate through each pixel in the second processed image of frame t-1, and obtain the pixel value of each pixel in the fused image in the same way and output it.

[0064] In a dual-channel architecture, the FPGA processes the Nth frame (the latest frame), while the ARM analyzes the (N-1)th frame (the previous frame). Due to the movement of surgical instruments or tissue, the content of these two frames may have slight differences in position. If conventional "alpha blending" is used to achieve the fusion between the two, that is, the marker box generated by the ARM based on the (N-1)th frame is directly superimposed on the Nth frame of the FPGA, the marker box will be "pasted" in the wrong position, resulting in a troublesome "ghosting" or "misalignment" phenomenon.

[0065] In this embodiment, when the pixel (x) in the second processed image of the (t-1)th frame 2,t-1 ,y 2,t-1 When the pixel value of (x) matches the preset mirror area identifier value, the FPGA discards this pixel from the ARM and selects the corresponding position (x) in the real-time image it has processed. 1,t ,y 1,t The pixels of ) are used as the final output; when the second processed image of frame t-1 contains pixels (x) 2,t-1 ,y 2,t-1 When the pixel value of the ARM is inconsistent with the preset mirror area identifier value, it means that this position is the UI element (such as text, arrow, or marker box) that the ARM wants to display. The FPGA will retain this pixel of the ARM as the final output.

[0066] The "pixel-level replacement" technology described in this embodiment cleverly avoids this problem, making the ARM's marker box and the real-time image output by the FPGA physically separate, eliminating misalignment and overlay, achieving dual-channel asynchronous processing and precise cross-frame fusion, and ensuring the accuracy and reliability of image fusion.

[0067] It should be understood that the "pixel-level replacement" operation in this embodiment is completed instantaneously by the FPGA hardware logic as the data flows through, without introducing any additional frame buffering or processing delay, thus maintaining the ultra-low latency characteristics of the "pass-through channel". The final fused image provides both a pure surgical field of view with no latency (through the FPGA) and simultaneously allows viewing of reference information provided by AI based on analysis from a slightly earlier moment (generated by the ARM). The two are mutually independent and clearly distinguishable.

[0068] In this embodiment, for the original endoscopic image of frame t, the total delay time from acquisition to generation and display of the fused image of the original endoscopic image of frame t satisfies: Ttotal(t)=Tcmos(t)+Ts1(t)+Tf1(t)-△Tpre+Thdmi(t); Wherein, Ttotal(t) is the total delay time from the acquisition of the original endoscopic image in frame t to the generation and display of the fused image, Tcmos(t) is the sampling time of the original endoscopic image in frame t, Ts1(t) is the transmission time of the original endoscopic image in frame t, Tf1(t) is the internal image transmission and reception time of the FPGA controller, and Thdmi(t) is the external image transmission time of the FPGA controller.

[0069] △Tpre is the advance time for the FPGA controller to read the first pixel of the first image signal in frame t. Since the FPGA controller has a certain processing cycle and data transmission time when processing images, by setting the advance reception time △Tpre and introducing a pre-reading strategy, the real-time performance of image display can be further optimized. This allows the FPGA controller to start reading the first image signal earlier, thus advancing the overall timing of image acquisition and display. The total delay time from acquiring the original endoscopic image to generating and displaying the fused image is effectively reduced, thus optimizing the image display effect.

[0070] For example, when the endoscope acquires the t-th frame of the original endoscopic image, the FPGA controller will begin operation ΔTpre time in advance when reading the first pixel of the first image signal of that frame. This gives the FPGA controller more time to process the image signal in subsequent processing, reducing delays caused by insufficient processing time.

[0071] The technical solution adopted in this embodiment effectively controls the total latency while achieving image fusion. On one hand, the powerful hardware processing capabilities of the FPGA controller and the pipelined image processing unit are used to quickly process the first image signal in real time. On the other hand, the ARM processor performs AI recognition and related processing on the second image signal, and adjusts the parameters of the first image signal processing in a timely manner through a feedback mechanism to optimize image quality. At the same time, the "pixel-level replacement" technology avoids the "ghosting" or "misalignment" phenomena that may occur in conventional fusion methods, ensuring the accuracy and reliability of image fusion without introducing additional processing latency.

[0072] The architecture and signal transmission process of this embodiment are as follows: Figure 4 As shown. In Figure 4In this process, the raw endoscopic image signal acquired by the CMOS sensor is split into two and transmitted synchronously to the FPGA controller and ARM processor. In the first image signal, after receiving the first image signal, the FPGA processes it through its built-in ISP unit to generate a first processed image. The second image signal is directly transmitted to the ARM processor, which then performs AI recognition and analysis on the image and generates UI information. Based on the AI ​​recognition results and UI information, a second processed image is generated and transmitted to the FPGA. The FPGA controller fuses the first and second processed images to generate a fused image, which is then directly transmitted to the display for display via the HDMI output driver.

[0073] Based on the above signal flow diagram, the overall delay timing diagram of a single frame of raw endoscopic image from acquisition to display in this embodiment can be obtained as follows: Figure 5 As shown, its total delay time satisfies: Ttotal(t)=Tcmos(t)+Ts1(t)+Tf1(t)-△Tpre+Thdmi(t); Among them, the sampling time Tcmos(t) of the t-th frame of the original endoscope image is equal to the aforementioned CMOS image exposure and sampling time Tcmos; the transmission time Ts1(t) of the t-th frame of the original endoscope image is equal to the aforementioned CMOS single-frame image transmission time; the internal image transmission and reception time Tf1(t) of the FPGA controller is equal to the aforementioned FPGA single-frame image reception and forwarding time; and the external image transmission time Thdmi(t) of the FPGA controller is equal to the aforementioned ARM single-frame image transmission time to the display. Based on the above delay timing diagram and combined with the actual measurement of the hardware and software system, it can be measured that the average delay from image acquisition to display is optimized to 50~60ms, which is significantly improved compared to the total delay of 100~120ms in the traditional solution.

[0074] Example 2 An endoscopic image display optimization system is applied to the endoscopic image display optimization method of Embodiment 1, such as... Figure 6 As shown, the system includes: An endoscope image acquisition module is used to acquire raw endoscope images in real time and split the raw endoscope images into a first image signal and a second image signal that are transmitted synchronously in real time. The image processing and analysis module is used to receive the first image signal in real time using an FPGA controller and process the first image signal in real time to obtain a first processed image; simultaneously, it uses an ARM processor to receive the second image signal in real time and processes the second image signal in real time to obtain a second processed image; it is also used to receive the second processed image in real time using the FPGA controller and fuse the first processed image and the second processed image in real time to generate a fused image. An image display output module is used to output the fused image.

[0075] In this embodiment, the original endoscopic image acquired in real time by the endoscopic image acquisition module is split into two parts. One part (i.e., the first image signal) is directly transmitted to the FPGA controller, and finally, the FPGA controller directly controls the image display, realizing a direct FPGA connection without involving the encryption and decryption process of image data, minimizing unnecessary processing steps, and achieving near real-time image display. Through the image processing and analysis module, the FPGA controller processes the first image signal in real time, and the other part (i.e., the second image signal) split into two by the ARM processor is processed in real time. Finally, the FPGA controller fuses the first and second processed images in real time. On the one hand, the entire architecture uses the FPGA controller as... The core processing unit can control fixed display latency as much as possible (e.g., reducing display latency to below 60ms, which can reduce latency by more than 30% compared to traditional solutions), maintaining stable image quality and performance during long, continuous surgeries, and overcoming the problem of image quality degradation in existing domestic endoscope products during long surgeries. On the other hand, it can utilize multiple intelligent image processing algorithms integrated into the FPGA controller to perform pipelined image processing, effectively improving image quality and image processing efficiency while achieving extremely low image transmission latency. It can achieve an ideal balance between image quality and image latency, ensuring a clear surgical field of view even in extreme surgical scenarios, and providing high-definition images with 4K resolution, ultra-wide color gamut, and high fluorescence sensitivity, thereby helping to improve surgical efficiency and safety.

[0076] The functions of each module in the endoscopic image display optimization system described in this embodiment are the same as the method steps of the endoscopic image display optimization method described in Embodiment 1. Therefore, for details not covered in this embodiment, please refer to Embodiment 1 and... Figures 1 to 5 The specific details will not be repeated here.

[0077] Example 3 This embodiment provides a device for optimizing the display of endoscopic images, such as... Figure 7 As shown, it includes: An image sensor is used to acquire raw endoscopic images in real time and to split the raw endoscopic images into a first image signal and a second image signal that are transmitted synchronously in real time. An FPGA controller, communicatively connected to the image sensor, is used to receive the first image signal transmitted by the image sensor in real time and process the first image signal in real time to obtain a first processed image. An ARM processor is communicatively connected to both the image sensor and the FPGA controller. It is used to receive the second image signal transmitted by the image sensor in real time and process the second image signal in real time to obtain a second processed image. The FPGA controller is also used to receive the second processed image transmitted by the ARM processor, and to fuse the first processed image and the second processed image in real time to generate a fused image; The display is communicatively connected to the FPGA controller and is used to receive and display the fused image.

[0078] In this embodiment, an image sensor, FPGA controller, ARM processor, and display work together to construct a highly efficient endoscopic image display optimization device. The image sensor, as the front end of the entire device, can acquire raw endoscopic images in a timely and accurate manner, and split them into a first image signal and a second image signal for synchronous transmission, providing a stable data source for subsequent processing. The FPGA controller, with its powerful hardware processing capabilities, processes the first image signal in real time at a fast speed, completing complex image processing tasks in a short time and ensuring the rapid generation of the first processed image. Simultaneously, the FPGA controller also undertakes the important task of image fusion, fusing the second processed image (processed by the ARM processor) with the first processed image in real time to generate a clear and accurate fused image. The ARM processor focuses on processing the second image signal, utilizing its advanced AI recognition technology and algorithms to perform in-depth image analysis and extract useful information; this information can provide doctors with important diagnostic basis, helping them better understand the patient's condition. The display, as the final output device, clearly displays the fused image, providing doctors with an intuitive surgical view. Its high resolution, ultra-wide color gamut, and high fluorescence sensitivity make the image display more realistic and clear, allowing doctors to observe the condition of the surgical site more accurately.

[0079] The clearly defined and collaborative display optimization device described in this embodiment ensures the accuracy and reliability of image fusion, effectively controls image display latency, and optimizes image display effects. In practical applications, this device can provide doctors with a clearer, more accurate, and real-time surgical field of view, helping to improve surgical efficiency and safety, and providing strong protection for patients' health.

[0080] Similarly, for details not covered in this embodiment, please refer to Embodiment 1, Embodiment 2, and... Figures 1 to 6 The specific details will not be repeated here.

[0081] Example 4 This embodiment also provides another endoscope image display optimization device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed, it implements the method steps in the endoscope image display optimization method of Embodiment 1.

[0082] By using a computer program stored in memory and running on a processor, the accuracy and reliability of image fusion are ensured. It also effectively controls the image display delay time and optimizes the image display effect, providing doctors with a clearer, more accurate, and real-time surgical field of vision. This helps improve surgical efficiency and safety and provides strong protection for patients' health.

[0083] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting all parts of the computer device through various interfaces and lines.

[0084] Memory can be used to store computer programs and / or models. The processor performs various functions of the computer device by running or executing the computer programs and / or models stored in the memory, and by accessing data stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function (e.g., sound playback, image playback, etc.); the data storage area can store data created based on the use of the mobile phone (e.g., audio data, video data, etc.). Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SmartMedia Cards (SMC), Secure Digital (SD) cards, Flash Cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0085] It should be understood that each block of a flowchart and / or block diagram, and combinations of blocks in a flowchart and / or block diagram, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that instructions executable by the processor of the computer or other programmable data processing device generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer programs may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] This embodiment also provides a computer storage medium, the computer storage medium including: at least one instruction, which, when executed by a computer, implements the method steps in the endoscopic image display optimization method of Embodiment 1.

[0089] By executing a computer storage medium containing at least one instruction, the accuracy and reliability of image fusion are ensured. It also effectively controls the image display delay time, optimizes the image display effect, and provides doctors with a clearer, more accurate, and real-time surgical field of vision, which helps to improve surgical efficiency and safety and provides strong protection for patients' health.

[0090] Similarly, for details not covered in this embodiment, please refer to Embodiment 1, Embodiment 2, Embodiment 3, and... Figures 1 to 7 The specific details will not be repeated here.

[0091] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for optimizing the display of endoscopic images, characterized in that, The method includes: The raw endoscopic images are acquired in real time and then split into a first image signal and a second image signal that are transmitted synchronously in real time. The first image signal is received in real time using an FPGA controller, and the first image signal is processed in real time to obtain a first processed image; simultaneously, the second image signal is received in real time using an ARM processor, and the second image signal is processed in real time to obtain a second processed image. The FPGA controller receives the second processed image in real time and fuses the first processed image and the second processed image in real time to generate a fused image and display it.

2. The method for optimizing the display of endoscopic images according to claim 1, characterized in that, The FPGA controller is equipped with an internal buffer; Receiving the first image signal using an FPGA controller includes: Using the FPGA controller, the first image signal is received in a pre-receive mode, such that when the internal buffer starts reading data from the first pixel of the first image signal in the current frame according to the pre-receive time, the last pixel of the first image signal in the current frame has already arrived in the FPGA controller. Let the first image signal of the current frame be the first image signal of frame t. Then the time point at which the internal buffer reads the first pixel of the first image signal of frame t satisfies: T_end_pixel(t)<T_read_start(t)+△Tpre; Where T_end_pixel(t) is the time point when the last pixel of the first image signal in frame t arrives at the FPGA controller, T_read_start(t) is the time point when the internal buffer reads the first pixel of the first image signal in frame t, and △Tpre is the preset advance reception time.

3. The method for optimizing the display of endoscopic images according to claim 2, characterized in that, Before receiving the first image signal in real time using the FPGA controller, the method further includes: The pre-set receiving time is configured in advance.

4. The method for optimizing the display of endoscopic images according to claim 2, characterized in that, The total delay time from acquiring the original endoscopic image in frame t to generating and displaying the fused image satisfies: Ttotal(t)=Tcmos(t)+Ts1(t)+Tf1(t)-△Tpre+Thdmi(t); Wherein, Ttotal(t) is the total delay time from the acquisition of the original endoscopic image in frame t to the generation and display of the fused image, Tcmos(t) is the sampling time of the original endoscopic image in frame t, Ts1(t) is the transmission time of the original endoscopic image in frame t, Tf1(t) is the internal image transmission and reception time of the FPGA controller, and Thdmi(t) is the external image transmission time of the FPGA controller.

5. The method for optimizing the display of endoscopic images according to claim 2, characterized in that, The FPGA controller is also configured with a pipelined image processing unit; The first image signal is processed in real time to obtain a first processed image, including: The first image signal is processed using the pipeline image processing unit to obtain the first processed image. The pipeline image processing includes at least one of the following: noise reduction, enhancement, color correction, and color restoration.

6. The method for optimizing the display of endoscopic images according to claim 1, characterized in that, For the original endoscopic image in frame t, when t≥2, the fused image obtained is specifically the image obtained by fusing the first processed image in frame t and the second processed image in frame t-1.

7. The method for optimizing the display of endoscopic images according to claim 6, characterized in that, For the original endoscopic image in frame t, the first processed image in frame t and the second processed image in frame (t-1) are fused in real time to generate a fused image, including: Compare the pixel value of each pixel in the second processed image of frame t-1 with the preset mirror area identifier value; Based on the comparison results, the pixel values ​​of the corresponding pixels in the first processed image in frame t or the second processed image in frame (t-1) are respectively used as the pixel values ​​of the fused image and output to complete the image fusion.

8. The method for optimizing the display of endoscopic images according to claim 7, characterized in that, Based on the comparison results, the pixel values ​​of the corresponding pixels in the first processed image of frame t or the second processed image of frame t-1 are respectively used as the pixel values ​​of the fused image and output, including: Select the pixel (x) in the second processed image of frame t-1. 2,t-1 ,y 2,t-1 ), when pixel (x 2,t-1 ,y 2,t-1 When the pixel value of (x) matches the preset mirror area identifier value, the selected pixel point (x) in the first processed image of frame t is selected. 2,t-1 ,y 2,t-1 The corresponding pixel (x) 1,t ,y 1,t The pixel values ​​of the fused image are used as the pixel values ​​of the fused image and output. When pixel (x) 2,t-1 ,y 2,t-1 When the pixel value of (x) is inconsistent with the preset mirror area identifier value, the pixel point (x) in the second processed image of frame t-1 is... 2,t-1 ,y 2,t-1 The pixel values ​​of the fused image are used as the output. Iterate through each pixel in the second processed image of frame t-1, and obtain the pixel value of each pixel in the fused image in the same way and output it.

9. The method for optimizing the display of endoscopic images according to claim 1, characterized in that, The second image signal is processed in real time to obtain a second processed image, including: Using the ARM processor, the second image signal is subjected to AI recognition to obtain the AI ​​recognition result; User interface information is generated based on the AI ​​recognition results; The second processed image is generated based on the AI ​​recognition results and the user interface information.

10. The method for optimizing the display of endoscopic images according to claim 9, characterized in that, The method further includes: Based on the AI ​​recognition results, the image processing parameters are adjusted during the real-time processing of the first image signal to optimize the real-time processing of the first image signal by the FPGA controller, resulting in an optimized first processed image.

11. A display optimization system for endoscopic images, characterized in that, The method for optimizing the display of endoscopic images as described in any one of claims 1 to 10 includes: An endoscope image acquisition module is used to acquire raw endoscope images in real time and split the raw endoscope images into a first image signal and a second image signal that are transmitted synchronously in real time. The image processing and analysis module is used to receive the first image signal in real time using an FPGA controller and process the first image signal in real time to obtain a first processed image; simultaneously, it uses an ARM processor to receive the second image signal in real time and processes the second image signal in real time to obtain a second processed image; it is also used to receive the second processed image in real time using the FPGA controller and fuse the first processed image and the second processed image in real time to generate a fused image. An image display output module is used to output the fused image.

12. A display optimization device for endoscopic images, characterized in that, include: An image sensor is used to acquire raw endoscopic images in real time and to split the raw endoscopic images into a first image signal and a second image signal that are transmitted synchronously in real time. An FPGA controller, communicatively connected to the image sensor, is used to receive the first image signal transmitted by the image sensor in real time and process the first image signal in real time to obtain a first processed image. An ARM processor is communicatively connected to both the image sensor and the FPGA controller. It is used to receive the second image signal transmitted by the image sensor in real time and process the second image signal in real time to obtain a second processed image. The FPGA controller is also used to receive the second processed image transmitted by the ARM processor, and to fuse the first processed image and the second processed image in real time to generate a fused image; The display is communicatively connected to the FPGA controller and is used to receive and display the fused image.

13. A computer storage medium, characterized in that, The computer storage medium includes at least one instruction that, when executed by a computer, implements the method steps of the method for optimizing the display of endoscopic images as described in any one of claims 1 to 10.

Images