A plug-and-play embedded region of interest real-time recognition overlay display device for existing microscope equipment

Abstract

The present application relates to the technical field of microscopic image auxiliary identification, in particular to a plug-and-play embedded region of interest real-time identification superimposed display device for existing microscopes, which accesses the video link between the existing microscope camera and the original display terminal, acquisition host or pathological workstation, completes real-time image access, image preprocessing, region of interest (ROI) identification, candidate region generation and superimposed display at the embedded end, thereby assisting the operator to quickly locate the key area in the microscopic field of view. The device does not change the original microscope optical path, does not replace the microscope camera, and does not modify the software interface of the original display terminal or pathological workstation, and belongs to the technical scheme of external intelligent enhancement for existing microscopes.

Description

Technical Field

[0001] This invention relates to the field of microscopic image-assisted recognition technology, and more specifically to a plug-and-play embedded region of interest real-time recognition and overlay display device for existing microscope equipment. Background Technology

[0002] Microscopes are essential observation tools in fields such as biomedicine, pathological diagnosis, cell analysis, materials observation, and industrial testing. Traditional optical microscopes are still widely used in hospital pathology departments, cytology screening centers, medical teaching laboratories, and primary healthcare institutions. Operators typically need to observe the current microscopic field of view through an eyepiece or monitor, relying on their experience to locate suspected lesions, densely packed cell areas, areas of nuclear atypia, tissue boundary areas, abnormally stained areas, or other areas requiring focused examination.

[0003] During the aforementioned observation process, due to the limited field of view under a microscope, the operator often needs to repeatedly move the stage, switch objectives, or adjust the focus to locate key areas. This process is significantly affected by the operator's experience, fatigue level, sample complexity, and the speed of field-of-view switching, easily leading to problems such as missing key areas, repeated observation, or low observation efficiency. These problems are particularly pronounced when reviewing images for extended periods, handling large sample sizes, or in grassroots institutions where there is a lack of real-time guidance from senior physicians.

[0004] Existing microscopic image-assisted recognition solutions mainly include server-based digital pathology analysis systems, ordinary microscope video acquisition equipment, PC-based auxiliary recognition software, and some dedicated hardware processing solutions. Server-based digital pathology analysis systems typically require digital slide scanners, high-performance graphics processor servers, or cloud analysis platforms, resulting in high deployment costs and requiring changes to the existing microscope observation workflow. PC-based auxiliary recognition software usually requires the operator to transfer images to a computer for processing or switch to a separate software interface to view the analysis results, making it difficult to provide real-time prompts directly during continuous microscopic observation. Ordinary microscope video acquisition equipment typically only has image acquisition, display, storage, or transmission functions, lacking the ability to perform region of interest identification, layer generation, and overlay display locally.

[0005] In recent years, deep learning and embedded image processing technologies have been applied to areas such as microscopic image recognition, cell image segmentation, digital pathology image analysis, and lightweight image recognition systems. For example, Ke Baosheng et al. proposed a deep learning-based method for detecting mitosis in live cells, which can identify and locate target cell regions in bright-field microscopic images; Xia Ping et al. proposed a fully convolutional deep learning network segmentation method for cell microscopic images with complex texture features; Chen Zhe conducted research on multi-stage image analysis methods for automatic analysis and interpretation of digital pathology images; Li Cong et al. designed an embedded image recognition system based on lightweight convolutional neural networks; and Wang Tingliang designed and implemented an ARM-based embedded image matching processing system. These studies indicate that intelligent microscopic image recognition and embedded image processing both have a certain research foundation, but existing research mainly focuses on algorithm models, offline image analysis, or general embedded image recognition systems, and has not yet fully solved the coordination problems between external access, real-time local ROI recognition, original image overlay display, and bypass pass-through in abnormal AI states in the image link of existing microscope equipment.

[0006] Furthermore, some existing solutions employ FPGAs or dedicated ASICs for real-time image processing. While FPGA solutions offer good real-time performance, their development cycles are lengthy, and algorithm updates and iterations are difficult. Dedicated ASIC solutions are costly and typically suitable for large-scale dedicated equipment, making it difficult to meet the need for low-cost retrofitting of existing microscope equipment. High-end microscope manufacturers offer closed intelligent recognition systems that are expensive and often require users to replace the entire microscope system or use specific brands of cameras, display terminals, and software platforms, making them incompatible with the widely deployed traditional microscope equipment. In summary, existing solutions still have shortcomings in terms of real-time performance, compatibility, flexibility, cost, and ease of deployment. There is an urgent need for a device that can be plugged into existing microscope equipment, perform real-time ROI recognition on an embedded device, and overlay the results onto the original video image.

[0007] Therefore, existing technologies still lack an external embedded device that can be directly inserted into existing microscope image links, retain the original display process, complete real-time region of interest identification and overlay display on the embedded end, and maintain the normal display of the original image when the artificial intelligence function is turned off, malfunctions, or is updated. Summary of the Invention

[0008] To address the shortcomings of the prior art, this invention provides a plug-and-play embedded real-time region of interest (ROI) identification and overlay display device for existing microscope equipment. This solves the problems of high equipment replacement costs, reliance on server or PC software, significant process modifications, insufficient real-time overlay capabilities, poor compatibility with existing equipment, and susceptibility of the original display link to AI processing links in existing microscopic image-assisted identification systems. The device is inserted between the existing microscope camera and the original display terminal, acquisition host, or pathology workstation. The real-time image signal output from the microscope camera first enters this device, and then is output from this device to the original display equipment. Without changing the microscope's optical path, replacing the microscope camera, or modifying the original display terminal's hardware configuration and software interface, this invention can complete real-time image access, local inference, ROI candidate region generation, and overlay display, allowing candidate key regions to be directly presented on the original display interface for review by doctors, technicians, teachers, or researchers.

[0009] The purpose of this invention is to provide a plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment. This device is positioned between the existing microscope camera and the existing display terminal, acquisition host, or pathology workstation, and includes: The image input interface unit is used to receive real-time microscope image signals output by existing microscope cameras; The video signal detection and adaptation unit is used to automatically identify the interface type, image resolution, frame rate, color format, image orientation, and output display parameters of the connected real-time microscope image signal, and configure the internal signal processing link accordingly to obtain the adapted digital image signal. The image frame buffer unit is used to temporarily store at least one complete frame of image data for the adapted digital image signal using a double or multiple buffering mechanism, so as to eliminate timing jitter between input and processing and obtain stable original image data. The image preprocessing unit is used to read the original image data from the image frame buffer unit and perform image preprocessing operations to obtain the preprocessed image; the preprocessing operations include color space conversion, size scaling, contrast enhancement, noise filtering, and region of interest cropping. The embedded AI inference unit performs real-time inference on the preprocessed image locally to identify multiple candidate ROI regions in the image; The ROI candidate region generation unit is used to obtain stable candidate ROI results from multiple candidate ROI regions based on preset confidence thresholds and region filtering rules. The ROI overlay display unit is used to convert candidate ROI results into a visualization layer and fuse them with the original microscope image frame to obtain the overlaid image signal; The video output unit converts the superimposed image signal into a video format compatible with the original monitor, acquisition host, or pathology workstation and outputs it. The embedded main control processing unit is used to control the status detection, task scheduling, parameter configuration and exception handling of each unit.

[0010] In one embodiment, the embedded main control processing unit further includes a video signal detection and adaptation unit, an image frame buffer unit, an embedded AI inference unit, an ROI overlay display unit, a video output unit, and a video bypass pass-through unit connected via GPIO, UART, I2C, SPI, or a high-speed bus to realize input state detection, inference state monitoring, overlay display control, and bypass switching control.

[0011] In one embodiment, the image frame caching unit sets a frame number or timestamp for the input image frame, the candidate region information output by the ROI candidate region generation unit is bound to the corresponding frame number or timestamp, and the ROI overlay display unit only overlays the candidate region information onto the corresponding original image frame or its synchronous output frame to reduce overlay misalignment caused by inference delay.

[0012] In one embodiment, the embedded AI inference unit is implemented using an embedded neural network processor, graphics processor, digital signal processor, or other computing unit for edge inference; the embedded AI inference unit loads a lightweight ROI recognition model package that has been trained and adapted to the embedded operating environment, identifies candidate key regions in the current microscope field of view, and outputs relevant information about the candidate regions.

[0013] In one embodiment, the structured data output by the embedded AI inference unit is written into a result cache and used by the ROI candidate region generation unit and the ROI overlay display unit; wherein, the structured data includes candidate ROI coordinates, category, confidence level or display level.

[0014] In one embodiment, obtaining stable candidate ROI results includes: The ROI candidate region generation unit receives the output of the embedded AI inference unit and generates a list of candidate regions of interest according to a preset confidence threshold and region filtering rules. For multiple candidate ROI regions output by the embedded AI inference unit, the ROI candidate region generation unit can filter low-confidence regions according to the confidence threshold and remove redundant regions with high overlap through non-maximum suppression, thereby generating stable candidate ROI results. These candidate ROI results are used for subsequent overlay display as an auxiliary observation prompt for the operator to review. The candidate region of interest list may include candidate region coordinates, region size, region category, confidence score, and display level.

[0015] In one embodiment, the visualization layer includes a semi-transparent rectangle, a label, a highlighted outline, or a numbered identifier; the ROI overlay display unit overlays the generated candidate ROI information onto the original image frame to form a fused display image. This overlay operation can be completed at the output end of the image frame buffer unit or on the video output path to ensure that the overlay information is synchronized with the original image.

[0016] In one embodiment, the video output unit may employ one or more of the following interfaces: HDMI, DisplayPort, USB UVC, SDI, or network video stream output. The video output unit outputs the superimposed image frames directly to the original monitor or pathology workstation via the HDMI interface, allowing the operator to observe the microscope image with ROI prompt information on the original display screen. Alternatively, the video output unit can simulate a standard video input source by using the USB UVC protocol to overlay the image frames, enabling the acquisition host or third-party image software to recognize this device as a camera device.

[0017] In one embodiment, the device further includes: The video bypass pass-through unit is located between the image input interface unit and the video output unit, and is connected in parallel with the AI ​​processing link consisting of the image frame buffer unit, the image preprocessing unit, the embedded AI inference unit, the ROI candidate region generation unit, and the ROI overlay display unit. It is used to ensure the normal output of the original microscope image when the AI ​​processing link is unavailable or the user turns off the overlay function.

[0018] In one embodiment, the video bypass pass-through unit is implemented by one or more of a high-speed video multiplexer, a video cross switch, an FPGA video selection module, an HDMI bypass chip, and a hardware relay, and its control terminal is connected to the embedded main control processing unit. When the AI ​​inference unit is working normally and the overlay display function is enabled, the video bypass pass-through unit will select the enhanced image signal output by the ROI overlay display unit to the video output unit. When the AI ​​inference unit is not enabled, malfunctions, the system is started or initialized, the model is updated, inference times out, the temperature is abnormal, or the user turns off the overlay display function, the video bypass pass-through unit prioritizes the original video pass-through path and directly outputs the original image signal received by the image input interface unit to the original display terminal or pathology workstation.

[0019] This invention provides a plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment, which has the following advantages compared to existing microscopic image-assisted identification systems: 1) Reduce the cost of upgrading existing microscope equipment. This invention can be directly inserted between existing microscope cameras and display terminals, acquisition hosts or pathology workstations, without the need to replace the entire microscope, microscope camera, monitor, digital slide scanner or server system, thereby reducing the cost of intelligent upgrading of existing microscope equipment in hospitals, laboratories and primary care institutions.

[0020] 2) Retain the original observation process. This invention does not change the microscope optical path, does not replace the original microscope camera, and does not modify the original display terminal hardware configuration and pathology workstation software interface, so that the operator can still observe microscope images in the original display interface, avoiding workflow interruptions caused by switching platforms, uploading images, or waiting for offline analysis results.

[0021] 3) Improved real-time auxiliary observation capabilities. This invention completes image access, image preprocessing, local inference, candidate ROI generation, and overlay display on the embedded device, so that candidate key areas are directly presented on the original display screen, and the operator can obtain auxiliary prompts during real-time observation.

[0022] 4) Improve the reliability of the display link. This invention sets up a video bypass direct-through unit, which can directly output the original microscope image signal to the original display terminal when the AI ​​inference unit is not enabled, is running abnormally, the system is starting up and initializing, the model is being updated, or the user disables the overlay display function, thus avoiding the impact on normal microscope observation due to the unavailability of the AI ​​processing link.

[0023] 5) Improved device compatibility. This invention can adapt to different microscope cameras and display terminals with various input methods such as HDMI, USB, Type-C, GigE Vision, Camera Link, MIPI CSI, SDI, or network video streams, and can output via HDMI, DisplayPort, USB UVC, SDI, or network video streams, facilitating connection to microscope equipment of different brands and models.

[0024] 6) Reduces the burden of manually searching for key areas. This invention provides auxiliary prompts for candidate ROIs, helping operators focus on key areas such as suspected lesion areas, densely celled areas, areas with nuclear atypia, tissue boundary areas, or abnormally stained areas, thus improving the efficiency of candidate area localization. It should be noted that this invention provides auxiliary prompts and does not directly replace the operator's professional judgment. Attached Figure Description

[0025] Figure 1 This is a diagram showing the overall hardware structure of the device of the present invention; Figure 2 This is a schematic diagram of the minimum system circuit for the embedded main control processing unit; Figure 3This is a schematic diagram of microscope image processing and ROI overlay display; Figure 4 This is a flowchart illustrating the operation of the device of the present invention. Detailed Implementation

[0026] In order to illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description is provided in conjunction with the embodiments.

[0027] The purpose of this invention is to provide a plug-and-play embedded real-time region of interest (ROI) identification and overlay display device for existing microscope equipment. This addresses the problems of existing microscopic image-assisted identification systems, such as high equipment replacement costs, reliance on server or PC software, significant process modifications, insufficient real-time overlay capabilities, poor compatibility with existing equipment, and susceptibility of the original display link to AI processing links. This device connects to the video link between the existing microscope camera and the existing display terminal, acquisition host, or pathology workstation. It performs real-time image access, image preprocessing, ROI identification, candidate region generation, and overlay display on the embedded end, thereby assisting the operator in quickly locating key areas in the microscopic field of view. This device does not alter the original microscope optical path, replace the microscope camera, or modify the software interface of the original display terminal or pathology workstation. It represents a technical solution for externally enhancing the intelligence of existing microscope equipment.

[0028] This invention sets up a video bypass direct-through mechanism. When the AI ​​function is turned off, the AI ​​inference unit malfunctions, the system is started or initialized, the model is updated, or the user turns off the overlay display function, the device can directly output the input raw microscope image signal to the original display terminal without going through the AI ​​inference unit and ROI overlay display unit, thereby ensuring that the original microscope image display process is not interrupted.

[0029] It should be noted that this device outputs candidate ROI (Region of Interest) suggestions for operator review and does not directly replace human diagnostic conclusions. This invention aims to reduce the burden of manually searching for key areas and improve the efficiency of candidate region localization. This device only identifies and visualizes candidate ROIs in microscopic images and does not output disease diagnostic conclusions; the final observation conclusion is made independently by the operator.

[0030] To achieve the above objectives, see Figure 1 As shown, a plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment is provided. This device is positioned between the existing microscope camera and the existing display terminal, acquisition host, or pathology workstation, and includes: The image input interface unit is used to receive real-time microscope image signals output by existing microscope cameras; For example, the image input interface unit is used to receive real-time microscope image signals output by an existing microscope camera. This unit can support one or more of the following interfaces: HDMI, USB, Type-C, GigE Vision, Camera Link, MIPI CSI, SDI, or network video streaming, to adapt to the output methods of different brands and models of microscope cameras.

[0031] The video signal detection and adaptation unit is used to automatically identify the interface type, image resolution, frame rate, color format, image orientation, and output display parameters of the connected real-time microscope image signal, and configure the internal signal processing link accordingly to obtain the adapted digital image signal. The video signal detection and adaptation unit is connected to the image input interface unit to automatically identify the interface type, image resolution, frame rate, color format, image orientation, and output display parameters of the input video signal, and configure the internal signal processing link accordingly. This allows users to connect to the device without making complex modifications to their existing microscopes, microscope cameras, monitors, or pathology workstations, thus achieving plug-and-play access.

[0032] The image frame buffer unit is used to temporarily store at least one complete frame of image data for the adapted digital image signal using a double or multiple buffering mechanism, so as to eliminate timing jitter between input and processing and obtain stable original image data. The image frame caching unit sets a frame number or timestamp for the input image frame. The candidate region information output by the ROI candidate region generation unit is bound to the corresponding frame number or timestamp. The ROI overlay display unit only overlays the candidate region information onto the corresponding original image frame or its synchronous output frame to reduce overlay misalignment caused by inference delay.

[0033] For example, the adapted digital image signal is sent to an image frame buffer unit. The image frame buffer unit employs a double-buffering or multi-buffering mechanism to temporarily store at least one complete frame of image data, eliminating timing jitter between input and processing, and providing a stable data source for subsequent image preprocessing, AI inference, and overlay display. Simultaneously, the image frame buffer unit also ensures the correspondence between the original image frame and the candidate ROI prompt information, enabling the subsequent ROI overlay display unit to overlay the candidate region prompt onto the corresponding original microscopic image frame, reducing overlay misalignment, image tearing, or output stuttering. In one implementation, the image frame buffer unit sets a frame number or timestamp for the input image frame. The candidate region information output by the ROI candidate region generation unit is bound to the corresponding frame number or timestamp. The ROI overlay display unit only overlays the candidate region information onto the corresponding original image frame or its synchronous output frame, reducing overlay misalignment caused by inference latency.

[0034] The image preprocessing unit is used to read the original image data from the image frame buffer unit and perform image preprocessing operations to obtain the preprocessed image; the preprocessing operations include color space conversion, size scaling, contrast enhancement, noise filtering, and region of interest cropping. For example, the image preprocessing unit reads the raw image data from the image frame buffer unit and performs necessary preprocessing operations on the image. These preprocessing operations include, but are not limited to, color space conversion, size scaling, contrast enhancement, noise filtering, and region of interest (ROI) cropping. The preprocessed image data is then sent to the embedded AI inference unit. The embedded AI inference unit is deployed inside the embedded device to perform real-time inference locally and identify candidate ROIs in the image. This inference unit can be implemented using an embedded neural network processor, graphics processor, digital signal processor, or other computing units that can be used for edge inference. The embedded AI inference unit loads a lightweight ROI recognition model package that has been trained and adapted to the embedded operating environment, identifies candidate key regions in the current microscope field of view, and outputs relevant information about the candidate regions. The focus of this invention is not on protecting the model training algorithm or training dataset separately, but on deploying the model package to the embedded device and enabling it to work in conjunction with real-time video acquisition, candidate ROI generation, overlay display, and bypass pass-through links.

[0035] The embedded AI inference unit performs real-time inference on the preprocessed image locally to identify multiple candidate ROI regions in the image; The embedded AI inference unit is implemented using an embedded neural network processor, graphics processor, digital signal processor, or other computing units used for edge inference. The embedded AI inference unit loads a lightweight ROI recognition model package that has been trained and adapted to the embedded operating environment, identifies candidate key regions in the current microscope field of view, and outputs relevant information about the candidate regions.

[0036] The structured data output by the embedded AI inference unit is written into the result cache and used by the ROI candidate region generation unit and the ROI overlay display unit; wherein, the structured data includes candidate ROI coordinates, category, confidence level or display level.

[0037] The embedded AI inference unit outputs structured data such as candidate ROI coordinates, categories, confidence levels, or display levels, which are then written to a result cache and accessed by the ROI candidate region generation unit and the ROI overlay display unit. Therefore, the AI ​​inference results are not output as independent diagnostic conclusions, but rather work collaboratively with the frame cache, ROI generation, overlay display, and video output links to provide real-time auxiliary prompts. In different observation scenarios, the embedded AI inference unit can load lightweight ROI recognition model packages adapted to sample types, observation magnification, or application requirements.

[0038] The ROI candidate region generation unit is used to obtain stable candidate ROI results from multiple candidate ROI regions based on preset confidence thresholds and region filtering rules. The process of obtaining stable candidate ROI results includes: The ROI candidate region generation unit receives the output of the embedded AI inference unit and generates a list of candidate regions of interest according to a preset confidence threshold and region filtering rules. For multiple candidate ROI regions output by the embedded AI inference unit, the ROI candidate region generation unit can filter low-confidence regions according to the confidence threshold and remove redundant regions with high overlap through non-maximum suppression, thereby generating stable candidate ROI results. These candidate ROI results are used for subsequent overlay display as an auxiliary observation prompt for the operator to review. The candidate region of interest list may include candidate region coordinates, region size, region category, confidence score, and display level.

[0039] For example, the ROI candidate region generation unit receives the output of the embedded AI inference unit and generates a candidate region of interest list according to a preset confidence threshold and region filtering rules. The candidate region of interest list may include information such as candidate region coordinates, region size, region category, confidence score, and display level. For multiple candidate regions output by the inference unit, the ROI candidate region generation unit can filter low-confidence regions according to the confidence threshold and remove redundant regions with high overlap through non-maximum suppression, thereby generating a relatively stable candidate ROI result. This candidate ROI result is used for subsequent overlay display as an auxiliary observation prompt for the operator to review.

[0040] The ROI overlay display unit is used to convert candidate ROI results into a visualization layer and fuse them with the original microscope image frame to obtain the overlaid image signal; The visualization layer includes a semi-transparent rectangle, labels, highlighted outlines, or numbered identifiers; the ROI overlay display unit overlays the generated candidate ROI information onto the original image frame to form a fused display image. This overlay operation can be completed at the output end of the image frame buffer unit or on the video output path to ensure that the overlay information is synchronized with the original image.

[0041] For example, the ROI overlay display unit is used to convert candidate region of interest information into a visualization layer and fuse it with the original microscope image frame. The visualization layer may include a semi-transparent rectangle, label, highlighted outline, numbered identifier, or other prompt graphics. The ROI overlay display unit overlays the generated candidate ROI information onto the original image frame to form a fused display image. This overlay operation can be completed at the output end of the image frame buffer unit or on the video output path to ensure that the overlay information is synchronized with the original image. In this way, the operator does not need to switch software interfaces or wait for offline analysis results to see the candidate ROI prompts on the original display interface.

[0042] The video output unit converts the superimposed image signal into a video format compatible with the original monitor, acquisition host, or pathology workstation and outputs it. The video output unit may adopt one or more of the following interfaces: HDMI, DisplayPort, USB UVC, SDI, or network video stream output interface; The video output unit outputs the superimposed image frames directly to the original monitor or pathology workstation via the HDMI interface, allowing the operator to observe the microscope image with ROI prompt information on the original display screen. Alternatively, the video output unit can simulate a standard video input source by using the USB UVC protocol to overlay the image frames, enabling the acquisition host or third-party image software to recognize this device as a camera device.

[0043] For example, the video output unit is used to convert the superimposed image signal into a video format compatible with the original display, acquisition host, or pathology workstation and output it. The video output unit can adopt one or more of HDMI, DisplayPort, USB UVC, SDI, or network video stream output interfaces. In one embodiment, the video output unit directly outputs the superimposed image frame to the original display or pathology workstation through the HDMI interface, so that the operator can observe the microscope image with ROI prompt information on the original display screen. In another embodiment, the video output unit can simulate the superimposed image frame as a standard video input source through the USB UVC protocol, so that the acquisition host or third-party image software can recognize this device as a camera device, thereby reducing the need for modification to the original software architecture.

[0044] The embedded main control processing unit is used to control the status detection, task scheduling, parameter configuration and exception handling of each unit.

[0045] The embedded main control processing unit also includes a video signal detection and adaptation unit, an image frame buffer unit, an embedded AI inference unit, an ROI overlay display unit, a video output unit, and a video bypass pass-through unit connected via GPIO, UART, I2C, SPI, or a high-speed bus, to realize input state detection, inference state monitoring, overlay display control, and bypass switching control.

[0046] The embedded main control processing unit, as the core control node of the device, is responsible for the status detection, task scheduling, parameter configuration, and anomaly handling of each functional unit. This unit can be implemented using an embedded microprocessor, microcontroller, system-on-a-chip, or edge computing module, and can run a real-time operating system or bare-metal program. As a specific implementation, the embedded main control processing unit can adopt a minimum system circuit, which includes a power supply and voltage regulation section, a clock section, a reset and startup configuration section, a debug interface, a serial communication interface, and an I2C communication interface. The power supply and voltage regulation section provides a stable operating voltage; the clock section provides the system clock; the reset and startup configuration section selects the reset and startup modes; the debug interface is used for program download and system debugging; and the serial and I2C communication interfaces are used to read the status of external modules, configure video processing parameters, or transmit operation logs. The embedded main control processing unit can also be connected to the video signal detection and adaptation unit, image frame buffer unit, embedded AI inference unit, ROI overlay display unit, video output unit, and video bypass pass-through unit via GPIO, UART, I2C, SPI, or high-speed bus to realize input state detection, inference state monitoring, overlay display control, and bypass switching control. It should be noted that... (See also...) Figure 2 The diagram shows only a minimum system circuit of the embedded main control processing unit and does not limit the specific chip model, pin name or package form, nor does it represent the complete hardware structure of the device of the present invention.

[0047] In one specific implementation, the embedded main control processing unit may include a power supply and voltage regulation circuit, a clock circuit, a reset and startup configuration circuit, a debugging interface, a serial communication interface, and an I2C communication interface, etc., to provide the main control processing unit with power supply, clock, reset, debugging, and external communication infrastructure.

[0048] Furthermore, combined Figure 1 The device hardware connection relationship shown is that, based on stable power supply, system clock, reset startup and debugging interface, the embedded main control processing unit establishes control and communication connections with each functional unit inside the device through GPIO, UART, I2C, SPI or high-speed bus, thereby realizing status detection, task scheduling, parameter configuration and exception handling of each functional unit.

[0049] See Figure 2The diagram shows the minimum system circuit of the embedded main control processing unit. The embedded main control processing unit controls the status detection, task scheduling, parameter configuration, and exception handling of each unit, as detailed below: In terms of status detection, the embedded main control processing unit can obtain the video access status of the image input interface unit, the interface type identification status of the video signal detection and adaptation unit, the image resolution and frame rate identification status, the buffer occupancy status of the image frame buffer unit, the model loading status and inference running status of the embedded AI inference unit, the overlay display status of the ROI overlay display unit, the output status of the video output unit, and the path selection status of the video bypass pass-through unit through GPIO level detection, I2C register reading, UART status feedback or high-speed bus feedback.

[0050] In terms of task scheduling, the embedded main control processing unit coordinates the sequential operation of functional units such as image input, signal adaptation, frame buffering, image preprocessing, local inference, ROI candidate region generation, ROI overlay display, and video output based on the input video signal status, image frame buffering status, and the running status of the embedded AI inference unit. When the image frame buffering unit completes buffering the current image frame, the embedded main control processing unit controls the image preprocessing unit to read the corresponding frame data. After preprocessing is complete, it controls the embedded AI inference unit to perform local inference. After the inference result is written to the result buffer, it controls the ROI candidate region generation unit to perform candidate region filtering and controls the ROI overlay display unit to overlay the candidate ROI information onto the corresponding original microscope image frame to ensure synchronization between the image frame and the ROI prompt information.

[0051] In terms of parameter configuration, the embedded main control processing unit can configure the image resolution, frame rate, color format, image orientation, caching mode, preprocessing parameters, ROI confidence threshold, region filtering rules, overlay display style, and output video format of the internal image processing link according to the input signal parameters fed back by the video signal detection and adaptation unit; it can also control the embedded AI inference unit to load or switch the corresponding lightweight ROI recognition model package according to different sample types, observation magnification, or application scenarios.

[0052] In terms of anomaly handling, when the embedded main control processing unit detects an input signal interruption, image frame buffer anomaly, AI inference timeout, model loading failure, temperature anomaly, overlay display anomaly, or the user disables the AI ​​overlay function, the embedded main control processing unit controls the video bypass pass-through unit to prioritize the original video pass-through path through GPIO or other control interfaces, so that the original microscope image signal received by the image input interface unit is directly output to the original display terminal, acquisition host, or pathology workstation; when the abnormal state is resolved and the AI ​​processing link returns to normal, the embedded main control processing unit then controls the video bypass pass-through unit to switch to the enhanced image output path after ROI overlay display.

[0053] In this invention, the device further includes: The video bypass pass-through unit is located between the image input interface unit and the video output unit, and is connected in parallel with the AI ​​processing link consisting of the image frame buffer unit, the image preprocessing unit, the embedded AI inference unit, the ROI candidate region generation unit, and the ROI overlay display unit. It is used to ensure the normal output of the original microscope image when the AI ​​processing link is unavailable or the user turns off the overlay function.

[0054] The video bypass pass-through unit is implemented through one or more of the following: high-speed video multiplexer, video cross switch, FPGA video selection module, HDMI bypass chip, and hardware relay, and its control terminal is connected to the embedded main control processing unit. When the AI ​​inference unit is working normally and the overlay display function is enabled, the video bypass pass-through unit will select the enhanced image signal output by the ROI overlay display unit to the video output unit. When the AI ​​inference unit is not enabled, malfunctions, the system is started or initialized, the model is updated, inference times out, the temperature is abnormal, or the user turns off the overlay display function, the video bypass pass-through unit prioritizes the original video pass-through path and directly outputs the original image signal received by the image input interface unit to the original display terminal or pathology workstation.

[0055] It should be noted that, Figure 1 The main hardware modules of this device and their interconnections are demonstrated, including an image input interface unit, a video signal detection and adaptation unit, an image frame buffer unit, an image preprocessing unit, an embedded AI inference unit, a ROI candidate region generation unit, an ROI overlay display unit, a video output unit, a video bypass pass-through unit, and an embedded main control processing unit. The main processing link is used to complete the microscopic image input, preprocessing, embedded AI inference, ROI generation, and overlay display; the bypass pass-through link is used to maintain the original microscopic image output when the AI ​​function is disabled or malfunctioning; and the embedded main control processing unit is used to control, schedule, and monitor the status of each functional unit.

[0056] Figure 1 This invention demonstrates an overall hardware connection relationship of the device, without limiting the specific chip models, number of interfaces, or integration methods of each functional unit. In actual implementation, the image input interface unit, video output unit, embedded AI inference unit, ROI overlay display unit, and video bypass pass-through unit can be integrated, split, or equivalently replaced according to the specific product solution. As long as the existing microscope image link access, local ROI recognition, overlay display, and original video bypass pass-through functions can be achieved, they all fall within the technical concept scope of this invention.

[0057] To further illustrate the plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment provided by the present invention, the following description is provided in conjunction with the accompanying drawings.

[0058] See Figure 3 As shown, after the original microscope image output by the microscope camera is connected to this device, it can be preprocessed according to the recognition requirements, such as local cropping, size scaling, contrast enhancement, and noise filtering, to form a locally cropped / preprocessed image. The embedded AI inference unit performs local inference on the preprocessed image and outputs a candidate ROI recognition image. The ROI overlay display unit overlays the candidate ROI recognition image onto the original or synchronous image frame in the form of a box selection, highlighting, or semi-transparent area to form an ROI overlay display output image.

[0059] See Figure 4 The diagram illustrates the working process of this device from connecting to an existing microscope image link to outputting and displaying images, including steps such as input signal recognition and adaptation, image frame buffering, image preprocessing, embedded AI local inference, ROI candidate region generation, ROI overlay display, video output, and bypass pass-through.

[0060] The overall working process of the device of the present invention is as follows: Figure 4 As shown, the main steps include device access, input signal recognition and adaptation, image frame buffering and preprocessing, embedded local inference, candidate ROI generation and overlay display, video output, and bypass pass-through.

[0061] Step 1: Connect the device to the existing microscope image link: The device of this invention is connected between an existing microscope camera and a pre-existing display terminal, acquisition host, or pathology workstation. This allows the real-time image signal output from the microscope camera to first enter the device of this invention, and then be output from the device to the existing display device. Through this connection method, the device of this invention does not require alteration of the microscope's optical path, nor does it require replacement of the microscope camera or display terminal.

[0062] Step 2: Input Signal Identification and Adaptation The image input interface unit receives the video signal output from the microscope camera. The video signal detection and adaptation unit automatically identifies the input interface type, resolution, frame rate, color format, and image orientation, and configures the internal processing links based on the identification results, so that the input video signal can be processed by the subsequent image frame buffer unit, image preprocessing unit, and embedded AI inference unit.

[0063] Step 3: Image frame buffering and preprocessing: The image frame buffer unit performs double or multiple buffering on the input image to temporarily store complete image frames and reduce timing jitter between input and processing. The image preprocessing unit reads image data from the image frame buffer unit and performs processing such as resizing, color space conversion, contrast enhancement, noise filtering, or region of interest cropping according to the input requirements of the embedded AI inference unit.

[0064] Step 4: Embedded Local Inference: The embedded AI inference unit loads a lightweight ROI recognition model package adapted to the embedded operating environment and performs local inference on the preprocessed image frames to identify candidate regions of interest in the current microscope field of view. The inference results may include candidate region coordinates, region size, region category, or confidence information.

[0065] Step 5: Candidate ROI Generation and Overlay Display: The ROI candidate region generation unit generates a list of candidate ROIs based on the output of the embedded AI inference unit, combined with preset confidence thresholds and region selection rules. The ROI overlay display unit overlays the candidate regions onto the original microscope image frame as semi-transparent rectangles, labels, or highlighted outlines to form an enhanced display image. This enhanced display image is used to draw the operator's attention to key areas that may require review.

[0066] Step Six: Video Output The video output unit converts the enhanced display image into a video format compatible with the existing monitor, acquisition host, or pathology workstation, and outputs it to the existing display device. Operators can directly observe microscope images with candidate ROI prompts in the existing display interface without switching to an additional software interface.

[0067] Step 7: Bypass Through: When the AI ​​function is disabled, the AI ​​inference unit malfunctions, the system starts up and initializes, the model is updated, inference times out, the temperature is abnormal, or the user disables the overlay display function, the video bypass unit directly outputs the input raw microscope image signal to the video output unit, bypassing the AI ​​inference and ROI overlay display links, thus ensuring that the original microscope image display is unaffected. In one specific implementation, the video bypass unit can be implemented using a high-speed video multiplexer, a video cross switch, an FPGA video selection module, an HDMI bypass chip, or a hardware relay. When the embedded main control processing unit detects that the AI ​​inference unit is working normally and the overlay display function is enabled, it controls the video bypass unit to output an enhanced video signal; when it detects that the AI ​​inference unit is not enabled, malfunctions, inference times out, the temperature is abnormal, or the overlay display function is disabled, it controls the video bypass unit to prioritize outputting the raw video signal.

[0068] Step 8: Control and Communication Implementation of the Embedded Main Control Processing Unit: In one specific implementation, the embedded main control processing unit is implemented using a minimum system circuit and establishes communication connections with various functional units within the device through a control interface. This minimum system circuit includes a power supply and voltage regulation section, a clock section, a reset and startup configuration section, and an external communication interface section.

[0069] The power supply and voltage regulation section includes an external power input terminal and a first-stage buck converter circuit to convert the external input voltage into the low-voltage power required for the main control processing unit to operate. The clock section includes an external crystal oscillator and its matching capacitor to provide a stable system clock for the main control processing unit. The reset and startup configuration section includes a reset circuit and a startup configuration circuit to implement system reset and startup mode selection.

[0070] The external communication interface section includes an SWD debugging interface, a UART serial communication interface, and an I2C communication interface. The SWD debugging interface is used for program downloading and system debugging; the UART serial communication interface is used for serial data communication with external modules; and the I2C communication interface, connected to the power supply via a pull-up resistor, is used for communication with external low-speed peripherals or configuration modules.

[0071] It should be noted that the above circuit is only a minimum system implementation of the embedded main control processing unit, and does not limit the specific main control chip model, pin name, number of interfaces, or package form used in this invention. In other embodiments, the embedded main control processing unit can also be implemented using other microcontrollers, embedded processors, system-on-a-chip, or edge computing modules, depending on the actual computing power and interface requirements.

[0072] Through the aforementioned minimum system circuit, the embedded main control processing unit can complete the control, status detection, parameter configuration, and communication scheduling of various functional units within the device, providing a control basis for subsequent image input, ROI recognition, overlay display, and bypass switching.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment, characterized in that, The device is installed between the existing microscope camera and the existing display terminal, acquisition host, or pathology workstation, and includes: The image input interface unit is used to receive real-time microscope image signals output by existing microscope cameras; The video signal detection and adaptation unit is used to automatically identify the interface type, image resolution, frame rate, color format, image orientation, and output display parameters of the connected real-time microscope image signal, and configure the internal signal processing link accordingly to obtain the adapted digital image signal. The image frame buffer unit is used to temporarily store at least one complete frame of image data for the adapted digital image signal using a double or multiple buffering mechanism, so as to eliminate timing jitter between input and processing and obtain stable original image data. The image preprocessing unit is used to read the original image data from the image frame buffer unit and perform image preprocessing operations to obtain the preprocessed image; the preprocessing operations include color space conversion, size scaling, contrast enhancement, noise filtering, and region of interest cropping. The embedded AI inference unit performs real-time inference on the preprocessed image locally to identify multiple candidate ROI regions in the image; The ROI candidate region generation unit is used to obtain stable candidate ROI results from multiple candidate ROI regions based on preset confidence thresholds and region filtering rules. The ROI overlay display unit is used to convert candidate ROI results into a visualization layer and fuse them with the original microscope image frame to obtain the overlaid image signal; The video output unit converts the superimposed image signal into a video format compatible with the original monitor, acquisition host, or pathology workstation and outputs it. The embedded main control processing unit is used to control the status detection, task scheduling, parameter configuration and exception handling of each unit.

2. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The embedded main control processing unit also includes a video signal detection and adaptation unit, an image frame buffer unit, an embedded AI inference unit, an ROI overlay display unit, a video output unit, and a video bypass pass-through unit connected via GPIO, UART, I2C, SPI, or a high-speed bus, to realize input state detection, inference state monitoring, overlay display control, and bypass switching control.

3. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The image frame caching unit sets a frame number or timestamp for the input image frame. The candidate region information output by the ROI candidate region generation unit is bound to the corresponding frame number or timestamp. The ROI overlay display unit only overlays the candidate region information onto the corresponding original image frame or its synchronous output frame to reduce overlay misalignment caused by inference delay.

4. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The embedded AI inference unit is implemented using an embedded neural network processor, graphics processor, digital signal processor, or other computing units used for edge inference. The embedded AI inference unit loads a lightweight ROI recognition model package that has been trained and adapted to the embedded operating environment, identifies candidate key regions in the current microscope field of view, and outputs relevant information about the candidate regions.

5. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The structured data output by the embedded AI inference unit is written into the result cache and used by the ROI candidate region generation unit and the ROI overlay display unit; wherein, the structured data includes candidate ROI coordinates, category, confidence level or display level.

6. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The process of obtaining stable candidate ROI results includes: The ROI candidate region generation unit receives the output of the embedded AI inference unit and generates a list of candidate regions of interest according to a preset confidence threshold and region filtering rules. For multiple candidate ROI regions output by the embedded AI inference unit, the ROI candidate region generation unit can filter low-confidence regions according to the confidence threshold and remove redundant regions with high overlap through non-maximum suppression, thereby generating stable candidate ROI results. These candidate ROI results are used for subsequent overlay display as an auxiliary observation prompt for the operator to review. The candidate region of interest list may include candidate region coordinates, region size, region category, confidence score, and display level.

7. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The visualization layer includes a semi-transparent rectangle, labels, highlighted outlines, or numbered identifiers; the ROI overlay display unit overlays the generated candidate ROI information onto the original image frame to form a fused display image. This overlay operation can be completed at the output end of the image frame buffer unit or on the video output path to ensure that the overlay information is synchronized with the original image.

8. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The video output unit may adopt one or more of the following interfaces: HDMI, DisplayPort, USB UVC, SDI, or network video stream output interface; The video output unit outputs the superimposed image frames directly to the original monitor or pathology workstation via the HDMI interface, allowing the operator to observe the microscope image with ROI prompt information on the original display screen. Alternatively, the video output unit can simulate a standard video input source by using the USB UVC protocol to overlay the image frames, enabling the acquisition host or third-party image software to recognize this device as a camera device.

9. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 1, characterized in that, The device also includes: The video bypass pass-through unit is located between the image input interface unit and the video output unit, and is connected in parallel with the AI ​​processing link consisting of the image frame buffer unit, the image preprocessing unit, the embedded AI inference unit, the ROI candidate region generation unit, and the ROI overlay display unit. It is used to ensure the normal output of the original microscope image when the AI ​​processing link is unavailable or the user turns off the overlay function.

10. The plug-and-play embedded region of interest real-time identification and overlay display device for existing microscope equipment according to claim 9, characterized in that, The video bypass pass-through unit is implemented through one or more of the following: high-speed video multiplexer, video cross switch, FPGA video selection module, HDMI bypass chip, and hardware relay, and its control terminal is connected to the embedded main control processing unit. When the AI ​​inference unit is working normally and the overlay display function is enabled, the video bypass pass-through unit will select the enhanced image signal output by the ROI overlay display unit to the video output unit. When the AI ​​inference unit is not enabled, malfunctions, the system is started or initialized, the model is updated, inference times out, the temperature is abnormal, or the user turns off the overlay display function, the video bypass pass-through unit prioritizes the original video pass-through path and directly outputs the original image signal received by the image input interface unit to the original display terminal or pathology workstation.

Images