Intelligent medical display system
Through the video overlay technology of the intelligent medical display system, the problem of insufficient image processing speed and stability in the prior art is solved, multi-function display and high-quality image output are realized, and real-time and accuracy of the surgical process are improved.
Patent Information
- Application Number
- PCT/CN2024/090491
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-12
- Filing Date
- 2024-04-29
- Publication Date
- 2024-11-21
AI Technical Summary
The existing medical display system has limitations in image processing speed, stability and versatility, and cannot switch multiple display modes, image quality degradation, system stability and delay problems, affecting the real-time and accuracy of the surgical process. sex.
An intelligent medical display system is designed, including a video acquisition module, an AI edge computing module, a marking layer output module and a video overlay module. Through video overlay technology, image quality and system stability are improved, multi-function display is realized, and image processing delay is reduced.
Three display modes are provided, which improve image quality and system stability, reduce image processing delays, improve real-time and accuracy during surgery, and support multimodal diagnosis and clinical technical practice.
Smart Images

Figure CN2024090491_21112024_PF_FP_ABST
Abstract
Description
An intelligent medical display system
[0001] Cross-reference instructions
[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 12, 2023, with application number 202310536781.5 and invention name “A Smart Medical Display System”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present invention belongs to the technical field of medical devices and relates to a medical display system, in particular to an intelligent medical display system. Background Art
[0004] With the rapid development of medical imaging and information technology, a growing number of AI-assisted diagnostic devices have emerged on the market. Most of these devices directly output video signals from medical imaging equipment to AI edge computing devices, which then perform image analysis and display the results within the independent devices. However, existing technologies have limitations in image processing speed, stability, and versatility.
[0005] Specifically, existing technologies use independent AI edge computing devices to analyze video signals in real time and output the analyzed images to a medical display, or to a separate display screen placed side by side with the display showing the original image. Although this solution can achieve real-time image analysis, it still has the following drawbacks:
[0006] 1. Single display function: Existing monitors can only display original video signals or signals processed by AI edge processing, and cannot switch between multiple display modes.
[0007] 2. Image quality degradation: Images processed by AI edge computing devices may cause the quality of the original images to degrade.
[0008] 3. System stability and delay issues: Due to the influence of the operating system and AI processing, the system may experience instability and delay issues, affecting the surgical process.
[0009] 4. Failure to gain the doctor's primary attention leads to missed or ineffective notifications. Furthermore, the simultaneous overlay and real-time selection switching of multimodal notification data processed by multiple algorithms cannot be achieved.
[0010] Therefore, in order to address the defects in the above-mentioned prior art, it is necessary to develop a new type of intelligent medical display system.
[0011] Summary of the Invention
[0012] In order to overcome the defects of the prior art, the present invention proposes an intelligent medical display system, which can improve image quality and system stability, reduce image processing delay, and improve the real-time performance and accuracy during surgery.
[0013] In order to achieve the above object, the present invention provides the following technical solutions:
[0014] An intelligent medical display system includes a video acquisition module, an AI edge computing module, a marker layer output module, a video overlay module and a display module, characterized in that:
[0015] The video acquisition module is used to receive video input signals from medical imaging equipment and transmit them to the AI edge computing module;
[0016] The AI edge computing module is used to analyze the video input signal to obtain lesion target detection messages, lesion segmentation messages and semantic scene detection messages and push them to the marking layer output module;
[0017] The marking layer output module includes an information processing submodule and an image pushing submodule. The information processing submodule is used to group and mark the messages pushed by the AI edge computing module to divide them into a lesion target detection message group, a lesion segmentation message group, and a semantic scene detection message group; the image pushing submodule is used to mark the background image according to the grouping and marking results of the information processing submodule to form a lesion target detection result video, a lesion segmentation result video, and a semantic scene detection result video, and output them to the video overlay module;
[0018] The video superposition module is used to superimpose the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the image push submodule and the video input signal from the video acquisition module and output the superimposed video to the display module;
[0019] The display module is used to display the superimposed video.
[0020] Preferably, the information processing submodule includes a first buffer and a lesion target detection buffer, a lesion segmentation buffer and a semantic scene detection buffer connected to the first buffer. The first buffer is connected to the AI edge computing module to receive and temporarily store the messages pushed by the AI edge computing module and parse the message headers of the messages. According to the message headers, the lesion target detection messages are temporarily stored in the lesion target detection buffer, the lesion segmentation messages are temporarily stored in the lesion segmentation buffer, and the semantic scene detection messages are temporarily stored in the semantic scene detection buffer, thereby dividing them into lesion target detection message group, lesion segmentation message group and semantic scene detection message group.
[0021] Preferably, the image push submodule includes a texture channel creation unit, a texture channel context creation unit, a background image drawing unit, a graphics drawing unit and a video output unit.
[0022] The texture channel creation unit is used to create a lesion target detection texture channel, a lesion segmentation texture channel and a semantic scene detection texture channel and bind them to three output ports;
[0023] The texture channel context creation unit is used to create a lesion target detection texture channel context, a lesion segmentation texture channel context and a semantic scene detection texture channel context for the three output ports respectively;
[0024] The background image drawing unit is used to draw background images in the lesion target detection texture channel, the lesion segmentation texture channel and the semantic scene detection texture channel respectively;
[0025] The graphics drawing unit draws a lesion target graphic on the background image in the lesion target detection texture channel based on the lesion target detection message group, draws a lesion segmentation graphic on the background image in the lesion segmentation texture channel based on the lesion segmentation message group, and draws a semantic scene graphic on the background image in the semantic scene detection texture channel based on the semantic scene detection message group;
[0026] The video output unit is used to form a lesion target detection result video with the background image and the lesion target graphic drawn thereon, form a lesion segmentation result video with the background image and the lesion segmentation graphic drawn thereon, form a semantic scene detection result video with the background image and the semantic scene detection graphic drawn thereon, and output them to the video overlay module respectively through the three output ports.
[0027] Preferably, the video overlay module includes four input terminals and one output terminal, the four input terminals are respectively used to receive the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the video output unit and the video input signal from the video acquisition module, and the output terminal is used to output the superimposed video to the display module.
[0028] Preferably, the video overlay module performs video overlay through the following steps:
[0029] 1) Establish and initialize a video input signal ring buffer, a lesion detection result video ring buffer, a lesion segmentation result video ring buffer, and a semantic scene detection result video ring buffer, and set the capacity of each of the ring buffers;
[0030] 2) creating a listening thread for the input channels of the four input terminals, so that when each input terminal receives a video input, it can write the frame data into the corresponding ring buffer, and when the ring buffer is full, it can pop out the head frame data while inserting the new frame data;
[0031] 3) creating staggered output, that is, when the video input signal ring buffer reaches the preset storage length for the first time, directly outputting the first frame data without any processing, so that the first frame data in the video input signal ring buffer is staggered by one frame in time with the first frames of data in the other three ring buffers;
[0032] 4) Output the superimposed and synthesized AI image.
[0033] Preferably, the step 4) specifically includes:
[0034] 4.1) Using the time taken by the staggered output to output the first frame of the video input signal, graphically overlay the first frame of data in the lesion detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene detection result video ring buffer to obtain a synthesized AI graphic prompt signal;
[0035] 4.2) Superimposing the synthesized AI graphic prompt signal with the frame data of the video input signal to finally obtain the superimposed and synthesized AI picture.
[0036] Preferably, the AI edge computing module includes a lesion target detection submodule, a lesion segmentation submodule and a semantic scene detection submodule. The lesion target detection submodule is used to perform frame-by-frame recognition on the video input signal to obtain the lesion detection object target, and convert the lesion detection object target into a message to obtain the lesion target detection message; the lesion segmentation submodule is used to perform frame-by-frame recognition on the video input signal to obtain the lesion segmentation object target, and convert the lesion segmentation object target into a message to obtain the lesion segmentation message; the semantic scene detection submodule is used to perform frame-by-frame recognition on the video input signal to obtain the semantic scene detection result, and convert the semantic scene detection result into a message to obtain the semantic scene detection message.
[0037] Preferably, the lesion target detection message includes a message header and a message body, wherein the message header is DET, and the message body is the coordinates of several lesion detection object targets, and multiple coordinates are separated by the "|" symbol; the lesion segmentation message includes a message header and a message body, wherein the message header is SEG, and the message body is a set of point coordinates of several lesion segmentation object targets, and multiple sets of point coordinate sets are separated by the "|" symbol; the semantic scene detection message includes a message header and a message body, wherein the message header is TXT, and the message body is prompt text corresponding to several semantic scenes, and multiple sets of prompt text are separated by the "|" symbol.
[0038] Preferably, the display module has an image input source switching button and the video acquisition module, AI edge computing module and video overlay module are all connected to the display module. The image input source switching button is used to determine whether the display module displays content from the video acquisition module, the AI edge computing module or the video overlay module.
[0039] Compared with the prior art, the intelligent medical display system of the present invention has one or more of the following beneficial technical effects:
[0040] 1. The present invention provides three display modes, which makes it multifunctional and provides more convenience for medical staff.
[0041] 2. This invention uses AI prompt signals to support multimodal diagnosis, graded diagnosis, combined application of imaging omics for disease diagnosis, and clinical technology practice, helping doctors improve the quality of medical operations.
[0042] 3. The present invention uses video overlay technology to realize the superimposed display of the original video input signal and the marked video output by the AI edge computing module, thereby improving image quality and system stability.
[0043] 4. The present invention reduces image processing delay and improves the real-time performance and accuracy during surgery. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] FIG1 is a schematic diagram showing the structure of the intelligent medical display system of the present invention.
[0045] FIG2 is a diagram showing the structure and workflow of the markup layer output module of the present invention.
[0046] FIG3 is a schematic diagram showing the structure of the image push submodule of the present invention.
[0047] FIG4 is a flow chart of video superposition performed by the video superposition module of the present invention. DETAILED DESCRIPTION
[0048] The present invention will be further described below with reference to the accompanying drawings and examples, and the contents of the examples are not intended to limit the scope of protection of the present invention.
[0049] In response to the shortcomings of medical display systems in the prior art, the present invention provides an intelligent medical display system that can improve image quality and system stability, reduce image processing delays, and improve the real-time performance and accuracy during surgery.
[0050] Figure 1 shows a schematic diagram of the structure of the intelligent medical display system of the present invention. As shown in Figure 1, the intelligent medical display system of the present invention includes a video acquisition module, an AI edge computing module, a marker layer output module, a video overlay module and a display module.
[0051] Among them, the video acquisition module is used to receive the video input signal from the medical imaging device and transmit it to the AI edge computing module.
[0052] Specifically, the video acquisition module can receive video input signals from various medical imaging devices, such as endoscopes, ultrasound, or laparoscopes, etc. Moreover, the video acquisition module can transmit the received video input signals to the AI edge computing module through a high-speed PCIE interface.
[0053] The AI edge computing module is used to analyze the video input signal to obtain lesion target detection messages, lesion segmentation messages and semantic scene detection messages and push them to the marking layer output module.
[0054] Specifically, the AI edge computing module includes a lesion target detection submodule, a lesion segmentation submodule and a semantic scene detection submodule.
[0055] The lesion target detection submodule can be constructed based on target detection algorithms such as yolov5, yolov7, and faster-rcnn. The steps for constructing the lesion target detection submodule are as follows:
[0056] 1. Collect medical images containing lesions, annotate the medical images with rectangular frames, and obtain a lesion target detection dataset.
[0057] 2. Build a neural network for lesion target detection.
[0058] In this paper, the YOLOv5 target detection network is used to build a lesion target detection neural network, which includes three groups of cross-stage residual network structures for feature extraction, one group of feature pyramid scaling structures for feature extraction and multi-scale feature map generation, one group of feature pyramid aggregation structures for feature extraction and multi-scale feature map fusion, and one group of target detection head structures for inferring the target frame position.
[0059] 3. Input the lesion target detection data set into the lesion target detection neural network, and obtain a lesion target detection algorithm, i.e., the lesion target detection submodule, through back-propagation training.
[0060] When in use, the lesion target detection submodule is used to perform frame-by-frame recognition on the video input signal to obtain a lesion detection object target, and the lesion detection object target is converted into a message to obtain the lesion target detection message.
[0061] The lesion target detection message includes a message header and a message body, wherein the message header is DET and the message body is the coordinates of several lesion detection targets, and the multiple coordinates are separated by the "|" symbol.
[0062] In the present invention, since the detection target is marked with a rectangular frame, the lesion detection object target is a rectangular frame, and the message body is the coordinates of the upper left vertex and the lower right vertex of the rectangular frame, and the coordinates of the two vertices are separated by the "|" symbol.
[0063] Finally, the lesion target detection submodule uses Socket technology to push the lesion target detection message to the marking layer output module.
[0064] The lesion segmentation submodule can be constructed based on image segmentation algorithms such as UNet, FCN, Deeplab, etc. The steps for constructing the lesion segmentation submodule are as follows:
[0065] 1. Collect medical images containing lesions, annotate the medical images with contour lines, and obtain a lesion segmentation dataset.
[0066] 2. Build a lesion segmentation neural network.
[0067] The present invention adopts the UNet segmentation network to build a lesion segmentation neural network, which includes a set of encoder modules, consisting of four groups of convolution blocks with decreasing scale, for feature extraction, and outputting four groups of feature maps; a set of decoder structures, consisting of four groups of convolution blocks with increasing scale, for feature fusion and reconstruction of segmentation results.
[0068] 3. Input the lesion segmentation data set into the lesion segmentation neural network, and obtain a lesion segmentation algorithm, i.e., the lesion segmentation submodule, through back-propagation training.
[0069] When in use, the lesion segmentation submodule identifies the video input signal frame by frame to obtain a lesion segmentation object target; and converts the lesion segmentation object target into a message to obtain the lesion segmentation message.
[0070] The lesion segmentation message includes a message header and a message body, the message header is SEG, and the message body is a set of point coordinates of several lesion segmentation object targets, and multiple sets of point coordinate sets are divided by the "|" symbol.
[0071] In the present invention, since the lesion segmentation is performed using polygons, the lesion segmentation object target is a polygon, and the message body is the coordinates of each vertex of the polygon, and the coordinates of each vertex are separated by a "|" symbol.
[0072] Finally, the lesion segmentation submodule uses Socket technology to push the lesion segmentation message to the marking layer output module.
[0073] The semantic scene detection submodule can be constructed based on semantic scene detection algorithms such as TSN, SlowFast, and X3D. The construction steps of the semantic scene detection submodule are as follows:
[0074] 1. Collect medical video clips and label them to obtain a semantic scene dataset. In this example, the labeling includes, for example, "Please flush the intestines" for an endoscopic fecal fluid clip, and "Please turn on the stained light for observation; a biopsy can be taken from the inner side of the ulcer embankment" for an ulcer lesion observation clip. It should be noted that the above examples are for illustrative purposes only and include, but are not limited to, the clips and labels described above.
[0075] 2. Build a semantic scene detection neural network.
[0076] The present invention adopts the TSM semantic scene detection network to build a semantic scene detection neural network, which includes a convolutional neural network with 16 groups of residual structures, which is used to receive input video clips and perform feature extraction frame by frame to obtain a frame-by-frame feature map; a group of spatiotemporal fusion modules, which is used to stagger the feature maps obtained frame by frame, that is, the first 1 / 3 of the frame-by-frame feature map is moved backward by one frame, and the last 1 / 3 of the frame-by-frame feature map is moved forward by one frame, so as to produce a staggered effect on the frame-by-frame feature map, and finally fuse the spatiotemporal feature map; a fully connected layer module, which is used to predict semantic category labels on the fused spatiotemporal feature map.
[0077] 3. Input the semantic scene data set into the semantic scene detection neural network, and obtain a semantic scene detection algorithm, i.e., the semantic scene detection submodule, through back-propagation training.
[0078] When in use, the semantic scene detection submodule performs frame-by-frame recognition on the video input signal to obtain a semantic scene detection result; and converts the semantic scene detection result into a message to obtain the semantic scene detection message.
[0079] The semantic scene message includes a message header and a message body, the message header is TXT, and the message body is prompt text corresponding to several semantic scenes, and multiple groups of prompt text are separated by the "|" symbol.
[0080] Finally, the semantic scene detection submodule uses Socket technology to push the semantic scene detection message to the markup layer output module.
[0081] The marking layer output module includes an information processing submodule and an image pushing submodule.
[0082] The information processing submodule is used to group and mark the messages pushed by the AI edge computing module to divide them into a lesion target detection message group, a lesion segmentation message group, and a semantic scene detection message group. The image push submodule is used to mark the background image based on the grouping and marking results of the information processing submodule to form a lesion target detection result video, a lesion segmentation result video, and a semantic scene detection result video, and output them to the video overlay module.
[0083] Specifically, as shown in FIG2 , the information processing submodule includes a first buffer and a lesion target detection buffer, a lesion segmentation buffer, and a semantic scene detection buffer connected to the first buffer.
[0084] Among them, the first buffer is connected to the AI edge computing module to receive and temporarily store the message pushed by the AI edge computing module and parse the message header of the message, and according to the message header, the lesion target detection message is temporarily stored in the lesion target detection buffer, the lesion segmentation message is temporarily stored in the lesion segmentation buffer, and the semantic scene detection message is temporarily stored in the semantic scene detection buffer, thereby dividing it into a lesion target detection message group, a lesion segmentation message group and a semantic scene detection message group.
[0085] That is, if the message header of the message pushed by the AI edge computing module is DET, then the first buffer determines that the message is a lesion target detection message and stores it in the lesion target detection buffer; if the message header of the message pushed by the AI edge computing module is SEG, then the first buffer determines that the message is a lesion segmentation message and stores it in the lesion segmentation buffer; if the message header of the message pushed by the AI edge computing module is TXT, then the first buffer determines that the message is a semantic scene detection message and stores it in the semantic scene detection buffer. As a result, the lesion target detection buffer temporarily stores multiple lesion target detection messages, that is, lesion target detection message groups; the lesion segmentation buffer temporarily stores multiple lesion segmentation messages, that is, lesion segmentation message groups; and the semantic scene detection buffer temporarily stores multiple semantic scene detection messages, that is, semantic scene detection message groups. Therefore, the grouping marking of the pushed messages is achieved.
[0086] In the present invention, the validity period of the lesion target detection buffer, lesion segmentation buffer, and semantic scene detection buffer is set to 16.66ms. It should be noted that 16.66ms is based on the access frequency of the image push submodule. In the present invention, the image push submodule adopts an access frequency of 60 frames per second, so 1000 / 60≈16.66. In actual use, the validity period is not limited to this and can vary according to the access frequency of the image push submodule.
[0087] At the same time, when receiving, the first buffer can receive the message pushed by the AI edge computing module through the Socker Client.
[0088] As shown in FIG3 , in the present invention, the image push submodule includes a texture channel creation unit, a texture channel context creation unit, a background image drawing unit, a graphics drawing unit and a video output unit.
[0089] The texture channel creation unit is used to create a lesion target detection texture channel, a lesion segmentation texture channel and a semantic scene detection texture channel and bind them to three output ports.
[0090] In the present invention, the texture channel creation unit can use GLFW software to create the lesion target detection texture channel, the lesion segmentation texture channel, and the semantic scene detection texture channel. Specifically, the glfwGetMonitorPos function can be used to select the video output port. The glfwGetMonitorPos function supports the serial number of the incoming video transmission. For example, the lesion target detection texture channel is bound to output port No. 1, that is, it is passed to getMonitors()[1]. Similarly, the lesion segmentation texture channel can be bound to output port No. 2, and the semantic scene detection texture channel can be bound to output port No. 3.
[0091] The texture channel context creation unit is used to create a lesion target detection texture channel context, a lesion separation texture channel context and a semantic scene detection texture channel context for the three output ports respectively.
[0092] In the present invention, the glfwMakeContextCurrent function in the GLFW software can be used to bind the three output ports to the OpenGL context respectively, thereby obtaining the lesion target detection texture channel context, the lesion separation texture channel context and the semantic scene detection texture channel context.
[0093] The background image drawing unit is used to draw background images in the lesion target detection texture channel, the lesion segmentation texture channel and the semantic scene detection texture channel respectively.
[0094] In the present invention, since 60 frames are rendered in a loop per second, the glClearColor function is used to set the clear value when the frame buffer of each frame is initialized. This function accepts four parameters, corresponding to the red, green, blue, and alpha (transparency) components. The present invention sets the background to white by default, that is, the input parameters are 1.0, 1.0, 1.0, 1.0. In other words, by using the glClearColor function and passing in the parameters 1.0, 1.0, 1.0, 1.0, a white background image can be drawn in each texture channel. The present invention uses a white background image, so that the background image will not affect the original video input signal when performing video overlay.
[0095] The graphics drawing unit draws the lesion target graphics on the background image in the lesion target detection texture channel based on the lesion target detection message group, draws the lesion segmentation graphics on the background image in the lesion segmentation texture channel based on the lesion segmentation message group, and draws the semantic scene graphics on the background image in the semantic scene detection texture channel based on the semantic scene message group.
[0096] Since the lesion target graphic is a rectangle and the lesion segmentation graphic is a polygon, their drawing principles and methods are the same and are introduced together below.
[0097] First, the lesion target detection texture channel reads the lesion target detection message group from the lesion target detection buffer frame by frame, and the lesion segmentation texture channel reads the lesion segmentation message group from the lesion segmentation buffer frame by frame, obtains the coordinates of each vertex in each message in the corresponding message group, and uses the glBufferData function to convert the vertex coordinates into vertex buffer objects (Vertex Buffer Objects) that can be interpreted by the GPU. Then, the glBindBuffer method is called to bind the vertex buffer object that can be interpreted by the GPU to the corresponding lesion target detection texture channel context or lesion segmentation texture channel context. Finally, the glDrawArrays method is called to complete the graphics drawing. This method will obtain vertex data from the corresponding context and draw the corresponding graphics on the background image of the current texture channel.
[0098] Specifically, the lesion target graphic is a rectangular frame, and the format of vertex coordinates is [(x0min, y0min), (x0max, y0max)], [(x1min, y1min), (x1max, y1max)]...[(xn'min, yn'min), (xn'max, yn'max)]. Wherein, each "[]" represents a graphic object, xmin and ymin represent the coordinates of the upper left corner of each graphic, and xmax and ymax represent the coordinates of the lower right corner of the graphic.
[0099] The lesion segmentation graph is a polygon, and the format of the coordinate vertices is [(x0,y0),(x1,y1),....,(xn,yn)],.....,[(x0,y0),(x1,y1),....,(xn,yn)]. Each "[]" represents a graphic object, which stores the coordinates of each vertex in the graph.
[0100] Since the semantic scene graph is a character graph, here we will separately introduce how to draw the character graph in the semantic scene detection texture channel. For the sake of convenience, the present invention will illustrate with an example. Assuming that the message body of the semantic scene detection message in the semantic scene detection buffer is "Please pay attention to flushing the intestines", the specific steps for drawing the corresponding character graph are:
[0101] First, traverse each character one by one, and according to the character value, match and search the corresponding texture object in the character texture set.
[0102] Next, calculate the character position based on the character length and position. Assuming the text is displayed at the center of a 1920*1080 screen, with a 14-pixel font size and a 7-word example, the vertex position coordinates for the first character are (896, 1060), (910, 1060). Similarly, assuming the character spacing is 2 pixels, the vertex position coordinates for the second character are (912, 1054), (926, 1054). The remaining character calculations are based on this approach.
[0103] Then, input the vertex coordinates of each character one by one, use the glBufferData function to convert the vertex coordinates into GPU-interpretable vertex buffer objects (Vertex Buffer Objects); and call the glBindBuffer method to bind the GPU-interpretable vertex buffer objects to the corresponding semantic scene detection texture channel context.
[0104] Finally, the character graphics are drawn by calling the glDrawArrays method, which retrieves vertex data from the corresponding context and draws the corresponding texture graphics content onto the background image of the current texture channel.
[0105] Among them, it should be known that the character texture set exists as fixed content, that is, it is executed each time the program is initialized. The specific creation steps include:
[0106] First, using the FreeType library, load the font file and call FT_Set_Pixel_Sizes to set the required font pixel size (the font is a conventional technical feature. In addition, regarding the font size, the present invention uses the following formula to calculate the font size: 10.5 points / 72 * 96 dpi = 14 pixels, that is, one point is equal to 1 / 72 inch. The present invention assumes that the font is displayed on a 96 dpi display);
[0107] Secondly, a character texture set is created. Specifically, since the present invention uses OpenGL as the graphics library, each character in the font file is traversed, the glBindTexture function is called to obtain a texture object, the glyph of each character is loaded using FreeType and converted into bitmap information, and the bitmap is copied to the texture object. The bitmap contains the texture content, coordinates, size, baseline position, and character width of each character;
[0108] Finally, a mapping set is established. In the present invention, key-value pairs are used to store data types, where the key is a character and the value is a texture object.
[0109] The video output unit is used to form a lesion target detection result video with the background image and the lesion target graphic drawn thereon, form a lesion segmentation result video with the background image and the lesion segmentation graphic drawn thereon, form a semantic scene detection result video with the background image and the semantic scene graphic drawn thereon, and output them to the video overlay module respectively through the three output ports.
[0110] Specifically, a frame buffer refresh function may be called to continuously output corresponding graphics to form a video output.
[0111] The video overlay module is used to overlay the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the image push submodule and the video input signal from the video acquisition module and output the overlaid video to the display module.
[0112] Specifically, the video overlay module includes four input terminals and one output terminal. The four input terminals are respectively used to receive the lesion target detection result video, lesion segmentation result video, and semantic scene detection result video from the video output unit, as well as the video input signals from the video acquisition module. The video overlay module overlays the received videos. The output terminal is used to output the overlaid video to the display module.
[0113] In the present invention, the video overlay module can be an FPGA video overlay module. When the FPGA video overlay module overlays the received video, the overlay order from the bottom layer to the top layer is: the video input signal from the video acquisition module, that is, the original video input screen → the lesion target detection result video → the lesion segmentation result video → the semantic scene detection result video, and finally the superimposed and synthesized AI screen is obtained.
[0114] As shown in FIG4 , the specific steps of the FPGA video overlay module for performing video overlay are as follows:
[0115] 1. Establish and initialize a video input signal ring buffer, a lesion detection result video ring buffer, a lesion segmentation result ring buffer, and a semantic scene detection result video ring buffer, and set the capacity of each of the ring buffers.
[0116] Each of the ring buffers needs to implement memory chip interface logic: connecting the DDR3 memory chip to the external memory interface of the FPGA video overlay module. The FPGA video overlay module uses the DDR3 controller IP core and the Memory Interface Generator (MIG) tool to generate the memory controller and interface. The MIG tool can generate the memory controller and interface based on the model and parameters of the memory chip. The capacity of the ring buffer, assuming that the received signal data is 1920*1080 pixels, 24-bit RGB graphics (occupying 3 bytes), and 3 frames are stored by default, is:
[0117] 1920*1080*3(bytes)*3(frames)=18662400 bytes.
[0118] 2. Create four groups of channel monitoring threads, that is, monitoring threads for the input channels of the four input terminals. This way, when each input terminal receives video input, it can be monitored and the frame data can be written to the corresponding ring buffer. When the ring buffer is full, the new frame data is inserted and the header frame data is popped out.
[0119] 3. Create staggered output. Staggered output can ensure low-latency output of the original image and obtain stable marking information.
[0120] The creation of dislocation output is specifically as follows:
[0121] First, when the video input signal ring buffer reaches the preset storage length for the first time, the first frame data therein is directly output without any processing.
[0122] In the present invention, the Xilinx HDMI 2.0 interface IP core can be used to implement the HDMI 2.0 output interface. Specifically, the DDR memory controller IP core reads video frame data from the DDR and passes it to a finite state machine (FSM). The FSM controls the order and timing of data reading and data transmission to the HDMI interface based on its internal state. At the same time, the FSM controls the generation of synchronization signals, including horizontal synchronization, vertical synchronization, and pixel enable signals. These synchronization signals determine the specific format and timing of the video frame, ultimately enabling the HDMI interface to output images in the correct format and speed.
[0123] Secondly, after the first frame data is output, subsequent new frame data will continue to enter the video input signal ring buffer. At this time, the first frame data of the video input signal ring buffer will be one frame time-shifted with the first frame data in the other three ring buffers.
[0124] 4. Output the superimposed and synthesized AI image.
[0125] First, using the time of outputting the first frame data of the video input signal occupied by the staggered output, the first frame data of the header in the lesion detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene detection result video ring buffer are graphically superimposed to obtain a synthesized AI graphic prompt signal (it should be noted that when implementing this step, whether the graphic superposition can be completed within 1 frame time, that is, 1 second / 60 frames ≈ 16.66ms, depends on the operating rate of the FPGA chip. For example, a too low-end FPGA chip may not meet this performance. The present invention uses a high-end FPGA chip to meet this performance).
[0126] In the present invention, the synthesized AI graphic prompt signal is obtained by the following steps: (1) taking out the first frame data from the lesion target detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene result video ring buffer, and traversing each pixel in turn according to the size information; (2) judging whether the three pixel values obtained are background colors (according to the above, the background color is assumed to be white 255, 255, 255); (3) if all are background colors, setting the pixel value of the AI graphic prompt signal at that position to the background color; if 1-3 of the three pixel values are non-background color values, setting the pixel value of the AI graphic prompt signal at that position according to the priority order. The priority order is: 1. Pixel value of the corresponding frame data of the semantic scene result video → 2. Pixel value of the corresponding frame data of the lesion segmentation result video → 3. Pixel value of the corresponding frame data of the lesion detection result video. (4) finally obtaining the synthesized AI graphic prompt signal.
[0127] The synthesized AI graphic prompt signal is then superimposed on the frame data of the video input signal. Specifically, the frame data of the video input signal is read from the video input signal ring buffer and compared with the synthesized AI graphic prompt signal pixel by pixel. If the value is the same as the background color (here, based on the above, it is assumed to be white 255, 255, 255), the pixel of the video input signal frame data is output; if the value is different from the background color, the pixel value is set as the synthesized AI graphic prompt signal, ultimately obtaining the superimposed synthesized AI image.
[0128] The display module is used to display the superimposed video.
[0129] Preferably, the internal structure of the display module includes a video input terminal chip, a video loop-out terminal chip, and an image input source switching button. The video acquisition module, AI edge computing module, and video overlay module are all connected to the video input terminal chip of the display module. The image input source switching button is used to determine whether the display module displays content from the video acquisition module, the AI edge computing module, or the video overlay module.
[0130] In this way, the display module can display according to the display mode selected by the user. The user can switch the display mode through the image input source switching button. Display mode 1 directly displays the original video input signal of the medical imaging device; display mode 2 displays the video signal with marking information processed by the AI edge computing module and output by the marking layer output module and superimposed with the original video input signal; display mode 3 is the operating system HDMI signal output by the AI edge computing module. Therefore, the present invention provides three display modes, which makes it have a multi-functional effect and provides more convenience for medical staff.
[0131] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art may, based on the principles of the present invention, modify or replace the technical solutions of the present invention with equivalents without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. An intelligent medical display system, comprising a video acquisition module, an AI edge computing module, a marking layer output module, a video overlay module and a display module, characterized in that: The video acquisition module is used to receive the video input signal from the medical imaging device and transmit it to the AI edge computing module; The AI edge computing module is used to analyze the video input signal to obtain lesion target detection messages, lesion segmentation messages and semantic scene detection messages and push them to the marking layer output module; The marking layer output module includes an information processing submodule and an image pushing submodule. The information processing submodule is used to group and mark the messages pushed by the AI edge computing module to divide them into a lesion target detection message group, a lesion segmentation message group, and a semantic scene detection message group; the image pushing submodule is used to mark the background image according to the grouping marking result of the information processing submodule to form a lesion target detection result video, a lesion segmentation result video, and a semantic scene detection result video, and output them to the video overlay module; The video superposition module is used to superimpose the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the image push submodule and the video input signal from the video acquisition module and output the superimposed video to the display module; The display module is used to display the superimposed video.
2. The intelligent medical display system according to claim 1, characterized in that: The information processing submodule includes a first buffer and a lesion target detection buffer, a lesion segmentation buffer and a semantic scene detection buffer connected to the first buffer. The first buffer is connected to the AI edge computing module to receive and temporarily store the messages pushed by the AI edge computing module and parse the message headers of the messages. According to the message headers, the lesion target detection messages are temporarily stored in the lesion target detection buffer, the lesion segmentation messages are temporarily stored in the lesion segmentation buffer, and the semantic scene detection messages are temporarily stored in the semantic scene detection buffer, thereby being divided into a lesion target detection message group, a lesion segmentation message group and a semantic scene detection message group.
3. The intelligent medical display system according to claim 2, characterized in that: The image push submodule includes a texture channel creation unit, a texture channel context creation unit, a background image drawing unit, a graphics drawing unit and a video output unit. The texture channel creation unit is used to create a lesion target detection texture channel, a lesion segmentation texture channel and a semantic scene detection texture channel and bind them to three output ports; The texture channel context creation unit is used to create a lesion target detection texture channel context, a lesion segmentation texture channel context and a semantic scene detection texture channel context for the three output ports respectively; The background image drawing unit is used to draw background images in the lesion target detection texture channel, the lesion segmentation texture channel and the semantic scene detection texture channel respectively; The graphics drawing unit draws a lesion target graphic on the background image in the lesion target detection texture channel based on the lesion target detection message group, draws a lesion segmentation graphic on the background image in the lesion segmentation texture channel based on the lesion segmentation message group, and draws a semantic scene graphic on the background image in the semantic scene detection texture channel based on the semantic scene detection message group; The video output unit is used to form a lesion target detection result video with the background image and the lesion target graphic drawn thereon, form a lesion segmentation result video with the background image and the lesion segmentation graphic drawn thereon, and form a semantic scene detection result video with the background image and the semantic scene detection graphic drawn thereon, and output them to the video overlay module respectively through the three output ports.
4. The intelligent medical display system according to claim 3, characterized in that: The video overlay module includes four input terminals and one output terminal. The four input terminals are respectively used to receive the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the video output unit and the video input signal from the video acquisition module. The output terminal is used to output the overlaid video to the display module.
5. The intelligent medical display system according to claim 4, characterized in that: The video overlay module performs video overlay through the following steps: 1) Establish and initialize a video input signal ring buffer, a lesion detection result video ring buffer, a lesion segmentation result video ring buffer, and a semantic scene detection result video ring buffer, and set the capacity of each of the ring buffers; 2) creating a listening thread for the input channels of the four input terminals, so that when each input terminal receives a video input, it can write frame data into the corresponding ring buffer respectively, and when the ring buffer is full, the head frame data is popped out while inserting new frame data; 3) creating a staggered output, that is, when the video input signal ring buffer reaches the preset storage length for the first time, the first frame data thereof is directly output without any processing, and at this time, the first frame data in the video input signal ring buffer is staggered by one frame in time with the first frame data in the other three ring buffers; 4) Output the superimposed and synthesized AI image.
6. The intelligent medical display system according to claim 5, characterized in that: The step 4) specifically includes: 4.1) Using the time of the first frame data output of the video input signal occupied by the dislocated output, the first frame data of the head of the lesion detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene detection result video ring buffer are graphically superimposed to obtain a synthesized AI graphic prompt signal; 4.2) Superimposing the synthesized AI graphic prompt signal with the frame data of the video input signal to finally obtain the superimposed synthesized AI picture.
7. The intelligent medical display system according to claim 1, characterized in that: The AI edge computing module includes a lesion target detection submodule, a lesion segmentation submodule and a semantic scene detection submodule. The lesion target detection submodule is used to identify the video input signal frame by frame to obtain the lesion detection object target, and convert the lesion detection object target into a message to obtain the lesion target detection message; the lesion segmentation submodule is used to identify the video input signal frame by frame to obtain the lesion segmentation object target, and convert the lesion segmentation object target into a message to obtain the lesion segmentation message; The semantic scene detection submodule is used to perform frame-by-frame recognition on the video input signal to obtain a semantic scene detection result, and convert the semantic scene detection result into a message to obtain the semantic scene detection message.
8. The intelligent medical display system according to claim 7, characterized in that: The lesion target detection message includes a message header and a message body, wherein the message header is DET, and the message body is the coordinates of several lesion detection object targets, and multiple coordinates are separated by the "|" symbol; the lesion segmentation message includes a message header and a message body, wherein the message header is SEG, and the message body is a set of point coordinates of several lesion segmentation object targets, and multiple sets of point coordinate sets are separated by the "|" symbol; the semantic scene detection message includes a message header and a message body, wherein the message header is TXT, and the message body is prompt text corresponding to several semantic scenes, and multiple sets of prompt text are separated by the "|" symbol.
9. The intelligent medical display system according to claim 1, characterized in that: The display module has an image input source switching button and the video acquisition module, AI edge computing module and video overlay module are all connected to the display module. The image input source switching button is used to determine whether the display module displays content from the video acquisition module, the AI edge computing module or the video overlay module.
Citation Information
Patent Citations
Pathological section real-time interpretation method, device and system
CN112132772A
Video synchronization processing method and system suitable for edge computing
CN113542842A
Ultrasonic imaging system and image processing equipment and method thereof
CN113645416A
Short video operation report generation method and device, computer equipment and storage medium
CN114496173A
Auxiliary decision-making method and device for minimally invasive surgery
CN114724682A
