An intelligent medical display system

Abstract

The present invention relates to an intelligent medical display system, which includes a video acquisition module, an AI edge computing module, a marking layer output module, a video overlay module and a display module, wherein the video acquisition module receives a video input signal and transmits it to the AI ​​edge computing module; the AI ​​edge computing module analyzes the video input signal to obtain a lesion target detection message, a lesion segmentation message and a semantic scene detection message and pushes them to the marking layer output module; the marking layer output module groups and marks the pushed messages and marks them on the background image according to the grouping marking results to form a lesion target detection result video, a lesion segmentation result video and a semantic scene detection result video, and outputs them to the video overlay module; the video overlay module is used for video overlay; and the display module performs display. It can improve image quality and system stability, reduce image processing delay, and improve real-time performance and accuracy during surgery.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical devices and relates to a medical display system, and in particular to an intelligent medical display system. Background Art

[0002] With the rapid development of medical imaging technology and information technology, more and more artificial intelligence-assisted diagnostic devices have appeared on the market. Most of them directly output the video signals of medical imaging equipment to AI edge computing devices, and then perform image analysis and display the results in independent devices. However, existing technologies have certain limitations in image processing speed, stability and versatility.

[0003] Specifically, in the prior art, an independent AI edge computing device is used to analyze the video signal in real time, and the analyzed image is output to a medical display, or output to an independent display screen and placed side by side with the display screen showing the original image. Although this solution can realize real-time image analysis, it still has the following defects:

[0004] 1. Single display function: Existing displays can only display original video signals or signals processed by AI edge, and cannot switch between multiple display modes.

[0005] 2. Image quality degradation: Images processed by AI edge computing devices may cause the original image quality to degrade.

[0006] 3. System stability and delay issues: Due to the influence of the operating system and AI processing, the system may experience instability and delay issues, affecting the surgical process.

[0007] 4. It is impossible to obtain the doctor's first attention, resulting in the omission of prompt information or failure to receive it effectively. At the same time, it is impossible to achieve the synchronous superposition and real-time selection switching of multi-modal prompt data after multi-channel algorithm processing.

[0008] Therefore, in view of the defects existing in the above-mentioned prior art, it is necessary to develop a new type of intelligent medical display system. Summary of the invention

[0009] In order to overcome the defects of the prior art, the present invention proposes an intelligent medical display system, which can improve image quality and system stability, reduce image processing delay, and improve the real-time performance and accuracy during surgery.

[0010] In order to achieve the above object, the present invention provides the following technical solutions:

[0011] An intelligent medical display system, comprising a video acquisition module, an AI edge computing module, a marking layer output module, a video overlay module and a display module, characterized in that:

[0012] The video acquisition module is used to receive the video input signal from the medical imaging device and transmit it to the AI ​​edge computing module;

[0013] The AI ​​edge computing module is used to analyze the video input signal to obtain lesion target detection messages, lesion segmentation messages and semantic scene detection messages and push them to the marking layer output module;

[0014] The marking layer output module includes an information processing submodule and an image pushing submodule. The information processing submodule is used to group and mark the messages pushed by the AI ​​edge computing module to divide them into a lesion target detection message group, a lesion segmentation message group, and a semantic scene detection message group; the image pushing submodule is used to mark the background image according to the grouping marking result of the information processing submodule to form a lesion target detection result video, a lesion segmentation result video, and a semantic scene detection result video, and output them to the video overlay module;

[0015] The video superposition module is used to superimpose the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the image push submodule and the video input signal from the video acquisition module and output the superimposed video to the display module;

[0016] The display module is used to display the superimposed video.

[0017] Preferably, the information processing submodule includes a first buffer and a lesion target detection buffer, a lesion segmentation buffer and a semantic scene detection buffer connected to the first buffer. The first buffer is connected to the AI ​​edge computing module to receive and temporarily store the messages pushed by the AI ​​edge computing module and parse the message headers of the messages. According to the message headers, the lesion target detection messages are temporarily stored in the lesion target detection buffer, the lesion segmentation messages are temporarily stored in the lesion segmentation buffer, and the semantic scene detection messages are temporarily stored in the semantic scene detection buffer, thereby being divided into lesion target detection message groups, lesion segmentation message groups and semantic scene detection message groups.

[0018] Preferably, the image push submodule includes a texture channel creation unit, a texture channel context creation unit, a background image drawing unit, a graphics drawing unit and a video output unit.

[0019] The texture channel creation unit is used to create a lesion target detection texture channel, a lesion segmentation texture channel and a semantic scene detection texture channel and bind them to three output ports;

[0020] The texture channel context creation unit is used to create a lesion target detection texture channel context, a lesion segmentation texture channel context and a semantic scene detection texture channel context for the three output ports respectively;

[0021] The background image drawing unit is used to draw background images in the lesion target detection texture channel, the lesion segmentation texture channel and the semantic scene detection texture channel respectively;

[0022] The graphics drawing unit draws a lesion target graphic on the background image in the lesion target detection texture channel based on the lesion target detection message group, draws a lesion segmentation graphic on the background image in the lesion segmentation texture channel based on the lesion segmentation message group, and draws a semantic scene graphic on the background image in the semantic scene detection texture channel based on the semantic scene detection message group;

[0023] The video output unit is used to form a lesion target detection result video with the background image and the lesion target graphic drawn thereon, form a lesion segmentation result video with the background image and the lesion segmentation graphic drawn thereon, and form a semantic scene detection result video with the background image and the semantic scene detection graphic drawn thereon, and output them to the video overlay module respectively through the three output ports.

[0024] Preferably, the video overlay module includes four input terminals and one output terminal, the four input terminals are respectively used to receive the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the video output unit and the video input signal from the video acquisition module, and the output terminal is used to output the superimposed video to the display module.

[0025] Preferably, the video overlay module performs video overlay through the following steps:

[0026] 1) Establish and initialize a video input signal ring buffer, a lesion detection result video ring buffer, a lesion segmentation result video ring buffer, and a semantic scene detection result video ring buffer, and set the capacity of each of the ring buffers;

[0027] 2) creating a listening thread for the input channels of the four input terminals, so that when each input terminal receives a video input, it can write frame data into the corresponding ring buffer respectively, and when the ring buffer is full, the head frame data is popped out while inserting new frame data;

[0028] 3) Creating staggered output, that is, when the video input signal ring buffer reaches the preset storage length for the first time, directly outputting the first frame data without any processing, so that the first frame data in the video input signal ring buffer is staggered by one frame in time with the first frame data in the other three ring buffers;

[0029] 4) Output the superimposed and synthesized AI image.

[0030] Preferably, the step 4) specifically includes:

[0031] 4.1) Using the time of the first frame data output of the video input signal occupied by the dislocated output, the first frame data of the head of the lesion detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene detection result video ring buffer are graphically superimposed to obtain a synthesized AI graphic prompt signal;

[0032] 4.2) Superimposing the synthesized AI graphic prompt signal with the frame data of the video input signal to finally obtain the superimposed synthesized AI picture.

[0033] Preferably, the AI ​​edge computing module includes a lesion target detection submodule, a lesion segmentation submodule and a semantic scene detection submodule. The lesion target detection submodule is used to perform frame-by-frame recognition on the video input signal to obtain a lesion detection object target, and convert the lesion detection object target into a message to obtain the lesion target detection message; the lesion segmentation submodule is used to perform frame-by-frame recognition on the video input signal to obtain a lesion segmentation object target, and convert the lesion segmentation object target into a message to obtain the lesion segmentation message; the semantic scene detection submodule is used to perform frame-by-frame recognition on the video input signal to obtain a semantic scene detection result, and convert the semantic scene detection result into a message to obtain the semantic scene detection message.

[0034] Preferably, the lesion target detection message includes a message header and a message body, wherein the message header is DET, and the message body is the coordinates of several lesion detection object targets, and the multiple coordinates are separated by the “|” symbol; the lesion segmentation message includes a message header and a message body, wherein the message header is SEG, and the message body is a set of point coordinates of several lesion segmentation object targets, and the multiple groups of point coordinate sets are separated by the “|” symbol; the semantic scene detection message includes a message header and a message body, wherein the message header is TXT, and the message body is prompt text corresponding to several semantic scenes, and the multiple groups of prompt text are separated by the “|” symbol.

[0035] Preferably, the display module has an image input source switching button and the video acquisition module, AI edge computing module and video overlay module are all connected to the display module, and the image input source switching button is used to determine whether the display module displays content from the video acquisition module, the AI ​​edge computing module or the video overlay module.

[0036] Compared with the prior art, the intelligent medical display system of the present invention has one or more of the following beneficial technical effects:

[0037] 1. The present invention provides three display modes, which makes it multifunctional and provides more convenience for medical staff.

[0038] 2. The present invention can support the combined application of imaging genomics for multimodal diagnosis, graded diagnosis, disease diagnosis, and clinical technology practice through AI prompt signals, helping doctors improve the quality of medical operations.

[0039] 3. The present invention uses video overlay technology to realize the overlay display of the original video input signal and the marked video output by the AI ​​edge computing module, thereby improving image quality and system stability.

[0040] 4. The present invention reduces image processing delay and improves real-time performance and accuracy during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the structure of the intelligent medical display system of the present invention.

[0042] Figure 2 It is the composition and workflow diagram of the marking layer output module of the present invention.

[0043] Figure 3 It is a schematic diagram of the composition of the image push submodule of the present invention.

[0044] Figure 4 It is a flow chart of video superposition performed by the video superposition module of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described below in conjunction with the accompanying drawings and embodiments, and the contents of the embodiments are not intended to limit the protection scope of the present invention.

[0046] In view of the deficiencies of the medical display system in the prior art, the present invention provides an intelligent medical display system, which can improve image quality and system stability, reduce image processing delay, and improve the real-time performance and accuracy during surgery.

[0047] Figure 1 FIG. 2 shows a schematic diagram of the structure of the intelligent medical display system of the present invention. Figure 1As shown, the intelligent medical display system of the present invention includes a video acquisition module, an AI edge computing module, a marking layer output module, a video overlay module and a display module.

[0048] Among them, the video acquisition module is used to receive the video input signal from the medical imaging device and transmit it to the AI ​​edge computing module.

[0049] Specifically, the video acquisition module can receive video input signals from various medical imaging devices, such as endoscopes, ultrasound, or laparoscopes, etc. Moreover, the video acquisition module can transmit the received video input signals to the AI ​​edge computing module through a high-speed PCIE interface.

[0050] The AI ​​edge computing module is used to analyze the video input signal to obtain lesion target detection messages, lesion segmentation messages and semantic scene detection messages and push them to the marking layer output module.

[0051] Specifically, the AI ​​edge computing module includes a lesion target detection submodule, a lesion segmentation submodule and a semantic scene detection submodule.

[0052] The lesion target detection submodule can be constructed based on target detection algorithms such as yolov5, yolov7, and faster-rcnn. The steps for constructing the lesion target detection submodule are as follows:

[0053] 1. Collect medical images containing lesions, annotate the medical images with rectangular frames, and obtain a lesion target detection dataset.

[0054] 2. Build a neural network for lesion target detection.

[0055] In the present invention, the yolov5 target detection network is used to build a lesion target detection neural network, which includes 3 groups of cross-stage residual network structures for feature extraction, 1 group of feature pyramid scaling structures for feature extraction and multi-scale feature map generation, 1 group of feature pyramid aggregation structures for feature extraction and multi-scale feature map fusion, and 1 group of target detection head structures for inferring the target frame position.

[0056] 3. Input the lesion target detection data set into the lesion target detection neural network, and obtain a lesion target detection algorithm, i.e., the lesion target detection submodule, through back-propagation training.

[0057] When in use, the lesion target detection submodule is used to identify the video input signal frame by frame to obtain the lesion detection object target, and the lesion detection object target is converted into a message to obtain the lesion target detection message.

[0058] The lesion target detection message includes a message header and a message body, wherein the message header is DET, and the message body is the coordinates of several lesion detection object targets, and the multiple coordinates are divided by the "|" symbol.

[0059] In the present invention, since the detection target is marked with a rectangular frame, the lesion detection object target is a rectangular frame, and the message body is the coordinates of the upper left vertex and the lower right vertex of the rectangular frame, and the coordinates of the two vertices are separated by a "|" symbol.

[0060] Finally, the lesion target detection submodule uses Socket technology to push the lesion target detection message to the marking layer output module.

[0061] The lesion segmentation submodule can be constructed based on image segmentation algorithms such as UNet, FCN, Deeplab, etc. The construction steps of the lesion segmentation submodule are:

[0062] 1. Collect medical images containing lesions, annotate the contours of the medical images, and obtain a lesion segmentation dataset.

[0063] 2. Build a lesion segmentation neural network.

[0064] The present invention adopts the UNet segmentation network to build a lesion segmentation neural network, which includes a set of encoder modules, which are composed of 4 groups of convolution blocks with decreasing scales, used for feature extraction, and output 4 groups of feature maps; a set of decoder structures, which are composed of 4 groups of convolution blocks with increasing scales, used for feature fusion and reconstruction of segmentation results.

[0065] 3. Input the lesion segmentation data set into the lesion segmentation neural network, and obtain a lesion segmentation algorithm, i.e., the lesion segmentation submodule, through back-propagation training.

[0066] When in use, the lesion segmentation submodule identifies the video input signal frame by frame to obtain a lesion segmentation object target; and converts the lesion segmentation object target into a message to obtain the lesion segmentation message.

[0067] The lesion segmentation message includes a message header and a message body, the message header is SEG, and the message body is a set of point coordinates of several lesion segmentation object targets, and multiple sets of point coordinate sets are divided by the "|" symbol.

[0068] In the present invention, since the lesion segmentation is performed by using polygons, the lesion segmentation object target is a polygon, and the message body is the coordinates of each vertex of the polygon, and the coordinates of each vertex are separated by a "|" symbol.

[0069] Finally, the lesion segmentation submodule uses Socket technology to push the lesion segmentation message to the marking layer output module.

[0070] The semantic scene detection submodule can be constructed based on semantic scene detection algorithms such as TSN, SlowFast, and X3D. The construction steps of the semantic scene detection submodule are:

[0071] 1. Collect medical video clips and label them to obtain a semantic scene dataset. In this embodiment, the labeling is such as the endoscope fecal water clip, the corresponding label is "please pay attention to flush the intestines", such as the ulcer lesion observation clip, the corresponding label is "please turn on the dyeing light for observation, and you can take a biopsy on the inside of the ulcer embankment", etc. It should be noted that the above is only an example, including but not limited to the above clips and labels.

[0072] 2. Build a semantic scene detection neural network.

[0073] The present invention adopts the TSM semantic scene detection network to build a semantic scene detection neural network, which includes a convolutional neural network with 16 groups of residual structures, which is used to receive the input video clips and perform feature extraction frame by frame to obtain a frame-by-frame feature map; a group of spatiotemporal fusion modules, which are used to stagger the feature maps obtained frame by frame, that is, the first 1 / 3 of the frame-by-frame feature map is moved backward by one frame, and the last 1 / 3 of the frame-by-frame feature map is moved forward by one frame, so as to produce a staggered effect on the frame-by-frame feature map, and finally fuse the spatiotemporal feature map; a fully connected layer module is used to predict the semantic category label of the fused spatiotemporal feature map.

[0074] 3. Input the semantic scene data set into the semantic scene detection neural network, and obtain a semantic scene detection algorithm, i.e., the semantic scene detection submodule, through back-propagation training.

[0075] When in use, the semantic scene detection submodule identifies the video input signal frame by frame to obtain a semantic scene detection result; and converts the semantic scene detection result into a message to obtain the semantic scene detection message.

[0076] The semantic scene message includes a message header and a message body, the message header is TXT, and the message body is prompt text corresponding to several semantic scenes, and multiple groups of prompt text are divided by the "|" symbol.

[0077] Finally, the semantic scene detection submodule uses Socket technology to push the semantic scene detection message to the markup layer output module.

[0078] The marking layer output module includes an information processing submodule and an image pushing submodule.

[0079] The information processing submodule is used to group and mark the messages pushed by the AI ​​edge computing module to divide them into a lesion target detection message group, a lesion segmentation message group, and a semantic scene detection message group. The image push submodule is used to mark the background image according to the grouping and marking results of the information processing submodule to form a lesion target detection result video, a lesion segmentation result video, and a semantic scene detection result video, and output them to the video overlay module.

[0080] Specifically, Figure 2 As shown, the information processing submodule includes a first buffer and a lesion target detection buffer, a lesion segmentation buffer and a semantic scene detection buffer connected to the first buffer.

[0081] Among them, the first buffer is connected to the AI ​​edge computing module to receive and temporarily store the message pushed by the AI ​​edge computing module and parse the message header of the message, and according to the message header, the lesion target detection message is temporarily stored in the lesion target detection buffer, the lesion segmentation message is temporarily stored in the lesion segmentation buffer, and the semantic scene detection message is temporarily stored in the semantic scene detection buffer, thereby dividing it into a lesion target detection message group, a lesion segmentation message group and a semantic scene detection message group.

[0082] That is, if the message header of the message pushed by the AI ​​edge computing module is DET, then the first buffer determines that the message is a lesion target detection message, and stores it in the lesion target detection buffer; if the message header of the message pushed by the AI ​​edge computing module is SEG, then the first buffer determines that the message is a lesion segmentation message, and stores it in the lesion segmentation buffer; if the message header of the message pushed by the AI ​​edge computing module is TXT, then the first buffer determines that the message is a semantic scene detection message, and stores it in the semantic scene detection buffer. Thus, the lesion target detection buffer temporarily stores multiple lesion target detection messages, that is, lesion target detection message groups; the lesion segmentation buffer temporarily stores multiple lesion segmentation messages, that is, lesion segmentation message groups; the semantic scene detection buffer temporarily stores multiple semantic scene detection messages, that is, semantic scene detection message groups. Therefore, the grouping marking of the pushed messages is realized.

[0083] In the present invention, the validity period of the lesion target detection buffer, the lesion segmentation buffer and the semantic scene detection buffer is set to 16.66ms. It should be noted that 16.66ms is based on the access frequency of the image push submodule. In the present invention, the image push submodule adopts an access frequency of 60 frames / second, so 1000 / 60≈16.66. In actual use, the validity period is not limited to this, and may vary according to the access frequency of the image push submodule.

[0084] At the same time, when receiving, the first buffer can receive the message pushed by the AI ​​edge computing module through the Socker Client.

[0085] like Figure 3 As shown, in the present invention, the image push submodule includes a texture channel creation unit, a texture channel context creation unit, a background image drawing unit, a graphics drawing unit and a video output unit.

[0086] The texture channel creation unit is used to create a lesion target detection texture channel, a lesion segmentation texture channel and a semantic scene detection texture channel and bind them to three output ports.

[0087] In the present invention, the texture channel creation unit can use GLFW software to create the lesion target detection texture channel, the lesion segmentation texture channel and the semantic scene detection texture channel. Specifically, the glfwGetMonitorPos function can be used to select the video output port. The glfwGetMonitorPos function supports the serial number of the incoming video transmission. For example, the lesion target detection texture channel is bound to the output port No. 1, that is, the getMonitors()[1] is passed in. Similarly, the lesion segmentation texture channel can be bound to the output port No. 2, and the semantic scene detection texture channel can be bound to the output port No. 3.

[0088] The texture channel context creation unit is used to create a lesion target detection texture channel context, a lesion separation texture channel context and a semantic scene detection texture channel context for the three output ports respectively.

[0089] In the present invention, the glfwMakeContextCurrent function in the GLFW software can be used to bind the three output ports to the OpenGL context respectively, so as to obtain the lesion target detection texture channel context, the lesion separation texture channel context and the semantic scene detection texture channel context.

[0090] The background image drawing unit is used to draw background images in the lesion target detection texture channel, the lesion segmentation texture channel and the semantic scene detection texture channel respectively.

[0091] In the present invention, since 60 frames are rendered in a loop per second, the glClearColor function is used to set the clear value when the frame buffer of each frame is initialized. This function receives four parameters, which correspond to the red, green, blue and alpha (transparency) components respectively. The present invention sets the background to white by default, that is, the input parameters are 1.0, 1.0, 1.0, 1.0. That is, by using the glClearColor function and passing in the parameters 1.0, 1.0, 1.0, 1.0, a white background image can be drawn in each texture channel. The present invention uses a white background image, so that the background image will not affect the original video input signal when the video is superimposed.

[0092] The graphics drawing unit draws a lesion target graphic on the background image in the lesion target detection texture channel based on the lesion target detection message group, draws a lesion segmentation graphic on the background image in the lesion segmentation texture channel based on the lesion segmentation message group, and draws a semantic scene graphic on the background image in the semantic scene detection texture channel based on the semantic scene message group.

[0093] Since the lesion target graphic is a rectangle and the lesion segmentation graphic is a polygon, their drawing principles and methods are the same and are introduced together below.

[0094] First, the lesion target detection texture channel reads the lesion target detection message group from the lesion target detection buffer frame by frame, and the lesion segmentation texture channel reads the lesion segmentation message group from the lesion segmentation buffer frame by frame, obtains the coordinates of each vertex in each message in the corresponding message group, and uses the glBufferData function to convert the vertex coordinates into vertex buffer objects (Vertex Buffer Objects) that can be interpreted by the GPU. Then, call the glBindBuffer method to bind the GPU-interpretable vertex buffer object to the corresponding lesion target detection texture channel context or lesion segmentation texture channel context. Finally, call the glDrawArrays method to complete the graphics drawing. This method will obtain vertex data from the corresponding context and draw the corresponding graphics on the background image of the current texture channel.

[0095] Specifically, the lesion target graphic is a rectangular frame, and the format of vertex coordinates is [(x0min, y0min), (x0max, y0max)], [(x1min, y1min), (x1max, y1max)]...[(xn'min, yn'min), (xn'max, yn'max)]. Wherein, each "[]" represents a graphic object, xmin and ymin represent the coordinates of the upper left corner of each graphic, and xmax and ymax represent the coordinates of the lower right corner of the graphic.

[0096] The lesion segmentation graph is a polygon, and the format of the coordinate vertices is [(x0,y0),(x1,y1),....,(xn,yn)],.....,[(x0,y0),(x1,y1),....,(xn,yn)]. Each "[]" represents a graphic object, storing the coordinates of each vertex in the graph.

[0097] Since the semantic scene graph is a character graph, here is a separate introduction on how to draw the character graph in the semantic scene detection texture channel. For the sake of ease of introduction, the present invention will illustrate by example, assuming that the message body of the semantic scene detection message in the semantic scene detection buffer is "Please pay attention to flushing the intestines", then the specific steps of drawing the corresponding character graph are:

[0098] First, traverse each character one by one, and according to the character value, match and search for the corresponding texture object in the character texture set.

[0099] Secondly, calculate the character position, based on the character length and position, assuming that the text displayed in the present invention is centered at the bottom of the 1920*1080 screen, the font size is 14 pixels, and the sample text is 7 words, then the vertex position coordinates of the first character are: (896,1060), (910,1060). Similarly, assuming that the character spacing is 2 pixels, then the vertex position coordinates of the second character are: (912,1054), (926,1054), and the remaining character calculation schemes are calculated based on this.

[0100] Then, the vertex coordinates of each character are input one by one, and the glBufferData function is used to convert the vertex coordinates into vertex buffer objects (Vertex Buffer Objects) that can be interpreted by the GPU; and the glBindBuffer method is called to bind the vertex buffer objects that can be interpreted by the GPU to the corresponding semantic scene detection texture channel context.

[0101] Finally, the glDrawArrays method is called to complete the character graphics drawing. This method obtains vertex data from the corresponding context and draws the corresponding texture graphics content onto the background image of the current texture channel.

[0102] Among them, it should be known that the character texture set exists as fixed content, that is, it is executed each time the program is initialized, and its specific creation steps include:

[0103] First, use the FreeType software library to load the font file and call FT_Set_Pixel_Sizes to set the required font pixel size (the font belongs to conventional technical features. In addition, regarding the font size, the present invention uses the following formula to calculate the font size: 10.5 pounds / 72*96dpi=14 pixels, that is, one pound is equal to 1 / 72 inch. The present invention assumes that it is displayed on a 96dpi display);

[0104] Secondly, create a character texture set. Specifically, since the graphics library used in the present invention is OpenGL, traverse each character in the font file, call the glBindTexture function to obtain the texture object, use FreeType to load the glyph of each character and convert it into bitmap information, and copy the bitmap to the texture object. The bitmap contains the texture content, coordinates, size, baseline position and character width of each character;

[0105] Finally, a mapping set is established. In the present invention, a key-value pair data type is used to store data, where the key is a character and the value is a texture object.

[0106] The video output unit is used to form a lesion target detection result video with the background image and the lesion target graphic drawn thereon, form a lesion segmentation result video with the background image and the lesion segmentation graphic drawn thereon, and form a semantic scene detection result video with the background image and the semantic scene graphic drawn thereon, and output them to the video overlay module respectively through the three output ports.

[0107] Specifically, a frame buffer refresh function may be called to continuously output corresponding graphics to form a video output.

[0108] The video overlay module is used to overlay the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the image push submodule and the video input signal from the video acquisition module and output the overlaid video to the display module.

[0109] Specifically, the video superposition module includes four input terminals and one output terminal. The four input terminals are respectively used to receive the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the video output unit and the video input signal from the video acquisition module. The video superposition module superimposes the received video. The output terminal is used to output the superimposed video to the display module.

[0110] In the present invention, the video overlay module can adopt an FPGA video overlay module. When the FPGA video overlay module overlays the received video, the overlay order from the bottom layer to the top layer is: the video input signal from the video acquisition module, that is, the original video input screen → the lesion target detection result video → the lesion segmentation result video → the semantic scene detection result video, and finally the superimposed and synthesized AI screen is obtained.

[0111] like Figure 4 As shown, the specific steps of the FPGA video overlay module for video overlay are:

[0112] 1. Establish and initialize a video input signal ring buffer, a lesion detection result video ring buffer, a lesion segmentation result ring buffer, and a semantic scene detection result video ring buffer, and set the capacity of each of the ring buffers.

[0113] Each of the ring buffers needs to implement the memory chip interface logic: connect the DDR3 memory chip to the external memory interface of the FPGA video overlay module. The DDR3 controller IP core and the Memory Interface Generator (MIG) tool are used in the FPGA video overlay module to generate the memory controller and interface. The MIG tool can generate the memory controller and interface according to the model and parameters of the memory chip. The capacity of the ring buffer, the present invention assumes that the received signal data is 1920*1080 pixels, 24-bit RGB graphics (occupying 3 bytes), and 3 frames are stored by default. The capacity of the ring buffer is:

[0114] 1920*1080*3(bytes)*3(frames)=18662400 bytes.

[0115] 2. Create 4 groups of channel monitoring threads, that is, the monitoring threads of the input channels of the four input terminals. In this way, when each input terminal receives video input, it can be monitored and the frame data can be written into the corresponding ring buffer respectively. When the ring buffer is full, the header frame data is popped out while inserting new frame data.

[0116] 3. Create staggered output. Staggered output can ensure low-latency output of the original image and obtain stable marking information.

[0117] The creation of dislocation output is specifically as follows:

[0118] Firstly, when the video input signal ring buffer reaches the preset storage length for the first time, the first frame data therein is directly output without any processing.

[0119] In the present invention, the IP core of Xilinx HDMI 2.0 interface can be used to implement HDMI 2.0 output interface. Specifically, the DDR memory controller IP core reads the video frame data from DDR and hands it to a finite state machine (FSM). FSM controls the order and time of data reading and data sending to the HDMI interface according to the internal state. At the same time, FSM controls the generation of synchronization signals, including horizontal synchronization, vertical synchronization and pixel enable signals. These synchronization signals determine the specific format and timing of the video frame, and finally enable the HDMI interface to output images in the correct format and speed.

[0120] Secondly, after the first frame data is output, subsequent new frame data will continue to enter the video input signal ring buffer. At this time, the first frame data of the video input signal ring buffer is dislocated by one frame in time with the first frame data in the other three ring buffers.

[0121] 4. Output the AI ​​image after superimposition and synthesis.

[0122] First, using the time of outputting the first frame data of the video input signal occupied by the staggered output, the first frame data of the head in the lesion detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene detection result video ring buffer are graphically superimposed to obtain a synthesized AI graphic prompt signal (it should be noted that when implementing this step, whether the graphic superposition can be completed within 1 frame time, that is, 1 second / 60 frames ≈ 16.66ms, depends on the operating rate of the FPGA chip. For example, a too low-end FPGA chip may not be able to meet this performance. The present invention uses a high-end FPGA chip to meet this performance).

[0123] In the present invention, the synthesized AI graphic prompt signal is obtained by the following steps: (1) taking out the first frame data from the lesion target detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene result video ring buffer respectively, and traversing each pixel in turn according to the size information; (2) judging whether the three pixel values ​​obtained are background colors (according to the above, the background colors are assumed to be white 255, 255, 255); (3) if they are all background colors, the pixel value of the AI ​​graphic prompt signal at this position is also set to the background color. If there are 1-3 non-background color values ​​among the three pixel values, the pixel value of the AI ​​graphic prompt signal at this position is set according to the priority order. The priority order is: 1. The semantic scene result video corresponds to the frame data pixel value → 2. The lesion segmentation result video corresponds to the frame data pixel value → 3. The lesion detection result video corresponds to the frame data pixel value. (4) Finally, the synthesized AI graphic prompt signal is obtained.

[0124] Then, the synthesized AI graphic prompt signal is superimposed with the frame data of the video input signal. Specifically, the frame data of the video input signal is read out from the video input signal ring buffer, and the pixel value is judged pixel by pixel with the synthesized AI graphic prompt signal. If the value is the same as the background color (here, according to the above, it is assumed to be white 255, 255, 255), the pixel of the frame data of the video input signal is output; if the value is different from the background color, the pixel value is set as the synthesized AI graphic prompt signal, and finally the superimposed synthesized AI picture is obtained.

[0125] The display module is used to display the superimposed video.

[0126] Preferably, the internal structure of the display module includes a video input terminal chip, a video loop-out terminal chip and an image input source switching button. The video acquisition module, the AI ​​edge computing module and the video overlay module are all connected to the video input terminal chip of the display module. The image input source switching button is used to determine whether the display module displays content from the video acquisition module, the AI ​​edge computing module or the video overlay module.

[0127] In this way, the display module can display according to the display mode selected by the user. The user can switch the display mode through the image input source switching button. Display mode 1 is to directly display the original video input signal of the medical imaging device; display mode 2 is to display the video signal with marking information processed by the AI ​​edge computing module and output by the marking layer output module and superimposed with the original video input signal; display mode 3 is the operating system HDMI signal output by the AI ​​edge computing module. Therefore, the present invention provides three display modes, which makes it have a multi-functional effect and provides more convenience for medical staff.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solution of the present invention according to the idea of ​​the present invention without departing from the essence and scope of the technical solution of the present invention.

Claims

1. An intelligent medical display system, comprising a video acquisition module, an AI edge computing module, a marking layer output module, a video overlay module and a display module, characterized in that: The video acquisition module is used to receive the video input signal from the medical imaging device and transmit it to the AI ​​edge computing module; The AI ​​edge computing module is used to analyze the video input signal to obtain lesion target detection messages, lesion segmentation messages and semantic scene detection messages and push them to the marking layer output module; The marking layer output module includes an information processing submodule and an image pushing submodule. The information processing submodule is used to group and mark the messages pushed by the AI ​​edge computing module to divide them into a lesion target detection message group, a lesion segmentation message group, and a semantic scene detection message group; the image pushing submodule is used to mark the background image according to the grouping marking result of the information processing submodule to form a lesion target detection result video, a lesion segmentation result video, and a semantic scene detection result video, and output them to the video overlay module; The video superposition module is used to superimpose the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the image push submodule and the video input signal from the video acquisition module and output the superimposed video to the display module; The display module is used to display the superimposed video; The image push submodule includes a texture channel creation unit, a texture channel context creation unit, a background image drawing unit, a graphics drawing unit and a video output unit. The texture channel creation unit is used to create a lesion target detection texture channel, a lesion segmentation texture channel and a semantic scene detection texture channel and bind them to three output ports; The texture channel context creation unit is used to create a lesion target detection texture channel context, a lesion segmentation texture channel context and a semantic scene detection texture channel context for the three output ports respectively; The background image drawing unit is used to draw background images in the lesion target detection texture channel, the lesion segmentation texture channel and the semantic scene detection texture channel respectively; The graphics drawing unit draws a lesion target graphic on the background image in the lesion target detection texture channel based on the lesion target detection message group, draws a lesion segmentation graphic on the background image in the lesion segmentation texture channel based on the lesion segmentation message group, and draws a semantic scene graphic on the background image in the semantic scene detection texture channel based on the semantic scene detection message group; The video output unit is used to form a lesion target detection result video with the background image and the lesion target graphic drawn thereon, form a lesion segmentation result video with the background image and the lesion segmentation graphic drawn thereon, and form a semantic scene detection result video with the background image and the semantic scene detection graphic drawn thereon, and output them to the video overlay module respectively through the three output ports.

2. The intelligent medical display system according to claim 1, characterized in that: The information processing submodule includes a first buffer and a lesion target detection buffer, a lesion segmentation buffer and a semantic scene detection buffer connected to the first buffer. The first buffer is connected to the AI ​​edge computing module to receive and temporarily store the messages pushed by the AI ​​edge computing module and parse the message headers of the messages. According to the message headers, the lesion target detection messages are temporarily stored in the lesion target detection buffer, the lesion segmentation messages are temporarily stored in the lesion segmentation buffer, and the semantic scene detection messages are temporarily stored in the semantic scene detection buffer, thereby being divided into a lesion target detection message group, a lesion segmentation message group and a semantic scene detection message group.

3. The intelligent medical display system according to claim 1, characterized in that: The video overlay module includes four input terminals and one output terminal. The four input terminals are respectively used to receive the lesion target detection result video, lesion segmentation result video and semantic scene detection result video from the video output unit and the video input signal from the video acquisition module. The output terminal is used to output the overlaid video to the display module.

4. The intelligent medical display system according to claim 3, characterized in that: The video overlay module performs video overlay through the following steps: 1) Establish and initialize a video input signal ring buffer, a lesion detection result video ring buffer, a lesion segmentation result video ring buffer, and a semantic scene detection result video ring buffer, and set the capacity of each of the ring buffers; 2) Create a listening thread for the input channels of the four input terminals, so that when each input terminal receives a video input, it can write the frame data into the corresponding ring buffer respectively, and when the ring buffer is full, the head frame data is popped out while inserting the new frame data; 3) Creating staggered output, that is, when the video input signal ring buffer reaches the preset storage length for the first time, the first frame data is directly output without any processing, at which time, the first frame data in the video input signal ring buffer is staggered by one frame in time with the first frame data in the other three ring buffers; 4) Output the superimposed and synthesized AI image.

5. The intelligent medical display system according to claim 4, characterized in that: The step 4) specifically includes: 4.1) Using the time of the first frame data output of the video input signal occupied by the dislocated output, the first frame data of the head of the lesion detection result video ring buffer, the lesion segmentation result video ring buffer, and the semantic scene detection result video ring buffer are graphically superimposed to obtain a synthesized AI graphic prompt signal; 4.2) Superimposing the synthesized AI graphic prompt signal with the frame data of the video input signal to finally obtain the superimposed synthesized AI picture.

6. The intelligent medical display system according to claim 1, characterized in that: The AI ​​edge computing module includes a lesion target detection submodule, a lesion segmentation submodule and a semantic scene detection submodule. The lesion target detection submodule is used to identify the video input signal frame by frame to obtain the lesion detection object target, and convert the lesion detection object target into a message to obtain the lesion target detection message; the lesion segmentation submodule is used to identify the video input signal frame by frame to obtain the lesion segmentation object target, and convert the lesion segmentation object target into a message to obtain the lesion segmentation message; The semantic scene detection submodule is used to perform frame-by-frame recognition on the video input signal to obtain a semantic scene detection result, and convert the semantic scene detection result into a message to obtain the semantic scene detection message.

7. The intelligent medical display system according to claim 6, characterized in that: The lesion target detection message includes a message header and a message body, wherein the message header is DET, and the message body is the coordinates of several lesion detection object targets, and multiple coordinates are separated by the "|" symbol; the lesion segmentation message includes a message header and a message body, wherein the message header is SEG, and the message body is a set of point coordinates of several lesion segmentation object targets, and multiple sets of point coordinate sets are separated by the "|" symbol; the semantic scene detection message includes a message header and a message body, wherein the message header is TXT, and the message body is prompt text corresponding to several semantic scenes, and multiple sets of prompt text are separated by the "|" symbol.

8. The intelligent medical display system according to claim 1, characterized in that: The display module has an image input source switching button and the video acquisition module, AI edge computing module and video overlay module are all connected to the display module. The image input source switching button is used to determine whether the display module displays content from the video acquisition module, the AI ​​edge computing module or the video overlay module.

Images