A real-time intraoperative laparoscopic image analysis device and method

Abstract

The present invention discloses a real-time intraoperative laparoscopic image analysis device and method, which belongs to the field of image analysis management technology, and includes: an artificial intelligence acceleration processor, a data temporary storage module, a laparoscopic video data input module, and a laparoscopic video data output module. The laparoscopic video data input module, the artificial intelligence acceleration processor, and the laparoscopic video data output module are connected in sequence, and the data temporary storage module is connected to the artificial intelligence acceleration processor; the artificial intelligence acceleration processor obtains the laparoscopic image video data transmitted by the laparoscopic video data input module, and extracts the laparoscopic image video data to obtain key frames, stores the key frames in the data temporary storage module, and performs image analysis on the key frames to obtain labeled images, and transmits the labeled images and the laparoscopic image video data to the laparoscopic video data output module for output. Through the above technical solution, the present invention can improve medical efficiency and accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image analysis management, and in particular relates to a real-time intraoperative laparoscopic image analysis device and method. Background Art

[0002] Minimally invasive surgery is becoming increasingly popular as a replacement for traditional open surgery. It can not only target specific parts of the body, such as adrenalectomy, but can also be used for conventional surgeries such as cancer surgery and endovascular surgery. Its application range is also very wide. Among them, minimally invasive surgeries such as laparoscopic cholecystectomy and laparoscopic appendectomy are already common in secondary public hospitals. As one of the most important tools for minimally invasive surgery, the laparoscope is called the "eye of minimally invasive surgery". It mainly enters the human body through a small incision to achieve functions such as image transmission, lighting, and subsequent surgical operations. Although minimally invasive surgery has various advantages, it is still facing the problem of lack of clinical physicians at this stage and cannot be widely popularized at the grassroots level and in towns and villages. The total number of various types of physicians is insufficient, and medical resources are unevenly distributed. Especially in the clinical aspect, it is difficult to operate in small cities, which has always been a common problem in the clinical medical industry. In addition, the training cycle of a clinical physician is long and the level requirements are high, resulting in a huge gap in the number of clinical physicians. Waiting in line for surgery has become the current situation.

[0003] In summary, the existing technology lacks effective collection and organization of intraoperative data, resulting in a waste of clinical data assets. In addition, the current clinical minimally invasive surgery has low medical efficiency and accuracy, resulting in slow surgical procedures for clinicians. As a result, large-scale clinical surgeries cannot be performed in areas with underdeveloped medical environments, leading to difficulties in sinking medical resources. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a real-time intraoperative laparoscopic image analysis device and method, which can effectively collect and use intraoperative clinical data. At the same time, the image data provided can provide relevant auxiliary information for clinical physicians, improve medical efficiency and accuracy, thereby reducing the pressure on clinical physicians and quickly sinking high-quality medical resources to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a real-time intraoperative laparoscopic image analysis device, comprising:

[0006] An artificial intelligence acceleration processor, a data temporary storage module, a laparoscope video data input module, and a laparoscope video data output module, wherein the laparoscope video data input module, the artificial intelligence acceleration processor, and the laparoscope video data output module are connected in sequence, and the data temporary storage module is connected to the artificial intelligence acceleration processor;

[0007] The artificial intelligence acceleration processor obtains the laparoscope image video data transmitted by the laparoscope video data input module, extracts the laparoscope image video data to obtain key frames, stores the key frames in the data temporary storage module, performs image analysis on the key frames to obtain annotated images, and transmits the annotated images and the laparoscope image video data to the laparoscope video data output module for output;

[0008] The artificial intelligence acceleration processor includes a data stream overall scheduling unit and an artificial intelligence video analysis operation unit, and the data stream overall scheduling unit is connected to the artificial intelligence video analysis operation unit;

[0009] The data stream overall scheduling unit transmits the laparoscope image video data to the laparoscope video data output module, extracts the laparoscope image video data to obtain the key frame, transmits the key frame to the data temporary storage module, and transmits the key frame to the artificial intelligence video analysis operation unit;

[0010] The artificial intelligence video analysis operation unit performs image analysis on the key frame to obtain the annotated image, and transmits the annotated image to the data stream overall scheduling unit, which transmits the annotated image to the laparoscope video data output module and the data temporary storage module;

[0011] The artificial intelligence video analysis operation unit performs image analysis on the key frame through the image key frame semantic segmentation AI analysis algorithm to obtain the labeled image with each pixel category labeled; wherein the key frame semantic segmentation AI analysis algorithm adopts a deep learning model;

[0012] After the artificial intelligence video analysis operation unit performs image analysis on the key frame, the following steps are further included:

[0013] The annotated image is blurred, and the blurred image is overlapped and stacked with the corresponding key frame to obtain an auxiliary image, and the auxiliary image is sent to the data stream overall scheduling unit, which transmits the auxiliary image to the laparoscope video data output module for output.

[0014] Optionally, the data stream overall scheduling unit extracts the laparoscope image video data at intervals of a fixed frame rate, and uses thread resources to perform batch scheduling on the key frames and the laparoscope image video data.

[0015] Optionally, an elastic memory is provided in the data temporary storage module, and the elastic memory performs slice-by-slice storage and transmission of the key frames, and the storage and transmission are controlled by the artificial intelligence acceleration processor;

[0016] The elastic storage stores and transmits the annotated images and the key frames, wherein the key frames at least include key frame acquisition time, operation type, department, and image type.

[0017] Optionally, the data stream overall scheduling unit includes a data scheduling unit for data stream scheduling and an extraction unit for extracting the key frame;

[0018] The extraction unit includes a global key frame extraction unit and a local key frame extraction unit, and the key frames include global key frames and local key frames;

[0019] The global key frame extraction unit is used to extract the global key frame based on an interval fixed frame rate or based on the change of the surgical stage, and the global key frame is at least used for postoperative record analysis;

[0020] The local key frame extraction unit is used to monitor the changes of surgical instruments or lesion areas in real time, and extract the local key frame by tracking the region of interest or analyzing the image significance, and the local key frame is at least used for real-time image analysis during surgery;

[0021] The data scheduling unit is at least used to transmit the global key frame to the data temporary storage module, and transmit the local key frame to the artificial intelligence video analysis operation unit.

[0022] Optionally, the extraction unit further includes:

[0023] A key frame duplication detection unit, used for comparing the extracted local key frame with the global key frame to detect whether there is duplication;

[0024] A key frame merging unit, configured to merge the repeated frames when it is detected that the local key frame and the global key frame are repeated;

[0025] The data stream overall scheduling unit also includes a key frame scheduling unit, which is used to obtain the type of data being scheduled by the data scheduling unit and control the priority of the data scheduling unit for scheduling different data according to the data type.

[0026] Optionally, the scheduling strategy of the key frame scheduling unit is:

[0027] ;

[0028] , , , , , ;

[0029] in:

[0030] Indicates keyframes;

[0031] Indicates The importance weight of each key frame;

[0032] Indicates The processing time of each key frame;

[0033] Indicates The timestamp of each key frame;

[0034] Indicates The timestamp of each key frame;

[0035] Indicates the current time;

[0036] Indicates the scheduling period, that is, the time interval for processing frames;

[0037] Indicates the frame interval time, indicating the maximum allowed frame interval;

[0038] Indicates processing key frames The amount of resources required;

[0039] Indicates the maximum computing resources available to the system;

[0040] , , are the priority weight coefficients of the corresponding items respectively;

[0041] Represents the scheduling queue, which contains the selected keyframes and is a collection of all keyframes to be scheduled A subset of .

[0042] Optionally, the The calculation formula is:

[0043] ;

[0044] in:

[0045] Represents the dynamically adjusted weight;

[0046] Represents nonlinear transformations or interactive calculations for each factor, including:

[0047] , is the proportion of the lesion area, which measures the importance of the lesion area in the key frame;

[0048] , is the surgical instrument interaction intensity, reflecting the interaction between the instrument and the lesion area;

[0049] , is the weight of the surgical stage, reflecting the importance of the surgical stage corresponding to the frame; represents the surgical stage scoring function;

[0050] , is the image saliency, which measures the distribution characteristics of visual saliency in the frame;

[0051] , is the motion characteristic of the lesion area, which measures the intensity of dynamic changes in the lesion area;

[0052] ,For redundancy detection, avoid selecting duplicate frames;

[0053] , is the dynamic characteristics of the interaction between the lesion area and the instrument;

[0054] in:

[0055] ;

[0056] The timestamp of the current key frame;

[0057] is the estimated total duration of the operation;

[0058] Dynamically adjust the weight;

[0059] is the basic weight, which indicates the initial weight distribution of each factor;

[0060] ;

[0061] Keyframe The area of ​​the lesion in

[0062] Keyframe The total area in

[0063] Keyframe the perimeter of the lesion area, indicating the boundary complexity;

[0064] represents the lesion area proportion function;

[0065] ;

[0066] Keyframe The overlapping area between the instrument and the lesion area;

[0067] Keyframe Exercise intensity of medium equipment;

[0068] is the maximum motion intensity of the equipment in all frames;

[0069] represents the surgical instrument interaction strength function;

[0070] ;

[0071] Keyframe Medium Pixels The strength of significance;

[0072] represents the image saliency function;

[0073] is the variance of the significant pixel intensity;

[0074] ;

[0075] is the pixel point in the lesion area movement displacement;

[0076] Represents the motion characteristic function;

[0077] ;

[0078] Keyframe With keyframes similarity;

[0079] represents the similarity function;

[0080] ;

[0081] Represents the dynamic characteristic function of the interaction between the lesion area and the instrument.

[0082] On the other hand, the present application also proposes a real-time intraoperative laparoscopic image analysis method, which is applied to the above-mentioned device, and the steps of the method include:

[0083] Acquire the laparoscope image video data transmitted by the laparoscope video data input module;

[0084] Extracting the laparoscope image video data to obtain key frames;

[0085] The key frame is stored in the data temporary storage module, and image analysis is performed on the key frame to obtain a marked image;

[0086] The annotated image and the laparoscope image video data are transmitted to the laparoscope video data output module for output.

[0087] Compared with the prior art, the present invention has the following advantages and technical effects:

[0088] In response to the problems raised by the above-mentioned background technology, the present application proposes a real-time intraoperative laparoscopic image analysis device to improve the medical efficiency and accuracy of clinical minimally invasive surgery, solve the current slow surgical process of clinical doctors, and solve the problem of the inability to conduct large-scale clinical surgery and the difficulty in sinking medical resources in areas with underdeveloped medical environments. Clinicians only need to be given certain basic training on the use of image analysis devices. With the powerful computing speed and data processing capabilities of the artificial intelligence algorithm embedded in the equipment, existing laparoscopic images can be analyzed quickly and accurately, providing physicians with intraoperative medical assistance, reducing the clinical surgical pressure of physicians, improving the accuracy of physicians' intraoperative operations, and realizing the sinking of clinical high-quality resources. In view of this, the technical solution of the present invention is specially proposed.

[0089] The present invention provides a real-time intraoperative laparoscopic image analysis device to solve the current problem of clinical doctors having no intraoperative assistance and difficulty in using intraoperative data. Through the artificial intelligence acceleration processor, key frame selection, artificial intelligence analysis and processing are performed on the laparoscopic image video data, and then the original video and analyzed video can be exported through the laparoscopic video output module, which plays a role in assisting intraoperative surgery and improving the surgical work efficiency and decision-making accuracy of surgeons. At the same time, the general artificial intelligence acceleration processor algorithm can also be updated by wireless transmission to improve the accuracy of the laparoscopic image analysis device. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 The present invention is a schematic structural diagram of a real-time intraoperative laparoscopic image analysis device according to an embodiment of the present invention.

[0091] Figure 2 Schematic diagram of the structure of an artificial intelligence acceleration processor according to an embodiment of the present invention.

[0092] Figure 3 The present invention is a flowchart of a real-time intraoperative laparoscopic image analysis method according to an embodiment of the present invention.

[0093] In the figure: 100, artificial intelligence acceleration processor; 200, data temporary storage module; 300, laparoscope video data input module; 400, laparoscope video data output module; 500, laparoscope image video input port; 600, laparoscope image video output port; 700, wireless transmission and reception module; 110, data flow overall scheduling unit; 120, artificial intelligence video analysis operation unit; 111, data scheduling unit; 112, extraction unit. DETAILED DESCRIPTION

[0094] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0095] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0096] Reference Figure 1 and Figure 2 The present invention provides a real-time intraoperative laparoscopic image analysis device, comprising:

[0097] The artificial intelligence acceleration processor 100, the data temporary storage module 200, the laparoscope video data input module 300, and the laparoscope video data output module 400 are connected in sequence, and the data temporary storage module 200 is connected to the artificial intelligence acceleration processor 100;

[0098] The artificial intelligence acceleration processor 100 obtains the laparoscope image video data transmitted by the laparoscope video data input module 300, extracts the laparoscope image video data to obtain key frames, stores the key frames in the data temporary storage module 200, and performs image analysis on the key frames to obtain labeled images, and transmits the labeled images and the laparoscope image video data to the laparoscope video data output module 400 for output.

[0099] Among them, the artificial intelligence acceleration processor 100 may refer to a processor specifically used to accelerate the operation of artificial intelligence algorithms, for example, it can be implemented using hardware such as GPU, FPGA or ASIC.

[0100] The data temporary storage module 200 refers to a storage device for temporarily storing data, and can be implemented by a cache, RAM, or SSD.

[0101] The laparoscope video data input module 300 refers to a device for receiving and transmitting laparoscope video data, and can be implemented by a video capture card or a dedicated interface.

[0102] The laparoscope video data output module 400 refers to a device for outputting processed video data, and can be implemented by a display, a projector or a dedicated display device.

[0103] In addition, in some specific embodiments, a laparoscopic image video input port 500 is provided on the laparoscopic video data input module 300 for receiving intraoperative laparoscopic image video data acquired from the laparoscopic equipment system. The laparoscopic image video input port 500 serves as a sensor device for receiving relevant image sensing signals. The laparoscopic video data input module 300 converts and processes the image sensing signals through relevant signals to generate laparoscopic image video data.

[0104] The laparoscope video data output module 400 is provided with a laparoscope image video output port 600, and the laparoscope image video output port 600 outputs the original video and the analyzed video through two ports respectively.

[0105] The innovation of this application is to use the artificial intelligence acceleration processor 100 to realize real-time processing and analysis of laparoscopic video data. By extracting key frames and performing image analysis, annotated images are generated to provide real-time visual auxiliary information for the surgical team. This method not only improves surgical efficiency and accuracy, but also saves valuable data for postoperative research and teaching.

[0106] Here’s how this application works:

[0107] First, the laparoscope video data input module 300 receives the laparoscope image video data transmitted by the laparoscope and transmits it to the artificial intelligence acceleration processor 100. The artificial intelligence acceleration processor 100 can adopt a parallel computing architecture and can efficiently process a large amount of laparoscope image video data. The artificial intelligence acceleration processor 100 extracts key frames from the received laparoscope image video data. The extraction of key frames is based on a preset algorithm, such as scene change detection or fixed time interval sampling.

[0108] The extracted key frames can be stored in the data temporary storage module 200. The data temporary storage module 200 uses high-speed storage technology to ensure fast reading and writing of data. The artificial intelligence acceleration processor 100 can directly perform in-depth analysis on the key frames or perform in-depth learning analysis on the stored key frames to identify key information such as important structures and potential risk areas in the operation. Based on the analysis results, annotated images are generated.

[0109] Finally, the annotated image and the original laparoscope image video data are transmitted to the laparoscope video data output module 400. The output module displays these data in an appropriate format on the display device in the operating room to provide real-time visual auxiliary information for the surgical team. During the whole process, the modules can communicate with each other through a high-speed data bus to ensure the real-time and reliability of data transmission.

[0110] Specifically, the annotated image may include adding annotations of lesion areas, surgical instruments, operation suggestion annotations, etc. to the original image.

[0111] Specifically, the artificial intelligence acceleration processor 100 includes a data stream overall scheduling unit 110 and an artificial intelligence video analysis operation unit 120, and the data stream overall scheduling unit 110 is connected to the artificial intelligence video analysis operation unit 120;

[0112] The data flow overall scheduling unit 110 transmits the laparoscope image video data to the laparoscope video data output module 400, extracts the laparoscope image video data to obtain key frames, transmits the key frames to the data temporary storage module 200, and transmits the key frames to the artificial intelligence video analysis operation unit 120;

[0113] The artificial intelligence video analysis operation unit 120 performs image analysis on the key frames to obtain labeled images, and transmits the labeled images to the data stream overall scheduling unit 110, which transmits the labeled images to the laparoscope video data output module 400 and the data temporary storage module 200.

[0114] Among them, in some specific embodiments, the data flow overall scheduling unit 110 can send the key frames directly to the artificial intelligence video analysis operation unit 120, or can send the key frames directly to the data temporary storage module 200, and then the artificial intelligence video analysis operation unit 120 obtains the key frames from the data temporary storage module 200. The data flow overall scheduling unit 110 can also send the key frames to the artificial intelligence video analysis operation unit 120 and the data temporary storage module 200 at the same time.

[0115] Among them, the artificial intelligence video analysis operation unit 120 can send the labeled image to the data flow overall scheduling unit 110, and can also send it to the data temporary storage module 200 for storage.

[0116] By subdividing the artificial intelligence acceleration processor 100 into a data flow overall scheduling unit 110 and an artificial intelligence video analysis operation unit 120, the division of labor and cooperation in data processing and analysis is achieved. The data flow overall scheduling unit 110 is responsible for the transmission, extraction and distribution of data, ensuring the efficient management of data flow. It transmits the laparoscope image video data to the output module, and at the same time extracts the key frames and distributes them to the data temporary storage module 200 and the artificial intelligence video analysis operation unit 120, realizing parallel processing of data. The artificial intelligence video analysis operation unit 120 focuses on intelligent analysis of key frames and generates annotated images. This division of labor enables the system to process large amounts of real-time video data more efficiently while ensuring the quality and speed of analysis.

[0117] In the technical solution of the present application, the data stream overall scheduling unit 110 can implement efficient data management in a variety of ways. For example, multi-threading technology can be used to process different data streams in parallel, or priority queues can be used to manage data of different importance. The data stream overall scheduling unit 110 can dynamically adjust the data processing strategy according to the system load and data characteristics. For example, when the system load is high, key frames are processed first and non-critical data are temporarily deferred.

[0118] The artificial intelligence video analysis operation unit 120 can use deep learning models, such as convolutional neural networks (CNN) or recurrent neural networks (RNN) to analyze key frames. These models can be optimized to adapt to specific types of surgical scenarios to improve the accuracy and efficiency of analysis. In addition, model quantization and pruning techniques can be used to reduce computational complexity, so that the analysis process can be performed in real time with limited hardware resources.

[0119] The connection between the data flow overall scheduling unit 110 and the artificial intelligence video analysis operation unit 120 can be realized through a high-speed internal bus or shared memory to ensure low latency and high bandwidth of data transmission. This connection method can minimize data transmission overhead and improve the response speed of the overall system.

[0120] In practical applications, the data stream overall scheduling unit 110 can dynamically adjust the extraction frequency of key frames according to different stages of the operation. For example, in the critical operation stage of the operation, the extraction frequency of key frames can be increased to ensure that more important information is captured. In the preparation or closing stage of the operation, the extraction frequency can be appropriately reduced to save system resources.

[0121] The artificial intelligence video analysis operation unit 120 can use a multi-model fusion method when processing key frames. For example, the target detection model can be used to identify surgical instruments, and the semantic segmentation model can be used to identify the lesion area, and then these results can be combined to generate the final annotated image. This method can improve the comprehensiveness and accuracy of the analysis.

[0122] When transmitting the annotated image to the laparoscope video data output module 400, the data stream overall scheduling unit 110 can dynamically adjust the image quality and frame rate of the transmission according to the network bandwidth and the performance of the display device. For example, when the network bandwidth is limited, the image quality of non-critical areas can be reduced while maintaining high definition in critical areas to ensure that the transmission of important information is not affected.

[0123] Specifically, the data stream overall scheduling unit 110 may extract the laparoscope image video data at intervals of a fixed frame rate, and utilize thread resources to perform batch scheduling of key frames and laparoscope image video data.

[0124] In the data flow overall scheduling unit 110, key frames of the laparoscope image video data are extracted at intervals of a fixed frame rate. The data flow overall scheduling unit 110 can transmit the key frames to the data temporary storage module 200 for storage, output the laparoscope image video data in the laparoscope video data input module 300, that is, input the original data to the laparoscope video data output module 400, and output the analysis result generated by the artificial intelligence video analysis operation unit 120, that is, the annotated image to the laparoscope video data output module 400 and other functions.

[0125] Batch scheduling can be performed by efficiently utilizing thread resources, such as using a time wheel algorithm for efficient scheduling, transferring key frames to the data storage module 200 for storage, and using the data storage module 200 to store, manage, and transmit all key frames, thereby efficiently managing various delays and periodic tasks of key frames, and quickly transferring the data in the data storage module 200 to the artificial intelligence video analysis operation unit 120 in the artificial intelligence acceleration processor 100 for analysis, thereby improving the efficiency of the artificial intelligence acceleration processor 100.

[0126] Specifically, the data stream overall scheduling unit 110 may output the laparoscope image video data, i.e., the input original data, to the laparoscope video data output module 400. In some specific embodiments, the data stream overall scheduling unit 110 may directly transmit the laparoscope image video data to the laparoscope video data output module 400;

[0127] In another specific embodiment, when the artificial intelligence video analysis operation unit 120 receives a key frame from the data temporary storage module 200, the data flow overall scheduling unit 110 will first transmit the laparoscopic image video data to the laparoscopic video data output module 400, and then the artificial intelligence video analysis operation unit 120 will perform image data analysis, so that the laparoscopic video data output module 400 obtains the original laparoscopic image video data. Specifically, when the key frame is transmitted and stored in the data temporary storage module 200, the laparoscopic image video data with a preset duration corresponding to the key frame can also be stored.

[0128] Specifically, the data stream overall scheduling unit 110 also includes outputting the annotated image to the laparoscope video data output module 400. After the artificial intelligence video analysis operation unit 120 analyzes the key frame, the obtained annotated image is transmitted to the data stream overall scheduling unit 110, and the data stream overall scheduling unit 110 transmits the annotated image to the laparoscope video data output module 400.

[0129] Further, in some preferred embodiments, the artificial intelligence video analysis operation unit 120 performs image analysis on the key frames through the image key frame semantic segmentation AI analysis algorithm to obtain a labeled image with each pixel category labeled;

[0130] After the artificial intelligence video analysis unit 120 performs image analysis on the key frames, the following steps are also performed:

[0131] The annotated image is blurred, and the blurred image is overlapped and stacked with the corresponding key frame to obtain an auxiliary image, which is sent to the data stream overall scheduling unit 110, and the data stream overall scheduling unit 110 transmits the auxiliary image to the laparoscope video data output module 400 for output.

[0132] Specifically, the artificial intelligence analysis operation unit has an embedded key frame semantic segmentation AI analysis algorithm, which can perform real-time, low-latency, and high-precision image semantic segmentation and annotation of key frames, identify the pixel categories in the intraoperative image pictures, and generate annotated images. The annotated images are output to the laparoscopic image video output port 600 through the data stream overall scheduling unit 110, and then transmitted back to the operating room screen. Through the relevant annotated images, professional auxiliary information is provided to the doctor during the operation, thereby assisting the doctor's operation during the operation and improving the accuracy of the doctor's operation. In addition, clinical doctors in areas with low professional time and poor medical conditions can obtain the relevant experience of doctors with high professional time through short-term real-time intraoperative laparoscopic image analysis device auxiliary technical training, so that they have the ability to quickly carry out minimally invasive laparoscopic surgery and realize the rapid sinking of high-quality medical clinical resources.

[0133] For the key frame semantic segmentation AI analysis algorithm, the ResU-Net++ network can be used for implementation. Exemplarily, the ResU-Net++ network includes an encoder structure and a decoder structure. The output of the encoder structure is connected to the input port of the decoder structure through an ASPP module. The output end of the decoder structure is sequentially provided with an ASPP module, a 2D convolutional layer and a Sigmoid function. The Sigmoid function is used to identify the category of each pixel in the image.

[0134] The encoder structure includes several encoders, wherein the encoder includes a compression and excitation module, a residual module and an attention module connected in sequence, wherein the residual module and the attention module are residually connected, and the residual module includes two convolution blocks of residual connection, wherein the convolution block includes a Batch Norm layer, a Relu layer and a convolution layer of residual connection.

[0135] The decoder structure includes several decoders, wherein the decoder includes an attention module, an upsampling module, a feature link module, a residual module and an addition module connected in sequence, wherein the output of the feature link module is linked to the input of the addition module. Exemplarily, the number of encoders and decoders is 4 and 3 respectively.

[0136] In the compression and excitation module, the input features of the compression and excitation module are compressed into feature vectors through global pooling; corresponding weights are generated for different channels of the input features of the compression and excitation module according to the feature vectors, and different channels are calculated according to the weights to generate the output features of the compression and excitation module. This module amplifies important features by attributing increased weights to information channels, while reducing noise and insignificant elements by reducing the corresponding channels. This enhancement and reduction of discriminability improves the operating efficiency of the network.

[0137] In the ASPP module, by establishing different expansion rates, multi-scale sampling is performed on the ASPP module to generate multi-scale gravity feature data.

[0138] In the attention module, the output of the attention module is the result of element-by-element multiplication of the feature mapping data and the attention coefficient, wherein the feature mapping data is the input data of the attention module, and the attention coefficient is generated by the weighted sum of the feature mapping data and the input data of the attention module through the ReLU function, convolution processing, Simgoid function and resampling.

[0139] Furthermore, the artificial intelligence video analysis operation unit 120 can have the function of fusing the annotated image with the original image of the key frame. First, the annotated image is blurred to improve the transparency of the segmented image, and then it is overlapped and stacked with the original image of the key frame to generate an auxiliary image. The auxiliary image replaces the annotated image through the data flow overall scheduling unit 110 and is output to the laparoscope image video output port 600, and then transmitted back to the operating room screen, which serves as a better way to present auxiliary images during surgery.

[0140] In some specific implementations, the auxiliary image may be transmitted to the data temporary storage module 200 for storage.

[0141] In some specific embodiments, the artificial intelligence video analysis operation unit 120 uses an intra-frame prediction mode that supports 20-33 directions to perform motion compensation processing and vector prediction, and encodes the video in H.265 or H.264 mode, which will reduce the video volume by about 30-45%, greatly reducing the burden of medical image transmission and storage;

[0142] In some specific embodiments, the artificial intelligence video analysis operation unit 120 is burned with the operation program of the real-time intraoperative laparoscopic image analysis device, which can receive the relevant updated algorithm through the wireless transmission and receiving module 700 to update and iterate the algorithm in time, further improve the efficiency of artificial intelligence algorithm analysis, and reduce the delay of artificial intelligence laparoscopic image analysis;

[0143] Further, in some preferred embodiments, an elastic memory is provided in the data temporary storage module 200, and the elastic memory performs slice-by-slice storage and transmission of key frames, and the storage and transmission are controlled by the artificial intelligence acceleration processor 100;

[0144] The elastic storage stores and transmits the annotated images and key frames, and the key frames at least include the key frame acquisition time, the type of surgery, the department and the image type.

[0145] The data temporary storage module 200 is provided with a built-in elastic memory for storing and transmitting the key frames of the intraoperative images in a slicing manner. It is controlled by the general artificial intelligence acceleration processor 100, and can save the key frames and annotated images during the operation at a set time interval, and store the annotated images and key frame image information in the built-in elastic memory. The image information of all key frames can include basic information such as the acquisition time of the key frames, the type of surgery, the department in which they are located, and the type of image.

[0146] Specifically, under the control of the artificial intelligence acceleration processor 100, according to the real-time network status, when the network signal is good, the data stored in the data temporary storage module 200 is transmitted to the medical data system through the wireless transmission and reception module 700, so as to realize the dynamic and stable transmission of medical intraoperative imaging data; when the network transmission condition is not good, the data is temporarily stored through the data temporary storage module 200, and then transmitted at night or when the network condition is good. This wireless peak-shifting transmission mechanism can effectively reduce the traffic burden of large-scale image transmission on the medical network during peak working hours.

[0147] The wireless transmission and receiving module 700 is controlled by the artificial intelligence acceleration processor 100 and has both input and output functions. The wireless transmission and receiving module 700 includes a transmission module and a receiving module. The transmission module is mainly responsible for wirelessly transmitting the key frames and other data after relevant analysis to the in-hospital medical data storage network; the receiving module mainly receives the updated artificial intelligence algorithm from the cloud server, and updates the algorithm in the device through the updated artificial intelligence algorithm, so that the device has a higher accuracy and faster analysis rate for the laparoscope artificial intelligence semantic analysis.

[0148] Specifically, the transmission module can adopt the RTN network transmission structure, which can improve transmission efficiency, improve robustness and error recovery capabilities, reduce real-time delays, reduce channel acquisition time and random access delays, and reduce complexity, effectively alleviating the burden of medical image transmission and storage, which is conducive to postoperative physicians obtaining the patient's intraoperative relevant imaging data as soon as possible, and improving the accuracy of postoperative doctors' decisions on the patient's subsequent recovery.

[0149] Specifically, the receiving module receives the updated algorithm from the cloud server, updates the intraoperative image key frame semantic segmentation AI analysis algorithm embedded in the general artificial intelligence acceleration processor 100, improves the algorithm operation efficiency, algorithm analysis accuracy, changes the operation structure framework and reduces hardware analysis energy consumption.

[0150] The laparoscopic video data output module 400 is provided with a laparoscopic image video output port 600. The laparoscopic image video output port 600 is composed of two ports, one of which is a first port, which can losslessly output the unprocessed original video data, namely, the laparoscopic image video data, and the other port is a second port, which outputs the annotated image analyzed by the above-mentioned artificial intelligence acceleration processor 100. The total delay of the laparoscopic image video data and the annotated image output is controlled within 10-80ms, which meets the low-latency intraoperative auxiliary requirements of real-time analysis of intraoperative images. The laparoscopic video data output module 400 is used to convert different data into a data format adapted to the visualization device corresponding to the port according to the output data of the port to be output, and dispatch the corresponding data to the corresponding port, and output the data to the visualization device through the port for display.

[0151] Under the control of the artificial intelligence acceleration processor 100, the laparoscopic video data output module 400 outputs the laparoscopic image video data and the annotated image through the first port and the second port of the laparoscopic image video output port 600, respectively. During the operation, the physician can see the original laparoscopic image video and the laparoscopic image analyzed video on two screens in the operating room, respectively, thereby improving the accuracy of the physician's decision-making during the operation.

[0152] Specifically, in some embodiments, the solution of the present application is implemented as follows:

[0153] The output interface of the existing laparoscope system is inserted into the laparoscope image video input port 500 through the corresponding data line to realize lossless laparoscope image data input, which is connected to the laparoscope video data input module 300 to realize lossless reception of laparoscope image video data;

[0154] The artificial intelligence acceleration processor 100 intercepts key frames at intervals. The artificial intelligence acceleration processor 100 intercepts the lossless receiving laparoscope image video data received by the laparoscope video data input module 300 according to the set time (for example, it can be 5 seconds, or it can be adjusted adaptively. It can intercept at different time points according to different stages of the operation. For example, the data volume of the stage before the operation may not be very important, so the interception time interval can be longer. The data volume of the key stage of the operation is very important, so the interception time interval can be shorter. In this way, the data storage space can be better allocated and the rational utilization rate of the data storage space can be improved.) The laparoscope video data input module 300 intercepts the lossless receiving laparoscope image video data, and performs fixed frame rate interception processing on the laparoscope image video data input to obtain the key frame of the lossless receiving laparoscope image video data;

[0155] The key frames are transmitted to the data temporary storage module 200 for storage, and the overall process control is performed through the artificial intelligence acceleration processor 100. The intraoperative image key frames are stored in a sharded manner through the built-in elastic memory of the data temporary storage module 200, and when the network signal is good, the intraoperative image key frames are transmitted to the medical data system through the wireless transmission and receiving module 700. The use of the staggered transmission mechanism can effectively reduce the traffic burden of image transmission on the medical network during the daytime working hours;

[0156] When the artificial intelligence acceleration processor 100 receives the key frame of the lossless laparoscope data from the data temporary storage module 200, the artificial intelligence acceleration processor 100 will first transmit the laparoscope image video data to the laparoscope video data output module 400, and then extract the key frame to perform image data analysis, so that the laparoscope image video output port 600 of the laparoscope video data output module 400 can output the laparoscope image video data;

[0157] The artificial intelligence video analysis operation unit 120 fuses the annotated image with the original image of the key frame. First, after analyzing the data through a high-performance multi-task cloud-distributed large-scale pre-trained AI model, the AI ​​semantic segmentation algorithm can be used to classify each pixel in the intraoperative image to obtain annotated images, thereby completing multiple types of clinical intraoperative data analysis and application tasks. After low-latency, high-precision semantic segmentation processing, the annotated image is output to the laparoscope video output port 600 through the laparoscope video output module, and then transmitted back to the operating room screen. Secondly, the image obtained by fusing the annotated image with the original image of the key frame can combine the advantages of the two, so that the analyzed image has improved transparency compared to the original image, and the image obtained after analysis by the general artificial intelligence acceleration processor 100 has improved visualization and is more convenient for doctors to analyze.

[0158] Data is transmitted through the wireless transmission and receiving module 700, which adopts the H.265 high-performance video transmission standard to reduce the transmission bit rate, fundamentally reduce the image size, and increase the transmission rate. It can achieve 4K high-definition digital image transmission in a network environment below 2Mbps, and use the RTN network architecture for transmission, which can control the network delay within 300ms, meeting the low-latency transmission requirements of key frames for real-time analysis of intraoperative images.

[0159] As an optional embodiment, the wireless transmission and reception module 700 updates the semantic segmentation algorithm in the general artificial intelligence acceleration processor 100. The updated algorithm is received from the cloud server, and the semantic segmentation AI analysis algorithm of the key frame of the intraoperative image embedded in the general artificial intelligence acceleration processor 100 is updated to improve the algorithm operation efficiency, algorithm analysis accuracy, and change the operation structure framework to reduce the hardware analysis energy consumption.

[0160] As an optional embodiment, the key frame and other data are wirelessly transmitted to the hospital's internal information system. The intraoperative images transmitted to the medical system for long-term backup can effectively alleviate the burden of medical image transmission and storage, which is conducive to doctors getting the patient's intraoperative images for analysis as soon as possible, and more convenient for personalized postoperative care and doctor-patient communication.

[0161] In some of the above-mentioned embodiments, during the implementation of the present application, there is still the problem of how to achieve efficient extraction and analysis of laparoscope image video data.

[0162] In this regard, the present application proposes that the data stream overall scheduling unit 110 includes a data scheduling unit 111 for data stream scheduling and an extraction unit 112 for extracting key frames;

[0163] The extraction unit 112 includes a global key frame extraction unit and a local key frame extraction unit, and the key frames include global key frames and local key frames;

[0164] The global key frame extraction unit is used to extract global key frames based on an interval fixed frame rate or based on changes in surgical stages, and the global key frames are at least used for postoperative record analysis;

[0165] The local key frame extraction unit is used to monitor the changes of surgical instruments or lesion areas in real time, and extract local key frames through interest region tracking or image saliency analysis. The local key frames are at least used for real-time image analysis during surgery.

[0166] The data scheduling unit 111 is at least used to transmit the global key frame to the data temporary storage module 200 , and transmit the local key frame to the artificial intelligence video analysis operation unit 120 .

[0167] The present application realizes efficient extraction and analysis of laparoscopic image video data by designing a data stream overall scheduling unit 110, including a data scheduling unit 111 and an extraction unit 112, wherein the extraction unit 112 includes a global key frame extraction unit and a local key frame extraction unit. The global key frame extraction unit extracts global key frames by fixing the frame rate or the change of the surgical stage, which is mainly used for postoperative record analysis. This method can ensure that key information is evenly obtained throughout the operation and provide comprehensive data support for postoperative analysis. The local key frame extraction unit focuses on real-time monitoring of changes in surgical instruments or lesion areas, extracts local key frames by tracking the region of interest or image significance analysis, and is mainly used for real-time image analysis during surgery. This method can capture key changes during the operation and provide doctors with timely visual feedback. The data scheduling unit 111 is responsible for transmitting the global key frame to the data temporary storage module 200 and transmitting the local key frame to the artificial intelligence video analysis operation unit 120. This shunting processing method can improve the efficiency of data processing and ensure that local key frames with high real-time requirements can be analyzed in a timely manner.

[0168] The data stream overall scheduling unit 110 of the present application includes a data scheduling unit 111 and an extraction unit 112, wherein the extraction unit 112 is further divided into a global key frame extraction unit and a local key frame extraction unit. This structural design can realize efficient extraction and analysis of laparoscope video data.

[0169] The global keyframe extraction unit can extract global keyframes in two ways: based on an interval fixed frame rate or based on changes in the surgical stage. For example, it can be set to extract a global keyframe every 30 frames or 60 frames, or extract a global keyframe when it is detected that the surgery enters a new stage. These global keyframes are mainly used for postoperative record analysis and can provide doctors with an overview of the entire surgical process.

[0170] The local keyframe extraction unit focuses on real-time monitoring of changes in surgical instruments or lesion areas. It can extract local keyframes through region of interest tracking or image saliency analysis. For example, a target tracking algorithm can be used to continuously monitor the position changes of surgical instruments, or an image saliency algorithm can be used to detect mutations in the lesion area. These local keyframes are mainly used for real-time image analysis during surgery, which can help doctors detect key changes in a timely manner.

[0171] The data scheduling unit 111 is responsible for transmitting different types of key frames to the corresponding modules. The global key frames are transmitted to the data temporary storage module 200, which can be used for long-term storage and subsequent analysis. The local key frames are transmitted to the artificial intelligence video analysis operation unit 120 for real-time analysis and feedback. This diversion processing method can improve the overall data processing efficiency.

[0172] Through this global and local combination of key frame extraction methods and a reasonable data scheduling strategy, the technical solution of this application can improve the extraction efficiency of key information and the real-time analysis while ensuring comprehensiveness. This not only helps to improve the accuracy and safety of surgery, but also provides rich data support for postoperative analysis and medical teaching.

[0173] The data scheduling unit 111 is responsible for transmitting the global key frame to the data temporary storage module 200 and transmitting the local key frame to the artificial intelligence video analysis operation unit 120. This diversion processing method improves the efficiency of data processing and ensures that the local key frame with high real-time requirements can be analyzed in time.

[0174] Therefore, the technical solution of this application not only solves the problem of how to achieve efficient extraction and analysis of laparoscopic image video data, but also improves the extraction efficiency of key information and the real-time analysis while ensuring comprehensiveness. This method provides doctors with more comprehensive and timely surgical information support, which helps to improve the quality and efficiency of surgery. At the same time, by collecting and analyzing these key frame data, it also provides valuable resources for medical teaching and research.

[0175] As a preferred embodiment, the global key frame extraction unit of the present application can be set to extract a global key frame every 60 frames. For example, in a 30-frame / second video stream, a global key frame is extracted every 2 seconds. At the same time, it can also judge the transition of the surgical stage by identifying the change of surgical instruments or the significant change of the surgical area, and extract an additional global key frame when the stage transition is detected.

[0176] The local keyframe extraction unit can use a deep learning-based target detection algorithm to track the position of surgical instruments. For example, the YOLOv5 algorithm can be used to detect the position of surgical instruments in real time, and a local keyframe is extracted when the position of the instrument changes significantly (such as the moving distance exceeds 10% of the image width). At the same time, an image saliency algorithm (such as the Itti-Koch model) can be used to detect mutations in the lesion area, and a local keyframe is also extracted when the saliency value exceeds a preset threshold (such as 0.8).

[0177] The data scheduling unit 111 can use a priority queue to manage different types of key frames. For example, local key frames can be given a higher processing priority to ensure that they can be quickly transmitted to the artificial intelligence video analysis operation unit 120 for real-time analysis. Global key frames can be given a lower priority and transmitted to the data temporary storage module 200 when system resources allow.

[0178] Through this specific implementation method, the present application can realize the efficient extraction and analysis of laparoscopic image video data in practical applications, and provide doctors with timely and comprehensive surgical information support.

[0179] Compared with the prior art, the technical solution of this application has the following advantages:

[0180] Balance between comprehensiveness and real-time performance: By using both global keyframe and local keyframe extraction methods, this application can capture detailed changes at critical moments while ensuring a comprehensive record of the entire surgical process. Compared with a single fixed frame rate extraction method, this method can better balance the needs of comprehensiveness and real-time performance.

[0181] Intelligent key frame extraction: The local key frame extraction unit of the present application extracts key frames by real-time monitoring of changes in surgical instruments or lesion areas. This method is more intelligent than simple fixed frame rate extraction and can more accurately capture key information during the operation.

[0182] Efficient data processing: By transferring global keyframes and local keyframes to different modules for processing, the present application achieves more efficient data processing. This split processing method can ensure that local keyframes with high real-time requirements can be analyzed in a timely manner without affecting the storage and subsequent analysis of global keyframes.

[0183] Strong adaptability: The technical solution of this application can be adjusted according to different surgical types and requirements. For example, the extraction frequency of global key frames can be adjusted according to the complexity of the surgery, or the extraction strategy of local key frames can be adjusted according to different surgical instruments and lesion types.

[0184] These advantages enable the technical solution of this application to better meet the needs of modern laparoscopic surgery for image data processing, and provide strong technical support for improving surgical quality and efficiency.

[0185] It is worth noting that in some specific embodiments, global keyframes can also be analyzed and output for display during surgery, and local keyframes can also be transmitted to the data storage module 200 for storage for postoperative analysis, which depends on whether the computing resources are sufficient.

[0186] In some of the above-mentioned embodiments, during the implementation of the present application, there are still problems of duplicate frames in the key frame extraction and scheduling process and data scheduling priority issues.

[0187] In this regard, the present application proposes that the extraction unit 112 further includes:

[0188] A key frame duplication detection unit is used to compare the extracted local key frames with the global key frames to detect whether there are duplications;

[0189] A key frame merging unit, used for merging repeated frames when it is detected that a local key frame is repeated with a global key frame;

[0190] The data stream overall scheduling unit 110 further includes a key frame scheduling unit, which is used to obtain the type of data being scheduled by the data scheduling unit 111 and control the priority of the data scheduling unit 111 in scheduling different data according to the data type.

[0191] For duplicate frame detection, the key frame duplication detection unit uses image feature comparison. Specifically, it can calculate the similarity between frames, and when the similarity exceeds a preset threshold (for example, 95%), it is determined to be a duplicate frame. The similarity calculation can be based on color histogram, texture features, or deep learning features.

[0192] After detecting the repetition, the key frame merging unit will retain the frame with the largest amount of information as the final key frame. The amount of information can be comprehensively determined by indicators such as image clarity and target area ratio. During the merging process, metadata such as the timestamp of the repeated frame will be associated and stored.

[0193] The key frame scheduling unit monitors the working status of the data scheduling unit 111 in real time. The current data types can be divided into two categories: real-time analysis data and storage archive data. A higher priority (such as a priority coefficient of 1.5) is given to the real-time analysis data to ensure timely processing. A lower priority (such as a priority coefficient of 0.8) is used for storage archive data.

[0194] By introducing frame duplication detection and intelligent merging mechanisms, the system can effectively avoid resource waste caused by duplicate frames. At the same time, differentiated scheduling strategies based on data types enable the system to reasonably utilize computing resources while ensuring real-time requirements. This data processing mechanism not only improves system operation efficiency, but also ensures timely analysis and storage of key information.

[0195] As a specific embodiment, the system can work like this in actual operation: when it is detected that the feature similarity between the local key frame and the global key frame reaches 98%, the system will trigger the merge operation. By calculating the clarity index of the two frames (Laplacian operator can be used), the frame with higher clarity is retained. At the same time, the time relationship of the merged frame is recorded for subsequent tracing. In terms of data scheduling, if the surgical instrument tracking data is currently being processed, the processing priority is automatically increased to ensure real-time response.

[0196] Compared with the prior art, the present application significantly reduces system resource consumption and improves data processing efficiency through the collaboration of duplicate frame detection and merging and intelligent scheduling.

[0197] Specifically, in some preferred embodiments, the scheduling strategy of the key frame scheduling unit is:

[0198] ;

[0199] , , , , , ;

[0200] in:

[0201] Indicates keyframes;

[0202] Indicates The importance weight of each key frame;

[0203] Indicates The processing time of each key frame;

[0204] Indicates The timestamp of each key frame;

[0205] Indicates The timestamp of each key frame;

[0206] Indicates the current time;

[0207] Indicates the scheduling period, that is, the time interval for processing frames;

[0208] Indicates the frame interval time, indicating the maximum allowed frame interval;

[0209] Indicates processing key frames The amount of resources required;

[0210] Indicates the maximum computing resources available to the system;

[0211] , , are the priority weight coefficients of the corresponding items respectively;

[0212] Represents the scheduling queue, which contains the selected keyframes and is a collection of all keyframes to be scheduled A subset of .

[0213] The calculation formula is to maximize the selected dispatch queue The total priority value of all key frames in is used for sorting and scheduling. The frames with higher priority are scheduled first. , , They are , , The priority weight is, Controls the influence of image importance weight. A larger value will pay more attention to the importance of key frames. Controls the effect of processing time on priority. When it is larger, it tends to select frames with shorter processing time. Controls the impact of time factors on priority. When it is larger, it tends to select the most recently generated keyframe.

[0214] Is a keyframe The importance weight is calculated based on the surgical stage, changes in the lesion area, etc. The larger the value, the higher the importance of the frame.

[0215] Is a keyframe The processing time indicates the time required to complete the processing of the frame. The smaller the value, the higher the priority.

[0216] Is a keyframe The timestamp indicates the time when the frame was captured and is used to calculate the newness of the frame.

[0217] , For real-time constraints, keyframes Processing time Must be less than the scheduling period , ensuring that the system can complete the processing of the frame within the scheduling cycle.

[0218] , It is a frame interval constraint, which indicates the maximum allowed time interval between two scheduled key frames. It is used to limit the interval between adjacent frames in the scheduling queue to ensure that the system works at a stable frame rate.

[0219] It is a resource utilization constraint, which means that the system cannot exceed the limit of its computing capacity when processing key frames.

[0220] Furthermore, in some preferred embodiments, The calculation formula is:

[0221] ;

[0222] in:

[0223] Represents the dynamically adjusted weight;

[0224] Represents nonlinear transformations or interactive calculations for each factor, including:

[0225] , is the proportion of the lesion area, which measures the importance of the lesion area in the key frame;

[0226] , is the surgical instrument interaction intensity, reflecting the interaction between the instrument and the lesion area;

[0227] , is the weight of the surgical stage, reflecting the importance of the surgical stage corresponding to the frame;

[0228] , is the image saliency, which measures the distribution characteristics of visual saliency in the frame;

[0229] , is the motion characteristic of the lesion area, which measures the intensity of dynamic changes in the lesion area;

[0230] ,For redundancy detection, avoid selecting duplicate frames;

[0231] , is the dynamic characteristics of the interaction between the lesion area and the instrument;

[0232] Among them, in some preferred embodiments:

[0233] ;

[0234] The timestamp of the current key frame;

[0235] is the estimated total duration of the operation;

[0236] Dynamically adjust the weight;

[0237] is the basic weight, which indicates the initial weight distribution of each factor;

[0238] ;

[0239] Keyframe The area of ​​the lesion in

[0240] Keyframe The total area in

[0241] Keyframe the perimeter of the lesion area, indicating the boundary complexity;

[0242] represents the lesion area proportion function;

[0243] ;

[0244] Keyframe The overlapping area between the instrument and the lesion area;

[0245] Keyframe Exercise intensity of medium equipment;

[0246] is the maximum motion intensity of the equipment in all frames;

[0247] represents the surgical instrument interaction strength function;

[0248] ;

[0249] Keyframe Medium Pixels The strength of significance;

[0250] represents the image saliency function;

[0251] is the variance of the significant pixel intensity;

[0252] ;

[0253] is the pixel point in the lesion area movement displacement;

[0254] Represents the motion characteristic function;

[0255] ;

[0256] Keyframe With keyframes similarity;

[0257] represents the similarity function;

[0258] ;

[0259] Represents the dynamic characteristic function of the interaction between the lesion area and the instrument.

[0260] Specifically:

[0261] ;

[0262] This part of the formula calculates the area ratio of the lesion area relative to the entire image, and further adjusts the weight according to the shape complexity of the area. Through logarithmic function processing, the extreme influence can be weakened to avoid the dominance of too large an area on importance.

[0263] ;

[0264] This part of the formula calculates the interaction between the instrument and the lesion area, and adjusts the interaction weight according to the intensity of the instrument movement. The larger the area where the instrument interacts with the lesion and the more obvious the movement, the higher the importance of the keyframe.

[0265] ;

[0266] This part of the formula uses a square function to enhance the importance of key stage frames according to the importance of the surgical stage. Higher scores result in higher weights. It represents the surgical stage scoring function, and set values ​​can be given in advance for different surgical stages, that is, different surgical stage scoring functions.

[0267] ;

[0268] This part of the formula calculates the ratio of the saliency of the lesion area to the saliency of the entire keyframe, and adds the variance of the saliency distribution to emphasize the importance of keyframes where the saliency area is more concentrated.

[0269] ;

[0270] This part of the formula calculates the average movement of pixels in the lesion area. The larger the movement, the more significant the change in the lesion area, and the more important the keyframe is.

[0271] ;

[0272] This part of the formula calculates the frame With frame If the current frame is highly similar to the previous frame, the redundancy correction term will reduce the weight of the frame to avoid the influence of repeated frames.

[0273] ;

[0274] This part of the formula calculates the dynamic interaction between the lesion area and the instrument. The more obvious the interaction between the instrument and the lesion and the regional motion, the higher the importance of the frame.

[0275] This application can more accurately evaluate the importance of key frames through the above calculation method. Specifically, the formula dynamically evaluates the criticality of each frame by comprehensively considering multi-dimensional factors to ensure that the selected key frames can accurately reflect the key image changes during surgery. The specific effects of the formula include:

[0276] Accuracy: Through the combination of multiple factors such as lesion area ratio, significance, instrument interaction, motion information, etc., the most important medical information in the key frame can be accurately captured to avoid missing key details.

[0277] Dynamic adaptability: The introduction of dynamic weights can be adaptively adjusted according to the needs of different stages of surgery (such as the opening stage, cutting stage, and suturing stage), highlighting the key frame characteristics of different stages.

[0278] Redundancy removal: Through the redundant correction term, the importance of repeated frames is effectively reduced, and frames with highly similar content in adjacent frames are prevented from occupying processing resources, thereby improving the diversity of key frames.

[0279] Comprehensive trade-off: The formula incorporates characteristics such as regional shape complexity, instrument dynamic interaction, and significant distribution variance, providing a comprehensive analysis of complex surgical scenarios and adapting to the imaging needs of various minimally invasive surgeries.

[0280] The solution of this application is a real-time intraoperative laparoscopic image analysis device, which aims to provide support for real-time intraoperative assistance and postoperative review by collecting, analyzing and outputting key image data. The core role of the above formula in this solution is the importance evaluation and screening of key frames, which specifically includes the following points:

[0281] Improve image analysis efficiency: Calculate the importance weights of image frames in real time, quickly filter out key frames from continuous video streams, reduce the processing of unnecessary frames, and optimize computing resource allocation.

[0282] Supports multi-dimensional image feature analysis: Combining multi-dimensional characteristics such as lesion area changes, instrument movement, significance and surgical stage, ensures that the selected key frames can reflect the most important medical information during the operation (such as lesion changes, instrument operation key points).

[0283] Meeting real-time requirements: Dynamically adjust weights and screening logic to ensure that key frames that meet current surgical requirements are selected at different surgical stages, providing accurate real-time imaging support for intraoperative decision-making.

[0284] Second, refer to Figure 3 The present application also proposes a real-time intraoperative laparoscopic image analysis method, which is applied to the above-mentioned device. The steps of the method include:

[0285] S110, acquiring laparoscope image video data transmitted by the laparoscope video data input module 300;

[0286] S120, extracting laparoscope image video data to obtain key frames;

[0287] S130, storing the key frame in the data temporary storage module 200, and performing image analysis on the key frame to obtain a labeled image;

[0288] S140, transmitting the annotated image and the laparoscope image video data to the laparoscope video data output module 400 for output.

[0289] Through the artificial intelligence acceleration processor 100, key frame selection, artificial intelligence analysis and processing are performed on the laparoscopic image video data, and then through the laparoscopic video data output module 400, the original video and the analyzed video can be exported, which plays a role in assisting the intraoperative surgery and improving the surgical work efficiency and decision-making accuracy of the surgeon. At the same time, the general artificial intelligence acceleration processor algorithm can also be updated by wireless transmission to improve the accuracy of the laparoscopic image analysis device.

[0290] In addition, in some preferred embodiments, a real-time intraoperative laparoscopic image analysis method proposed in the present application also includes method steps executed by any module or unit in the above-mentioned real-time intraoperative laparoscopic image analysis device.

[0291] The invention concept is described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the core idea of ​​the invention. It should be pointed out that any obvious modification, equivalent substitution or other improvement made by a person of ordinary skill in the art without departing from the invention concept should be included in the protection scope of the present invention.

[0292] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A real-time intraoperative laparoscopic image analysis device, characterized in that: include: An artificial intelligence acceleration processor, a data temporary storage module, a laparoscope video data input module, and a laparoscope video data output module, wherein the laparoscope video data input module, the artificial intelligence acceleration processor, and the laparoscope video data output module are connected in sequence, and the data temporary storage module is connected to the artificial intelligence acceleration processor; The artificial intelligence acceleration processor obtains the laparoscope image video data transmitted by the laparoscope video data input module, extracts the laparoscope image video data to obtain key frames, stores the key frames in the data temporary storage module, performs image analysis on the key frames to obtain annotated images, and transmits the annotated images and the laparoscope image video data to the laparoscope video data output module for output; The artificial intelligence acceleration processor includes a data stream overall scheduling unit and an artificial intelligence video analysis operation unit, and the data stream overall scheduling unit is connected to the artificial intelligence video analysis operation unit; The data stream overall scheduling unit transmits the laparoscope image video data to the laparoscope video data output module, extracts the laparoscope image video data to obtain the key frame, transmits the key frame to the data temporary storage module, and transmits the key frame to the artificial intelligence video analysis operation unit; The artificial intelligence video analysis operation unit performs image analysis on the key frame to obtain the annotated image, and transmits the annotated image to the data stream overall scheduling unit, which transmits the annotated image to the laparoscope video data output module and the data temporary storage module; The artificial intelligence video analysis operation unit performs image analysis on the key frame through the image key frame semantic segmentation AI analysis algorithm to obtain the labeled image with each pixel category labeled; wherein the key frame semantic segmentation AI analysis algorithm adopts a deep learning model; After the artificial intelligence video analysis operation unit performs image analysis on the key frame, the following steps are further included: Blurring the annotated image, overlapping and stacking the blurred image with the corresponding key frame to obtain an auxiliary image, sending the auxiliary image to the data stream overall scheduling unit, and the data stream overall scheduling unit transmitting the auxiliary image to the laparoscope video data output module for output; The data stream overall scheduling unit includes a data scheduling unit for data stream scheduling and an extraction unit for extracting the key frame; The extraction unit includes a global key frame extraction unit and a local key frame extraction unit, and the key frames include global key frames and local key frames; The global key frame extraction unit is used to extract the global key frame based on an interval fixed frame rate or based on the change of the surgical stage, and the global key frame is at least used for postoperative record analysis; The local key frame extraction unit is used to monitor the changes of surgical instruments or lesion areas in real time, and extract the local key frame by tracking the region of interest or analyzing the image significance, and the local key frame is at least used for real-time image analysis during surgery; The data scheduling unit is at least used to transmit the global key frame to the data temporary storage module, and transmit the local key frame to the artificial intelligence video analysis operation unit; The data stream overall scheduling unit also includes a key frame scheduling unit, which is used to obtain the data type being scheduled by the data scheduling unit, and control the priority of the data scheduling unit for scheduling different data according to the data type; The scheduling strategy of the key frame scheduling unit is: ； ， ， ， ， ， ； in: Indicates keyframes; Indicates The importance weight of each key frame; Indicates The processing time of each key frame; Indicates The timestamp of each key frame; Indicates The timestamp of each key frame; Indicates the current time; Indicates the scheduling period, that is, the time interval for processing frames; Indicates the frame interval time, indicating the maximum allowed frame interval; Indicates processing key frames The amount of resources required; Indicates the maximum computing resources available to the system; , , are the priority weight coefficients of the corresponding items respectively; Represents the scheduling queue, which contains the selected keyframes and is a collection of all keyframes to be scheduled A subset of .

2. A real-time intraoperative laparoscopic image analysis device according to claim 1, characterized in that: The data stream overall scheduling unit extracts the laparoscope image video data at intervals of a fixed frame rate, and uses thread resources to perform batch scheduling on the key frames and the laparoscope image video data.

3. The real-time intraoperative laparoscopic image analysis device according to claim 1, characterized in that: The data temporary storage module is provided with an elastic memory, which performs slice-by-slice storage and transmission of the key frames, and the storage and transmission are controlled by the artificial intelligence acceleration processor; The elastic storage stores and transmits the annotated images and the key frames, wherein the key frames at least include key frame acquisition time, operation type, department, and image type.

4. The real-time intraoperative laparoscopic image analysis device according to claim 1, characterized in that: The extraction unit also includes: A key frame duplication detection unit, used for comparing the extracted local key frame with the global key frame to detect whether there is duplication; The key frame merging unit is used to merge the repeated frames when it is detected that the local key frame and the global key frame are repeated.

5. The real-time intraoperative laparoscopic image analysis device according to claim 1, characterized in that: Said The calculation formula is: ； in: Represents the dynamically adjusted weight; Represents nonlinear transformations or interaction calculations for each factor, including: , is the proportion of the lesion area, which measures the importance of the lesion area in the key frame; , is the surgical instrument interaction intensity, reflecting the interaction between the instrument and the lesion area; , is the weight of the surgical stage, reflecting the importance of the surgical stage corresponding to the frame; represents the surgical stage scoring function; , is the image saliency, which measures the distribution characteristics of visual saliency in the frame; , is the motion characteristic of the lesion area, which measures the intensity of dynamic changes in the lesion area; ,For redundancy detection, avoid selecting duplicate frames; , is the dynamic characteristics of the interaction between the lesion area and the instrument; in: ； The timestamp of the current key frame; is the estimated total duration of the operation; Dynamically adjust the weight; is the basic weight, which indicates the initial weight distribution of each factor; ； Keyframe The area of ​​the lesion in Keyframe The total area in Keyframe the perimeter of the lesion area, indicating the boundary complexity; represents the lesion area proportion function; ； Keyframe The overlapping area between the instrument and the lesion area; Keyframe Exercise intensity of medium equipment; is the maximum motion intensity of the equipment in all frames; represents the surgical instrument interaction strength function; ； Keyframe Medium Pixels The strength of significance; represents the image saliency function; is the variance of the significant pixel intensity; ； is the pixel point in the lesion area movement displacement; Represents the motion characteristic function; ； Keyframe With keyframes similarity; represents the similarity function; ； Represents the dynamic characteristic function of the interaction between the lesion area and the instrument.

6. A real-time intraoperative laparoscopic image analysis method, applied to the device according to any one of claims 1 to 5, characterized in that: The steps of the method include: Acquire the laparoscope image video data transmitted by the laparoscope video data input module; Extracting the laparoscope image video data to obtain key frames; The key frame is stored in the data temporary storage module, and image analysis is performed on the key frame to obtain a marked image; The annotated image and the laparoscope image video data are transmitted to the laparoscope video data output module for output.

Images