Minimally invasive endoscope image-text report generation system

Abstract

The invention belongs to the technical field of medical equipment, and discloses a minimally invasive endoscope image-text report generation system which comprises an image acquisition device and a report generation system, the report generation system comprises a patient information access module, a key frame image screening module and a report generation module, and the image acquisition device is in communication connection with the report generation system; the image acquisition device is used for acquiring an operation process image through the minimally invasive endoscope and sending the image to the report generation system; the patient information access module is used for acquiring basic information of a patient from a medical information system of a hospital; the key frame image screening module is used for screening key frame images from the operation process images according to the basic information of the patient; the report generation module is used for generating an operation image-text report according to the basic information of the patient and the key frame image; therefore, the user can be assisted to quickly generate the image-text report of the minimally invasive surgery, and the generation efficiency of the image-text report of the minimally invasive surgery is improved.

Description

Technical Field

[0001] The present application relates to the field of medical equipment technology, and in particular to a minimally invasive laparoscopic graphic report generation system. Background Art

[0002] The graphic reports of minimally invasive surgery can directly reflect the degree of the patient's lesions and the intraoperative treatment, provide a diagnostic basis for the patient's subsequent treatment, and assist in patient care; they can also serve as the main evidence for postoperative doctor-patient communication, reduce doctor-patient conflicts, and reduce duplication and waste of medical resources. In addition, the graphic reports of minimally invasive surgery can more conveniently spread the theoretical knowledge and rich clinical experience of professional doctors to other young doctors, which is conducive to strengthening the training effect of medical students.

[0003] However, current graphic reports of minimally invasive surgeries are generally generated manually. When generating graphic reports, doctors are required to manually select key frame images from the video shot during the operation to insert into the graphic reports. The screening process takes a lot of time, which not only increases the workload of doctors, but also has low generation efficiency. Summary of the invention

[0004] The purpose of this application is to provide a minimally invasive laparoscopic graphic report generation system that can assist users in quickly generating graphic reports of minimally invasive surgery, thereby improving the efficiency of generating graphic reports of minimally invasive surgery.

[0005] The present application provides a minimally invasive laparoscope graphic report generation system, including an image acquisition device and a report generation system, wherein the report generation system includes a patient information access module, a key frame image screening module and a report generation module, and the image acquisition device and the report generation system are communicatively connected; The image acquisition device is used to acquire surgical process images through a minimally invasive laparoscope and send the images to the report generation system; The patient information access module is used to obtain basic patient information from the hospital's medical information system; The key frame image screening module is used to screen key frame images from the surgical process images according to the basic information of the patient; The report generation module is used to generate a surgical graphic report based on the patient's basic information and the key frame images.

[0006] The minimally invasive laparoscopic graphic report generation system can automatically screen key frame images, reducing the time spent by users in screening key frame images from surgical process images, thereby assisting users in quickly generating graphic reports of minimally invasive surgery and improving the efficiency of generating graphic reports of minimally invasive surgery.

[0007] Preferably, the image acquisition device comprises a central control processing module, a minimally invasive laparoscope, a memory, a wireless communication module and a command triggering device, and the minimally invasive laparoscope, the memory, the wireless communication module and the command triggering device are all electrically connected to the central control processing module; Minimally invasive laparoscopes are used to collect images of the surgical process; The instruction triggering device is used to generate prompt instructions under the operation of the chief surgeon; The central control processing module is used to record the surgical process image in the memory, and when receiving the prompt instruction, mark the surgical process image with key frames, and send the surgical process image to the report generation system through the wireless communication module.

[0008] During the operation, the surgeon in charge can issue prompt instructions through the command trigger device at critical moments based on his own experience, thereby marking the frame image at the corresponding moment as a key frame image. The marked key frame image usually contains important information, which can complement the key frame image automatically screened by the system, thereby reducing the probability of missing the frame image containing important information.

[0009] Preferably, the central control processing module is also used to adjust the image acquisition frame rate of the minimally invasive laparoscope in different surgical stages according to the patient's basic information, and the surgical stages include non-important surgical stages and important surgical stages, so that the image acquisition frame rate in non-important surgical stages is lower than the image acquisition frame rate in important surgical stages.

[0010] Since the storage space of the memory of the image acquisition device is limited, reducing the frame rate of image acquisition in non-important surgical stages can save storage space for storing more useful information.

[0011] Preferably, when the central control processing module adjusts the image acquisition frame rate of the minimally invasive laparoscope at different surgical stages according to the basic information of the patient, the central control processing module executes: Acquire a moving distance cutoff value according to the basic information of the patient; Obtaining the real-time moving distance of the minimally invasive laparoscope; Comparing the real-time moving distance with the moving distance cutoff value to determine the category of the current surgical stage; If the current surgical stage is a non-important surgical stage, the minimally invasive laparoscope is controlled to capture the surgical process images at a first preset frame rate. If the current surgical stage is an important surgical stage, the minimally invasive laparoscope is controlled to capture the surgical process images at a second preset frame rate. The first preset frame rate is less than the second preset frame rate.

[0012] Preferably, when recording the surgical process image in the memory, the central control processing module executes: When the remaining storage space of the memory is lower than a preset storage space threshold, estimating the data size of the surgical process image of this operation according to the basic information of the patient; Determining the historical surgical process images to be deleted and the frame images to be deleted according to the data size of the key frame images marked in the historical surgical process images in the memory and the estimated data size of the surgical process images of the current surgery; When the memory is full, each frame image to be deleted of each historical surgical process image to be deleted is deleted in the order of recording time, so as to reserve storage space for storing the surgical process images collected in real time.

[0013] Preferably, when the key frame image screening module screens the key frame images from the surgical process images according to the basic information of the patient, the key frame image screening module performs: Establishing a key frame image set, and adding the frame images marked as key frame images in the surgical process image to the key frame image set; Using a pre-trained image feature recognition model, key features are recognized for each frame image in the surgical process image that is not marked as a key frame image, to obtain key features contained in the corresponding frame image; Calculating a feature content evaluation function of each frame image not marked as a key frame image according to key features contained in each frame image not marked as a key frame image; Whether each frame image not marked as a key frame image is a key frame image is determined according to the feature content evaluation function, and the frame images determined to be key frame images are added to the key frame image set.

[0014] Preferably, according to the key features contained in each frame image not marked as a key frame image, calculating the feature content evaluation function of each frame image not marked as a key frame image specifically includes: Calculate the matching degree between each key feature contained in the frame image that is not marked as a key frame image and the standard key feature of the same type; The characteristic content evaluation function of the frame image not marked as a key frame image is calculated according to the following formula: ; in, is the feature content evaluation function of the frame image not marked as a key frame image, m is the number of key features contained in the frame image not marked as a key frame image, is the matching degree corresponding to the i-th key feature contained in the frame image not marked as a key frame image, is the weight factor corresponding to the i-th key feature contained in the frame image that is not marked as a key frame image.

[0015] Preferably, determining whether each frame image not marked as a key frame image is a key frame image according to the feature content evaluation function specifically includes: According to the magnitude of the characteristic content evaluation function, each frame image that is not marked as a key frame image is sorted in descending order; According to the sorting result, the first N frame images in the sorting are determined as key frame images, where N is a preset positive integer.

[0016] Preferably, the report generation module includes a report picture generation module and a report document generation module; The report image generation module is used to assist the user in selecting the key frame image, and to annotate the selected key frame image to generate a report image; The report document generation module is used to obtain the corresponding report template according to the patient's basic information, and after calling the report image generation module to perform key frame image selection and annotation processing, the generated report image is inserted into the corresponding position in the report template.

[0017] Preferably, the surgical graphic report includes part or all of the patient's basic information, graphic process information, pathological biopsy information, postoperative diagnosis information, and doctor's advice information.

[0018] Beneficial effects: The minimally invasive laparoscopic graphic report generation system provided in the present application can automatically filter key frame images, reducing the time spent by users in filtering key frame images from surgical process images, thereby assisting users in quickly generating graphic reports of minimally invasive surgery and improving the efficiency of generating graphic reports of minimally invasive surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A structural schematic diagram of a minimally invasive laparoscopic graphic report generation system provided in an embodiment of the present application.

[0020] Figure 2 It is a structural schematic diagram of the image acquisition device.

[0021] Figure 3 A schematic diagram of an exemplary surgical report.

[0022] Explanation of reference numerals: 1. Image acquisition device; 101. Central control processing module; 102. Minimally invasive laparoscope; 103. Memory; 104. Wireless communication module; 105. Command trigger device; 2. Report generation system; 201. Patient information access module; 202. Key frame image screening module; 203. Report generation module; 204. Report picture generation module; 205. Report document generation module; 206. Printing transmission module; 90. Medical information system. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0024] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] Please refer to Figure 1 , Figure 2 In some embodiments of the present application, a minimally invasive laparoscope graphic report generation system includes an image acquisition device 1 and a report generation system 2, wherein the report generation system 2 includes a patient information access module 201, a key frame image screening module 202 and a report generation module 203, and the image acquisition device 1 and the report generation system 2 are in communication connection; The image acquisition device 1 is used to acquire surgical process images through the minimally invasive laparoscope 102 and send them to the report generation system 2; The patient information access module 201 is used to obtain basic patient information from the hospital's medical information system 90; The key frame image screening module 202 is used to screen key frame images from the surgical process images according to the basic information of the patient; The report generation module 203 is used to generate a surgical graphic report based on the patient's basic information and key frame images.

[0026] The minimally invasive laparoscopic graphic report generation system can automatically screen key frame images, reducing the time spent by users in screening key frame images from surgical process images, thereby assisting users in quickly generating graphic reports of minimally invasive surgery and improving the efficiency of generating graphic reports of minimally invasive surgery.

[0027] The basic patient information includes part or all of the patient's name, gender, age, height and weight, department, surgeon, surgery type, fee type, medical history, clinical manifestations, diagnosis and treatment, and necessary materials related to disease diagnosis or surgery (e.g., at least one of physical examination results, imaging images, pathological images, and various biochemical test results), but is not limited thereto. All of this information is recorded in the hospital's medical information system 90, and can be retrieved from the medical information system 90 through the patient information access module 201.

[0028] Specifically, see Figure 2 The image acquisition device 1 includes a central control processing module 101, a minimally invasive laparoscope 102, a memory 103, a wireless communication module 104 and a command trigger device 105, and the minimally invasive laparoscope 102, the memory 103, the wireless communication module 104 and the command trigger device 105 are all electrically connected to the central control processing module 101; The minimally invasive laparoscope 102 is used to collect images of the surgical process; The instruction triggering device 105 is used to generate prompt instructions under the operation of the surgeon; The central control processing module 101 is used to record the surgical process images in the memory 103, and when receiving the prompt instruction, it marks the surgical process images as key frames, and sends the surgical process images to the report generation system 2 through the wireless communication module 104 (therefore, the report generation system 2 is provided with a corresponding communication module for communicating with the wireless communication module 104).

[0029] During the operation, the surgeon in charge can issue prompt instructions through the instruction trigger device 105 at critical moments based on his own experience, thereby marking the frame image at the corresponding moment as a key frame image. The marked key frame image usually contains important information and can complement the key frame image automatically screened by the system, thereby reducing the probability of missing the frame image containing important information.

[0030] Among them, the minimally invasive laparoscope 102 and the memory 103 are prior art and will not be described in detail here. The central control processing module 101 may adopt an MCU module, but is not limited thereto. The wireless communication module 104 may adopt a WIFI module, a 4G module, a 5G module, a Bluetooth module, a zigbee module, etc., but is not limited thereto. The command trigger device 105 may be, but is not limited to, a button module (such as a hand-pressed button module, a foot-operated button module, etc.) or a voice recognition module; if the command trigger device 105 is a button module, the prompt command is a button trigger command, and if the command trigger device 105 is a voice recognition module, the prompt command is a voice command.

[0031] The minimally invasive laparoscope 102 may collect images of the surgical process at a constant image collection frame rate, or may adjust the image collection frame rate at different surgical stages according to actual needs.

[0032] For example, in some embodiments, the central control processing module 101 is also used to adjust the image acquisition frame rate of the minimally invasive laparoscope 102 at different surgical stages according to the basic information of the patient. The surgical stage includes a non-important surgical stage and an important surgical stage, so that the image acquisition frame rate of the non-important surgical stage is lower than the image acquisition frame rate of the important surgical stage. Among them, the important surgical stage refers to a stage with a high correlation with the surgery, and the non-important surgical stage refers to a stage with a low correlation with the surgery. Generally, the important surgical stage includes the process from the minimally invasive laparoscope 102 reaching the surgical target site (for example, when performing gastric surgery, the surgical target site is the stomach) to completing the surgical treatment of the surgical target site, and the non-important surgical stage includes the movement process of the minimally invasive laparoscope 102 before reaching the surgical target site, but it is not limited to this, and can be specifically set according to actual needs. Since the storage space of the memory 103 of the image acquisition device 1 is limited, reducing the image acquisition frame rate of the non-important surgical stage can save storage space to store more useful information.

[0033] Among them, the surgeon can switch the image acquisition frame rate through the command trigger device 105, so that the command trigger device 105 is also used to generate a switching command under the operation of the surgeon, so that the image acquisition frame rate of the minimally invasive laparoscope 102 is switched between a first preset frame rate and a second preset frame rate, and the first preset frame rate is less than the second preset frame rate. After the minimally invasive laparoscope 102 is started, the first preset frame rate is used as the default frame rate to acquire images during the surgical process. When the central control processing module 101 receives the switching command, it controls the minimally invasive laparoscope 102 to switch the image acquisition frame rate to the second preset frame rate. After receiving the switching command again, it controls the minimally invasive laparoscope 102 to switch the image acquisition frame rate back to the first preset frame rate (hereinafter, this image acquisition frame rate adjustment mode is referred to as the manual adjustment mode). The surgeon can switch autonomously according to actual needs. If the command trigger device 105 is a button module, the switching command is a button trigger command, and if the command trigger device 105 is a voice recognition module, the switching command is a voice command.

[0034] The system may also automatically switch the image acquisition frame rate to avoid the surgeon being distracted from manual switching; thus, in some embodiments, when the central control processing module 101 adjusts the image acquisition frame rate of the minimally invasive laparoscope 102 at different surgical stages according to the patient's basic information, the following is performed (hereinafter, this image acquisition frame rate adjustment mode is referred to as the automatic adjustment mode): Obtain the moving distance cutoff value based on the patient's basic information; Obtaining the real-time moving distance of the minimally invasive laparoscope 102; Compare the real-time moving distance with the moving distance cutoff value to determine the category of the current surgical stage; If the current surgical stage is a non-important surgical stage, the minimally invasive laparoscope 102 is controlled to capture surgical process images at a first preset frame rate. If the current surgical stage is an important surgical stage, the minimally invasive laparoscope 102 is controlled to capture surgical process images at a second preset frame rate.

[0035] By performing automatic adjustments in the above manner, the surgeon does not need to manually switch the image acquisition frame rate, allowing him to concentrate more on the operation.

[0036] When the automatic adjustment mode is adopted, the basic information of the patient obtained needs to include the patient's gender, age, height and weight, and type of surgery. The moving distance demarcation value is obtained based on the basic information of the patient, specifically including: determining the surgical target site according to the type of surgery, estimating the total moving distance required for the minimally invasive laparoscope 102 to reach the surgical target site according to the patient's gender, age, and height and weight (the total moving distance is the path distance from the human body entrance to the surgical target site, for example, when the surgical target site is the stomach, the total moving distance required to move from the patient's mouth through the esophagus to the stomach is estimated; it can be estimated by a pre-trained deep learning model, but is not limited to this), and calculating the moving distance demarcation value based on the total moving distance (for example, multiplying the total moving distance by a preset proportional coefficient less than 1 to obtain the moving distance demarcation value, the preset proportional coefficient can be taken as 0.8, but is not limited to this).

[0037] An acceleration sensor may be provided in the minimally invasive laparoscope 102 to obtain the real-time acceleration of the minimally invasive laparoscope 102 , and the real-time moving distance of the minimally invasive laparoscope 102 may be obtained by performing a secondary integration of the real-time acceleration.

[0038] When the real-time moving distance is less than the moving distance threshold, the current surgical stage is determined to be a non-important surgical stage. When the real-time moving distance is greater than or equal to the moving distance threshold, the current surgical stage is determined to be an important surgical stage.

[0039] Among them, the surgeon can select the image acquisition frame rate adjustment mode before the operation. For example, a human-computer interaction module can be set on the image acquisition device 1, and the surgeon selects the image acquisition frame rate adjustment mode through the human-computer interaction module; if the surgeon selects the manual adjustment mode, the system will not automatically adjust the image acquisition frame rate during the operation. If the surgeon selects the automatic adjustment mode, the system automatically adjusts the image acquisition frame rate. In fact, for some types of operations, the total moving distance required for the minimally invasive laparoscope 102 to reach the target surgical site is small, and it is not suitable to use the above-mentioned automatic adjustment mode. Therefore, in some embodiments, the central control processing module 101 is also used to determine whether the automatic adjustment mode is applicable to this operation according to the type of operation when the surgeon selects the automatic adjustment mode before the operation. If not applicable, the selection result is changed to the manual adjustment mode, and a notification message is sent to inform the surgeon of the change result.

[0040] In some preferred embodiments, when the central control processing module 101 records the surgical process image in the memory 103, it executes: When the remaining storage space of the memory 103 is lower than a preset storage space threshold (which can be set according to actual needs), the data size of the surgical process image of this operation is estimated according to the basic information of the patient; Determine the historical surgical process images to be deleted and their frame images to be deleted according to the data size of the key frame images marked in the historical surgical process images (i.e., the surgical process images of the previously completed surgery) in the memory 103 and the estimated data size of the surgical process images of the current surgery; When the memory 103 is full, each frame image of each historical surgical process image to be deleted is deleted in the order of recording time, so as to reserve storage space for storing the surgical process images collected in real time.

[0041] The data size of the surgical process images of this operation is estimated based on the basic information of the patient, and the data size of the surgical process images of this operation is estimated specifically based on the type of operation (which can be estimated through big data statistics).

[0042] According to the data size of the key frame images marked in the historical surgical process images in the memory 103 and the estimated data size of the surgical process images of the current surgery, the historical surgical process images to be deleted and the frame images to be deleted are determined, specifically including: The estimated data size of the surgical process image of the current surgery is multiplied by a preset redundancy coefficient (the preset redundancy coefficient is greater than 1 and can be set according to actual needs, such as 1.2, but not limited thereto) to obtain a first data size; Calculate the sum of the data sizes of the key frame images marked in each historical surgical process image as the key data size of the corresponding historical surgical process image; Subtract the corresponding key data size from the total data size of each historical surgical process image to obtain the non-key data size of the corresponding historical surgical process image; If the sum of the non-critical data sizes of all historical surgical process images is not less than the first data size, then, according to the order of the recording time of each historical surgical process image, each historical surgical process image is sequentially determined as a historical surgical process image to be deleted, until the sum of the non-critical data sizes of all historical surgical process images to be deleted is not less than the first data size; and unmarked frame images (i.e., frame images not marked as key frame images) in each historical surgical process image to be deleted are determined as frame images to be deleted; If the sum of the non-critical data sizes of all historical surgical process images is less than the first data size, all historical surgical process images are determined as historical surgical process images to be deleted, and the unmarked frame images in each historical surgical process image to be deleted are used as frame images to be deleted, and according to the chronological order of the recording times of each historical surgical process image to be deleted, the marked key frame images in each historical surgical process image to be deleted are added in sequence as frame images to be deleted of the corresponding historical surgical process image to be deleted, until the sum of the data sizes of all frame images to be deleted is not less than the first data size.

[0043] Thus, when the memory 103 is full, each frame image to be deleted of each historical surgical process image to be deleted is deleted in the order of recording time to reserve storage space for storing the surgical process image acquired in real time; that is, every time a new image is acquired, a frame image to be deleted is deleted in a first-in-first-out manner to reserve storage space for storing the new image. By performing storage control of the surgical process image in the above manner, the key information in the historical surgical process image can be retained as much as possible while the surgical process image of the current operation is completely preserved for the user to check.

[0044] In some possible implementations, when the central control processing module 101 receives a prompt instruction, it marks the surgical process image as a key frame, specifically: marking the frame image collected in a preset time period (which can be set according to actual needs, for example, 2s, but not limited to this) with the moment when the prompt instruction is received as the midpoint as a key frame image.

[0045] After the key frame marking is completed, the surgical process image is sent to the report generating system 2, so that the surgical process image received by the report generating system 2 contains the marking information of the key frame image.

[0046] In this embodiment, when the key frame image screening module 202 screens the key frame images from the surgical process images according to the basic information of the patient, it executes: Establishing a key frame image set, and adding frame images marked as key frame images in the surgical process images to the key frame image set; Using a pre-trained image feature recognition model, key features of each frame image that is not marked as a key frame image in the surgical process image are recognized to obtain the key features contained in the corresponding frame image; Calculating a feature content evaluation function of each frame image not marked as a key frame image according to key features contained in each frame image not marked as a key frame image; Whether each frame image not marked as a key frame image is a key frame image is determined according to the feature content evaluation function, and the frame image determined to be a key frame image is added to the key frame image set.

[0047] Among them, each type of surgery generally has multiple key features (such as lesion characteristics of the target site, surgical wound characteristics of the target site, local shape characteristics of the target site, etc., but not limited to these), and the key features of different types of surgery may be partially the same or completely different.

[0048] Among them, according to each type of surgery, the corresponding surgical images can be collected and key features can be annotated to form a surgical image data set, and then the surgical image data sets of each type of surgery can be used to train the image feature recognition model to obtain the pre-trained image feature recognition model corresponding to each type of surgery. Thus, the pre-trained image feature recognition model is used to perform key feature recognition on each frame image in the surgical process image that is not marked as a key frame image, specifically including: calling the corresponding pre-trained image feature recognition model according to the surgical type of this surgery, and using the called pre-trained image feature recognition model to perform key feature recognition on each frame image in the surgical process image that is not marked as a key frame image.

[0049] Furthermore, according to the key features contained in each frame image not marked as a key frame image, a feature content evaluation function of each frame image not marked as a key frame image is calculated, specifically including: Calculate the matching degree between each key feature contained in the frame image not marked as a key frame image and the standard key feature of the same type; specifically, for each key feature, the corresponding standard key feature is pre-recorded to form a standard key feature library for each type of surgery. In actual application, image matching is performed between each key feature contained in the frame image not marked as a key frame image and the standard key feature of the same type in the standard key feature library corresponding to the surgical type of this surgery, so as to obtain the matching degree between each key feature and the standard key feature of the same type; The feature content evaluation function of the frame image that is not marked as a key frame image is calculated according to the following formula: ; in, is the feature content evaluation function of the frame image not marked as a key frame image, m is the number of key features contained in the frame image not marked as a key frame image, is the matching degree corresponding to the i-th key feature contained in the frame image that is not marked as a key frame image, is the weight factor corresponding to the ith key feature contained in the frame image that is not marked as a key frame image (the weight factor corresponding to each key feature can be set according to actual needs). Therefore, the larger the feature content evaluation function is, the more key features are contained in the corresponding frame image and the more important the key features are, the more it should be used as a key frame image.

[0050] Among them, the matching degree between the key features and the standard key features can be calculated in the following way: Scaling the key feature image (i.e., the image of the key feature) so that the pixel size of the scaled key feature image is the same as that of the corresponding standard key feature image (i.e., the image of the corresponding standard key feature); The scaled key feature image and the standard key feature image are converted into grayscale images (the method of converting into grayscale images is a prior art and will not be described in detail here), to obtain a first grayscale image and a second grayscale image; The pixel values ​​of the first grayscale image and the second grayscale image are normalized respectively, and the specific calculation formula is: ; ; ; ; in, is the pixel value of the (i, j) pixel (i.e., the pixel at the i-th row and j-th column) of the first grayscale image after normalization. is the pixel value of the (i, j) pixel of the first grayscale image before normalization, is the maximum pixel value of the pixels of the first grayscale image before normalization, is the pixel value of the (i, j) pixel of the second grayscale image after normalization, is the pixel value of the pixel (i, j) of the second grayscale image before normalization, is the maximum pixel value of the pixel points of the second grayscale image before normalization, M is the horizontal coordinate pixel size of the standard key feature image, and N is the vertical coordinate pixel size of the standard key feature image; Using windows of the same size and the same sliding window step size, sliding window sampling is performed on the first grayscale image after normalization and the second grayscale image after normalization to obtain respective multiple window images (i.e., images intercepted by the window each time); The matching coefficients between each window image of the first grayscale image after normalization and each window image of the second grayscale image after normalization are calculated according to the following formula: ; ; ; in, is the matching coefficient between the kth window image of the normalized first grayscale image and the qth window image of the normalized second grayscale image, K is the total number of window images of the normalized first grayscale image (which is also equal to the total number of window images of the normalized second grayscale image), is the pixel value of the (i, j) pixel of the kth window image of the first grayscale image after normalization, is the pixel value of the pixel point (i, j) of the q-th window image of the second grayscale image after normalization processing, o is the horizontal pixel size of the window, and p is the vertical pixel size of the window; Calculate the effective matching coefficients corresponding to the respective window images of the first grayscale image after normalization processing according to the following formula: ; where is the effective matching coefficient corresponding to the k-th window image of the first grayscale image after normalization processing, represents the maximum value of the matching coefficients between the k-th window image of the first grayscale image after normalization processing and the respective window images of the second grayscale image; Perform an ascending order sorting on the effective matching coefficients, delete the effective matching coefficients before the Q-th one after sorting, and calculate the average value of the remaining effective matching coefficients as the matching degree between the key feature and the standard key feature; where Q is a preset positive integer (Q < K), and can be specifically set according to actual needs.

[0051] Actually, the calculation method of the matching degree between the key feature and the standard key feature is not limited to the above method. For example, existing technologies such as the structural similarity (SSIM) index, normalized gray-scale matching, and feature-based matching algorithms can also be used to calculate the matching degree between the key feature and the standard key feature, but not limited to this.

[0052] In some embodiments, determine whether each frame image not marked as a key frame image is a key frame image according to the feature content evaluation function, specifically including: determining the frame images with the feature content evaluation function exceeding a preset evaluation function threshold (which can be set according to actual needs) as key frame images.

[0053] In some other embodiments, determine whether each frame image not marked as a key frame image is a key frame image according to the feature content evaluation function, specifically including: Perform a descending order sorting on each frame image not marked as a key frame image according to the size of the feature content evaluation function; According to the sorting result, determine the frame images before the N-th one after sorting as key frame images, where N is a preset positive integer (which can be set according to actual needs).

[0054] Thus, the number of key frame images screened is determined by the preset N, which can avoid having too many key frame images, resulting in the user having to select the target key frame image from a large number of key frame images when generating a surgical text and image report, and is beneficial to improving the efficiency of the user's selection of the target key frame image; in addition, it can also avoid having too few key frame images, resulting in the absence of important images, and improve the probability that the user can select the most needed frame images.

[0055] Furthermore, after completing the construction of the key frame image set, the similarity between the key frame images in the key frame image set can also be calculated. For key frame images whose similarity deviation is within a preset tolerance range (which can be set according to actual needs), only one of the key frame images is retained (for example, the one with the largest feature content evaluation function is retained); since the image information contained in the key frame images whose similarity deviation is within the preset tolerance range is basically equivalent, retaining only one of them can also avoid the loss of important images, thereby ensuring that no important images are lost while reducing the number of key frame images and improving the efficiency of users in selecting target key frame images.

[0056] Specifically, see Figure 1 , the report generation module 203 includes a report picture generation module 204 and a report document generation module 205; The report image generation module 204 is used to assist the user in selecting a key frame image (i.e., selecting a target key frame image), and to annotate the selected key frame image to generate a report image; The report document generation module 205 is used to obtain the corresponding report template according to the basic information of the patient, and after calling the report image generation module 204 to perform key frame image selection and annotation processing, the generated report image is inserted into the corresponding position of the report template.

[0057] Among them, the report template includes several report image insertion positions, and the report generation system 2 also includes a human-computer interaction device. The user can edit the obtained report template through the human-computer interaction device, including clicking the report image insertion position. When the report image insertion position is clicked, the report image generation module 204 generates a key frame image display interface for the user to select the target key frame image. After the user selects the target key frame image, an image editing interface is generated and the target key frame image is displayed in the image editing interface. The image editing interface provides a marking function for marking the target key frame image (for example, at least one marking function such as circling, captioning, and local magnification). After the user completes the marking of the target key frame image, the marked target key frame image is inserted as a report image at the clicked report image insertion position. Thereby, it can assist the user to quickly complete the selection, marking and insertion operations of each target key frame image, and improve the generation efficiency of the surgical graphic report.

[0058] Among them, the surgical graphic report includes part or all of the patient's basic information, graphic process information, pathological biopsy information, postoperative diagnosis information, and doctor's advice information.

[0059] The graphic and text flow information includes text surgical flow information and corresponding report pictures. Therefore, generally, the surgical graphic and text report needs to include graphic and text flow information.

[0060] The pathological biopsy information includes at least one of the following: the macroscopic appearance of the biopsy tissue (including the overall appearance of the biopsy tissue, the biopsy site, the shape of the suspected cancer, whether the edge is clear, the size of the biopsy tissue, and part or all of the weight), the microscopic observation information (including the image under the microscope and the description of the biopsy tissue at the molecular level), and the professional diagnosis information given by the pathology department. The pathological biopsy information can be obtained from the medical information system 90 and inserted into the report template.

[0061] The postoperative diagnosis information includes at least one of the following: diagnosis information of diseases observable during surgery, information on basic pathological processes (such as the occurrence of hypoxia, shock, etc.) caused by the surgery, and information on surgical complications. The postoperative diagnosis information can be obtained from the medical information system 90 and inserted into the report template.

[0062] The doctor's advice information includes at least one of postoperative recovery advice (including prevention and treatment of complications, dietary recommendations and dietary taboos, exercise methods and frequencies, etc.), medication guidance, and reexamination advice. The doctor's advice information can be obtained from the medical information system 90 and inserted into the report template.

[0063] For example Figure 3 In the figure, an exemplary surgical graphic report is shown, which includes basic patient information, graphic process information, postoperative diagnosis information and doctor's advice information. In the figure, Part 1 is the basic patient information, Part 2 is the report picture in the graphic process information, Part 3 is the text surgical process information in the graphic process information, Part 4 is the postoperative diagnosis information, and Part 5 is the doctor's advice information.

[0064] In some possible implementations, the report generation system 2 further includes a printing transmission module 206, which is used to send the generated surgical graphic report to the medical information system 90, so that the medical information system 90 prints and stores the surgical graphic report.

[0065] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0066] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A minimally invasive laparoscopic graphic report generation system, characterized in that: It comprises an image acquisition device (1) and a report generation system (2), wherein the report generation system (2) comprises a patient information access module (201), a key frame image screening module (202) and a report generation module (203), and the image acquisition device (1) and the report generation system (2) are communicatively connected; The image acquisition device (1) is used to acquire surgical process images through a minimally invasive laparoscope (102) and send the images to the report generation system (2); The patient information access module (201) is used to obtain basic patient information from the hospital's medical information system (90); The key frame image screening module (202) is used to screen key frame images from the surgical process images according to the basic information of the patient; The report generation module (203) is used to generate a surgical graphic report based on the patient's basic information and the key frame images.

2. The minimally invasive laparoscopic graphic report generation system according to claim 1, characterized in that: The image acquisition device (1) comprises a central control processing module (101), a minimally invasive laparoscope (102), a memory (103), a wireless communication module (104) and a command trigger device (105); the minimally invasive laparoscope (102), the memory (103), the wireless communication module (104) and the command trigger device (105) are all electrically connected to the central control processing module (101); The minimally invasive laparoscope (102) is used to collect images of the surgical process; The instruction triggering device (105) is used to generate a prompt instruction under the operation of the surgeon; The central control processing module (101) is used to record the surgical process image in the memory (103), and when receiving the prompt instruction, mark the surgical process image with key frames, and send the surgical process image to the report generation system (2) via the wireless communication module (104).

3. The minimally invasive laparoscopic graphic report generation system according to claim 2, characterized in that: The central control processing module (101) is also used to adjust the image acquisition frame rate of the minimally invasive laparoscope (102) at different surgical stages according to the basic information of the patient, wherein the surgical stages include non-important surgical stages and important surgical stages, so that the image acquisition frame rate of the non-important surgical stage is lower than the image acquisition frame rate of the important surgical stage.

4. The minimally invasive laparoscopic graphic report generation system according to claim 3, characterized in that: When the central control processing module (101) adjusts the image acquisition frame rate of the minimally invasive laparoscope (102) at different surgical stages according to the basic information of the patient, the central control processing module (101) executes: Acquire a moving distance cutoff value according to the basic information of the patient; Acquiring the real-time moving distance of the minimally invasive laparoscope (102); Comparing the real-time moving distance with the moving distance cutoff value to determine the category of the current surgical stage; If the current surgical stage is a non-important surgical stage, the minimally invasive laparoscope (102) is controlled to capture surgical process images at a first preset frame rate; if the current surgical stage is an important surgical stage, the minimally invasive laparoscope (102) is controlled to capture surgical process images at a second preset frame rate; the first preset frame rate is less than the second preset frame rate.

5. The minimally invasive laparoscopic graphic report generation system according to claim 2, characterized in that: When recording the surgical process image in the memory (103), the central control processing module (101) executes: When the remaining storage space of the memory (103) is lower than a preset storage space threshold, estimating the data size of the surgical process image of the current operation based on the basic information of the patient; Determining the historical surgical process images to be deleted and the frame images to be deleted based on the data size of the key frame images marked in the historical surgical process images in the memory (103) and the estimated data size of the surgical process images of the current surgery; When the memory (103) is full, each frame image to be deleted of each historical surgical process image to be deleted is deleted in the order of recording time, so as to reserve storage space for storing surgical process images collected in real time.

6. The minimally invasive laparoscopic graphic report generation system according to claim 2, characterized in that: When the key frame image screening module (202) screens the key frame image from the surgical process image according to the basic information of the patient, the key frame image screening module (202) performs: Establishing a key frame image set, and adding the frame images marked as key frame images in the surgical process image to the key frame image set; Using a pre-trained image feature recognition model, key features are recognized for each frame image in the surgical process image that is not marked as a key frame image, to obtain key features contained in the corresponding frame image; Calculating a feature content evaluation function of each frame image not marked as a key frame image according to key features contained in each frame image not marked as a key frame image; Whether each frame image not marked as a key frame image is a key frame image is determined according to the feature content evaluation function, and the frame images determined to be key frame images are added to the key frame image set.

7. The minimally invasive laparoscopic graphic report generation system according to claim 6, characterized in that: According to the key features contained in each frame image not marked as a key frame image, the feature content evaluation function of each frame image not marked as a key frame image is calculated, specifically including: Calculate the matching degree between each key feature contained in the frame image that is not marked as a key frame image and the standard key feature of the same type; The characteristic content evaluation function of the frame image not marked as a key frame image is calculated according to the following formula: ； in, is the feature content evaluation function of the frame image not marked as a key frame image, m is the number of key features contained in the frame image not marked as a key frame image, is the matching degree corresponding to the i-th key feature contained in the frame image not marked as a key frame image, is the weight factor corresponding to the i-th key feature contained in the frame image that is not marked as a key frame image.

8. The minimally invasive laparoscopic graphic report generation system according to claim 7, characterized in that: Determining whether each frame image not marked as a key frame image is a key frame image according to the feature content evaluation function specifically includes: According to the magnitude of the characteristic content evaluation function, each frame image that is not marked as a key frame image is sorted in descending order; According to the sorting result, the first N frame images in the sorting are determined as key frame images, where N is a preset positive integer.

9. The minimally invasive laparoscopic graphic report generation system according to claim 1, characterized in that: The report generation module (203) includes a report picture generation module (204) and a report document generation module (205); The report image generation module (204) is used to assist the user in selecting the key frame image, and to perform annotation processing on the selected key frame image to generate a report image; The report document generation module (205) is used to obtain the corresponding report template according to the basic information of the patient, and after calling the report image generation module (204) to perform key frame image selection and annotation processing, the generated report image is inserted into the corresponding position in the report template.

10. The minimally invasive laparoscopic graphic report generation system according to claim 1, characterized in that: The surgical graphic report includes part or all of the patient's basic information, graphic process information, pathological biopsy information, postoperative diagnosis information, and doctor's advice information.

Images