A minimally invasive endoscopic graphic report generation system

Abstract

This application belongs to the technical field of medical devices and discloses a minimally invasive endoscopic graphic report generation system, which includes an image acquisition device and a report generation system. The report generation system includes a patient information access module, a key frame image screening module, and a report generation module. The image acquisition device and the report generation system are communicatively connected. The image acquisition device is used to collect surgical process images through a minimally invasive endoscope and send them to the report generation system. The patient information access module is used to obtain the basic patient information from the hospital's medical information system. The key frame image screening module is used to screen key frame images from the surgical process images according to the basic patient information. The report generation module is used to generate a surgical graphic report according to the basic patient information and the key frame images, so as to assist users in quickly generating graphic reports for minimally invasive surgeries, thereby improving the generation efficiency of graphic reports for minimally invasive surgeries.

Description

Technical Field

[0001] This application relates to the technical field of medical devices, and more particularly, to a minimally invasive endoscopic graphic report generation system. Background Art

[0002] The graphic report of minimally invasive surgery can directly reflect the degree of the patient's lesion and the intraoperative treatment situation, provide a diagnostic basis for the patient's subsequent treatment, and assist in patient care; it can also serve as the main evidence for doctor-patient communication after surgery, reduce doctor-patient conflicts, and reduce the duplication and waste of medical resources. In addition, the graphic report of minimally invasive surgery can more conveniently spread the theoretical knowledge and rich clinical experience of professional doctors to other young doctors, which is beneficial to strengthening the training effect of medical students.

[0003] However, currently, the graphic reports of minimally invasive surgeries are generally generated manually. When generating a graphic report, doctors need to manually select key frame images from the videos taken during the surgery for insertion into the graphic report. The screening process takes a lot of time, which not only increases the workload of doctors but also results in low generation efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a minimally invasive endoscopic graphic report generation system, which can assist users in quickly generating graphic reports of minimally invasive surgeries, thereby improving the generation efficiency of graphic reports of minimally invasive surgeries.

[0005] This application provides a minimally invasive endoscopic graphic report generation system, including an image acquisition device and a report generation system. The report generation system includes a patient information access module, a key frame image screening module, and a report generation module. The image acquisition device and the report generation system are communicatively connected;

[0006] The image acquisition device is used to acquire intraoperative images through a minimally invasive endoscope and send them to the report generation system;

[0007] The patient information access module is used to obtain the basic information of the patient from the hospital's medical information system;

[0008] The key frame image screening module is used to screen key frame images from the intraoperative images according to the basic information of the patient;

[0009] The report generation module is used to generate a surgical graphic report according to the basic information of the patient and the key frame images.

[0010] This minimally invasive endoscopic graphic report generation system can automatically screen key frame images, reduce the time-consuming for users to screen key frame images from intraoperative images, thereby assisting users in quickly generating graphic reports of minimally invasive surgeries, and can improve the generation efficiency of graphic reports of minimally invasive surgeries.

[0011] Preferably, the image acquisition device includes a central control and processing module, a minimally invasive endoscope, a memory, a wireless communication module, and an instruction trigger device. The minimally invasive endoscope, the memory, the wireless communication module, and the instruction trigger device are all electrically connected to the central control and processing module;

[0012] The minimally invasive endoscope is used to collect images of the surgical process;

[0013] The instruction trigger device is used to generate a prompt instruction under the operation of the surgeon;

[0014] The central control and processing module is used to record the surgical process images in the memory, and when receiving the prompt instruction, mark key frames of the surgical process images, and send the surgical process images to the report generation system through the wireless communication module.

[0015] During the surgical process, the surgeon can issue a prompt instruction through the instruction trigger device at a critical moment according to his own experience, so as to mark the frame image at the corresponding moment as a key frame image. The marked key frame images usually contain important information, which can complement the key frame images automatically screened by the system and reduce the probability of missing frame images containing important information.

[0016] Preferably, the central control and processing module is further used to adjust the image acquisition frame rate of the minimally invasive endoscope at different surgical stages according to the basic patient information. The surgical stages include unimportant surgical stages and important surgical stages, so that the image acquisition frame rate in the unimportant surgical stage is lower than that in the important surgical stage.

[0017] Since the storage space of the memory of the image acquisition device is limited, reducing the image acquisition frame rate in the unimportant surgical stage can save storage space for storing more useful information.

[0018] Preferably, when the central control and processing module adjusts the image acquisition frame rate of the minimally invasive endoscope at different surgical stages according to the basic patient information, it executes:

[0019] Obtain the moving distance demarcation value according to the basic patient information;

[0020] Obtain the real-time moving distance of the minimally invasive endoscope;

[0021] Compare the real-time moving distance with the moving distance demarcation value to determine the category of the current surgical stage;

[0022] If the current surgical stage is a non-important surgical stage, the minimally invasive laparoscope is controlled to capture the surgical process images at a first preset frame rate. If the current surgical stage is an important surgical stage, the minimally invasive laparoscope is controlled to capture the surgical process images at a second preset frame rate. The first preset frame rate is less than the second preset frame rate.

[0023] Preferably, when recording the surgical process image in the memory, the central control processing module executes:

[0024] When the remaining storage space of the memory is lower than a preset storage space threshold, estimating the data size of the surgical process image of this operation according to the basic information of the patient;

[0025] Determining the historical surgical process images to be deleted and the frame images to be deleted according to the data size of the key frame images marked in the historical surgical process images in the memory and the estimated data size of the surgical process images of the current surgery;

[0026] When the memory is full, each frame image to be deleted of each historical surgical process image to be deleted is deleted in the order of recording time, so as to reserve storage space for storing the surgical process images collected in real time.

[0027] Preferably, when the key frame image screening module screens the key frame images from the surgical process images according to the basic information of the patient, the key frame image screening module performs:

[0028] Establishing a key frame image set, and adding the frame images marked as key frame images in the surgical process image to the key frame image set;

[0029] Using a pre-trained image feature recognition model, key features are recognized for each frame image in the surgical process image that is not marked as a key frame image, to obtain key features contained in the corresponding frame image;

[0030] Calculating a feature content evaluation function of each frame image not marked as a key frame image according to key features contained in each frame image not marked as a key frame image;

[0031] Whether each frame image not marked as a key frame image is a key frame image is determined according to the feature content evaluation function, and the frame images determined to be key frame images are added to the key frame image set.

[0032] Preferably, according to the key features contained in each frame image not marked as a key frame image, calculating the feature content evaluation function of each frame image not marked as a key frame image specifically includes:

[0033] Calculate the matching degree between each key feature included in the frame image not marked as a key frame image and the standard key feature of the same type;

[0034] Calculate the feature content evaluation function of the frame image not marked as a key frame image according to the following formula:

[0035] ;

[0036] where, is the feature content evaluation function of the frame image not marked as a key frame image, m is the number of key features included in the frame image not marked as a key frame image, is the matching degree corresponding to the i-th key feature included in the frame image not marked as a key frame image, is the weight factor corresponding to the i-th key feature included in the frame image not marked as a key frame image.

[0037] Preferably, determine whether each frame image not marked as a key frame image is a key frame image according to the feature content evaluation function, specifically including:

[0038] Sort each frame image not marked as a key frame image in descending order according to the size of the feature content evaluation function;

[0039] According to the sorting result, determine the first N frame images as key frame images, where N is a preset positive integer.

[0040] Preferably, the report generation module includes a report picture generation module and a report document generation module;

[0041] The report picture generation module is used to assist the user in selecting the key frame images, and perform annotation processing on the selected key frame images to generate report pictures;

[0042] The report document generation module is used to obtain the corresponding report template according to the patient's basic information, and after calling the report picture generation module to perform the selection operation and annotation processing of the key frame images, insert the generated report pictures into the corresponding positions in the report template.

[0043] Preferably, the surgical graphic report includes some or all of the patient's basic information, graphic process information, pathological biopsy information, postoperative diagnosis information, and doctor's advice information.

[0044] Beneficial effects: The minimally invasive laparoscopic graphic report generation system provided by this application can automatically screen key frame images, reduce the time-consuming for users to screen key frame images from surgical process images, thereby assisting users in quickly generating graphic reports for minimally invasive surgeries, and improving the generation efficiency of graphic reports for minimally invasive surgeries. Brief Description of the Drawings

[0045] Figure 1 FIG. is a schematic structural diagram of a minimally invasive endoscopic graphic report generation system provided by an embodiment of the present application.

[0046] Figure 2 FIG. is a schematic structural diagram of an image acquisition device.

[0047] Figure 3 FIG. is a schematic diagram of an exemplary surgical graphic report.

[0048] Reference Numeral Description: 1. Image acquisition device; 101. Central control processing module; 102. Minimally invasive endoscope; 103. Memory; 104. Wireless communication module; 105. Instruction trigger device; 2. Report generation system; 201. Patient information access module; 202. Key frame image screening module; 203. Report generation module; 204. Report picture generation module; 205. Report document generation module; 206. Printing and transmission module; 90. Medical information system. Detailed Embodiments

[0049] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0050] It should be noted that: similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0051] Please refer to Figure 1 、 Figure 2 , a minimally invasive endoscopic graphic report generation system in some embodiments of the present application includes an image acquisition device 1 and a report generation system 2. The report generation system 2 includes a patient information access module 201, a key frame image screening module 202, and a report generation module 203. The image acquisition device 1 and the report generation system 2 are communicatively connected;

[0052] The image acquisition device 1 is used to acquire surgical process images through the minimally invasive endoscope 102 and send them to the report generation system 2;

[0053] The patient information access module 201 is used to obtain the basic patient information from the hospital's medical information system 90;

[0054] The key frame image screening module 202 is used to screen key frame images from the surgical process images according to the basic patient information;

[0055] The report generation module 203 is used to generate a surgical illustrated report according to the basic patient information and the key frame images.

[0056] This minimally invasive endoscope illustrated report generation system can automatically screen key frame images, reduce the time-consuming for users to screen key frame images from the surgical process images, thereby assisting users to quickly generate an illustrated report for minimally invasive surgery, and can improve the generation efficiency of the illustrated report for minimally invasive surgery.

[0057] Among them, the basic patient information includes some or all of the patient's name, gender, age, height and weight, department, surgeon, type of surgery, type of charge, patient's past medical history, clinical manifestations, diagnosis and treatment conditions, and necessary materials related to disease diagnosis or surgery (such as at least one of physical examination results, imaging pictures, pathological pictures, various biochemical test results, etc.), but not limited to this. All these information are recorded in the hospital's medical information system 90, and these information can be retrieved from the medical information system 90 through the patient information access module 201.

[0058] Specifically, see Figure 2 , the image acquisition device 1 includes a central control processing module 101, a minimally invasive endoscope 102, a memory 103, a wireless communication module 104 and an instruction trigger device 105. The minimally invasive endoscope 102, the memory 103, the wireless communication module 104 and the instruction trigger device 105 are all electrically connected to the central control processing module 101;

[0059] The minimally invasive endoscope 102 is used to acquire surgical process images;

[0060] The instruction trigger device 105 is used to generate a prompt instruction under the operation of the surgeon;

[0061] The central control processing module 101 is used to record the surgical process images in the memory 103, and when receiving the prompt instruction, mark the key frames of the surgical process images, and send the surgical process images to the report generation system 2 through the wireless communication module 104 (thus, a corresponding communication module is set on the report generation system 2 for communicating with the wireless communication module 104).

[0062] During the operation, the surgeon can, based on their own experience, trigger a prompt instruction through the instruction trigger device 105 at a critical moment, thereby marking the frame image at the corresponding moment as a key frame image. The marked key frame images usually contain important information, which can complement the key frame images automatically selected by the system and reduce the probability of missing frame images containing important information.

[0063] Among them, the minimally invasive endoscope 102 and the memory 103 are prior arts and will not be elaborated here. The central control processing module 101 can adopt an MCU module, but is not limited thereto. The wireless communication module 104 can adopt a WIFI module, a 4G module, a 5G module, a Bluetooth module, a zigbee module, etc., but is not limited thereto. The instruction trigger device 105 can be, but is not limited to, a button module (such as a hand-pressed button module, a foot-pressed button module, etc.) or a voice recognition module; if the instruction trigger device 105 is a button module, the prompt instruction is a button trigger instruction, and if the instruction trigger device 105 is a voice recognition module, the prompt instruction is a voice instruction.

[0064] Among them, when the minimally invasive endoscope 102 collects the surgical process images, it can collect them at a constant image acquisition frame rate, or adjust the image acquisition frame rate at different surgical stages according to actual needs.

[0065] For example, in some embodiments, the central control processing module 101 is further configured to adjust the image acquisition frame rate of the minimally invasive endoscope 102 at different surgical stages according to the patient's basic information. The surgical stages include non-important surgical stages and important surgical stages, and the image acquisition frame rate in the non-important surgical stages is lower than that in the important surgical stages. Among them, the important surgical stage refers to the stage with a relatively high correlation with the operation, and the non-important surgical stage refers to the stage with a relatively low correlation with the operation. Generally, the important surgical stage includes the process from when the minimally invasive endoscope 102 reaches the surgical target site (for example, when performing a gastric surgery, the surgical target site is the stomach) to the completion of the surgical treatment of the surgical target site, and the non-important surgical stage includes the moving process before the minimally invasive endoscope 102 reaches the surgical target site, but is not limited thereto, and can be specifically set according to actual needs. Since the storage space of the memory 103 of the image acquisition device 1 is limited, reducing the image acquisition frame rate in the non-important surgical stages can save storage space for storing more useful information.

[0066] Among them, the operating surgeon can trigger the image acquisition frame rate switching through the instruction trigger device 105. Thus, the instruction trigger device 105 is also used to generate a switching instruction under the operation of the operating surgeon, so that the image acquisition frame rate of the minimally invasive endoscope 102 can be switched between a first preset frame rate and a second preset frame rate, and the first preset frame rate is less than the second preset frame rate. After the minimally invasive endoscope 102 is started, the first preset frame rate is used as the default frame rate for image acquisition during the surgical process. When the central control processing module 101 receives the switching instruction, it controls the minimally invasive endoscope 102 to switch the image acquisition frame rate to the second preset frame rate. After receiving the switching instruction again, it controls the minimally invasive endoscope 102 to switch the image acquisition frame rate back to the first preset frame rate (hereinafter, this image acquisition frame rate adjustment mode is called the manual adjustment mode). The operating surgeon can switch independently according to actual needs. If the instruction trigger device 105 is a button module, the switching instruction is a button trigger instruction. If the instruction trigger device 105 is a voice recognition module, the switching instruction is a voice instruction.

[0067] The image acquisition frame rate can also be automatically switched by the system to avoid the operating surgeon being distracted to perform manual switching. Thus, in some embodiments, when the central control processing module 101 adjusts the image acquisition frame rate of the minimally invasive endoscope 102 at different surgical stages according to the patient's basic information, it performs (hereinafter, this image acquisition frame rate adjustment mode is called the automatic adjustment mode):

[0068] Obtain the moving distance demarcation value according to the patient's basic information;

[0069] Obtain the real-time moving distance of the minimally invasive endoscope 102;

[0070] Compare the real-time moving distance with the moving distance demarcation value to determine the category of the current surgical stage;

[0071] If the current surgical stage is a non-critical surgical stage, control the minimally invasive endoscope 102 to perform image shooting during the surgical process at the first preset frame rate. If the current surgical stage is a critical surgical stage, control the minimally invasive endoscope 102 to perform image shooting during the surgical process at the second preset frame rate.

[0072] Through the above automatic adjustment method, the operating surgeon does not need to manually switch the image acquisition frame rate, so that he can concentrate more on the operation.

[0073] When the automatic adjustment mode is adopted, the basic patient information obtained needs to include the patient's gender, age, height and weight, and the type of surgery. The moving distance threshold value is obtained according to the basic patient information, specifically including: determining the surgical target site according to the type of surgery, and estimating the total moving distance required for the minimally invasive endoscope 102 to reach the surgical target site according to the patient's gender, age, height and weight (the total moving distance is the path distance from the body entry point to the surgical target site. For example, when the surgical target site is the stomach, estimate the total moving distance required to move from the patient's oral cavity through the esophagus to the stomach; it can be estimated through a pre-trained deep learning model, but not limited to this), and calculating the moving distance threshold value according to the total moving distance (for example, multiplying the total moving distance by a preset proportionality coefficient less than 1 to obtain the moving distance threshold value, and the preset proportionality coefficient can be taken as 0.8, but not limited to this).

[0074] Among them, an acceleration sensor can be set in the minimally invasive endoscope 102 to obtain the real-time acceleration of the minimally invasive endoscope 102, and the real-time moving distance of the minimally invasive endoscope 102 can be obtained by performing double integration on the real-time acceleration.

[0075] When the real-time moving distance is less than the moving distance threshold value, it is determined that the current surgical stage is a non-important surgical stage. When the real-time moving distance is greater than or equal to the moving distance threshold value, it is determined that the current surgical stage is an important surgical stage.

[0076] Among them, the surgeon can select the image acquisition frame rate adjustment mode before the operation. For example, a human-computer interaction module can be set on the image acquisition device 1, and the surgeon selects the image acquisition frame rate adjustment mode through the human-computer interaction module; if the surgeon selects the manual adjustment mode, during the operation, the system will not automatically adjust the image acquisition frame rate. If the surgeon selects the automatic adjustment mode, the system will automatically adjust the image acquisition frame rate. In fact, for some types of surgeries, the total moving distance required for the minimally invasive endoscope 102 to reach the surgical target site is relatively small, and it is not suitable to use the above automatic adjustment mode. Therefore, in some embodiments, the central control processing module 101 is further configured to, when the surgeon selects the automatic adjustment mode before the operation, determine whether the current operation is applicable to the automatic adjustment mode according to the type of surgery. If not, change the selection result to the manual adjustment mode and send a notification message to inform the surgeon of the change result.

[0077] In some preferred embodiments, when the central control processing module 101 records the surgical process images in the memory 103, it performs:

[0078] When the remaining storage space in the memory 103 is lower than the preset storage space threshold value (which can be set according to actual needs), estimate the data size of the surgical process images of this operation according to the basic patient information;

[0079] Determine the historical surgical process images to be deleted and their frame images to be deleted according to the data size of the key frame images marked in the historical surgical process images (i.e., the surgical process images of the previously completed surgery) in the memory 103 and the estimated data size of the surgical process images of the current surgery;

[0080] When the memory 103 is full, each frame image of each historical surgical process image to be deleted is deleted in the order of recording time, so as to reserve storage space for storing the surgical process images collected in real time.

[0081] The data size of the surgical process images of this operation is estimated based on the basic information of the patient, and the data size of the surgical process images of this operation is estimated specifically based on the type of operation (which can be estimated through big data statistics).

[0082] According to the data size of the key frame images marked in the historical surgical process images in the memory 103 and the estimated data size of the surgical process images of the current surgery, the historical surgical process images to be deleted and the frame images to be deleted are determined, specifically including:

[0083] The estimated data size of the surgical process image of the current surgery is multiplied by a preset redundancy coefficient (the preset redundancy coefficient is greater than 1 and can be set according to actual needs, such as 1.2, but not limited thereto) to obtain a first data size;

[0084] Calculate the sum of the data sizes of the key frame images marked in each historical surgical process image as the key data size of the corresponding historical surgical process image;

[0085] Subtract the corresponding key data size from the total data size of each historical surgical process image to obtain the non-key data size of the corresponding historical surgical process image;

[0086] If the sum of the non-critical data sizes of all historical surgical process images is not less than the first data size, then, according to the order of the recording time of each historical surgical process image, each historical surgical process image is sequentially determined as a historical surgical process image to be deleted, until the sum of the non-critical data sizes of all historical surgical process images to be deleted is not less than the first data size; and unmarked frame images (i.e., frame images not marked as key frame images) in each historical surgical process image to be deleted are determined as frame images to be deleted;

[0087] If the sum of the sizes of non-critical data of all historical surgical procedure images is less than the first data size, then all historical surgical procedure images are determined as historical surgical procedure images to be deleted. The unmarked frame images in each historical surgical procedure image to be deleted are used as frame images to be deleted. According to the chronological order of the recording times of each historical surgical procedure image to be deleted, the marked key frame images in each historical surgical procedure image to be deleted are sequentially added as frame images to be deleted corresponding to the historical surgical procedure image to be deleted until the sum of the data sizes of all frame images to be deleted is not less than the first data size.

[0088] Thus, when the memory 103 is full, the frame images to be deleted in each historical surgical procedure image to be deleted are deleted in chronological order of the recording time to free up storage space for storing the surgically acquired real-time images. That is, whenever a new image is acquired, one frame image to be deleted is deleted in a first-in, first-out manner to free up storage space for storing the new image. By controlling the storage of surgical procedure images in the above manner, while completely preserving the surgical procedure images of the current surgery, the key information in the historical surgical procedure images can be retained as much as possible for the user to review.

[0089] In some possible implementation manners, when the central control processing module 101 receives a prompt instruction, it marks the key frames of the surgical procedure images, and specifically executes: marking the frame images acquired in a preset time period (which can be set according to actual needs, for example, 2s, but not limited to this) with the moment when the prompt instruction is received as the midpoint as key frame images.

[0090] After completing the key frame marking, the surgical procedure images are sent to the report generation system 2, so that the surgical procedure images received by the report generation system 2 contain the marking information of the key frame images.

[0091] In this embodiment, when the key frame image screening module 202 screens key frame images from the surgical procedure images according to the basic patient information, it executes:

[0092] Establishing a key frame image set, and adding the frame images marked as key frame images in the surgical procedure images to the key frame image set;

[0093] Using a pre-trained image feature recognition model to perform key feature recognition on each frame image in the surgical procedure images that is not marked as a key frame image to obtain the key features included in the corresponding frame image;

[0094] Calculating the feature content evaluation function of each frame image that is not marked as a key frame image according to the key features included in each frame image that is not marked as a key frame image;

[0095] Determine whether each frame image that is not marked as a key frame image is a key frame image according to the feature content evaluation function, and add the frame images determined to be key frame images to the key frame image set.

[0096] Generally, there are multiple key features for each type of surgery (such as lesion features of the surgical target site, surgical wound surface features of the surgical target site, local shape features of the surgical target site, etc., but not limited to these). The key features of different types of surgery may be partially the same or completely different.

[0097] Among them, for each type of surgery, the corresponding surgical images can be collected, key feature annotation is performed to form a surgical image data set, and then the image feature recognition models are trained respectively with the surgical image data sets of various types of surgery to obtain the pre-trained image feature recognition models corresponding to various types of surgery. Thus, the pre-trained image feature recognition models are used to perform key feature recognition on each frame image in the surgical process image that is not marked as a key frame image, specifically including: calling the corresponding pre-trained image feature recognition model according to the surgical type of this surgery, and using the called pre-trained image feature recognition model to perform key feature recognition on each frame image in the surgical process image that is not marked as a key frame image.

[0098] Furthermore, according to the key features included in each frame image that is not marked as a key frame image, calculate the feature content evaluation function of each frame image that is not marked as a key frame image, specifically including:

[0099] Calculate the matching degree between each key feature included in the frame image that is not marked as a key frame image and the standard key feature of the same type; specifically, for each key feature, the corresponding standard key feature is pre-recorded in advance to form a standard key feature library for each type of surgery. In practical applications, match each key feature included in the frame image that is not marked as a key frame image with the standard key feature of the same type in the standard key feature library corresponding to the surgical type of this surgery, so as to obtain the matching degree between each key feature and the standard key feature of the same type.

[0100] Calculate the feature content evaluation function of the frame image that is not marked as a key frame image according to the following formula:

[0101] ;

[0102] Among them, is the feature content evaluation function of the frame image that is not marked as a key frame image, m is the number of key features included in the frame image that is not marked as a key frame image, is the matching degree corresponding to the i-th key feature included in the frame image that is not marked as a key frame image, It is the weight factor corresponding to the i-th key feature included in the frame image that is not marked as a key frame image (the weight factor corresponding to each type of key feature can be set according to actual needs). Thus, the larger the feature content evaluation function, the more key features are included in the corresponding frame image and the more important the included key features are, and it is more suitable to be used as a key frame image.

[0103] Among them, the matching degree between the key feature and the standard key feature can be calculated in the following way:

[0104] Perform a scaling process on the key feature image (i.e., the image of the key feature) so that the pixel size of the scaled key feature image is the same as that of the corresponding standard key feature image (i.e., the image of the corresponding standard key feature);

[0105] Convert the scaled key feature image and the standard key feature image into grayscale images (the method of converting to grayscale images is a prior art and will not be elaborated here), obtaining a first grayscale image and a second grayscale image;

[0106] Normalize the pixel values of the first grayscale image and the second grayscale image respectively. The specific calculation formula is:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] Among them, is the pixel value of the (i, j) pixel point (i.e., the pixel point in the i-th row and j-th column) of the first grayscale image after normalization, is the pixel value of the (i, j) pixel point of the first grayscale image before normalization, is the maximum pixel value of the pixel points of the first grayscale image before normalization, is the pixel value of the (i, j) pixel point of the second grayscale image after normalization, is the pixel value of the (i, j) pixel point of the second grayscale image before normalization, is the maximum pixel value of the pixel points of the second grayscale image before normalization, M is the horizontal pixel size of the standard key feature image, and N is the vertical pixel size of the standard key feature image;

[0112] Use windows of the same size and the same sliding window step length to perform sliding window sampling on the first grayscale image after normalization and the second grayscale image after normalization respectively, obtaining multiple window images (i.e., the images intercepted by the window each time);

[0113] Calculate the matching coefficient between each window image of the first grayscale image after normalization and each window image of the second grayscale image after normalization according to the following formula:

[0114] ;

[0115] ;

[0116] ;

[0117] where, is the matching coefficient between the k-th window image of the first grayscale image after normalization and the q-th window image of the second grayscale image after normalization. K is the total number of window images of the first grayscale image after normalization (which is also equal to the total number of window images of the second grayscale image after normalization), is the pixel value of the (i, j) pixel point of the k-th window image of the first grayscale image after normalization, is the pixel value of the (i, j) pixel point of the q-th window image of the second grayscale image after normalization. o is the horizontal pixel size of the window, and p is the vertical pixel size of the window;

[0118] Calculate the effective matching coefficient corresponding to each window image of the first grayscale image after normalization according to the following formula:

[0119] ;

[0120] where, is the effective matching coefficient corresponding to the k-th window image of the first grayscale image after normalization, represents the maximum value of the matching coefficients between the k-th window image of the first grayscale image after normalization and each window image of the second grayscale image after normalization;

[0121] Sort the effective matching coefficients in ascending order. After deleting the first Q effective matching coefficients before sorting, calculate the average value of the remaining effective matching coefficients as the matching degree between the key feature and the standard key feature; where Q is a preset positive integer (Q < K), and can be specifically set according to actual needs.

[0122] Actually, the calculation method of the matching degree between the key feature and the standard key feature is not limited to the above method. For example, existing technologies such as the structural similarity (SSIM) index, normalized grayscale matching, and feature-based matching algorithms can also be used to calculate the matching degree between the key feature and the standard key feature, but not limited to this.

[0123] In some embodiments, it is determined whether each frame image not marked as a key-frame image is a key-frame image according to a feature content evaluation function. Specifically, it includes: determining a frame image whose feature content evaluation function exceeds a preset evaluation function threshold (which can be set according to actual needs) as a key-frame image.

[0124] In other embodiments, it is determined whether each frame image not marked as a key-frame image is a key-frame image according to a feature content evaluation function. Specifically, it includes:

[0125] Descendingly sort each frame image not marked as a key-frame image according to the magnitude of the feature content evaluation function;

[0126] According to the sorting result, determine the first N frame images as key-frame images, where N is a preset positive integer (which can be set according to actual needs).

[0127] Thus, the number of key-frame images obtained by screening is determined by the preset N, which can avoid having too many key-frame images, resulting in the need for the user to select target key-frame images from a large number of key-frame images when generating a surgical text and picture report, and is beneficial to improving the efficiency of the user's selection of target key-frame images; in addition, it can also avoid having too few key-frame images, resulting in the lack of important images, and increase the probability that the user can select the most needed frame images.

[0128] Further, after the construction of the key-frame image set is completed, the similarity between the key-frame images in the key-frame image set can also be calculated. For key-frame images with a similarity deviation within a preset tolerance range (which can be set according to actual needs), only one of the key-frame images is retained (for example, retain the one with the largest feature content evaluation function); since the image information contained in key-frame images with a similarity deviation within the preset tolerance range is basically equivalent, retaining only one of them can also avoid the lack of important images, so that while ensuring that important images are not missing, the number of key-frame images can be reduced, and the efficiency of the user's selection of target key-frame images can be improved.

[0129] Specifically, see Figure 1 , the report generation module 203 includes a report picture generation module 204 and a report document generation module 205;

[0130] The report picture generation module 204 is used to assist the user in performing a selection operation on the key-frame images (i.e., the operation of selecting target key-frame images), and perform annotation processing on the selected key-frame images to generate report pictures;

[0131] The report document generation module 205 is used to obtain a corresponding report template according to the patient's basic information, and after calling the report picture generation module 204 to perform the selection operation and annotation processing of the key-frame images, insert the generated report pictures into the corresponding positions in the report template.

[0132] Among them, the report template includes several report picture insertion positions. The report generation system 2 further includes a human-computer interaction device. The user can perform editing operations on the obtained report template through the human-computer interaction device, including clicking on the report picture insertion positions. When a report picture insertion position is clicked, the report picture generation module 204 generates a key frame image display interface for the user to select a target key frame image. After the user selects the target key frame image, an image editing interface is generated and the target key frame image is displayed in the image editing interface. The image editing interface provides an annotation function for annotating the target key frame image (such as at least one annotation function of drawing circles, captions, local magnification, etc.). After the user completes the annotation of the target key frame image, the annotated target key frame image is inserted as a report picture at the clicked report picture insertion position. Thus, it can assist the user to quickly complete the selection, annotation, and insertion operations of each target key frame image, improving the generation efficiency of the surgical graphic report.

[0133] Among them, the surgical graphic report includes some or all of the patient's basic information, graphic process information, pathological biopsy information, postoperative diagnosis information, and doctor's advice information.

[0134] The graphic process information includes text surgical process information and corresponding report pictures. Thus, generally, the surgical graphic report needs to include graphic process information.

[0135] The pathological biopsy information includes at least one of the macroscopic appearance of the biopsy tissue (including some or all of the whole picture of the biopsy tissue, the biopsy site, the shape of the suspected cancerous tumor, whether the edge is clear, the size and weight of the biopsy tissue), microscopic observation information (including microscopic images and descriptions of the biopsy tissue at the molecular level), and professional diagnosis information given by the pathology department. The pathological biopsy information can be obtained from the medical information system 90 and inserted into the report template.

[0136] The postoperative diagnosis information includes at least one of the diagnosis information of the diseases observable during the operation, the basic pathological processes caused by the surgical process (such as the occurrence of hypoxia, shock, etc.), and surgical complication information. The postoperative diagnosis information can be obtained from the medical information system 90 and inserted into the report template.

[0137] The doctor's advice information includes at least one of postoperative recovery advice (including prevention and treatment of complications, diet recommendations and taboos, exercise methods and frequencies, etc.), medication guidance, and review advice. The doctor's advice information can be obtained from the medical information system 90 and inserted into the report template.

[0138] For example Figure 3In it, an exemplary surgical illustrated report is shown. The surgical illustrated report includes patient basic information, illustrated process information, postoperative diagnosis information, and doctor's advice information. In the figure, Part1 is the patient basic information, Part2 is the report picture in the illustrated process information, Part3 is the text surgical process information in the illustrated process information, Part4 is the postoperative diagnosis information, and Part5 is the doctor's advice information.

[0139] In some possible implementation manners, the report generation system 2 further includes a printing and transmission module 206, and the printing and transmission module 206 is configured to send the generated surgical illustrated report to the medical information system 90 so that the medical information system 90 prints and stores the surgical illustrated report.

[0140] In this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0141] The above are only embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A minimally invasive endoscopic graphic report generation system, characterized in that, It includes an image acquisition device (1) and a report generation system (2). The report generation system (2) includes a patient information access module (201), a key-frame image screening module (202), and a report generation module (203). The image acquisition device (1) and the report generation system (2) are communicatively connected; The image acquisition device (1) is configured to acquire surgical process images through a minimally invasive endoscope (102) and send them to the report generation system (2); The patient information access module (201) is configured to obtain patient basic information from the hospital's medical information system (90); The key-frame image screening module (202) is configured to screen key-frame images from the surgical process images according to the patient basic information; The report generation module (203) is configured to generate a surgical illustrated report according to the patient basic information and the key-frame images; The image acquisition device (1) includes a central control processing module (101), a minimally invasive endoscope (102), a memory (103), a wireless communication module (104), and an instruction trigger device (105). The minimally invasive endoscope (102), the memory (103), the wireless communication module (104), and the instruction trigger device (105) are all electrically connected to the central control processing module (101); The minimally invasive endoscope (102) is configured to acquire surgical process images; The instruction trigger device (105) is configured to generate a prompt instruction under the operation of the surgeon; The central control processing module (101) is configured to record the surgical process images in the memory (103), perform key-frame marking on the surgical process images when receiving the prompt instruction, and send the surgical process images to the report generation system (2) through the wireless communication module (104); When the key-frame image screening module (202) screens key-frame images from the surgical process images according to the patient basic information, it performs: Establish a key-frame image set and add the frame images marked as key-frame images in the surgical process images to the key-frame image set; Use a pre-trained image feature recognition model to perform key feature recognition on each frame image in the surgical process images that is not marked as a key-frame image, and obtain the key features included in the corresponding frame image; The key features include the lesion features of the surgical target site, the surgical wound features of the surgical target site, and the local shape features of the surgical target site; According to the key features included in each frame image that is not marked as a key-frame image, calculate the feature content evaluation function of each frame image that is not marked as a key-frame image; Determine whether each frame image that is not marked as a key-frame image is a key-frame image according to the feature content evaluation function, and add the frame images determined to be key-frame images to the key-frame image set; According to the key features included in each frame image that is not marked as a key-frame image, calculate the feature content evaluation function of each frame image that is not marked as a key-frame image, specifically including: Calculate the matching degree between each key feature contained in the frame image that is not marked as a key frame image and the standard key feature of the same type; The characteristic content evaluation function of the frame image not marked as a key frame image is calculated according to the following formula: ； Among them, is the feature content evaluation function of the frame image not marked as a key frame image, m is the number of key features included in the frame image not marked as a key frame image, is the matching degree corresponding to the i-th key feature included in the frame image not marked as a key frame image, is the weight factor corresponding to the i-th key feature included in the frame image not marked as a key frame image; The central control processing module (101) is further used to adjust the image acquisition frame rate of the minimally invasive laparoscope (102) at different surgical stages according to the basic information of the patient, wherein the surgical stages include non-important surgical stages and important surgical stages, so that the image acquisition frame rate of the non-important surgical stage is lower than the image acquisition frame rate of the important surgical stage; When the central control processing module (101) adjusts the image acquisition frame rate of the minimally invasive laparoscope (102) at different surgical stages according to the basic information of the patient, the central control processing module (101) executes: Acquire a moving distance cutoff value according to the basic information of the patient; Acquiring the real-time moving distance of the minimally invasive laparoscope (102); Comparing the real-time moving distance with the moving distance cutoff value to determine the category of the current surgical stage; If the current surgical stage is a non-important surgical stage, the minimally invasive laparoscope (102) is controlled to capture surgical process images at a first preset frame rate; if the current surgical stage is an important surgical stage, the minimally invasive laparoscope (102) is controlled to capture surgical process images at a second preset frame rate; the first preset frame rate is less than the second preset frame rate.

2. The minimally invasive laparoscope graphic report generation system according to claim 1, wherein When recording the surgical process image in the memory (103), the central control processing module (101) executes: When the remaining storage space of the memory (103) is lower than a preset storage space threshold, estimating the data size of the surgical process image of the current operation based on the basic information of the patient; Determining the historical surgical process images to be deleted and the frame images to be deleted based on the data size of the key frame images marked in the historical surgical process images in the memory (103) and the estimated data size of the surgical process images of the current surgery; When the memory (103) is full, each frame image to be deleted of each historical surgical process image to be deleted is deleted in the order of recording time, so as to reserve storage space for storing surgical process images collected in real time.

3. The minimally invasive laparoscope graphic report generation system according to claim 1, characterized in that, Determining whether each frame image not marked as a key frame image is a key frame image according to the feature content evaluation function specifically includes: According to the magnitude of the characteristic content evaluation function, each frame image that is not marked as a key frame image is sorted in descending order; According to the sorting result, the first N frame images in the sorting are determined as key frame images, where N is a preset positive integer.

4. The minimally invasive laparoscope graphic report generation system according to claim 1, characterized in that, The report generation module (203) includes a report picture generation module (204) and a report document generation module (205); The report image generation module (204) is used to assist the user in selecting the key frame image, and to perform annotation processing on the selected key frame image to generate a report image; The report document generation module (205) is configured to obtain a corresponding report template according to the basic patient information, and after calling the report picture generation module (204) to perform the operation of selecting key-frame images and annotation processing, insert the generated report pictures into corresponding positions in the report template.

5. The minimally invasive laparoscope graphic report generation system according to claim 1, wherein The surgical graphic report includes some or all of the basic patient information, graphic process information, pathological biopsy information, postoperative diagnosis information, and doctor's advice information.

Images