Method for automatically generating violation operation report and electronic equipment

Through multiple three-dimensional feature extraction and feature vector transformation, illegal operations in surgical videos are identified, which solves the problem of ignoring detailed features in two-dimensional feature recognition, and improves the accuracy of surgical reports of illegal operations.

CN118942612APending Publication Date: 2024-11-12QINGDAO HISENSE MEDICAL EQUIP
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310526253.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, when identifying surgical videos illegally based on two-dimensional convolutional network model, detailed characteristics are ignored, resulting in a low accuracy rate of illegal operation surgical reports.

Method used

Multiple continuous three-dimensional feature extraction layers are used to perform three-dimensional feature extraction on the surgical video for multiple consecutive times. Through the three-dimensional surgical feature map and feature vector recognition surgical operations, the recognition accuracy of illegal operations is improved.

Benefits of technology

The detailed grasping of the actions during the surgery from a spatial perspective has improved the accuracy of identification of illegal operations and further improved the accuracy of generated illegal operation reports.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942612A_ABST
    Figure CN118942612A_ABST
Patent Text Reader

Abstract

The invention provides an automatic generation method of a violation operation report and electronic equipment. The accuracy of a generated operation report is improved. Comprising the following steps: on the basis of the sequence of each intermediate operation video in a target operation video obtained by performing video grouping on operation videos, performing continuous multi-time three-dimensional feature extraction on each intermediate operation video in sequence by using a plurality of continuous three-dimensional feature extraction layers to obtain a plurality of three-dimensional operation feature maps of each intermediate operation video; for any intermediate operation video, converting the plurality of three-dimensional operation feature maps of the intermediate operation video into three-dimensional operation feature vectors by using a full connection layer; performing surgical operation identification on the intermediate surgical video based on the three-dimensional surgical feature vector by using a classifier to obtain the probability that the surgical operation in the intermediate surgical video is each preset surgical operation; obtaining violation operation of the intermediate operation video according to the probability of each operation; and obtaining an operation report of the operation video through the violation operation of each intermediate operation video.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] In the medical field, surgical process information can be integrated through information data, digital media information image recognition, and voice recognition technologies to automatically generate surgical reports of illegal operations, avoiding manual completion of surgical reports. To generate surgical reports of illegal operations, it is necessary to first analyze the human body movements of the doctors performing the operation in the surgical video to determine the illegal operations during the operation, and then generate corresponding illegal operation surgical reports based on the illegal operations.

[0003] In the prior art, the main method for determining illegal operations is to use a two-dimensional convolutional network model to extract two-dimensional features from the surgical video, and identify illegal operations during the operation based on the extracted two-dimensional features. However, since the images of each surgical operation during the surgical operation do not change significantly, and the surgical operations are all actions with small changes, the identification of illegal operations based on two-dimensional features will ignore the detailed features, which will result in a low accuracy rate of the generated illegal operation surgical report. Summary of the invention

[0004] In an exemplary embodiment of the present disclosure, a method for automatically generating a surgical report of an illegal operation and an electronic device are provided to improve the accuracy of the generated surgical report of an illegal operation.

[0005] A first aspect of the present disclosure provides a method for automatically generating a surgical report of an illegal operation, the method comprising:

[0006] Grouping the surgical videos for which the surgical report is to be generated to obtain a target surgical video, wherein the target surgical video includes a plurality of intermediate surgical videos, and the duration of any of the intermediate surgical videos is shorter than the duration of the target surgical video;

[0007] Based on the sequence of the intermediate surgical videos in the target surgical video, three-dimensional feature extraction is performed on the intermediate surgical videos in sequence for multiple times using multiple continuous three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of the intermediate surgical videos, wherein the number of the three-dimensional feature extraction layers is the same as the number of times three-dimensional feature extraction is performed on any one of the intermediate surgical videos;

[0008] For any intermediate surgical video, using a fully connected layer to convert multiple three-dimensional surgical feature maps of the intermediate surgical video into feature vectors to obtain a three-dimensional surgical feature vector; and

[0009] Using a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector, and obtaining the probability that the surgical operation in the intermediate surgical video is each pre-set surgical operation; and,

[0010] According to the probability of each surgical operation, the illegal operation in the intermediate surgical video is obtained;

[0011] The surgical report of the surgical video is obtained by detecting the illegal operations in each intermediate surgical video in the target surgical video.

[0012] In this embodiment, multiple continuous three-dimensional feature extraction layers are used to extract three-dimensional features of the surgical video for generating the surgical report for multiple times to obtain multiple three-dimensional surgical feature maps, and three-dimensional surgical feature vectors are determined based on the multiple three-dimensional surgical feature maps to obtain illegal operations in the surgical video, and finally an illegal surgical report is generated based on the illegal operations. In the embodiment of the present application, multiple continuous three-dimensional feature extraction layers are used to extract three-dimensional spatial features of the surgical video for multiple times, so that the actions during the operation can be finely captured from a spatial perspective, and more subtle actions can be identified, thereby improving the recognition accuracy of illegal operations during the operation and further improving the accuracy of the generated illegal operation surgical report.

[0013] In one embodiment, the plurality of continuous three-dimensional feature extraction layers are connected end to end, and any one of the three-dimensional feature extraction layers includes a three-dimensional convolution layer and a pooling layer;

[0014] The method of extracting three-dimensional features of each intermediate surgical video in sequence by using multiple continuous three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of each intermediate surgical video includes:

[0015] For any 3D feature extraction layer, a 3D convolution layer in the 3D feature extraction layer is used to perform 3D feature extraction on a plurality of images to be processed corresponding to any intermediate surgical video, to obtain a plurality of first intermediate 3D feature maps, wherein the plurality of images to be processed are surgical video images in the intermediate surgical video or a plurality of second intermediate 3D feature maps output by a 3D feature extraction layer located in an upper layer of the 3D feature extraction layer; and,

[0016] Performing a downsampling operation on the plurality of first intermediate three-dimensional feature maps by using a pooling layer in the three-dimensional feature extraction layer to obtain a plurality of second intermediate three-dimensional feature maps, wherein the sizes of the plurality of second intermediate three-dimensional feature maps are specified sizes;

[0017] A second intermediate three-dimensional feature map obtained by a last three-dimensional feature extraction layer among the multiple continuous three-dimensional feature extraction layers is determined as the multiple three-dimensional surgical feature maps of the intermediate surgical video.

[0018] In the embodiment of the present application, the structure of any three-dimensional feature extraction layer is set to a combination of a three-dimensional convolution layer and a pooling layer. This is because after the three-dimensional convolution layer extracts multiple features, the pooling layer is used to reduce the feature dimension of the three-dimensional convolution layer output, so as to reduce the amount of calculation, reduce the probability of overfitting, and improve the accuracy of generating surgical reports of illegal operations.

[0019] In one embodiment, the plurality of images to be processed are determined in the following manner:

[0020] For any three-dimensional feature extraction layer, if the three-dimensional feature extraction layer is the first layer of the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are the surgical video images in the intermediate surgical video; or,

[0021] If the three-dimensional feature extraction layer is not the first layer among the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are multiple second intermediate three-dimensional feature maps output by the three-dimensional feature extraction layer located in the upper layer of the three-dimensional feature extraction layer.

[0022] In one embodiment, based on the sequence of the intermediate surgical videos in the target surgical video, the method further comprises:

[0023] The target surgical video is filtered using a filter to obtain a filtered target surgical video, and the filtered target surgical video is determined as the target surgical video.

[0024] In the embodiment of the present application, before performing three-dimensional feature processing on the target surgical video, it is necessary to first filter the target surgical video to filter out the noise in the target surgical video, thereby further improving the accuracy of the generated surgical report.

[0025] In one embodiment, obtaining the surgical report of the surgical video by detecting the illegal operations in each intermediate surgical video in the target surgical video includes:

[0026] For any intermediate surgical video in the target surgical video, based on the number of frames of the intermediate surgical video in the target surgical video and the illegal operation corresponding to the intermediate surgical video, obtaining an intermediate surgical report of the intermediate surgical video;

[0027] According to the intermediate surgical reports of each intermediate surgical video, the surgical report of the surgical video is obtained.

[0028] In one embodiment, the step of grouping the surgical videos for which surgical reports are to be generated to obtain target surgical videos includes:

[0029] Using a pre-trained video grouping model to group the surgical video to obtain the target surgical video; or,

[0030] The surgical video is grouped according to the total number of frames of the surgical video and the number of preset video groupings to obtain the target surgical video.

[0031] A second aspect of the present disclosure provides an electronic device, wherein the processor and the memory are connected via a bus;

[0032] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0033] Grouping the surgical videos for which the surgical report is to be generated to obtain a target surgical video, wherein the target surgical video includes a plurality of intermediate surgical videos, and the duration of any of the intermediate surgical videos is shorter than the duration of the target surgical video;

[0034] Based on the sequence of the intermediate surgical videos in the target surgical video, three-dimensional feature extraction is performed on the intermediate surgical videos in sequence for multiple times using multiple continuous three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of the intermediate surgical videos, wherein the number of the three-dimensional feature extraction layers is the same as the number of times three-dimensional feature extraction is performed on any one of the intermediate surgical videos;

[0035] For any intermediate surgical video, using a fully connected layer to convert multiple three-dimensional surgical feature maps of the intermediate surgical video into feature vectors to obtain a three-dimensional surgical feature vector; and

[0036] Using a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector, and obtaining the probability that the surgical operation in the intermediate surgical video is each pre-set surgical operation; and,

[0037] According to the probability of each surgical operation, the illegal operation in the intermediate surgical video is obtained;

[0038] The surgical report of the surgical video is obtained by detecting the illegal operations in each intermediate surgical video in the target surgical video.

[0039] In one embodiment, the plurality of continuous three-dimensional feature extraction layers are connected end to end, and any one of the three-dimensional feature extraction layers includes a three-dimensional convolution layer and a pooling layer;

[0040] The processor performs the three-dimensional feature extraction of each intermediate surgical video in sequence for multiple consecutive times using multiple consecutive three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of each intermediate surgical video, and is specifically configured as follows:

[0041] For any 3D feature extraction layer, a 3D convolution layer in the 3D feature extraction layer is used to perform 3D feature extraction on a plurality of images to be processed corresponding to any intermediate surgical video, to obtain a plurality of first intermediate 3D feature maps, wherein the plurality of images to be processed are surgical video images in the intermediate surgical video or a plurality of second intermediate 3D feature maps output by a 3D feature extraction layer located in an upper layer of the 3D feature extraction layer; and,

[0042] Performing a downsampling operation on the plurality of first intermediate three-dimensional feature maps by using a pooling layer in the three-dimensional feature extraction layer to obtain a plurality of second intermediate three-dimensional feature maps, wherein the sizes of the plurality of second intermediate three-dimensional feature maps are specified sizes;

[0043] A second intermediate three-dimensional feature map obtained by a last three-dimensional feature extraction layer among the multiple continuous three-dimensional feature extraction layers is determined as the multiple three-dimensional surgical feature maps of the intermediate surgical video.

[0044] In one embodiment, the processor is further configured to:

[0045] The multiple images to be processed are determined by:

[0046] For any three-dimensional feature extraction layer, if the three-dimensional feature extraction layer is the first layer of the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are the surgical video images in the intermediate surgical video; or,

[0047] If the three-dimensional feature extraction layer is not the first layer among the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are multiple second intermediate three-dimensional feature maps output by the three-dimensional feature extraction layer located in the upper layer of the three-dimensional feature extraction layer.

[0048] In one embodiment, the processor is further configured to:

[0049] Based on the sequence of the intermediate surgical videos in the target surgical video, multiple continuous three-dimensional feature extraction layers are used to perform three-dimensional feature extraction on each intermediate surgical video in sequence for multiple times. Before obtaining multiple three-dimensional surgical feature maps of each intermediate surgical video, the target surgical video is filtered by a filter to obtain a filtered target surgical video, and the filtered target surgical video is determined as the target surgical video.

[0050] In one embodiment, the processor performs the illegal operation through each intermediate surgical video in the target surgical video to obtain a surgical report of the surgical video, and is specifically configured as follows:

[0051] For any intermediate surgical video in the target surgical video, based on the number of frames of the intermediate surgical video in the target surgical video and the illegal operation corresponding to the intermediate surgical video, obtaining an intermediate surgical report of the intermediate surgical video;

[0052] According to the intermediate surgical reports of each intermediate surgical video, the surgical report of the surgical video is obtained.

[0053] In one embodiment, the processor performs the video grouping for the surgical video for which the surgical report is to be generated to obtain the target surgical video, and is specifically configured as follows:

[0054] Using a pre-trained video grouping model to group the surgical video to obtain the target surgical video; or,

[0055] The surgical video is grouped according to the total number of frames of the surgical video and the number of preset video groupings to obtain the target surgical video.

[0056] According to a third aspect provided by an embodiment of the present disclosure, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 This is one of the schematic diagrams of application scenarios according to one embodiment of the present disclosure;

[0059] Figure 2This is a second schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0060] Figure 3 This is one of the flow charts of a method for automatically generating a surgical report of an illegal operation according to an embodiment of the present disclosure;

[0061] Figure 4 is a schematic diagram of a surgical video according to an embodiment of the present disclosure;

[0062] Figure 5 A schematic diagram of video grouping according to an embodiment of the present disclosure;

[0063] Figure 6 is a schematic structural diagram of a three-dimensional feature extraction layer according to an embodiment of the present disclosure;

[0064] Figure 7 is a schematic diagram of the structure of multiple three-dimensional feature extraction layers according to an embodiment of the present disclosure;

[0065] Figure 8 is a schematic diagram of a surgical report according to an embodiment of the present disclosure;

[0066] Fig. 9 A schematic diagram of the structure of a neural network model according to an embodiment of the present disclosure;

[0067] Fig.10 It is a flowchart of a method for automatically generating a report of an illegal operation according to an embodiment of the present disclosure;

[0068] Fig.11 A schematic diagram of a device for automatically generating a report of an illegal operation according to an embodiment of the present disclosure;

[0069] Fig.12 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0071] In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0072] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0073] In the prior art, the main method for determining illegal operations is to use a two-dimensional convolutional network model to extract two-dimensional features from the surgical video, and identify illegal operations during the operation based on the extracted two-dimensional features. However, since the images of each surgical operation during the surgical operation do not change significantly, and the surgical operations are all actions with small changes, the identification of illegal operations based on two-dimensional features will ignore the detailed features, which will result in a low accuracy rate of the generated illegal operation surgical report.

[0074] Therefore, the application proposes a method for automatically generating a surgical report of illegal operations, by using multiple continuous three-dimensional feature extraction layers to extract three-dimensional features of the surgical video for which the surgical report is to be generated to obtain multiple three-dimensional surgical feature maps, and based on the three-dimensional surgical feature vectors determined by the multiple three-dimensional surgical feature maps, the illegal operations in the surgical video are obtained, and finally the illegal surgical report is generated based on the illegal operations. In the embodiment of the present application, by using multiple continuous three-dimensional feature extraction layers to extract three-dimensional spatial features of the surgical video for multiple times, the actions during the operation can be finely captured from a spatial perspective, and more subtle actions can be identified, thereby improving the recognition accuracy of illegal operations during the operation and further improving the accuracy of the generated surgical report of illegal operations.

[0075] Figure 1 The following is a schematic diagram showing an application scenario of the method for automatically generating a report of an illegal operation provided by an embodiment of the present application; Figure 1 As shown, the application scenario is described by taking the electronic device as a server as an example. However, the embodiment of the present application does not limit the electronic device, and the electronic device can be a server and a terminal device, etc., which can be set according to actual conditions. The application scenario includes a camera 110, a server 120 and a terminal device 130. The server 120 can be implemented by a single server or by multiple servers. The server 120 can be implemented by a physical server or by a virtual server.

[0076] In a possible application scenario, the camera 110 sends the surgical video for generating a surgical report to the server 120. After receiving the surgical video for generating the surgical report, the server 120 groups the surgical video for generating the surgical report to obtain a target surgical video, wherein the target surgical video includes multiple intermediate surgical videos, and the duration of any intermediate surgical video is less than the duration of the target surgical video; then the server 120 uses multiple continuous three-dimensional feature extraction layers to perform three-dimensional feature extraction on each intermediate surgical video in sequence based on the sequence of the intermediate surgical videos in the target surgical video to obtain multiple three-dimensional surgical feature maps of each intermediate surgical video, wherein the number of the three-dimensional feature extraction layers is the same as the number of times the three-dimensional feature extraction is performed on any intermediate surgical video; for any intermediate surgical video, the server 120 uses a fully connected layer to convert the multiple three-dimensional surgical feature maps of the intermediate surgical video into feature vectors to obtain a three-dimensional surgical feature vector, and uses a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector to obtain the probability that the surgical operation in the intermediate surgical video is the pre-set surgical operation. Then, the server 120 obtains the illegal operations in the intermediate surgical video according to the probability of each surgical operation, and obtains the surgical report of the surgical video through the illegal operations in each intermediate surgical video in the target surgical video. Finally, the surgical report of the surgical video is sent to the terminal device 130 for display.

[0077] like Figure 2, which is a schematic diagram of another application scenario in an embodiment of the present application, wherein the application scenario includes a camera 110 and a terminal device 130 . The camera 110 sends the surgical video for generating a surgical report to the terminal device 130. After receiving the surgical video for generating a surgical report, the terminal device 130 groups the surgical video for generating a surgical report to obtain a target surgical video, wherein the target surgical video includes multiple intermediate surgical videos, and the duration of any intermediate surgical video is less than the duration of the target surgical video. Then, based on the sequence of the intermediate surgical videos in the target surgical video, the terminal device 130 uses multiple continuous three-dimensional feature extraction layers to extract three-dimensional features on each intermediate surgical video in sequence for multiple times to obtain multiple three-dimensional surgical feature maps of each intermediate surgical video, wherein the number of the three-dimensional feature extraction layers is the same as the number of times the three-dimensional feature extraction is performed on any intermediate surgical video. For any intermediate surgical video, the terminal device 130 uses a fully connected layer to convert the multiple three-dimensional surgical feature maps of the intermediate surgical video into feature vectors to obtain a three-dimensional surgical feature vector, and uses a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector to obtain the probability that the surgical operation in the intermediate surgical video is the pre-set surgical operation. Then, the terminal device 130 obtains the illegal operations in the intermediate surgical video according to the probabilities of the surgical operations, and obtains and displays the surgical report of the surgical video through the illegal operations in the intermediate surgical videos in the target surgical video.

[0078] in, Figure 1 The server 120 and the terminal device 130 can exchange information through a communication network, wherein the communication mode adopted by the communication network can be divided into a wireless communication mode or a wired communication mode.

[0079] Exemplarily, the server 120 can access the network through cellular mobile communication technology to communicate with the terminal device 130, wherein the cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th Generation Mobile Networks, 5G) technology.

[0080] Optionally, the server 120 may access the network and communicate with the terminal device 130 via short-range wireless communication, wherein the short-range wireless communication method, for example, includes Wireless Fidelity (Wi-Fi) technology.

[0081] Among them, the description in this application only details a single camera 110, a single server 120, and a single terminal device 130, but those skilled in the art should understand that the illustrated camera 110, server 120, and terminal device 130 are intended to represent the operations of the camera 110, server 120, and terminal device 130 involved in the technical solution of this application. It does not imply any limitation on the number, type, or location of the camera 110, server 120, and terminal device 130. It should be noted that if additional modules are added to the illustrated environment or individual modules are removed therefrom, the underlying concepts of the exemplary embodiments of this application will not be changed.

[0082] It should be noted that the automatic generation method of illegal operation report proposed in this application is not only applicable to Figure 1 and Figure 2 The application scenario shown is also applicable to any device that automatically generates reports of surgical operations that violate regulations.

[0083] The following describes the method for automatically generating a report on an illegal operation in an exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate understanding of the method and principle of the present application, and the implementation of the present application is not limited in this respect.

[0084] like Figure 3 FIG. 1 is a flow chart of the method for automatically generating a surgical report of an illegal operation disclosed in the present invention, which may include the following steps:

[0085] Step 301: grouping the surgical videos for which surgical reports are to be generated to obtain target surgical videos, wherein the target surgical videos include a plurality of intermediate surgical videos, and the duration of any of the intermediate surgical videos is shorter than the duration of the target surgical video;

[0086] In order to further improve the accuracy of the generated surgical report, in one embodiment, before executing step 301, it is necessary to perform a preprocessing operation on the surgical video. The preprocessing operation includes frame reduction and cropping. For example, the frame reduction operation can be to reduce the frame rate of a surgical video with a frame rate of 60 to a surgical video with a frame rate of 30 frames or a surgical video with a frame rate of 24 frames. Cropping operations, such as laparoscopic surgical videos, will have black frames around them, which are useless information. It will also affect the accuracy of feature recognition and judgment. Therefore, the black edges of the video image can be cropped and the image size can be adjusted to a uniform size.

[0087] It should be noted that the preprocessing operation in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application is not limited here.

[0088] In the embodiment of the present application, the surgical video for which the surgical report is to be generated is selected by the user, such as Figure 4 As shown, Figure 4 The surgical report includes multiple surgical videos, and the user can select the video for which the surgical report needs to be generated from the multiple surgical videos. The surgical video selected by the user is the surgical video for which the surgical report is to be generated.

[0089] In one embodiment, the surgical videos for which surgical reports are to be generated may be grouped in the following two ways:

[0090] Method 1: Use a pre-trained video grouping model to group the surgical videos to obtain the target surgical videos.

[0091] Among them, the video grouping model in the embodiment of the present application is a neural network model, and the neural network model can be set according to the specific actual situation. The embodiment of the present application does not limit the neural network model.

[0092] Method 2: Grouping the surgical video according to the total number of frames of the surgical video and the number of preset video groupings to obtain the target surgical video.

[0093] In one embodiment, the total number of frames of the surgical video is divided by the number of groups to obtain the number of frames of a group of videos, and then the surgical videos are grouped based on the number of frames of the group of videos.

[0094] For example, if the number of groups is 5 and the total number of frames of the surgical video is 20, the number of frames in one group of videos is 4, and 5 intermediate surgical videos are obtained. Figure 5 As shown, the left image is each image frame in the surgical video, and the right image is the target surgical video after the surgical video is grouped. Figure 5 It can be seen that the 1st to 4th frames correspond to the intermediate surgical video A, the 5th to 8th frames correspond to the intermediate surgical video B, the 9th to 12th frames correspond to the intermediate surgical video C, the 13th to 16th frames correspond to the intermediate surgical video D, and the 17th to 20th frames correspond to the intermediate surgical video E.

[0095] It should be noted that the number of video groupings in the embodiment of the present application is pre-set and can be set specifically according to actual conditions. The embodiment of the present application does not limit the number of video groupings.

[0096] In order to reduce the amount of calculation and improve the efficiency of generating illegal operation surgical reports, in one embodiment, before executing step 302, the target surgical video is filtered using a filter to obtain a filtered target surgical video, and the filtered target surgical video is determined as the target surgical video.

[0097] Among them, the filter in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the filter here.

[0098] Step 302: Based on the sequence of the intermediate surgical videos in the target surgical video, use a plurality of continuous 3D feature extraction layers to sequentially perform 3D feature extraction on the intermediate surgical videos for multiple times to obtain a plurality of 3D surgical feature graphs for the intermediate surgical videos, wherein the number of the 3D feature extraction layers is the same as the number of times the 3D feature extraction is performed on any intermediate surgical video;

[0099] The sequence of the intermediate surgical videos in the embodiment of the present application is the time sequence of the intermediate surgical videos in the target surgical video, for example, as described above. Figure 5 The order of the intermediate surgical videos in the target surgical video is: intermediate surgical video A, intermediate surgical video B, intermediate surgical video C, intermediate surgical video D, and intermediate surgical video E.

[0100] like Figure 6 As shown, it is a schematic diagram of the structure of the three-dimensional feature extraction layer in the embodiment of the present application. Figure 6 It can be seen that the three-dimensional feature extraction layer 600 includes a three-dimensional convolution layer 601 and a pooling layer 602. Then any three-dimensional feature extraction layer can perform three-dimensional feature extraction in the following manner:

[0101] The three-dimensional convolutional layer 601 in the three-dimensional feature extraction layer 600 is used to perform three-dimensional feature extraction on multiple images to be processed corresponding to any intermediate surgical video to obtain multiple first intermediate three-dimensional feature maps, wherein the multiple images to be processed are each surgical video image in the intermediate surgical video or multiple second intermediate three-dimensional feature maps output by the three-dimensional feature extraction layer located in the upper layer of the three-dimensional feature extraction layer 600; and the pooling layer 602 in the three-dimensional feature extraction layer 600 is used to perform a downsampling operation on the multiple first intermediate three-dimensional feature maps to obtain multiple second intermediate three-dimensional feature maps, wherein the size of the multiple second intermediate three-dimensional feature maps is a specified size; the second intermediate three-dimensional feature map obtained by the three-dimensional feature extraction layer located in the last layer of the multiple consecutive three-dimensional feature extraction layers is determined as the multiple three-dimensional surgical feature maps of the intermediate surgical video.

[0102] In one embodiment, the plurality of images to be processed are determined in the following manner:

[0103] For any three-dimensional feature extraction layer, if the three-dimensional feature extraction layer is the first layer among the multiple continuous three-dimensional feature extraction layers, then the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are the surgical video images in the intermediate surgical video; or if the three-dimensional feature extraction layer is not the first layer among the multiple continuous three-dimensional feature extraction layers, then the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are the multiple second intermediate three-dimensional feature maps output by the three-dimensional feature extraction layer located in the upper layer of the three-dimensional feature extraction layer.

[0104] Among them, the multiple continuous three-dimensional feature extraction layers in the embodiment of the present application are connected end to end.

[0105] like Figure 7 As shown, Figure 7 In the figure, five three-dimensional feature extraction layers are used as an example for explanation, including three-dimensional feature extraction layer 1, three-dimensional feature extraction layer 2, three-dimensional feature extraction layer 3, three-dimensional feature extraction layer 4 and three-dimensional feature extraction layer 5. Among them:

[0106] If the 3D feature extraction layer 1 is the first 3D feature extraction layer, then the multiple images to be processed corresponding to the 3D feature extraction layer 1 are the surgical video images in the intermediate surgical video. If the 3D feature extraction layer 2, 3D feature extraction layer 3, 3D feature extraction layer 4 and 3D feature extraction layer 5 are non-first 3D feature extraction layers, then the multiple images to be processed corresponding to the 3D feature extraction layer 2 are the multiple second intermediate 3D feature maps output by the 3D feature extraction layer 1. The multiple images to be processed corresponding to the 3D feature extraction layer 3 are the multiple second intermediate 3D feature maps output by the 3D feature extraction layer 2. The multiple images to be processed corresponding to the 3D feature extraction layer 4 are the multiple second intermediate 3D feature maps output by the 3D feature extraction layer 3. The multiple images to be processed corresponding to the 3D feature extraction layer 5 are the multiple second intermediate 3D feature maps output by the 3D feature extraction layer 4.

[0107] It should be noted that the number of three-dimensional feature extraction layers in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the number of three-dimensional feature extraction layers.

[0108] Step 303: for any intermediate surgical video, using a fully connected layer to convert multiple 3D surgical feature maps of the intermediate surgical video into feature vectors to obtain a 3D surgical feature vector;

[0109] Step 304: using a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector, and obtaining the probability that the surgical operation in the intermediate surgical video is each preset surgical operation;

[0110] Among them, each surgical operation in the embodiment of the present application is pre-set, for example, the surgical operation may include exposure, dissection, hemostasis knotting, suturing and drainage operations. In the embodiment of the present application, the classifier outputs the probability that the surgical operation in the intermediate surgical video is the corresponding surgical operation. If the pre-set surgical operations are: surgical operation A, surgical operation B, surgical operation C. The result obtained by the classifier is the probability that the surgical operation in the intermediate surgical video is surgical operation A, the probability that the surgical operation in the intermediate surgical video is surgical operation B, and the probability that the surgical operation in the intermediate surgical video is surgical operation C.

[0111] It should be noted that: each surgical operation in the embodiment of the present application can be set according to the specific actual situation, and the embodiment of the present application does not limit the surgical operation. In addition, the classifier in the embodiment of the present application uses a softmax classifier, but the classifier in the embodiment of the present application can be set according to the actual situation, and the embodiment of the present application does not limit the classifier here.

[0112] Step 305: Obtaining illegal operations in the intermediate surgical video according to the probabilities of the surgical operations;

[0113] In one embodiment, the illegal operation of the intermediate surgical video can be obtained in the following two ways:

[0114] Method 1: The surgical operation corresponding to the probability with the largest value is determined as the target surgical operation of the intermediate surgical video; if the probability of the target surgical operation is less than a first specified threshold, the target surgical operation is determined to be an illegal operation. Otherwise, it is determined that there is no illegal operation in the intermediate surgical video.

[0115] For example, if the results obtained for the intermediate surgical video 1 are: surgical operation A: 0.6, surgical operation B: 0.3, surgical operation C: 0.4, then the target surgical operation in the intermediate surgical video 1 is surgical operation A. If the first specified threshold is 0.7, then the probability of surgical operation A is less than the specified threshold, and surgical operation A is determined to be an illegal operation, then the surgical operation in the intermediate surgical video 1 is surgical operation A, and surgical operation A is an illegal operation.

[0116] Method 2: Determine each surgical operation with a probability greater than a second specified threshold as each surgical operation to be confirmed, and after setting each surgical operation to be confirmed as the pre-set surgical operation, return to the sequence of each intermediate surgical video in the target surgical video, perform three-dimensional feature extraction on each intermediate surgical video in turn for multiple times, and obtain a plurality of three-dimensional surgical feature maps of each intermediate surgical video, and obtain the probability that the surgical operation in the intermediate surgical video is each surgical operation to be confirmed; determine the surgical operation corresponding to the probability with the largest numerical value as the target surgical operation of the intermediate surgical video; if the probability of the target surgical operation is less than the first specified threshold, determine that the target surgical operation is an illegal operation.

[0117] For example, if the results obtained in the intermediate surgical video 1 are: surgical operation A: 0.4, surgical operation B: 0.4, surgical operation C: 0.2. If the second specified threshold is 0.3, the surgical operations to be confirmed are surgical operation A and surgical operation B. Then surgical operation A and surgical operation B are set as the preset surgical operations, and the sequence of the intermediate surgical videos in the target surgical video is returned, and the three-dimensional feature extraction is performed on the intermediate surgical videos in sequence for multiple consecutive times to obtain the steps of multiple three-dimensional surgical feature maps of each intermediate surgical video. Then, the results obtained in the intermediate surgical video 1 are: surgical operation A: 0.6, surgical operation B: 0.4. Then surgical operation A is the target surgical operation of the intermediate surgical video. If the first specified threshold is 0.7, the probability of surgical operation A is less than the specified threshold, and surgical operation A is determined to be an illegal operation. Then, the surgical operation in the intermediate surgical video 1 is surgical operation A, and surgical operation A is an illegal operation.

[0118] It should be noted that the second specified threshold in the embodiment of the present application is less than the first specified threshold, but the specific values ​​of the first specified threshold and the second specified threshold are not limited in the embodiment of the present application, and the first specified threshold and the second specified threshold in the embodiment of the present application can be set according to actual conditions. In addition, the sum of the probabilities of each surgical operation in the embodiment of the present application is 1.

[0119] Step 306: Obtain a surgery report of the surgery video through illegal operations in each intermediate surgery video in the target surgery video.

[0120] In one embodiment, step 306 can be specifically implemented as follows: for any intermediate surgical video in the target surgical video, based on the number of frames of the intermediate surgical video in the target surgical video and the illegal operation corresponding to the intermediate surgical video, obtain the intermediate surgical report of the intermediate surgical video; according to the intermediate surgical reports of each intermediate surgical video, obtain the surgical report of the surgical video.

[0121] The intermediate surgery report of any intermediate surgery video includes the illegal operation in the intermediate surgery video and the frame number range of the intermediate surgery video in the target surgery video, and the frame number range is determined as the video frame number interval of the illegal operation. Then, each illegal operation and the video frame number interval corresponding to each illegal operation are displayed in the surgery report.

[0122] In the embodiment of the present application, the surgical report includes information such as illegal operations, video frame intervals of illegal operations, patient information, surgical information, doctors and nurses, drugs and consumables.

[0123] like Figure 8 The following is a schematic diagram of the surgical report. Figure 8 It can be seen that the surgical report shows that there are illegal suturing operations in the 1st to 10th frames of the surgical video, and there are illegal drainage operations in the 31st to 50th frames.

[0124] The identification of illegal operations in the embodiment of the present application is completed based on a neural network model. The structure of the neural network model in the embodiment of the present application is described below. Fig. 9 As shown, it is a schematic diagram of the structure of the neural network model in the embodiment of the present application, and the model includes a three-dimensional feature extraction layer 901, a three-dimensional feature extraction layer 902, a three-dimensional feature extraction layer 903, a three-dimensional feature extraction layer 904, a three-dimensional feature extraction layer 905, a fully connected layer 906 and a classifier 907. And any three-dimensional feature extraction layer includes a three-dimensional convolution layer and a pooling layer. That is, the three-dimensional feature extraction layer 901 includes a three-dimensional convolution layer 9011 and a pooling layer 9012. The three-dimensional feature extraction layer 902 includes a three-dimensional convolution layer 9021 and a pooling layer 9022. The three-dimensional feature extraction layer 903 includes a three-dimensional convolution layer 9031 and a pooling layer 9032. The three-dimensional feature extraction layer 904 includes a three-dimensional convolution layer 9041 and a pooling layer 9042. The three-dimensional feature extraction layer 905 includes a three-dimensional convolution layer 9051 and a pooling layer 9052.

[0125] The number of convolution kernels of the three-dimensional convolution layer 9011 in the embodiment of the present application is 64. The number of convolution kernels of the three-dimensional convolution layer 902 is 1128, the number of convolution kernels of the three-dimensional convolution layer 903 is 256, the number of convolution kernels of the three-dimensional convolution layer 904 is 256, and the number of convolution kernels of the three-dimensional convolution layer 905 is 256, and the convolution kernel size of each three-dimensional convolution layer is 3*3*3. The size of the kernel of the pooling layer 9012 in the embodiment of the present application is 1*1*1. The size of the kernel of other pooling layers is 2*2*2. However, the number and size of the kernels of each three-dimensional convolution layer and the pooling layer in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application is no longer limited here. And any three-dimensional convolution kernel in the embodiment of the present application is performed from 5 channels when performing feature extraction, and each channel is image grayscale, x gradient, y gradient, x streamer, and y streamer, respectively.

[0126] Next, the method of identifying illegal operations in surgical videos in the embodiment of the present application is described in combination with the neural network model:

[0127] First, the target surgical video is input into the neural network model. For any intermediate surgical video in the target surgical video, the three-dimensional convolution layer 9011 in the three-dimensional feature extraction layer 901 is first used to extract features from multiple images in the intermediate surgical video to obtain multiple first intermediate three-dimensional feature maps, and then the multiple first intermediate three-dimensional feature maps are respectively downsampled using the pooling layer 9012 to obtain multiple second intermediate three-dimensional feature maps. The second intermediate three-dimensional feature map is then input into the three-dimensional feature extraction layer 902 for feature extraction, that is, the three-dimensional convolution layer 9021 in the three-dimensional feature extraction layer 902 is first used to extract features from multiple images in the intermediate surgical video to obtain multiple first intermediate three-dimensional feature maps, and then the multiple first intermediate three-dimensional feature maps are respectively downsampled using the pooling layer 9022 to obtain multiple second intermediate three-dimensional feature maps. Then, the second intermediate 3D feature map obtained by the 3D feature extraction layer 902 is input into the 3D feature extraction layer 903 for feature extraction, that is, the 3D convolution layer 9031 in the 3D feature extraction layer 903 is first used to extract features from multiple images in the intermediate surgical video to obtain multiple first intermediate 3D feature maps, and then the multiple first intermediate 3D feature maps are respectively downsampled by the pooling layer 9032 to obtain multiple second intermediate 3D feature maps, and the 3D convolution layer 9041 in the 3D feature extraction layer 904 is used to extract features from multiple images in the intermediate surgical video to obtain multiple first intermediate 3D feature maps, and then the multiple first intermediate 3D feature maps are respectively downsampled by the pooling layer 9042 to obtain multiple second intermediate 3D feature maps. Then, the 3D convolution layer 9051 in the 3D feature extraction layer 905 is used to extract features from multiple images in the intermediate surgical video to obtain multiple first intermediate 3D feature maps, and then the multiple first intermediate 3D feature maps are respectively downsampled by the pooling layer 9052 to obtain multiple second intermediate 3D feature maps. Then, the fully connected layer 906 is used to convert the multiple second intermediate three-dimensional feature maps obtained in the three-dimensional feature extraction layer 905 into feature vectors to obtain the surgical feature vector. Finally, the classifier 907 is used to identify the surgical operation of the intermediate surgical video based on the surgical feature vector to obtain the probability that the surgical operation in the intermediate surgical video is each of the surgical operations.

[0128] The following is a brief introduction to the training method of the neural network model described above: surgical video training samples are input into the neural network model to obtain predicted violation operations corresponding to multiple surgical videos in the surgical video training samples; wherein the surgical video training samples include multiple surgical videos and labeled violation operations corresponding to the surgical videos; a loss value is obtained based on the predicted violation operations of the multiple surgical videos and the labeled violation operations, and if the loss value is greater than a specified loss value, the specified parameters of the neural network model are adjusted, and then the step of inputting the surgical video training samples into the neural network model is returned until the loss value is no greater than the specified loss value, and the training of the neural network model is terminated.

[0129] It should be noted that the specified loss value and the specified parameters in the embodiment of the present application can be set according to the specific actual situation, and the embodiment of the present application is not limited here.

[0130] In order to further understand the technical solution of the present disclosure, Fig.10 A detailed description may include the following steps:

[0131] Step 1001: grouping surgical videos for generating surgical reports to obtain target surgical videos, wherein the target surgical videos include multiple intermediate surgical videos, and the duration of any intermediate surgical video is shorter than the duration of the target surgical video;

[0132] Step 1002: filtering the target surgical video using a filter to obtain a filtered target surgical video, and determining the filtered target surgical video as the target surgical video;

[0133] Step 1003: Based on the sequence of the intermediate surgical videos in the target surgical video, use a plurality of continuous 3D feature extraction layers to sequentially perform 3D feature extraction on the intermediate surgical videos for multiple times to obtain a plurality of 3D surgical feature graphs for the intermediate surgical videos, wherein the number of the 3D feature extraction layers is the same as the number of times the 3D feature extraction is performed on any intermediate surgical video;

[0134] Step 1004: using a fully connected layer to convert multiple 3D surgical feature maps of the intermediate surgical video into feature vectors to obtain a 3D surgical feature vector;

[0135] Step 1005: using a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector, and obtaining the probability that the surgical operation in the intermediate surgical video is each preset surgical operation;

[0136] Step 1006: Obtaining illegal operations in the intermediate surgical video according to the probabilities of the surgical operations;

[0137] Step 1007: for any intermediate surgical video in the target surgical video, based on the number of frames of the intermediate surgical video in the target surgical video and the illegal operation corresponding to the intermediate surgical video, obtaining an intermediate surgical report of the intermediate surgical video;

[0138] Step 1008: Obtain the surgical report of the surgical video according to the intermediate surgical report of each intermediate surgical video.

[0139] Based on the same inventive concept, the automatic generation of the illegal operation report described above can also be realized by an automatic generation device for illegal operation report. The effect of the automatic generation device for illegal operation report is similar to that of the above method, which will not be described in detail here.

[0140] Fig.11 The present invention is a schematic diagram of the structure of an apparatus for automatically generating a report of an illegal operation according to an embodiment of the present invention.

[0141] like Fig.11 As shown, the automatic generation device 1100 of the illegal operation surgical report of the present disclosure may include a video grouping module 1110, a three-dimensional feature extraction module 1120, a feature vector conversion module 1130, a classification module 1140, a illegal operation determination module 1150 and a surgical report generation module 1160.

[0142] The video grouping module 1110 is used to group the surgical videos for which the surgical report is to be generated to obtain a target surgical video, wherein the target surgical video includes a plurality of intermediate surgical videos, and the duration of any of the intermediate surgical videos is shorter than the duration of the target surgical video;

[0143] The 3D feature extraction module 1120 is used to perform 3D feature extraction on each intermediate surgical video in sequence for multiple times using multiple continuous 3D feature extraction layers based on the sequence of each intermediate surgical video in the target surgical video, so as to obtain multiple 3D surgical feature graphs for each intermediate surgical video, wherein the number of the 3D feature extraction layers is the same as the number of times the 3D feature extraction is performed on any intermediate surgical video;

[0144] A feature vector conversion module 1130 is used to convert a plurality of three-dimensional surgical feature maps of any intermediate surgical video into a feature vector using a fully connected layer to obtain a three-dimensional surgical feature vector; and

[0145] The classification module 1140 is used to use a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector, and obtain the probability that the surgical operation in the intermediate surgical video is a preset surgical operation; and

[0146] A violation operation determination module 1150, configured to obtain the violation operation of the intermediate surgical video according to the probability of each surgical operation;

[0147] The surgery report generating module 1160 is used to obtain the surgery report of the surgery video through the illegal operations of each intermediate surgery video in the target surgery video.

[0148] In one embodiment, the plurality of continuous three-dimensional feature extraction layers are connected end to end, and any one of the three-dimensional feature extraction layers includes a three-dimensional convolution layer and a pooling layer;

[0149] The three-dimensional feature extraction module 1120 is specifically used for:

[0150] For any 3D feature extraction layer, a 3D convolution layer in the 3D feature extraction layer is used to perform 3D feature extraction on a plurality of images to be processed corresponding to any intermediate surgical video, to obtain a plurality of first intermediate 3D feature maps, wherein the plurality of images to be processed are surgical video images in the intermediate surgical video or a plurality of second intermediate 3D feature maps output by a 3D feature extraction layer located in an upper layer of the 3D feature extraction layer; and,

[0151] Performing a downsampling operation on the plurality of first intermediate three-dimensional feature maps by using a pooling layer in the three-dimensional feature extraction layer to obtain a plurality of second intermediate three-dimensional feature maps, wherein the sizes of the plurality of second intermediate three-dimensional feature maps are specified sizes;

[0152] A second intermediate three-dimensional feature map obtained by a last three-dimensional feature extraction layer among the multiple continuous three-dimensional feature extraction layers is determined as the multiple three-dimensional surgical feature maps of the intermediate surgical video.

[0153] In one embodiment, the three-dimensional feature extraction module 1120 is further used to:

[0154] For any three-dimensional feature extraction layer, if the three-dimensional feature extraction layer is the first layer of the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are the surgical video images in the intermediate surgical video; or,

[0155] If the three-dimensional feature extraction layer is not the first layer among the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are multiple second intermediate three-dimensional feature maps output by the three-dimensional feature extraction layer located in the upper layer of the three-dimensional feature extraction layer.

[0156] In one embodiment, the apparatus further comprises:

[0157] The filtering module 1170 is used to perform three-dimensional feature extraction on each intermediate surgical video in sequence based on the sequence of the intermediate surgical videos in the target surgical video by using multiple continuous three-dimensional feature extraction layers, and before obtaining multiple three-dimensional surgical feature maps of each intermediate surgical video, filter the target surgical video by using a filter to obtain a filtered target surgical video, and determine the filtered target surgical video as the target surgical video.

[0158] In one embodiment, the surgical report generating module 1160 is specifically used to:

[0159] For any intermediate surgical video in the target surgical video, based on the number of frames of the intermediate surgical video in the target surgical video and the illegal operation corresponding to the intermediate surgical video, obtaining an intermediate surgical report of the intermediate surgical video;

[0160] According to the intermediate surgical reports of each intermediate surgical video, the surgical report of the surgical video is obtained.

[0161] In one embodiment, the video grouping module 1110 is specifically configured to:

[0162] Using a pre-trained video grouping model to group the surgical video to obtain the target surgical video; or,

[0163] The surgical video is grouped according to the total number of frames of the surgical video and the number of preset video groupings to obtain the target surgical video.

[0164] After introducing a method and device for automatically generating a surgical report of an illegal operation according to an exemplary embodiment of the present invention, next, an electronic device according to another exemplary embodiment of the present invention is introduced.

[0165] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0166] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for automatically generating a surgical report of an illegal operation according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 3 Steps 301-306 shown in .

[0167] Refer to the following Fig.12 The electronic device 1200 according to this embodiment of the present invention is described. Fig.12 The electronic device 1200 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0168] like Fig.12 As shown, the electronic device 1200 is in the form of a general electronic device. The components of the electronic device 1200 may include but are not limited to: the at least one processor 1201, the at least one computer storage medium 1202, and a bus 1203 connecting different system components (including the computer storage medium 1202 and the processor 1201).

[0169] Bus 1203 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0170] Computer storage media 1202 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1221 and / or cache storage media 1222 , and may further include read-only computer storage media (ROM) 1223 .

[0171] The computer storage medium 1202 may also include a program / utility 1225 having a set (at least one) of program modules 1224, such program modules 1224 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0172] The electronic device 1200 may also communicate with one or more external devices 1204 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1205. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1206. As shown, the network adapter 1206 communicates with other modules for the electronic device 1200 via a bus 1203. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0173] In some possible embodiments, various aspects of the method for automatically generating a surgical report on an illegal operation provided by the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method for automatically generating a surgical report on an illegal operation according to various exemplary embodiments of the present disclosure described above in this specification.

[0174] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access computer storage medium (RAM), a read-only computer storage medium (ROM), an erasable programmable read-only computer storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only computer storage medium (CD-ROM), an optical computer storage medium, a magnetic computer storage medium, or any suitable combination of the above.

[0175] The program product for automatically generating a report of an illegal operation in an embodiment of the present disclosure can be a portable compact disk read-only computer storage medium (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus, or a device.

[0176] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0177] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0178] Program code for performing the disclosed operations may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device may be connected to the user electronic device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0179] It should be noted that although several modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0180] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0181] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk computer storage media, CD-ROM, optical computer storage media, etc.) containing computer-usable program codes.

[0182] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable computer storage medium produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0185] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for automatically generating a surgical report of an illegal operation. It is characterized in that The method comprises: Grouping the surgical videos for which the surgical report is to be generated to obtain a target surgical video, wherein the target surgical video includes a plurality of intermediate surgical videos, and the duration of any of the intermediate surgical videos is shorter than the duration of the target surgical video; Based on the sequence of the intermediate surgical videos in the target surgical video, three-dimensional feature extraction is performed on the intermediate surgical videos in sequence for multiple times using multiple continuous three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of the intermediate surgical videos, wherein the number of the three-dimensional feature extraction layers is the same as the number of times three-dimensional feature extraction is performed on any one of the intermediate surgical videos; For any intermediate surgical video, a fully connected layer is used to convert multiple three-dimensional surgical feature maps of the intermediate surgical video into feature vectors to obtain a three-dimensional surgical feature vector; and Using a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector, and obtaining the probability that the surgical operation in the intermediate surgical video is each pre-set surgical operation; and, According to the probability of each surgical operation, the illegal operation in the intermediate surgical video is obtained; The surgical report of the surgical video is obtained by detecting the illegal operations in each intermediate surgical video in the target surgical video.

2. The method according to claim 1, It is characterized in that The multiple continuous three-dimensional feature extraction layers are connected end to end, and any three-dimensional feature extraction layer includes a three-dimensional convolution layer and a pooling layer; The method of extracting three-dimensional features of each intermediate surgical video in sequence by using multiple continuous three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of each intermediate surgical video includes: For any 3D feature extraction layer, a 3D convolution layer in the 3D feature extraction layer is used to perform 3D feature extraction on a plurality of images to be processed corresponding to any intermediate surgical video, to obtain a plurality of first intermediate 3D feature maps, wherein the plurality of images to be processed are surgical video images in the intermediate surgical video or a plurality of second intermediate 3D feature maps output by a 3D feature extraction layer located in an upper layer of the 3D feature extraction layer; and, Using a pooling layer in the three-dimensional feature extraction layer to perform a downsampling operation on the plurality of first intermediate three-dimensional feature maps to obtain a plurality of second intermediate three-dimensional feature maps, wherein the sizes of the plurality of second intermediate three-dimensional feature maps are specified sizes; A second intermediate three-dimensional feature map obtained by a last three-dimensional feature extraction layer among the multiple continuous three-dimensional feature extraction layers is determined as the multiple three-dimensional surgical feature maps of the intermediate surgical video.

3. The method according to claim 2, It is characterized in that The multiple images to be processed are determined by: For any three-dimensional feature extraction layer, if the three-dimensional feature extraction layer is the first layer of the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are the surgical video images in the intermediate surgical video; or, If the three-dimensional feature extraction layer is not the first layer among the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are multiple second intermediate three-dimensional feature maps output by the three-dimensional feature extraction layer located in the upper layer of the three-dimensional feature extraction layer.

4. The method according to claim 1, It is characterized in that Before the method further comprises: performing a plurality of consecutive 3D feature extractions on each of the intermediate surgical videos based on the sequence of the intermediate surgical videos in the target surgical video by using a plurality of consecutive 3D feature extraction layers to obtain a plurality of 3D surgical feature maps of each of the intermediate surgical videos; The target surgical video is filtered using a filter to obtain a filtered target surgical video, and the filtered target surgical video is determined as the target surgical video.

5. The method according to claim 1, It is characterized in that The step of obtaining a surgical report of the surgical video by detecting illegal operations in each intermediate surgical video in the target surgical video includes: For any intermediate surgical video in the target surgical video, based on the number of frames of the intermediate surgical video in the target surgical video and the illegal operation corresponding to the intermediate surgical video, obtaining an intermediate surgical report of the intermediate surgical video; According to the intermediate surgical reports of each intermediate surgical video, the surgical report of the surgical video is obtained.

6. The method according to claim 1, It is characterized in that The step of grouping the surgical videos for which the surgical reports are to be generated to obtain target surgical videos includes: Using a pre-trained video grouping model to group the surgical video to obtain the target surgical video; or, The surgical video is grouped according to the total number of frames of the surgical video and the number of preset video groupings to obtain the target surgical video.

7. An electronic device, It is characterized in that comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Grouping the surgical videos for which the surgical report is to be generated to obtain a target surgical video, wherein the target surgical video includes a plurality of intermediate surgical videos, and the duration of any of the intermediate surgical videos is shorter than the duration of the target surgical video; Based on the sequence of the intermediate surgical videos in the target surgical video, three-dimensional feature extraction is performed on the intermediate surgical videos in sequence for multiple times using multiple continuous three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of the intermediate surgical videos, wherein the number of the three-dimensional feature extraction layers is the same as the number of times three-dimensional feature extraction is performed on any one of the intermediate surgical videos; For any intermediate surgical video, using a fully connected layer to convert multiple three-dimensional surgical feature maps of the intermediate surgical video into feature vectors to obtain a three-dimensional surgical feature vector; and Using a classifier to identify the surgical operation of the intermediate surgical video based on the three-dimensional surgical feature vector, and obtaining the probability that the surgical operation in the intermediate surgical video is each pre-set surgical operation; and, According to the probability of each surgical operation, the illegal operation in the intermediate surgical video is obtained; The surgical report of the surgical video is obtained by detecting the illegal operations in each intermediate surgical video in the target surgical video.

8. The electronic device according to claim 7, It is characterized in that The multiple continuous three-dimensional feature extraction layers are connected end to end, and any three-dimensional feature extraction layer includes a three-dimensional convolution layer and a pooling layer; The processor performs the three-dimensional feature extraction of each intermediate surgical video in sequence for multiple consecutive times using multiple consecutive three-dimensional feature extraction layers to obtain multiple three-dimensional surgical feature graphs of each intermediate surgical video, and is specifically configured as follows: For any 3D feature extraction layer, a 3D convolution layer in the 3D feature extraction layer is used to perform 3D feature extraction on a plurality of images to be processed corresponding to any intermediate surgical video, to obtain a plurality of first intermediate 3D feature maps, wherein the plurality of images to be processed are surgical video images in the intermediate surgical video or a plurality of second intermediate 3D feature maps output by a 3D feature extraction layer located in an upper layer of the 3D feature extraction layer; and, Using a pooling layer in the three-dimensional feature extraction layer to perform a downsampling operation on the plurality of first intermediate three-dimensional feature maps to obtain a plurality of second intermediate three-dimensional feature maps, wherein the sizes of the plurality of second intermediate three-dimensional feature maps are specified sizes; A second intermediate three-dimensional feature map obtained by a last three-dimensional feature extraction layer among the multiple continuous three-dimensional feature extraction layers is determined as the multiple three-dimensional surgical feature maps of the intermediate surgical video.

9. The electronic device according to claim 8, It is characterized in that The processor is further configured to: The multiple images to be processed are determined by: For any three-dimensional feature extraction layer, if the three-dimensional feature extraction layer is the first layer of the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are the surgical video images in the intermediate surgical video; or, If the three-dimensional feature extraction layer is not the first layer among the multiple continuous three-dimensional feature extraction layers, the multiple images to be processed for three-dimensional feature extraction by the three-dimensional feature extraction layer are multiple second intermediate three-dimensional feature maps output by the three-dimensional feature extraction layer located in the upper layer of the three-dimensional feature extraction layer.

10. The electronic device according to claim 7, It is characterized in that The processor is further configured to: Based on the sequence of the intermediate surgical videos in the target surgical video, multiple continuous three-dimensional feature extraction layers are used to perform three-dimensional feature extraction on each intermediate surgical video in sequence for multiple times. Before obtaining multiple three-dimensional surgical feature maps of each intermediate surgical video, the target surgical video is filtered by a filter to obtain a filtered target surgical video, and the filtered target surgical video is determined as the target surgical video.

Citation Information

Cited By

  • Method and system for automatic generation of surgical record based on dense video description

    CN120475215B