Laparoscopic surgery video processing method and electronic device

Abstract

The present application discloses a method and electronic device for processing laparoscopic surgery videos, belonging to the field of video processing technology. The method comprises: determining the histogram difference corresponding to each video frame in the laparoscopic surgery video, determining multiple video clips from the laparoscopic surgery video based on the histogram difference corresponding to each video frame, each video clip corresponding to a potential video frame, determining the kurtosis and skewness corresponding to each video clip based on the histogram difference corresponding to the video frames included in the multiple video clips, and determining multiple scene nodes in the laparoscopic surgery video based on the kurtosis and skewness corresponding to the multiple video clips. The present application can automatically mark multiple surgical scenes for the laparoscopic surgery video by analyzing the red component histogram of each video frame in the laparoscopic surgery video, thereby greatly simplifying the viewing process of the laparoscopic surgery video and facilitating the later production and editing of the laparoscopic surgery video.

Description

Technical Field

[0001] The present application relates to the technical field of video processing, and in particular to a method and electronic device for processing laparoscopic surgery videos. Background Art

[0002] At present, in digital operating rooms, the entire process of laparoscopic surgery can be recorded to obtain laparoscopic surgery videos. Laparoscopic surgery videos can help doctors review the surgical process, facilitate doctors to improve the surgical process and surgical techniques, and can also be used as medical teaching materials.

[0003] Normally, a fully recorded laparoscopic surgery video may take several hours or even more than ten hours. However, for doctors, only a few dozen minutes are truly of reference value. Therefore, when watching laparoscopic surgery videos, doctors need to frequently drag the progress bar to find the required content, which makes the process of watching the video very troublesome for doctors, and it is not convenient to produce and edit the laparoscopic surgery video later. Therefore, there is an urgent need for a laparoscopic surgery video processing method to accurately locate different scenes in the laparoscopic surgery video, thereby simplifying the video viewing process and facilitating the production and editing of the laparoscopic surgery video later. Summary of the invention

[0004] The present application provides a method for processing laparoscopic surgery videos and an electronic device, which can locate different scenes in laparoscopic surgery videos. The technical solution is as follows:

[0005] On the one hand, a method for processing a laparoscopic surgery video is provided, the method comprising:

[0006] Determine a histogram difference corresponding to each video frame in the laparoscopic surgery video, wherein the histogram difference refers to a difference in a red component histogram between the corresponding video frame and an adjacent previous video frame;

[0007] Based on the histogram difference corresponding to each video frame, a plurality of video segments are determined from the laparoscopic surgery video, each video segment corresponds to a potential video frame, and the potential video frame refers to a video frame in which a surgical scene has a potential change;

[0008] Determining the kurtosis and skewness corresponding to each video clip based on histogram differences corresponding to the video frames included in the multiple video clips;

[0009] Based on the kurtosis and skewness corresponding to the multiple video clips, multiple scene nodes in the laparoscopic surgery video are determined, and the multiple scene nodes are used to mark the surgical scenes in the laparoscopic surgery video.

[0010] In another aspect, an electronic device is provided, the electronic device comprising a processor, the processor being configured to:

[0011] Determine a histogram difference corresponding to each video frame in the laparoscopic surgery video, wherein the histogram difference refers to a difference in a red component histogram between the corresponding video frame and an adjacent previous video frame;

[0012] Based on the histogram difference corresponding to each video frame, a plurality of video segments are determined from the laparoscopic surgery video, each video segment corresponds to a potential video frame, and the potential video frame refers to a video frame in which a surgical scene has a potential change;

[0013] Determining the kurtosis and skewness corresponding to each video clip based on histogram differences corresponding to the video frames included in the multiple video clips;

[0014] Based on the kurtosis and skewness corresponding to the multiple video clips, multiple scene nodes in the laparoscopic surgery video are determined, and the multiple scene nodes are used to mark the surgical scenes in the laparoscopic surgery video.

[0015] On the other hand, a device for processing a laparoscopic surgery video is provided, the device comprising:

[0016] A first determination module is used to determine a histogram difference corresponding to each video frame in the laparoscopic surgery video, wherein the histogram difference refers to a difference in a red component histogram between the corresponding video frame and an adjacent previous video frame;

[0017] A second determination module is used to determine a plurality of video segments from the laparoscopic surgery video based on a histogram difference corresponding to each video frame, each video segment corresponds to a potential video frame, and the potential video frame refers to a video frame in which a surgical scene has a potential change;

[0018] A third determination module, configured to determine the kurtosis and skewness corresponding to each video segment based on the histogram differences corresponding to the video frames included in the plurality of video segments;

[0019] The fourth determination module is used to determine a plurality of scene nodes in the laparoscopic surgery video based on the kurtosis and skewness corresponding to the plurality of video clips, and the plurality of scene nodes are used to mark the surgical scenes in the laparoscopic surgery video.

[0020] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned laparoscopic surgery video processing method are implemented.

[0021] On the other hand, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned method for processing laparoscopic surgery videos.

[0022] The technical solution provided by this application can at least bring the following beneficial effects:

[0023] By analyzing the red component histogram of each video frame in the laparoscopic surgery video, the histogram difference corresponding to each video frame is determined. Based on the histogram difference corresponding to each video frame, multiple video clips in which the surgical scene has potential changes can be determined from the laparoscopic surgery video. Based on the kurtosis and skewness corresponding to the multiple video clips, the changes in the surgical scene in each video clip and whether the images before and after the surgical scene change are stable can be judged, thereby accurately screening multiple scene nodes in the laparoscopic surgery video according to the kurtosis and skewness characteristics corresponding to the video clips when the surgical scene changes. The method provided by the embodiment of the present application can automatically mark multiple surgical scenes for the laparoscopic surgery video, thereby greatly simplifying the viewing process of the laparoscopic surgery video and facilitating the later production and editing of the laparoscopic surgery video. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is a flow chart of a method for processing a laparoscopic surgery video provided in an embodiment of the present application;

[0026] Figure 2 is a flowchart of another method for processing laparoscopic surgery video provided in an embodiment of the present application;

[0027] Figure 3 is a flowchart of another method for processing laparoscopic surgery video provided in an embodiment of the present application;

[0028] Figure 4 is a flowchart of another method for processing laparoscopic surgery video provided in an embodiment of the present application;

[0029] Figure 5 It is a schematic diagram of an interface of a method for processing a laparoscopic surgery video provided in an embodiment of the present application;

[0030] Figure 6 It is a structural schematic diagram of a laparoscopic surgery video processing device provided in an embodiment of the present application;

[0031] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0033] Before explaining in detail the method for processing laparoscopic surgery videos provided in the embodiment of the present application, the application scenarios involved in the embodiment of the present application are first introduced.

[0034] Since a complete laparoscopic surgery may take several hours or even more than ten hours, and the laparoscopic surgery video obtained by recording the entire process of the laparoscopic surgery may also take several hours or even more than ten hours, when viewing the laparoscopic surgery video, in order to find the content that interests them, viewers often need to repeatedly drag the progress bar of the laparoscopic surgery video or watch the laparoscopic surgery video for a long time, which will take a lot of time to do some useless actions. The method provided in the embodiment of the present application can process the laparoscopic surgery video, and then determine the scene node in the laparoscopic surgery video. Through the scene node, different scenes of the laparoscopic surgery can be accurately located, simplifying the steps of video viewing and facilitating post-editing.

[0035] The method provided in the embodiments of the present application can be executed by any device with video processing function, for example, the device can be a PC (Personal Computer), a mobile phone, a PDA (Personal Digital Assistant), a Pocket PC (PPC), a tablet computer, a server, etc.

[0036] It should be noted that the application scenarios and execution entities described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new application scenarios and devices, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0037] Next, the method for processing laparoscopic surgery videos provided in the embodiment of the present application is explained in detail. Figure 1 This is a flowchart of a method for processing laparoscopic surgery videos provided in an embodiment of the present application. Figure 1 , the method comprises the following steps.

[0038] Step 101: Determine a histogram difference corresponding to each video frame in a laparoscopic surgery video, where the histogram difference refers to a difference in a red component histogram between the corresponding video frame and its adjacent previous video frame.

[0039] The main process of laparoscopic surgery includes anesthesia, perforation, endoscope removal, blood vessel treatment, lesion removal, complete hemostasis, endoscope removal, and resuscitation. The key to this laparoscopic surgery is blood vessel treatment, lesion removal, and complete hemostasis, and all these key steps are performed through minimally invasive holes. Therefore, the main scenes in the laparoscopic surgery video are accompanied by the presence of blood vessels and blood, and the main scene is mainly red. When preparing and finishing work, the overall scene of the operating room will appear in the laparoscopic surgery video. The overall scene of the operating room is mainly green. Therefore, the red component histogram of each video frame in the laparoscopic surgery video can be analyzed to determine the histogram difference corresponding to each video frame in the laparoscopic surgery video. Among them, there are many ways to determine the histogram difference corresponding to each video frame in the laparoscopic surgery video. Next, two of them will be introduced.

[0040] For the first implementation, please refer to Figure 2 For any video frame in the laparoscopic surgery video, determine the red component histogram of the video frame. Count the number of pixels in the red component histogram of the video frame whose red value is greater than the red threshold to obtain the number of red pixels corresponding to the video frame. Determine the absolute value of the difference between the number of red pixels corresponding to the video frame and the number of red pixels corresponding to the previous adjacent video frame of the video frame to obtain the histogram difference corresponding to the video frame.

[0041] In practical applications, the total number of pixels in each video frame of a laparoscopic surgery video is related to the resolution of the video. The greater the resolution of the laparoscopic surgery video, the greater the total number of pixels in each video frame, which will result in a greater number of pixels with red values ​​greater than the red threshold in the red component histogram of the video frame corresponding to the main scene of the laparoscopic surgery. Therefore, the number of red pixels obtained by the above statistics can be normalized to reduce the amount of calculation and increase the calculation speed.

[0042] In some embodiments, the number of red pixels corresponding to the video frame can be normalized based on the number of red pixels corresponding to the video frame and the total number of pixels of the video frame. That is, the number of red pixels corresponding to the video frame is divided by the total number of pixels of the video frame to obtain the normalized number of red pixels. After normalizing the number of red pixels corresponding to each video frame in the same manner, the absolute value of the difference between the normalized number of red pixels corresponding to the video frame and the normalized number of red pixels corresponding to the previous adjacent video frame of the video frame can be determined, and the absolute value of the difference can be determined as the histogram difference corresponding to the video frame.

[0043] The second implementation method is to determine the red component histogram of any video frame in the laparoscopic surgery video. Count the number of pixels in the red component histogram of the video frame whose red values ​​are multiple reference red values ​​to obtain multiple numbers of red pixels, and the reference red value is greater than the red threshold and less than or equal to the maximum red value in the red component histogram of the video frame. Determine the absolute value of the difference between the number of red pixels corresponding to the same reference red value in the video frame and the previous adjacent video frame of the video frame to obtain multiple absolute values ​​of the difference. The sum of the multiple absolute values ​​of the difference is determined as the histogram difference corresponding to the video frame.

[0044] For ease of understanding, the process of determining the histogram difference corresponding to the video frame in the second implementation manner described above can be represented by the following formula (1).

[0045]

[0046] In the above formula (1), D m represents the histogram difference corresponding to the mth video frame in the laparoscopic surgery video, r represents the reference red value, t represents the red threshold, T represents the maximum red value in the red component histogram of the video frame, H m (r) represents the number of pixels with red value r in the red component histogram of the mth video frame in the laparoscopic surgery video, H m-1 (r) represents the number of pixels with red value r in the red component histogram of the m-1th video frame in the laparoscopic surgery video.

[0047] Similar to the first implementation method mentioned above, when the resolution of the laparoscopic surgery video is relatively large, the number of multiple red value pixels obtained by the above statistics can be normalized to reduce the amount of calculation and improve the operation speed. In some embodiments, the number of multiple red pixels corresponding to the video frame can be normalized based on the number of multiple red pixels corresponding to the video frame and the total number of pixels of the video frame. That is, the number of multiple red pixels corresponding to the video frame is divided by the total number of pixels of the video frame to obtain the normalized number of multiple red pixels. After normalizing the number of multiple red pixels corresponding to each video frame in the same manner, the absolute value of the difference between the normalized number of red pixels corresponding to the same reference red value in the video frame and the previous adjacent video frame of the video frame can be determined to obtain multiple difference absolute values, and then the sum of the multiple difference absolute values ​​is determined as the histogram difference corresponding to the video frame.

[0048] Among them, in the above two implementations, the red threshold is set in advance, and in different situations, the red threshold can be adjusted according to different needs. And in the above second implementation, the reference red value is greater than the red threshold and less than or equal to the maximum red value in the red component histogram of the video frame, that is, the reference red value can be determined based on the red threshold and the maximum red value in the red component histogram of the video frame. For example, the red threshold is 200, and the maximum red value in the red component histogram of the video frame is 255. In this case, the reference red value can be all values ​​between 200 and 255.

[0049] In some embodiments, before determining the histogram difference corresponding to each video frame in the laparoscopic surgery video, the laparoscopic surgery video may be sampled at a certain frequency to determine the histogram difference corresponding to each video frame in the sampled laparoscopic surgery video.

[0050] As an example, the laparoscopic surgery video may be sampled at a reference frequency. The reference frequency is set in advance and may be adjusted according to different requirements in different situations. For example, the reference frequency may be 1 Hz (Hertz). In this case, the laparoscopic surgery video may be sampled at a frequency of 1 Hz, that is, one video frame of the laparoscopic surgery video is sampled every second.

[0051] Step 102: Based on the histogram difference corresponding to each video frame, multiple video segments are determined from the laparoscopic surgery video, each video segment corresponds to a potential video frame, and the potential video frame refers to a video frame with potential changes in the surgical scene.

[0052] In some embodiments, for any video frame in the laparoscopic surgery video, if the histogram difference corresponding to the video frame is greater than a second difference threshold, the video frame is taken as a potential video frame, and a segment consisting of multiple video frames adjacent to the video frame before and after in the laparoscopic surgery video is taken as a video segment.

[0053] If the histogram difference corresponding to the video frame is greater than the second difference threshold, it means that the number of red pixels corresponding to the video frame is significantly different from the number of red pixels corresponding to the previous video frame of the video frame, that is, the video frame may be a node where the scene changes during laparoscopic surgery. Therefore, the video frame can be used as a potential video frame, and a segment composed of multiple video frames adjacent to the video frame before and after in the laparoscopic surgery video can be used as a video segment.

[0054] It should be noted that the segment composed of multiple video frames adjacent to the front and back of the video frame may include the first Q video frames adjacent to the video frame, the last Q video frames adjacent to the video frame, and the video frame. Among them, Q is a value set in advance, and in different situations, it can also be adjusted according to different needs. For example, Q can be set to 30, that is, the first 30 video frames adjacent to the video frame, the last 30 video frames adjacent to the video frame, and the segment composed of the video frame in the laparoscopic surgery video are taken as a video segment.

[0055] Of course, the number of video frames before the video frame and the number of video frames after the video frame in the video segment may also be different. That is, the segment composed of multiple video frames adjacent to the video frame before and after may include the first Q video frames adjacent to the video frame, the last P video frames adjacent to the video frame, and the video frame. Among them, Q and P are different values, and Q and P are pre-set values, and can be adjusted according to different requirements in different situations. For example, Q can be set to 30 and P can be set to 25.

[0056] The second difference threshold is set in advance, for example, the second difference threshold may be 45. Moreover, in different situations, it may be adjusted according to different requirements.

[0057] After analyzing each video frame in the laparoscopic surgery video in the same way, multiple video clips can be obtained, and each video clip corresponds to a potential video frame. However, since the surgical steps in laparoscopic surgery are limited, the number of changes in the scene of the laparoscopic surgery is also limited, and the potential video frames corresponding to the multiple video clips are not all caused by changes in the scene of the laparoscopic surgery, but may also be caused by reasons such as the doctor's hand occlusion. Therefore, in some embodiments, based on the histogram difference corresponding to each potential video frame, the potential video frames can be sorted in order from high to low in the histogram difference, and then the video clips corresponding to the first Y potential video frames in the sorted results are obtained from the multiple video clips. That is, the video clips corresponding to the Y potential video frames with the largest histogram difference in the laparoscopic surgery video are obtained.

[0058] Here, Y is a value set in advance, and can be adjusted according to different requirements in different situations. For example, Y can be set to 50.

[0059] When the scene of a laparoscopic surgery video changes, there will be a large difference between the number of red pixels corresponding to a certain video frame in the laparoscopic surgery video and the number of red pixels corresponding to the previous video frame of the video frame. Therefore, by obtaining the video segments corresponding to the Y potential video frames with the largest histogram difference in the laparoscopic surgery video, that is, obtaining the Y potential video frames with the largest difference in the number of red pixels corresponding to the potential video frames in the laparoscopic surgery video and the number of red pixels corresponding to the previous video frame of the potential video frame, the scene nodes where the surgical scene has actually changed can be more accurately screened out, and the potential video frames that cause interference can be removed.

[0060] During the operation, the position of the laparoscope may need to be changed. Since the position of the laparoscope cannot be determined in one step, it needs to be adjusted repeatedly. In this case, the picture of the video frame in the laparoscopic surgery video will also change multiple times, so that multiple adjacent video clips may be determined from the laparoscopic surgery video in the above manner, but in fact, the surgical scene has only changed once. Therefore, the video clips obtained above can be screened again. Among them, there are many ways to screen the obtained video clips again, and two of them are introduced below.

[0061] For the first implementation, please refer to Figure 3 For any two adjacent video clips, determine the number of video frames between the potential video frames corresponding to the two video clips. If the number of video frames between the potential video frames corresponding to the two video clips is less than the interval number threshold, then discard the video clip with the smaller histogram difference of the potential video frames corresponding to the two video clips. If the number of video frames between the potential video frames corresponding to the two video clips is greater than or equal to the interval number threshold, then retain the two video clips.

[0062] If the number of video frames spaced between potential video frames corresponding to two adjacent video segments is less than the interval number threshold, it means that the two video segments are adjacent video segments, so the video segment with the smaller histogram difference of the potential video frames corresponding to the two video segments is discarded. If the number of video frames spaced between potential video frames corresponding to two adjacent video segments is greater than or equal to the interval number threshold, it means that the two video segments are not adjacent video segments, so the two video segments are retained.

[0063] In a second implementation, for any two adjacent video clips, it is determined whether the two video clips have overlapping video frames. If the two video clips have overlapping video frames, the video clip with the smaller histogram difference of the corresponding potential video frames in the two video clips is discarded. If the two video clips do not have overlapping video frames, the two video clips are retained.

[0064] If there are overlapping video frames between the two video clips, it means that the two video clips are adjacent video clips, so the video clip with the smaller histogram difference of the corresponding potential video frames in the two video clips is discarded. If there are no overlapping video frames between the two video clips, it means that the two video clips are not adjacent video clips, so the two video clips are retained.

[0065] In the case where the number of video frames contained in the video clip is large, the possibility of any two video clips overlapping is large. Even if the two video clips overlap, it cannot be said that the two video clips are adjacent video clips. Therefore, in some embodiments, if there are overlapping video frames between the two video clips, the number of overlapping video frames is determined. If the number of overlapping video frames is less than the overlapping number threshold, the two video clips are retained. If the number of overlapping video frames is greater than or equal to the overlapping number threshold, the video clip with the smaller histogram difference of the corresponding potential video frames in the two video clips is discarded. If there are no overlapping video frames between the two video clips, the two video clips are retained.

[0066] If there are overlapping video frames between the two video clips, and the number of overlapping video frames is less than the overlapping number threshold, it means that the two video clips are not adjacent video clips, so the two video clips are retained. If the number of overlapping video frames is greater than or equal to the overlapping number threshold, it means that the two video clips are adjacent video clips, so the video clip with the smaller histogram difference of the corresponding potential video frames in the two video clips is discarded. If there are no overlapping video frames between the two video clips, it also means that the two video clips are not adjacent video clips, so the two video clips are retained.

[0067] The interval number threshold is set in advance, for example, the interval number threshold may be 2. In different cases, it may be adjusted according to different requirements. The overlap number threshold is also set in advance, for example, the overlap number threshold may be 3. In different cases, it may be adjusted according to different requirements.

[0068] Step 103: Determine the kurtosis and skewness corresponding to each video segment based on the histogram differences corresponding to the video frames included in the multiple video segments.

[0069] In some embodiments, for any video segment among the multiple video segments, the kurtosis corresponding to the video segment is determined according to the following formula (2) based on the histogram difference corresponding to the video frames included in the video segment.

[0070]

[0071] In the above formula (2), K refers to the kurtosis corresponding to the video clip, n refers to the number of video frames contained in the video clip, and x i It refers to the histogram difference corresponding to the i-th video frame in the video clip. It refers to the average value of the histogram differences corresponding to all video frames in the video clip.

[0072] In some embodiments, for any video segment among the multiple video segments, based on the histogram difference corresponding to the video frames included in the video segment, the skewness corresponding to the video segment is determined according to the following formula (3).

[0073]

[0074] In the above formula (3), S refers to the skewness corresponding to the video clip, n refers to the number of video frames contained in the video clip, and x i It refers to the histogram difference corresponding to the i-th video frame in the video clip. It refers to the average value of the histogram differences corresponding to all video frames in the video clip.

[0075] The above method of determining the kurtosis and skewness corresponding to the video clip is only an example. In practical applications, the kurtosis and skewness corresponding to the video clip may also be determined in other ways.

[0076] Step 104: Based on the kurtosis and skewness corresponding to the multiple video clips, determine multiple scene nodes in the laparoscopic surgery video, and the multiple scene nodes are used to mark the surgical scenes in the laparoscopic surgery video.

[0077] Since the change of the surgical scene is a slow process, the histogram difference corresponding to each video frame in the video clip will first increase and then decrease and gradually become slow, which is close to the normal distribution. The kurtosis corresponding to the video clip is used to indicate the steepness of the distribution of the histogram difference corresponding to the video clip, and the skewness is used to indicate the distribution of the histogram difference corresponding to the video clip, wherein the skewness equals 0 for normal distribution, the skewness greater than 0 for right-skewed distribution, and the skewness less than 0 for left-skewed distribution. Therefore, the higher the kurtosis corresponding to the video clip, that is, the steeper the distribution of the histogram difference corresponding to the video clip, the more significant the change of the surgical scene in the video clip, the closer the skewness corresponding to the video clip is to 0, that is, the closer the distribution of the histogram difference corresponding to the video clip is to the normal distribution, which means that the picture before and after the surgical scene changes in the video clip is more stable. Based on this, for any video clip among the multiple video clips, it can be judged whether the potential video frame corresponding to the video clip is a scene node where the surgical scene changes based on the kurtosis and skewness corresponding to the video clip.

[0078] In some embodiments, video clips whose kurtosis is higher than a kurtosis threshold and whose absolute value of skewness is lower than a skewness threshold can be screened out from the multiple video clips, and multiple scene nodes in the laparoscopic surgery video can be determined based on potential video frames corresponding to the screened video clips.

[0079] As an example, the positions of potential video frames corresponding to the screened video clips can be directly determined as multiple scene nodes in the laparoscopic surgery video. That is, when only video clips with kurtosis higher than the kurtosis threshold and the absolute value of skewness lower than the skewness threshold are screened, the positions of potential video frames corresponding to each screened video clip can be directly determined as scene nodes in the laparoscopic surgery video.

[0080] During laparoscopic surgery, the screen may change temporarily due to external instrument access, and the multiple scene nodes determined based on kurtosis and skewness may include scene nodes where the screen changes temporarily. Since the scene nodes that really need to be determined are the nodes where the surgical scene changes due to the different positions of the laparoscope, as another example, the filtered video clips can be filtered again. That is, please refer to Figure 4 For any video clip selected, determine the difference in the red component histogram between the potential video frame corresponding to the video clip and the first video frame in the laparoscopic surgery video to obtain a first histogram difference, the first video frame is located before the potential video frame corresponding to the video clip and is M video frames away from the potential video frame corresponding to the video clip, M is an integer greater than 1. Determine the difference in the red component histogram between the potential video frame corresponding to the video clip and the second video frame in the laparoscopic surgery video to obtain a second histogram difference, the second video frame is located after the potential video frame corresponding to the video clip and is M video frames away from the potential video frame corresponding to the video clip. If the absolute value of the difference between the first histogram difference and the second histogram difference is higher than the first difference threshold, the position of the potential video frame corresponding to the video clip is used as a scene node in the laparoscopic surgery video. If the absolute value of the difference between the first histogram difference and the second histogram difference is lower than the first difference threshold, the position of the potential video frame corresponding to the video clip will not be used as a scene node in the laparoscopic surgery video, that is, the potential video frame corresponding to the video clip will be discarded.

[0081] If the absolute value of the difference between the first histogram difference and the second histogram difference is higher than the first difference threshold, it means that the number of pixel points of the red component of the M video frames before the potential video frame and the M video frames after the potential video frame is relatively different, that is, the potential video frame is a node where the surgical scene actually changes. In this case, the position of the potential video frame is used as a scene node in the laparoscopic surgery video. If the absolute value of the difference between the first histogram difference and the second histogram difference is lower than the first difference threshold, it means that the number of pixel points of the red component of the M video frames before the potential video frame and the M video frames after the potential video frame is relatively different, that is, the potential video frame is generated due to the connection of external instruments, etc. In this case, it means that the potential video frame is not a node where the surgical scene actually changes, so the potential video frame is discarded.

[0082] Among them, the first difference threshold is set in advance, for example, the first difference threshold can be 0.5. In different situations, it can also be adjusted according to different requirements. M is also a value set in advance, and in different situations, it can also be adjusted according to different requirements. For example, M can be set to 20, that is, the first video frame is located before the potential video frame corresponding to the video clip and is 20 video frames away from the potential video frame corresponding to the video clip, and the second video frame is located after the potential video frame corresponding to the video clip and is 20 video frames away from the potential video frame corresponding to the video clip.

[0083] In the embodiments of the present application, "higher than" can be understood as greater than, or greater than or equal to. When "higher than" is understood as greater than, "lower than" can be understood as less than or equal to. When "higher than" is understood as greater than or equal to, "lower than" can be understood as less than.

[0084] In some embodiments, when multiple scene nodes in a laparoscopic surgery video are determined, the multiple scene nodes can be marked in the playback progress bar of the laparoscopic surgery video. Figure 5 , Figure 5 This is a schematic diagram of the interface of the processed laparoscopic surgery video, in which five scene nodes are marked in the playback progress bar of the laparoscopic surgery video.

[0085] In an embodiment of the present application, by analyzing the red component histogram of each video frame in the laparoscopic surgery video, and then determining the histogram difference corresponding to each video frame, based on the histogram difference corresponding to each video frame, multiple video clips in which the surgical scene has potential changes can be determined from the laparoscopic surgery video, and based on the kurtosis and skewness corresponding to the multiple video clips, the change of the surgical scene in each video clip and whether the picture before and after the surgical scene changes can be judged. Therefore, according to the kurtosis and skewness characteristics corresponding to the video clip when the surgical scene changes, multiple scene nodes in the laparoscopic surgery video can be accurately screened out. The method provided by the embodiment of the present application can automatically mark multiple surgical scenes for the laparoscopic surgery video, thereby greatly simplifying the viewing process of the laparoscopic surgery video, and facilitating the later production and editing of the laparoscopic surgery video.

[0086] In addition, by obtaining the video segments corresponding to the Y potential video frames with the largest histogram difference in the laparoscopic surgery video, and determining whether the multiple video segments are adjacent video segments, the scene nodes where the surgical scene actually changes can be screened more accurately, and the video segments that cause interference can be removed. Moreover, some potential scene changes caused by external instrument access and other situations can also be screened to obtain more accurate scene nodes, thereby achieving accurate positioning of different scenes in the laparoscopic surgery video.

[0087] Figure 6 This is a schematic diagram of the structure of a laparoscopic surgery video processing device provided in an embodiment of the present application. The laparoscopic surgery video processing device can be implemented by software, hardware or a combination of both to become part or all of the laparoscopic surgery video processing device. Please refer to Figure 6 The device includes: a first determination module 601, a second determination module 602, a third determination module 603 and a fourth determination module 604.

[0088] The first determination module 601 is used to determine the histogram difference corresponding to each video frame in the laparoscopic surgery video, where the histogram difference refers to the difference between the red component histogram of the corresponding video frame and its adjacent previous video frame. The detailed implementation process refers to the corresponding content in the above-mentioned embodiments, which will not be repeated here.

[0089] The second determination module 602 is used to determine multiple video segments from the laparoscopic surgery video based on the histogram difference corresponding to each video frame, each video segment corresponds to a potential video frame, and the potential video frame refers to a video frame with potential changes in the surgical scene. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.

[0090] The third determination module 603 is used to determine the kurtosis and skewness corresponding to each video segment based on the histogram differences corresponding to the video frames included in the multiple video segments. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.

[0091] The fourth determination module 604 is used to determine multiple scene nodes in the laparoscopic surgery video based on the kurtosis and skewness corresponding to the multiple video clips, and the multiple scene nodes are used to mark the surgical scenes in the laparoscopic surgery video. The detailed implementation process refers to the corresponding content in the above embodiments, which will not be repeated here.

[0092] Optionally, the fourth determining module 604 includes:

[0093] A screening unit, used for screening out, from a plurality of video clips, a video clip whose kurtosis is higher than a kurtosis threshold and whose absolute value of skewness is lower than a skewness threshold;

[0094] The first determination unit is used to determine multiple scene nodes in the laparoscopic surgery video based on potential video frames corresponding to the screened video clips.

[0095] Optionally, the first determining unit is specifically configured to:

[0096] For any screened video clip, determine the difference of the red component histogram between the potential video frame corresponding to the video clip and the first video frame in the laparoscopic surgery video to obtain a first histogram difference, where the first video frame is located before the potential video frame and is M video frames away from the potential video frame, where M is an integer greater than 1;

[0097] Determine a difference in a red component histogram between the potential video frame and a second video frame in the laparoscopic surgery video to obtain a second histogram difference, the second video frame being located after the potential video frame and M video frames away from the potential video frame;

[0098] If the absolute value of the difference between the first histogram difference and the second histogram difference is higher than the first difference threshold, the position of the potential video frame is taken as a scene node in the laparoscopic surgery video.

[0099] Optionally, the third determining module 603 is used to:

[0100] For any video segment among the multiple video segments, based on the histogram difference corresponding to the video frames included in the video segment, the kurtosis corresponding to the video segment is determined according to the following formula;

[0101]

[0102] In the above formula, K refers to the kurtosis corresponding to the video clip, n refers to the number of video frames contained in the video clip, and xi It refers to the histogram difference corresponding to the i-th video frame in the video clip. It refers to the average value of the histogram differences corresponding to all video frames in the video clip.

[0103] Optionally, the third determining module 603 is further specifically configured to:

[0104] For any video segment among the multiple video segments, based on the histogram difference corresponding to the video frames included in the video segment, the skewness corresponding to the video segment is determined according to the following formula;

[0105]

[0106] In the above formula, S refers to the skewness corresponding to the video clip, n refers to the number of video frames contained in the video clip, and x i It refers to the histogram difference corresponding to the i-th video frame in the video clip. It refers to the average value of the histogram differences corresponding to all video frames in the video clip.

[0107] Optionally, the second determining module 602 is specifically configured to:

[0108] For any video frame in the laparoscopic surgery video, if the histogram difference corresponding to the video frame is greater than the second difference threshold, the video frame is taken as a potential video frame, and a segment consisting of multiple video frames adjacent to the video frame before and after in the laparoscopic surgery video is taken as a video segment.

[0109] Optionally, the first determining module 601 is specifically configured to:

[0110] For any video frame in the laparoscopic surgery video, determine a red component histogram of the video frame;

[0111] Counting the number of pixels whose red values ​​are greater than the red threshold in the red component histogram of the video frame to obtain the number of red pixels corresponding to the video frame;

[0112] The absolute value of the difference between the number of red pixels corresponding to the video frame and the number of red pixels corresponding to a previous adjacent video frame of the video frame is determined to obtain a histogram difference corresponding to the video frame.

[0113] Optionally, the first determining module 601 is further specifically configured to:

[0114] For any video frame in the laparoscopic surgery video, determine a red component histogram of the video frame;

[0115] Counting the number of pixels whose red values ​​in the red component histogram of the video frame are respectively a plurality of reference red values, to obtain a plurality of numbers of red pixels, wherein the reference red value is greater than a red threshold value and less than or equal to a maximum red value in the red component histogram of the video frame;

[0116] Determine an absolute value of a difference between the number of red pixels corresponding to the same reference red value in the video frame and a previous adjacent video frame of the video frame to obtain a plurality of absolute values ​​of the difference;

[0117] The sum of the absolute values ​​of the multiple differences is determined as the histogram difference corresponding to the video frame.

[0118] In an embodiment of the present application, by analyzing the red component histogram of each video frame in the laparoscopic surgery video, and then determining the histogram difference corresponding to each video frame, based on the histogram difference corresponding to each video frame, multiple video clips in which the surgical scene has potential changes can be determined from the laparoscopic surgery video, and based on the kurtosis and skewness corresponding to the multiple video clips, the change of the surgical scene in each video clip and whether the picture before and after the surgical scene changes can be judged. Therefore, according to the kurtosis and skewness characteristics corresponding to the video clip when the surgical scene changes, multiple scene nodes in the laparoscopic surgery video can be accurately screened out. The method provided by the embodiment of the present application can automatically mark multiple surgical scenes for the laparoscopic surgery video, thereby greatly simplifying the viewing process of the laparoscopic surgery video, and facilitating the later production and editing of the laparoscopic surgery video.

[0119] In addition, by obtaining the video segments corresponding to the Y potential video frames with the largest histogram difference in the laparoscopic surgery video, and determining whether the multiple video segments are adjacent video segments, the scene nodes where the surgical scene actually changes can be screened more accurately, and the video segments that cause interference can be removed. Moreover, some potential scene changes caused by external instrument access and other situations can also be screened to obtain more accurate scene nodes, thereby achieving accurate positioning of different scenes in the laparoscopic surgery video.

[0120] It should be noted that: the laparoscopic surgery video processing device provided in the above embodiment only uses the division of the above functional modules as an example when processing laparoscopic surgery videos. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the laparoscopic surgery video processing device provided in the above embodiment and the laparoscopic surgery video processing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0121] Figure 7700 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 700 includes a central processing unit (CPU) 701, a system memory 704 including a random access memory (RAM) 702 and a read-only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the central processing unit 701. The electronic device 700 also includes a basic input / output system (I / O system) 706 that helps transmit information between various devices in the computer, and a large-capacity storage device 707 for storing an operating system 713, an application program 714, and other program modules 715.

[0122] The basic input / output system 706 includes a display 708 for displaying information and an input device 709 such as a mouse and a keyboard for user inputting information. The display 708 and the input device 709 are connected to the central processing unit 701 through an input / output controller 710 connected to the system bus 705. The basic input / output system 706 may also include an input / output controller 710 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 710 also provides output to a display screen, a printer, or other types of output devices.

[0123] The mass storage device 707 is connected to the central processing unit 701 through a mass storage controller (not shown) connected to the system bus 705. The mass storage device 707 and its associated computer-readable media provide non-volatile storage for the electronic device 700. That is, the mass storage device 707 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.

[0124] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 704 and mass storage device 707 can be collectively referred to as memory.

[0125] According to various embodiments of the present application, the electronic device 700 can also be connected to a remote computer on the network through a network such as the Internet. That is, the electronic device 700 can be connected to the network 712 through the network interface unit 711 connected to the system bus 705, or the network interface unit 711 can also be used to connect to other types of networks or remote computer systems (not shown).

[0126] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0127] In some embodiments, a computer-readable storage medium is further provided, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for processing laparoscopic surgery videos in the above embodiment are implemented. For example, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0128] It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0129] It should be understood that all or part of the steps to implement the above embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-mentioned computer readable storage medium.

[0130] That is, in some embodiments, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned method for processing laparoscopic surgery videos.

[0131] It should be understood that the "at least one" mentioned herein refers to one or more, and "multiple" refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0132] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the laparoscopic surgery video involved in the embodiments of this application was obtained with full authorization.

[0133] The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing laparoscopic surgery video, characterized in that: The method comprises: Determine a histogram difference corresponding to each video frame in the laparoscopic surgery video, wherein the histogram difference refers to a difference in a red component histogram between the corresponding video frame and an adjacent previous video frame; Based on the histogram difference corresponding to each video frame, a plurality of video segments are determined from the laparoscopic surgery video, each video segment corresponds to a potential video frame, and the potential video frame refers to a video frame in which a surgical scene has a potential change; Determining the kurtosis and skewness corresponding to each video clip based on histogram differences corresponding to the video frames included in the multiple video clips; Based on the kurtosis and skewness corresponding to the multiple video clips, multiple scene nodes in the laparoscopic surgery video are determined, and the multiple scene nodes are used to mark the surgical scenes in the laparoscopic surgery video.

2. The method according to claim 1, characterized in that The determining of a plurality of scene nodes in the laparoscopic surgery video based on the kurtosis and skewness corresponding to the plurality of video clips comprises: Screening out video clips from the multiple video clips, the video clips having a kurtosis higher than a kurtosis threshold and an absolute value of a skewness lower than a skewness threshold; Based on the potential video frames corresponding to the screened video clips, multiple scene nodes in the laparoscopic surgery video are determined.

3. The method according to claim 2, characterized in that The determining of a plurality of scene nodes in the laparoscopic surgery video based on potential video frames corresponding to the screened video clips includes: For any screened video clip, determine the difference of the red component histogram between the potential video frame corresponding to the any video clip and the first video frame in the laparoscopic surgery video to obtain a first histogram difference, where the first video frame is located before the potential video frame and is M video frames away from the potential video frame, where M is an integer greater than 1; Determine a difference in a red component histogram between the potential video frame and a second video frame in the laparoscopic surgery video to obtain a second histogram difference, wherein the second video frame is located after the potential video frame and is M video frames away from the potential video frame; If the absolute value of the difference between the first histogram difference and the second histogram difference is higher than a first difference threshold, the position of the potential video frame is taken as a scene node in the laparoscopic surgery video.

4. The method according to any one of claims 1 to 3, characterized in that: Determining the kurtosis corresponding to each video segment based on histogram differences corresponding to video frames included in the multiple video segments includes: For any video segment among the multiple video segments, based on a histogram difference corresponding to a video frame included in the any video segment, determine the kurtosis corresponding to the any video segment according to the following formula; In the above formula, K refers to the kurtosis corresponding to any video clip, n refers to the number of video frames contained in any video clip, and x i refers to the histogram difference corresponding to the i-th video frame in any video clip, It refers to the average value of the histogram differences corresponding to all video frames in any video segment.

5. The method according to any one of claims 1 to 3, characterized in that: Determining the skewness corresponding to each video segment based on histogram differences corresponding to video frames included in the multiple video segments includes: For any video segment among the multiple video segments, based on a histogram difference corresponding to a video frame included in the any video segment, determine the skewness corresponding to the any video segment according to the following formula; In the above formula, S refers to the skewness corresponding to any video clip, n refers to the number of video frames contained in any video clip, and x i refers to the histogram difference corresponding to the i-th video frame in any video clip, It refers to the average value of the histogram differences corresponding to all video frames in any video segment.

6. The method according to claim 1, characterized in that The determining of a plurality of video segments from the laparoscopic surgery video based on the histogram difference corresponding to each video frame includes: For any video frame in the laparoscopic surgery video, if the histogram difference corresponding to any video frame is greater than the second difference threshold, then the any video frame is taken as a potential video frame, and a segment consisting of multiple video frames adjacent to and before the any video frame in the laparoscopic surgery video is taken as a video segment.

7. The method according to claim 1, characterized in that The determining of the histogram difference corresponding to each video frame in the laparoscopic surgery video includes: For any video frame in the laparoscopic surgery video, determine a red component histogram of the any video frame; Counting the number of pixels whose red values ​​are greater than a red threshold in a red component histogram of any video frame to obtain the number of red pixels corresponding to any video frame; An absolute value of a difference between the number of red pixels corresponding to any one of the video frames and the number of red pixels corresponding to a previous adjacent video frame of the any one of the video frames is determined to obtain a histogram difference corresponding to the any one of the video frames.

8. The method according to claim 1, characterized in that The determining of the histogram difference corresponding to each video frame in the laparoscopic surgery video includes: For any video frame in the laparoscopic surgery video, determine a red component histogram of the any video frame; Counting the number of pixel points whose red values ​​in the red component histogram of any video frame are respectively multiple reference red values ​​to obtain multiple numbers of red pixel points, wherein the reference red value is greater than a red threshold value and less than or equal to a maximum red value in the red component histogram of any video frame; Determine an absolute value of a difference between the number of red pixels corresponding to the same reference red value in any one of the video frames and a previous adjacent video frame of the any one of the video frames to obtain a plurality of absolute values ​​of the difference; The sum of the absolute values ​​of the multiple differences is determined as the histogram difference corresponding to any one of the video frames.

9. An electronic device, characterized in that: The electronic device comprises a processor, wherein the processor is configured to: Determine a histogram difference corresponding to each video frame in the laparoscopic surgery video, wherein the histogram difference refers to a difference in a red component histogram between the corresponding video frame and an adjacent previous video frame; Based on the histogram difference corresponding to each video frame, a plurality of video segments are determined from the laparoscopic surgery video, each video segment corresponds to a potential video frame, and the potential video frame refers to a video frame in which a surgical scene has a potential change; Determining the kurtosis and skewness corresponding to each video clip based on histogram differences corresponding to the video frames included in the multiple video clips; Based on the kurtosis and skewness corresponding to the multiple video clips, multiple scene nodes in the laparoscopic surgery video are determined, and the multiple scene nodes are used to mark the surgical scenes in the laparoscopic surgery video.

10. The electronic device according to claim 9, characterized in that: The processor is further configured to: Screening out video clips from the multiple video clips, the video clips having a kurtosis higher than a kurtosis threshold and an absolute value of a skewness lower than a skewness threshold; Based on the potential video frames corresponding to the screened video clips, multiple scene nodes in the laparoscopic surgery video are determined.

Images