Video processing method, computer device and computer readable storage medium

Abstract

The application discloses a video processing method, a computer device and a computer readable storage medium. The video processing method comprises the following steps: calculating the average gray value of each frame of a target video to obtain a gray value sequence; inputting the gray value sequence into a time sequence model to obtain a predicted category corresponding to each frame of the target video, wherein the predicted category comprises an information category and a non-information category, and the time sequence model is a deep learning model; dividing the target video into an information segment and a non-information segment according to the predicted category corresponding to each frame of the target video; and removing the non-information segment of the target video. The application can reduce the video memory resource required when the time sequence model is calculated, thereby supporting longer time sequence surgical video processing and making the modeling capability of the time sequence model for long-time continuous segments better.

Description

Technical Field

[0001] This application relates to the field of interventional surgery video processing, and more particularly to video processing methods, computer equipment, and computer-readable storage media. Background Technology

[0002] In the current field of interventional surgical video processing, the automatic identification of "effective angiographic segments" still faces significant challenges. Most current work relies on manual frame-by-frame annotation to filter out clinically valuable image segments, a process that is not only inefficient but also highly subjective and repetitive. Although some automated methods have been proposed to assist in identification, their performance in real-world scenarios still falls short of the requirements for high-precision clinical applications.

[0003] Existing automatic identification methods mainly include the following categories: (1) Inter-frame difference method: Detects whether there is motion in the picture by calculating the pixel difference between consecutive frames; (2) Optical flow method and background modeling technology: Identifying active regions in videos based on inter-frame motion information; (3) Similarity-based loop detection: using image histograms, structural similarity (SSIM) and other indicators to identify content repetition or static segments.

[0004] While these existing technologies are effective in general video analysis tasks, capable of identifying some still frames or repetitive content, they suffer from high false positive and false negative rates when dealing with medical video processing, particularly in typical scenarios such as repeated playback of grayscale angiography images during interventional surgery where there are subtle dynamic changes, minor variations in the brightness of medical device interfaces and parameter windows, and overall image brightness, but the structure remains largely consistent. This is because existing technologies tend to misclassify real moving content as static frames in low-dynamic scenes, struggle to capture repetitive segments with minimal visual differences, and cannot analyze the relationship between non-information segments and context from a temporal perspective.

[0005] With the development of artificial intelligence, deep learning models (such as Convolutional Neural Networks (CNN), Long Short-Term Memory Networks (LSTM), and Visual Transformers) have demonstrated excellent performance in image and video classification and segmentation tasks, and are widely used in medical image analysis. However, the automatic identification of loop segments and relatively static segments in interventional surgical videos still faces the following core bottlenecks: (1) The lack of standardized public datasets and annotation specifications limits the training and validation of algorithms on large-scale, reusable data; (2) Manual annotation is costly, inefficient, and requires a high level of medical knowledge, making it difficult to construct a large-scale, high-quality training set; (3) Long-term video processing is limited by video memory resources. When inputting long video segments, deep learning models are prone to "video memory exhaustion", which limits the model's ability to model long-term continuous segments. Summary of the Invention

[0006] The purpose of this application is to provide a video processing method, computer device, and computer-readable storage medium that can save video memory resources and improve the ability of deep learning models to model long-term continuous segments when removing non-information segments from surgical videos.

[0007] To achieve the above objectives, this application provides a video processing method for removing non-information segments from surgical videos, comprising: The average grayscale value of each frame of the target video is calculated to obtain a grayscale average value sequence; The grayscale average value sequence is input into a time series model to obtain the predicted category corresponding to each frame of the target video. The predicted category includes informational and non-informational categories. The time series model is a deep learning model. Based on the predicted category corresponding to each frame of the target video, the target video is divided into information segments and non-information segments; Remove the non-information segments from the target video.

[0008] Since the grayscale average sequence includes the temporal feature information of the target video, this application can divide the target video into information segments and non-information segments based on the corresponding grayscale average sequence. Compared with technical solutions that divide the target video based on its frame sequence, this application can reduce the memory resources required for temporal model calculations, thereby supporting the processing of surgical videos with longer time sequences and improving the modeling ability of temporal models for long-term continuous segments. Furthermore, because this application utilizes the grayscale average sequence of the target video for calculation, it consumes fewer resources and can be deployed on ordinary hospital terminals or edge devices, resulting in high availability.

[0009] Optionally, before removing each of the non-information segments from the target video, the method further includes: The segments in the target video whose duration is less than a first preset time value are merged into adjacent segments.

[0010] Optionally, dividing the target video into information segments and non-information segments according to the predicted category corresponding to each of the frames includes: Based on the prediction category corresponding to each frame of the target video, determine the start and end timestamps of each non-information segment; Based on the start timestamp and the end timestamp, the target video is divided into non-information segments.

[0011] Optionally, the non-information class includes static classes and cyclic classes, and the non-information fragment includes static fragments and cyclic fragments.

[0012] Optionally, the time series model is a 1D U-net model.

[0013] Before calculating the average grayscale value of each frame of the target video to obtain the average grayscale value sequence, the following steps are also included: Acquire surgical video data and assign corresponding category labels to the frame icons of each surgical video data. Generate several simulated grayscale value sequences, and assign the corresponding category label to the sequence value of each simulated grayscale value sequence; The time series model is pre-trained using the simulated grayscale value sequences described above; The time series model is fine-tuned using the surgical video data.

[0014] Optionally, the simulated grayscale value sequence includes a simulated static sequence; The generation of the simulated static sequence includes: Generate a first sequence, wherein each sequence value in the first sequence corresponds to the same feature value; Perturbation values ​​are added to each sequence value of the first sequence to obtain the simulated static sequence.

[0015] Optionally, the simulated grayscale value sequence includes a simulated cyclic sequence; The generation of the simulated cyclic sequence includes: Randomly generate a second sequence; The second sequence is copied several times, and perturbation values ​​are added to each sequence value of the second sequence. The simulated cyclic sequence is obtained by merging each of the second sequences.

[0016] Optionally, the pre-training of the time series model using each of the simulated grayscale value sequences includes: The simulated grayscale value sequence is input into a time series model to obtain the predicted category corresponding to each sequence value of the simulated grayscale value sequence; The first loss function value is calculated based on the category label corresponding to each sequence value of the predicted category and the simulated gray value sequence; The time series model is optimized based on the first loss function value.

[0017] Optionally, fine-tuning the time-series model using the surgical video data includes: The third sequence is obtained by calculating the average grayscale value of each frame of each surgical video data. The third sequence is input into the time series model to obtain the predicted category corresponding to each sequence value of the third sequence; The second loss function value is calculated based on the category label corresponding to each sequence value of the predicted category and the third sequence; The time series model is optimized based on the value of the second loss function.

[0018] To achieve the above objectives, this application also provides a computer device, including a memory and a processor; The memory is connected to the processor. The memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the above-described video processing method or the above-described training method.

[0019] To achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the video processing method or the training method described above. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the video processing method according to an embodiment of this application.

[0021] Figure 2 This is a partial flowchart illustrating the video processing method according to an embodiment of this application.

[0022] Figure 3 This is another flowchart illustrating the video processing method according to an embodiment of this application.

[0023] Figure 4 This is a schematic diagram of the user interface of the surgical video editing software according to an embodiment of this application.

[0024] Figures 5 to 9 This is a partial flowchart of the video processing method according to an embodiment of this application.

[0025] Figure 10 A schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0026] To explain in detail the technical content, structural features, objectives and effects of this application, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0027] In the field of smart healthcare, various surgical videos are constantly being collected and used for business scenarios such as interventional video compression, summary extraction, knowledge archiving, and clinical review. Generally, the acquired surgical videos include many informational segments (i.e., segments containing valid information, such as segments of the normal surgical procedure) and non-informational segments (i.e., segments that do not contain valid information, such as still or looping segments). The non-informational segments are not helpful in handling the aforementioned business scenarios and consume a significant amount of memory. Therefore, after acquiring the surgical videos, it is necessary to remove the non-informational segments. However, surgical videos are typically quite long (e.g., 2 hours long with up to 200,000 frames). If the surgical video is directly divided using current technology, the number of frames to process is excessive, resulting in slow processing speed. Furthermore, in some cases, it's necessary to extract longer frame sequences to determine if non-information segments exist (e.g., inputting 300 frames at once to a deep learning model). This often leads to insufficient GPU memory, preventing deep learning models from processing long videos and limiting their ability to model long, continuous segments. This results in the following drawbacks: the model can only handle pixel-level repetitions and cannot truly understand semantic-level loops; it is insensitive to contrast grayscale images, leading to misjudgments or missed judgments; it cannot determine whether the content is repeated by the system (e.g., segments where the device is stuck in an infinite loop); and it can only recognize extremely static states and cannot identify "low-dynamic" or "pseudo-dynamic" frame sequences (e.g., the video image moves only slightly but is essentially still).

[0028] This application discovers that non-information segments and information segments in surgical videos differ in their grayscale value sequences. Therefore, by capturing the temporal characteristics of the grayscale value sequences, non-information segments and information segments can be distinguished. Specifically, this application discloses the following embodiments: Example 1 Please see Figure 1 This application discloses a video processing method for removing non-information segments from surgical videos, including: S1, calculate the average grayscale value of each frame of the target video to obtain the average grayscale value sequence, and use the average grayscale value as the image feature of each frame.

[0029] Optionally, before step 1, the process further includes preprocessing the target video, such as extracting frames from the target video at a frame rate of 1fps or higher, and performing grayscale normalization on the target video.

[0030] S3 inputs the grayscale average value sequence into the temporal model to obtain the predicted category corresponding to each frame of the target video. The predicted category includes informational and non-informational categories. The temporal model is a deep learning model.

[0031] Specifically, the time series model is a 1D U-net model. Of course, it is not limited to this.

[0032] S5: Based on the predicted categories corresponding to each frame of the target video, the target video is divided into information segments and non-information segments.

[0033] Please see Figure 2 Specifically, step S5 includes: S51, based on the prediction category corresponding to each frame of the target video, determine the start and end timestamps of each non-information segment.

[0034] S52, based on the start timestamp and end timestamp, divide the target video into non-information segments.

[0035] It is understandable that when non-information segments are identified, the remaining segments of the target video are information segments. Therefore, it is also possible to divide the target video into information segments by defining the start and end timestamps of each information segment, and then, based on these timestamps, divide the target video into corresponding information segments, leaving the remaining segments as non-information segments.

[0036] Specifically, non-information categories include static and looping categories, and non-information segments include static segments and looping segments. Static segments are segments where the image remains still, and their grayscale average value typically remains relatively constant. Looping segments are segments that are played repeatedly, and their grayscale average value typically shows a set of grayscale average values ​​appearing consecutively. Informational segments, however, behave differently in their grayscale average value sequence compared to static and looping segments; for example, they might exhibit a more random set of grayscale average values. Therefore, different types of segments correspond to different grayscale average value characteristics, and the grayscale average value sequence corresponding to the target video can be used to classify the target video.

[0037] S7 removes non-information segments from the target video.

[0038] It is understandable that after completing step S5, step S7 may not necessarily be executed; other steps could also be performed.

[0039] Specifically, the information segments are spliced ​​together to obtain the target video after removing the non-information segments.

[0040] Optionally, the target video with non-information segments removed can be saved to a local or cloud database.

[0041] Please see Figure 3 Specifically, before step S7, the following is also included: S6, merge segments in the target video whose duration is less than a first preset time value into adjacent segments. This application finds that segments with excessively short durations may be due to misidentification by the timing model; therefore, segments with durations less than the first preset time value can be merged into the preceding or following adjacent segment. It can be understood that if the duration of the first segment of the target video is less than the first preset time value, then that segment is merged into the next adjacent segment. If the duration of the last segment of the target video is less than the first preset time value, then that segment is merged into the preceding adjacent segment.

[0042] Please see Figure 4 In a specific example, the obtained start and end timestamps can be displayed on the user's video editing software, allowing the user to refer to these timestamps to determine the parts of the target video that need to be removed. This application assists users in removing non-information segments from the target video by dividing the video into start and end timestamps for each information segment or non-information segment, thereby improving the efficiency of removing non-information segments from the target video.

[0043] Since the grayscale average sequence includes the temporal feature information of the target video, this application can divide the target video into information segments and non-information segments based on the corresponding grayscale average sequence. Compared with technical solutions that divide the target video based on its frame sequence, this application can reduce the memory resources required for temporal model calculations, thereby supporting the processing of surgical videos with longer time sequences and improving the modeling ability of temporal models for long-term continuous segments. Furthermore, since this application utilizes the grayscale average sequence of the target video for calculation, it consumes fewer resources and can be deployed on ordinary hospital terminals or edge devices, resulting in high availability.

[0044] To obtain the aforementioned time-series model through training, a large number of surgical videos are typically required. However, in reality, relevant surgical video resources are scarce, and there is a lack of standardized public datasets and annotation specifications, limiting the training and validation of deep learning algorithms on large-scale, reusable data. Manually annotating surgical videos is costly, inefficient, and requires a high level of medical knowledge, making it difficult to construct large-scale, high-quality training sets. Therefore, in this field, it is common to encounter situations where the amount of data in the dataset is insufficient to meet the training requirements of deep learning models.

[0045] For this, please refer to Figure 5 Specifically, the procedure before step S1 includes: S11, acquire surgical video data and assign corresponding category labels to the frame icons of each surgical video data.

[0046] S12 generates several simulated grayscale value sequences and assigns corresponding category labels to the sequence values ​​of each simulated grayscale value sequence.

[0047] It is understandable that there is no specific order between steps S11 and S12; step S12 can be executed first, followed by step S11.

[0048] Please see Figure 6 More specifically, the simulated grayscale value sequence includes a simulated still sequence, and the generation of the simulated still sequence includes: S121, Generate the first sequence, where each sequence value in the first sequence corresponds to the same feature value.

[0049] More specifically, a random algorithm is used to generate the time length and frame rate (FPS) corresponding to the first sequence, and the length of the first sequence is determined based on the time length and frame rate. The first sequence is then generated based on its length. The time length of the first sequence is within a first preset range (e.g., 2 to 10 seconds), and the frame rate is within a second preset range (e.g., 30 to 60 frames per second).

[0050] S122, add perturbation values ​​to each sequence value of the first sequence to obtain a simulated static sequence. By adding small perturbations (such as Gaussian perturbations), the scene of the device being stationary or the picture not changing in real video can be simulated, making the simulation more realistic.

[0051] Please see Figure 7 More specifically, the simulated grayscale value sequence includes a simulated cyclic sequence, and the generation of the simulated cyclic sequence includes: S123, randomly generate the second sequence.

[0052] S124, the second sequence is copied several times, and perturbation values ​​are added to each sequence value of each second sequence. The perturbation values ​​can be achieved by generating additive Gaussian noise and brightness perturbation, to simulate phenomena such as parameter interface refresh and repeated playback of key steps commonly seen in surgical videos.

[0053] Optionally, the difference between the upper and lower limits of the disturbance value is less than the maximum preset difference, such as the difference between the upper and lower limits of the disturbance value being less than 20.

[0054] S125, merge each second sequence to obtain a simulated cyclic sequence.

[0055] It is understandable that steps S121 to S122 and steps S123 to S125 have no specific order. Alternatively, steps S123 to S125 can be executed first, followed by steps S121 to S122.

[0056] In the process of generating simulated data, this application does not need to generate an entire image, but only needs to generate a simulated grayscale value sequence according to actual needs. Therefore, the simulated grayscale value sequence generated by this application has good controllability, structure and diversity, which can closely resemble the loop and static patterns in real intervention videos and alleviate the dilemma of relying on manual annotation.

[0057] S13, pre-train the time series model using each simulated gray value sequence.

[0058] Please see Figure 8 More specifically, step S13 includes: S131, input the simulated gray value sequence into the time series model to obtain the predicted category corresponding to each sequence value of the simulated gray value sequence.

[0059] S132, calculate the first loss function value based on the category label corresponding to each sequence value of the predicted category and the simulated gray value sequence.

[0060] More specifically, the first loss value is calculated using the cross-entropy loss function. The labels output by the time series model can be 0 (representing the information class mentioned above), 1 (representing the cyclic class mentioned above), and 2 (representing the static class mentioned above), but are not limited to these.

[0061] S133, optimize the time series model based on the first loss function value.

[0062] Among them, the optimization function used to optimize the time series model can be Adam, and the learning rate can be set to 0.001. The learning rate optimization method can use the warm-up strategy.

[0063] Optionally, steps S131 to S133 are repeated until the first loss function value reaches the convergence condition. The convergence condition includes the training epoch reaching a preset training epoch value and / or the first loss function value being less than a first convergence preset value.

[0064] S14, fine-tuning the time series model using surgical video data.

[0065] Please see Figure 9 More specifically, step S14 includes: S141, calculate the average grayscale value of each frame of each surgical video data to obtain the third sequence.

[0066] Optionally, before performing step S141, data augmentation may be performed on each frame of the surgical video data.

[0067] S142, input the third sequence into the time series model to obtain the predicted category corresponding to each sequence value of the third sequence.

[0068] S143, calculate the second loss function value based on the predicted category and the category label corresponding to each sequence value of the third sequence.

[0069] In a specific example, the cross-entropy loss function can be used to calculate the second loss value. The labels output by the time series model can be 0 (representing the information class mentioned above), 1 (representing the cyclic class mentioned above), and 2 (representing the static class mentioned above), but are not limited to these.

[0070] S144, optimize the time series model based on the value of the second loss function.

[0071] In a specific example, the optimization function used to optimize the time series model can be Adam, and the learning rate can be set to 0.0001, or the learning rate optimization method can be omitted.

[0072] Optionally, steps S141 to S144 are repeated until the second loss function value reaches the convergence condition. The convergence condition includes the training epoch reaching a preset training epoch value and / or the second loss function value being less than a second convergence preset value.

[0073] The simulated data in this application is a grayscale value sequence, which is relatively easy to simulate and can therefore be generated in large quantities, thus solving the problem of insufficient real data. Furthermore, this application uses a large amount of simulated data to pre-train the time-series model, and then fine-tunes it on smaller amounts of real surgical videos, which helps improve the generalization ability and robustness of the deep learning model. Experimental results show that this application can handle ultra-long interventional videos exceeding 2 hours in length and with up to 200,000 frames, achieving high training and inference speeds while maintaining recognition accuracy.

[0074] Example 2 Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 10 As shown, the computer device may include a processor 201 and a memory 202. The memory 202 is connected to the processor 201 and is used to store computer programs. The processor 201 calls the computer programs to cause the computer device to execute the aforementioned video processing method. Furthermore, the computer device may also include at least one communication bus 203. The communication bus 203 is used to enable communication between components. The memory 202 may be a high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0075] Example 3 This application also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute the video processing method described above.

[0076] Example 4 This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned video processing method.

[0077] It should be understood that, in the embodiments of this application, the processor may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0078] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to computer program instructions. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0079] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0080] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the scope of this application shall still fall within the scope of this application.

Claims

1. A video processing method for removing non-informational segments from surgical videos, characterized in that, include: The average grayscale value of each frame of the target video is calculated to obtain the average grayscale value sequence; The grayscale average value sequence is input into a time series model to obtain the predicted category corresponding to each frame of the target video. The predicted category includes informational and non-informational categories. The time series model is a deep learning model. Based on the predicted category corresponding to each frame of the target video, the target video is divided into information segments and non-information segments; Remove the non-information segments from the target video.

2. The video processing method as described in claim 1, characterized in that, The process further includes, prior to removing each of the non-information segments from the target video: The segments in the target video whose duration is less than a first preset time value are merged into adjacent segments.

3. The video processing method as described in claim 1, characterized in that, The step of dividing the target video into information segments and non-information segments according to the predicted category corresponding to each frame includes: Based on the prediction category corresponding to each frame of the target video, determine the start and end timestamps of each non-information segment; Based on the start timestamp and the end timestamp, the target video is divided into non-information segments.

4. The video processing method as described in claim 1, characterized in that, The non-information classes include static classes and cyclic classes, and the non-information fragments include static fragments and cyclic fragments.

5. The video processing method as described in claim 1, characterized in that, The time series model is the 1D U-net model.

6. The video processing method as described in claim 1, characterized in that, Before calculating the average grayscale value of each frame of the target video to obtain the average grayscale value sequence, the following steps are also included: Acquire surgical video data and assign corresponding category labels to the frame icons of each surgical video data. Generate several simulated grayscale value sequences, and assign the corresponding category label to the sequence value of each simulated grayscale value sequence; The time series model is pre-trained using the simulated grayscale value sequences described above; The time series model is fine-tuned using the surgical video data.

7. The video processing method as described in claim 6, characterized in that, The simulated grayscale value sequence includes a simulated static sequence; The generation of the simulated static sequence includes: Generate a first sequence, wherein each sequence value in the first sequence corresponds to the same feature value; Perturbation values ​​are added to each sequence value of the first sequence to obtain the simulated static sequence.

8. The video processing method as described in claim 6 or 7, characterized in that, The simulated grayscale value sequence includes a simulated cyclic sequence; The generation of the simulated cyclic sequence includes: Randomly generate a second sequence; The second sequence is copied several times, and perturbation values ​​are added to each sequence value of the second sequence. The simulated cyclic sequence is obtained by merging each of the second sequences.

9. The video processing method as described in claim 6, characterized in that, The pre-training of the time series model using the simulated grayscale value sequences includes: The simulated grayscale value sequence is input into a time series model to obtain the predicted category corresponding to each sequence value of the simulated grayscale value sequence; The first loss function value is calculated based on the category label corresponding to each sequence value of the predicted category and the simulated gray value sequence; The time series model is optimized based on the first loss function value.

10. The video processing method as described in claim 6, characterized in that, The step of fine-tuning the time-series model using the surgical video data includes: The third sequence is obtained by calculating the average grayscale value of each frame of each surgical video data. The third sequence is input into the time series model to obtain the predicted category corresponding to each sequence value of the third sequence; The second loss function value is calculated based on the category label corresponding to each sequence value of the predicted category and the third sequence; The time series model is optimized based on the value of the second loss function.

11. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the video processing method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the video processing method according to any one of claims 1 to 10.

Images