A video processing method and apparatus

Abstract

The application discloses a video processing method and device, and relates to the technical field of computers. A specific implementation manner of the method comprises the following steps: processing video data by using an optical flow algorithm to generate optical flow image data; taking the video data and the optical flow image data as spatial flow branch input and time flow branch input respectively, evaluating the importance of the video data by using an importance evaluation model to generate an importance score of the video data, wherein the importance evaluation model is constructed based on a dual-flow convolutional network, and the spatial flow branch and the time flow branch adopt a lightweight convolutional neural network; extracting important video data from the video data according to the importance scores of the video data, and performing specific video processing operations based on the important video data. The implementation manner can evaluate a video with unobvious foreground change, quickly process multiple video frames, and has simple steps, short time consumption and high efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a video processing method and apparatus. Background Technology

[0002] Assessing the importance of video frames, and measuring the significance of each frame, can aid in operations such as keyframe extraction, video segmentation, and cropping. Currently, there are two main approaches to assessing the importance of video frames: Approach 1 evaluates the importance of video frames based on characteristics such as target motion to extract keyframes; Approach 2 extracts video frame features using a deep classification network, focusing only on the current video frame.

[0003] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0004] Option 1 cannot evaluate videos with inconspicuous foreground changes, while Option 2 can only process a single video frame and cannot fully utilize the information of consecutive frames. Processing multiple video frames involves many steps, a complex process, and a long processing time. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a video processing method and apparatus that can evaluate videos with insignificant foreground changes, quickly process multiple video frames, and are simple, time-saving, and highly efficient.

[0006] To achieve the above objectives, a video processing method is provided according to one aspect of the present invention.

[0007] A video processing method includes: processing video data using an optical flow algorithm to generate optical flow image data; using the video data and the optical flow image data as inputs to a spatial flow branch and a temporal flow branch, respectively, evaluating the importance of the video data using an importance evaluation model to generate an importance score for the video data, wherein the importance evaluation model is constructed based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch employ lightweight convolutional neural networks; extracting important video data from each of the video data based on the importance scores of each video data, and performing specific video processing operations based on the important video data.

[0008] Optionally, the step of using the video data and the optical flow image data as inputs to the spatial flow branch and the temporal flow branch, respectively, and evaluating the importance of the video data through an importance evaluation model includes: inputting the video data and the optical flow image data into the spatial flow branch and the temporal flow branch of the importance evaluation model, respectively; the importance evaluation model is constructed based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch employ lightweight convolutional neural networks; obtaining the spatial information and temporal information of the video data from the outputs of the spatial flow branch and the temporal flow branch through global average pooling; and generating a score for evaluating the importance of the video data through a fully connected layer based on the spatial information and the temporal information.

[0009] Optionally, before evaluating the importance of the video data using the importance assessment model, the process includes: performing frame-level annotation on the video sample set to obtain the frame-level annotation score of the video sample set; and using the annotated video sample set as training samples to train the importance assessment model.

[0010] Optionally, the step of performing frame-level annotation on the video sample set to obtain the frame-level annotation score of the video sample set includes: using a panoramic segmentation network as a feature extractor to extract features from the video sample set and generate feature vectors of video frames in the video sample set; and calculating the frame-level annotation score of the video sample set based on the feature vectors of video frames in the video sample set using the inter-frame difference method.

[0011] Optionally, training the importance assessment model using the labeled video sample set as training samples includes: taking the video sample set as input and the frame-level annotation score of the video sample set as output, and training the importance assessment model.

[0012] Optionally, before processing the video data using the optical flow algorithm to generate optical flow image data, the process includes: grouping all the video frames to be evaluated to obtain one or more groups of the video data.

[0013] According to another aspect of the present invention, a video processing apparatus is provided.

[0014] A video processing apparatus includes: an optical flow image data generation module, used to process video data using an optical flow algorithm to generate optical flow image data; an importance assessment module, used to take the video data and the optical flow image data as spatial flow branch inputs and temporal flow branch inputs respectively, and evaluate the importance of the video data using an importance assessment model to generate an importance score for the video data, wherein the importance assessment model is constructed based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch employ lightweight convolutional neural networks; and based on the importance scores of each video data, important video data is extracted from each video data, and specific video processing operations are performed based on the important video data.

[0015] Optionally, the importance assessment module is further configured to: input the video data and the optical flow image data into the spatial flow branch and temporal flow branch of the importance assessment model, respectively, wherein the importance assessment model is constructed based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch employ lightweight convolutional neural networks; obtain the spatial information and temporal information of the video data from the outputs of the spatial flow branch and the temporal flow branch through global average pooling; and generate a score for assessing the importance of the video data through a fully connected layer based on the spatial information and temporal information.

[0016] Optionally, it also includes a model training module, used for: performing frame-level annotation on the video sample set to obtain the frame-level annotation score of the video sample set; and using the annotated video sample set as training samples to train the importance evaluation model.

[0017] Optionally, the model training module is further configured to: use the panoramic segmentation network as a feature extractor to extract features from the video sample set and generate feature vectors for video frames in the video sample set; and calculate the frame-level annotation score of the video sample set based on the feature vectors of the video frames in the video sample set using the inter-frame difference method.

[0018] Optionally, the model training module is further configured to: take the video sample set as input and the frame-level annotation score of the video sample set as output, and train the importance evaluation model.

[0019] Optionally, it also includes a video frame grouping module, used to: group all the video frames to be evaluated to obtain one or more groups of video data.

[0020] According to another aspect of the present invention, an electronic device is provided.

[0021] An electronic device includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the video processing method provided in the embodiments of the present invention.

[0022] According to another aspect of the present invention, a computer-readable medium is provided.

[0023] A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the video processing method provided in the embodiments of the present invention.

[0024] One embodiment of the above invention has the following advantages or beneficial effects: Optical flow algorithms are used to process video data to generate optical flow image data; the video data and optical flow image data are used as inputs to the spatial flow branch and temporal flow branch, respectively; the importance of the video data is evaluated using an importance assessment model to generate an importance score for the video data. The importance assessment model is constructed based on a two-stream convolutional network, and the spatial flow branch and temporal flow branch employ lightweight convolutional neural networks; based on the importance score of each video data point, important video data is extracted from each video data point, and specific video processing operations are performed based on the important video data. This approach can evaluate videos with insignificant foreground changes, quickly process multiple video frames, and is simple, time-efficient, and highly effective.

[0025] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0026] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0027] Figure 1 This is a schematic diagram of the main steps of a video processing method according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the architecture of an importance assessment model according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the frame-level annotation score curve of a video sample set according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the main modules of a video processing apparatus according to an embodiment of the present invention;

[0031] Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0032] Figure 6This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0033] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] Figure 1 This is a schematic diagram of the main steps of a video processing method according to an embodiment of the present invention.

[0035] like Figure 1 As shown, the video data importance assessment method of one embodiment of the present invention mainly includes the following steps S101 to S102.

[0036] Step S101: Process the video data using an optical flow algorithm to generate optical flow image data.

[0037] Video data can be in RGB image format, and optical flow image data can be obtained from RGB images using optical flow algorithms. Various commonly used optical flow algorithms can be used.

[0038] Before generating optical flow image data, all video frames to be evaluated can be grouped using optical flow algorithms to obtain one or more sets of video data.

[0039] Step S102: The video data and optical flow image data are used as inputs to the spatial flow branch and the temporal flow branch, respectively. The importance of the video data is evaluated by the importance evaluation model to generate an importance score for the video data. The importance evaluation model is based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch use lightweight convolutional neural networks.

[0040] Using video data and optical flow image data as inputs to the spatial flow branch and temporal flow branch, respectively, and evaluating the importance of the video data through an importance assessment model, the process can include: inputting the video data and optical flow image data into the spatial flow branch and temporal flow branch of the importance assessment model, respectively; the importance assessment model being constructed based on a dual-stream convolutional network; the spatial flow branch and temporal flow branch employing lightweight convolutional neural networks, preferably using a MobileNetv2 architecture; obtaining the spatial and temporal information of the video data from the outputs of the spatial flow branch and temporal flow branch through global average pooling; and generating a score for evaluating the importance of the video data through a fully connected layer based on the spatial and temporal information.

[0041] Before evaluating the importance of video data using an importance assessment model, the process may include: performing frame-level annotation on the video sample set to obtain frame-level annotation scores for the video sample set; and using the annotated video sample set as training samples to train the importance assessment model.

[0042] Performing frame-level annotation on a video sample set to obtain a frame-level annotation score for the video sample set can include: using a panoramic segmentation network as a feature extractor to extract features from the video sample set and generate feature vectors for the video frames in the video sample set; and calculating the frame-level annotation score of the video sample set based on the feature vectors of the video frames in the video sample set using the inter-frame difference method.

[0043] Using labeled video sample sets as training samples to train an importance assessment model can include: taking the video sample set as input and the frame-level annotation scores of the video sample set as output, and then training the importance assessment model.

[0044] Step S103: Based on the importance score of each video data, extract important video data from each video data set, and perform specific video processing operations based on the important video data. These specific video processing operations may include video segmentation and video editing. The specific methods for video segmentation and video editing can adopt common video segmentation and editing methods, which will not be detailed in this invention.

[0045] Figure 2 This is a schematic diagram of the architecture of an importance assessment model according to an embodiment of the present invention.

[0046] like Figure 2As shown, in one embodiment, an importance evaluation model including spatial and temporal branches is constructed based on a two-stream convolutional network and the MobileNetv2 architecture. Specifically, a two-stream convolutional network based on MobileNetv2 is built, which includes two branches: spatial and temporal. The input to the spatial branch is the RGB image (i.e., video data) of each video frame, responsible for extracting features representing the spatial information of the video frame. The input to the temporal branch is the optical flow image (i.e., optical flow image data) corresponding to each video frame, responsible for extracting features representing the temporal information of the video frame. The optical flow image contains the motion information of the foreground target in consecutive frames, and the corresponding optical flow image of the video frame can be obtained using an optical flow algorithm. Both the spatial and temporal branches adopt the MobileNetv2 architecture, selecting features of dimension [1×1×320] obtained through global average pooling to represent spatial and temporal information. The features of the two branches are concatenated to obtain the final feature of dimension [1×1×640]. The final feature is passed through a fully connected (fc) layer to obtain the importance score of the video frame. MobileNet is a lightweight deep neural network proposed by Google for embedded devices such as mobile phones. MobileNetv2 is the second generation, which is more efficient than the first generation. MobileNetv2 still uses the depthwise separable convolution in the first generation. The difference is that MobileNetv2 introduces residual structures and bottleneck layers.

[0047] In one embodiment, a panoramic segmentation network is used as a feature extractor to extract features from a video sample set and generate feature vectors for video frames within the set. The frame-level annotation score for the video sample set is then calculated based on these feature vectors using the inter-frame difference method. The annotated video sample set is then used as training samples to train an importance evaluation model. Specifically, the panoramic segmentation network can be a deep learning network based on an improved SpatialFlow panoramic segmentation algorithm. This network serves as the feature extractor for the video frames in the video sample set, generating feature vectors for each frame. The extracted features sufficiently represent the foreground and background information of the video frames. The absolute value of the inter-frame difference is calculated based on the feature vectors of the video frames using the inter-frame difference method, and this absolute value can be directly used as the frame-level annotation score for the video sample set. Higher similarity between video frames results in a lower frame-level annotation score. In calculating the frame-level annotation score of a video frame using the inter-frame difference method, three adjacent video frames can be used. Specifically, when calculating the score of the second frame, the absolute difference between the features of the third and second frames is calculated first, and then the absolute difference between the features of the second and first frames is calculated. The average of these two absolute values ​​is taken as the frame-level annotation score of the second frame. The frame-level annotation score curve of a video sample set according to an embodiment of this invention is shown below. Figure 3 As shown, the vertical axis corresponds to the score, and the horizontal axis corresponds to the video frame.

[0048] In one embodiment, a video sample set is used as input, and the frame-level annotation scores of the video sample set are used as output to train an importance assessment model. Specifically, the regression target of the importance score predicted by the importance assessment model is the frame-level annotation score of the video sample set; that is, the loss is calculated between the predicted importance score and the frame-level annotation score of the video sample set, and the network model is trained.

[0049] In one embodiment, the importance assessment model can process multiple video frames simultaneously. Before using the importance assessment model, all video frames to be evaluated can be grouped to obtain one or more sets of video data. Specifically, the batch_size parameter is set, which is the number of video frames included in each set of video data. Each set of video frames forms the input [batch_size, 3, H, W], where H, W, and 3 are the height, width, and number of channels of the video frame, respectively. Generally, batch_size can be set to 25, meaning that the importance scores of 25 video frames can be predicted in each forward inference.

[0050] Figure 4 This is a schematic diagram of the main modules of a video processing apparatus according to an embodiment of the present invention.

[0051] like Figure 4As shown, the importance assessment 400 of video data in one embodiment of the present invention mainly includes: an optical flow image data generation module 401 and an importance assessment module 402.

[0052] The optical flow image data generation module 401 is used to process video data using an optical flow algorithm to generate optical flow image data.

[0053] The importance assessment module 402 is used to take video data and optical flow image data as inputs to the spatial flow branch and the temporal flow branch, respectively, and evaluate the importance of the video data through the importance assessment model to generate an importance score for the video data. The importance assessment model is built based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch adopt lightweight convolutional neural networks.

[0054] The video processing module 403 is used to extract important video data from each video data according to the importance score of each video data, and to perform specific video processing operations based on the important video data.

[0055] In one embodiment, the importance assessment module is further configured to: input video data and optical flow image data into the spatial flow branch and temporal flow branch of the importance assessment model, respectively; the importance assessment model is constructed based on a dual-stream convolutional network, and the spatial flow branch and temporal flow branch employ lightweight convolutional neural networks; obtain the spatial information and temporal information of the video data from the outputs of the spatial flow branch and the temporal flow branch through global average pooling; and generate a score for assessing the importance of the video data through a fully connected layer based on the spatial information and temporal information.

[0056] In one embodiment, the system further includes a model training module, configured to: perform frame-level annotation on the video sample set to obtain frame-level annotation scores for the video sample set; and use the annotated video sample set as training samples to train an importance assessment model.

[0057] In one embodiment, the model training module is further configured to: use the panoramic segmentation network as a feature extractor to extract features from the video sample set and generate feature vectors for video frames in the video sample set; and calculate the frame-level annotation score of the video sample set based on the feature vectors of the video frames in the video sample set using the inter-frame difference method.

[0058] In one embodiment, the model training module is further configured to: take a video sample set as input and the frame-level annotation scores of the video sample set as output, and train an importance evaluation model.

[0059] In one embodiment, a video frame grouping module is further included, which is used to: group all the video frames to be evaluated to obtain one or more groups of video data.

[0060] Furthermore, the specific implementation details of the video processing device in the embodiments of the present invention have been described in detail in the above video processing method, so the details will not be repeated here.

[0061] Figure 5 An exemplary system architecture 500 is shown for which the video processing method or video processing apparatus of the present invention can be applied.

[0062] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0063] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0064] Terminal devices 501, 502, and 503 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0065] Server 505 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 501, 502, and 503 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device.

[0066] It should be noted that the video processing method provided in the embodiments of the present invention is generally executed by server 505, and correspondingly, the video processing device is generally located in server 505.

[0067] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0068] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing terminal devices or servers of the present invention. Figure 6The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0069] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0070] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0071] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0072] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0074] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including an optical flow image data generation module, an importance assessment module, and a video processing module. The names of these modules do not necessarily limit the module itself; for example, the optical flow image data generation module may also be described as "a module for processing video data using an optical flow algorithm to generate optical flow image data."

[0075] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to: process video data using an optical flow algorithm to generate optical flow image data; use the video data and optical flow image data as spatial flow branch inputs and temporal flow branch inputs, respectively, to evaluate the importance of the video data using an importance evaluation model, generating an importance score for the video data, wherein the importance evaluation model is constructed based on a two-stream convolutional network, and the spatial flow branch and temporal flow branch employ lightweight convolutional neural networks; extract important video data from each video data according to the importance score of each video data, and perform specific video processing operations based on the important video data.

[0076] According to the technical solution of this invention, video data is processed using an optical flow algorithm to generate optical flow image data. The video data and optical flow image data are used as inputs to the spatial flow branch and temporal flow branch, respectively. An importance evaluation model is used to assess the importance of the video data, generating an importance score. The importance evaluation model is constructed based on a two-stream convolutional network, and the spatial flow and temporal flow branches employ lightweight convolutional neural networks. Based on the importance score of each video data point, important video data is extracted from each video data point, and specific video processing operations are performed based on this important video data. This method can evaluate videos with insignificant foreground changes, quickly process multiple video frames, and is simple, time-efficient, and highly effective.

[0077] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video processing method, characterized in that, include: Optical flow algorithms are used to process video data to generate optical flow image data; The video data and the optical flow image data are used as inputs to the spatial flow branch and the temporal flow branch, respectively. The importance of the video data is evaluated using an importance assessment model to generate an importance score for the video data. The importance assessment model is built based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch use lightweight convolutional neural networks. Both the spatial flow branch and the temporal flow branch adopt the Mobilenetv2 structure, which introduces residual structures and bottleneck layers. Based on the importance score of each video data, important video data is extracted from each video data, and specific video processing operations are performed based on the important video data. The step of using the video data and optical flow image data as inputs to the spatial flow branch and temporal flow branch, respectively, and evaluating the importance of the video data through an importance evaluation model includes: inputting the video data and optical flow image data into the spatial flow branch and temporal flow branch of the importance evaluation model, respectively; obtaining the spatial information and temporal information of the video data from the outputs of the spatial flow branch and temporal flow branch through global average pooling; and generating a score for evaluating the importance of the video data through a fully connected layer based on the spatial information and temporal information. Before evaluating the importance of the video data using the importance assessment model, the process includes: performing frame-level annotation on the video sample set to obtain the frame-level annotation score of the video sample set; and using the annotated video sample set as training samples to train the importance assessment model.

2. The method according to claim 1, characterized in that, The step of performing frame-level annotation on the video sample set to obtain the frame-level annotation score of the video sample set includes: The panoramic segmentation network is used as a feature extractor to extract features from the video sample set and generate feature vectors for video frames in the video sample set. The frame-level annotation score of the video sample set is calculated based on the feature vectors of the video frames in the video sample set using the inter-frame difference method.

3. The method according to claim 1, characterized in that, The step of using the labeled video sample set as training samples to train the importance assessment model includes: The importance assessment model is trained by taking the video sample set as input and the frame-level annotation score of the video sample set as output.

4. The method according to claim 1, characterized in that, Before processing the video data using the optical flow algorithm to generate optical flow image data, the process includes: All video frames to be evaluated are grouped to obtain one or more sets of video data.

5. A video processing apparatus, characterized in that, include: The optical flow image data generation module is used to process video data using optical flow algorithms to generate optical flow image data; An importance assessment module is used to take the video data and the optical flow image data as inputs to the spatial flow branch and the temporal flow branch, respectively, and evaluate the importance of the video data through an importance assessment model to generate an importance score for the video data. The importance assessment model is constructed based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch employ lightweight convolutional neural networks. Both the spatial flow branch and the temporal flow branch adopt the MobileNetv2 structure, which incorporates residual structures and bottleneck layers. The video processing module is used to extract important video data from each of the video data according to the importance score of each video data, and to perform specific video processing operations based on the important video data; The importance assessment module is further configured to: input the video data and the optical flow image data into the spatial flow branch and temporal flow branch of the importance assessment model, respectively; the importance assessment model is constructed based on a two-stream convolutional network, and the spatial flow branch and the temporal flow branch employ lightweight convolutional neural networks; obtain the spatial information and temporal information of the video data from the outputs of the spatial flow branch and the temporal flow branch through global average pooling; and generate a score for assessing the importance of the video data through a fully connected layer based on the spatial information and temporal information. It also includes a model training module, used for: performing frame-level annotation on the video sample set to obtain the frame-level annotation score of the video sample set; and using the annotated video sample set as training samples to train the importance assessment model.

6. The apparatus according to claim 5, characterized in that, The model training module is also used for: The panoramic segmentation network is used as a feature extractor to extract features from the video sample set and generate feature vectors for video frames in the video sample set. The frame-level annotation score of the video sample set is calculated based on the feature vectors of the video frames in the video sample set using the inter-frame difference method.

7. The apparatus according to claim 5, characterized in that, The model training module is also used for: The importance assessment model is trained by taking the video sample set as input and the frame-level annotation score of the video sample set as output.

8. The apparatus according to claim 5, characterized in that, It also includes a video frame grouping module, used for: All video frames to be evaluated are grouped to obtain one or more sets of video data.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images