Video processing method and device, electronic device and computer-readable storage medium

Abstract

The present application discloses a video processing method and apparatus, an electronic device, and a computer-readable storage medium. The method comprises: obtaining a video to be processed; obtaining a segmentation probability of each video frame of the video to be processed based on the video features of the video to be processed, wherein the segmentation probability represents the probability of a video frame being an endpoint of a video segment, wherein the endpoints include a start frame and an end frame; selecting two or more video frames from the video to be processed whose segmentation probability is greater than or equal to a segmentation threshold, wherein the two or more video frames include a first video frame and a second video frame with adjacent timestamps; and intercepting a video segment between the first video frame and the second video frame from the video to be processed to obtain a target video segment.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] By segmenting the video to be processed, we can separate the video segments belonging to different categories in the video to be processed. Then, we can perform subsequent processing based on the segmented video segments, such as video splicing, video editing, and video synthesis. Therefore, how to segment the video to be processed is of great significance. Summary of the Invention

[0003] The present application provides a video processing method and device, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, a video processing method is provided, the method comprising:

[0005] Get the video to be processed;

[0006] Obtaining, based on the video features of the video to be processed, a segmentation probability of each video frame of the video to be processed, wherein the segmentation probability represents a probability of a video frame being an endpoint of a video segment, wherein the endpoints include a start frame and an end frame;

[0007] Selecting two or more video frames whose segmentation probability is greater than or equal to a segmentation threshold from the video to be processed, wherein the two or more video frames include a first video frame and a second video frame with adjacent timestamps;

[0008] A video segment between the first video frame and the second video frame is intercepted from the video to be processed to obtain a target video segment.

[0009] In combination with any embodiment of the present application, before obtaining the target video segment, the method further includes:

[0010] Obtaining, based on the video features of the video to be processed, a category probability of each video frame from the first video frame to the second video frame, wherein the category probability represents a probability of a category to which the video frame belongs;

[0011] After obtaining the target video segment, the method further includes:

[0012] The category of the target video segment is obtained according to the category probability of each video frame from the first video frame to the second video frame.

[0013] In combination with any embodiment of the present application, obtaining the category of the target video segment according to the category probability of each video frame from the first video frame to the second video frame includes:

[0014] summing the category probabilities of each category in the target video segment according to the category probabilities of each video frame from the first video frame to the second video frame to obtain a sum of the category probabilities of each category;

[0015] The category corresponding to the maximum value of the category probability sum is determined as the category of the target video segment.

[0016] In combination with any embodiment of the present application, obtaining the segmentation probability of each video frame of the video to be processed according to the video features of the video to be processed includes:

[0017] Determining, based on the video features, an image difference between two adjacent video frames in the video to be processed;

[0018] The segmentation probability of the two adjacent video frames is determined according to the image difference, and the segmentation probability of the two adjacent video frames is positively correlated with the image difference.

[0019] In conjunction with any embodiment of the present application, the video feature includes a color feature;

[0020] Determining the image difference between two adjacent video frames in the video to be processed according to the video features includes:

[0021] Determining a color difference between two adjacent video frames in the video to be processed according to the color feature;

[0022] According to the color difference between two adjacent video frames in the video to be processed, the image difference between two adjacent video frames in the video to be processed is obtained, and the image difference is positively correlated with the color difference.

[0023] In combination with any embodiment of the present application, obtaining the segmentation probability of each video frame of the video to be processed according to the video features of the video to be processed includes:

[0024] Determining, based on the video features, a correlation between two non-adjacent video frames in the video to be processed;

[0025] According to the correlation, a segmentation probability of each video frame between the two non-adjacent video frames is obtained, and the segmentation probability of each video frame between the two non-adjacent video frames is negatively correlated with the correlation.

[0026] In conjunction with any embodiment of the present application, before determining the correlation between two non-adjacent video frames in the video to be processed based on the video features, the method further includes:

[0027] Determining the clarity of each video frame in the video to be processed;

[0028] Determining the correlation between two non-adjacent video frames in the video to be processed according to the video features includes:

[0029] In the case that there is a video frame whose definition is less than or equal to a definition threshold value among two adjacent frames in the video to be processed, the correlation between two non-adjacent video frames in the video to be processed is determined.

[0030] In combination with any embodiment of the present application, before obtaining the segmentation probability of each video frame of the video to be processed based on the video features of the video to be processed, the method further includes:

[0031] The video to be processed is processed using a neural network to obtain the video features of the video to be processed.

[0032] In a second aspect, a video processing device is provided, the device comprising:

[0033] An acquisition unit, used to acquire the video to be processed;

[0034] A first processing unit is configured to obtain a segmentation probability of each video frame of the video to be processed based on the video features of the video to be processed, wherein the segmentation probability represents a probability of a video frame being an endpoint of a video segment, where the endpoint includes a start frame and an end frame;

[0035] A second processing unit is configured to select two or more video frames having a segmentation probability greater than or equal to a segmentation threshold from the video to be processed, wherein the two or more video frames include a first video frame and a second video frame with adjacent timestamps;

[0036] The interception unit is configured to intercept a video segment between the first video frame and the second video frame from the video to be processed to obtain a target video segment.

[0037] In combination with any embodiment of the present application, the first processing unit is further configured to obtain a category probability of each video frame from the first video frame to the second video frame based on the video features of the video to be processed, wherein the category probability represents a probability of the category to which the video frame belongs;

[0038] The second processing unit is further configured to obtain a category of the target video segment according to a category probability of each video frame from the first video frame to the second video frame.

[0039] In combination with any embodiment of the present application, the second processing unit is configured to:

[0040] summing the category probabilities of each category in the target video segment according to the category probabilities of each video frame from the first video frame to the second video frame to obtain a sum of the category probabilities of each category;

[0041] The category corresponding to the maximum value of the category probability sum is determined as the category of the target video segment.

[0042] In combination with any embodiment of the present application, the first processing unit is configured to:

[0043] Determining, based on the video features, an image difference between two adjacent video frames in the video to be processed;

[0044] The segmentation probability of the two adjacent video frames is determined according to the image difference, and the segmentation probability of the two adjacent video frames is positively correlated with the image difference.

[0045] In conjunction with any embodiment of the present application, the video feature includes a color feature;

[0046] The first processing unit is configured to:

[0047] Determining a color difference between two adjacent video frames in the video to be processed according to the color feature;

[0048] According to the color difference between two adjacent video frames in the video to be processed, the image difference between two adjacent video frames in the video to be processed is obtained, and the image difference is positively correlated with the color difference.

[0049] In combination with any embodiment of the present application, the first processing unit is configured to:

[0050] Determining, based on the video features, a correlation between two non-adjacent video frames in the video to be processed;

[0051] According to the correlation, a segmentation probability of each video frame between the two non-adjacent video frames is obtained, and the segmentation probability of each video frame between the two non-adjacent video frames is negatively correlated with the correlation.

[0052] In combination with any embodiment of the present application, the first processing unit is further configured to:

[0053] Determining the clarity of each video frame in the video to be processed;

[0054] In the case that there is a video frame whose definition is less than or equal to a definition threshold value among two adjacent frames in the video to be processed, the correlation between two non-adjacent video frames in the video to be processed is determined.

[0055] In combination with any embodiment of the present application, the first processing unit is further used to process the video to be processed using a neural network to obtain the video features of the video to be processed.

[0056] According to a third aspect, an electronic device is provided, the video feature of which is that it includes: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0057] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0058] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect above and any possible implementation method thereof.

[0059] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the method of the above-mentioned first aspect and any possible implementation thereof.

[0060] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0061] In an embodiment of the present application, when a video processing device obtains a video to be processed, it obtains the segmentation probability of each video frame of the video to be processed based on the video features of the video to be processed, wherein the segmentation probability represents the probability of a video frame being the endpoint of a video segment, and the endpoints include a start frame and an end frame. Then, by selecting two or more video frames whose segmentation probability is greater than or equal to a segmentation threshold from the video to be processed, two or more video frames that can serve as endpoints are selected from the video to be processed. Finally, two frames with adjacent timestamps in the two or more video frames are respectively used as the start frame and the end frame, and the target video segment is intercepted from the video to be processed, thereby achieving segmentation of the video to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0063] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0064] Figure 1 A flowchart of a video processing method provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of the structure of a hollow deep neural network module provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of the structure of a color similarity calculation module provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0072] References to "embodiments" herein mean that a particular video feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0073] The embodiment of the present application is executed by a video processing device, wherein the video processing device can be any electronic device capable of executing the technical solution disclosed in the embodiment of the method of the present application. Optionally, the video processing device can be one of the following: a mobile phone, a computer, a tablet computer, or a wearable smart device.

[0074] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 This is a flowchart of a video processing method provided in an embodiment of the present application.

[0075] 101. Get the video to be processed.

[0076] In the embodiments of the present application, the video to be processed can be any video. The video to be processed can be an offline video or an online video. An offline video can be a video captured by a camera or a mobile smart device. An online video can be a video captured in real time by a camera. The video to be processed can be a video containing any content, for example, a video of a basketball game, another example, a video of a dance, or another example, a video containing both a dance and a basketball game.

[0077] In one implementation of obtaining a video to be processed, a video processing device receives the video to be processed input by a user through an input component, wherein the input component includes at least one of the following: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0078] In another implementation of obtaining a video to be processed, a video processing device receives the video to be processed sent by a terminal. The terminal may be any of the following: a mobile phone, a computer, a tablet computer, or a server.

[0079] In yet another implementation of obtaining the video to be processed, the video processing device obtains the video to be processed by downloading the video from the Internet.

[0080] In another implementation of obtaining the video to be processed, a communication connection is established between the video processing device and the camera, and the camera obtains the video captured by the camera as the video to be processed through the communication connection.

[0081] 102. Obtain a segmentation probability of each video frame of the video to be processed according to the video features of the video to be processed.

[0082] In the embodiment of the present application, the segmentation probability represents the probability of a video frame being an endpoint of a video segment, where the endpoint includes a start frame and an end frame.

[0083] Optionally, endpoints are used to distinguish video segments of different categories. Specifically, in the video to be processed, endpoints are the dividing lines between video segments of different categories. For example, the video to be processed includes the first frame, the second frame, the third frame, the fourth frame, the fifth frame, and the sixth frame, where the first frame, the third frame, the fourth frame, and the sixth frame are endpoints. Then, based on the endpoints, the video to be processed can be divided into a first video segment and a second video segment, where the first video segment includes the first frame, the second frame, and the third frame, and the second video segment includes the fourth frame, the fifth frame, and the sixth frame. In this case, the first video segment and the second video segment have different categories.

[0084] In an embodiment of the present application, the category of the video segment represents the category of the content of the video segment. For example, the category of the first video segment is dance, and the category of the second video segment is food. Then, the content of the first video segment is related to dance, and the content of the second video segment is related to food.

[0085] In an embodiment of the present application, the video features of the video to be processed carry feature information of the video frames in the video to be processed and timestamp information of the video frames in the video to be processed. The feature information of the video frames includes at least one of the following information: texture information of the video frame, color information of the video frame, shape information of objects within the video frame, brightness information of the video frame, and spatial relationship information of different objects within the video frame. The timestamp of the video frame can be determined based on the timestamp information of the video frame, where the timestamp of the video frame represents the playback time of the video frame in the video to be processed.

[0086] For example, a video to be processed includes the first frame, the second frame, and the third frame, where the timestamp of the first frame is t1, the timestamp of the second frame is t2, and the timestamp of the third frame is t3. In this case, the video features of the video to be processed carry the feature information of the first frame, the feature information of the second frame, and the feature information of the third frame, and the timestamp of the first frame is t1, the timestamp of the second frame is t2, and the timestamp of the third frame is t3.

[0087] In a possible implementation, the video processing device obtains video features of the video to be processed by performing feature extraction processing on the video to be processed.

[0088] When the video processing device obtains the video features of the video to be processed based on the video to be processed, it can determine the feature information of each video frame based on the video features of the video to be processed, and then use the feature information of each video frame to determine the segmentation probability of each video frame in the video to be processed.

[0089] 103. Select two or more video frames whose segmentation probability is greater than or equal to a segmentation threshold from the video to be processed.

[0090] In the embodiment of the present application, the greater the segmentation probability of a video frame, the greater the probability that the video frame is an endpoint. Therefore, in order to improve the accuracy of the endpoint determined from the video to be processed, a video frame with a large segmentation probability can be selected as the endpoint. The video processing device determines whether the segmentation probability of a video frame is large or small based on the segmentation threshold. Specifically, if the segmentation probability of a video frame is greater than or equal to the segmentation threshold, it means that the segmentation probability of the video frame is large, and then the video frame is determined to be an endpoint. If the segmentation probability of a video frame is less than the segmentation threshold, it means that the segmentation probability of the video frame is small, and then the video frame is determined not to be an endpoint.

[0091] The video processing device can select two or more video frames whose segmentation probability is greater than or equal to the segmentation threshold from the video to be processed based on the segmentation probability of each video frame in the video to be processed, that is, select two or more video frames that can serve as endpoints from the video to be processed.

[0092] The two or more video frames selected by the video processing device from the video to be processed as endpoints include a first video frame and a second video frame with adjacent timestamps. That is, the first video frame and the second video frame are any two frames with adjacent timestamps in the two or more video frames. It should be understood that the fact that the first video frame and the second video frame have adjacent timestamps in the two or more video frames does not mean that the first video frame and the second video frame have adjacent timestamps in the video to be processed.

[0093] For example, a video to be processed includes a first frame, a second frame, a third frame, a fourth frame, a fifth frame, and a sixth frame. The video processing device selects two or more video frames from the video to be processed that can serve as endpoints, including the first frame, the third frame, the fourth frame, and the sixth frame. In this case, the first frame and the third frame are two frames with adjacent timestamps among the two or more video frames, the third frame and the fourth frame are two frames with adjacent timestamps among the two or more video frames, and the third frame and the sixth frame are two frames with adjacent timestamps among the two or more video frames.

[0094] 104 . Cut out a video segment between the first video frame and the second video frame from the video to be processed to obtain a target video segment.

[0095] Since the first video frame and the second video frame are both endpoints, the video processing device can obtain the target video segment by intercepting the video segment between the first video frame and the second video frame from the video to be processed.

[0096] For example, the video to be processed includes the first frame, the second frame, the third frame, the fourth frame, the fifth frame, and the sixth frame, and the two or more video frames selected by the video processing device from the video to be processed that can serve as endpoints include: the first frame, the third frame, the fourth frame, and the sixth frame. In the case where the first video frame is the first frame and the second video frame is the third frame, the target video segment intercepted from the video to be processed by the video processing device includes the first frame, the second frame, and the third frame. In the case where the first video frame is the third frame and the second video frame is the fourth frame, the target video segment intercepted from the video to be processed by the video processing device includes the third frame and the fourth frame. In the case where the first video frame is the fourth frame and the second video frame is the sixth frame, the target video segment intercepted from the video to be processed by the video processing device includes the fourth frame, the fifth frame, and the sixth frame.

[0097] In an embodiment of the present application, when a video processing device obtains a video to be processed, it obtains the segmentation probability of each video frame of the video to be processed based on the video features of the video to be processed, wherein the segmentation probability represents the probability of a video frame being the endpoint of a video segment, and the endpoints include a start frame and an end frame. Then, by selecting two or more video frames whose segmentation probability is greater than or equal to a segmentation threshold from the video to be processed, two or more video frames that can serve as endpoints are selected from the video to be processed. Finally, two frames with adjacent timestamps in the two or more video frames are respectively used as the start frame and the end frame, and the target video segment is intercepted from the video to be processed, thereby achieving segmentation of the video to be processed.

[0098] As an optional implementation manner, the video processing device further performs the following steps before obtaining the target video segment:

[0099] 201. Obtain a category probability of each video frame from the first video frame to the second video frame according to the video features of the video to be processed.

[0100] In the embodiment of the present application, the category probability represents the probability of the category to which the video frame belongs, wherein the category to which the video frame belongs represents the category of the content of the video frame. For example, if the category of the video frame is dance, then the content of the video frame includes dance.

[0101] In the embodiment of the present application, each video frame from the first video frame to the second video frame includes the first video frame, the second video frame, and the video frames between the first video frame and the second video frame in the to-be-processed video. That is, each video frame from the first video frame to the second video frame is a video frame in the target video segment. For example, if the to-be-processed video includes the first frame, the second frame, the third frame, the fourth frame, the fifth frame, and the sixth frame, and the first video frame is the first frame and the second video frame is the third frame, then each video frame from the first video frame to the second video frame includes: the first frame, the second frame, and the third frame.

[0102] Since the video features of the video to be processed include feature information of each video frame, the video processing device determines the feature information of each video frame from the first video frame to the second video frame based on the video features of the video to be processed, and then uses the feature information of each video frame from the first video frame to the second video frame to determine the category probability of each video frame from the first video frame to the second video frame.

[0103] For example, the video to be processed includes the first frame, the second frame, the third frame, the fourth frame, the fifth frame, and the sixth frame, and each video frame from the first video frame to the second video frame includes: the first frame, the second frame, and the third frame. Then the category probability of each video frame from the first video frame to the second video frame includes: the probability of the category of the first frame, the probability of the category of the second frame, and the probability of the category of the third frame.

[0104] In this embodiment, after obtaining the target video segment, the video processing device further performs the following steps:

[0105] 202. Obtain a category of the target video segment according to the category probabilities of the video frames in the target video segment.

[0106] The video processing device can determine the category of each video frame based on the category probability of each video frame, and further, can obtain the category of the target video segment based on the category of each video frame. In one possible implementation, the video processing device determines the category of each video frame based on the category probability of each video frame. Based on the categories of each video frame, the category corresponding to the largest number of video frames is determined as the category of the target video segment.

[0107] For example, the target video segment includes video frames a, b, and c. Based on the category probabilities of the video frames, the video processing device determines that the category of video frame a is dance, the category of video frame b is dance, and the category of video frame c is food. In this case, the number of video frames classified as dance is 2, and the number of video frames classified as food is 1. Therefore, the video processing device determines that the category of the target video is dance.

[0108] In this embodiment, the video processing device obtains the category probability of each video frame from the first video frame to the second video frame based on the video features of the video to be processed, and then, when the target video is intercepted from the video to be processed, the category of the target video can be obtained based on the category probability of each video frame from the first video frame to the second video frame.

[0109] As an optional implementation manner, the video processing device performs the following steps during the execution of step 202:

[0110] 301 . According to the category probabilities of the video frames from the first video frame to the second video frame, sum the category probabilities of each category in the target video segment to obtain a sum of the category probabilities of each category.

[0111] 302. Determine the category corresponding to the maximum value of the sum of the category probabilities as the category of the target video segment.

[0112] For example, a target video segment includes video frames a, b, and c. The class probabilities for video frame a include a probability of 0.8 for dance and a probability of 0.2 for food. The class probabilities for video frame b include a probability of 0.6 for dance and a probability of 0.4 for food. The class probabilities for video frame c include a probability of 0.5 for dance and a probability of 0.5 for food. In this case, the categories in the target video segment include dance and food. The sum of the class probabilities for dance is 0.8 + 0.6 + 0.5 = 1.9, and the sum of the class probabilities for food is 0.2 + 0.4 + 0.5 = 1.1. Because the sum of the class probabilities for dance is greater than the sum of the class probabilities for food, the sum of the class probabilities for dance is the maximum of the class probabilities. The category corresponding to the maximum class probabilities is dance, indicating that the target video segment is dance.

[0113] In this embodiment, the video processing device sums the class probabilities of each category in the target video segment based on the class probabilities of each video frame from the first video frame to the second video frame to obtain the sum of the class probabilities for each category. The class corresponding to the maximum value of the sum of the class probabilities is then determined as the class of the target video segment. In this manner, the class of the target video segment is obtained based on the class probabilities of each video frame in the target video segment.

[0114] As an optional implementation manner, the video processing device performs the following steps during the process of executing step 102:

[0115] 401. Determine an image difference between two adjacent video frames in the video to be processed based on the video features.

[0116] In the embodiment of the present application, the image difference represents the difference in image content. For example, the image difference between video frame a and video frame b represents the difference in semantics between video frame a and video frame b.

[0117] Optionally, the image difference is a numerical value, that is, the video processing device quantifies the difference in image content between two adjacent frames in the video to be processed based on the video features to obtain the image difference. In one possible implementation, a larger numerical value of the image difference indicates a greater difference in image content.

[0118] In an embodiment of the present application, two adjacent video frames in the video to be processed are two frames with adjacent timestamps in the video to be processed. For example, the video to be processed includes a first frame, a second frame, and a third frame. At this time, the two adjacent video frames in the video to be processed can be the first frame and the second frame, and the two adjacent video frames in the video to be processed can also be the second frame and the third frame.

[0119] Since the video features carry the feature information of each video frame in the video to be processed, the video processing device can determine the feature information of two adjacent video frames in the video to be processed based on the video features, and then use the feature information of two adjacent video frames in the video to be processed to determine the image difference between the two adjacent video frames in the video to be processed.

[0120] 402. Determine the segmentation probability of the two adjacent video frames according to the image difference.

[0121] The greater the image difference between two adjacent video frames, the greater the probability that the two video frames belong to video segments of different categories. Therefore, the video processing device can determine the segmentation probability of the two adjacent video frames based on the image difference between the two adjacent video frames.

[0122] In the embodiment of the present application, the probability of segmenting two adjacent video frames is positively correlated with the image difference between the two adjacent video frames, that is, the greater the image difference between the two adjacent video frames, the greater the probability of segmenting the two adjacent video frames.

[0123] In this embodiment, the video processing device determines the image difference between two adjacent video frames in the video to be processed based on the video features. Then, when the segmentation probability of the two adjacent video frames is positively correlated with the image difference between the two adjacent video frames, the segmentation probability of the two adjacent video frames is determined based on the image difference. This allows the segmentation probability of each video frame in the video to be processed to be determined separately, and also improves the accuracy of the segmentation probability of each video frame in the video to be processed.

[0124] As an optional implementation, the video feature includes a color feature. The video processing device performs the following steps during the execution of step 401:

[0125] 501. Determine a color difference between two adjacent video frames in the video to be processed based on the color feature.

[0126] In the embodiment of the present application, the color difference represents a difference in image color. Optionally, the color difference is a numerical value, i.e., the video processing device quantifies the difference in image color between two adjacent frames in the video to be processed based on the color characteristics to obtain the color difference. In one possible implementation, a larger numerical value of the color difference represents a greater difference in image color.

[0127] 502. Obtain the image difference between the two adjacent video frames in the video to be processed according to the color difference between the two adjacent video frames in the video to be processed.

[0128] The greater the color difference between two adjacent video frames, the greater the difference in image content between the two video frames. Therefore, the video processing device can determine the image difference between the two adjacent video frames based on the color difference between the two adjacent video frames.

[0129] In the embodiment of the present application, the image difference between two adjacent video frames is positively correlated with the color difference between the two adjacent video frames, that is, the greater the color difference between the two adjacent video frames, the greater the image difference between the two adjacent video frames.

[0130] In this embodiment, the video processing device determines the color difference between two adjacent video frames in the video to be processed based on the video features. If the image difference between the two adjacent video frames and the color difference between the two adjacent video frames are positively correlated, the image difference between the two adjacent video frames is determined based on the color difference between the two adjacent video frames, thereby improving the accuracy of the image difference between the two adjacent video frames.

[0131] As an optional implementation manner, the video processing device performs the following steps during the process of executing step 102:

[0132] 601. Determine the correlation between two non-adjacent video frames in the video to be processed based on the video features.

[0133] In the embodiment of the present application, the two non-adjacent video frames in the video to be processed are two video frames with non-adjacent timestamps in the video to be processed. For example, the video to be processed includes a first frame, a second frame, and a third frame, and the two non-adjacent video frames in the video to be processed are the first frame and the third frame.

[0134] In an embodiment of the present application, the correlation between two non-adjacent video frames represents the correlation between the image content of the two non-adjacent video frames. Optionally, the correlation between two non-adjacent video frames is the similarity between the two non-adjacent video frames. For example, two non-adjacent video frames are video frame a and video frame b. If the image content of video frame a and the image content of video frame b are both dance, then the correlation between video frame a and video frame b is high. If the image content of video frame a is dance and the image content of video frame b is food, then the correlation between video frame a and video frame b is low.

[0135] Since the video features carry the feature information of each video frame in the video to be processed, the video processing device can determine the feature information of two non-adjacent video frames in the video to be processed based on the video features, and then use the feature information of the two non-adjacent video frames in the video to be processed to determine the correlation between the two non-adjacent video frames in the video to be processed.

[0136] 602. Obtain a segmentation probability of each video frame between the two non-adjacent video frames according to the correlation.

[0137] The greater the correlation between two non-adjacent video frames, the lower the probability that the two video frames belong to different categories of video segments, that is, the greater the probability that the two non-adjacent video frames belong to the same video segment, that is, the greater the probability that the video frame located between the two non-adjacent video frames belongs to the same video segment, that is, the lower the probability of segmenting the video frames located between the two non-adjacent video frames. Therefore, the video processing device can determine the segmentation probability of each video frame between the two non-adjacent video frames based on the correlation between the two non-adjacent video frames. Specifically, when the segmentation probability of the video frames between the two non-adjacent video frames satisfies a negative correlation relationship with the above-mentioned correlation, the video processing device obtains the segmentation probability of each video frame between the two non-adjacent video frames based on the correlation between the two non-adjacent video frames.

[0138] In this embodiment, the video processing device determines the correlation between two non-adjacent video frames in the video to be processed based on the aforementioned video features. If the probability of segmenting the video frames between the two non-adjacent video frames is negatively correlated with the aforementioned correlation, the segmentation probability of each video frame between the two non-adjacent video frames can be obtained based on the correlation between the two non-adjacent video frames, thereby improving the accuracy of the segmentation probability of each video frame between the two non-adjacent video frames.

[0139] As an optional implementation manner, the video processing device further performs the following steps before performing step 601:

[0140] 701. Determine the clarity of each video frame in the video to be processed.

[0141] After executing step 701, the video processing device performs the following steps during the execution of step 601:

[0142] 702. When there is a video frame whose clarity is less than or equal to a clarity threshold between two adjacent frames in the video to be processed, determine a correlation between two non-adjacent video frames in the video to be processed.

[0143] The clarity of a video frame affects the accuracy of its feature information. Specifically, low video frame clarity is positively correlated with the accuracy of its feature information. That is, the lower the video frame clarity, the lower the accuracy of its feature information. Therefore, if one of two adjacent video frames has low clarity, determining the segmentation probability between the two frames based on their feature information will result in low segmentation accuracy.

[0144] In an embodiment of the present application, in order to improve the accuracy of the segmentation probability, the video processing device determines the correlation between two non-adjacent video frames in the video to be processed when there is a video frame with low clarity between two adjacent frames in the video to be processed, so that the segmentation probability of each video frame between the two non-adjacent video frames can be obtained based on the correlation between the two non-adjacent video frames.

[0145] In an embodiment of the present application, the video processing device determines whether the clarity of a video frame is low or high based on a clarity threshold. Specifically, if the clarity of a video frame is less than or equal to the clarity threshold, it indicates that the clarity of the video frame is low, and if the clarity of a video frame is greater than the clarity threshold, it indicates that the clarity of the video frame is high. Therefore, when there is a video frame with a clarity less than or equal to the clarity threshold between two adjacent frames in the video to be processed, the video processing device determines the correlation between two non-adjacent video frames in the video to be processed, and then, based on the correlation between the two non-adjacent video frames, obtains the segmentation probability of each video frame between the two non-adjacent video frames, thereby improving the accuracy of the segmentation probability of each video frame between the two non-adjacent video frames.

[0146] Steps 401 and 402, and steps 601 and 602 respectively provide two different implementations for determining the segmentation probability of each video frame in the video to be processed. As an optional implementation, the video processing device combines the two implementations for determining the segmentation probability provided by steps 401 and 402, and steps 601 and 602, to determine the segmentation probability of each video frame in the video to be processed.

[0147] In one possible implementation, the video processing device determines the probability of segmenting the video frames in the video to be processed based on the image difference and the correlation, after determining the image difference between two adjacent video frames in the video to be processed and the correlation between two non-adjacent video frames in the video to be processed. The segmentation probability of the video frames in the video to be processed is positively correlated with the image difference, and the segmentation probability of the video frames in the video to be processed is negatively correlated with the correlation.

[0148] For example, a video to be processed includes a first frame, a second frame, and a third frame. The video processing device determines an image difference between the first frame and the second frame, as well as a correlation between the first frame and the third frame. The device then determines a segmentation probability for the second frame based on the image difference and the correlation. The segmentation probability for the second frame is positively correlated with the image difference, and negatively correlated with the correlation.

[0149] In this embodiment, the video processing device determines the segmentation probability of each video frame in the video to be processed by combining the two embodiments of determining the segmentation probability provided by steps 401 and 402 and steps 601 and 602, thereby improving the accuracy of the segmentation probability of each video frame.

[0150] Optionally, the video processing device combines the two implementation methods of determining the segmentation probability provided by steps 401 and 402, and steps 601 and 602, and performs steps 501 and 502 during the execution of step 401 to further improve the accuracy of the segmentation probability of each video frame when determining the segmentation probability of each video frame in the video to be processed.

[0151] Optionally, the video processing device combines the two implementation methods of determining the segmentation probability provided by steps 401 and 402, and steps 601 and 602, and executes step 701 before executing step 601 when determining the segmentation probability of each video frame in the video to be processed, and executes step 702 during the execution of step 601 after executing step 701, so as to further improve the accuracy of the segmentation probability of each video frame.

[0152] Optionally, the video processing device combines the two implementation methods of determining the segmentation probability provided by steps 401 and 402, and steps 601 and 602, and executes steps 501 and 502 during the execution of step 401 to determine the segmentation probability of each video frame in the video to be processed, and executes step 701 before executing step 601, and executes step 702 during the execution of step 601 after executing step 701, thereby further improving the accuracy of the segmentation probability of each video frame.

[0153] As an optional implementation, before executing step 102, the video processing device obtains the video features of the video to be processed by executing the following steps: 801. Process the video to be processed using a neural network to obtain the video features of the video to be processed.

[0154] In the embodiment of the present application, the neural network has the ability to extract the features of the video. Figure 2 is a structural diagram of a neural network, such as Figure 2 As shown in the figure, the neural network includes: a dilated deep convolution neural network module (DDCNN), an average pooling module (Avg pooling), a color similarity calculation module (Color Similarities), a connection layer (Concat), a temporal self-attention module (Temporal Self-attention), a fully connected layer (Dense), and a sigmoid function (sigmoid).

[0155] Figure 3 This is a structural diagram of a hollow deep neural network module, such as Figure 3 As shown in the figure, DDCNN includes four convolution kernels (Conv) of size 1×3×3 and four dilated convolution kernels (Dilation) of size 3×1×1. It should be understood that the convolution kernels of size 1×3×3 and the dilated convolution kernels of size 3×1×1 are both three-dimensional convolution kernels, and the size of the three-dimensional convolution kernel can be expressed as T×H×W, where T is time, H is height, and W is width.

[0156] Four convolution kernels of size 1×3×3 process the input data of DDCNN respectively, and then four dilated convolution kernels of size 3×1×1 process the output data of four convolution kernels of size 1×3×3 respectively. Among them, one dilated convolution kernel of size 3×1×1 processes the output data of one convolution kernel of size 1×3×3, and any two different dilated convolution kernels of size 3×1×1 process different data. Finally, the output data of DDCNN (i.e. Figure 3 Output in ).

[0157] Figure 4 is a structural diagram of a color similarity calculation module, such as Figure 4As shown in the figure, the four global average pooling modules process the input data of the color similarity calculation module respectively, and then concatenate the output data of the four global average pooling modules. The concatenated data is then processed by the fully connected layer (Dense). The pre-similarity calculation module (Cosine Similarities) calculates the color similarity of the output data of the fully connected layer, and finally processes the color similarity through the fully connected layer (Dense) to obtain the output data of the color similarity calculation module (i.e. Figure 4 It should be understood that the output data of the color similarity calculation module is color similarity, specifically, the color similarity of two video frames. The color similarity is negatively correlated with the above-mentioned color difference, that is, the color difference can be determined based on the color similarity.

[0158] The temporal self-attention module can determine the correlation between two non-adjacent video frames by executing the above 601, thereby determining the segmentation probability across frames in the temporal dimension. Optionally, the temporal self-attention module is expressed by the following formula:

[0159]

[0160] Among them, Q, K, and V all represent the features of the video frame in the video to be processed. It should be understood that Q, K, and V can be the same, that is, Q, K, and V can be the features of the same video frame. Any two of Q, K, and V can also be the same, that is, Q and K can be the features of the same video frame, but Q and K are different from V, or Q and V can be the features of the same video frame, but Q and V are different from K, or K and V can be the features of the same video frame, but K and V are different from Q. Q, K, and V can also be the features of three different video frames in the video to be processed. k is the number of channels of Q, and softmax(·) represents the logistic regression (softmax) function.

[0161] When the video to be processed is input to Figure 2 After the neural network shown in Figure 2The neural network shown processes the video to be processed, and the data output by the temporal self-attention module is the video feature of the video to be processed. Then, the video features of the video to be processed are processed through two different recognition branches (each recognition branch includes a fully connected layer and a sigmoid function), and the segmentation probability of each video frame in the video to be processed and the category probability of each video frame in the video to be processed are output respectively. In other words, the neural network can sequentially realize feature extraction processing of the video to be processed to obtain the video features of the video to be processed, and based on the video features of the video to be processed, the target video segment can be intercepted from the video to be processed and the category of the target video segment can be determined, which can reduce the cost of intercepting the target video segment from the video to be processed and the cost of determining the category of the target video segment.

[0162] Optionally, before using the neural network to perform feature extraction on the video to be processed, the neural network is trained. The training process includes: using the neural network to process the training video to obtain video features of the training video. The video features of the training video are processed by two different recognition branches to obtain the predicted segmentation probability of each video frame in the training video and the category probability of each video frame in the training video. According to the predicted segmentation probability of each video frame in the training video, the training video is segmented to obtain at least one segmented video segment. According to the category probability of each video frame in the training video, the category of each segmented video segment is determined respectively, and then the category of the segmented video segment is used as the predicted category of each video frame in the segmented video segment, thereby obtaining the predicted category of each video frame in the training video.

[0163] Determine a first difference between the predicted segmentation probability of each video frame in the training video and the actual segmentation probability of each video frame in the training video, and optionally determine the first difference through a cross entropy loss function. Determine a second difference between the predicted category of each video frame in the training video and the actual category of each video frame in the training video, and optionally determine the second difference through a cross entropy loss function. Based on the first difference and the second difference, obtain a training loss, wherein the first difference is positively correlated with the training loss, and the second difference is positively correlated with the training loss. Based on the training loss, adjust the parameters of the neural network until the training loss converges, thereby completing the training of the neural network.

[0164] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0165] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0166] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0167] See also Figure 5 , Figure 5 This is a structural diagram of a video processing device provided in an embodiment of the present application. The video processing device 1 includes: an acquisition unit 11, a first processing unit 12, a second processing unit 13, and an interception unit 14. Specifically:

[0168] An acquisition unit 11 is used to acquire a video to be processed;

[0169] A first processing unit 12 is configured to obtain a segmentation probability of each video frame of the video to be processed based on the video features of the video to be processed, wherein the segmentation probability represents a probability of a video frame being an endpoint of a video segment, where the endpoints include a start frame and an end frame;

[0170] A second processing unit 13 is configured to select two or more video frames whose segmentation probability is greater than or equal to a segmentation threshold from the video to be processed, wherein the two or more video frames include a first video frame and a second video frame with adjacent timestamps;

[0171] The interception unit 14 is configured to intercept a video segment between the first video frame and the second video frame from the video to be processed to obtain a target video segment.

[0172] In combination with any embodiment of the present application, the first processing unit 12 is further configured to obtain a category probability of each video frame from the first video frame to the second video frame based on the video features of the video to be processed, wherein the category probability represents a probability of the category to which the video frame belongs;

[0173] The second processing unit 13 is further configured to obtain a category of the target video segment according to the category probability of each video frame from the first video frame to the second video frame.

[0174] In combination with any embodiment of the present application, the second processing unit 13 is configured to:

[0175] summing the category probabilities of each category in the target video segment according to the category probabilities of each video frame from the first video frame to the second video frame to obtain a sum of the category probabilities of each category;

[0176] The category corresponding to the maximum value of the category probability sum is determined as the category of the target video segment.

[0177] In combination with any embodiment of the present application, the first processing unit 12 is configured to:

[0178] Determining, based on the video features, an image difference between two adjacent video frames in the video to be processed;

[0179] The segmentation probability of the two adjacent video frames is determined according to the image difference, and the segmentation probability of the two adjacent video frames is positively correlated with the image difference.

[0180] In conjunction with any embodiment of the present application, the video feature includes a color feature;

[0181] The first processing unit 12 is configured to:

[0182] Determining a color difference between two adjacent video frames in the video to be processed according to the color feature;

[0183] According to the color difference between two adjacent video frames in the video to be processed, the image difference between two adjacent video frames in the video to be processed is obtained, and the image difference is positively correlated with the color difference.

[0184] In combination with any embodiment of the present application, the first processing unit 12 is configured to:

[0185] Determining, based on the video features, a correlation between two non-adjacent video frames in the video to be processed;

[0186] According to the correlation, a segmentation probability of each video frame between the two non-adjacent video frames is obtained, and the segmentation probability of each video frame between the two non-adjacent video frames is negatively correlated with the correlation.

[0187] In combination with any embodiment of the present application, the first processing unit 12 is further configured to:

[0188] Determining the clarity of each video frame in the video to be processed;

[0189] In the case that there is a video frame whose definition is less than or equal to a definition threshold value among two adjacent frames in the video to be processed, the correlation between two non-adjacent video frames in the video to be processed is determined.

[0190] In combination with any embodiment of the present application, the first processing unit 12 is further configured to process the video to be processed using a neural network to obtain the video features of the video to be processed.

[0191] In an embodiment of the present application, when a video processing device obtains a video to be processed, it obtains the segmentation probability of each video frame of the video to be processed based on the video features of the video to be processed, wherein the segmentation probability represents the probability of a video frame being the endpoint of a video segment, and the endpoints include a start frame and an end frame. Then, by selecting two or more video frames whose segmentation probability is greater than or equal to a segmentation threshold from the video to be processed, two or more video frames that can serve as endpoints are selected from the video to be processed. Finally, two frames with adjacent timestamps in the two or more video frames are respectively used as the start frame and the end frame, and the target video segment is intercepted from the video to be processed, thereby achieving segmentation of the video to be processed.

[0192] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0193] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, connection through various interfaces, transmission lines, buses, etc.

[0194] The processor 21 may include one or more processors, for example, one or more central processing units (CPUs). In the case where the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Alternatively, the processor 21 may be a processor group consisting of multiple CPUs, wherein the multiple processors are coupled to each other via one or more buses. Alternatively, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0195] The memory 22 can be used to store computer program instructions and various computer program codes, including program codes for executing the solution of the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0196] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0197] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0198] It is understandable that Figure 6 Only a simplified design of an electronic device is shown. In actual applications, the electronic device may further include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the scope of protection of the present application.

[0199] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here. Those skilled in the art will also clearly understand that the descriptions of the various embodiments of this application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, reference can be made to the descriptions of other embodiments.

[0201] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some video features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection of some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0202] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0203] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0204] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0205] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by a computer program instructing related hardware to perform the processes. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A video processing method, characterized in that: The method comprises: Get the video to be processed; Inputting the video to be processed into a neural network so that the neural network processes the video to be processed, obtaining video features of the video to be processed through a temporal self-attention module in the neural network, processing the video features of the video to be processed through a first recognition branch in the neural network and a second recognition branch in the neural network, and outputting a segmentation probability of each video frame in the video to be processed and a category probability of each video frame in the video to be processed, respectively, wherein the segmentation probability represents the probability of a video frame being an endpoint of a video segment, the endpoint including a start frame and an end frame, and the category probability represents the probability of the category to which the video frame belongs; Selecting two or more video frames whose segmentation probability is greater than or equal to a segmentation threshold from the video to be processed, wherein the two or more video frames include a first video frame and a second video frame with adjacent timestamps in the two or more video frames; intercepting a video segment between the first video frame and the second video frame from the video to be processed to obtain a target video segment; The category of the target video segment is obtained according to the category probability of each video frame from the first video frame to the second video frame.

2. The method according to claim 1, characterized in that Obtaining the category of the target video segment according to the category probabilities of the video frames from the first video frame to the second video frame includes: summing the category probabilities of each category in the target video segment according to the category probabilities of each video frame from the first video frame to the second video frame to obtain a sum of the category probabilities of each category; The category corresponding to the maximum value of the category probability sum is determined as the category of the target video segment.

3. The method according to claim 1 or 2, characterized in that The first recognition branch in the neural network and the second recognition branch in the neural network process the video features of the video to be processed, and output the segmentation probability of each video frame in the video to be processed and the category probability of each video frame in the video to be processed respectively, including: Processing the video features of the video to be processed by the first recognition branch in the neural network and the second recognition branch in the neural network, outputting the category probability of each video frame in the video to be processed and determining the image difference between two adjacent video frames in the video to be processed; The segmentation probability of the two adjacent video frames is determined according to the image difference, and the segmentation probability of the two adjacent video frames is positively correlated with the image difference.

4. The method according to claim 3, characterized in that The video features include color features; Determining the image difference between two adjacent video frames in the video to be processed according to the video features includes: Determining a color difference between two adjacent video frames in the video to be processed according to the color feature; According to the color difference between two adjacent video frames in the video to be processed, the image difference between two adjacent video frames in the video to be processed is obtained, and the image difference is positively correlated with the color difference.

5. The method according to claim 1 or 2, characterized in that The first recognition branch in the neural network and the second recognition branch in the neural network process the video features of the video to be processed, and output the segmentation probability of each video frame in the video to be processed and the category probability of each video frame in the video to be processed respectively, including: Processing the video features of the video to be processed by the first recognition branch in the neural network and the second recognition branch in the neural network, outputting the category probability of each video frame in the video to be processed and determining the correlation between two non-adjacent video frames in the video to be processed; According to the correlation, a segmentation probability of each video frame between the two non-adjacent video frames is obtained, and the segmentation probability of each video frame between the two non-adjacent video frames is negatively correlated with the correlation.

6. The method according to claim 5, characterized in that Before determining the correlation between two non-adjacent video frames in the video to be processed based on the video features, the method further includes: Determining the clarity of each video frame in the video to be processed; Determining the correlation between two non-adjacent video frames in the video to be processed according to the video features includes: In the case that there is a video frame whose definition is less than or equal to a definition threshold value among two adjacent frames in the video to be processed, the correlation between two non-adjacent video frames in the video to be processed is determined.

7. A video processing device, characterized in that: The device comprises: An acquisition unit, used to acquire the video to be processed; a first processing unit, configured to input the video to be processed into a neural network so that the neural network processes the video to be processed, obtain video features of the video to be processed via a temporal self-attention module in the neural network, process the video features of the video to be processed via a first recognition branch in the neural network and a second recognition branch in the neural network, and output a segmentation probability of each video frame in the video to be processed and a category probability of each video frame in the video to be processed, respectively, wherein the segmentation probability represents the probability of a video frame being an endpoint of a video segment, wherein the endpoint includes a start frame and an end frame, and the category probability represents the probability of a category to which the video frame belongs; A second processing unit is configured to select two or more video frames having a segmentation probability greater than or equal to a segmentation threshold from the video to be processed, wherein the two or more video frames include a first video frame and a second video frame with adjacent timestamps in the two or more video frames; a capture unit, configured to capture a video segment between the first video frame and the second video frame from the video to be processed to obtain a target video segment; The second processing unit is further configured to obtain a category of the target video segment according to a category probability of each video frame from the first video frame to the second video frame.

8. An electronic device, wherein the video feature comprises: A processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images