Video clip generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a video clip generation method and device, equipment and a storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring a motion video, and segmenting the motion video to obtain a plurality of video clips; obtaining each frame of image in the video clip, and performing feature extraction on each frame of image to obtain optical flow information corresponding to each frame of image; analyzing the optical flow information to obtain an optical flow vector, and determining a label corresponding to the optical flow information according to the optical flow vector; determining image features of each frame of image in the video clip according to the optical flow information and the corresponding label; and determining an image corresponding to the key action frame from the video clip according to the image feature of each frame of image in the video clip, and generating a target video clip according to the image corresponding to the key action frame. The objective of the invention is to solve the problem that the key action frame cannot be accurately identified due to many interference factors in the related technology.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for generating a video clip, an apparatus for generating a video clip, a computer device, and a computer-readable storage medium. Background Art

[0002] At present, sports videos usually contain a large number of sports scenes and actions. For some viewers, they may only want to pay attention to the highlights in sports videos. For them, watching the entire game video to get the highlights may be too lengthy. By identifying key action frames, we can achieve video summaries and extract highlights, focusing on the most important moments and providing a more compact and exciting viewing experience.

[0003] However, the traditional method of identifying key action frames is to identify them through the original video. However, this recognition method has many interference factors and low recognition accuracy. Summary of the invention

[0004] The present application provides a method for generating a video clip, a device for generating a video clip, a computer device and a computer-readable storage medium, aiming to solve the problem in the related art that key action frames cannot be accurately identified due to many interference factors.

[0005] To achieve the above object, the present application provides a method for generating a video clip, the method comprising:

[0006] Acquire a motion video, and segment the motion video to obtain multiple video clips;

[0007] Obtain each frame of the image in each of the video clips, and perform feature extraction on each frame of the image to obtain optical flow information corresponding to each frame of the image;

[0008] Parsing the optical flow information to obtain an optical flow vector, and determining a label corresponding to the optical flow information according to the optical flow vector;

[0009] Determining image features of each frame of the video clip according to the optical flow information and its corresponding label;

[0010] Predicting each frame of the image in the video clip according to the image features of each frame of the image in the video clip to obtain a key frame probability of each frame of the image;

[0011] An image corresponding to a key action frame is determined according to the key frame probability of each frame of the image, and a target video segment is generated according to the image corresponding to the key action frame.

[0012] To achieve the above object, the present application provides a device for generating a video clip, the device for generating a video clip comprising:

[0013] A video segmentation module is used to obtain a motion video and segment the motion video to obtain multiple video clips;

[0014] An optical flow information generation module is used to obtain each frame of the image in each of the video clips, and perform feature extraction on each frame of the image to obtain optical flow information corresponding to each frame of the image;

[0015] A label generation module, used to parse the optical flow information to obtain an optical flow vector, and determine a label corresponding to the optical flow information according to the optical flow vector;

[0016] A feature extraction module, used to determine the image features of each frame of the video clip according to the optical flow information and its corresponding label;

[0017] A key frame prediction module, used to predict each frame of the image in the video clip according to the image features of each frame of the image in the video clip, and obtain a key frame probability of each frame of the image;

[0018] The video segment generation module is used to determine the image corresponding to the key action frame according to the key frame probability of each frame of the image, and generate a target video segment according to the image corresponding to the key action frame.

[0019] To achieve the above-mentioned purpose, the present application provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the method for generating a video clip as described above when executing the computer program.

[0020] To achieve the above objectives, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method for generating a video clip as described above.

[0021] The video clip generation method, video clip generation device, computer equipment and computer-readable storage medium disclosed in the embodiment of the present application obtain a motion video and segment the motion video to obtain multiple video clips; obtain each frame image in the video clip, and extract features for each frame image to obtain the optical flow information corresponding to each frame image; parse the optical flow information to obtain an optical flow vector, and determine the label corresponding to the optical flow information according to the optical flow vector; determine the image features of each frame image in the video clip according to the optical flow information and its corresponding label; determine the image corresponding to the key action frame from the video clip according to the image features of each frame image in the video clip. In this way, by obtaining the optical flow information corresponding to each frame image, the probability of each frame being a key action frame can be predicted through the optical flow information, and the key information can be extracted more clearly, and the interference information can be eliminated, so that the recognition rate of the action can be greatly improved, and then the recognition accuracy of the key action frame can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 It is a scene schematic diagram of a method for generating a video clip provided in an embodiment of the present application;

[0024] Figure 2 It is a flowchart of a method for generating a video clip provided in an embodiment of the present application;

[0025] Figure 3 is a schematic block diagram of a device for generating a video clip provided in an embodiment of the present application;

[0026] Figure 4 It is a schematic block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0029] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] At present, for sports videos such as badminton games, the research significance of video frame detection of batting action in videos includes sports technology analysis and improvement, action recognition and classification, decision support and tactical analysis, and training progress tracking and evaluation. However, the traditional method of identifying key action frames is to identify them through the original video, but this identification method has many interference factors and low recognition accuracy.

[0031] The present application proposes a method for generating video clips, a device for generating video clips, a computer device and a computer-readable storage medium, which can effectively capture the relationship between video blocks and the interactive relationship between video and text, so that the complete video can be accurately retrieved through the video clips, the reliability and efficiency of video retrieval can be improved, and the application scenarios are flexible.

[0032] The method can be applied to a server, and of course can also be applied to a terminal device, to generate a corresponding image classification model, wherein the terminal device may include a fixed terminal such as a mobile phone, a tablet computer, a personal digital assistant (PDA), etc. The server can be, for example, a separate server or a server cluster. However, for ease of understanding, the following embodiment will be described in detail with a method for generating a video clip applied to a server.

[0033] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] like Figure 1 As shown, the method for generating a video clip provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. The application environment includes a terminal device 110 and a server 120, wherein the terminal device 110 can communicate with the server 120 through a network. Specifically, the server 120 obtains a motion video, and performs segmentation processing on the motion video to obtain multiple video clips; obtains each frame image in the video clip, and performs feature extraction on each frame image to obtain optical flow information corresponding to each frame image; parses the optical flow information to obtain an optical flow vector, and determines a label corresponding to the optical flow information according to the optical flow vector; determines the image features of each frame image in the video clip according to the optical flow information and its corresponding label; determines the image corresponding to the key action frame from the video clip according to the image features of each frame image in the video clip, generates a target video clip according to the image corresponding to the key action frame, and sends the target video clip to the terminal device 110. Among them, the server 120 can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 110 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0035] See also Figure 2 , Figure 2 This is a schematic flow chart of a method for generating a video clip provided in an embodiment of the present application. The method for generating a video clip can obtain the optical flow information corresponding to each frame of the image, thereby predicting the probability of each frame being a key action frame through the optical flow information, and can extract key information more clearly and exclude interference information, thereby greatly improving the recognition rate of the action, and further improving the recognition accuracy of the key action frame.

[0036] like Figure 2 As shown, the method for generating the video clip includes steps S101 to S106.

[0037] S101: Obtain a motion video, and segment the motion video to obtain a plurality of video clips.

[0038] The sports video may include a badminton match video, a football match video, a basketball match video, etc. The video clip may be a complete attack clip or a highlight clip in the sports video.

[0039] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0040] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0041] In some embodiments, target detection processing is performed on the motion video to obtain first target position information, second target position information, and third target position information; if the first target position information is detected to coincide with the second target position information at the first moment, the first moment is used as the initial moment of the segment; if the first target position information is detected to coincide with the third target position information at the second moment, the second moment is used as the end moment of the segment; and a video segment is generated based on the initial moment of the segment and the end moment of the segment. In this way, a video segment can be accurately obtained from the motion video clip based on the first target position information, the second target position information, and the third target position information.

[0042] The first target position information, the second target position information and the third target position information correspond to different detection targets. The first target position information may be the position information of the first detection target in the video, and the first detection target may be badminton, football, basketball, etc. The second target position information may be the position information of the second detection target in the video, and the second detection target may be hands, feet, etc. The third target position information may be the position information of the third detection target in the video, and the third detection target may be a venue floor, door frame, basket, etc.

[0043] Exemplarily, taking badminton video as an example, the first detection target is a badminton, the second detection target is a hand, and the third detection target is a venue floor. If the first target position information is detected to coincide with the second target position information at the first moment, it can be considered that the badminton is in the player's hand, and the player is preparing to kick off, so the first moment is taken as the initial moment of the segment; if the first target position information is detected to coincide with the third target position information at the second moment, it can be considered that the badminton is on the venue floor, and the round is considered to have ended, so the second moment is taken as the end moment of the segment; since the initial moment of the segment and the end moment of the segment can be respectively considered as the beginning and end of a batting round, a video segment can be generated through the initial moment of the segment and the end moment of the segment, and so on, multiple video segments can be generated based on each round.

[0044] Exemplarily, taking a football video as an example, the first detection target is the football, the second detection target is the foot, and the third detection target is the goal frame. If the first target position information is detected to coincide with the second target position information at the first moment, it can be considered that the football is on the player's foot, that is, the player is ready to touch the ball, so the first moment is taken as the initial moment of the segment; if the first target position information is detected to coincide with the third target position information at the second moment, it can be considered that the football is near the goal frame, that is, a threatening attack is generated and the round is ended, so the second moment is taken as the end moment of the segment; since the initial moment of the segment and the end moment of the segment can be respectively considered as the beginning and end of an offensive round, a video segment can be generated through the initial moment of the segment and the end moment of the segment, and so on, multiple video segments can be generated based on each round.

[0045] It should be noted that the above-mentioned method for generating video segments is only an example, and any motion video and corresponding targets can be adjusted accordingly to perform video segmentation to generate video segments.

[0046] S102: Obtain each frame of image in each video clip, and perform feature extraction on each frame of image to obtain optical flow information corresponding to each frame of image.

[0047] The video clip may include multiple frames of images. Optical flow information is a concept in object motion detection in the field of view, which is used to describe the motion of an observed target, surface or edge caused by the motion relative to the observer.

[0048] In some embodiments, for any frame of image, the pixel position information of the image and the pixel position information of the adjacent images are determined; and the optical flow information corresponding to the image is generated based on the pixel position information of the image and the adjacent images. Thus, the optical flow information corresponding to the image can be accurately generated based on the pixel position information of the image and the adjacent images.

[0049] The pixel position information of the image can be used to represent the position information of each pixel in the image. The adjacent images can be used to represent images of adjacent frames, such as the image of the previous frame and the image of the next frame.

[0050] Specifically, for any frame image, determine the position information of the pixel points of the frame image and the position information of the pixel points of the adjacent frame image; based on the pixel position information of the frame image and the adjacent frame image, the optical flow information corresponding to the frame image can be extracted according to the displacement difference between the pixel points in the frame image and the adjacent frame image, and so on, so as to calculate the optical flow information corresponding to each frame image in the video clip.

[0051] In some embodiments, the image and the adjacent image are converted into their respective corresponding grayscale images; based on the grayscale images, the optical flow vectors of each pixel between the image and the adjacent image are calculated; the eigenvalues ​​of each optical flow vector are calculated, and the eigenvalues ​​of each optical flow vector are used as the optical flow information corresponding to the image. In this way, the optical flow information corresponding to the image can be accurately extracted.

[0052] Specifically, two consecutive frames of images can be converted into their corresponding grayscale images, a specific function can be called to calculate the optical flow vector of each pixel between the two grayscale images, all optical flow vectors can be traversed, the sizes and angles of all optical flow vectors can be calculated, and they can be saved as eigenvalues.

[0053] For example, for the i-th segment X i , taking T as the size of a sliding window, the optical flow information of each frame is obtained, which is expressed as follows:

[0054] I=(I1,I2,…I T :I t ∈R d×d×O )

[0055] Among them I t ∈R d×d×O The d×d represents the size of the image, and O represents the dimension of the optical flow feature.

[0056] S103, parsing the optical flow information to obtain an optical flow vector, and determining a label corresponding to the optical flow information according to the optical flow vector.

[0057] Specifically, the optical flow information may be parsed to obtain the characteristic value of the optical flow vector, that is, the size and angle of the optical flow vector, and the label corresponding to the optical flow information may be determined according to the size and angle of the optical flow vector.

[0058] The labels corresponding to the optical flow information may include positive labels and negative labels. Taking a badminton game video as an example, the positive label may be the optical flow information corresponding to a batting action, and the negative label may be the optical flow information corresponding to a non-batting action.

[0059] Exemplarily, the label corresponding to each frame of image is as follows:

[0060] e=(e1,e2,…e T :e t ∈R 2 )

[0061] Among them, e t ∈R 2 Since a badminton match video is used as an example in the embodiment of the present application, only two actions are identified, namely, a hitting action and a non-hitting action. The specific settings can be made according to actual conditions and are not specifically limited here.

[0062] S104: Determine the image features of each frame of the video clip according to the optical flow information and its corresponding label.

[0063] The image feature may be a high-dimensional optical flow feature of each frame of an image in a video clip.

[0064] In some embodiments, based on the feature extractor, feature extraction is performed based on the optical flow information and its corresponding labels to obtain low-dimensional optical flow features; the low-dimensional feature matrix channel is subjected to dimensionality-upgrading processing to obtain high-dimensional optical flow features, and the high-dimensional optical flow features are used as image features corresponding to the optical flow information. In this way, key information can be extracted more clearly based on the optical flow information and interference information can be eliminated, which can greatly improve the recognition rate of actions.

[0065] The feature extractor may be a feature extractor corresponding to a model such as a MobileNet model, the low-dimensional optical flow feature is an optical flow feature of a low-dimensional subspace manifold region, and the high-dimensional optical flow feature is an optical flow feature of a high-dimensional subspace manifold region. Generally speaking, when the input dimension is low-dimensional such as 2 or 3, the output loses more information than the input; however, when the input dimension is high-dimensional such as 15-30, the output retains more information of the input. Therefore, when the input dimension is small, the corresponding information loss is very serious, and when the input dimension is large, the input manifold can be restored better.

[0066] Exemplarily, the image features of each frame of image can be extracted through optical flow information and its corresponding labels. Specifically, a model can be used to extract the image features of each frame of image, which is expressed as follows:

[0067] [F t ,F t-1 ,F t-2 ,…,F t-T ]=Encoder([I t ,I t-1 ,I t-2 ,…,I t-T ])

[0068] Among them, Encoder() represents a feature extractor, which can be the feature extractor corresponding to the MobileNet model. t ,F t-1 ,F t-2 ,…,F t-T ] is the image feature of each frame image.

[0069] S105 . Predict each image frame in the video clip according to the image features of each image frame in the video clip to obtain a key frame probability of each image frame.

[0070] The key action frame may be an image frame corresponding to key information, such as an image frame corresponding to a batting action, an image frame corresponding to a wonderful shot, etc. The key frame probability may be used to indicate the probability that each frame is a key action frame.

[0071] Specifically, whether each frame of the image in the video clip is a key action frame can be predicted based on the image features of each frame of the image in the video clip, thereby obtaining the key frame probability of each frame of the image.

[0072] In some embodiments, for the image features of any frame of image, the image features of the image and the adjacent images are obtained; based on the image features of the image and the adjacent images, the action in the image is predicted to obtain the key frame probability of the image. Thus, the key frame probability of the image can be accurately predicted based on the image features.

[0073] Specifically, the image features of the current frame image and the adjacent frame images may be obtained; based on the image features of the current frame image and the adjacent frame images, the action in the current frame image is predicted to obtain the key frame probability of the image.

[0074] For example, the probability that each frame image is an image corresponding to a key action frame can be calculated, where the probability P of the current frame image being an image corresponding to a key action frame is t The calculation is as follows:

[0075] P t =LSTM([F t ,F t-1 ,F t-2 ,…,F t-T ])

[0076] Among them, it is necessary to input the image features of adjacent frame images, for example, the image features of the previous T frames can be input to calculate the probability P that the current frame image is the image corresponding to the key action frame t .

[0077] S106 , determining an image corresponding to a key action frame according to the key frame probability of each frame of image, and generating a target video segment according to the image corresponding to the key action frame.

[0078] Among them, the images corresponding to the key action frames can be used to combine and generate target video clips, and the target video clips can be corresponding video clips in scenes such as highlights and highlight moments.

[0079] Specifically, one or more frames of images with the highest probability of key frames can be used as images corresponding to key action frames, or one or more frames of images with a probability of key frames higher than a preset threshold can be used as images corresponding to key action frames, without specific limitation, and then the images corresponding to the key action frames are combined and edited to generate the target video clip. In this way, the images corresponding to the key action frames can be accurately determined, so that the key information can be clearly extracted and the interference information can be eliminated, which can greatly improve the recognition rate of the action, so as to accurately generate the target video clip.

[0080] In some embodiments, it is determined whether the key frame probability of each frame of the image exceeds a preset probability value; if there is a frame of the image whose key frame probability exceeds the preset probability value, the frame of the image is used as the image corresponding to the key action frame.

[0081] The preset probability value may be any probability value, which may be set according to actual conditions and is not specifically limited here.

[0082] For example, if the key frame probabilities of each frame image are determined to be 90%, 80% and 70% respectively, the image corresponding to the key frame probability of 90% can be used as the image corresponding to the key action frame. If the preset probability value is 75%, the images corresponding to the key frame probabilities of 90% and 80% can be used as the images corresponding to the key action frame.

[0083] The video clip generation method, video clip generation device, computer equipment and computer-readable storage medium disclosed in the embodiment of the present application obtain a motion video and segment the motion video to obtain multiple video clips; obtain each frame image in the video clip, and extract features for each frame image to obtain the optical flow information corresponding to each frame image; parse the optical flow information to obtain an optical flow vector, and determine the label corresponding to the optical flow information according to the optical flow vector; determine the image features of each frame image in the video clip according to the optical flow information and its corresponding label; determine the image corresponding to the key action frame from the video clip according to the image features of each frame image in the video clip. In this way, by obtaining the optical flow information corresponding to each frame image, the probability of each frame being a key action frame can be predicted through the optical flow information, and the key information can be extracted more clearly, and the interference information can be eliminated, so that the recognition rate of the action can be greatly improved, and then the recognition accuracy of the key action frame can be improved.

[0084] See also Figure 3 , Figure 3 It is a schematic block diagram of a video segment generation device provided in an embodiment of the present application. The video segment generation device can be configured in a server to execute the aforementioned video segment generation method.

[0085] like Figure 3 As shown, the video segment generation device 200 includes: a video segmentation module 201, an optical flow information generation module 202, a label generation module 203, a feature extraction module 204, a key frame prediction module 205 and a video segment generation module 206.

[0086] The video segmentation module 201 is used to obtain a motion video and segment the motion video to obtain multiple video clips;

[0087] The optical flow information generating module 202 is used to obtain each frame of the image in each of the video clips, and perform feature extraction on each frame of the image to obtain the optical flow information corresponding to each frame of the image;

[0088] A label generation module 203 is used to parse the optical flow information to obtain an optical flow vector, and determine a label corresponding to the optical flow information according to the optical flow vector;

[0089] A feature extraction module 204, configured to determine the image features of each frame of the video clip according to the optical flow information and its corresponding label;

[0090] A key frame prediction module 205 is used to predict each frame of the image in the video segment according to the image features of each frame of the image in the video segment, and obtain a key frame probability of each frame of the image;

[0091] The video segment generating module 206 is used to determine the image corresponding to the key action frame according to the key frame probability of each frame of the image, and generate a target video segment according to the image corresponding to the key action frame.

[0092] In some embodiments, the video segmentation module 201 is further used to perform target detection processing on the motion video to obtain first target position information, second target position information and third target position information; if the first target position information is detected to coincide with the second target position information at the first moment, the first moment is used as the initial moment of the segment; if the first target position information is detected to coincide with the third target position information at the second moment, the second moment is used as the end moment of the segment; and a video segment is generated according to the initial moment of the segment and the end moment of the segment.

[0093] In some embodiments, the optical flow information generation module 202 is also used to determine the pixel position information of the image and the pixel position information of the adjacent images for any frame of the image; and generate the optical flow information corresponding to the image based on the pixel position information of the image and the adjacent images.

[0094] In some embodiments, the optical flow information generation module 202 is also used to convert the image and the adjacent images into corresponding grayscale images; based on the grayscale images, calculate the optical flow vector of each pixel point between the image and the adjacent images; calculate the eigenvalue of each of the optical flow vectors, and use the eigenvalue of each of the optical flow vectors as the optical flow information corresponding to the image.

[0095] In some embodiments, the feature extraction module 204 is also used to perform feature extraction based on the optical flow information and its corresponding labels based on the feature extractor to obtain low-dimensional optical flow features; perform dimensionality upscaling on the low-dimensional feature matrix channel to obtain high-dimensional optical flow features, and use the high-dimensional optical flow features as image features corresponding to the optical flow information.

[0096] In some embodiments, the key frame prediction module 205 is also used to obtain the image features of any frame image and the image features of the image and the adjacent images; based on the image features of the image and the adjacent images, the action in the image is predicted to obtain the key frame probability of the image.

[0097] In some embodiments, the video segment generation module 206 is further configured to determine whether the key frame probability of each frame of the image exceeds a preset probability value; if there is a frame of the image with a key frame probability exceeding the preset probability value, the image of the frame is used as the image corresponding to the key action frame.

[0098] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0100] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.

[0101] See also Figure 4 , Figure 4 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0102] like Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0103] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for generating a video segment.

[0104] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0105] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for generating a video segment.

[0106] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0108] Among them, in some embodiments, the processor is used to run a computer program stored in the memory to implement the following steps: obtain a motion video, and segment the motion video to obtain multiple video clips; obtain each frame image in each of the video clips, and extract features for each frame image to obtain optical flow information corresponding to each frame image; parse the optical flow information to obtain an optical flow vector, and determine a label corresponding to the optical flow information based on the optical flow vector; determine image features of each frame image in the video clip based on the optical flow information and its corresponding label; predict each frame image in the video clip based on the image features of each frame image in the video clip to obtain a key frame probability of each frame image; determine the image corresponding to the key action frame based on the key frame probability of each frame image, and generate a target video clip based on the image corresponding to the key action frame.

[0109] In some embodiments, the processor is also used to perform target detection processing on the motion video to obtain first target position information, second target position information and third target position information; if it is detected at a first moment that the first target position information coincides with the second target position information, the first moment is used as the initial moment of the segment; if it is detected at a second moment that the first target position information coincides with the third target position information, the second moment is used as the end moment of the segment; and a video segment is generated based on the initial moment of the segment and the end moment of the segment.

[0110] In some embodiments, the processor is further used to determine the pixel position information of the image and the pixel position information of the adjacent image for any frame of the image; and generate the optical flow information corresponding to the image based on the pixel position information of the image and the adjacent image.

[0111] In some embodiments, the processor is also used to convert the image and the adjacent images into corresponding grayscale images; based on the grayscale images, calculate the optical flow vector of each pixel point between the image and the adjacent images; calculate the eigenvalue of each of the optical flow vectors, and use the eigenvalue of each of the optical flow vectors as the optical flow information corresponding to the image.

[0112] In some embodiments, the processor is also used to perform feature extraction based on the optical flow information and its corresponding labels based on a feature extractor to obtain low-dimensional optical flow features; perform dimensionality upscaling on the low-dimensional feature matrix channel to obtain high-dimensional optical flow features, and use the high-dimensional optical flow features as image features corresponding to the optical flow information.

[0113] In some embodiments, the processor is further used to obtain image features of any frame of image and image features of adjacent images; predict the action in the image based on the image features and image features of adjacent images to obtain the key frame probability of the image.

[0114] In some embodiments, the processor is further used to determine whether the key frame probability of each frame of the image exceeds a preset probability value; if there is a frame of the image whose key frame probability exceeds the preset probability value, the image of this frame is used as the image corresponding to the key action frame.

[0115] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any method for generating a video clip provided in the embodiment of the present application is implemented.

[0116] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0117] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0118] The present invention refers to a new application model of computer technologies such as storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. of the blockchain language model. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a video clip, characterized in that: The method comprises: Acquire a motion video, and segment the motion video to obtain multiple video clips; Obtain each frame of the image in each of the video clips, and perform feature extraction on each frame of the image to obtain optical flow information corresponding to each frame of the image; Parsing the optical flow information to obtain an optical flow vector, and determining a label corresponding to the optical flow information according to the optical flow vector; Determining image features of each frame of the video clip according to the optical flow information and its corresponding label; Predicting each frame of the image in the video clip according to the image features of each frame of the image in the video clip to obtain a key frame probability of each frame of the image; An image corresponding to a key action frame is determined according to the key frame probability of each frame of the image, and a target video segment is generated according to the image corresponding to the key action frame.

2. The method according to claim 1, characterized in that The segmentation process of the motion video to obtain a plurality of video clips includes: Performing target detection processing on the motion video to obtain first target position information, second target position information, and third target position information, wherein the first target position information, the second target position information, and the third target position information correspond to different detection targets; If it is detected at a first moment that the first target position information coincides with the second target position information, the first moment is used as the initial moment of the segment; If it is detected at a second moment that the first target position information coincides with the third target position information, the second moment is taken as the end moment of the segment; A video segment is generated according to the segment start time and the segment end time.

3. The method according to claim 1, characterized in that The extracting features of each frame of the image to obtain the optical flow information corresponding to each frame of the image includes: For any frame of the image, determining pixel position information of the image and pixel position information of adjacent images; Optical flow information corresponding to the image is generated according to pixel position information of the image and the adjacent images.

4. The method according to claim 3, characterized in that The step of generating optical flow information corresponding to each frame of the image according to pixel position information of the image and the adjacent images includes: Converting the image and the adjacent images into respective corresponding grayscale images; Based on the grayscale image, calculating the optical flow vector of each pixel point between the image and the adjacent image; The eigenvalues ​​of the optical flow vectors are calculated, and the eigenvalues ​​of the optical flow vectors are used as the optical flow information corresponding to the image.

5. The method according to claim 1, characterized in that: The step of determining the image features of each frame of the image in the video clip according to the optical flow information and its corresponding label includes: Based on the feature extractor, feature extraction is performed according to the optical flow information and its corresponding labels to obtain low-dimensional optical flow features; The low-dimensional feature matrix channel is subjected to dimensionality increase processing to obtain a high-dimensional optical flow feature, and the high-dimensional optical flow feature is used as the image feature corresponding to the optical flow information.

6. The method according to claim 1, characterized in that The step of predicting each frame of the image in the video clip according to the image features of each frame of the image in the video clip to obtain a key frame probability of each frame of the image includes: For image features of any frame of image, obtain image features of the image and adjacent images; The action in the image is predicted according to the image features and the image features of adjacent images to obtain the key frame probability of the image.

7. The method according to claim 1, characterized in that The step of determining the image corresponding to the key action frame according to the key frame probability of each frame of the image includes: Determine whether the key frame probability of each frame of the image exceeds a preset probability value; If there is a frame of the image whose key frame probability exceeds a preset probability value, the image of the frame is used as the image corresponding to the key action frame.

8. A device for generating a video clip, characterized in that: The video clip generation device comprises: A video segmentation module is used to obtain a motion video and segment the motion video to obtain multiple video clips; An optical flow information generation module is used to obtain each frame of the image in each of the video clips, and perform feature extraction on each frame of the image to obtain optical flow information corresponding to each frame of the image; A label generation module, used to parse the optical flow information to obtain an optical flow vector, and determine a label corresponding to the optical flow information according to the optical flow vector; A feature extraction module, used to determine the image features of each frame of the video clip according to the optical flow information and its corresponding label; A key frame prediction module, used to predict each frame of the image in the video clip according to the image features of each frame of the image in the video clip, and obtain a key frame probability of each frame of the image; The video segment generation module is used to determine the image corresponding to the key action frame according to the key frame probability of each frame of the image, and generate a target video segment according to the image corresponding to the key action frame.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement, when executing the computer program: A method for generating a video clip as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method for generating a video segment according to any one of claims 1 to 7.

Images