Video description information determination method and device, electronic equipment and storage medium

Abstract

The invention provides a video description information determination method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring video data, wherein the video data comprises at least two video frames; determining the posture of the camera when each video frame is shot; according to postures of a camera when different video frames in the video data are shot, determining parameter data when the camera moves in a shooting process of the video data, the parameter data being used for representing kinematics characteristics of the camera; according to the parameter data, mirror moving information corresponding to the video data is determined, and the mirror moving information is used for representing the change track of the video data in the shooting process when the camera moves; and performing text description on the video content of the video data according to the mirror moving information to obtain video description information of the video data, the video description information being used for representing the text description of the video content of the video data.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device, electronic device and storage medium for determining video description information. Background Art

[0002] With the explosive growth of videos, the demand for video content analysis is also growing. Video content analysis can be performed using video description information, where video description information refers to text descriptions of video content such as objects, scenes, and events.

[0003] At present, video processing can be performed based on video description information, such as video search, video editing and video recommendation, etc. Therefore, it is very important to determine the video description information. In the related art, the determination of video description information mainly focuses on the semantic understanding of video content, for example, by identifying information such as objects, scenes and events in the video, and determining the video description information based on this information. However, the video description information determined in this way is not comprehensive enough, resulting in low efficiency of video processing based on video description information. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, electronic device and storage medium for determining video description information, so as to solve the problem that the video description information obtained by the video description method in the related art is not comprehensive enough, resulting in low efficiency of video processing when the video description information is applied. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, a method for determining video description information is first provided, the method comprising:

[0006] Acquire video data, where the video data includes at least two video frames;

[0007] Determining the posture of the camera when each of the video frames is captured;

[0008] Determine parameter data of the camera in motion during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, wherein the parameter data is used to represent the kinematic characteristics of the camera;

[0009] Determine, according to the parameter data, camera movement information corresponding to the video data, wherein the camera movement information is used to indicate a change trajectory of the video data when the camera moves during shooting;

[0010] A text description is performed on the video content of the video data according to the camera movement information to obtain video description information of the video data, where the video description information is used to represent a text description of the video content of the video data.

[0011] In a second aspect of the present invention, a device for determining video description information is provided, the device comprising:

[0012] A first acquisition module, used to acquire video data, wherein the video data includes at least two video frames;

[0013] A first determination module, used to determine the posture of the camera when each video frame is shot;

[0014] A second determination module is used to determine parameter data of the camera movement during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, wherein the parameter data is used to represent the kinematic characteristics of the camera;

[0015] A third determination module, used to determine the camera movement information corresponding to the video data according to the parameter data, wherein the camera movement information is used to indicate the change track of the video data when the camera moves during the shooting process;

[0016] A text description module is used to perform a text description on the video content of the video data according to the camera movement information to obtain video description information of the video data, wherein the video description information is used to represent a text description of the video content of the video data.

[0017] In the third aspect of the implementation of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; the processor is used to implement any of the above-mentioned methods for determining the video description information when executing the program stored in the memory.

[0018] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute any of the above-mentioned methods for determining video description information.

[0019] In a fifth aspect of the implementation of the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for determining video description information.

[0020] In an embodiment of the present invention, by acquiring video data, the video data includes at least two video frames; determining the posture of the camera when each of the video frames is shot; determining parameter data of the camera when the camera moves during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, the parameter data is used to represent the kinematic characteristics of the camera; according to the parameter data, determining the mirror movement information corresponding to the video data, the mirror movement information is used to represent the change trajectory of the camera when the video data moves during the shooting process; according to the mirror movement information, the video content of the video data is described in text to obtain the video description information of the video data, the video description information is used to represent the text description of the video content of the video data. In this way, the video description information can accurately describe the movement trajectory, angle change trajectory, etc. of the camera, making the video description information more comprehensive, thereby improving the efficiency of video processing according to the video description information. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0022] Figure 1 is a flow chart of a method for determining video description information in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the structure of a device for determining video description information in an embodiment of the present invention;

[0024] Figure 3 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] See also Figure 1 , Figure 1 Flow chart of a method for determining video description information in an embodiment of the present invention. The method for determining video description information includes:

[0027] Step S101: acquiring video data, where the video data includes at least two video frames.

[0028] The video data may be any video data, or may be video data in which the camera moves during the shooting process, wherein the camera moves during the shooting process, which may refer to: the movement of the camera position and the change of the camera angle during the shooting process.

[0029] The playback time of the video data can be arbitrary and is not specifically limited here. In some embodiments, such as a video generation scenario, the video generation model can be trained by annotating the video description information of the video data. In this scenario, the playback time of the video data is usually relatively short. If the acquired video is a long video, the long video can be truncated to obtain multiple video clips with relatively short playback time, and each video clip can be used as the video data of this embodiment.

[0030] It can obtain video data shot in real time, receive video data sent by other electronic devices, and obtain video data on video platforms.

[0031] In some embodiments, for example, the video data is video data in which the camera has motion during the shooting process. The camera position and angle recorded during the shooting process can be used to determine whether the camera has motion during the shooting process. If it is determined that the camera has motion during the shooting process, the captured video can be used as the video data in this embodiment. The camera posture corresponding to the captured video frame can also be calculated by a Simultaneous Localization And Mapping (SLAM) algorithm, so that if it is determined that the camera has motion during the shooting process according to the camera posture, the captured video can be used as the video data in this embodiment.

[0032] The at least two video frames may be all video frames in the video data, and the at least two video frames may also be video frames obtained by performing frame extraction on the video data, which is not specifically limited here.

[0033] Step S102, determining the posture of the camera when each of the video frames is captured.

[0034] In some embodiments, the posture of the camera when each video frame is shot can be determined by the camera position and angle recorded when the video frame is shot. In some embodiments, the posture of the camera corresponding to the video frame can be calculated by a SLAM algorithm. In some embodiments, the posture of the camera when the video frame is shot can be determined by a first model, wherein the first model can be a deep learning model.

[0035] The camera posture may include at least one of the camera position and angle. Assume the position of the camera in the i-th video frame, that is, the position of the camera when the i-th video frame is captured, denoted by P i =(x i ,y i ,z i ) means, x i ,y i and zi They can be the coordinates of the camera in the x-axis, y-axis and z-axis directions in the pre-constructed xyz coordinate system, and the angle is θ i =(α i ,β i ,γ i ) indicates that α i , β i and γ i They can be the roll angle, pitch angle and heading angle of the camera respectively.

[0036] When the camera posture when each video frame is shot is determined, the camera posture when each video frame is shot may be aggregated in ascending order of the timestamp of each video frame, so as to obtain the first posture data.

[0037] Step S103, determining parameter data of the camera in motion during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, wherein the parameter data is used to represent the kinematic characteristics of the camera.

[0038] Among them, kinematic characteristics refer to the characteristics of the movement rules of the moving parts and their changing rules. Kinematic characteristics include information such as movement time, movement speed, movement direction, movement trajectory, etc., and are one of the important contents of kinematic research.

[0039] In some embodiments, the parameter data of the camera moving during the video data shooting process can be determined based on the posture change of two adjacent video frames in the first posture data. In some embodiments, the parameter data of the camera moving during the video data shooting process can be determined based on the posture change of two adjacent video frames in the second posture data, and the second posture data is obtained by smoothing the first posture data.

[0040] In some embodiments, the parameter data may include the movement speed, movement direction and rotation angle of the camera during the video frame shooting process. Assume that the positions of the camera when two adjacent video frames are shot are P i =(x i ,y i ,z i ) and P i+1 =(x i+1 ,y i+1 ,z i+1 ), the angles are θ i =(α i ,β i ,γ i ) and θ i+1 =(α i+1 ,β i+1 ,γ i+1). When the camera is shooting the i-th video frame, its movement speed v i It can be calculated by the following formula (1): i The rotation angle Δθ can be calculated by the following formula (2): i It can be calculated by the following formula (3).

[0041]

[0042] Δθ i =(α i+1 -α i ,β i+1 -β i ,γ i+1 -γ i ) (3)

[0043] The parameters of the camera when it moves during the process of shooting each video frame can be aggregated according to the time stamp of each video frame from small to large, so as to obtain the parameter data of the camera when it moves during the video data shooting process.

[0044] Step S104: determining the camera movement information corresponding to the video data according to the parameter data, wherein the camera movement information is used to indicate the changing trajectory of the video data when the camera moves during the shooting process.

[0045] The change trajectory of the camera during movement may include at least one of a movement change trajectory and a rotation angle change trajectory.

[0046] In some embodiments, the camera movement information during the video frame shooting process can be determined based on the position change and rotation angle change of two adjacent video frames in the first posture information.

[0047] In some embodiments, the target video frame in the video data can be determined based on the parameter data; and the camera movement method when the target video frame is shot can be determined based on the parameter data when the camera moves during the shooting of the target video frame; and the camera movement information can be determined based on the camera movement method when each target video frame in the video data is shot. Among them, the camera movement information is determined by identifying the key camera movements during the video shooting process, and aggregating the identified key camera movements to obtain the camera movement information corresponding to the video data. In some embodiments, the camera movement method during the shooting of each video frame can be aggregated in the order of the timestamp of each video frame from small to large to obtain the camera movement information during the shooting of the video data.

[0048] In some embodiments, trajectory fitting can be performed according to the movement speed and direction of the camera during the video frame shooting process to obtain the movement change trajectory of the camera during the video data shooting process, and trajectory fitting can be performed according to the rotation angle of the camera during the video frame shooting process to obtain the rotation angle change trajectory of the camera during the video data shooting process.

[0049] For example, when determining the position change of two adjacent video frames based on the movement speed of the camera during the video frame shooting process, it can be determined that the camera movement method during the video frame shooting process is lens movement, and the direction of lens movement can be determined by the movement direction. If the movement direction is from left to right, the lens movement direction can be from left to right. Accordingly, the camera movement information can indicate that the movement change trajectory of the camera during movement includes lens movement from left to right.

[0050] For another example, when determining the change in the rotation angle of two adjacent video frames, it can be determined that the camera movement method during the video frame shooting process is lens rotation, and the camera movement direction can be determined according to the angle of change in the rotation angle. If the rotation angle of the roll angle changes, the camera movement information during the video frame shooting process can indicate: the change trajectory when the camera moves includes the camera rotating the lens according to the roll angle. If the rotation angle of the pitch angle changes, the camera movement information during the video frame shooting process can indicate: the change trajectory when the camera moves includes the lens rotating according to the pitch angle. If the rotation angle of the heading angle changes, the camera movement information during the video frame shooting process can indicate: the change trajectory when the camera moves includes the lens rotating according to the heading angle.

[0051] For example, when it is determined that the rotation angle change or movement speed change of two adjacent video frames exceeds a preset range, it can be determined that the camera movement method during the video frame shooting process is lens scene switching. The camera movement information during the video frame shooting process can indicate: the change trajectory of the camera movement includes lens scene switching according to the rotation angle change or movement speed change.

[0052] Step S105 , performing a text description on the video content of the video data according to the camera movement information to obtain video description information of the video data, wherein the video description information is used to represent a text description of the video content of the video data.

[0053] The video description information may include camera movement description information, which is used to represent the text description of the camera movement used to shoot the video content of the video data. The video description information may also include semantic description information, which is used to represent the text description of the semantics of the video content of the video data.

[0054] In some embodiments, the camera movement information corresponding to the video data can be processed by an existing or new algorithm to generate video description information of the video data. For example, the camera movement information can be analyzed to determine whether there is a video frame in which the camera movement mode has changed during the shooting process, and then the camera movement mode during the shooting process of the video frame can be connected with the camera movement mode during the shooting process of the previous video frame of the video frame through a conjunction, thereby generating the camera movement description information of the video data.

[0055] For example, the playing time of the video data is 10 seconds, and the video data includes video frame 1, video frame 2, ..., video frame 100, including 10 video frames per second. Among them, the mirror movement information during the shooting process of video frame 1, video frame 2, ..., video 10 indicates that the camera moves the lens from left to right, the mirror movement information during the shooting process of video 11, video 12, ..., video 20 indicates that the camera moves the lens from back to front, and the mirror movement information during the shooting process of video 21, video 22, ..., video 100 indicates that the camera rotates the lens according to the pitch angle. In this scenario, the video frames in which the camera movement mode has changed can be identified as video frame 11 and video frame 21. By setting the conjunction "then", the camera movement information of different camera movement modes are connected. Accordingly, the generated camera movement description information of the video data may include "when shooting the video frame with a timestamp of 1 second, the camera first moves the lens from left to head, and then, when shooting the video frame with a timestamp of 11 seconds, the camera moves the lens from back to front, and then, when shooting the video frame with a timestamp of 21 seconds, the camera rotates the lens according to the pitch angle."

[0056] In some embodiments, adjectives representing the speed of movement can be added to the camera movement description information by quantifying the movement speed and the rotation angle change speed, so that the description of the camera movement information is more specific. In some embodiments, the movement speed and the rotation angle change speed can be quantified by setting corresponding thresholds. When the movement speed is less than a threshold, it can be indicated that the movement speed is slow, when the movement speed is greater than another threshold, it can be indicated that the movement speed is fast, and when the movement speed is between two thresholds, it can be indicated that the movement speed is normal speed.

[0057] For example, the camera movement information during the shooting process of video frame 1, video frame 2, ..., video 10 indicates that the camera moves the lens from left to right at a speed of 1 m / s, the camera movement information during the shooting process of video 11, video 12, ..., video 20 indicates that the camera moves the lens from back to front at a speed of 0.1 m / s, and the camera movement information during the shooting process of video 21, video 22, ..., video 100 indicates that the camera rotates the lens at a pitch angle at a speed of 1 degree / second. In this scenario, the generated camera movement description information of the video data may be "when shooting a video frame with a time stamp of 1 second, the camera first moves the lens from left to head at a faster speed, then, when shooting a video frame with a time stamp of 11 seconds, the camera moves the lens from back to front at a slower speed, then, when shooting a video frame with a time stamp of 21 seconds, the camera rotates the lens at a slower speed according to the pitch angle".

[0058] In some embodiments, the camera movement information corresponding to the video data can be input into the big model for natural language processing, and the big model can output the camera movement description information that is consistent with human understanding and reading.

[0059] In some embodiments, the text description of the change trajectory of the camera during movement can also be added before or after the text description of the camera movement mode, or to other positions. For example, the generated video description information of the video data can also include "during the video data shooting process, the camera moves the lens along the trajectory from left to right and from back to front, and then rotates the lens according to the change trajectory of the pitch angle".

[0060] It should be noted that the camera movement description information in the video description information can be deployed before or after the semantic description information, or the description of the same video frame in the camera movement description information and the semantic description information can be deployed correspondingly, which is not specifically limited here.

[0061] In this embodiment, in the embodiment of the present invention, by acquiring video data, the video data includes at least two video frames; determining the posture of the camera when each of the video frames is shot; according to the posture of the camera when different video frames in the video data are shot, determining the parameter data of the camera when the camera moves during the video data shooting process, the parameter data is used to represent the kinematic characteristics of the camera; according to the parameter data, determining the mirror movement information corresponding to the video data, the mirror movement information is used to represent the change trajectory of the camera when the video data moves during the shooting process; according to the mirror movement information, the video content of the video data is described in text to obtain the video description information of the video data, the video description information is used to represent the text description of the video content of the video data. In this way, the video description information can accurately describe the movement trajectory, angle change trajectory, etc. of the camera, making the video description information more comprehensive, thereby improving the efficiency of video processing according to the video description information.

[0062] Optionally, the step S104 specifically includes:

[0063] Determine a target video frame in the video data according to the parameter data; and determine a camera movement mode when shooting the target video frame according to the parameter data when the camera moves during the shooting of the target video frame;

[0064] The camera movement information is determined according to the camera movement mode when each target video frame in the video data is shot.

[0065] The target video frame in the video data can be determined according to the motion speed and rotation angle of the camera during the process of shooting the video frame. In some embodiments, when the motion speed of the video frame is greater than 0, the video frame can be determined as the target video frame. In some embodiments, when the rotation angle of the video frame is greater than 0 during the shooting, the video frame can be determined as the target video frame.

[0066] In some embodiments, the type of lens movement when shooting the target video frame can be determined based on the characteristics of the camera's movement speed and rotation angle when shooting the video frame, and the camera movement method when shooting the video frame can be determined in combination with the movement direction.

[0067] For example, when the camera is shooting a video frame, when the motion speed is low, it can be determined that the type of lens motion when the video frame is shot is lens movement. When the motion speed is high and exceeds the first preset threshold, it can be determined that the type of lens motion when the video frame is shot is lens scene switching, that is, due to the motion speed, the lens switches from one scene to another. When the rotation angle is low, it can be determined that the type of lens motion when the video frame is shot is lens rotation. When the rotation angle is high and exceeds the second preset threshold, it can be determined that the type of lens motion when the video frame is shot is lens scene switching, that is, due to the rotation angle, the lens switches from one scene to another.

[0068] Afterwards, the camera movement methods when shooting different target video frames can be aggregated, so as to obtain the camera movement information corresponding to the video data. In some embodiments, since a plurality of camera movement methods may be used when shooting a target video frame, in this scenario, the camera movement methods for the same type of lens movement when shooting different target video frames can be aggregated respectively, so as to obtain the camera movement information corresponding to the video data. For example, the camera movement methods for lens movement when shooting different target video frames can be aggregated, and the camera movement methods for lens rotation when shooting different target video frames can be aggregated to obtain the camera movement information corresponding to the video data.

[0069] In this embodiment, the target video frame in the video data is determined according to the parameter data; the camera movement mode when the target video frame is shot is determined according to the parameter data when the camera moves during the shooting process of the target video frame; the camera movement information is determined according to the camera movement mode when each target video frame in the video data is shot. That is, the key camera movement when the video data is shot is identified by the parameter data, so that the camera movement information during the video data shooting process can be marked more simply and accurately, and the description accuracy of the camera movement information is improved.

[0070] Optionally, determining a target video frame in the video data according to the parameter data; and determining a camera movement mode when shooting the target video frame according to the parameter data when the camera moves during shooting of the target video frame, includes at least one of the following:

[0071] When the movement speed of the camera when shooting the first video frame in the video data is greater than zero and less than or equal to a first preset threshold, determine that the first video frame is a target video frame; determine, based on the movement speed and the movement direction of the camera when shooting the first video frame, that the camera movement mode when shooting the first video frame includes a first camera movement mode, the first camera movement mode instructs the camera to move the lens according to the movement direction and the movement speed, and the parameter data includes the movement speed and the movement direction;

[0072] When the rotation angle of the camera when shooting the second video frame in the video data is greater than zero and less than or equal to a second preset threshold, determine that the second video frame is a target video frame; determine, based on the rotation angle, that the camera movement mode when shooting the second video frame includes a second camera movement mode, the second camera movement mode instructs the camera to rotate the lens according to the rotation angle, and the parameter data includes the rotation angle;

[0073] When the movement speed of the camera when shooting the third video frame in the video data is greater than a first preset threshold, determining that the third video frame is a target video frame; and determining that the camera movement mode during the shooting of the third video frame includes a third camera movement mode, and the third camera movement mode instructs the camera to switch the lens scene according to the movement speed;

[0074] When the rotation angle of the camera when shooting the fourth video frame in the video data is greater than the second preset threshold, the fourth video frame is determined to be the target video frame; and it is determined that the camera movement mode during the shooting of the fourth video frame includes a fourth camera movement mode, and the fourth camera movement mode instructs the camera to switch the lens scene according to the rotation angle.

[0075] The first preset threshold and the second preset threshold can be set according to actual conditions and are not specifically limited here.

[0076] For a video frame, if camera movement is used when it is shot, it will be reflected in the kinematic characteristics, that is, this embodiment can determine the target video frame in the video data according to the characteristics of the movement speed and rotation angle of the camera in the process of shooting the video frame, and determine the type of lens movement when the target video frame is shot according to the characteristics of the movement speed and rotation angle of the camera in the process of shooting the video frame, and determine the camera movement method when the target video frame is shot in combination with the movement direction, thereby improving the accuracy of determining the camera movement method when shooting the video frame.

[0077] It should be noted that there may be multiple camera movement modes when shooting a target video frame. For example, the camera movement modes when shooting a target video frame may include at least two of the first camera movement mode, the second camera movement mode, the third camera movement mode, and the fourth camera movement mode. In this scenario, the camera movement modes for the same type of lens movement when shooting different target video frames can be aggregated respectively, so as to obtain the camera movement information corresponding to the video data. For example, the camera movement modes for lens movement when shooting different target video frames can be aggregated, and the camera movement modes for lens rotation when shooting different target video frames can be aggregated to obtain the camera movement information corresponding to the video data.

[0078] Optionally, the step 102 specifically includes:

[0079] Input each of the video frames into the first model to perform a posture prediction operation to obtain the posture of the camera when each of the video frames was shot;

[0080] The posture prediction operation includes: extracting features from the video frame to obtain a feature vector of the video frame, and determining the posture of the camera when the video frame was shot according to the feature vector.

[0081] The first model may be a deep learning model, such as a convolutional neural network (CNN), and the CNN convolutional neural network may be used to determine the posture of the video frame. The CNN convolutional neural network may include a convolution module and a regression module.

[0082] The video frame can be input into the CNN convolutional neural network. Accordingly, the convolution module can extract features from the video frame to obtain a feature vector of the video frame. The regression module can determine the posture of the camera when the video frame was shot based on the feature vector of the video frame.

[0083] It should be noted that before using the CNN convolutional neural network, the CNN convolutional neural network needs to be trained. The video data of the camera's operating behavior during the shooting process can be obtained and used as training data to train the CNN convolutional neural network to learn the mapping relationship between the camera posture when the video frame is shot and the feature vector of the video frame. Therefore, when using the CNN convolutional neural network, the posture of the camera when the video frame was shot can be regressed according to the feature vector of the video frame.

[0084] In this way, the accuracy of determining the camera posture when capturing video frames can be improved.

[0085] Optionally, before step S103, the method further includes:

[0086] Smoothing the first posture data to obtain second posture data, wherein the first posture data includes the determined posture of the camera when each of the video frames is shot;

[0087] The step S103 specifically includes:

[0088] According to the posture of the camera when different video frames are shot in the second posture data, parameter data of the camera moving during the video data shooting process is determined.

[0089] In this embodiment, the first posture data can be smoothed to obtain the second posture data, and the parameter data of the camera in motion during the video data shooting process can be determined according to the posture of the camera when different video frames are shot in the second posture data. In this way, the camera posture estimation result can be smoothed to remove noise, which can improve the accuracy of determining the parameter data when the camera moves during the video data shooting process, thereby improving the accuracy of determining the mirror movement information corresponding to the video data, and further improving the accuracy of determining the video description information of the video data.

[0090] Optionally, after step 105, the method further includes:

[0091] In a case where the video description information does not include preset information, using the video data and the video description information as training data for the second model, wherein the preset information indicates that the video data includes a fifth video frame, and the fifth video frame is shot by using a camera movement method of shot scene switching;

[0092] In the case where the video description information includes preset information, the video description information is segmented according to the preset information; and the video data is segmented according to the fifth video frame indicated by the preset information; and the segmented different video data and the corresponding video description information are used as training data for the second model;

[0093] Training the second model according to the training data;

[0094] Among them, the video description information is the label in the training data, the video data is the input sample in the training data, the second model is used to generate video description information according to the input video data, and the video description information output by the second model includes camera movement description information, and the camera movement description information is used to represent the text description of the camera movement used in the video content of the video data.

[0095] Among them, the second model can be a generation model of video description information, which can be a deep learning model.

[0096] The preset information indicates the fifth video frame with transition in the video data, and can identify whether the video description information includes keywords, such as switching, transition and other keywords. When the preset keywords such as switching and transition are matched in the video description information, it means that the video description information includes the preset information, otherwise it means that the video description information does not include the preset information.

[0097] When the video description information does not include preset information, it means that the video data is a video of a single scene. In this case, the video data and the video description information of the video data can be used as training data for the second model.

[0098] When the video description information includes preset information, it means that the scene corresponding to the video data includes at least two scenes, and it also means that the video data includes a fifth video frame shot by using a camera movement method of switching scenes. Since the second model usually generates video description information for video data in a single scene with better results, for video data including at least two scenes, the video description information can be segmented according to the preset information; and the video data can be segmented according to the fifth video frame indicated by the preset information; and the segmented different video data and the corresponding video description information are used as training data for the second model.

[0099] Among them, when segmenting the information, the camera description information and the semantic description information in the video description information need to be segmented accordingly according to the preset information to obtain the video description information of the video data in different scenes, which can improve the accuracy of the second model training.

[0100] Afterwards, the second model can be trained according to the training data. When the second model is trained, video description information of the video data can be generated by the second model, and the generated video description information includes camera movement description information, so that the video data can be automatically annotated with the video description information by the second model, thereby improving the efficiency of annotating the video description information of the video data.

[0101] In addition, the video description information generated by the second model can be applied to the video platform to perform video processing such as video search, video editing, and video recommendation based on the video description information. The video description information automatically generated by the second model can also be used as input samples in the training data of the video generation model to train the video generation model. The video generation model can be applied to short video scenarios to directly generate short videos without transitions for the video platform.

[0102] In the related art, it is difficult to accurately reproduce the video shooting process because it is impossible to determine the camera lens movement mode, movement trajectory and rotation angle change, that is, it is impossible to restore the shooting process well according to the video description information. In this embodiment, by automatically annotating the video description information, and the video description information includes the camera movement description information, the second model is trained by using the automatically annotated video description information as a label, and the trained second model is used to automatically generate video description information, which is used as an input sample to train the video generation model, so that the video generation model has the ability to accurately reproduce the video shooting process according to the video description information, so as to generate a better video and meet the user's demand for video.

[0103] like Figure 2 As shown, the embodiment of the present invention further provides a device 200 for determining video description information, including:

[0104] A first acquisition module 201, configured to acquire video data, wherein the video data includes at least two video frames;

[0105] A first determination module 202, used to determine the posture of the camera when each of the video frames is captured;

[0106] A second determination module 203, configured to determine parameter data of the camera movement during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, wherein the parameter data is used to represent the kinematic characteristics of the camera;

[0107] A third determination module 204 is used to determine the camera movement information corresponding to the video data according to the parameter data, wherein the camera movement information is used to indicate the change track of the video data when the camera moves during the shooting process;

[0108] The text description module 205 is used to perform a text description on the video content of the video data according to the camera movement information to obtain video description information of the video data, wherein the video description information is used to represent a text description of the video content of the video data.

[0109] Optionally, the third determining module 204 includes:

[0110] A first determining unit is used to determine a target video frame in the video data according to the parameter data; and determine a camera movement mode when shooting the target video frame according to the parameter data when the camera moves during the shooting of the target video frame;

[0111] The second determining unit is used to determine the camera movement information according to the camera movement mode when each target video frame in the video data is shot.

[0112] Optionally, the first determining unit is specifically configured to:

[0113] When the movement speed of the camera when shooting the first video frame in the video data is greater than zero and less than or equal to a first preset threshold, determine that the first video frame is a target video frame; determine, based on the movement speed and the movement direction of the camera when shooting the first video frame, that the camera movement mode when shooting the first video frame includes a first camera movement mode, the first camera movement mode instructs the camera to move the lens according to the movement direction and the movement speed, and the parameter data includes the movement speed and the movement direction;

[0114] When the rotation angle of the camera when shooting the second video frame in the video data is greater than zero and less than or equal to a second preset threshold, determine that the second video frame is a target video frame; determine, based on the rotation angle, that the camera movement mode when shooting the second video frame includes a second camera movement mode, the second camera movement mode instructs the camera to rotate the lens according to the rotation angle, and the parameter data includes the rotation angle;

[0115] When the movement speed of the camera when shooting the third video frame in the video data is greater than a first preset threshold, determining that the third video frame is a target video frame; and determining that the camera movement mode during the shooting of the third video frame includes a third camera movement mode, and the third camera movement mode instructs the camera to switch the lens scene according to the movement speed;

[0116] When the rotation angle of the camera when shooting the fourth video frame in the video data is greater than the second preset threshold, the fourth video frame is determined to be the target video frame; and it is determined that the camera movement mode during the shooting of the fourth video frame includes a fourth camera movement mode, and the fourth camera movement mode instructs the camera to switch the lens scene according to the rotation angle.

[0117] Optionally, the first determining module 202 is specifically configured to:

[0118] Input each of the video frames into the first model to perform a posture prediction operation to obtain the posture of the camera when each of the video frames was shot;

[0119] The posture prediction operation includes: extracting features from the video frame to obtain a feature vector of the video frame, and determining the posture of the camera when the video frame was shot according to the feature vector.

[0120] Optionally, the device further comprises:

[0121] A smoothing processing module, used for smoothing the first posture data to obtain second posture data, wherein the first posture data includes the determined posture of the camera when each of the video frames is shot;

[0122] The second determination module is specifically used to determine parameter data of the camera movement during the video data shooting process according to the posture of the camera when shooting different video frames in the second posture data.

[0123] Optionally, the device further comprises:

[0124] a second acquisition module, configured to use the video data and the video description information as training data for a second model when the video description information does not include preset information, wherein the preset information indicates that the video data includes a fifth video frame, and the fifth video frame is shot by using a camera movement method of shot scene switching;

[0125] a segmentation module, configured to segment the video description information according to the preset information when the video description information includes preset information; and segment the video data according to the fifth video frame indicated by the preset information; and use the segmented different video data and the corresponding video description information as training data for the second model;

[0126] A training module, used for training the second model according to the training data;

[0127] Among them, the video description information is the label in the training data, the video data is the input sample in the training data, the second model is used to generate video description information according to the input video data, and the video description information output by the second model includes camera movement description information, and the camera movement description information is used to represent the text description of the camera movement used in the video content of the video data.

[0128] The embodiment of the present invention further provides an electronic device, such as Figure 3 As shown, it includes a processor 301 , a communication interface 302 , a memory 303 and a communication bus 304 , wherein the processor 301 , the communication interface 302 , and the memory 303 communicate with each other via the communication bus 304 .

[0129] Memory 303, used for storing computer programs;

[0130] The processor 301 is used to execute the program stored in the memory 303 to implement the following steps:

[0131] Acquire video data, wherein the video data includes at least two video frames;

[0132] Determining the posture of the camera when each of the video frames is captured;

[0133] Determine parameter data of the camera in motion during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, wherein the parameter data is used to represent the kinematic characteristics of the camera;

[0134] Determine, according to the parameter data, camera movement information corresponding to the video data, wherein the camera movement information is used to indicate a change trajectory of the video data when the camera moves during shooting;

[0135] A text description is performed on the video content of the video data according to the camera movement information to obtain video description information of the video data, where the video description information is used to represent a text description of the video content of the video data.

[0136] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0137] Determine a target video frame in the video data according to the parameter data; and determine a camera movement mode when shooting the target video frame according to the parameter data when the camera moves during the shooting of the target video frame;

[0138] The camera movement information is determined according to the camera movement mode when each target video frame in the video data is shot.

[0139] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0140] When the movement speed of the camera when shooting the first video frame in the video data is greater than zero and less than or equal to a first preset threshold, determine that the first video frame is a target video frame; determine, based on the movement speed and the movement direction of the camera when shooting the first video frame, that the camera movement mode when shooting the first video frame includes a first camera movement mode, the first camera movement mode instructs the camera to move the lens according to the movement direction and the movement speed, and the parameter data includes the movement speed and the movement direction;

[0141] When the rotation angle of the camera when shooting the second video frame in the video data is greater than zero and less than or equal to a second preset threshold, determine that the second video frame is a target video frame; determine, based on the rotation angle, that the camera movement mode when shooting the second video frame includes a second camera movement mode, the second camera movement mode instructs the camera to rotate the lens according to the rotation angle, and the parameter data includes the rotation angle;

[0142] When the movement speed of the camera when shooting the third video frame in the video data is greater than a first preset threshold, determining that the third video frame is a target video frame; and determining that the camera movement mode during the shooting of the third video frame includes a third camera movement mode, and the third camera movement mode instructs the camera to switch the lens scene according to the movement speed;

[0143] When the rotation angle of the camera when shooting the fourth video frame in the video data is greater than the second preset threshold, the fourth video frame is determined to be the target video frame; and it is determined that the camera movement mode during the shooting of the fourth video frame includes a fourth camera movement mode, and the fourth camera movement mode instructs the camera to switch the lens scene according to the rotation angle.

[0144] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0145] Input each of the video frames into the first model to perform a posture prediction operation to obtain the posture of the camera when each of the video frames was shot;

[0146] The posture prediction operation includes: extracting features from the video frame to obtain a feature vector of the video frame, and determining the posture of the camera when the video frame was shot according to the feature vector.

[0147] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0148] Smoothing the first posture data to obtain second posture data, wherein the first posture data includes the determined posture of the camera when each of the video frames is shot;

[0149] According to the posture of the camera when different video frames are shot in the second posture data, parameter data of the camera moving during the video data shooting process is determined.

[0150] Optionally, when the computer program is executed by the processor 301, it is further used for:

[0151] In a case where the video description information does not include preset information, using the video data and the video description information as training data for the second model, wherein the preset information indicates that the video data includes a fifth video frame, and the fifth video frame is shot by using a camera movement method of shot scene switching;

[0152] In the case where the video description information includes preset information, the video description information is segmented according to the preset information; and the video data is segmented according to the fifth video frame indicated by the preset information; and the segmented different video data and the corresponding video description information are used as training data for the second model;

[0153] Training the second model according to the training data;

[0154] Among them, the video description information is the label in the training data, the video data is the input sample in the training data, the second model is used to generate video description information according to the input video data, and the video description information output by the second model includes camera movement description information, and the camera movement description information is used to represent the text description of the camera movement used in the video content of the video data.

[0155] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0156] The communication interface is used for communication between the above electronic device and other devices.

[0157] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the aforementioned processor.

[0158] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0159] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the method for determining video description information described in any of the above embodiments.

[0160] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for determining video description information described in any one of the above embodiments.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0162] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0163] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for determining video description information, characterized in that: The method comprises: Acquire video data, where the video data includes at least two video frames; Determining the posture of the camera when each of the video frames is captured; Determine parameter data of the camera in motion during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, wherein the parameter data is used to represent the kinematic characteristics of the camera; Determine, according to the parameter data, camera movement information corresponding to the video data, wherein the camera movement information is used to indicate a change trajectory of the video data when the camera moves during shooting; A text description is performed on the video content of the video data according to the camera movement information to obtain video description information of the video data, where the video description information is used to represent a text description of the video content of the video data.

2. The method according to claim 1, characterized in that: Determining the camera movement information corresponding to the video data according to the parameter data includes: Determine a target video frame in the video data according to the parameter data; and determine a camera movement mode when shooting the target video frame according to the parameter data when the camera moves during the shooting of the target video frame; The camera movement information is determined according to the camera movement mode when each target video frame in the video data is shot.

3. The method according to claim 2, characterized in that The step of determining a target video frame in the video data according to the parameter data; and determining a camera movement mode when shooting the target video frame according to the parameter data when the camera moves during shooting of the target video frame, includes at least one of the following: When the movement speed of the camera when shooting the first video frame in the video data is greater than zero and less than or equal to a first preset threshold, determine that the first video frame is a target video frame; determine, based on the movement speed and the movement direction of the camera when shooting the first video frame, that the camera movement mode when shooting the first video frame includes a first camera movement mode, the first camera movement mode instructs the camera to move the lens according to the movement direction and the movement speed, and the parameter data includes the movement speed and the movement direction; When the rotation angle of the camera when shooting the second video frame in the video data is greater than zero and less than or equal to a second preset threshold, determine that the second video frame is a target video frame; determine, based on the rotation angle, that the camera movement mode when shooting the second video frame includes a second camera movement mode, the second camera movement mode instructs the camera to rotate the lens according to the rotation angle, and the parameter data includes the rotation angle; When the movement speed of the camera when shooting the third video frame in the video data is greater than a first preset threshold, determining that the third video frame is a target video frame; and determining that the camera movement mode during the shooting of the third video frame includes a third camera movement mode, and the third camera movement mode instructs the camera to switch the lens scene according to the movement speed; When the rotation angle of the camera when shooting the fourth video frame in the video data is greater than the second preset threshold, the fourth video frame is determined to be the target video frame; and it is determined that the camera movement mode during the shooting of the fourth video frame includes a fourth camera movement mode, and the fourth camera movement mode instructs the camera to switch the lens scene according to the rotation angle.

4. The method according to claim 1, characterized in that: Determining the posture of the camera when each video frame is shot includes: Input each of the video frames into the first model to perform a posture prediction operation to obtain the posture of the camera when each of the video frames was shot; The posture prediction operation includes: extracting features from the video frame to obtain a feature vector of the video frame, and determining the posture of the camera when the video frame was shot according to the feature vector.

5. The method according to claim 1, characterized in that Before determining the parameter data of the camera movement during the video data shooting process according to the posture of the camera when shooting different video frames in the video data, the method further includes: Smoothing the first posture data to obtain second posture data, wherein the first posture data includes the determined posture of the camera when each of the video frames is shot; The step of determining parameter data of the camera movement during the video data shooting process according to the posture of the camera when different video frames in the video data are shot includes: According to the posture of the camera when different video frames are shot in the second posture data, parameter data of the camera moving during the video data shooting process is determined.

6. The method according to claim 1, characterized in that After performing a text description of the video content of the video data according to the camera movement information to obtain the video description information of the video data, the method further includes: In a case where the video description information does not include preset information, using the video data and the video description information as training data for the second model, wherein the preset information indicates that the video data includes a fifth video frame, and the fifth video frame is shot by using a camera movement method of shot scene switching; In the case where the video description information includes preset information, the video description information is segmented according to the preset information; and the video data is segmented according to the fifth video frame indicated by the preset information; and the segmented different video data and the corresponding video description information are used as training data for the second model; Training the second model according to the training data; Among them, the video description information is the label in the training data, the video data is the input sample in the training data, the second model is used to generate video description information according to the input video data, and the video description information output by the second model includes camera movement description information, and the camera movement description information is used to represent the text description of the camera movement used in the video content of the video data.

7. A device for determining video description information, characterized in that: The device comprises: A first acquisition module, used to acquire video data, wherein the video data includes at least two video frames; A first determination module, used to determine the posture of the camera when each video frame is shot; A second determination module is used to determine parameter data of the camera movement during the video data shooting process according to the posture of the camera when different video frames in the video data are shot, wherein the parameter data is used to represent the kinematic characteristics of the camera; A third determination module, used to determine the camera movement information corresponding to the video data according to the parameter data, wherein the camera movement information is used to indicate the change track of the video data when the camera moves during the shooting process; A text description module is used to perform a text description on the video content of the video data according to the camera movement information to obtain video description information of the video data, wherein the video description information is used to represent a text description of the video content of the video data.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images