Video focusing model training method, video focusing method, device and equipment
By constructing a video focus model, using optical flow estimation network and dynamic focus network, the blur or out of focus problems caused by fast moving objects in video shooting are solved, and accurate prediction of focus point position and clarity is achieved, and video quality is improved.
Patent Information
- Application Number
- CN202411675407.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-02-25
AI Technical Summary
During video shooting, fast moving objects cause autofocus to fail to follow the object in time, resulting in blurring or out of focus in moving objects in the video.
By constructing a video focus model, using pre-trained optical flow estimation network and dynamic focus network, combining feature extraction and information prediction network, focal length prediction and loss function optimization are performed to train a video focus model that can adjust the focal length in time.
It improves the accurate prediction ability of focus point position and clarity, reduces blur or out of focus of moving objects in the video, and improves the clarity and quality of the video.
Smart Images
Figure CN119520988A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video shooting technology, and in particular to a video focusing model training method, a video focusing method, a device and equipment. Background Art
[0002] In traditional video shooting, the focal length is usually fixed, causing the clarity of the object in the video to change with its distance from the camera. Especially when shooting moving objects, due to the changes in their movement speed and direction, it is often impossible to ensure that the clarity and focal length of the video remain consistent, resulting in the moving objects in the video often appearing blurred or out of focus. Traditional algorithms mainly adjust the focal length based on image contrast or phase difference and other features. However, these methods still have certain limitations. Especially for fast-moving objects, the autofocus often cannot adjust the focal length in time to follow the movement of the object, resulting in blur or out of focus. Summary of the invention
[0003] The main technical problem solved by the present invention is that for fast-moving objects, the automatic focus during video shooting cannot adjust the focal length in time to follow the movement of the object, resulting in blurring or out-of-focus of the moving object in the video.
[0004] According to the first aspect, an embodiment provides a video focus model training method, including:
[0005] Constructing a video focusing model to be trained based on a pre-trained optical flow estimation network and a preset dynamic focusing network; the dynamic focusing network includes a feature extraction network and an information prediction network;
[0006] Inputting the training sample data into the feature extraction network to obtain a feature map corresponding to the training sample data, and performing optical flow estimation processing on the feature map according to the optical flow estimation network to obtain optical flow estimation information; the training sample data includes a video sequence containing adjacent frames;
[0007] The feature map and the optical flow estimation information are merged into final feature information, and the information prediction network is used to output a focal length prediction value corresponding to the final feature information;
[0008] A corresponding target loss function is constructed according to the focal length prediction value, and the video focusing model to be trained is trained based on the target loss function to obtain a trained video focusing model.
[0009] In some embodiments, the training of the to-be-trained video focusing model based on the target loss function to obtain a trained video focusing model includes:
[0010] Calculating a loss value according to the target loss function, and determining a difference between the loss value and a preset loss threshold;
[0011] When the loss value is greater than the loss threshold, the weight parameters in the dynamic focus network in the video focus model to be trained are adjusted, and the video focus model to be trained and the training sample data are used for retraining after the weight parameters in the dynamic focus network are adjusted, until the loss value calculated after the retraining is less than or equal to the loss threshold, and the video focus model to be trained after the weight parameters in the dynamic focus network are adjusted is used as the trained video focus model;
[0012] When the loss value is less than or equal to the loss threshold, the video focusing model to be trained is used as a trained video focusing model.
[0013] In some embodiments, fusing the feature map and the optical flow estimation information into final feature information includes:
[0014] The output feature data in each dimension of the feature map is concatenated with the optical flow feature data in the same dimension in the optical flow estimation information to obtain the fused final feature information.
[0015] In some embodiments, the objective loss function is expressed as:
[0016]
[0017] Among them, L1 represents the target loss function, x represents the focal length prediction value, y represents the preset focal length true value, and n represents the total number of data corresponding to the final feature information.
[0018] In some embodiments, inputting the training sample data into the feature extraction network includes:
[0019] Acquire a first image frame and a second image frame in the training sample data; the first image frame and the second image frame are adjacent frames;
[0020] The first image frame and the second image frame are respectively input into the feature extraction network; wherein the weight parameters of the feature extraction network remain consistent when the first image frame and the second image frame are input.
[0021] In some embodiments, the pre-trained optical flow estimation network is trained in the following manner:
[0022] Acquire at least two adjacent reference image frames, and calculate a pixel displacement value between the at least two adjacent reference image frames;
[0023] At least two adjacent reference image frames are used as training data sets, and the pixel displacement values are used as labels to train a preset optical flow estimation network to obtain a trained optical flow estimation network.
[0024] According to the second aspect, an embodiment provides a video focusing method, including:
[0025] Acquire a video sequence of a target object; the video sequence includes the moving target object captured by a video capture device;
[0026] Inputting the video sequence into a trained video focusing model to obtain a target focal length parameter;
[0027] The target focal length parameter is used to adjust the device focal length of the video shooting device.
[0028] According to a third aspect, an embodiment provides a video focusing device, comprising:
[0029] A parameter determination module is used to obtain a video sequence of a target object; the video sequence includes the moving target object captured by a video capture device; the video sequence is input into a trained video focusing model to obtain a target focal length parameter;
[0030] The focal length adjustment module is used to adjust the device focal length of the video shooting device using the target focal length parameter.
[0031] According to a fourth aspect, an embodiment provides a video focus model training device, including:
[0032] Memory, used to store programs;
[0033] A processor is used to implement a video focusing model training method by executing a program stored in the memory.
[0034] According to the fifth aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement a video focusing model training method.
[0035] According to the video focusing model training method, video focusing method, video focusing device, video focusing model training equipment and computer-readable storage medium of the above-mentioned embodiments, since the optical flow estimation information obtained by estimating the pre-trained optical flow estimation network is introduced when training the video focusing model, the optical flow estimation information is obtained to help the model understand the motion trajectory of objects in the video, and the video focusing model to be trained is trained based on the optical flow estimation information, so that the trained video focusing model can improve the ability to accurately predict the focus position and clarity, thereby improving the clarity and quality of the video. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a flowchart of the video focusing model training method according to an embodiment of the present application;
[0037] Figure 2 is a flowchart of a training method of a pre-trained optical flow estimation network in one embodiment;
[0038] Figure 3 A flowchart of inputting training sample data into a feature extraction network in one embodiment;
[0039] Figure 4 A flowchart of training a video focusing model to be trained based on a target loss function to obtain a trained video focusing model in an embodiment;
[0040] Figure 5 A flowchart of a video focusing method according to an embodiment;
[0041] Figure 6 The figure is a schematic structural diagram of a video focusing device according to an embodiment. DETAILED DESCRIPTION
[0042] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.
[0043] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.
[0044] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).
[0045] Please refer to Figure 1 In an embodiment of the present invention, a video focusing model training method is provided, including steps S10 to S40, which are described in detail below.
[0046] Step S10: constructing a video focusing model to be trained based on the pre-trained optical flow estimation network and the preset dynamic focusing network.
[0047] Please refer to Figure 2 In some embodiments, the optical flow estimation network pre-trained in step S10 is trained by steps S11 to S12, which are described in detail below.
[0048] Step S11: obtaining at least two adjacent reference image frames, and calculating a pixel displacement value between the at least two adjacent reference image frames.
[0049] Step S12: using at least two adjacent reference image frames as training data sets and pixel displacement values as labels, training a preset optical flow estimation network to obtain a trained optical flow estimation network.
[0050] In some embodiments, the preset optical flow estimation network is a convolutional neural network, and the convolutional neural network is used to pre-train the optical flow estimation network, so that the pre-trained optical flow estimation network can predict the pixel displacement between adjacent frames in a video sequence, thereby inferring the movement of the object.
[0051] In some embodiments, the preset dynamic focusing network includes a feature extraction network and an information prediction network.
[0052] Step S20: input the training sample data into the feature extraction network to obtain a feature map corresponding to the training sample data, perform optical flow estimation processing on the feature map according to the optical flow estimation network to obtain optical flow estimation information.
[0053] In some embodiments, the dynamic focus network in the video focus model to be trained includes a feature extraction network and an information prediction network, and the training sample data includes a video sequence containing adjacent frames.
[0054] Please refer to Figure 3 In some embodiments, step S20 inputs the training sample data into the feature extraction network, including steps S21 to S22, which are described in detail below.
[0055] Step S21: Acquire the first image frame and the second image frame in the training sample data.
[0056] In some embodiments, the first image frame and the second image frame are adjacent frames.
[0057] Step S22: input the first image frame and the second image frame into the feature extraction network respectively.
[0058] In some embodiments, a feature extraction network with shared weights is used to extract multi-scale features of the training sample data, that is, the weight parameters of the feature extraction network remain consistent when the first image frame and the second image frame are input, thereby obtaining a feature map corresponding to the training sample data.
[0059] In some embodiments, the optical flow estimation network performs optical flow estimation processing on the feature map to obtain optical flow estimation information, which can effectively describe the motion of the video frame sequence. Since the optical flow estimation network is a pre-trained network, the weight of the optical flow estimation network can be frozen during the training phase to promote the mutual fusion of the dynamic focusing network and the optical flow information, which helps the video focusing model to be trained to better integrate the information of the moving object, and better understand the dynamic changes of the video frame, thereby improving the clarity and quality of the video.
[0060] Step S30: The feature map and the optical flow estimation information are merged into the final feature information, and the focal length prediction value corresponding to the final feature information is outputted using the information prediction network.
[0061] In some embodiments, the output feature data of each dimension in the feature map is spliced with the optical flow feature data of the same dimension in the optical flow estimation information to obtain the final fused feature information. The final feature information is passed through an information prediction network (MLP, MultilayerPerceptron) to output a predicted focal length prediction value. The method of using the fused final feature information for prediction can help the information of the structure of the optical flow estimation network and the dynamic focusing network to be more coupled.
[0062] Step S40: constructing a corresponding target loss function according to the focal length prediction value, and training the video focusing model to be trained based on the target loss function to obtain a trained video focusing model.
[0063] In some embodiments, the objective loss function is expressed as:
[0064]
[0065] Among them, L1 represents the target loss function, x represents the focal length prediction value, y represents the preset focal length true value, and n represents the total number of data corresponding to the final feature information.
[0066] Please refer to Figure 4 In some embodiments, step S40 trains the video focusing model to be trained based on the target loss function to obtain a trained video focusing model, including steps S41 to S43, which are described in detail below.
[0067] Step S41: Calculate the loss value according to the target loss function, and determine the size between the loss value and a preset loss threshold.
[0068] Step S42: When the loss value is greater than the loss threshold, the weight parameters in the dynamic focusing network in the video focusing model to be trained are adjusted, and the video focusing model to be trained and the training sample data after adjusting the weight parameters in the dynamic focusing network are used for re-training until the loss value calculated after the re-training is less than or equal to the loss threshold. The video focusing model to be trained after adjusting the weight parameters in the dynamic focusing network is used as the trained video focusing model.
[0069] In some embodiments, in order to maintain the stability of the optical flow estimation network, the weights of the optical flow estimation network need to be frozen during training to train the dynamic focusing network. That is, when the loss value is greater than the loss threshold, the weight parameters of the dynamic focusing network in the video focusing model to be trained are adjusted, and the dynamic focusing network with the adjusted weight parameters is re-trained until the loss value calculated after the re-training is less than or equal to the loss threshold, thereby obtaining a trained video focusing model.
[0070] Step S43: when the loss value is less than or equal to the loss threshold, the video focusing model to be trained is used as the trained video focusing model.
[0071] In some embodiments, the focal length prediction value obtained by aggregating and predicting the optical flow estimation results and the feature map is used to train the video focusing model to be trained, thereby achieving precise control of the focal length adjustment.
[0072] Please refer to Figure 5 An embodiment of the present invention provides a video focusing method, including steps S50 to S70, which are described in detail below.
[0073] Step S50: Acquire a video sequence of the target object.
[0074] In some embodiments, the video sequence includes a moving target object captured by a video capture device.
[0075] Step S60: input the video sequence into the trained video focusing model to obtain the target focal length parameters.
[0076] In some embodiments, a trained video focusing model is obtained through the video focusing model training method of steps S10 to S40, and the video sequence is input into the trained video focusing model to obtain the target focal length parameter.
[0077] Step S70: adjusting the device focal length of the video shooting device using the target focal length parameter.
[0078] In some embodiments, the focus of the video capture device is adjusted in real time using the target focus parameter during the video shooting process to adapt to the movement of the object in the video, thereby effectively solving the blur or out-of-focus problem that cannot be solved by the traditional video focusing method, and realizing adaptive clear focus adjustment of the moving object in the video. At the same time, because the trained video focusing model incorporates the optical flow estimation information, the tracking accuracy of the moving object in the video can be improved, making the video content clearer and more coherent.
[0079] Please refer to Figure 6 An embodiment of the present invention provides a video focusing device, including a parameter determination module 10 and a focal length adjustment module 20, which are described in detail below.
[0080] The parameter determination module 10 is used to obtain a video sequence of a target object, input the video sequence into a trained video focusing model, and obtain a target focal length parameter.
[0081] In some embodiments, the video sequence includes a moving target object captured by a video capture device, a trained video focusing model is obtained by a video focusing model training method, and the video sequence is input into the trained video focusing model to obtain a target focal length parameter.
[0082] The focal length adjustment module 20 is used to adjust the device focal length of the video shooting device using the target focal length parameter.
[0083] In some embodiments, the focus of the video capture device is adjusted in real time using the target focus parameter during the video shooting process to adapt to the movement of the object in the video, thereby effectively solving the blur or out-of-focus problem that cannot be solved by the traditional video focusing method, and realizing adaptive clear focus adjustment of the moving object in the video. At the same time, because the trained video focusing model incorporates the optical flow estimation information, the tracking accuracy of the moving object in the video can be improved, making the video content clearer and more coherent.
[0084] Some embodiments provide a video focusing model training device, including: a memory for storing programs; and a processor for implementing a video focusing model training method by executing the programs stored in the memory.
[0085] In some embodiments, the video focusing model training method is described in detail below:
[0086] A video focusing model to be trained is constructed based on a pre-trained optical flow estimation network and a preset dynamic focusing network; wherein the dynamic focusing network includes a feature extraction network and an information prediction network;
[0087] Input the training sample data into the feature extraction network to obtain a feature map corresponding to the training sample data, and perform optical flow estimation processing on the feature map according to the optical flow estimation network to obtain optical flow estimation information; wherein the training sample data includes a video sequence containing adjacent frames;
[0088] The feature map and optical flow estimation information are fused into the final feature information, and the focal length prediction value corresponding to the final feature information is output using the information prediction network;
[0089] A corresponding target loss function is constructed according to the focal length prediction value, and the video focusing model to be trained is trained based on the target loss function to obtain a trained video focusing model.
[0090] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.
[0091] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.
Claims
1. A video focusing model training method, characterized in that: include: A video focusing model to be trained is constructed based on a pre-trained optical flow estimation network and a preset dynamic focusing network; The dynamic focusing network includes a feature extraction network and an information prediction network; Inputting the training sample data into the feature extraction network to obtain a feature map corresponding to the training sample data, and performing optical flow estimation processing on the feature map according to the optical flow estimation network to obtain optical flow estimation information; The training sample data includes a video sequence containing adjacent frames; The feature map and the optical flow estimation information are merged into final feature information, and the information prediction network is used to output a focal length prediction value corresponding to the final feature information; A corresponding target loss function is constructed according to the focal length prediction value, and the video focusing model to be trained is trained based on the target loss function to obtain a trained video focusing model.
2. The video focusing model training method according to claim 1, characterized in that: The step of training the to-be-trained video focusing model based on the target loss function to obtain a trained video focusing model includes: Calculating a loss value according to the target loss function, and determining a difference between the loss value and a preset loss threshold; When the loss value is greater than the loss threshold, the weight parameters in the dynamic focus network in the video focus model to be trained are adjusted, and the video focus model to be trained and the training sample data are used for retraining after the weight parameters in the dynamic focus network are adjusted, until the loss value calculated after the retraining is less than or equal to the loss threshold, and the video focus model to be trained after the weight parameters in the dynamic focus network are adjusted is used as the trained video focus model; When the loss value is less than or equal to the loss threshold, the video focusing model to be trained is used as a trained video focusing model.
3. The video focusing model training method according to claim 1, characterized in that: The step of fusing the feature map and the optical flow estimation information into final feature information includes: The output feature data in each dimension of the feature map is concatenated with the optical flow feature data in the same dimension in the optical flow estimation information to obtain the fused final feature information.
4. The video focusing model training method according to claim 1, characterized in that: The expression of the objective loss function is: Among them, L1 represents the target loss function, x represents the focal length prediction value, y represents the preset focal length true value, and n represents the total number of data corresponding to the final feature information.
5. The video focusing model training method according to claim 1, characterized in that: The step of inputting the training sample data into the feature extraction network comprises: Acquire a first image frame and a second image frame in the training sample data; the first image frame and the second image frame are adjacent frames; The first image frame and the second image frame are respectively input into the feature extraction network; wherein the weight parameters of the feature extraction network remain consistent when the first image frame and the second image frame are input.
6. The video focusing model training method according to claim 1, characterized in that: The pre-trained optical flow estimation network is trained in the following way: Acquire at least two adjacent reference image frames, and calculate a pixel displacement value between the at least two adjacent reference image frames; At least two adjacent reference image frames are used as training data sets, and the pixel displacement values are used as labels to train a preset optical flow estimation network to obtain a trained optical flow estimation network.
7. A video focusing method, characterized in that: include: Obtain a video sequence of the target object; The video sequence includes the moving target object captured by a video capture device; Inputting the video sequence into a trained video focusing model obtained by the video focusing model training method according to any one of claims 1 to 6 to obtain a target focal length parameter; The target focal length parameter is used to adjust the device focal length of the video shooting device.
8. A video focusing device, characterized in that: include: A parameter determination module, used to obtain a video sequence of a target object; The video sequence includes the moving target object captured by a video capture device; Inputting the video sequence into a trained video focusing model obtained by the video focusing model training method according to any one of claims 1 to 6 to obtain a target focal length parameter; The focal length adjustment module is used to adjust the device focal length of the video shooting device using the target focal length parameter.
9. A video focusing model training device, characterized in that: include: Memory, used to store programs; A processor, configured to implement the video focusing model training method as described in any one of claims 1 to 6 by executing the program stored in the memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which can be executed by a processor to implement the video focusing model training method as described in any one of claims 1-6.