Video processing method and device, equipment and storage medium

Abstract

The invention provides a video processing method, apparatus and device, and a storage medium. The method comprises the steps of obtaining a to-be-processed collection image and pose information corresponding to the collection image; inputting the pose information corresponding to the collected image into a pose prediction model, and predicting a virtual pose corresponding to the collected image; and performing optimization processing on the acquired image according to the virtual pose and the pose information to obtain target image information. According to the scheme, the preset pose prediction model is obtained through training according to the sample actual pose, the sample historical actual pose, the sample cache actual pose and the corresponding sample virtual pose of the corresponding collection time of the sample collection image, and the virtual pose corresponding to the actual pose is obtained through prediction according to the pose information of the collection image. And the acquired image is optimized according to the virtual pose and the actual pose, that is, the acquired image is optimized in combination with the pose information corresponding to the acquired image, so that the accuracy of video optimization is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video processing method, device, equipment and storage medium. Background Art

[0002] With the development of camera technology, the application scenarios of cameras are becoming increasingly extensive and are not limited to shooting and processing in stable positions. For example, when shooting images, devices such as mobile phones and sports cameras will cause video jitter due to factors such as movement, which affects the video viewing experience. Therefore, the requirements for video optimization processing are also strict; among them, video optimization processing is mainly achieved by transforming the jittery video into a stable video. In related technologies, the jitter state of each image frame in the video is generally monitored based on a large number of preset prior thresholds, and video optimization processing is performed, resulting in low accuracy in video optimization processing. Summary of the invention

[0003] The present application provides a video processing method, apparatus, device and storage medium, aiming to solve the problem of low accuracy of video optimization processing in the background technology and improve the accuracy of video optimization processing.

[0004] In a first aspect, the present application provides a video processing method, comprising:

[0005] Acquire a collected image to be processed and position information corresponding to the collected image;

[0006] Inputting the posture information corresponding to the collected image into a posture prediction model to predict the virtual posture corresponding to the collected image;

[0007] The collected image is optimized according to the virtual posture and the posture information to obtain target image information.

[0008] In a possible implementation of the present application, the posture information includes the actual posture when the captured image is collected, the historical actual posture collected before the actual posture, the cached actual posture collected after the actual posture, and the historical virtual posture corresponding to the historical actual posture;

[0009] The obtaining of the collected image to be processed and the posture information corresponding to the collected image includes:

[0010] Determine the actual posture and the historical actual posture according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and each gyroscope data frame acquired by the preset gyroscope;

[0011] Determining a historical virtual pose according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and the historical virtual quaternion set;

[0012] The actual pose, the historical actual pose, the cached actual pose and the historical virtual pose are set as pose information.

[0013] In a possible implementation of the present application, determining the actual posture and the historical actual posture according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and each gyroscope data frame acquired by the preset gyroscope includes:

[0014] Acquire a collected image to be processed, a collection time and a collection period of the collected image;

[0015] According to the acquisition time and the acquisition cycle, determine the historical acquisition time for acquiring the historical acquisition image before the acquisition time, and the cache acquisition time for acquiring the cache acquisition image after the acquisition time;

[0016] According to the acquisition time of the acquired image, a target data frame is extracted from each gyroscope data frame acquired by a preset gyroscope, and the target data frame is converted into a target quaternion frame, and the target quaternion frame is set as an actual posture;

[0017] Extracting a historical data frame from each of the gyroscope data frames according to the historical acquisition time, converting the historical data frame into a historical quaternion frame, and setting the historical quaternion frame as a historical actual posture;

[0018] A cache data frame is extracted from each of the gyroscope data frames according to the cache acquisition time, and the cache data frame is converted into a cache quaternion frame, and the cache quaternion frame is set as a cache actual posture.

[0019] In a possible implementation of the present application, before setting the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture as posture information, the method includes:

[0020] If the target data frame corresponding to any target time point among the acquisition time, the historical acquisition time and the cached acquisition time is not extracted from each of the gyroscope data frames, interpolation processing is performed on the four data values ​​one acquisition cycle away from the target time point, and a new quaternion frame obtained by the interpolation processing is set as the quaternion frame corresponding to the target time point, wherein the target data frame includes at least one of the actual posture, the historical actual posture or the cached actual posture;

[0021] For each of the historical acquisition times, a relative quaternion frame between the historical quaternion frame corresponding to the historical acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the historical quaternion frame corresponding to the historical acquisition time;

[0022] For each of the cache acquisition time, a relative quaternion frame between the cache quaternion frame corresponding to the cache acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the cache quaternion frame corresponding to the cache acquisition time.

[0023] In a possible implementation of the present application, determining the historical virtual pose according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and the historical virtual quaternion set includes:

[0024] Extracting a historical virtual quaternion frame with the same timestamp as the historical acquisition time from the historical virtual quaternion set;

[0025] For each of the historical collection time, if a historical virtual quaternion frame with the same timestamp as the historical collection time is not extracted, determining the historical virtual quaternion frame corresponding to the historical collection time according to the historical virtual quaternion frame corresponding to the timestamp adjacent to the historical collection time;

[0026] A relative quaternion frame between the historical virtual quaternion frame corresponding to each of the historical acquisition times and the target quaternion frame is calculated, and the relative quaternion frame is set as the historical virtual posture corresponding to each of the historical acquisition times.

[0027] In a possible implementation of the present application, the step of optimizing the captured image according to the virtual posture and the actual posture to obtain target image information includes:

[0028] Performing grid division processing on the collected image to obtain multiple image regions;

[0029] For each of the image regions, determining an acquisition time of the image region according to a rolling time corresponding to each pixel point in the image region, and determining an actual position of the image region according to the acquisition time of the image region;

[0030] Calculating region mapping information according to the actual position and posture of the region and the virtual position and posture;

[0031] According to the region mapping information of each image region, mapping processing is performed on the image region to obtain target image information.

[0032] In a possible implementation of the present application, inputting the posture information corresponding to the acquired image into a preset posture prediction model to predict a virtual posture corresponding to the acquired image includes:

[0033] Inputting the posture information into the attention mechanism module of the posture prediction model for weighted processing, and outputting weighted frame feature information;

[0034] Inputting the frame feature information into the recursive neural network module in the posture prediction model for feature extraction, and outputting feature information;

[0035] The feature information is input into a fully connected module in the pose prediction model to predict a virtual pose corresponding to the captured image.

[0036] In a possible implementation of the present application, the posture prediction model is obtained by training the sample actual posture when collecting the sample collection image, the sample historical actual posture collected before the sample actual posture, the sample cache actual posture collected after the sample actual posture, and the sample historical virtual posture corresponding to the sample historical actual posture;

[0037] The steps of constructing the preset posture prediction model training include:

[0038] Acquire sample pose information, wherein the sample pose information includes an actual pose sample when the sample capture image is collected, a sample historical actual pose collected before the sample actual pose, a sample future actual pose collected after the sample actual pose, and a sample historical virtual pose corresponding to the sample historical actual pose;

[0039] Training the initial prediction model according to the sample pose information to obtain a training prediction model;

[0040] Until the sample virtual pose predicted by the training prediction model meets the convergence condition, a preset pose prediction model is obtained.

[0041] In a possible implementation of the present application, the initial prediction model is trained according to the sample pose information to obtain a training prediction model, including:

[0042] Calculate a smoothing loss parameter according to the sample virtual pose predicted by the training prediction model, the acquisition period of the sample acquisition image, and a preset first loss function;

[0043] Calculating an anti-distortion loss parameter according to the actual pose of the sample, the virtual pose of the sample, and a preset second loss function;

[0044] Calculating an anti-cross-border loss parameter and an over-cropping loss parameter according to a preset cropping rate, the sample virtual pose, and the sample acquisition image;

[0045] A total loss parameter is calculated according to the smoothing loss parameter, the anti-distortion loss parameter, the anti-cross-boundary loss parameter and the over-clipping loss parameter;

[0046] The initial prediction model is modified according to the total loss parameter to obtain a training prediction model.

[0047] In a second aspect, the present application provides a video processing device, including:

[0048] An acquisition module, used to acquire a collected image to be processed and position information corresponding to the collected image;

[0049] A prediction module, used to input the posture information corresponding to the collected image into a preset posture prediction model to predict the virtual posture corresponding to the collected image;

[0050] A processing module is used to optimize the collected image according to the virtual posture and the posture information to obtain target image information.

[0051] In a third aspect, the present application provides a video processing device, the video processing device comprising:

[0052] one or more processors;

[0053] Memory; and

[0054] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps in any one of the video processing methods.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in any one of the video processing methods described above.

[0056] The present application provides a video processing method, device, equipment and storage medium, which obtains a captured image to be processed and the posture information corresponding to the captured image; inputs the posture information corresponding to the captured image into a preset posture prediction model to predict the virtual posture corresponding to the captured image; and optimizes the captured image according to the virtual posture and the posture information to obtain target image information. This solution analyzes the posture information corresponding to the captured image through a prediction model, predicts the virtual posture corresponding to the actual posture, and further optimizes the captured image according to the virtual posture and the actual posture to obtain the target image to generate a video, and optimizes the captured image in combination with the posture information corresponding to the captured image to improve the accuracy of the video optimization processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 is a scene schematic diagram of the video processing method provided in an embodiment of the present application;

[0059] Figure 2 is a schematic diagram of an embodiment of a video processing method provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of one implementation flow of the posture prediction model training in the video processing method provided in the implementation scheme of the present application;

[0061] Figure 4 A schematic diagram of a flow chart of one implementation scheme for obtaining posture information in a video processing method provided in the implementation scheme of the present application;

[0062] Figure 5 A schematic diagram of a flow chart of one implementation scheme for obtaining a historical virtual posture in a video processing method provided in an implementation scheme of the present application;

[0063] Figure 6 A schematic diagram of a flow chart of one implementation scheme for obtaining a target image in a video processing method provided in an implementation scheme of the present application;

[0064] Figure 7 A schematic diagram of the structure of one implementation scheme of the preset prediction model in the video processing method provided in the implementation scheme of the present application;

[0065] Figure 8 A schematic diagram of another implementation structure of the preset prediction model in the video processing method provided in the implementation scheme of the present application;

[0066] Fig. 9 A schematic diagram of one of the implementation schemes of the structure of the Attention network layer in the video processing method provided in the implementation scheme of the present application;

[0067] Fig.10 is a schematic diagram of the structure of an embodiment of a video processing device provided in an embodiment of the present application;

[0068] Fig.11 It is a schematic diagram of the structure of an embodiment of a video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0070] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0071] In the embodiment of the present application, "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0072] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0073] Among them, video stabilization, that is, video optimization processing, is to transform the shaking video into a stable video. The main video stabilization methods include motion estimation, motion smoothing and rendering modules. By smoothing the motion trajectory of the camera, the original image is transformed into the corresponding stabilized image, and then cropped to obtain the stabilized video. Traditional methods have many prior thresholds, and a large number of post-processing operations are performed based on these thresholds. In some scenarios, they will fail or not be smooth enough.

[0074] The embodiment of the present application provides a video processing method, device, equipment and computer-readable storage medium, which uses deep learning to smooth the camera motion information measured by the gyroscope corresponding to the captured image to obtain a smoother motion trajectory, and estimates the virtual pose of the current frame through multiple frames of real camera poses and multiple frames of virtual camera poses. Then, the captured image is rendered to obtain a stabilized video through the conversion relationship between the virtual pose and the actual pose, which is described in detail below.

[0075] The video processing method in the embodiment of the present invention is applied to a video processing device, which is arranged in a video processing device. The video processing device is provided with one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the video processing method; the video processing device can be a terminal, such as a mobile phone or a tablet computer, and the video processing device can also be a server, or a service cluster composed of multiple servers.

[0076] like Figure 1 As shown, Figure 1 This is a scene diagram of the video processing method of an embodiment of the present application. The video processing scene in the embodiment of the present invention includes a video processing device 100 (a video processing device is integrated in the video processing device 100), and a computer-readable storage medium corresponding to video processing runs in the video processing device 100 to execute the steps of video processing.

[0077] Understandably, Figure 1 The video processing device in the scenario of the video processing method shown, or the devices included in the video processing device, does not constitute a limitation on the embodiments of the present invention, that is, the number of devices and types of devices included in the scenario of the video processing method, or the number of devices and types of devices included in each device, do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be regarded as equivalent replacements or derivatives of the technical solution claimed to be protected in the embodiments of the present invention.

[0078] The video processing device 100 in the embodiment of the present invention is mainly used for: obtaining a captured image to be processed, and obtaining the posture information corresponding to the captured image, the posture information including the actual posture when the captured image is captured, the historical actual posture captured before the actual posture, the cached actual posture captured after the actual posture, and the historical virtual posture corresponding to the historical actual posture; inputting the posture information corresponding to the captured image into a preset posture prediction model to predict the virtual posture corresponding to the captured image; the posture prediction model is obtained by training the sample actual posture when the sample is captured, the sample historical actual posture captured before the sample actual posture, the sample cached actual posture captured after the sample actual posture, and the sample historical virtual posture corresponding to the sample historical actual posture; optimizing the captured image according to the virtual posture and the actual posture to obtain a target image, and generating a video according to the target image.

[0079] In the embodiment of the present invention, the video processing device 100 may be an independent video processing device, or a video processing device network or a video processing device cluster composed of video processing devices. For example, the video processing device 100 described in the embodiment of the present invention includes but is not limited to a computer, a network host, a single network video processing device, a set of multiple network video processing devices, or a cloud video processing device composed of multiple video processing devices. The cloud video processing device is composed of a large number of computers or network video processing devices based on cloud computing.

[0080] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer video processing devices as shown in , or a network connection relationship of the video processing devices, such as Figure 1 Only one video processing device is shown. It can be understood that the scenario of the video processing method can also include one or more other video processing devices, which are not limited here. The video processing device 100 can also include a memory for storing data, for example, storing image information obtained by shooting.

[0081] In addition, in the scenario of the video processing method of the present application, the video processing device 100 may be provided with a display device, or the video processing device 100 may not be provided with a display device and may be communicated with an external display device 200, and the display device 200 is used to output the result of the execution of the video processing method in the video processing device. The video processing device 100 may access a background database 300 (the background database may be in the local memory of the video processing device, or the background database may also be set in the cloud), and the background database 300 may store information related to video processing, for example, images collected in the background database 300, or pre-set historical virtual postures.

[0082] It should be noted that Figure 1 The scene diagram of the video processing method shown is only an example. The scene of the video processing method described in the embodiment of the present invention is to more clearly illustrate the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention.

[0083] Based on the scenario of the above video processing method, an embodiment of the video processing method is proposed.

[0084] like Figure 2 FIG. 2 is a flow chart of an embodiment of a video processing method in an embodiment of the present application. The video processing method includes steps S201-S203:

[0085] S201. Obtain a collected image to be processed and position information corresponding to the collected image.

[0086] Among them, the captured image to be processed, that is, the imaging image frame captured by the sensor that needs to be optimized; it can be understood that when cameras, mobile phones and other imaging devices capture images or videos, they first capture optical image information through image sensors, and further convert the optical image information into electronic signals to generate the captured image, and then perform anti-shake processing (that is, optimization processing) on ​​the captured images (captured images to be processed) in turn as a video frame of a video, and then obtain a video with all the video frames.

[0087] The posture information includes the actual posture when the captured image is collected, the historical actual posture collected before the actual posture, the cached actual posture collected after the actual posture, and the historical virtual posture corresponding to the historical actual posture.

[0088] It can be understood that the acquisition time, that is, the imaging time of the acquired image, exemplarily, the camera CMOS sensor images in sequence according to the pixel array, and finally obtains a complete acquired image, that is, the acquisition time of the acquired image has a certain process, and the acquisition time, that is, the complete image generation time of the acquired image.

[0089] Among them, the actual posture, that is, the device posture of a camera, mobile phone or other imaging device when capturing images. It can be understood that the actual posture can be acquired by a gyroscope or acceleration sensor installed in the imaging device.

[0090] Among them, the virtual posture, that is, the posture of the target image after the optimized processing of the acquired image, can be understood that the target image after posture correction (to avoid image jitter) is equivalent to trajectory smoothing processing, that is, a corrected stable posture corresponding to the target image, and the virtual posture is the stable posture, that is, the virtual posture is equivalent to the posture after the actual posture is optimized.

[0091] Furthermore, the actual posture corresponding to the acquisition time, that is, the posture of the device at the time point corresponding to the acquisition time.

[0092] Among them, the historical actual posture, that is, the device posture collected before the time point corresponding to the collection time; it can be understood that the historical virtual posture may include multiple, for example, the historical virtual posture includes the actual posture collected historically at an interval of one collection period-N collection periods from the collection time of the actual posture, that is, the historical actual posture may include N.

[0093] It can be understood that during the video stabilization process, the camera CMOS sensor images to obtain a captured image, which is further processed. However, due to the different imaging frequencies of the processor and the camera CMOS sensor, there is generally a delay, that is, a cached captured image is generated (the cached captured image is the captured image queued for processing). Therefore, when a certain captured image to be processed is processed, there are multiple captured images queued for processing corresponding to the captured image, that is, as cached captured images, an actual posture cache will also be collected for the capture time corresponding to each cached captured image, that is, the cached actual posture is obtained.

[0094] Further, it can be understood that the cached actual posture, that is, the device posture acquired after the time point corresponding to the acquisition time (corresponding to the cached acquisition time of the cached acquisition image); it can be understood that the cached virtual posture can include multiple, exemplarily, the cached virtual posture includes the actual postures acquired in the cache at an interval of one acquisition cycle-N acquisition cycles from the acquisition time of the actual posture, that is, the cached actual postures can include N.

[0095] Among them, the historical virtual posture, that is, the virtual posture obtained after mapping the historical actual posture, and the virtual posture represents the corresponding posture after the historical collected image is optimized, and is not the actual posture. That is, it can be understood that the mapping relationship between the historical collected image and the historical target image after the historical collected image is optimized can be represented by the historical virtual posture and the historical actual posture.

[0096] Furthermore, in some other implementation schemes of the present application, the image acquisition frequency of the acquired image can be set to N times the acquisition frequency of the actual posture, that is, to ensure that the acquisition time of each acquired image can capture the corresponding actual posture; the real stable posture and the acquired image can also be further time-aligned according to the acquisition time of the actual posture and the acquisition time of the acquired image, which is not specifically limited in the present application.

[0097] Furthermore, in some other implementation schemes of the present application: the acquiring of the acquired image to be processed and the posture information corresponding to the acquired image includes: determining the actual posture and the historical actual posture according to the acquisition time of the acquired image, the acquisition period of the acquired image, and each gyroscope data frame acquired by the preset gyroscope; determining the historical virtual posture according to the acquisition time of the acquired image, the acquisition period of the acquired image, and the historical virtual quaternion set; setting the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture as posture information.

[0098] S202: Inputting the posture information corresponding to the collected image into a posture prediction model to predict a virtual posture corresponding to the collected image.

[0099] The posture prediction model is obtained by training the sample actual posture when collecting the sample image, the sample historical actual posture collected before the sample actual posture, the sample cache actual posture collected after the sample actual posture, and the sample historical virtual posture corresponding to the sample historical actual posture.

[0100] The sample acquisition images, that is, acquisition images acquired historically, can exemplarily be several acquisition images acquired historically set as sample acquisition images.

[0101] It is understandable that the device posture collected at the acquisition time corresponding to each sample acquisition image (i.e., each historically acquired acquisition image) is used as the sample actual posture, that is, each sample acquisition image corresponds to the sample actual posture. Before the acquisition time of each sample acquisition image, the device posture collected at the corresponding time point at intervals of N acquisition cycles is used as the sample historical actual posture. The sample actual posture may include multiple, generally, the acquisition time interval between each sample historical actual posture and the sample actual posture is one acquisition cycle, two acquisition cycles, ... N acquisition cycles.

[0102] Among them, the sample historical virtual posture, that is, the virtual posture obtained after mapping processing of the sample historical actual posture, it can be understood that the mapping relationship between the sample historical actual posture and the virtual posture can be determined by the mapping relationship between the historical acquisition image corresponding to the time of the sample historical actual posture and the historical target image corresponding to the time of the sample historical actual posture. Exemplarily, for a certain sample historical actual posture, the mapping relationship between the historical acquisition image corresponding to the time of the sample historical actual posture and the historical target image (the image after the historical acquisition image is optimized) corresponding to the time of the sample historical actual posture is used to perform time mapping processing on the sample historical actual posture to obtain the corresponding sample historical virtual posture as the sample posture information corresponding to the sample acquisition image. Further, the sample posture information is input into the initial prediction model for model writing training to obtain the prediction model.

[0103] In one of the implementation schemes of the present application, after obtaining the posture information of the acquired image to be processed, the posture information is input into the prediction model to obtain the virtual posture of the posture acquisition device corresponding to the acquisition time of the acquired image.

[0104] S203: Optimize the collected image according to the virtual posture and the posture information to obtain target image information.

[0105] Among them, the target image information, that is, the image after optimization processing of the captured image, that is, the image after posture correction (to avoid image jitter), that is, the virtual posture is equivalent to the posture after the actual posture is optimized.

[0106] The actual posture is the actual device posture at the acquisition time corresponding to the acquired image, and the target image is the image after the acquired image is optimized.

[0107] Specifically, after predicting the virtual posture corresponding to the acquired image, determine the correspondence between the virtual posture and the actual posture acquired at the acquisition time, and optimize the acquired image according to the correspondence. Exemplarily, map the acquired image to the corresponding stabilized image position according to the correspondence to achieve optimization processing; then obtain the target image corresponding to the acquired image, that is, obtain a video frame, and further generate a video based on all the obtained target images.

[0108] It can be understood that video generation is to generate a complete video by collecting multiple image frames (image frames are equivalent to each collected image). In the implementation scheme of the present application, after obtaining a collected image, that is, obtaining the posture information corresponding to the collected image, further inputting the posture information into a preset prediction model to obtain a virtual posture corresponding to the collected image, and then optimizing the collected image according to the virtual posture and the actual posture to obtain a target image, and generating a video based on the target image.

[0109] Further, based on the above implementation scheme, see Figure 3 , Figure 3 A schematic flow chart of one implementation scheme of posture prediction model training in the video processing method provided in the implementation scheme of the present application, specifically including steps S301-S303:

[0110] S301. Obtain sample pose information, wherein the sample pose information includes an actual pose sample when collecting a sample acquisition image, a sample historical actual pose collected before the sample actual pose, a sample future actual pose collected after the sample actual pose, and a sample historical virtual pose corresponding to the sample historical actual pose.

[0111] The sample acquisition images may include multiple ones, and corresponding sample pose information is generated for each sample acquisition image.

[0112] Exemplarily, in one embodiment of the present application, for each sample acquisition image, by obtaining the sample actual pose corresponding to the sample acquisition time of the sample acquisition image, obtaining the sample historical actual pose corresponding to the historical acquisition time of the N historical acquisition images before the sample acquisition time, obtaining the cached actual pose corresponding to the cached acquisition time of the P cached acquisition images after the acquisition time, and obtaining the historical virtual pose corresponding to the M historical acquisition times before the acquisition time. The actual pose corresponding to the acquisition time, the sample historical actual pose corresponding to the historical acquisition time of the N historical acquisition images before the sample acquisition time, the cached actual pose corresponding to the cached acquisition time of the P cached acquisition images after the acquisition time, and the historical virtual pose corresponding to the M historical acquisition times are set as the sample pose information corresponding to the sample acquisition image, that is, a sample pose information is generated for each sample acquisition image, and the sample pose information for the sample acquisition image may include multiple.

[0113] Furthermore, in some other implementation schemes of the present application, for each sample acquisition image, by obtaining the actual posture of the sample collected corresponding to the sample acquisition time of the sample acquisition image, obtaining the historical actual posture of the sample collected corresponding to the historical acquisition time of the N historical acquisition images before the sample acquisition time, obtaining the cache actual posture collected corresponding to the cache acquisition time of the P cache acquisition images after the acquisition time, and obtaining the historical virtual posture corresponding to the M historical acquisition times before the acquisition time, the algorithm is further used to perform time alignment correction on the actual posture of each sample, or the historical actual posture of the sample, or the historical virtual posture of the sample, or posture relative calculation processing, and the posture information after the correction processing is set as the sample posture information corresponding to the sample acquisition image, wherein, for the implementation scheme of performing time alignment correction on the actual posture of each sample, or the historical actual posture of the sample, or the historical virtual posture of the sample, and posture relative calculation processing by the algorithm, refer to the actual implementation scheme of obtaining the posture information of the acquired image described below, and this application does not make specific limitations.

[0114] S302: Train an initial prediction model according to the sample pose information to obtain a training prediction model.

[0115] Furthermore, for the sample pose information of each sample acquisition image, the pose information is respectively input into the initial prediction model for pose prediction, and the corresponding sample virtual pose of the sample acquisition time (the acquisition time corresponding to the sample acquisition image) is output, and then the mapping matrix is ​​calculated according to the actual pose at the acquisition time and the predicted virtual pose, the corner point positions corresponding to the four vertices of the acquisition image in the stabilized image are calculated according to the mapping matrix, and the total loss is calculated according to the corner point positions, the virtual pose and the actual pose, and the training prediction model is adjusted according to the total loss to obtain the training prediction model.

[0116] S303, until the sample virtual pose predicted by the training prediction model meets the convergence condition, a preset pose prediction model is obtained.

[0117] The above steps are further repeated to output the sample pose information corresponding to the sample acquisition image to the training prediction model for prediction, and the training prediction model is trained until the virtual pose obtained by the training prediction meets the convergence condition, thereby obtaining the pose prediction model.

[0118] Specifically, the present application also provides a specific implementation scheme for training an initial prediction model according to the sample pose information to obtain a training prediction model, comprising the steps of:

[0119] (1) Calculating a smoothing loss parameter according to the sample virtual pose predicted by the training prediction model, the acquisition period of the sample acquisition image, and a preset first loss function.

[0120] Wherein, the first loss function includes:

[0121]

[0122] Specifically, the R v (t) is to input the pose information into the initial prediction model for pose prediction, and output the sample acquisition time (the acquisition time corresponding to the sample acquisition image), the △T is the sample acquisition period corresponding to the sample acquisition image, and the adjacent virtual frame four elements are close to

[0123] Wherein, the first loss function also includes:

[0124]

[0125] Specifically, the R v (t) is the virtual pose of the sample, △T is the acquisition period corresponding to the acquired image, and the differences between the four elements of adjacent virtual frames are relatively close.

[0126] Among them, the and is the smoothing loss parameter.

[0127] (2) Calculating an anti-distortion loss parameter according to the actual pose of the sample, the virtual pose of the sample, and a preset second loss function.

[0128] The anti-distortion loss parameter is calculated according to the second loss function, and the second loss function is:

[0129]

[0130] Specifically, R r Refers to the actual pose corresponding to the sample acquisition image, R v Refers to the virtual pose corresponding to the sample acquisition image, Ω(R v ,R r ) refers to the spherical angle between the actual pose and the virtual pose.

[0131] (3) Calculating an anti-cross-boundary loss parameter and an over-cropping loss parameter according to a preset cropping rate, the sample virtual pose, and the sample acquisition image.

[0132] Specifically, after predicting the sample virtual pose, the mapping relationship between the sample actual pose of the sample acquisition image and the sample virtual pose is calculated, and the four corner points corresponding to the acquisition image are mapped to the coordinates corresponding to the stabilized image according to the mapping relationship to obtain the corner point coordinates after stabilization. The anti-cross-boundary loss parameters and over-cropping loss parameters are further calculated based on the corner point coordinates and the preset clipping rate.

[0133] Specifically, in one embodiment of the present application, the preset clipping rate includes a first preset clipping rate and a second preset clipping rate, wherein the first preset clipping rate is used to constrain out-of-bounds clipping, and the second preset clipping rate is used to preset over-clipped out-of-bounds clipping, wherein the anti-out-of-bounds loss parameter and the over-clipped loss parameter specifically include the steps of:

[0134] a. According to the mapping relationship between the sample virtual posture and the sample actual posture, the corner point information of the sample acquisition image is mapped to obtain the mapped corner point information.

[0135] Among them, the mapping relationship between the sample virtual pose and the sample actual pose, that is, the projection matrix between the sample virtual pose and the sample actual pose, can be calculated according to the rotation matrices corresponding to the virtual pose and the sample actual pose respectively, combined with the camera intrinsic parameters.

[0136] b. calculating an anti-crossing loss parameter according to the mapped corner point information and theoretical corner point information of the sample acquisition image corresponding to a first preset cropping rate;

[0137] For example, constraints are imposed on the four corner points corresponding to the first preset cropping rate of 0.08:

[0138] The four corner points after transformation cannot cross the four corner points corresponding to the cropping rate of 0.04. Construct loss and get

[0139] c. Calculate an over-cropping loss parameter according to the mapped corner point information and theoretical corner point information of the sample acquisition image corresponding to a second preset cropping rate.

[0140] For example, constraints are imposed on the four corner points corresponding to the second preset cropping rate of 0.15:

[0141] The four corner points after transformation cannot cross the four corner points corresponding to the cropping rate of 0.15. Construct loss and get

[0142] It can be understood that the above-mentioned values ​​of the first preset cropping rate and the second preset cropping rate are only for example purposes, and do not limit the specific values ​​of the first preset cropping rate and the second preset cropping rate in the technical solution of the present application.

[0143] Furthermore, the constructed loss can adjust the FOV (Field of View) size of the stabilized video. For example, in order to obtain a larger FOV, the above parameters can be adjusted from 0.08, 0.04, 0.15 to 0.03, 0.01, 0.08.

[0144] (4) calculating a total loss parameter according to the smoothing loss parameter, the anti-distortion loss parameter, the anti-cross-boundary loss parameter, and the over-clipping loss parameter;

[0145] Furthermore, the total loss parameter is calculated according to the total loss calculation formula:

[0146]

[0147] (5) The initial prediction model is modified according to the total loss parameter to obtain a training prediction model.

[0148] Exemplarily, after obtaining the total loss parameter, the training prediction model is modified according to the size relationship between the total loss parameter and a preset loss parameter threshold until the sample virtual pose predicted by the training prediction model meets the convergence condition, thereby obtaining a preset pose prediction model.

[0149] Further, based on the above implementation scheme, see Figure 4 , Figure 4 A schematic flow chart of one implementation scheme for obtaining posture information in a video processing method provided in the implementation scheme of the present application, specifically including steps S401-S406:

[0150] S401, obtaining a captured image to be processed, a capture time and a capture cycle of the captured image, and determining a historical capture time for capturing historical captured images before the capture time and a cache capture time for capturing cached captured images after the capture time according to the capture time and the capture cycle.

[0151] Specifically, in one of the time schemes of the present application, after acquiring the acquired image, according to the acquisition time of the acquired image, the historical acquisition time of an acquisition cycle similar to the acquisition time...the historical acquisition time of N acquisition cycles are searched, and the cached acquisition time of an acquisition cycle similar to the acquisition time...the cached acquisition time of P acquisition cycles are searched. Exemplarily, the N is 10, the P is also 10, the P can also be 5, etc., and the present application does not make any specific limitation.

[0152] S402, extracting a target data frame from each gyroscope data frame collected by a preset gyroscope according to the collection time of the collected image, converting the target data frame into a target quaternion frame, and setting the target quaternion frame as an actual posture.

[0153] Each gyroscope data frame includes data detected by a gyroscope, exemplarily including vectors (x, y, z) corresponding to three axes of the gyroscope.

[0154] Specifically, the image acquisition period is obtained, and the image frame acquisition period is determined according to the image frame acquisition period, and the posture frame acquisition frequency / period of the gyroscope data frame is preset according to the image (image frame) acquisition period to ensure that one frame of the gyroscope data frame can be acquired at the acquisition time of each acquired image.

[0155] Among them, the target data frame is the gyroscope data frame corresponding to the acquisition time of the acquired image. Specifically, after obtaining the image to be processed, the gyroscope data frame corresponding to the acquisition time of the acquired image is obtained, and further, the gyroscope data frame is converted into a target quaternion frame, and the target quaternion frame is set as the actual posture.

[0156] S403, extracting a historical data frame from each of the gyroscope data frames according to the historical acquisition time, converting the historical data frame into a historical quaternion frame, and setting the historical quaternion frame as a historical actual posture.

[0157] Further, according to the historical acquisition time, historical gyroscope data frames corresponding to the historical acquisition time of N historical acquisition images before the acquisition time are obtained from each of the gyroscope data frames to obtain a historical data frame.

[0158] It can be understood that the gyroscope data frame carries a timestamp corresponding to the acquisition time, and the corresponding historical data frame can be extracted by comparing the timestamp with the historical acquisition time, and the extracted historical data frame can be further converted into a quaternion frame, and the historical quaternion frame can be set as the actual historical posture.

[0159] S404, extracting a cached data frame from each of the gyroscope data frames according to the cached acquisition time, converting the cached data frame into a cached quaternion frame, and setting the cached quaternion frame as a cached actual posture.

[0160] Further, according to the cache acquisition time, cache gyroscope data frames acquired corresponding to the cache acquisition time of P cache acquisition images after the acquisition time are acquired from each of the gyroscope data frames to obtain a cache data frame.

[0161] It can be understood that the gyroscope data frame carries a timestamp corresponding to the acquisition time, and the corresponding cache data frame can be extracted by comparing the timestamp with the cache acquisition time, and the extracted cache data frame is further converted into a quaternion frame, and the cached quaternion frame is set to cache the actual posture.

[0162] S405, extracting a historical virtual quaternion frame from a historical virtual quaternion set according to the historical collection time, and determining a historical virtual pose according to the historical virtual quaternion frame;

[0163] Specifically, the virtual frame buffer is searched according to the acquisition time of the historical acquisition time, and the historical virtual quaternion frame (ie, virtual pose frame) corresponding to the acquisition time in the virtual frame buffer is obtained to obtain the virtual pose.

[0164] It can be understood that the number of the virtual frame buffer can be the same as or different from the number of historical actual postures, that is, part or all of the historical acquisition time can be selected from the historical acquisition time, the historical virtual quaternion frame can be extracted, and the historical virtual quaternion frame can be extracted from the historical virtual quaternion set, and further determined as the historical virtual posture based on the historical virtual quaternion frame. Exemplarily, the historical virtual quaternion frame can be directly set as the historical virtual posture, or the historical virtual quaternion frame can be directly processed to obtain the historical virtual posture.

[0165] Exemplary: searching the virtual frame buffer for M frames of historical virtual quaternion frames with an interval sampling frequency before the current timestamp as the virtual pose.

[0166] S406: Set the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture as posture information.

[0167] Specifically, after obtaining the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture, the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture are set as the posture information corresponding to the acquired image, and the posture information corresponding to the acquired image is input into the posture prediction model to predict the virtual posture corresponding to the acquired image.

[0168] It can be understood that a quaternion frame includes a quaternion corresponding to a gyroscope data frame, wherein a quaternion consists of a real part and three imaginary parts, usually expressed as q=w+xi+yj+zk, where w is the real part, x, y and z are the imaginary parts. Here, i, j and k are quaternion units.

[0169] Furthermore, for each gyroscope data frame, the quaternion corresponding to the gyroscope data frame is calculated according to a preset quaternion calculation formula, and each of the gyroscope data frames is converted into a quaternion frame as the actual posture corresponding to the captured image.

[0170] It can be understood that, in the above implementation scheme, before setting the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture as posture information, the following steps are specifically included:

[0171] (1) If the target data frame corresponding to any target time point among the acquisition time, the historical acquisition time and the cached acquisition time is not extracted from each of the gyroscope data frames, interpolation processing is performed on the four data values ​​that are one acquisition cycle away from the target time point, and a new quaternion frame obtained by the interpolation processing is set as the quaternion frame corresponding to the target time point, and the target data frame includes at least one of the actual posture, the historical actual posture or the cached actual posture.

[0172] It is understandable that there may be an error between the gyroscope time and the time of the image acquisition, or the gyroscope acquisition may skip frames (missed acquisition). Therefore, in order to ensure the continuity of the gyroscope data frame, after obtaining the acquisition time of the acquired image, determine the historical acquisition time of the historical acquired image acquired before the acquisition time and the cached acquisition time of the cached acquired image acquired after the acquisition time according to the acquisition time and the acquisition cycle of the acquired image; and based on the acquisition time, find an aligned (i.e., the same time) frame from multiple gyroscope data frames acquired by the preset gyroscope as the target data frame acquired at the same time as the acquisition time of the acquired image, convert the target data frame into a target quaternion frame, and set the target quaternion frame to the actual posture. Furthermore, according to the historical acquisition time and the cached acquisition, A gyroscope data frame is searched at a target time point, and if a target data frame is found, the target data frame is converted into a quaternion frame. During the search process, if the gyroscope data frame of the target time point (including the acquisition time, the historical acquisition time and any target time point in the cache acquisition time) is found, the searched gyroscope data frame is determined as the target data frame of the corresponding target time point. If the target data frame of the target time point is not found, the gyroscope data frame corresponding to the time point separated from the target time point by one acquisition cycle is searched, and the gyroscope data frame is converted into a quaternion frame. The quaternion data frame separated from the target time point by one acquisition cycle is interpolated, and the new quaternion frame obtained by the interpolation is set as the quaternion frame corresponding to the target time point. If the target time point is the acquisition time, the quaternion frame corresponding to the target time point is the actual posture.

[0173] Specifically, the specific implementation method of interpolating four data frames that are one acquisition cycle away from the target time point is not specifically limited in this application. Exemplarily, proportional interpolation is performed on the time ratio relationship between the timestamps (acquisition time / the historical acquisition time / the cached acquisition time) corresponding to the four data frames that are one acquisition cycle away from the target time point to obtain the quaternion frame corresponding to the timestamp, or interpolation is performed based on the average of the current timestamp and the previous and next timestamps.

[0174] (2) For each of the historical acquisition times, a relative quaternion frame between the historical quaternion frame corresponding to the historical acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the historical quaternion frame corresponding to the historical acquisition time.

[0175] Furthermore, after determining the quaternion frames corresponding to the acquisition time, the historical acquisition time, and the cached acquisition time, for each of the historical acquisition times, a relative quaternion frame between the historical quaternion frame corresponding to the historical acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the historical quaternion frame corresponding to the historical acquisition time.

[0176] The relative quaternion may be calculated by subtracting or dividing quaternions. In the embodiment of the present application, the relative quaternion is obtained by dividing quaternions.

[0177] (3) For each of the cache acquisition times, a relative quaternion frame between the cache quaternion frame corresponding to the cache acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the cache quaternion frame corresponding to the cache acquisition time.

[0178] The relative quaternion may be calculated by subtracting or dividing quaternions. In the embodiment of the present application, the relative quaternion is obtained by dividing quaternions.

[0179] Specifically, in the implementation scheme of the present application, the timestamp of the acquired image (i.e., as the acquisition time) is obtained from the frame information of the acquired image, and the current frame (target data frame) and the N frames before and after the sampling frequency (sampling period) are searched from the gyroscope data according to the timestamp. The corresponding gyroscope information is converted into four elements, and the relative four elements with the current frame are calculated. When searching for gyroscope data with a timestamp, if the timestamp being searched is equal to the timestamp of the gyroscope data, the data corresponding to the timestamp being searched is the gyroscope data; if it is not equal, the gyroscope four-element data corresponding to the timestamps before and after the timestamp being searched is interpolated.

[0180] Further, based on the above implementation scheme, see Figure 5 , Figure 5 A schematic flow chart of one implementation scheme for obtaining a historical virtual posture in a video processing method provided in the implementation scheme of the present application, specifically including steps S501-S503:

[0181] S501 , extracting a historical virtual quaternion frame whose timestamp is the same as the historical collection time from the historical virtual quaternion set.

[0182] It can be understood that the timestamp of the historical virtual quaternion frame, that is, the timestamp corresponding to the historical actual quaternion frame (historical actual posture frame) corresponding to the historical virtual quaternion frame, that is, the acquisition time of the historical acquired image corresponding to the historical actual quaternion frame.

[0183] Furthermore, it is understandable that due to the possibility of frame drop (gyroscope data frame), the timestamp in the gyroscope data frame will be discontinuous, and the timestamp of the gyroscope data frame will be used when updating the cache of the historical virtual quaternion frame. Therefore, the timestamp of the virtual frame cache will also be discontinuous, resulting in peak jumps in the pose prediction results, and obvious stuttering in the corresponding stabilized video; therefore, after obtaining the historical acquisition time, the historical virtual quaternion frame is extracted for each timestamp of the historical virtual quaternion frame corresponding to the historical acquisition time; each historical acquisition time is compared with the timestamp of each historical virtual quaternion frame in the historical virtual quaternion set, and the historical virtual quaternion frame with the same timestamp as the historical acquisition time is extracted from the historical virtual quaternion set.

[0184] S502: For each of the historical collection time, if a historical virtual quaternion frame with the same timestamp as the historical collection time is not extracted, determine the historical virtual quaternion frame corresponding to the historical collection time according to the historical virtual quaternion frame corresponding to the timestamp adjacent to the historical collection time.

[0185] That is, taking the acquisition time as the benchmark, look forward at the sampling period to search for multiple frames of historical virtual quaternion frames. If a historical virtual quaternion frame with the same timestamp as the historical acquisition time is extracted, the last two frames of historical virtual frames in the historical virtual quaternion set are used to calculate the virtual pose; it can be understood that the historical virtual quaternion frames are arranged according to the timestamp, that is, the last frame is closest to the acquisition time, generally one acquisition period, and the current acquisition time is two sampling periods. The second to last and third to last frames of historical virtual frames are used to calculate the virtual pose, and so on. If the historical acquisition time is between the timestamps of these two frames, use these two frames for time ratio interpolation to obtain the virtual pose to be found. Furthermore, if the historical acquisition time is not between the timestamps of the two theoretically corresponding frames, use these two frames of quaternions for mean interpolation to obtain the virtual pose to be found.

[0186] S503, calculating a relative quaternion frame between the historical virtual quaternion frame corresponding to each of the historical collection times and the target quaternion frame, and setting the relative quaternion frame as the historical virtual posture corresponding to each of the historical collection times.

[0187] Specifically, after obtaining a new historical virtual quaternion frame, for each historical virtual quaternion frame, a relative quaternion frame between the historical virtual quaternion frame and the target quaternion frame is calculated, and the relative quaternion frame is set as the historical virtual pose corresponding to each historical acquisition time.

[0188] Further, in another embodiment of the present application, according to the historical acquisition time, extracting a historical virtual quaternion frame from the historical virtual quaternion set, and determining the historical virtual pose according to the historical virtual quaternion frame specifically includes:

[0189] (1) searching from the historical virtual quaternion set a target historical virtual quaternion frame corresponding to a target historical acquisition time that is one acquisition cycle away from the acquisition time interval, and a historical virtual quaternion frame corresponding to a historical acquisition time before the target historical acquisition time;

[0190] (2) sorting the historical virtual quaternion frames in chronological order to obtain a historical virtual quaternion frame sequence, wherein the target historical virtual quaternion frame is located at the end of the historical virtual quaternion frame sequence, denoted as S position;

[0191] (3) calculating the time difference between the timestamp of the historical virtual quaternion frame of the Mth frame in the historical virtual quaternion frame sequence and S times the acquisition period to obtain a target timestamp;

[0192] (4) If the target timestamp is between the timestamp of the historical virtual quaternion frame of the Mth frame and the timestamp of the historical virtual quaternion frame of the M-1th frame, proportional interpolation processing is performed according to the historical virtual quaternion frame of the Mth frame and the historical virtual quaternion frame of the M-1th frame to obtain the historical virtual quaternion frame corresponding to the target timestamp;

[0193] (5) generating a new historical virtual pose sequence according to the historical virtual quaternion frame corresponding to the target timestamp;

[0194] (6) Calculating a relative quaternion frame between each of the historical virtual quaternion frames and the target quaternion frame in the new historical virtual pose sequence, and setting the relative quaternion frame to the historical virtual pose corresponding to each of the historical acquisition times.

[0195] Further, based on the above implementation scheme, see Figure 6 , Figure 6 A schematic flow chart of one implementation scheme for obtaining a target image in a video processing method provided in the implementation scheme of the present application, specifically comprising steps S601-S604:

[0196] S601: Perform grid division processing on the collected image to obtain multiple image regions.

[0197] Specifically, the acquired image is subjected to grid division processing, that is, the acquired image is divided according to pixel rows or pixel columns; illustratively, if the sensor is row imaging, grid division is performed according to pixel rows, and each pixel row generates an image area.

[0198] S602: for each of the image regions, determine the acquisition time of the image region according to the rolling time corresponding to each pixel in the image region, and determine the actual position and posture of the image region according to the acquisition time of the image region.

[0199] The actual position and posture of the region refers to the actual position and posture of the acquisition device when imaging the image region.

[0200] It can be understood that CMOS sensor imaging has a corresponding rolling time for each pixel, that is, each image area includes multiple rolling times. In the technical solution of the present application, for each of the image areas, the timestamp of the image area, that is, the acquisition time of the image area, is determined according to the rolling time corresponding to each pixel point in the image area, to ensure that the imaging time of the image area is closer to the actual time, and then according to the imaging timestamp of the image area, the posture data frame of the acquisition device is searched, and the actual posture of the device corresponding to the imaging timestamp of the image area, that is, the actual posture of the area, is determined according to the timestamp of the posture data frame.

[0201] It is understandable that the timestamp of the image area can be determined by calculating the average of the rolling time, or can be the rolling time of the last pixel in the image area, which is not specifically limited in the present application.

[0202] S603: Calculate region mapping information according to the actual position and posture of the region and the virtual position and posture.

[0203] Among them, the region mapping information, that is, the region mapping matrix.

[0204] Furthermore, after determining the actual posture of the device corresponding to the imaging time of the image area (actual posture of the area), the area mapping matrix is ​​calculated according to the actual posture of the area and the virtual posture, that is, the actual posture of the area is converted into a real area four-element frame, and the virtual quaternion frame (virtual posture) and the real area four-element frame are converted into a real projection matrix and a virtual area projection matrix according to the camera intrinsic parameters of the acquisition device, and then the mapping matrix between them is calculated according to the real projection matrix and the area virtual projection matrix as the area mapping matrix.

[0205] S604: Perform mapping processing on the image area according to the area mapping information of each image area to obtain target image information.

[0206] Furthermore, after obtaining the region mapping matrix, the corresponding image region is mapped according to each region mapping matrix to obtain a target image; that is, the original image is rendered into a corresponding stabilized image (target image) through multiple mapping matrices, and the target images are combined into a video.

[0207] Furthermore, based on any of the above implementation schemes, the present application also provides an implementation scheme for virtual posture correction, which includes the following steps after inputting the target posture corresponding to the acquired image into a preset prediction model and predicting the virtual posture corresponding to the acquired image:

[0208] (1) Calculating an initial mapping matrix according to the predicted virtual pose and the actual pose corresponding to the acquired image.

[0209] Specifically, after the virtual pose corresponding to the actual pose is obtained according to the preset prediction model, the initial mapping matrix is ​​calculated according to the virtual pose and the virtual pose, that is, according to the camera intrinsic parameters of the acquisition device, the virtual quaternion frame (virtual pose) and the real quaternion frame (real pose) are converted into a real projection matrix and a virtual projection matrix, and then the mapping matrix between them is calculated according to the real projection matrix and the virtual projection matrix as the initial mapping matrix.

[0210] (2) determining, according to the initial mapping matrix, that the four corner points of the acquired image are located at the corner point positions in the stabilized initial target image, and if the corner point positions are within a preset cropping frame position range, performing interpolation correction on the virtual pose;

[0211] Among them, the method for interpolating and correcting the virtual posture includes iteratively adjusting the virtual posture and the real posture in a certain step interpolation manner, exemplarily: taking values ​​from 0 to 1 at a certain interval (such as 0.1) as interpolation coefficients to interpolate.

[0212] (3) If, according to a new initial mapping matrix corresponding to the interpolated and corrected virtual posture, it is determined that the four corner points of the acquired image are located outside the preset cropping frame position range in the initial target image after stabilization, the interpolated and corrected virtual posture is set as the posture, and the step of optimizing the acquired image according to the virtual posture and the actual posture is performed according to the corrected virtual posture to obtain the target image.

[0213] Further, based on the above implementation scheme, see Figure 7 , Figure 7 A schematic diagram of one implementation structure of the preset prediction model in the video processing method provided in the implementation scheme of this application:

[0214] Specifically, the preset prediction model includes an attention mechanism module, a recursive neural network module (i.e., an LSTM network) and a fully connected module; wherein the real frame quaternion and the virtual frame quaternion described in the figure, i.e., the pose information of the captured image.

[0215] The step of inputting the pose information into a preset prediction model to predict the virtual pose of the target image comprises the following steps:

[0216] (1) inputting the posture information into the attention mechanism module of the posture prediction model for weighted processing, and outputting weighted frame feature information;

[0217] (2) inputting the frame feature information into the recursive neural network module in the posture prediction model to extract features and output feature information;

[0218] (3) Inputting the feature information into the fully connected module in the pose prediction model to predict the virtual pose corresponding to the captured image.

[0219] For further information, see Figure 8 , Figure 8 A schematic diagram of the structure of another implementation scheme of the preset prediction model in the video processing method provided in the implementation scheme of the present application, wherein the attention mechanism module can be an Attention network layer; the Attention network layer can also include a Concat layer before it, which is used to splice the target posture; the recursive neural network module, that is, the LSTM network, LSTM can use multiple layers, for example, see the LSTMCell (512) layer in the figure; the fully connected module, that is, the fully connected layer (see FC-256, FC-4), can use multiple layers, and the extracted features are predicted through the fully connected layer The virtual posture of the actual posture corresponding to the captured image.

[0220] Specifically, the four-element information (posture information) of multiple frames is input into the attention mechanism module to obtain weighted frame feature information. For example, a one-dimensional convolution of size 4 is performed first, and then softmax is performed to obtain the weight, and then the residual structure is used to obtain the final frame feature information; the weighted multi-frame information is converted into a one-dimensional vector and input into the LSTM network to extract features, and the LSTM can use multiple layers; the extracted features are further passed through the fully connected layer to predict the virtual posture of the camera of the current frame (corresponding to the acquired image), and the fully connected layer can use multiple layers.

[0221] For details, see Fig. 9 , Fig. 9 A schematic diagram of one of the implementation schemes of the structure of the Attention network layer in the video processing method provided in the implementation scheme of the present application;

[0222] For example, the input is X(b,n,4), where b is the batch; n is the number of frames, such as 10 past frames, 1 current frame, 10 future frames, and 2 virtual frames, 10+10+1+2=23, b=23; 4 refers to the quaternion. The Conv1d layer first performs a one-dimensional convolution of size 4, and the softmax layer performs softmax to obtain the weights, and then uses the residual structure to obtain the final frame feature information.

[0223] The above-mentioned implementation scheme of the present application provides a communication video processing method, which obtains a captured image to be processed, and obtains the posture information corresponding to the captured image, the posture information includes the actual posture when the captured image is captured, the historical actual posture captured before the actual posture, the cached actual posture captured after the actual posture, and the historical virtual posture corresponding to the historical actual posture; the posture information corresponding to the captured image is input into a preset posture prediction model to predict the virtual posture corresponding to the captured image; the posture prediction model is obtained by training the actual posture samples when the sample images are captured, the sample historical actual postures captured before the sample actual postures, the sample cached actual postures captured after the sample actual postures, and the sample historical virtual postures corresponding to the sample historical actual postures; the captured image is optimized according to the virtual posture and the actual posture to obtain a target image, and a video is generated according to the target image. This scheme obtains a preset posture prediction model based on the sample actual posture of the sample at the corresponding collection time of the sample collected image, the sample historical actual posture, the sample cache actual posture and the corresponding sample virtual posture training, and obtains a prediction model that can predict the virtual posture, and further analyzes the corresponding posture information of the collected image through the model, that is, predicts the virtual posture that the actual posture should be in after stabilization, and further, optimizes the collected image according to the virtual posture and the actual posture, so as to obtain the target image to generate the video, that is, the stabilized virtual posture is predicted by combining the actual posture of the collected image and the posture information of the previous and next time periods and the trajectory change information of the historical virtual posture and the correlation between the actual posture and the virtual posture, and further optimizes the collected image according to the mapping relationship between the predicted virtual posture and the actual posture, so that the optimization processing of the collected image can be adaptively processed corresponding to the motion information, thereby improving the accuracy of the video optimization processing.

[0224] In order to better implement the video processing method in the embodiment of the present application, based on the video processing method, the embodiment of the present application also provides a video processing device, such as Fig.10 As shown, the video processing device includes modules 101-103:

[0225] An acquisition module 101 is used to acquire a captured image to be processed and position information corresponding to the captured image;

[0226] A prediction module 102, configured to input the posture information corresponding to the collected image into a posture prediction model to predict a virtual posture corresponding to the collected image;

[0227] The processing module 103 is used to optimize the collected image according to the virtual posture and the posture information to obtain target image information.

[0228] In one embodiment of the present application, the posture information includes the actual posture when the captured image is collected, the historical actual posture collected before the actual posture, the cached actual posture collected after the actual posture, and the historical virtual posture corresponding to the historical actual posture;

[0229] The acquisition module 101 is used to acquire the acquired image to be processed and the posture information corresponding to the acquired image, and specifically includes:

[0230] Determine the actual posture and the historical actual posture according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and each gyroscope data frame acquired by the preset gyroscope;

[0231] Determining a historical virtual pose according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and the historical virtual quaternion set;

[0232] The actual pose, the historical actual pose, the cached actual pose and the historical virtual pose are set as pose information.

[0233] In one embodiment of the present application, the acquisition module 101 is used to determine the actual posture and the historical actual posture according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and each gyroscope data frame acquired by the preset gyroscope, specifically including:

[0234] Acquire a collected image to be processed, a collection time and a collection period of the collected image;

[0235] According to the acquisition time and the acquisition cycle, determine the historical acquisition time for acquiring the historical acquisition image before the acquisition time, and the cache acquisition time for acquiring the cache acquisition image after the acquisition time;

[0236] According to the acquisition time of the acquired image, a target data frame is extracted from each gyroscope data frame acquired by a preset gyroscope, and the target data frame is converted into a target quaternion frame, and the target quaternion frame is set as an actual posture;

[0237] Extracting a historical data frame from each of the gyroscope data frames according to the historical acquisition time, converting the historical data frame into a historical quaternion frame, and setting the historical quaternion frame as a historical actual posture;

[0238] A cache data frame is extracted from each of the gyroscope data frames according to the cache acquisition time, and the cache data frame is converted into a cache quaternion frame, and the cache quaternion frame is set as a cache actual posture.

[0239] In one embodiment of the present application, the acquisition module 101 is used to set the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture as posture information, and specifically includes:

[0240] If the target data frame corresponding to any target time point among the acquisition time, the historical acquisition time and the cached acquisition time is not extracted from each of the gyroscope data frames, interpolation processing is performed on the four data values ​​one acquisition cycle away from the target time point, and a new quaternion frame obtained by the interpolation processing is set as the quaternion frame corresponding to the target time point, wherein the target data frame includes at least one of the actual posture, the historical actual posture or the cached actual posture;

[0241] For each of the historical acquisition times, a relative quaternion frame between the historical quaternion frame corresponding to the historical acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the historical quaternion frame corresponding to the historical acquisition time;

[0242] For each of the cache acquisition time, a relative quaternion frame between the cache quaternion frame corresponding to the cache acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the cache quaternion frame corresponding to the cache acquisition time.

[0243] In one embodiment of the present application, the acquisition module 101 is used to determine the historical virtual pose according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and the historical virtual quaternion set, specifically including:

[0244] Extracting a historical virtual quaternion frame with the same timestamp as the historical acquisition time from the historical virtual quaternion set;

[0245] For each of the historical collection time, if a historical virtual quaternion frame with the same timestamp as the historical collection time is not extracted, determining the historical virtual quaternion frame corresponding to the historical collection time according to the historical virtual quaternion frame corresponding to the timestamp adjacent to the historical collection time;

[0246] A relative quaternion frame between the historical virtual quaternion frame corresponding to each of the historical acquisition times and the target quaternion frame is calculated, and the relative quaternion frame is set as the historical virtual posture corresponding to each of the historical acquisition times.

[0247] In one embodiment of the present application, the processing module 103 is used to optimize the collected image according to the virtual posture and the actual posture to obtain target image information, specifically including:

[0248] Performing grid division processing on the collected image to obtain multiple image regions;

[0249] For each of the image regions, determining an acquisition time of the image region according to a rolling time corresponding to each pixel point in the image region, and determining an actual position of the image region according to the acquisition time of the image region;

[0250] Calculating region mapping information according to the actual position and posture of the region and the virtual position and posture;

[0251] According to the region mapping information of each image region, mapping processing is performed on the image region to obtain target image information.

[0252] In one embodiment of the present application, the prediction module 102 is used to input the posture information corresponding to the collected image into a preset posture prediction model to predict the virtual posture corresponding to the collected image, specifically including:

[0253] Inputting the posture information into the attention mechanism module of the posture prediction model for weighted processing, and outputting weighted frame feature information;

[0254] Inputting the frame feature information into the recursive neural network module in the posture prediction model for feature extraction, and outputting feature information;

[0255] The feature information is input into a fully connected module in the pose prediction model to predict a virtual pose corresponding to the captured image.

[0256] In one embodiment of the present application, the posture prediction model is obtained by training the sample actual posture when collecting the sample collection image, the sample historical actual posture collected before the sample actual posture, the sample cache actual posture collected after the sample actual posture, and the sample historical virtual posture corresponding to the sample historical actual posture;

[0257] The video processing device also includes a training module for constructing a preset posture prediction model training, specifically including:

[0258] Acquire sample pose information, wherein the sample pose information includes an actual pose sample when the sample capture image is collected, a sample historical actual pose collected before the sample actual pose, a sample future actual pose collected after the sample actual pose, and a sample historical virtual pose corresponding to the sample historical actual pose;

[0259] Training the initial prediction model according to the sample pose information to obtain a training prediction model;

[0260] Until the sample virtual pose predicted by the training prediction model meets the convergence condition, a preset pose prediction model is obtained.

[0261] In one embodiment of the present application, the training module is used to train the initial prediction model according to the sample pose information to obtain the training prediction model, specifically including:

[0262] Calculate a smoothing loss parameter according to the sample virtual pose predicted by the training prediction model, the acquisition period of the sample acquisition image, and a preset first loss function;

[0263] Calculating an anti-distortion loss parameter according to the actual pose of the sample, the virtual pose of the sample, and a preset second loss function;

[0264] Calculating an anti-cross-border loss parameter and an over-cropping loss parameter according to a preset cropping rate, the sample virtual pose, and the sample acquisition image;

[0265] A total loss parameter is calculated according to the smoothing loss parameter, the anti-distortion loss parameter, the anti-cross-boundary loss parameter and the over-clipping loss parameter;

[0266] The initial prediction model is modified according to the total loss parameter to obtain a training prediction model.

[0267] In one embodiment of the present application, the training module is used to calculate the anti-cross-border loss parameter and the over-cropping loss parameter according to the preset cropping rate, the sample virtual pose, and the sample acquisition image, and specifically includes:

[0268] According to the mapping relationship between the sample virtual pose and the sample actual pose, mapping the corner point information of the sample captured image to obtain mapped corner point information;

[0269] Calculating an anti-crossing loss parameter according to the mapped corner point information and theoretical corner point information of the sample acquisition image corresponding to a first preset cropping rate;

[0270] An over-cropping loss parameter is calculated according to the mapped corner point information and theoretical corner point information of the sample acquisition image corresponding to a second preset cropping rate.

[0271] The embodiment of the present invention also provides a video processing device, such as Fig.11 As shown, Fig.11 It is a schematic diagram of the structure of an embodiment of a video processing device provided in an embodiment of the present application.

[0272] Video processing equipment includes:

[0273] one or more processors;

[0274] Memory; and

[0275] One or more applications, wherein the one or more applications are stored in the memory and are configured so that the processor executes the steps of the video processing method described in any of the above-mentioned video processing method embodiments.

[0276] Specifically, the video processing device may include one or more processing core processors 1001, one or more computer-readable storage media memories 1002, a power supply 1003, an input unit 1004 and other components. Those skilled in the art will understand that Fig.10 The video processing device structure shown in the figure does not constitute a limitation on the video processing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0277] The processor 1001 is the control center of the video processing device. It uses various interfaces and lines to connect various parts of the entire video processing device. By running or executing software programs and / or modules stored in the memory 1002, and calling data stored in the memory 1002, it executes various functions of the video processing device and processes data, thereby monitoring the video processing device as a whole. It can be understood that the processor 1001 transmits signals with the controller. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 1001.

[0278] The memory 1002 can be used to store software programs and modules. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the video processing device, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0279] In some embodiments of the present application, the video processing device can be implemented in the form of a computer program. The computer program can be Fig.11 The memory of the video processing device can store various program modules constituting the video processing device, such as: Fig.10 The acquisition module 101, prediction module 102, and processing module 103 are shown. The computer program composed of various program modules enables the processor to execute the steps of the video processing method of each embodiment of the present application described in this specification.

[0280] For example, Fig.11 The video processing device shown can be Fig.10 The acquisition module 101 in the video processing device shown executes step S201. The video processing device can execute step S202 through the prediction module 102. The video processing device can execute step S203 through the processing module 103. The video processing device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the video processing device is used to provide computing and control capabilities. The memory of the video processing device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the video processing device is used to communicate with an external video processing device through a network connection. When the computer program is executed by the processor, a video processing method is implemented.

[0281] The video processing device also includes a power supply 1003 for supplying power to each component. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 1003 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0282] The video processing device may further include an input unit 1004, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0283] Although not shown, the video processing device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 1001 in the video processing device will load the executable files corresponding to the processes of one or more application programs into the memory 1002 according to the following instructions, and the processor 1001 will run the application programs stored in the memory 1002, thereby realizing various functions, as follows:

[0284] Acquire a collected image to be processed and position information corresponding to the collected image;

[0285] Inputting the posture information corresponding to the collected image into a posture prediction model to predict the virtual posture corresponding to the collected image;

[0286] The collected image is optimized according to the virtual posture and the posture information to obtain target image information.

[0287] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0288] To this end, an embodiment of the present invention provides a computer-readable storage medium (hereinafter referred to as a storage medium), which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any video processing method provided in an embodiment of the present invention. For example, the computer program may be loaded by a processor to execute the following steps:

[0289] Acquire a collected image to be processed and position information corresponding to the collected image;

[0290] Inputting the posture information corresponding to the collected image into a posture prediction model to predict the virtual posture corresponding to the collected image;

[0291] The collected image is optimized according to the virtual posture and the posture information to obtain target image information.

[0292] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0293] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0294] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0295] The above is a detailed introduction to a video processing method, device, equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A video processing method, characterized in that: include: Acquire a collected image to be processed and position information corresponding to the collected image; Inputting the posture information corresponding to the collected image into a posture prediction model to predict the virtual posture corresponding to the collected image; The collected image is optimized according to the virtual posture and the posture information to obtain target image information.

2. The video processing method according to claim 1, characterized in that: The posture information includes the actual posture when the captured image is collected, the historical actual posture collected before the actual posture, the cached actual posture collected after the actual posture, and the historical virtual posture corresponding to the historical actual posture; The obtaining of the collected image to be processed and the posture information corresponding to the collected image includes: Determine the actual posture and the historical actual posture according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and each gyroscope data frame acquired by the preset gyroscope; Determining a historical virtual pose according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and the historical virtual quaternion set; The actual pose, the historical actual pose, the cached actual pose and the historical virtual pose are set as pose information.

3. The video processing method according to claim 2, characterized in that: The determining the actual posture and the historical actual posture according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and each gyroscope data frame acquired by the preset gyroscope includes: Acquire a collected image to be processed, a collection time and a collection period of the collected image; According to the acquisition time and the acquisition cycle, determine the historical acquisition time for acquiring the historical acquisition image before the acquisition time, and the cache acquisition time for acquiring the cache acquisition image after the acquisition time; According to the acquisition time of the acquired image, a target data frame is extracted from each gyroscope data frame acquired by a preset gyroscope, and the target data frame is converted into a target quaternion frame, and the target quaternion frame is set as an actual posture; Extracting a historical data frame from each of the gyroscope data frames according to the historical acquisition time, converting the historical data frame into a historical quaternion frame, and setting the historical quaternion frame as a historical actual posture; A cache data frame is extracted from each of the gyroscope data frames according to the cache acquisition time, and the cache data frame is converted into a cache quaternion frame, and the cache quaternion frame is set as a cache actual posture.

4. The video processing method according to claim 3, characterized in that: Before setting the actual posture, the historical actual posture, the cached actual posture and the historical virtual posture as posture information, the method includes: If the target data frame corresponding to any target time point among the acquisition time, the historical acquisition time and the cached acquisition time is not extracted from each of the gyroscope data frames, interpolation processing is performed on the four data values ​​one acquisition cycle away from the target time point, and a new quaternion frame obtained by the interpolation processing is set as the quaternion frame corresponding to the target time point, wherein the target data frame includes at least one of the actual posture, the historical actual posture or the cached actual posture; For each of the historical acquisition times, a relative quaternion frame between the historical quaternion frame corresponding to the historical acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the historical quaternion frame corresponding to the historical acquisition time; For each of the cache acquisition time, a relative quaternion frame between the cache quaternion frame corresponding to the cache acquisition time and the target quaternion frame is calculated, and the relative quaternion frame is updated to the cache quaternion frame corresponding to the cache acquisition time.

5. The video processing method according to claim 2, characterized in that: Determining the historical virtual pose according to the acquisition time of the acquired image, the acquisition cycle of the acquired image, and the historical virtual quaternion set includes: Extracting a historical virtual quaternion frame with the same timestamp as the historical acquisition time from the historical virtual quaternion set; For each of the historical collection time, if a historical virtual quaternion frame with the same timestamp as the historical collection time is not extracted, determining the historical virtual quaternion frame corresponding to the historical collection time according to the historical virtual quaternion frame corresponding to the timestamp adjacent to the historical collection time; A relative quaternion frame between the historical virtual quaternion frame corresponding to each of the historical acquisition times and the target quaternion frame is calculated, and the relative quaternion frame is set as the historical virtual posture corresponding to each of the historical acquisition times.

6. The video processing method according to claim 1, characterized in that: The step of optimizing the collected image according to the virtual posture and the actual posture to obtain target image information includes: Performing grid division processing on the collected image to obtain multiple image regions; For each of the image regions, determining an acquisition time of the image region according to a rolling time corresponding to each pixel point in the image region, and determining an actual position of the image region according to the acquisition time of the image region; Calculating region mapping information according to the actual position and posture of the region and the virtual position and posture; According to the region mapping information of each image region, mapping processing is performed on the image region to obtain target image information.

7. The video processing method according to claim 1, characterized in that: The step of inputting the posture information corresponding to the collected image into a preset posture prediction model to predict the virtual posture corresponding to the collected image includes: Inputting the posture information into the attention mechanism module of the posture prediction model for weighted processing, and outputting weighted frame feature information; Inputting the frame feature information into the recursive neural network module in the posture prediction model for feature extraction, and outputting feature information; The feature information is input into a fully connected module in the pose prediction model to predict a virtual pose corresponding to the captured image.

8. The video processing method according to any one of claims 1 to 7, characterized in that: The posture prediction model is obtained by training the sample actual posture when collecting the sample image, the sample historical actual posture collected before the sample actual posture, the sample cache actual posture collected after the sample actual posture, and the sample historical virtual posture corresponding to the sample historical actual posture; The steps of constructing the preset posture prediction model training include: Acquire sample pose information, wherein the sample pose information includes an actual pose sample when the sample capture image is collected, a sample historical actual pose collected before the sample actual pose, a sample future actual pose collected after the sample actual pose, and a sample historical virtual pose corresponding to the sample historical actual pose; Training the initial prediction model according to the sample pose information to obtain a training prediction model; Until the sample virtual pose predicted by the training prediction model meets the convergence condition, a preset pose prediction model is obtained.

9. The video processing method according to claim 8, characterized in that: The initial prediction model is trained according to the sample pose information to obtain a training prediction model, including: Calculate a smoothing loss parameter according to the sample virtual pose predicted by the training prediction model, the acquisition period of the sample acquisition image, and a preset first loss function; Calculating an anti-distortion loss parameter according to the actual pose of the sample, the virtual pose of the sample, and a preset second loss function; Calculating an anti-cross-border loss parameter and an over-cropping loss parameter according to a preset cropping rate, the sample virtual pose, and the sample acquisition image; A total loss parameter is calculated according to the smoothing loss parameter, the anti-distortion loss parameter, the anti-cross-boundary loss parameter and the over-clipping loss parameter; The initial prediction model is modified according to the total loss parameter to obtain a training prediction model.

10. A video processing device, characterized in that: include: An acquisition module, used to acquire a collected image to be processed and position information corresponding to the collected image; A prediction module, used to input the posture information corresponding to the collected image into a preset posture prediction model to predict the virtual posture corresponding to the collected image; A processing module is used to optimize the collected image according to the virtual posture and the posture information to obtain target image information.

11. A video processing device, characterized in that: The video processing device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to implement the steps in the video processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the video processing method according to any one of claims 1 to 9.

Images