Video processing method, device, computer equipment and storage medium

Abstract

The present application provides a video processing method, device, computer equipment and storage medium, the method comprising: obtaining a panoramic video stream to be processed, obtaining the corresponding current original camera posture from the current panoramic video frame, performing horizontal angle correction on the current original camera posture to obtain the corrected horizontal camera posture, performing equidistant cylindrical projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting the target object, and tracking the target object, encoding the video frame after target tracking, and generating a planar video about the target object. The present application can automatically generate a special effect planar video of a drone tracking perspective without the user's complicated video editing operations.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing technology, and specifically to a video processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of video processing technology, people are becoming more and more interested in various special video processing effects, especially image processing involving moving objects. For example, when shooting a moving person from a drone’s perspective, due to the large field of view under the drone’s perspective, it is easy for multiple target objects to appear within the field of view, making it impossible to accurately determine the target object that the photographer needs to follow. In addition, in traditional shooting and post-processing of video, the creation method of drone’s perspective special effects is also very cumbersome, requiring users to perform complex post-production, which is difficult for ordinary users. Therefore, there is an urgent need for a special effects video processing method for drone’s perspective that is simple to shoot and does not require users to perform complex post-processing operations. Summary of the invention

[0003] The embodiments of the present application provide a video processing method, apparatus, computer equipment and storage medium. The technical problem to be solved is that, in view of the defects of the prior art, the existing special effects video processing method of the drone tracking perspective is complicated and cumbersome, and requires the user to have high professional video processing capabilities.

[0004] In order to solve the above technical problems, a technical solution adopted by the present application is: to provide a video processing method, apparatus, computer equipment and storage medium, wherein the video processing method comprises:

[0005] Get the panoramic video stream to be processed;

[0006] Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture;

[0007] Performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting a target object, and tracking the target object;

[0008] The video frames after target tracking are encoded to generate a planar video about the target object.

[0009] In a specific implementation, the performing horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture includes:

[0010] The Kalman filter is used to track the current original camera posture corresponding to the current panoramic video frame of the panoramic video stream to obtain a first camera rotation matrix R t , according to the first camera rotation matrix R t Rotating the current original camera posture to obtain a smooth camera posture;

[0011] According to the camera's inertial sensor data, the second camera rotation matrix R is calculated s , the second camera rotation matrix Rs is, when the Z axis of the camera coordinate system is in the same direction as the gravity vector in an ideal state, the rotation matrix of the camera coordinate system relative to the inertial coordinate system;

[0012] According to the first camera rotation matrix R t and the second camera rotation matrix Rs, a horizontal correction matrix R is calculated, and a horizontal angle correction is performed on the current original camera posture according to the horizontal correction matrix R to obtain a corrected horizontal camera posture.

[0013] In a specific implementation, the second camera rotation matrix R is calculated based on the inertial sensor data of the camera. s ,include:

[0014] The inertial sensor data includes accelerometer data, which is the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system. s =[a sx , a sy , a sz ] T ;

[0015] In the ideal state, the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system is s is a0, a0=[0,0,g] T , where g = 9.81 m / s 2 ;

[0016] Then it satisfies that a s =R s *a0;

[0017] Calculate the second camera rotation matrix R s for: ;

[0018] in, is the pitch angle of the camera coordinate system relative to the inertial coordinate system, is the roll angle of the camera coordinate system relative to the inertial coordinate system.

[0019] In a specific implementation, the first camera rotation matrix R t And the second camera rotation matrix R s , calculate the horizontal correction matrix R, including: the first camera rotation matrix R t 、The second camera rotation matrix R s , and the horizontal correction matrix R satisfies: R = R s -1 R t .

[0020] In a specific embodiment, the panoramic video frame corresponding to the corrected horizontal camera posture is expanded by equidistant cylindrical projection, and target detection is performed in a set expansion range after the expansion, including: the panoramic video frame corresponding to the corrected horizontal camera posture is expanded by equidistant cylindrical projection to a planar rectangular video frame, the planar rectangular video frame has a longitude and latitude coordinate system, the abscissa u of the longitude and latitude coordinate system linearly maps the longitude of the panoramic video frame, and the value range is [−π,π], and the ordinate v of the longitude and latitude coordinate system linearly maps the latitude of the panoramic video frame, and the value range is [−π / 2, π / 2]; target detection is performed in a set latitude area in the expanded planar rectangular video frame.

[0021] In a specific implementation, the set latitude area range is [-70°, 30°].

[0022] In a specific implementation, the target object is an object with the largest size detected during target detection within the set expansion range.

[0023] In a specific implementation, a video processing device is also included, and the video processing device includes:

[0024] An acquisition module, wherein the acquisition module is used to acquire a panoramic video stream to be processed;

[0025] A horizontal correction module, the horizontal correction module is used to obtain the corresponding current original camera posture from the current panoramic video frame of the panoramic video stream, perform horizontal angle correction on the current original camera posture, and obtain a corrected horizontal camera posture;

[0026] A tracking module, the tracking module is used to perform equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and perform target detection in a set expansion range after expansion, detect the target object, and track the target object;

[0027] A generation module is used to encode the video frames after target tracking to generate a planar video about the target object.

[0028] In a specific implementation, a computer device is also included, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the video processing method described in any of the above specific implementations when executing the computer program.

[0029] In a specific implementation, it also includes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video processing method described in any of the above specific implementations are implemented.

[0030] The beneficial effects of the present application are as follows: Different from the prior art, the embodiments of the present application provide a video processing method, apparatus, computer equipment and storage medium, wherein the video processing method includes:

[0031] Get the panoramic video stream to be processed;

[0032] Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture;

[0033] Performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting a target object, and tracking the target object;

[0034] The video frames after target tracking are encoded to generate a planar video about the target object.

[0035] It can be seen from the above technical scheme that the embodiment of the present application has the following advantages: by performing horizontal angle correction processing on the panoramic video frame by setting a horizontal correction algorithm, a plane video frame with a stable horizontal angle is obtained, and the plane video frame with a horizontal angle is expanded by equidistant cylindrical projection, so that the possible position of the target object can be represented by a latitude range, thereby narrowing the search range of the target tracking algorithm and improving the accuracy of target detection and tracking, thereby realizing the automatic generation of a special effects plane video from the perspective of a drone follow-up shot, simulating the shooting effect of a drone follow-up shot.

[0036] This application improves the efficiency of producing special effects two-dimensional videos from the perspective of drone tracking. Users can automatically generate special effects two-dimensional videos from the perspective of drone tracking through algorithms without complex shooting techniques and complex post-video processing capabilities. A new way to play panoramic videos is added, which improves user playability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the implementation modes of the present application, the drawings required for use in the description of the implementation modes will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a schematic diagram of an implementation flow of a video processing method provided in Example 1 of the present application;

[0039] Figure 2 This is an example schematic diagram of the position relationship between a user and a camera in a video processing method provided in Embodiment 1 of the present application;

[0040] Figure 3 It is a schematic diagram of the module composition of a video processing device provided in Embodiment 2 of the present application;

[0041] Figure 4 It is a schematic diagram of the internal structure of a computer device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0042] The present application is further described in detail below in conjunction with the accompanying drawings and implementation methods. It is particularly noted that the following implementation methods are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following implementation methods are only some implementation methods of the present application rather than all implementation methods. All other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0043] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to these processes, methods, products or devices.

[0044] The above terms are only for the convenience of description and should not be construed as limitations on the present technical solution.

[0045] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0046] The specific implementation of the present application is described in detail below in conjunction with specific embodiments.

[0047] Embodiment 1

[0048] Figure 1 The implementation process of a video processing method provided in Example 1 of the present application is shown. For the convenience of explanation, only the part related to the embodiment of the present application is shown, which is described in detail as follows.

[0049] Get the panoramic video stream to be processed;

[0050] It should be noted that the embodiments of the present application are applicable to video image processing devices such as video display and acquisition. The panoramic video stream is panoramic video data captured and collected by a panoramic camera.

[0051] Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture;

[0052] It should be noted that the camera posture is a three-dimensional spatial information, and the three-dimensional spatial information corresponding to the current camera posture can be represented by quaternion or Euler angle or rotation matrix to obtain the current original camera posture corresponding to the current panoramic video frame.

[0053] It should be understood that in some specific calculations, in order to facilitate calculation, it is necessary to convert the representation of the camera posture. For example, in spherical linear interpolation calculations, if the camera posture is represented by Euler angles or rotation matrices, in order to facilitate calculation, the quaternion representation corresponding to the camera posture can be obtained through conversion calculation. Since the conversion calculations between quaternions, Euler angles, and rotation matrices are common knowledge in the field, they will not be elaborated here.

[0054] One specific implementation method is that the horizontal angle correction is performed on the current original camera posture to obtain a corrected horizontal camera posture, including:

[0055] The Kalman filter is used to track the current original camera posture corresponding to the current panoramic video frame of the panoramic video stream to obtain a first camera rotation matrix R t , according to the first camera rotation matrix R t Rotating the current original camera posture to obtain a smooth camera posture;

[0056] According to the camera's inertial sensor data, the second camera rotation matrix R is calculated s , the second camera rotation matrix R s is, when the Z axis of the camera coordinate system is in the same direction as the gravity vector in an ideal state, the rotation matrix of the camera coordinate system relative to the inertial coordinate system;

[0057] According to the first camera rotation matrix R t And the second camera rotation matrix R s , a horizontal correction matrix R is calculated, and the horizontal angle correction is performed on the current original camera posture according to the horizontal correction matrix R to obtain a corrected horizontal camera posture.

[0058] It should be noted that the first camera rotation matrix R t is the rotation matrix of the original camera posture of the camera coordinate system in the inertial coordinate system, which represents the rotation matrix formed by the rotation of the camera around three axes in the inertial coordinate system from the initial moment (e.g., the initial shooting moment), that is, the original camera posture moment, to moment t. The current panoramic video frame is rotated by the first camera rotation matrix R t The rotation transformation is used to correct the camera posture and obtain a smooth camera posture of the camera, thereby achieving anti-shake of the panoramic video frame.

[0059] The second camera rotation matrix R s is the rotation matrix of the camera coordinate system relative to the inertial coordinate system when the Z axis of the camera coordinate system is aligned with the direction of the gravity vector in an ideal state. At time s, when the Z axis of the camera coordinate system is aligned with the direction of the gravity vector in an ideal state, it represents the rotation matrix of the camera coordinate system relative to the inertial coordinate system from the initial moment (for example, the initial shooting moment), that is, the original camera posture moment, to time s.

[0060] One specific implementation method is that the second camera rotation matrix R is calculated based on the inertial sensor data of the camera. s ,include:

[0061] The inertial sensor data includes accelerometer data, which is the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system. s =[a sx , a sy , a sz ] T ;

[0062] In the ideal state, the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system is s

[0063] is a0, a0=[0,0,g] T , where g = 9.81 m / s 2 ;

[0064] Then it satisfies that a s =R s *a0;

[0065] Calculate the second camera rotation matrix R s for:

[0066] ;

[0067] in, is the pitch angle of the camera coordinate system relative to the inertial coordinate system, is the roll angle of the camera coordinate system relative to the inertial coordinate system.

[0068] One specific implementation method is that according to the first camera rotation matrix R t And the second camera rotation matrix R s , the horizontal correction matrix R is calculated, including:

[0069] The first camera rotation matrix R t 、The second camera rotation matrix R s , and the horizontal correction matrix R satisfies: R = R s -1 R t .

[0070] It should be noted that the horizontal correction matrix R is obtained by first transforming the current original camera posture through the first camera rotation matrix R t Rotate so that the posture of the camera coordinate system is corrected in the inertial coordinate system, so as to achieve a smooth transition of the camera posture, obtain a stable panoramic video frame, and achieve the effect of picture anti-shake, and then pass the second camera rotation matrix R s By performing a reverse rotation, the Z axis of the camera coordinate system is made to coincide with the direction of gravity, so that the captured horizontal line can remain horizontal in the panoramic video frame, thereby simulating the effect of shooting while the drone is flying and moving on a horizontal plane.

[0071] Then the corresponding horizontal correction matrix R satisfies: R = R s -1 R t .

[0072] One specific implementation method is that the panoramic video frame corresponding to the corrected horizontal camera posture is expanded by equirectangular projection, and target detection is performed in a set expansion range after expansion, including:

[0073] Performing equidistant cylindrical projection on the panoramic video frame corresponding to the corrected horizontal camera posture to expand it into a planar rectangular video frame, wherein the planar rectangular video frame has a longitude and latitude coordinate system, wherein the abscissa u of the longitude and latitude coordinate system linearly maps the longitude of the panoramic video frame, and the value range is [−π,π], and the ordinate v of the longitude and latitude coordinate system linearly maps the latitude of the panoramic video frame, and the value range is [−π / 2, π / 2];

[0074] Target detection is performed in a set latitude area in the expanded planar rectangular video frame.

[0075] It should be noted that the panoramic video frame is mapped in a panoramic spherical model, which has a 3D stereoscopic coordinate system corresponding to the spherical coordinate system of the camera. The coordinate origin of the stereoscopic coordinate system is at the center of the sphere, the X-axis points to the front of the sphere, the Y-axis points to the top of the sphere, and the Z-axis points to the right of the sphere. The stereoscopic coordinate system conforms to the right-hand system. Points on the sphere can be represented by longitude ϕ and latitude θ. The longitude ϕ ranges from [−π,π], and the latitude θ ranges from [−π / 2, π / 2]. The longitude ϕ is defined as the angle of counterclockwise rotation along the X-axis, and the latitude θ is defined as the angle of rotation from the equator to the Y-axis.

[0076] On a unit sphere (i.e., the radius of the sphere is 1), the conversion relationship between coordinates (X, Y, Z) and (ϕ, θ) is as follows:

[0077] X = cosθcosϕ;

[0078] Y = sinθ;

[0079] Z = -cosθsinϕ;

[0080] ϕ=tan -1 (Z / X);

[0081] θ = sin -1 (Y / (X 2 +Y 2 +Z 2 ) 1 / 2 );

[0082] The purpose of projection is to map the pixels on the spherical surface of the panoramic ball model to a two-dimensional plane (uv longitude and latitude coordinate system), that is, to convert (X, Y, Z) coordinates into (u, v) coordinates. The abscissa u of the longitude and latitude coordinate system linearly maps the longitude ϕ of the panoramic video frame, with a value range of [−π, π], and the ordinate v of the longitude and latitude coordinate system linearly maps the latitude θ of the panoramic video frame, with a value range of [−π / 2, π / 2].

[0083] In one specific implementation manner, the set latitude area range is [-70°, 30°].

[0084] It should be noted that the set latitude area range [-70°, 30°] is an empirical value obtained by the applicant through a large number of experiments. Detecting and tracking target objects within the set latitude area range [-70°, 30°] can greatly reduce the amount of calculated data and improve the accuracy of target detection and tracking.

[0085] In one specific implementation manner, the target object is an object with the largest size detected during target detection within the set expansion range.

[0086] It should be noted that the target object is the user itself, and the distance between the user and the camera is the shortest, such as Figure 2 As shown, Figure 2 This is an example diagram of the positional relationship between the user and the camera. Therefore, when target detection is performed within the set expansion range, the largest target detected is the user. The user holds the selfie stick to shoot, and the camera performs target detection within the set expansion range. The largest target detected is the user. Therefore, the user can be quickly detected and tracked, and a special effects plane video from the drone's perspective can be automatically and quickly generated.

[0087] The video frames after target tracking are encoded to generate a planar video about the target object.

[0088] It should be noted that the generated planar video about the target object means that in the planar video, the target object is in the central area of ​​the video frame of the planar video and occupies a set proportion, preferably 70%, so as to highlight the target object and form a special effects planar video from the perspective of the drone.

[0089] Embodiment 2

[0090] Figure 3 A schematic diagram of the module composition of a video processing device provided in the second embodiment of the present application is shown. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0091] A video processing device according to an embodiment of the present application includes:

[0092] An acquisition module, wherein the acquisition module is used to acquire a panoramic video stream to be processed;

[0093] A horizontal correction module, the horizontal correction module is used to obtain the corresponding current original camera posture from the current panoramic video frame of the panoramic video stream, perform horizontal angle correction on the current original camera posture, and obtain a corrected horizontal camera posture;

[0094] A tracking module, the tracking module is used to perform equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and perform target detection in a set expansion range after expansion, detect the target object, and track the target object;

[0095] A generation module is used to encode the video frames after target tracking to generate a planar video about the target object.

[0096] A video processing device according to an embodiment of the present application is used to implement a video processing method provided in any specific implementation manner of the first embodiment of the present application:

[0097] Get the panoramic video stream to be processed;

[0098] Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture;

[0099] Performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting a target object, and tracking the target object;

[0100] The video frames after target tracking are encoded to generate a planar video about the target object.

[0101] It should be noted that each module in the above video processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0102] It should be noted that, for technical details not fully described in this embodiment, reference may be made to the video processing method provided in any specific implementation manner in the above-mentioned embodiment 1, and will not be repeated here.

[0103] Embodiment 3

[0104] In an embodiment of the present application, a computer device is provided. Figure 4 A schematic diagram of the internal structure of a computer device.

[0105] The computer device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the video processing method in any specific implementation manner provided in Embodiment 1 of the present application, specifically including:

[0106] Get the panoramic video stream to be processed;

[0107] Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture;

[0108] Performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting a target object, and tracking the target object;

[0109] The video frames after target tracking are encoded to generate a planar video about the target object.

[0110] It should be noted that the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a video processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0111] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0112] It should be noted that for technical details that are not described in detail in this embodiment, reference can be made to the panoramic video editing method provided in the above-mentioned embodiment 1, and will not be repeated here.

[0113] Embodiment 4

[0114] In an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video processing method in any specific implementation manner provided in the first embodiment of the present application is implemented, specifically including:

[0115] Get the panoramic video stream to be processed;

[0116] Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture;

[0117] Performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting a target object, and tracking the target object;

[0118] The video frames after target tracking are encoded to generate a planar video about the target object.

[0119] The computer-readable storage medium of the embodiments of the present application may include any entity or device, recording medium that can carry computer program code, for example, ROM / RAM, magnetic disk, optical disk, flash memory and other memories.

[0120] It should be noted that, for technical details not fully described in this embodiment, reference may be made to the video processing method provided in any specific implementation manner in the above-mentioned embodiment 1, and will not be described in detail here.

[0121] Different from the prior art, the embodiments of the present application provide a video processing method, apparatus, computer device and storage medium, wherein the video processing method includes:

[0122] Get the panoramic video stream to be processed;

[0123] Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture;

[0124] Performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting a target object, and tracking the target object;

[0125] The video frames after target tracking are encoded to generate a planar video about the target object.

[0126] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0127] By performing horizontal angle correction processing on the panoramic video frame using a set horizontal correction algorithm, a plane video frame with a stable horizontal angle is obtained, and the plane video frame with a horizontal angle is expanded by equidistant cylindrical projection, so that the possible position of the target object can be represented by a latitude range, thereby narrowing the search range of the target tracking algorithm and improving the accuracy of target detection and tracking, thereby automatically generating a special effects plane video from the perspective of a drone follow-up shot, simulating the shooting effect of a drone follow-up shot.

[0128] This application improves the efficiency of producing special effects two-dimensional videos from the perspective of drone tracking. Users can automatically generate special effects two-dimensional videos from the perspective of drone tracking through algorithms without complex shooting techniques and complex post-video processing capabilities. A new way to play panoramic videos is added, which improves user playability.

[0129] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0131] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0132] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Get the panoramic video stream to be processed; Acquire a corresponding current original camera posture from a current panoramic video frame of the panoramic video stream, and perform horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture; Performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection in a set expansion range after expansion, detecting a target object, and tracking the target object; Encoding the video frames after target tracking to generate a planar video of the target object; The performing horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture includes: The Kalman filter is used to track the current original camera posture corresponding to the current panoramic video frame of the panoramic video stream to obtain a first camera rotation matrix R t , according to the first camera rotation matrix R t Rotating the current original camera posture to obtain a smooth camera posture; According to the camera's inertial sensor data, the second camera rotation matrix R is calculated s , the second camera rotation matrix R s is, when the Z axis of the camera coordinate system is in the same direction as the gravity vector in an ideal state, the rotation matrix of the camera coordinate system relative to the inertial coordinate system; According to the first camera rotation matrix R t And the second camera rotation matrix R s , calculate and obtain a horizontal correction matrix R, perform horizontal angle correction on the current original camera posture according to the horizontal correction matrix R, and obtain a corrected horizontal camera posture; The second camera rotation matrix R is calculated based on the inertial sensor data of the camera. s ,include: The inertial sensor data includes accelerometer data, which is the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system. s =[a sx ,a sy ,a sz ] T ; In the ideal state, the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system is s is a0, a0=[0,0,g] T , where g = 9.81 m / s 2 ; Then it satisfies that a s =R s *a0; Calculate the second camera rotation matrix R s for: Wherein, θ is the pitch angle of the camera coordinate system relative to the inertial coordinate system, and φ is the roll angle of the camera coordinate system relative to the inertial coordinate system; The first camera rotation matrix R t And the second camera rotation matrix R s , the horizontal correction matrix R is calculated, including: The first camera rotation matrix R t , the second camera rotation matrix R s , and the horizontal correction matrix R satisfies: R=R s -1 R t ; The step of performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection within a set expansion range after expansion, comprises: Performing equidistant cylindrical projection on the panoramic video frame corresponding to the corrected horizontal camera posture to expand it into a planar rectangular video frame, wherein the planar rectangular video frame has a longitude and latitude coordinate system, wherein the abscissa u of the longitude and latitude coordinate system linearly maps the longitude of the panoramic video frame, and the value range is [-π,π], and the ordinate v of the longitude and latitude coordinate system linearly maps the latitude of the panoramic video frame, and the value range is [-π / 2,π / 2]; Target detection is performed in a set latitude area in the expanded planar rectangular video frame.

2. A video processing method according to claim 1, characterized in that: The set latitude area range is [-70°, 30°].

3. A video processing method according to claim 1 or 2, characterized in that: The target object is an object with the largest size detected during target detection within the set expansion range.

4. A video processing device, characterized in that: The video processing device comprises: An acquisition module, wherein the acquisition module is used to acquire a panoramic video stream to be processed; A horizontal correction module, the horizontal correction module is used to obtain the corresponding current original camera posture from the current panoramic video frame of the panoramic video stream, perform horizontal angle correction on the current original camera posture, and obtain a corrected horizontal camera posture; The performing horizontal angle correction on the current original camera posture to obtain a corrected horizontal camera posture includes: The Kalman filter is used to track the current original camera posture corresponding to the current panoramic video frame of the panoramic video stream to obtain a first camera rotation matrix R t , according to the first camera rotation matrix R t Rotating the current original camera posture to obtain a smooth camera posture; According to the camera's inertial sensor data, the second camera rotation matrix R is calculated s , the second camera rotation matrix R s is, when the Z axis of the camera coordinate system is in the same direction as the gravity vector in an ideal state, the rotation matrix of the camera coordinate system relative to the inertial coordinate system; According to the first camera rotation matrix R t And the second camera rotation matrix R s , calculate and obtain a horizontal correction matrix R, perform horizontal angle correction on the current original camera posture according to the horizontal correction matrix R, and obtain a corrected horizontal camera posture; The second camera rotation matrix R is calculated based on the inertial sensor data of the camera. s ,include: The inertial sensor data includes accelerometer data, which is the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system. s =[a sx ,a sy ,a sz ] T ; In the ideal state, the acceleration a of the three axes of the camera coordinate system relative to the inertial coordinate system is s is a0, a0=[0,0,g] T , where g = 9.81 m / s 2 ; Then it satisfies that a s =R s *a0; Calculate the second camera rotation matrix R s for: Wherein, θ is the pitch angle of the camera coordinate system relative to the inertial coordinate system, and φ is the roll angle of the camera coordinate system relative to the inertial coordinate system; The first camera rotation matrix R t And the second camera rotation matrix R s , the horizontal correction matrix R is calculated, including: The first camera rotation matrix R t , the second camera rotation matrix R s , and the horizontal correction matrix R satisfies: R=R s -1 R t ; A tracking module, the tracking module is used to perform equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and perform target detection in a set expansion range after expansion, detect the target object, and track the target object; The step of performing equirectangular projection expansion on the panoramic video frame corresponding to the corrected horizontal camera posture, and performing target detection within a set expansion range after expansion, comprises: Performing equidistant cylindrical projection on the panoramic video frame corresponding to the corrected horizontal camera posture to expand it into a planar rectangular video frame, wherein the planar rectangular video frame has a longitude and latitude coordinate system, wherein the abscissa u of the longitude and latitude coordinate system linearly maps the longitude of the panoramic video frame, and the value range is [-π,π], and the ordinate v of the longitude and latitude coordinate system linearly maps the latitude of the panoramic video frame, and the value range is [-π / 2,π / 2]; Performing target detection in a set latitude area in the expanded planar rectangular video frame; A generation module is used to encode the video frames after target tracking to generate a planar video about the target object.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the video processing method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 3 are implemented.

Images