Video encoding method, device, electronic device and medium

Abstract

The present application discloses a video encoding method, apparatus, electronic device, and medium, belonging to the field of video processing technology. The method comprises: obtaining relative position information of a first video frame and a second video frame, the first video frame being the video frame to be encoded and the second video frame being the reference video frame; determining, based on the relative position information, a motion vector and residual information of a first pixel block, the first pixel block being the pixel block to be encoded in the first video frame; and encoding the motion vector and residual information.

Description

Technical Field

[0001] The present application belongs to the field of video processing technology, and specifically relates to a video encoding method, device, electronic device and medium. Background Art

[0002] Typically, during video encoding, if you want to encode a pixel block in a video frame, you can first use a global motion search algorithm to search for a matching block that matches the pixel block in the reference video frame, and then determine the motion vector and residual of the pixel block based on the matching block, so that you can encode the pixel block based on the motion vector and residual.

[0003] However, since a matching block matching the pixel block to be encoded needs to be searched in the reference video frame, it may take a long time to search for the matching block, and thus it may take a long time to encode the pixel block to be encoded.

[0004] This results in a longer time consumption for video encoding. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a video encoding method, device, electronic device and medium, which can solve the problem of long time consumption in video encoding.

[0006] In a first aspect, an embodiment of the present application provides a video encoding method, which includes: obtaining relative posture information of a first video frame and a second video frame, the first video frame being a video frame to be encoded, and the second video frame being a reference video frame; determining a motion vector and residual information of a first pixel block based on the relative posture information, the first pixel block being a pixel block to be encoded in the first video frame; and encoding the motion vector and residual information.

[0007] In a second aspect, embodiments of the present application provide a video encoding device comprising: an acquisition module, a determination module, and an encoding module. The acquisition module is configured to acquire relative pose information of a first video frame and a second video frame, where the first video frame is the video frame to be encoded and the second video frame is the reference video frame. The determination module is configured to determine a motion vector and residual information of a first pixel block based on the relative pose information acquired by the acquisition module. The encoding module is configured to encode the motion vector and residual information determined by the determination module.

[0008] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0012] In an embodiment of the present application, the electronic device can obtain the relative posture information of the first video frame and the second video frame, and determine the motion vector and residual information of the first pixel block to be encoded in the first video frame based on the relative posture information, so that the electronic device can encode the motion vector and residual information. Since the electronic device can directly determine the motion vector and residual information of the first pixel block based on the obtained relative posture information, without the need for the electronic device to search for a pixel block matching the first pixel block in the second video frame and then determine the motion vector and residual information of the first pixel block, the time consumed by the electronic device in determining the motion vector and residual information of the first pixel block can be reduced, thereby reducing the time consumed by the electronic device in encoding the first pixel block. In this way, the time consumed by the electronic device in video encoding can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flow charts of the video encoding method provided in the embodiment of the present application;

[0014] Figure 2 This is the second flow chart of the video encoding method provided in the embodiment of the present application;

[0015] Figure 3 This is the third flow chart of the video encoding method provided in the embodiment of the present application;

[0016] Figure 4 This is the fourth flow chart of the video encoding method provided in the embodiment of the present application;

[0017] Figure 5 This is the fifth flow chart of the video encoding method provided in the embodiment of the present application;

[0018] Figure 6 This is the sixth flow chart of the video encoding method provided in the embodiment of the present application;

[0019] Figure 7 This is the seventh flowchart of the video encoding method provided in the embodiment of the present application;

[0020] Figure 8 Schematic diagram of the mapping relationship between the first pixel block and the second pixel block provided in an embodiment of the present application;

[0021] Figure 9 is a structural diagram of a video encoding device provided in an embodiment of the present application;

[0022] Figure 10 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0023] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0025] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0026] The video encoding method, device, electronic device, and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0027] In the related art, during video encoding, if a pixel block in a video frame is to be encoded, a matching block that matches the pixel block can be searched one by one among all pixel blocks in a reference video frame, using the size of the pixel block as a sliding window. The motion vector and residual of the pixel block are then determined based on the matching block, and the pixel block can be encoded based on the motion vector and residual. However, because a matching block that matches the pixel block must be searched one by one among all pixel blocks in the reference video frame, using the size of the pixel block as a sliding window, a large number of pixel blocks must be searched, resulting in a large amount of computation required to find the matching block. This results in a long time required to determine the motion vector and residual of the pixel block, which in turn requires a long time to encode the pixel block. Consequently, video encoding takes a long time.

[0028] In an embodiment of the present application, if a certain pixel block in a certain video frame is to be encoded, the electronic device can directly obtain the relative posture information of the certain video frame and the reference video frame, and determine the motion vector and residual information of the certain pixel block based on the relative posture information, and then encode the motion vector and residual information. It can be understood that the electronic device can directly determine the motion vector and residual information of the certain pixel block based on the relative posture information of the certain video frame and the reference video frame, without the need for the electronic device to use the size of the certain pixel block as a sliding window and search one by one for a matching block that matches the certain pixel block in all pixel blocks of the second video frame. Therefore, the time consumed by the electronic device to determine the motion vector and residual information of the certain pixel block can be reduced, thereby reducing the time consumed by the electronic device to encode the certain pixel block. In this way, the time consumed by the electronic device to perform video encoding can be reduced.

[0029] Figure 1 FIG. 1 shows a flow chart of a video encoding method provided by an embodiment of the present application. Figure 1 As shown, the video encoding method provided in the embodiment of the present application may include the following steps 101 to 103.

[0030] Step 101: The electronic device obtains relative posture information of a first video frame and a second video frame.

[0031] In the embodiment of the present application, the electronic device may be any of the following: a mobile phone, a tablet computer, a laptop computer, a wearable device, an extended reality (XR) device, etc. Among them, the XR device may specifically be an XR head-mounted display device, such as XR glasses, an XR helmet, etc.

[0032] In the embodiment of the present application, the first video frame is a video frame to be encoded, and the second video frame is a reference video frame.

[0033] The first video frame may specifically be a video frame currently being encoded by the electronic device; the second video frame may be a specific video frame, or may be a previous video frame of the first video frame.

[0034] It should be noted that the above “previous video frame of the first video frame” can be understood as: a video frame before the first video frame in the acquired video frame sequence.

[0035] Optionally, in an embodiment of the present application, when an XR application is enabled on an electronic device, the electronic device's camera may be enabled and N video frames may be acquired through the camera, so that the electronic device may encode the N video frames and acquire relative pose information when encoding the first video frame, where N is a positive integer. Specifically, the N video frames may all be XR video frames.

[0036] It can be understood that the first video frame and the second video frame can both be video frames among the N video frames.

[0037] Optionally, in an embodiment of the present application, the electronic device may first obtain one pose information of the first video frame in the first coordinate system and another pose information of the second video frame in the first coordinate system, and then determine the relative pose information based on the one pose information and the other pose information.

[0038] The electronic device may obtain the position information of the first video frame and the position information of the second video frame in the first coordinate system through the following examples.

[0039] In one example, the electronic device may directly perform calculations based on any one of the first video frame and the second video frame to obtain the position information of any one of the video frames in the first coordinate system.

[0040] In another example, the electronic device can calculate the posture information of any video frame in the first coordinate system based on any video frame among the first video frame and the second video frame, and the motion information corresponding to any video frame (i.e., the motion information obtained by the inertial measurement unit (IMU) when obtaining any video frame).

[0041] In another example, the electronic device can calculate the pose information of any video frame in the first coordinate system based on the motion tracking image frame and the motion information corresponding to the motion tracking image frame. It should be noted that the timing of the motion tracking image frame is synchronized with the timing of any video frame between the first video frame and the second video frame. The motion information corresponding to the motion tracking image frame refers to the motion information obtained by the IMU when obtaining the motion tracking image frame.

[0042] Among them, when a user makes a video call through an XR application, such as a holographic video call, a remote expert system, or telemedicine, the electronic device can collect motion tracking image frames through at least one motion tracking camera, such as a fisheye camera, and synchronize the timing of the motion tracking image frame with the timing of any video frame through a timing synchronization module, so that the electronic device can obtain the position information of any video frame in the first coordinate system based on the motion tracking image frame and the motion information corresponding to the motion tracking image frame.

[0043] In another example, the electronic device can directly obtain the posture information of the previous video frame of any one of the first video frame and the second video frame in the first coordinate system, and then calculate the posture information of any one video frame in the first coordinate system based on the posture information of the previous video frame in the first coordinate system and the motion information corresponding to any one video frame. The motion information corresponding to the above-mentioned any one video frame refers to the motion information obtained by the IMU when obtaining any one video frame.

[0044] In an embodiment of the present application, while the electronic device obtains the posture information of any one of the first video frame and the second video frame in the first coordinate system, the electronic device can also obtain the posture information of another video frame (i.e., the video frame other than the any one of the first video frame and the second video frame) through the above-mentioned multiple examples.

[0045] The manner in which the electronic device obtains the position and posture information of any video frame in the first coordinate system may be the same as or different from the manner in which the electronic device obtains the position and posture information of another video frame in the first coordinate system.

[0046] Exemplarily, the electronic device can perform calculations based on the first video frame to obtain the posture information of the first video frame in the first coordinate system; and perform calculations based on the second video frame to obtain the posture information of the second video frame in the first coordinate system; that is, the way in which the electronic device obtains the posture information of the first video frame in the first coordinate system is the same as the way in which the electronic device obtains the posture information of the second video frame in the first coordinate system.

[0047] As another example, the electronic device can perform calculations based on the first video frame to obtain the posture information of the first video frame in the first coordinate system; and calculate the posture information of the second video frame in the first coordinate system based on the second video frame and the motion information corresponding to the second video frame, that is, the way in which the electronic device obtains the posture information of the first video frame in the first coordinate system is different from the way in which the electronic device obtains the posture information of the second video frame in the first coordinate system.

[0048] Step 102: The electronic device determines the motion vector and residual information of the first pixel block according to the relative posture information.

[0049] In the embodiment of the present application, the first pixel block is a pixel block to be encoded in the first video frame.

[0050] It can be understood that the first pixel block may specifically be a pixel block that the electronic device is currently encoding.

[0051] The following will illustrate an example of a process in which the electronic device determines the motion vector and residual information of the first pixel block.

[0052] Optionally, in the embodiment of the present application, Figure 1 ,like Figure 2 As shown, the above step 102 can be specifically implemented through the following steps 102a and 102b.

[0053] Step 102a: The electronic device determines, based on the relative posture information, a second pixel block in the second video frame that matches the first pixel block.

[0054] In an embodiment of the present application, since in the first video frame and the second video frame, any pair of matching feature points (pixel blocks) will be located on each other's epipolar lines, that is, any pair of matching feature points (pixel blocks) satisfies the epipolar constraint, the electronic device can determine the pixel block in the second video frame that matches the first pixel block based on the relative posture information.

[0055] Optionally, in an embodiment of the present application, the electronic device may use the position information of the first pixel block to determine the position information of the second pixel block based on the relative posture information to determine the second pixel block.

[0056] The position information of the first pixel block may be: position information of the first pixel block in the first video frame; the position information of the second pixel block may be: position information of the second pixel block in the second video frame.

[0057] Optionally, in the embodiment of the present application, the above step 102a can be specifically implemented through the following steps 102a1 and 102a2.

[0058] Step 102a1: The electronic device determines mapping parameters according to the relative posture information and the target intrinsic parameter matrix.

[0059] In an embodiment of the present application, the above-mentioned target intrinsic parameter matrix is ​​the intrinsic parameter matrix of the first camera, and the above-mentioned first video frame and second video frame are obtained by the first camera; the above-mentioned mapping parameters are used to indicate the mapping relationship between the pixel block of the first video frame and the pixel block of the second video frame.

[0060] The first camera may be a color (red, green, blue, RGB) camera.

[0061] Further optionally, in an embodiment of the present application, the relative posture information shown may include a translation vector and a rotation matrix, such as the third translation vector and the third rotation matrix in the following embodiment, the translation vector is: the relative translation vector in the first coordinate system when acquiring the first video frame and when acquiring the second video frame, and the rotation matrix is: the relative rotation matrix in the first coordinate system when acquiring the first video frame and when acquiring the second video frame; thereby, the electronic device can use the epipolar constraint equation to calculate the fourth translation vector based on the translation vector, and use the epipolar constraint equation to calculate the fourth rotation matrix based on the rotation matrix to obtain the mapping parameters. Wherein, the first coordinate system can specifically be a world coordinate system.

[0062] It can be understood that the mapping parameters include a fourth translation vector and a fourth rotation matrix.

[0063] It should be noted that for the description of the epipolar constraint equation, reference may be made to the specific description in the relevant technology, which will not be elaborated in detail in the embodiments of the present application.

[0064] Step 102a2: The electronic device calculates the position information of the second pixel block according to the position information of the first pixel block and the mapping parameters.

[0065] Further optionally, the position information of the first pixel block may specifically be: coordinate information of the first pixel block in the first video frame; the position information of the second pixel block may specifically be: coordinate information of the second pixel block in the second video frame.

[0066] Further optionally, in an embodiment of the present application, the electronic device can translate the coordinate information of the first pixel block according to the fourth translation vector, and rotate the coordinate information of the first pixel block according to the fourth rotation matrix to obtain the coordinate information of the second pixel block.

[0067] It can be seen that when the electronic device determines the second pixel block corresponding to the first pixel block, the electronic device can directly determine the mapping parameters based on the relative posture information and the target intrinsic parameter matrix, and use the mapping parameters to directly calculate the position information of the second pixel block based on the position information of the first pixel block, without searching for the second pixel block matching the first pixel block in the second video frame. Therefore, the time spent on searching for the second pixel block can be reduced, and thus the time spent on video encoding by the electronic device can be reduced.

[0068] Step 102b: The electronic device determines the motion vector and residual information of the first pixel block according to the position information of the first pixel block and the second pixel block.

[0069] Further optionally, in an embodiment of the present application, the electronic device may calculate the motion vector of the first pixel block based on the position information of the first pixel block and the position information of the second pixel block.

[0070] Specifically, the electronic device may determine the difference between the coordinate information of the first pixel block and the coordinate information of the second pixel block as the motion vector of the first pixel block.

[0071] In the embodiment of the present application, the above residual information is used to indicate the residual between the first pixel block and the second pixel block.

[0072] Further optionally, in an embodiment of the present application, the above-mentioned residual information may specifically include: luminance residual information and color difference residual information; so that the electronic device can determine the difference between the luminance component of the first pixel block and the luminance component of the second pixel block as luminance residual information; and determine the difference between the color difference component of the first pixel block and the color difference component of the second pixel block as color difference residual information to determine the residual information.

[0073] In an embodiment of the present application, the electronic device can directly determine the second pixel block that matches the first pixel block in the second video frame based on the relative posture information, without searching for the second pixel block that matches the first pixel block in the second video frame. Therefore, the electronic device can quickly determine the motion vector and residual information of the first pixel block based on the position information of the first pixel block and the second pixel block, thereby improving the efficiency of the electronic device in obtaining the motion vector and residual information of the first pixel block.

[0074] Step 103: The electronic device encodes the motion vector and residual information.

[0075] It should be noted that, for the description of how the electronic device encodes the motion vector and residual information, reference may be made to the specific description in the relevant technology, which will not be elaborated in detail in the embodiments of the present application.

[0076] It can be understood that since the brightness and color difference between the first video frame and the second video frame are similar, the residual information may contain more zero values. Therefore, when the electronic device encodes the motion vector and the residual information, the more zero values ​​can be compressed to improve the efficiency of video compression.

[0077] Optionally, in an embodiment of the present application, after the electronic device encodes the motion vector and residual information, the electronic device may transmit the encoded video stream to a receiver of the XR application.

[0078] In an embodiment of the present application, when an electronic device encodes a first pixel block in a first coded video frame, the electronic device can obtain relative posture information and directly determine the motion vector and residual information of the first pixel block based on the relative posture information, so that the electronic device can encode the motion vector and residual information.

[0079] In the video encoding method provided by the embodiment of the present application, the electronic device can obtain the relative posture information of the first video frame and the second video frame, and determine the motion vector and residual information of the first pixel block to be encoded in the first video frame based on the relative posture information, so that the electronic device can encode the motion vector and residual information. Since the electronic device can directly determine the motion vector and residual information of the first pixel block based on the acquired relative posture information, without the need for the electronic device to search for a pixel block matching the first pixel block in the second video frame and then determine the motion vector and residual information of the first pixel block, the time taken by the electronic device to determine the motion vector and residual information of the first pixel block can be reduced, thereby reducing the time taken by the electronic device to encode the first pixel block. In this way, the search time for the matching block can be reduced, and the time taken by the electronic device to perform video encoding can be reduced.

[0080] Furthermore, since the electronic device does not need to search for the second pixel block that matches the first pixel block in the second video frame, the consumption of computing power and electricity caused by the search algorithm can be avoided; and when the user makes a video call through an XR application (such as a holographic video call, a remote expert system, telemedicine, etc.), the time spent by the electronic device for video encoding is reduced, and therefore, the real-time performance of the video call can be improved.

[0081] The following describes in detail how electronic devices obtain relative posture information.

[0082] Optionally, in the embodiment of the present application, the first video frame and the second video frame are acquired by the second camera. Figure 1 ,like Figure 3 As shown, before the above step 101, the video encoding method provided in the embodiment of the present application may further include the following step 201, and the above step 101 may be specifically implemented through the following step 101a.

[0083] Step 201: The electronic device obtains first posture information and second posture information.

[0084] In an embodiment of the present application, the first pose information is the pose information of the first video frame in the first coordinate system; the second pose information is the pose information of the second video frame in the first coordinate system.

[0085] The first coordinate system may specifically be a world coordinate system.

[0086] Further optionally, in the embodiment of the present application, the second camera may be specifically an RGB camera. The second camera may be the same camera as the first camera.

[0087] The following describes in detail how the electronic device obtains the first posture information or the second posture information.

[0088] Optionally, in the embodiment of the present application, Figure 3 ,like Figure 4 As shown, the above step 201 can be specifically implemented through the following step 201a.

[0089] Step 201a: The electronic device calculates and obtains first target posture information based on image information of the first target video frame.

[0090] In the embodiment of the present application, the first target video frame is the first video frame or the second video frame.

[0091] In an embodiment of the present application, when the first target video frame is the first video frame, the first target posture information is the first posture information; when the first target video frame is the second video frame, the first target posture information is the second posture information.

[0092] Further optionally, in an embodiment of the present application, the image information of the first target video frame may specifically include at least one of the following: position information of feature points in the first target video frame, and depth information of the first target video frame.

[0093] Further optionally, in an embodiment of the present application, the electronic device may use a simultaneous localization and mapping (SLAM) algorithm to calculate the first target pose information based on the image information of the first target video frame.

[0094] Since the electronic device can directly calculate the first pose information (or second pose information) based on the image information of the first video frame (or second video frame), so that the electronic device can quickly obtain the relative pose information, the time consumed by the electronic device for video encoding can be further reduced.

[0095] Optionally, in the embodiment of the present application, Figure 3 ,like Figure 5 As shown, the above step 201 can be specifically implemented through the following steps 201b to 201d.

[0096] Step 201b: The electronic device obtains the second target video frame and the first motion information.

[0097] In the embodiment of the present application, the second target video frame is the first video frame or the second video frame.

[0098] In the embodiment of the present application, the above-mentioned first motion information is the motion information obtained by the IMU when the second camera obtains the second target video frame.

[0099] Further optionally, in an embodiment of the present application, the first motion information may be at least one of the following: acceleration information, angular velocity information.

[0100] Further optionally, in an embodiment of the present application, when a user watches an XR image or makes a video call through an XR application, the electronic device can obtain a second target video frame through a second camera and obtain the first motion information through the IMU.

[0101] Step 201c: The electronic device calculates third posture information based on the second target video frame, the first motion information, and the first extrinsic parameter matrix.

[0102] In the embodiment of the present application, the above-mentioned first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU; the above-mentioned third posture information is the posture information of the IMU in the first coordinate system.

[0103] Further optionally, in an embodiment of the present application, a first extrinsic parameter matrix is ​​pre-stored in the electronic device, so that the electronic device can directly obtain the first extrinsic parameter matrix and calculate the third posture information based on the second target video frame, the first motion information and the first extrinsic parameter matrix.

[0104] Further optionally, in an embodiment of the present application, the electronic device may use a SLAM algorithm to calculate the third pose information based on the second target video frame, the first motion information and the first extrinsic parameter matrix.

[0105] Step 201d: The electronic device calculates the second target pose information based on the third pose information and the first extrinsic parameter matrix.

[0106] In an embodiment of the present application, when the second target video frame is the first video frame, the second target posture information is the first posture information; when the second target video frame is the second video frame, the second target posture information is the second posture information.

[0107] It should be noted that, for the description of how the electronic device calculates the second target posture information based on the third posture information and the first external parameter matrix, please refer to the specific description in the relevant technology, and the embodiments of the present application will not be repeated here.

[0108] It can be seen that since the electronic device can obtain the first motion information when obtaining the first video frame or the second video frame, the electronic device can directly calculate the first posture information or the second posture information, so that the electronic device can quickly obtain the relative posture information, thereby further reducing the time spent by the electronic device for video encoding.

[0109] Optionally, in the embodiment of the present application, Figure 3 ,like Figure 6As shown, the above step 201 can be specifically implemented through the following steps 201e to 201g.

[0110] Step 201e: The electronic device obtains the motion tracking image frame and the second motion information.

[0111] In the embodiment of the present application, the second motion information is motion information obtained by the IMU when at least one third camera obtains a motion tracking image frame.

[0112] Further optionally, the at least one third camera may specifically include X third cameras, where X is a positive even number, wherein each third camera may be a motion tracking camera.

[0113] Exemplarily, the X third cameras may be two third cameras, or four third cameras, etc.

[0114] Further optionally, in an embodiment of the present application, the second motion information may be at least one of the following: acceleration information, angular velocity information.

[0115] Further optionally, in an embodiment of the present application, the above-mentioned motion tracking image frame can be a first motion tracking image frame or a second motion tracking image frame; so that when a user watches XR images or makes a video call through an XR application, the electronic device can obtain the first video frame through the second camera, and obtain the first motion tracking image frame through at least one third camera, and obtain the second motion information through the IMU; or, the electronic device can obtain the second video frame through the second camera, and obtain the second motion tracking image frame through at least one third camera, and obtain the second motion information through the IMU.

[0116] Step 201f: The electronic device calculates and obtains fourth posture information based on the motion tracking image frame, the second motion information, and at least one second extrinsic parameter matrix.

[0117] In an embodiment of the present application, the above-mentioned fourth posture information is the posture information of the IMU in the first coordinate system, and the above-mentioned at least one second external parameter matrix is ​​the external parameter matrix between at least one third camera and the IMU.

[0118] Further optionally, in an embodiment of the present application, the electronic device may use a SLAM algorithm to calculate the fourth pose information based on the motion tracking image frame, the second motion information and at least one second extrinsic parameter matrix.

[0119] Step 201g: The electronic device calculates the third target pose information based on the fourth pose information and the first extrinsic parameter matrix.

[0120] In the embodiment of the present application, the above-mentioned first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU.

[0121] In an embodiment of the present application, when the third target video frame is the first video frame, the third target posture information is the first posture information; when the third target video frame is the second video frame, the third target posture information is the second posture information; the above-mentioned motion tracking image frame and the third target video frame are time-synchronized.

[0122] It can be understood that when the third target video frame is the first video frame, the motion tracking image frame is the first motion tracking image frame; when the third target video frame is the second video frame, the motion tracking image frame is the second motion tracking image frame.

[0123] It can be seen that since the electronic device can obtain the second motion information when obtaining the motion tracking image frame that is time-synchronized with the first video frame (or the second video frame), the electronic device can directly calculate the first posture information or the second posture information, so that the electronic device can quickly obtain the relative posture information, thereby further reducing the time spent by the electronic device for video encoding.

[0124] Optionally, in the embodiment of the present application, Figure 3 ,like Figure 7 As shown, the above step 201 can be specifically implemented by the following steps 201h to 201j

[0125] Step 201h: The electronic device obtains second posture information and third motion information.

[0126] In the embodiment of the present application, the third motion information is the motion information obtained by the IMU when the second camera obtains the first video frame.

[0127] Further optionally, in an embodiment of the present application, the third motion information may be at least one of the following: acceleration information, angular velocity information.

[0128] In the embodiment of the present application, the second video frame is the previous video frame of the first video frame.

[0129] When a user watches XR images or makes a video call through an XR application, the electronic device can obtain the first video frame through the second camera and obtain the third motion information through the IMU.

[0130] Step 201i: The electronic device obtains fifth posture information by performing integral calculation based on the second posture information and the third motion information.

[0131] In this embodiment of the present application, the fifth posture information is the posture information of the IMU in the first coordinate system.

[0132] Step 201j: The electronic device calculates the first pose information based on the fifth pose information and the first extrinsic parameter matrix.

[0133] In the embodiment of the present application, the above-mentioned first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU.

[0134] It should be noted that the electronic device can use steps 201h to 201j above to obtain the second pose information. That is, the electronic device can obtain the pose information and the fourth motion information of the previous video frame of the second video frame in the first coordinate system, and integrate the pose information and the fourth motion information of the previous video frame in the first coordinate system to obtain the pose information of the IMU in the first coordinate system. The electronic device can then calculate the second pose information based on the pose information and the first extrinsic parameter matrix.

[0135] It can be seen that since the electronic device can directly obtain the posture information of the second video frame in the first coordinate system, and the third motion information obtained by the IMU when the second camera obtains the first video frame, the electronic device can directly calculate the first posture information or the second posture information, so that the electronic device can quickly obtain the relative posture information, thereby further reducing the time spent by the electronic device for video encoding.

[0136] Step 101a: The electronic device determines relative posture information based on the first posture information and the second posture information.

[0137] Further optionally, in an embodiment of the present application, the electronic device may use a preset algorithm to determine relative posture information based on the first posture information and the second posture information.

[0138] It can be seen that since the electronic device can directly determine the relative posture information based on the first posture information and the second posture information, the accuracy of obtaining the relative posture information can be improved. In this way, the accuracy of the electronic device in determining the second pixel block corresponding to the first pixel block can be improved.

[0139] The following will take the preset algorithms including the first preset algorithm and the second preset algorithm as an example for illustration.

[0140] Optionally, in an embodiment of the present application, the first pose information includes a first translation vector and a first rotation matrix, wherein the first translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the first video frame, and the first rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the first video frame; the second pose information includes a second translation vector and a second rotation matrix, wherein the second translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the second video frame, and the second rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the second video frame. Specifically, step 101a can be implemented by steps 101a1 to 101a3 described below.

[0141] Step 101a1: The electronic device determines a third rotation matrix according to the first rotation matrix and the second rotation matrix.

[0142] Further optionally, in an embodiment of the present application, the first coordinate axis may specifically be an X-axis, or a Y-axis, or a Z-axis.

[0143] Further optionally, in an embodiment of the present application, the electronic device may adopt a first preset algorithm to determine a third rotation matrix according to the first rotation matrix and the second rotation matrix.

[0144] Specifically, the first preset algorithm can be: R=R1 -1 R2;

[0145] Among them, R is the third rotation matrix, R1 is the first rotation matrix, and R2 is the second rotation matrix.

[0146] Step 101a2: The electronic device calculates a third translation vector according to the first translation vector, the second translation vector, and the first rotation matrix.

[0147] Further optionally, in an embodiment of the present application, the electronic device may adopt a second preset algorithm to determine the third translation vector based on the first translation vector, the second translation vector and the first rotation matrix.

[0148] Specifically, the second preset algorithm can be: t=R1 -1 (t2-t1);

[0149] Wherein, t is the third translation vector, t1 is the first translation vector, t2 is the second translation vector, and R2 is the first rotation matrix.

[0150] In an embodiment of the present application, the above-mentioned relative posture information includes a third rotation matrix and a third translation vector.

[0151] Thus, it can be seen that the electronic device can accurately determine the third translation vector and the third rotation matrix based on the first translation vector, the first rotation matrix, the second translation vector and the second rotation matrix to improve the accuracy of determining the relative posture information.

[0152] It should be noted that the way in which the electronic device obtains the first posture information (or the second posture information) may be the same as or different from the way in which the electronic device obtains the second posture information (or the first posture information).

[0153] Exemplarily, the electronic device may obtain the first posture information in the manner of Example 1, and obtain the second posture information in the manner of Example 1, that is, the manner in which the electronic device obtains the first posture information is the same as the manner in which it obtains the second posture information.

[0154] As another example, the electronic device may obtain the first posture information using the method of Example 1 and obtain the second posture information using the method of Example 3, that is, the way the electronic device obtains the first posture information is different from the way it obtains the second posture information.

[0155] The following will use two different scenarios to specifically illustrate the video encoding method provided in the embodiments of the present application.

[0156] Scenario 1: Users watch XR images through XR applications.

[0157] When a user watches XR images through an XR application, the electronic device can turn on the camera of the electronic device (for example, the first camera or the second camera) to collect N video frames, and obtain N motion information through the IMU. The N motion information corresponds to the N video frames one by one. Then, when encoding the first video frame among the N video frames, the electronic device can read the first extrinsic parameter matrix between the camera (that is, the first camera or the second camera) and the IMU, and calculate the first position information of the first video frame in the first coordinate system based on the first video frame, the motion information corresponding to the first video frame, and the first extrinsic parameter matrix, and calculate the second position information of the second video frame in the first coordinate system based on the second video frame (that is, the previous video frame of the first video frame in the N video frames), the motion information corresponding to the second video frame, and the first extrinsic parameter matrix. Thus, the electronic device can determine the relative posture information of the first video frame and the second video frame based on the first posture information and the second posture information; at this time, the electronic device can read the target intrinsic parameter matrix of the electronic device's camera (i.e., the first camera or the second camera), and determine the mapping parameters based on the relative posture information and the target intrinsic parameter matrix, and the mapping parameters are used to indicate the mapping relationship between the pixel block of the first video frame and the pixel block of the second video frame, and then the electronic device can calculate the position information of the second pixel block based on the position information of the first pixel block and the mapping parameters, and determine the motion vector and residual information of the first pixel block based on the position information of the first pixel block and the second pixel block, and then the electronic device can encode the motion vector and residual information.

[0158] For example, Figure 8 As shown, assuming that the first pixel block is a 4*4 pixel block, the position information of the first pixel block in the first video frame (for example, I1) is (x, y), so that the electronic device can calculate the position information of the second pixel block in the second video frame (for example, I2) based on the position information of the first pixel block and the mapping parameter. The position information of the second pixel block in I2 is (x ’ ,y ’ ).

[0159] Scenario 2: Users make video calls through XR applications.

[0160] When a user makes a video call through an XR application, the electronic device can turn on the camera of the electronic device (for example, the first camera or the second camera) to collect N video frames, and turn on at least one third camera of the electronic device to collect N motion tracking image frames respectively, and obtain N motion information through the IMU, and the N motion information and the N motion tracking image frames correspond one to one. In this way, the electronic device can first synchronize the timing of each video frame and the timing of each motion tracking image frame, and then read the extrinsic parameter matrix between the at least one third camera and the IMU. In this way, the electronic device can determine the first pose information of the first video frame in the first coordinate system based on the first motion tracking image frame synchronized with the timing of the first video frame, the motion information corresponding to the first motion tracking image frame, and at least one second extrinsic parameter matrix, and determine the second pose information of the second video frame in the first coordinate system based on the second motion tracking image frame synchronized with the timing of the second video frame, the motion information corresponding to the second motion tracking image frame, and at least one second extrinsic parameter matrix. Thus, the electronic device can determine the relative pose information of the first video frame and the second video frame based on the first pose information and the second pose information; at this time, the electronic device can read the target intrinsic parameter matrix of the electronic device's camera (i.e., the first camera or the second camera), and determine the mapping parameters based on the relative pose information and the target intrinsic parameter matrix, and the mapping parameters are used to indicate the mapping relationship between the pixel block of the first video frame and the pixel block of the second video frame. Then, the electronic device can calculate the position information of the second pixel block based on the position information of the first pixel block and the mapping parameters, and determine the motion vector and residual information of the first pixel block based on the position information of the first pixel block and the second pixel block. Then, the electronic device can encode the motion vector and residual information, and transmit the encoded video stream to the receiver of the XR application.

[0161] The video encoding method provided in the embodiment of the present application can be executed by a video encoding device. In the embodiment of the present application, the video encoding method is executed by a video encoding device as an example to illustrate the video encoding device provided in the embodiment of the present application.

[0162] Figure 9 FIG. 1 shows a possible structural diagram of a video encoding device involved in an embodiment of the present application. Figure 9 As shown, the video encoding device 60 may include: an acquisition module 61 , a determination module 62 and an encoding module 63 .

[0163] The acquisition module 61 is configured to obtain relative pose information of a first video frame and a second video frame, where the first video frame is the video frame to be encoded and the second video frame is the reference video frame. The determination module 62 is configured to determine the motion vector and residual information of the first pixel block based on the relative pose information obtained by the acquisition module 61. The encoding module 63 is configured to encode the motion vector and residual information determined by the determination module 62.

[0164] In one possible implementation, the above-mentioned determination module 62 is specifically used to determine the second pixel block that matches the first pixel block in the second video frame based on the relative posture information; and determine the motion vector and residual information of the first pixel block based on the position information of the first pixel block and the second pixel block.

[0165] In one possible implementation, the above-mentioned determination module 62 is specifically used to determine the mapping parameters based on the relative posture information and the target intrinsic parameter matrix, where the target intrinsic parameter matrix is ​​the intrinsic parameter matrix of the first camera, and the first video frame and the second video frame are obtained by the first camera; the mapping parameters are used to indicate the mapping relationship between the pixel block of the first video frame and the pixel block of the second video frame; and the position information of the second pixel block is calculated based on the position information of the first pixel block and the mapping parameters.

[0166] In one possible implementation, the first and second video frames are acquired by a second camera. The acquisition module 61 is specifically configured to acquire first pose information and second pose information, where the first pose information is pose information of the first video frame in a first coordinate system, and the second pose information is pose information of the second video frame in the first coordinate system. The determination module 62 is specifically configured to determine relative pose information based on the first and second pose information acquired by the acquisition module 61.

[0167] In one possible implementation, the determination module 62 is further configured to calculate first target pose information based on image information of the first target video frame. When the first target video frame is the first video frame, the first target pose information is the first pose information; and when the first target video frame is the second video frame, the first target pose information is the second pose information.

[0168] In one possible implementation, the acquisition module 61 is further configured to acquire a second target video frame and first motion information, where the first motion information is the motion information acquired by the IMU when the second camera acquires the target video frame. The determination module 62 is further configured to calculate third pose information based on the second target video frame, the first motion information, and the first extrinsic parameter matrix acquired by the acquisition module 61, where the third pose information is the pose information of the IMU in the first coordinate system, and the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU; and to calculate second target pose information based on the third pose information and the first extrinsic parameter matrix. When the second target video frame is the first video frame, the second target pose information is the first pose information; and when the second target video frame is the second video frame, the second target pose information is the second pose information.

[0169] In one possible implementation, the acquisition module 61 is further configured to acquire motion tracking image frames and second motion information, where the second motion information is motion information acquired by the IMU when at least one third camera acquires the motion tracking image frames. The determination module 62 is further configured to calculate fourth pose information based on the motion tracking image frames acquired by the acquisition module 61, the second motion information, and at least one second extrinsic parameter matrix, where the fourth pose information is the pose information of the IMU in the first coordinate system, and the at least one second extrinsic parameter matrix is ​​the extrinsic parameter matrix between the at least one third camera and the IMU; and to calculate third target pose information based on the fourth pose information and the first extrinsic parameter matrix, where the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU. When the third target video frame is the first video frame, the third target pose information is the first pose information; when the third target video frame is the second video frame, the third target pose information is the second pose information; and the motion tracking image frame and the third target video frame are time-synchronized.

[0170] In one possible implementation, the acquisition module 61 is further configured to acquire second pose information and third motion information, where the third motion information is motion information acquired by the IMU when the second camera acquires the first video frame. The determination module 62 is further configured to integrate the second pose information and the third motion information acquired by the acquisition module 61 to obtain fifth pose information, where the fifth pose information is the pose information of the IMU in the first coordinate system; and to calculate first pose information based on the fifth pose information and a first extrinsic parameter matrix, where the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU.

[0171] In one possible implementation, the first pose information includes a first translation vector and a first rotation matrix, and the second pose information includes a second translation vector and a second rotation matrix. The determination module 62 is specifically configured to calculate a third rotation matrix based on the first rotation matrix and the second rotation matrix; and calculate a third translation vector based on the first translation vector, the second translation vector, and the first rotation matrix. The relative pose information includes a third rotation matrix and a third translation vector; the first translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the first video frame, and the first rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the first video frame; the second translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the second video frame, and the second rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the second video frame.

[0172] The video encoding device provided in the embodiment of the present application can directly determine the motion vector and residual information of the first pixel block based on the acquired relative posture information, without the need for the video encoding device to search for a pixel block matching the first pixel block in the second video frame and then determine the motion vector and residual information of the first pixel block. Therefore, the time spent by the video encoding device in determining the motion vector and residual information of the first pixel block can be reduced, thereby reducing the time spent by the video encoding device in encoding the first pixel block. In this way, the time spent by the video encoding device in performing video encoding can be reduced.

[0173] The video encoding device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0174] The video encoding device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0175] The video encoding device provided in the embodiment of the present application can achieve Figures 1 to 8 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0176] Optionally, in the embodiment of the present application, Figure 10 As shown, an embodiment of the present application also provides an electronic device 70, including a processor 71 and a memory 72, wherein the memory 72 stores a program or instruction that can be run on the processor 71. When the program or instruction is executed by the processor 71, the various process steps of the above-mentioned video encoding method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0177] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0178] Figure 11 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0179] The electronic device 1100 includes but is not limited to components such as a radio frequency unit 1101 , a network module 1102 , an audio output unit 1103 , an input unit 1104 , a sensor 1105 , a display unit 1106 , a user input unit 1107 , an interface unit 1108 , a memory 1109 , and a processor 1110 .

[0180] Those skilled in the art will understand that the electronic device 1100 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0181] Among them, processor 1110 is used to obtain relative posture information of a first video frame and a second video frame, where the first video frame is a video frame to be encoded and the second video frame is a reference video frame; determine the motion vector and residual information of a first pixel block based on the relative posture information, where the first pixel block is a pixel block to be encoded in the first video frame; and encode the motion vector and residual information.

[0182] The electronic device provided in the embodiment of the present application can directly determine the motion vector and residual information of the first pixel block based on the acquired relative posture information, without the need for the electronic device to search for a pixel block matching the first pixel block in the second video frame and then determine the motion vector and residual information of the first pixel block. Therefore, the time taken by the electronic device to determine the motion vector and residual information of the first pixel block can be reduced, thereby reducing the time taken by the electronic device to encode the first pixel block. In this way, the time taken by the electronic device to perform video encoding can be reduced.

[0183] Optionally, in an embodiment of the present application, the processor 1110 is specifically used to determine a second pixel block that matches the first pixel block in the second video frame based on relative posture information; and determine the motion vector and residual information of the first pixel block based on position information of the first pixel block and the second pixel block.

[0184] It can be seen that since the electronic device can directly determine the second pixel block that matches the first pixel block in the second video frame based on the relative posture information, without having to search for the second pixel block that matches the first pixel block in the second video frame, the electronic device can quickly determine the motion vector and residual information of the first pixel block based on the position information of the first pixel block and the second pixel block, thereby improving the efficiency of the electronic device in obtaining the motion vector and residual information of the first pixel block.

[0185] Optionally, in an embodiment of the present application, the processor 1110 is specifically used to determine mapping parameters based on relative posture information and a target intrinsic parameter matrix, where the target intrinsic parameter matrix is ​​the intrinsic parameter matrix of the first camera, and the first video frame and the second video frame are acquired by the first camera; the mapping parameters are used to indicate a mapping relationship between a pixel block of the first video frame and a pixel block of the second video frame; and the position information of the second pixel block is calculated based on the position information of the first pixel block and the mapping parameters.

[0186] It can be seen that when the electronic device determines the second pixel block corresponding to the first pixel block, the electronic device can directly determine the mapping parameters based on the relative posture information and the target intrinsic parameter matrix, and use the mapping parameters to directly calculate the position information of the second pixel block based on the position information of the first pixel block, without searching for the second pixel block matching the first pixel block in the second video frame. Therefore, the time spent on searching for the second pixel block can be reduced, and thus the time spent on video encoding by the electronic device can be reduced.

[0187] Optionally, in an embodiment of the present application, the first video frame and the second video frame are acquired by a second camera.

[0188] The processor 1110 is further configured to obtain first pose information and second pose information, where the first pose information is pose information of the first video frame in the first coordinate system, and the second pose information is pose information of the second video frame in the first coordinate system.

[0189] The processor 1110 is specifically configured to determine relative posture information based on the first posture information and the second posture information.

[0190] It can be seen that since the electronic device can directly determine the relative posture information based on the first posture information and the second posture information, the accuracy of obtaining the relative posture information can be improved. In this way, the accuracy of the electronic device in determining the second pixel block corresponding to the first pixel block can be improved.

[0191] Optionally, in an embodiment of the present application, the processor 1110 is further configured to calculate the first target posture information based on the image information of the first target video frame.

[0192] Wherein, when the first target video frame is the first video frame, the first target posture information is the first posture information; when the first target video frame is the second video frame, the first target posture information is the second posture information.

[0193] It can be seen that since the electronic device can directly calculate the first pose information (or second pose information) based on the image information of the first video frame (or second video frame), so that the electronic device can quickly obtain the relative pose information, the time consumed by the electronic device for video encoding can be further reduced.

[0194] Optionally, in an embodiment of the present application, the processor 1110 is also used to obtain a second target video frame and first motion information, where the first motion information is the motion information obtained by the IMU when the second camera obtains the target video frame; and based on the second target video frame, the first motion information and the first extrinsic parameter matrix, calculate the third pose information, where the third pose information is the pose information of the IMU in the first coordinate system, and the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU; and, based on the third pose information and the first extrinsic parameter matrix, calculate the second target pose information.

[0195] Among them, when the second target video frame is the first video frame, the second target posture information is the first posture information; when the second target video frame is the second video frame, the second target posture information is the second posture information.

[0196] It can be seen that since the electronic device can obtain the first motion information when obtaining the first video frame or the second video frame, the electronic device can directly calculate the first posture information or the second posture information, so that the electronic device can quickly obtain the relative posture information, thereby further reducing the time spent by the electronic device for video encoding.

[0197] Optionally, in an embodiment of the present application, the processor 1110 is further used to obtain motion tracking image frames and second motion information, where the second motion information is the motion information obtained by the IMU when at least one third camera obtains the motion tracking image frame; fourth pose information is calculated based on the motion tracking image frame, the second motion information and at least one second extrinsic parameter matrix, where the fourth pose information is the pose information of the IMU in the first coordinate system, and the at least one second extrinsic parameter matrix is ​​the extrinsic parameter matrix between at least one third camera and the IMU; third target pose information is calculated based on the fourth pose information and the first extrinsic parameter matrix, where the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU.

[0198] Among them, when the third target video frame is the first video frame, the third target posture information is the first posture information; when the third target video frame is the second video frame, the third target posture information is the second posture information; the motion tracking image frame and the third target video frame are time-synchronized.

[0199] It can be seen that since the electronic device can obtain the second motion information when obtaining the motion tracking image frame that is time-synchronized with the first video frame (or the second video frame), the electronic device can directly calculate the first posture information or the second posture information, so that the electronic device can quickly obtain the relative posture information, thereby further reducing the time spent by the electronic device for video encoding.

[0200] Optionally, in an embodiment of the present application, the processor 1110 is also used to obtain second posture information and third motion information, where the third motion information is the motion information obtained by the IMU when the second camera obtains the first video frame; and based on the second posture information and the third motion information, fifth posture information is obtained by integral calculation, where the fifth posture information is the posture information of the IMU in the first coordinate system; and, based on the fifth posture information and the first extrinsic parameter matrix, the first posture information is calculated, where the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU.

[0201] It can be seen that since the electronic device can directly obtain the posture information of the second video frame in the first coordinate system, and the third motion information obtained by the IMU when the second camera obtains the first video frame, the electronic device can directly calculate the first posture information or the second posture information, so that the electronic device can quickly obtain the relative posture information, thereby further reducing the time spent by the electronic device for video encoding.

[0202] Optionally, in an embodiment of the present application, the first pose information includes a first translation vector and a first rotation matrix, and the second pose information includes a second translation vector and a second rotation matrix.

[0203] The processor 1110 is specifically configured to calculate a third rotation matrix based on the first rotation matrix and the second rotation matrix; and calculate a third translation vector based on the first translation vector, the second translation vector, and the first rotation matrix.

[0204] Among them, the above-mentioned relative posture information includes a third rotation matrix and a third translation vector; the above-mentioned first translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the first video frame, and the above-mentioned first rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the first video frame; the above-mentioned second translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the second video frame, and the above-mentioned second rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the second video frame.

[0205] Thus, it can be seen that the electronic device can accurately determine the third translation vector and the third rotation matrix based on the first translation vector, the first rotation matrix, the second translation vector and the second rotation matrix to improve the accuracy of determining the relative posture information.

[0206] It should be understood that in an embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0207] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0208] Processor 1110 may include one or more processing units. Optionally, processor 1110 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1110.

[0209] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned video encoding method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0210] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0211] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video encoding method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0212] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0213] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video encoding method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0214] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0215] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0216] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A video encoding method, characterized in that: The method comprises: Obtaining relative pose information of a first video frame and a second video frame, where the first video frame is a video frame to be encoded and the second video frame is a reference video frame; Determining a motion vector and residual information of a first pixel block according to the relative posture information, where the first pixel block is a pixel block to be encoded in the first video frame; encoding the motion vector and the residual information; The determining, according to the relative posture information, the motion vector and residual information of the first pixel block includes: Determining, according to the relative posture information, a second pixel block in the second video frame that matches the first pixel block; The motion vector and residual information of the first pixel block are determined according to the position information of the first pixel block and the second pixel block.

2. The method according to claim 1, characterized in that The determining, according to the relative posture information, a second pixel block in the second video frame that matches the first pixel block includes: Determining mapping parameters based on the relative pose information and a target intrinsic parameter matrix, where the target intrinsic parameter matrix is ​​an intrinsic parameter matrix of a first camera, and the first video frame and the second video frame are acquired by the first camera; the mapping parameters are used to indicate a mapping relationship between a pixel block of the first video frame and a pixel block of the second video frame; The position information of the second pixel block is calculated based on the position information of the first pixel block and the mapping parameter.

3. The method according to claim 1, characterized in that The first video frame and the second video frame are acquired by a second camera, and acquiring relative pose information of the first video frame and the second video frame includes: Obtaining first pose information and second pose information, wherein the first pose information is pose information of the first video frame in a first coordinate system, and the second pose information is pose information of the second video frame in the first coordinate system; The obtaining of relative pose information of the first video frame and the second video frame includes: The relative posture information is determined according to the first posture information and the second posture information.

4. The method according to claim 3, characterized in that The method further comprises: Calculating first target pose information based on image information of the first target video frame; Wherein, in a case where the first target video frame is the first video frame, the first target posture information is the first posture information; When the first target video frame is the second video frame, the first target posture information is the second posture information.

5. The method according to claim 3, characterized in that The method further comprises: Acquire a second target video frame and first motion information, where the first motion information is motion information acquired by an inertial measurement unit (IMU) when the second camera acquires the target video frame; Calculate third pose information according to the second target video frame, the first motion information, and a first extrinsic parameter matrix, where the third pose information is pose information of the IMU in the first coordinate system, and the first extrinsic parameter matrix is ​​an extrinsic parameter matrix between the second camera and the IMU; Calculating second target pose information according to the third pose information and the first extrinsic parameter matrix; Wherein, when the second target video frame is the first video frame, the second target posture information is the first posture information; When the second target video frame is the second video frame, the second target posture information is the second posture information.

6. The method according to claim 3, characterized in that The method further comprises: Acquire a motion tracking image frame and second motion information, where the second motion information is motion information acquired by the IMU when at least one third camera acquires the motion tracking image frame; Calculate fourth pose information based on the motion tracking image frame, the second motion information, and at least one second extrinsic parameter matrix, where the fourth pose information is pose information of the IMU in the first coordinate system, and the at least one second extrinsic parameter matrix is ​​an extrinsic parameter matrix between the at least one third camera and the IMU; Calculating third target pose information according to the fourth pose information and a first extrinsic parameter matrix, where the first extrinsic parameter matrix is ​​an extrinsic parameter matrix between the second camera and the IMU; Wherein, when the third target video frame is the first video frame, the third target posture information is the first posture information; When the third target video frame is the second video frame, the third target posture information is the second posture information; The motion tracking image frame and the third target video frame are time-synchronized.

7. The method according to claim 3, characterized in that The method further comprises: Acquire second position information and third motion information, where the third motion information is motion information acquired by the IMU when the second camera acquires the first video frame; Integrate and calculate the fifth pose information based on the second pose information and the third motion information, where the fifth pose information is the pose information of the IMU in the first coordinate system; The first pose information is calculated based on the fifth pose information and a first extrinsic parameter matrix, where the first extrinsic parameter matrix is ​​an extrinsic parameter matrix between the second camera and the IMU.

8. The method according to claim 3, characterized in that The first posture information includes a first translation vector and a first rotation matrix, and the second posture information includes a second translation vector and a second rotation matrix; The determining the relative posture information according to the first posture information and the second posture information includes: Calculating a third rotation matrix based on the first rotation matrix and the second rotation matrix; Calculating a third translation vector according to the first translation vector, the second translation vector and the first rotation matrix; Wherein, the relative posture information includes the third rotation matrix and the third translation vector; The first translation vector is a translation vector of the second camera relative to the origin of the first coordinate system when acquiring the first video frame, and the first rotation matrix is ​​a rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the first video frame; The second translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the second video frame, and the second rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the second video frame.

9. A video encoding device, characterized in that: The video encoding device includes: an acquisition module, a determination module and an encoding module; The acquisition module is configured to acquire relative posture information of a first video frame and a second video frame, wherein the first video frame is a video frame to be encoded and the second video frame is a reference video frame; The determining module is configured to determine the motion vector and residual information of the first pixel block according to the relative posture information acquired by the acquiring module; The encoding module is configured to encode the motion vector and the residual information determined by the determination module; The determination module is specifically used to determine a second pixel block in the second video frame that matches the first pixel block based on the relative posture information; and determine the motion vector and residual information of the first pixel block based on the position information of the first pixel block and the second pixel block.

10. The video encoding device according to claim 9, wherein: The determining module is specifically configured to determine mapping parameters based on the relative pose information and a target intrinsic parameter matrix, where the target intrinsic parameter matrix is ​​an intrinsic parameter matrix of a first camera, and the first video frame and the second video frame are acquired by the first camera; The mapping parameter is used to indicate a mapping relationship between the first video frame pixel block and the second video frame pixel block; and the position information of the second pixel block is calculated based on the position information of the first pixel block and the mapping parameter.

11. The video encoding device according to claim 9, wherein The first video frame and the second video frame are acquired by a second camera; The acquisition module is specifically configured to acquire first pose information and second pose information, wherein the first pose information is pose information of the first video frame in a first coordinate system, and the second pose information is pose information of the second video frame in the first coordinate system; The determining module is specifically configured to determine the relative posture information based on the first posture information and the second posture information acquired by the acquiring module.

12. The video encoding device according to claim 11, wherein: The determining module is further configured to calculate first target pose information based on image information of the first target video frame; Wherein, in a case where the first target video frame is the first video frame, the first target posture information is the first posture information; When the first target video frame is the second video frame, the first target posture information is the second posture information.

13. The video encoding device according to claim 11, wherein: The acquisition module is further configured to acquire a second target video frame and first motion information, where the first motion information is motion information acquired by an inertial measurement unit (IMU) when the second camera acquires the target video frame; The determining module is further configured to calculate third pose information obtained by the acquiring module based on the second target video frame, the first motion information, and the first extrinsic parameter matrix, wherein the third pose information is the pose information of the IMU in the first coordinate system, and the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU; and calculating second target pose information based on the third pose information and the first extrinsic parameter matrix; Wherein, when the second target video frame is the first video frame, the second target posture information is the first posture information; When the second target video frame is the second video frame, the second target posture information is the second posture information.

14. The video encoding device according to claim 11, wherein: The acquisition module is further configured to acquire a motion tracking image frame and second motion information, where the second motion information is motion information acquired by the IMU when at least one third camera acquires the motion tracking image frame; The determination module is further configured to calculate fourth pose information based on the motion tracking image frame acquired by the acquisition module, the second motion information, and at least one second extrinsic parameter matrix, where the fourth pose information is the pose information of the IMU in the first coordinate system, and the at least one second extrinsic parameter matrix is ​​the extrinsic parameter matrix between the at least one third camera and the IMU; and calculate third target pose information based on the fourth pose information and the first extrinsic parameter matrix, where the first extrinsic parameter matrix is ​​the extrinsic parameter matrix between the second camera and the IMU; Wherein, when the third target video frame is the first video frame, the third target posture information is the first posture information; When the third target video frame is the second video frame, the third target posture information is the second posture information; The motion tracking image frame and the third target video frame are time-synchronized.

15. The video encoding device according to claim 11, wherein The acquisition module is further configured to acquire second posture information and third motion information, wherein the third motion information is motion information acquired by the IMU when the second camera acquires the first video frame; The determining module is further configured to obtain fifth pose information by performing integral calculation based on the second pose information and the third motion information obtained by the obtaining module, where the fifth pose information is the pose information of the IMU in the first coordinate system; The first pose information is calculated based on the fifth pose information and the first extrinsic parameter matrix, where the first extrinsic parameter matrix is ​​an extrinsic parameter matrix between the second camera and the IMU.

16. The video encoding device according to claim 11, wherein The first posture information includes a first translation vector and a first rotation matrix, and the second posture information includes a second translation vector and a second rotation matrix; The determining module is specifically configured to calculate a third rotation matrix based on the first rotation matrix and the second rotation matrix; and calculate a third translation vector based on the first translation vector, the second translation vector, and the first rotation matrix; Wherein, the relative posture information includes the third rotation matrix and the third translation vector; The first translation vector is a translation vector of the second camera relative to the origin of the first coordinate system when acquiring the first video frame, and the first rotation matrix is ​​a rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the first video frame; The second translation vector is the translation vector of the second camera relative to the origin of the first coordinate system when acquiring the second video frame, and the second rotation matrix is ​​the rotation matrix of the second camera relative to the first coordinate axis of the first coordinate system when acquiring the second video frame.

17. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the video encoding method according to any one of claims 1 to 8 are implemented.

18. A readable storage medium, characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the video encoding method according to any one of claims 1 to 8 are implemented.

Images