Methods for object implantation in multi-degree-of-freedom video, electronic devices and storage media

Abstract

This application provides a method, electronic device, and storage medium for object implantation in multi-degree-of-freedom (DOF) video. The method includes: obtaining the coordinates of a real camera matrix in a real-world coordinate system, wherein the real camera matrix is ​​used to capture multi-DOF video; constructing a virtual coordinate system based on the coordinates of the real camera matrix in the real-world coordinate system, wherein the virtual coordinate system is the coordinate system corresponding to the virtual camera matrix in virtual space; mapping the trajectory information of the real camera matrix in the real-world coordinate system to the trajectory information of the virtual camera matrix in the virtual coordinate system; and associating the trajectory information of the virtual camera matrix in the virtual coordinate system with a virtual object to be implanted in the multi-DOF video to obtain implantation information for implanting the multi-DOF video. This application provides a foundation for accurate matching between virtual objects and multi-DOF video.

Description

Technical Field

[0001] This application relates to the field of video technology, specifically to a method for object implantation in multi-degree-of-freedom video, an electronic device, and a storage medium. Background Technology

[0002] Multi-degree-of-freedom (MDO) video is a type of immersive video that allows users to adjust their viewing angle, thus providing an immersive viewing experience. During rendering, MDO video can incorporate virtual objects such as AR (Augmented Reality) elements. However, due to the variable perspective of MDO video, traditional techniques for embedding virtual objects in video often struggle to accurately match these perspective changes, frequently resulting in mismatches between the embedded virtual objects and the video frame. Therefore, providing a solution for object embedding in MDO video that lays the foundation for accurate matching between virtual objects and the video has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method for object implantation into multi-degree-of-freedom video, an electronic device, and a storage medium, so as to provide a basis for accurate matching of virtual objects and multi-degree-of-freedom video.

[0004] To achieve the above objectives, the embodiments of this application provide the following technical solutions.

[0005] In a first aspect, embodiments of this application provide a method for object implantation in multi-degree-of-freedom video, including:

[0006] Obtain the coordinates of the real camera matrix in the real space coordinate system, wherein the real camera matrix is ​​used to capture multi-degree-of-freedom video;

[0007] Based on the coordinates of the real camera matrix in the real space coordinate system, a virtual space coordinate system is constructed, which is the coordinate system corresponding to the virtual camera matrix in the virtual space;

[0008] The trajectory information of the real camera matrix in the real space coordinate system is mapped to the trajectory information of the virtual camera matrix in the virtual space coordinate system;

[0009] The trajectory information of the virtual camera matrix in the virtual space coordinate system is associated with the virtual object into which the multi-degree-of-freedom video is implanted to obtain the implantation information for implanting the multi-degree-of-freedom video.

[0010] In a second aspect, embodiments of this application provide an electronic device, including: at least one memory and at least one processor; the memory stores one or more computer-executable instructions, and the processor invokes the one or more computer-executable instructions to execute the object implantation method for multi-degree-of-freedom video as described in the first aspect above.

[0011] Thirdly, embodiments of this application provide a storage medium that stores one or more computer-executable instructions, which, when executed, implement the object implantation method for multi-degree-of-freedom video as described in the first aspect above.

[0012] This application embodiment can construct a virtual space coordinate system mapped to the real space coordinate system based on the coordinates of the real camera matrix in the real space coordinate system, thereby reproducing the shooting space of the real camera matrix in the virtual space. This maps the trajectory information of the real camera matrix in the real space coordinate system to the trajectory information of the virtual camera matrix in the virtual space coordinate system, achieving the reproduction of the trajectory of the real camera matrix in the virtual space and matching the movement of the real camera matrix with the movement of the virtual camera matrix. Furthermore, the trajectory information of the virtual camera matrix in the virtual space coordinate system is associated with a virtual object embedded in the multi-degree-of-freedom video to obtain embedding information for embedding in the multi-degree-of-freedom video. Since the embedding information carries the trajectory information of the virtual object and the virtual camera matrix in the virtual space coordinate system, when the virtual object is embedded into the multi-degree-of-freedom video, the virtual object embedded in the multi-degree-of-freedom video can accurately match each shooting angle of the real camera matrix through the associated trajectory information of the virtual camera matrix in the virtual space coordinate system, achieving precise matching between the virtual object and the multi-degree-of-freedom video. This application embodiment constructs a virtual spatial coordinate system that maps to the real spatial coordinate system where the real camera matrix is ​​located, and reproduces the trajectory of the real camera matrix in the virtual spatial coordinate system. This enables virtual objects to accurately match the various shooting angles of the real camera matrix based on the camera trajectory reproduced in the virtual spatial coordinate system, providing a foundation for the accurate matching of virtual objects with multi-degree-of-freedom videos and making it possible for virtual objects to accurately match the changes in the viewing angle of multi-degree-of-freedom videos. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1A This is a schematic diagram of the setup for a real camera matrix.

[0015] Figure 1B Example image for embedding AR elements in multi-degree-of-freedom video.

[0016] Figure 2A This is a schematic diagram illustrating the execution phase of the object implantation method provided in the embodiments of this application.

[0017] Figure 2B This is a schematic diagram illustrating the mapping between the real space coordinate system and the virtual space coordinate system provided in the embodiments of this application.

[0018] Figure 3A A flowchart of the object implantation method provided in the embodiments of this application.

[0019] Figure 3B A flowchart illustrating the method for constructing a virtual spatial coordinate system provided in this application embodiment.

[0020] Figure 3C This is an example diagram of the translation and rotation matrix of a real camera in a real-world coordinate system.

[0021] Figure 4 This is a flowchart illustrating a method for associating trajectory information of a virtual camera matrix with a virtual object, as provided in an embodiment of this application.

[0022] Figure 5 A flowchart illustrating the method for embedding multi-degree-of-freedom video into a virtual object according to an embodiment of this application.

[0023] Figure 6 An example diagram illustrating the process of embedding AR elements into a 6DOF video as provided in the embodiments of this application.

[0024] Figure 7 A block diagram of an object implantation device for multi-degree-of-freedom video provided in an embodiment of this application.

[0025] Figure 8 This is a block diagram of an electronic device. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] The multi-degree-of-freedom (DOF) video referred to in this application includes, for example, three-degree-of-freedom (3CoF) video and six-degree-of-freedom (6DoF) video. Three-DOF video has translational degrees of freedom along the X, Y, and Z axes (also described as yaw, pitch, and roll), meaning it allows users to adjust their viewing angle by translating along the X, Y, and Z axes. A typical example of three-DOF video is panoramic video. Six-DOF video, in addition to having translational degrees of freedom along the X, Y, and Z axes, also has rotational degrees of freedom around the X, Y, and Z axes. In other words, six-DOF video allows users to adjust their viewing angle not only by translating along the X, Y, and Z axes but also by rotating around the X, Y, and Z axes.

[0028] Multi-degree-of-freedom video can be captured using a matrix of real cameras formed by multiple real cameras. For ease of explanation, let's take filming a basketball game as an example. Figure 1A An exemplary schematic diagram of a real camera matrix setup is shown. Figure 1A As shown, the photographer can set up n real cameras C1 to Cn along a certain path in the shooting space of a basketball game (the specific value of n can be determined according to the actual situation). Figure 1A In this diagram, Cm represents the middle real camera among n real cameras, and Cn-1 represents the real camera preceding the nth real camera Cn. These n real cameras, C1 to Cn, can be set along an arc. Real cameras C1 to Cn can capture video from different angles, and by combining the parameters of C1 to Cn, multi-degree-of-freedom video can be obtained. For example, the texture maps captured by each real camera C1 to Cn, the parameters of each real camera (including intrinsic and extrinsic parameters), and the depth maps of each real camera can form multi-angle video data. Based on this multi-angle video data, virtual viewpoints of the real camera matrix can be generated, thus providing a multi-degree-of-freedom video experience. Since the real camera matrix includes multiple real cameras, the virtual viewpoints of the real cameras can include the virtual viewpoints of each real camera.

[0029] It should be noted that a virtual viewpoint is a three-dimensional concept representing the shooting angle of a real camera, which can be represented using multi-degree-of-freedom coordinates. Taking six degrees of freedom as an example, a virtual viewpoint can be represented using six-degree-of-freedom coordinates, where the spatial position of the virtual viewpoint can be represented as (x, y, z), and the viewing angle can be represented as three rotational directions.

[0030] When rendering and displaying multi-degree-of-freedom videos, there is often a need to incorporate virtual objects such as AR elements to enrich the video's content or increase its appeal. For ease of explanation, let's take... Figure 1A Taking the example of shooting a basketball game using a real camera matrix, AR elements (such as textures) that express the game score can be embedded into the multi-degree-of-freedom video. Figure 1B An example diagram illustrating the embedding of AR elements into multi-degree-of-freedom video is shown, such as... Figure 1BAs shown, "3:2" AR elements (such as...) can be embedded into the footage captured by multi-degree-of-freedom video. Figure 1B (As shown in the virtual frame), to display rich and multi-degree-of-freedom video content.

[0031] Multi-degree-of-freedom (MDOF) videos have the characteristic of changing perspectives. If AR elements and other virtual objects are implanted into fixed positions in MDOF videos using traditional special effects methods, it will be difficult for the implanted virtual objects to match the changing perspectives of the MDOF video. This will result in the implanted virtual objects not aligning with the MDOF video's image, causing visual defects for the user when watching the MDOF video.

[0032] To address the aforementioned issues, this application provides a novel object implantation scheme for multi-degree-of-freedom videos. By constructing a virtual spatial coordinate system that maps to the real-world coordinate system of the real camera matrix, and reproducing the trajectory of the real camera matrix within the virtual spatial coordinate system, the virtual object can accurately match the various shooting positions and angles of the real camera matrix based on the camera trajectory reproduced in the virtual spatial coordinate system, thus providing a foundation for the accurate matching of virtual objects with multi-degree-of-freedom videos.

[0033] In some embodiments, Figure 2A An exemplary schematic diagram illustrates the execution phase of the object implantation method for multi-degree-of-freedom video provided in an embodiment of this application. For example... Figure 2A As shown, the execution stages provided in this application embodiment may include: virtual space coordinate system construction stage 211, trajectory reconstruction stage 212, and virtual object association stage 213.

[0034] In the virtual space coordinate system construction stage 211, embodiments of this application can construct a virtual space coordinate system that maps to the real space coordinate system where the real camera matrix resides. It should be noted that the real space coordinate system is the coordinate system of the real camera matrix, such as the coordinate system corresponding to the shooting space of the real camera matrix; the virtual space coordinate system is the coordinate system corresponding to the virtual camera matrix. A real camera matrix can have a corresponding virtual camera matrix in the virtual space; for example, one real camera corresponds to one virtual camera in the virtual space, and multiple virtual cameras form a virtual camera matrix. In some embodiments, the real camera can be used for video shooting, and the virtual camera can be used for image rendering.

[0035] For ease of explanation, Figure 2B An exemplary diagram illustrating the mapping between a real-world coordinate system and a virtual coordinate system is shown. For example... Figure 2BAs shown, the real-world coordinate system x'y'z'o' (where 0 represents the origin) can be mapped to the constructed virtual coordinate system x'y'z'o', thereby reproducing the shooting space of the real camera matrix in the virtual coordinate system x'y'z'o'. In some embodiments, the virtual coordinate system can be constructed based on the coordinates of the real camera matrix in the real-world coordinate system.

[0036] In the trajectory reconstruction stage 212, embodiments of this application can reconstruct the trajectory of a virtual camera matrix corresponding to the trajectory of the real camera matrix in a virtual space coordinate system to obtain the trajectory information of the virtual camera matrix in multi-degree-of-freedom video. In some embodiments, the trajectory of the real camera matrix represents the trajectory information of the real camera matrix in each frame of multi-degree-of-freedom video (e.g., in a six-degree-of-freedom video, the trajectory information corresponding to each virtual viewpoint in each frame of the six-degree-of-freedom video, where one virtual viewpoint corresponds to the shooting angle of one real camera). Based on the construction of the virtual space coordinate system, embodiments of this application can map the trajectory information of the virtual camera matrix in each frame of multi-degree-of-freedom video in the virtual space coordinate system based on the trajectory information of the real camera matrix in each frame of multi-degree-of-freedom video in the real space coordinate system, thereby realizing the corresponding mapping between the trajectory of the virtual camera matrix in the virtual space coordinate system and the trajectory of the real camera matrix in the real space coordinate system.

[0037] Based on the virtual space coordinate system construction stage 211 and the trajectory reconstruction stage 212, this embodiment of the application can construct a virtual space coordinate system that maps to the real space coordinate system, and determine the trajectory information of the virtual camera matrix in the multi-degree-of-freedom video in the virtual space coordinate system, so that the trajectory information of the virtual camera matrix in the multi-degree-of-freedom video corresponds to the trajectory information of the real camera matrix in the multi-degree-of-freedom video, thereby realizing the reproduction of the trajectory of the real camera matrix in the virtual space coordinate system, and achieving the matching of the motion of the real camera matrix with the motion of the virtual camera matrix.

[0038] In the virtual object association stage 213, embodiments of this application can associate the trajectory information of the virtual camera matrix in the multi-degree-of-freedom video with the virtual object embedded in the multi-degree-of-freedom video, thereby providing embedding information for embedding in the multi-degree-of-freedom video. Based on the correspondence between the trajectory information of the virtual camera matrix in the multi-degree-of-freedom video and the trajectory information of the real camera matrix, by associating the virtual object embedded in the multi-degree-of-freedom video with the trajectory information of the virtual camera matrix, embodiments of this application can enable the virtual object to accurately match each shooting angle of the real camera matrix, providing a foundation for accurate matching between the virtual object and the multi-degree-of-freedom video.

[0039] As Figure 2A Optional implementations of the execution phase shown. Figure 3AAn exemplary flowchart of a method for object implantation in multi-degree-of-freedom video provided in an embodiment of this application is illustrated. This method flowchart can be implemented by an electronic device with data processing capabilities, such as a server. The server can be a server that processes video captured by a real camera matrix to obtain multi-degree-of-freedom video, or it can be a standalone server. (Refer to...) Figure 3A The method process may include the following steps.

[0040] In step S310, the coordinates of the real camera matrix in the real space coordinate system are obtained. The real camera matrix is ​​used to capture multi-degree-of-freedom video.

[0041] This application embodiment can set up a real camera matrix for shooting multi-degree-of-freedom video, based on the characteristics of the shooting location. For example, a six-degree-of-freedom real camera matrix can be set up for shooting six-degree-of-freedom video. The set up real camera matrix can have corresponding coordinates in the real space coordinate system. Since the real camera matrix includes multiple real cameras, the coordinates of the real camera matrix in the real space coordinate system can include the coordinates of the multiple real cameras in the real space coordinate system, which is used to express the distribution position of the multiple real cameras in the real space coordinate system.

[0042] Based on a real camera matrix set up at the shooting location, this application embodiment can mark the real camera matrix on-site and transmit the trajectory and parameters of the real camera matrix after the test is passed. In some embodiments, the coordinates of the real camera matrix can be carried in the parameters of the real camera matrix. This application embodiment can obtain the coordinates of the real camera matrix in the real space coordinate system by reading the parameters of the real camera matrix.

[0043] The purpose of camera calibration is to determine the camera's parameter values. The parameters that need to be calibrated can be divided into intrinsic and extrinsic parameters. In some embodiments, the parameters of the real camera matrix may include both extrinsic and intrinsic parameters. Extrinsic parameters represent the camera's parameters in the real-world coordinate system, such as its position and rotation direction. Intrinsic parameters represent parameters related to the camera's inherent characteristics, such as its focal length and pixel size. Alternatively, embodiments of this application can read the extrinsic parameters of the real camera matrix to obtain the coordinates of the real camera matrix in the real-world coordinate system, for example, obtaining the coordinates of each real camera in the real-world coordinate system from the extrinsic parameters of each real camera.

[0044] In step S320, a virtual space coordinate system is constructed based on the coordinates of the real camera matrix in the real space coordinate system. The virtual space coordinate system is the coordinate system corresponding to the virtual camera matrix in the virtual space.

[0045] After obtaining the coordinates of the real camera matrix in the real space coordinate system, embodiments of this application can construct a virtual space coordinate system that has a mapping relationship with the real space coordinate system, thereby realizing the shooting space reproduction of the real camera matrix in the virtual space. This virtual space can be configured with a virtual camera matrix corresponding to the real camera matrix, and the virtual camera matrix can be used for image rendering. In some embodiments, the virtual camera matrix may include multiple virtual cameras, with one virtual camera corresponding to one real camera.

[0046] In some embodiments, both the real space coordinate system and the virtual space coordinate system can be three-dimensional coordinate systems. The real space coordinate system can be represented as xyzo, and the virtual space coordinate system can be represented as x'y'z'o'. In this application embodiment, the origin o', x' axis, y' axis and z' axis of the virtual space coordinate system can be constructed based on the coordinates of multiple real cameras in the real space coordinate system, so as to realize the construction of the virtual space coordinate system.

[0047] As an optional implementation, combined Figure 1A As shown, there are n real cameras C1 to Cn in the real camera matrix, and Cm is the middle real camera among the n real cameras (e.g., the middle real camera among the n real cameras). In this embodiment, the origin o' and the x-axis basis vector (denoted by x') of the virtual space coordinate system can be constructed based on the coordinates of the first real camera C1 and the last real camera Cn in the real camera matrix; the z-axis basis vector (denoted by z') of the virtual space coordinate system can be constructed based on the coordinates of the middle real camera Cm, the first real camera C1 and the last real camera Cn; the y-axis basis vector (denoted by y') of the virtual space coordinate system can be constructed based on the x-axis basis vector (x') and the z-axis basis vector (z'); thus, the virtual space coordinate system x'y'z'o' is formed based on the origin o', the x-axis basis vector (x'), the y-axis basis vector (y') and the z-axis basis vector (z').

[0048] In some embodiments, the virtual space coordinate system constructed in this application may have a mapping relationship with the real space coordinate system. For example, based on the real space coordinate system and the virtual space coordinate system, this application can determine the mapping relationship between coordinates in the real space coordinate system and the virtual space coordinate system. In some embodiments, this mapping relationship can be represented by a mapping matrix. For example, coordinates in the real space coordinate system can be converted into coordinates in the virtual space coordinate system based on this mapping matrix. It should be noted that after constructing the virtual space coordinate system based on the coordinates of the real camera matrix in the real space coordinate system, this application determines the mapping relationship (e.g., mapping matrix) between the real space coordinate system and the virtual space coordinate system based on the constructed virtual space coordinate system.

[0049] In step S330, the trajectory information of the virtual camera matrix in the virtual space coordinate system is determined based on the trajectory information of the real camera matrix in the real space coordinate system.

[0050] The trajectory information of a real camera matrix in the real-world coordinate system can represent the trajectory of the real camera matrix during multi-degree-of-freedom video capture in the real-world coordinate system. For example, it can represent the trajectory of the real camera matrix during each frame of multi-degree-of-freedom video capture in the real-world coordinate system. Since the real camera matrix comprises multiple real cameras, its trajectory information in the real-world coordinate system can include the individual trajectory information of each real camera in the real-world coordinate system.

[0051] After constructing a virtual spatial coordinate system, this embodiment of the application, based on the mapping relationship between the virtual spatial coordinate system and the real spatial coordinate system, can map the trajectory information of the real camera matrix in the real spatial coordinate system to the trajectory information of the virtual camera matrix in the virtual spatial coordinate system. This allows the trajectory of the real camera matrix in the real spatial coordinate system to be reproduced using the trajectory of the virtual camera matrix in the virtual spatial coordinate system. Alternatively, this embodiment can map the trajectory information of each frame of multi-degree-of-freedom video of the real camera matrix in the real spatial coordinate system to the trajectory information of each frame of video of the virtual camera matrix in the virtual spatial coordinate system. For example, if the real camera matrix includes multiple real cameras, this embodiment can map the trajectory information of each real camera in each frame of multi-degree-of-freedom video to the trajectory information of the corresponding virtual camera in each frame of multi-degree-of-freedom video in the virtual spatial coordinate system.

[0052] In some further embodiments, the trajectory information of the real camera matrix in a frame of multi-degree-of-freedom video can be expressed by the translation and rotation matrices of the real camera matrix at each virtual viewpoint in the frame of multi-degree-of-freedom video. In this embodiment, the translation and rotation matrices of the real camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video can be mapped to the translation and rotation matrices of the virtual camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video, thereby obtaining the trajectory information of the virtual camera matrix in each frame of video in the virtual space coordinate system. Alternatively, based on the shooting angle of a real camera corresponding to a virtual viewpoint, this embodiment can map the translation and rotation matrices of each real camera at the virtual viewpoint in each frame of multi-degree-of-freedom video to the corresponding translation and rotation matrices of the virtual camera matrix at the virtual viewpoint in each frame of multi-degree-of-freedom video.

[0053] In step S340, the trajectory information of the virtual camera matrix in the virtual space coordinate system is associated with the virtual object into which the multi-degree-of-freedom video is implanted, so as to obtain the implantation information for implanting the multi-degree-of-freedom video.

[0054] In some embodiments, this application can associate the trajectory information of each frame of video in the virtual space coordinate system with the virtual object implanted in each frame of multi-degree-of-freedom video to obtain implantation information for each frame of multi-degree-of-freedom video. Furthermore, based on the implantation information of each frame of multi-degree-of-freedom video, when this application embeds a virtual object into each frame of multi-degree-of-freedom video, the virtual object can achieve precise matching with the multi-degree-of-freedom video based on the associated trajectory information of the virtual camera matrix. It is understood that after mapping the trajectory information of multiple real cameras in each frame of multi-degree-of-freedom video to the trajectory information of corresponding virtual cameras in each frame of multi-degree-of-freedom video in the virtual space coordinate system, the trajectory information of each virtual camera can correspond to the trajectory information of each real camera. Therefore, when a virtual object is implanted into each frame of multi-degree-of-freedom video, the virtual object can correspond to the trajectory information of each real camera through the associated trajectory information of each virtual camera. Thus, when the viewpoint changes in each frame of multi-degree-of-freedom video, this application can enable the virtual object to adjust accordingly based on the viewpoint change in each frame of multi-degree-of-freedom video.

[0055] This application embodiment can construct a virtual space coordinate system mapped to the real space coordinate system based on the coordinates of the real camera matrix in the real space coordinate system, thereby reproducing the shooting space of the real camera matrix in the virtual space. This maps the trajectory information of the real camera matrix in the real space coordinate system to the trajectory information of the virtual camera matrix in the virtual space coordinate system, achieving the reproduction of the trajectory of the real camera matrix in the virtual space and matching the movement of the real camera matrix with the movement of the virtual camera matrix. Furthermore, the trajectory information of the virtual camera matrix in the virtual space coordinate system is associated with a virtual object embedded in the multi-degree-of-freedom video to obtain embedding information for embedding in the multi-degree-of-freedom video. Since the embedding information carries the trajectory information of the virtual object and the virtual camera matrix in the virtual space coordinate system, when the virtual object is embedded into the multi-degree-of-freedom video, the virtual object embedded in the multi-degree-of-freedom video can accurately match each shooting angle of the real camera matrix through the associated trajectory information of the virtual camera matrix in the virtual space coordinate system, achieving precise matching between the virtual object and the multi-degree-of-freedom video. This application embodiment constructs a virtual spatial coordinate system that maps to the real spatial coordinate system where the real camera matrix is ​​located, and reproduces the trajectory of the real camera matrix in the virtual spatial coordinate system. This enables virtual objects to accurately match the various shooting angles of the real camera matrix based on the camera trajectory reproduced in the virtual spatial coordinate system, providing a foundation for the accurate matching of virtual objects with multi-degree-of-freedom videos and making it possible for virtual objects to accurately match the changes in the viewing angle of multi-degree-of-freedom videos.

[0056] After labeling the real camera matrix and obtaining its coordinates in the real-world coordinate system using its parameters (e.g., extrinsic parameters), embodiments of this application can construct a virtual coordinate system to reproduce the shooting space of the real camera matrix in virtual space. Figure 3B An exemplary flowchart illustrating a method for constructing a virtual spatial coordinate system provided in an embodiment of this application is shown. Figure 3B As shown, the method flow may include the following steps.

[0057] In step S321, the center of the line connecting the coordinates of the first real camera and the last real camera is determined as the origin of the virtual space coordinate system.

[0058] The real camera matrix includes multiple real cameras. Based on the coordinates of the first and last real cameras in the real-world coordinate system, embodiments of this application can determine the line connecting the coordinates of the first and last real cameras, and the midpoint of this line is determined as the origin of the virtual-space coordinate system. In one implementation example, combined with... Figure 1A As shown, in this embodiment of the application, the midpoint of the line connecting the coordinates of the first real camera C1 and the last real camera Cn can be used as the origin o' of the virtual space coordinate system.

[0059] It should be noted that determining the origin of the virtual space coordinate system by using the center of the line connecting the coordinates of the first and last real cameras is only one possible implementation. The center of the line connecting the coordinates of the first and last real cameras can be considered as the center of the entire shooting scene. In this embodiment, the origin of the virtual space coordinate system can also be determined by the center of the line connecting the coordinates of other real cameras, as long as the center of the line connecting the coordinates of other real cameras corresponds to or is close to the center of the entire shooting scene. In other possible implementations, this embodiment can also choose the center of the line connecting the coordinates of the second and second-to-last real cameras as the origin of the virtual space coordinate system.

[0060] In step S322, the x-axis basis vector of the virtual space coordinate system is determined based on the vector regularization result from the coordinates of the first real camera to the coordinates of the last real camera.

[0061] Based on the coordinates of the first and last real cameras in the real-world coordinate system, embodiments of this application can determine the vector regularization result from the coordinates of the first real camera to the coordinates of the last real camera, thereby using it as the x-axis basis vector of the virtual-space coordinate system. In one implementation example, combined with... Figure 1AAs shown, in this embodiment of the application, the vector from the coordinates of the first real camera C1 to the coordinates of the last real camera Cn can be regularized, and the result is used as the x-axis basis vector (x') of the virtual space coordinate system.

[0062] In step S323, the cross product of the coordinate line vectors connecting the intermediate real camera with the first real camera and the last real camera is performed, and the regularization of the cross product result is used as the z-axis basis vector of the virtual space coordinate system.

[0063] Based on the coordinates of the intermediate real camera and the first real camera in the real space coordinate system, embodiments of this application can determine the coordinate connection vector between the intermediate real camera and the first real camera; based on the coordinates of the intermediate real camera and the last real camera in the real space coordinate system, embodiments of this application can determine the coordinate connection vector between the intermediate real camera and the last real camera; then, the coordinate connection vector between the intermediate real camera and the first real camera, and the coordinate connection vector between the intermediate real camera and the last real camera are cross-producted, and the cross-product result is regularized; the regularized result can then be used as the z-axis basis vector of the virtual space coordinate system. In one implementation example, combined with Figure 1A As shown, in this embodiment of the application, the coordinate connection vectors of the intermediate real camera Cm and the first real camera C1 and the last real camera Cn can be cross-producted, and the regularized result of the cross-product is used as the z-axis basis vector (z') of the virtual space coordinate system.

[0064] In step S324, the x-axis basis vector and z-axis basis vector of the virtual space coordinate system are cross-producted to obtain the y-axis basis vector of the virtual space coordinate system.

[0065] In this embodiment of the application, after obtaining the x-axis basis vector and z-axis basis vector of the virtual space coordinate system, the two can be cross-multiplied to obtain the y-axis basis vector of the virtual space coordinate system.

[0066] In step S325, a virtual spatial coordinate system is constructed based on the origin, x-axis basis vector, y-axis basis vector, and z-axis basis vector of the virtual spatial coordinate system.

[0067] This application embodiment can construct a virtual space coordinate system that maps to the real space coordinate system based on the coordinates of the real camera matrix in the real space coordinate system, thereby reproducing the shooting space of the real camera matrix in the virtual space and providing a foundation for subsequent trajectory reconstruction of the real camera matrix in the virtual space.

[0068] In some embodiments, the trajectory information of the real camera matrix in each frame of multi-DOF video can be expressed by the translation and rotation matrices corresponding to each virtual viewpoint of the real camera matrix in each frame of multi-DOF video. Since the real camera matrix includes multiple real cameras, each virtual viewpoint of the real camera matrix in a frame of multi-DOF video can correspond to the shooting angle of each real camera; for example, one virtual viewpoint corresponds to the shooting angle of one real camera. It should be noted that the translation and rotation matrix corresponding to the virtual viewpoint of the camera in a frame of video represents the rotation and translation required for the content of the real world captured by the camera in that frame to fall onto the camera coordinates.

[0069] This embodiment of the application can, in a virtual space coordinate system, map the translation and rotation matrices of the real camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video to the translation and rotation matrices of the virtual camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video, based on the mapping relationship (e.g., a mapping matrix) between the real space coordinate system and the virtual space coordinate system, thereby obtaining the trajectory information of the virtual camera matrix at each frame of multi-degree-of-freedom video. Alternatively, this embodiment of the application can multiply the mapping matrix by the translation and rotation matrices of the real camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video to obtain the translation and rotation matrices of the virtual camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video. For example, if the real camera matrix includes multiple real cameras, this embodiment of the application can multiply the mapping matrix by the translation and rotation matrices corresponding to the virtual viewpoints of each real camera in each frame of multi-degree-of-freedom video to obtain the translation and rotation matrices corresponding to the virtual viewpoints of each virtual camera in each frame of multi-degree-of-freedom video.

[0070] In one implementation example Figure 3C An example diagram illustrating the translation and rotation matrix of a real camera in a real-world coordinate system is shown. Figure 3C As shown, after setting up real cameras C1 to Cn, in a frame of multi-degree-of-freedom video, the translation and rotation matrices corresponding to the virtual viewpoints of real cameras C1 to Cn are rt1 to rtb, respectively. Based on the mapping matrix M that maps the real space coordinate system xyzo to the virtual space coordinate system x'y'z'o', this embodiment of the application can multiply the translation and rotation matrices rt1 to rtn corresponding to the virtual viewpoints of real cameras C1 to Cn in a frame of multi-degree-of-freedom video by the mapping matrix M, thereby obtaining the translation and rotation matrices corresponding to the virtual viewpoints of each virtual camera in a frame of multi-degree-of-freedom video.

[0071] After obtaining the trajectory information of the virtual camera matrix in the virtual space coordinate system (e.g., the translation and rotation matrix corresponding to the virtual viewpoint of each virtual camera in each frame of multi-degree-of-freedom video), embodiments of this application can associate the trajectory information of the virtual camera matrix in the virtual space coordinate system with the virtual object to be embedded in the multi-degree-of-freedom video to obtain the embedding information. In some embodiments, embodiments of this application can create virtual objects such as AR elements, and associate at least one channel information of the virtual object with the trajectory information of the virtual camera matrix in the virtual space coordinate system to realize the association between the trajectory information of the virtual camera matrix and the virtual object. As an optional implementation, Figure 4 An exemplary flowchart illustrates a method for associating trajectory information of a virtual camera matrix with virtual objects, as provided in an embodiment of this application. Figure 4 As shown, the method flow may include the following steps.

[0072] In step S410, at least one channel information of the virtual object is extracted, and the at least one channel information is aggregated into a channel information sequence of the virtual object.

[0073] In some embodiments, at least one channel information of the virtual object includes at least one of RGB (red, green, blue) channel information, Depth channel information, and Alpha (transparency) channel information. Embodiments of this application can extract at least one channel information of the virtual object by extracting at least one of the RGB, Depth, and Alpha channel information. Optionally, the RGB channel information of the virtual object can be a background image of the virtual object containing color information; the Depth channel information can be the linearized depth information of the virtual object in virtual space, which corresponds to the depth captured by a real camera in real space; the Alpha channel information is the transparency information of the virtual object in virtual space, and the transparency of virtual objects of various materials can be extracted using an inverse blending algorithm.

[0074] In some embodiments, this application can render the image of a virtual object onto an RGBA-supporting texture and sample the RGB and Alpha channel information of the virtual object; the Depth channel information of the virtual object can be obtained by acquiring the depth texture of a real camera matrix and calculating the linear depth. In further embodiments, the virtual object may include AR elements, such as AR models / panels, AR particles / effects, AR elements with non-transparent / semi-transparent materials, UI (user interface design) textures, etc. When calculating the Depth channel information of the virtual object, for AR particles, since AR particles usually have no depth information, an additional rendering pass can be added when rendering AR particles to fill the depth information in the form of opaque panels, while the color drawing of this pass is masked to avoid affecting the acquisition of color and transparency information.

[0075] After obtaining at least one of the RGB channel information, Depth channel information, and Alpha channel information of the virtual object, embodiments of this application can aggregate the at least one channel information into a channel information sequence, which may include the at least one channel information. In an optional implementation, embodiments of this application can aggregate the RGB channel information, Depth channel information, and Alpha channel information of the virtual object into a multi-channel information sequence; of course, in other possible implementations, embodiments of this application may also render only the RGB channel information of the virtual object, such as using only the RGB channel information of the virtual object as a channel information sequence. In this case, since depth-of-field fusion in three-dimensional space is not performed, the realism will be somewhat lost.

[0076] In step S420, the channel information sequence of the virtual object is associated with the trajectory information of the virtual camera matrix in the virtual space coordinate system to obtain the implantation information.

[0077] After obtaining the channel information sequence of the virtual object, embodiments of this application can associate the channel information sequence of the virtual object with the trajectory information of the virtual camera matrix in the virtual space coordinate system. For example, the channel information sequence of the virtual object to be implanted in each frame of multi-degree-of-freedom video is associated with the trajectory information of the virtual camera matrix in each frame of multi-degree-of-freedom video to obtain the implantation information for each frame of multi-degree-of-freedom video. In some embodiments, the implantation information may be a data packet structure or a data sequence structure, used to represent the association between the trajectory information of the virtual camera matrix in the virtual space coordinate system and the channel information sequence of the virtual object.

[0078] After obtaining the embedding information for the multi-degree-of-freedom video, embodiments of this application can embed virtual objects into the multi-degree-of-freedom video, for example, embedding virtual objects into each frame of the multi-degree-of-freedom video. As an optional implementation, Figure 5An exemplary flowchart of a method for embedding virtual objects into multi-degree-of-freedom video according to an embodiment of this application is shown. Figure 5 As shown, the method flow may include the following steps.

[0079] In step S510, the intrinsic parameters of the real camera matrix are synchronized with the virtual camera matrix.

[0080] Implanting virtual objects into multi-degree-of-freedom (DOF) video can be considered as the process of rendering virtual objects within the video. Based on the rendering capabilities of a virtual camera matrix, to ensure that the rendered result visually closely matches the real-world image of the multi-DOF video, embodiments of this application can synchronize the intrinsic parameters of the virtual camera matrix with those of the real camera matrix. For example, embodiments of this application can read the intrinsic parameters of the real camera matrix, thereby synchronizing the focal length and distortion parameters of the real camera matrix with those of the virtual camera matrix.

[0081] In step S520, the virtual objects in the implanted information are rendered into a multi-degree-of-freedom video using the intrinsic parameters of the virtual camera matrix synchronization. The virtual objects rendered into the multi-degree-of-freedom video are associated with the trajectory information of the virtual camera matrix in the virtual space coordinate system carried in the implanted information.

[0082] After synchronizing the intrinsic parameters of the virtual camera matrix, based on the virtual object to be implanted carried in the implantation information, this embodiment of the application can use the intrinsic parameters of the virtual camera matrix synchronization to render the virtual object into a multi-degree-of-freedom video, and make the rendered virtual object associated with the trajectory information of the virtual camera matrix in the virtual space coordinate system carried in the implantation information, so that after the virtual object is rendered into the multi-degree-of-freedom video, it can be adjusted based on the perspective changes of the multi-degree-of-freedom video.

[0083] As an optional implementation, if the implanted information carries a channel information sequence of the virtual object, and the channel information sequence includes RGB channel information, Depth channel information, and Alpha channel information, then the embodiments of this application can use the intrinsic parameters of the virtual camera matrix synchronization to perform depth occlusion and culling of RGB channel information in the multi-degree-of-freedom video based on the Depth channel information and Alpha channel information of the virtual object, so that the RGB channel information and the multi-degree-of-freedom video produce a realistic fusion, thereby realizing the rendering of the virtual object into the multi-degree-of-freedom video.

[0084] As an optional implementation, the fusion formula between the channel information sequence of the virtual object and the multi-degree-of-freedom video can be as follows:

[0085] Color = [Depth] ar / Depth img ]*RGB img+[Depth img / Depth ar ]*(RGB ar +(1-Alpha ar )*RGB img ); where Color represents the fusion result of the virtual object's channel information sequence and the multi-degree-of-freedom video, and Depth ar This represents the depth information of the rendered virtual object. img This represents the depth information of the raw footage from multi-degree-of-freedom video captured by a real camera matrix, in RGB format. ar RGB represents the color information of the rendered virtual object. img Alpha represents the original color information of multi-degree-of-freedom video captured by a real camera matrix. ar This represents the alpha channel information (transparency information) of the rendered virtual object.

[0086] In other words, the embodiments of this application can use the integer division result (0 or 1) between the depth of the virtual object and the depth of the multi-degree-of-freedom video as the basis for elimination to determine whether to use the original image of the multi-degree-of-freedom video or the fusion result of the multi-degree-of-freedom video and the virtual object.

[0087] In one implementation example, virtual objects are used as AR elements. Figure 6 An exemplary diagram illustrating the process of embedding AR elements into a 6DoF video according to an embodiment of this application is shown. Figure 6 As shown, embedding AR elements into a 6DoF video can include the following process.

[0088] A real camera matrix can be set up, for example, a six-degree-of-freedom camera matrix and related servers can be set up according to the characteristics of the shooting location.

[0089] The trajectory and parameter calibration of the real camera matrix are performed (62). For example, the real camera matrix is ​​calibrated on-site, and the trajectory and intrinsic and extrinsic parameter information of the real camera matrix are transmitted back after the test is passed.

[0090] Spatial and Trajectory Reconstruction 63: A reconstruction algorithm is used to perform 3D reconstruction of the real-world space and shooting trajectory of the actual camera matrix (the specific process can be described in the relevant sections above, and will not be elaborated here). This establishes a highly accurate matching relationship between the virtual space and the real space, generating a unified standard virtual space coordinate system and reproducing the shooting trajectory of the actual camera matrix within the virtual space coordinate system. In one example, the virtual space coordinate system can also be referred to as a unified standard space coordinate system.

[0091] 6DoF video shooting 64: After setting up and calibrating a real camera matrix, this embodiment of the application can capture 6DoF video footage in real space to complete 6DoF video shooting.

[0092] AR element creation 65: After completing the spatial and trajectory reconstruction, the embodiments of this application can construct multiple sets of matching AR content elements based on the target of the 6DoF video captured this time.

[0093] After the AR element is created, this embodiment can extract the RGB channel information, Depth channel information, and Alpha channel information of the AR element and aggregate them to form a multi-channel information sequence of the AR element. The multi-channel information sequence of the AR element can be associated with the trajectory information of the virtual camera matrix in the virtual space coordinate system to form a reusable AR sequence. The AR sequence can be regarded as a form of implanted information. The AR sequence can be a data packet structure or a data sequence structure. For example, the AR sequence can indicate the multi-channel information sequence of the AR element to be implanted in each frame of 6DoF video, as well as the translation and rotation matrix of the associated virtual camera matrix at each virtual viewpoint in each frame of 6DoF video.

[0094] The rendering object is fused with the 6DoF video 67. Based on a reusable AR sequence, the embodiments of this application can mix the multi-channel information sequence in the AR sequence with the 6DoF positive film, and perform depth occlusion and culling according to the depth and transparency information, thereby producing a positive film after AR elements are embedded in the 6DoF video.

[0095] As an optional implementation, this embodiment of the application can construct multiple sets of AR content elements based on the target of the captured 6DoF video after the real camera matrix is ​​set up. Based on the reconstructed virtual space coordinate system and trajectory, reusable AR sequences can be determined for subsequent embedding into the 6DoF video. In other possible implementations, this embodiment of the application can also create customized AR elements after capturing the 6DoF video, and determine reusable AR sequences based on the reconstructed virtual space coordinate system and trajectory. This allows AR elements to be embedded into the 6DoF video based on the AR sequences, fulfilling the customization requirements of the AR elements.

[0096] This application embodiment can reproduce the shooting space and trajectory in virtual space based on the parameters of the real camera matrix, enabling virtual objects such as AR elements to match various shooting angles of the real camera matrix. This overcomes the problem that traditional methods of embedding virtual objects into videos cannot be applied to perspective changes in multi-degree-of-freedom videos. This application embodiment constructs a virtual space coordinate system to determine and render the multi-channel information sequence of virtual objects, allowing virtual objects to be efficiently integrated with the original multi-degree-of-freedom video footage. This not only overcomes the time-consuming and costly problem of post-production special effects but also achieves the reusability of virtual object materials; one set of virtual object materials can be used in multiple scenes.

[0097] The following describes the object implantation apparatus for multi-degree-of-freedom video provided in the embodiments of this application. The apparatus described below can be considered as the functional modules required for an electronic device to implement the object implantation method for multi-degree-of-freedom video provided in the embodiments of this application. The apparatus described below can be referred to in correspondence with the description above.

[0098] Figure 7 A block diagram of a multi-degree-of-freedom video object implantation device provided in an embodiment of this application is shown. Figure 7 As shown, the device may include:

[0099] The coordinate acquisition module 710 is used to acquire the coordinates of the real camera matrix in the real space coordinate system, wherein the real camera matrix is ​​used to capture multi-degree-of-freedom video;

[0100] The virtual space construction module 720 is used to construct a virtual space coordinate system based on the coordinates of the real camera matrix in the real space coordinate system. The virtual space coordinate system is the coordinate system corresponding to the virtual camera matrix in the virtual space.

[0101] The trajectory reconstruction module 730 is used to map the trajectory information of the real camera matrix in the real space coordinate system to the trajectory information of the virtual camera matrix in the virtual space coordinate system.

[0102] The association module 740 is used to associate the trajectory information of the virtual camera matrix in the virtual space coordinate system with the virtual object into which the multi-degree-of-freedom video is implanted, so as to obtain the implantation information for implanting the multi-degree-of-freedom video.

[0103] In some embodiments, the real camera matrix includes multiple real cameras; the virtual space construction module 720 is used to construct a virtual space coordinate system based on the coordinates of the real camera matrix in the real space coordinate system, including:

[0104] Based on the coordinates of the first and last real cameras, construct the origin and x-axis basis vector of the virtual space coordinate system;

[0105] Based on the coordinates of the intermediate real camera, the first real camera, and the last real camera, construct the z-axis basis vector of the virtual space coordinate system;

[0106] Based on the x-axis basis vector and z-axis basis vector of the virtual space coordinate system, construct the y-axis basis vector of the virtual space coordinate system;

[0107] A virtual spatial coordinate system is formed based on the origin, x-axis basis vector, y-axis basis vector, and z-axis basis vector of the virtual spatial coordinate system.

[0108] In some embodiments, the virtual space construction module 720 is used to construct the origin of the virtual space coordinate system based on the coordinates of the first and last real cameras, including:

[0109] The center of the line connecting the coordinates of the first and last real cameras is defined as the origin of the virtual space coordinate system.

[0110] In some embodiments, the virtual space construction module 720 is used to construct the x-axis basis vector of the virtual space coordinate system based on the coordinates of the first and last real cameras, including:

[0111] Based on the vector regularization result from the coordinates of the first real camera to the coordinates of the last real camera, the x-axis basis vector of the virtual space coordinate system is determined.

[0112] In some embodiments, the virtual space construction module 720 is used to construct the z-axis basis vector of the virtual space coordinate system based on the coordinates of the intermediate real camera, the first real camera, and the last real camera, including:

[0113] The cross product is performed between the coordinate vectors of the intermediate real camera and the coordinates of the first and last real cameras, and the regularized result of the cross product is used as the z-axis basis vector of the virtual space coordinate system.

[0114] In some embodiments, the virtual space construction module 720 is used to construct the y-axis basis vector of the virtual space coordinate system based on the x-axis basis vector and the z-axis basis vector of the virtual space coordinate system, including:

[0115] The y-axis basis vector of the virtual space coordinate system is obtained by cross-product of the x-axis basis vector and the z-axis basis vector.

[0116] In some embodiments, the trajectory reconstruction module 730 is used to map the trajectory information of the real camera matrix in the real space coordinate system to the trajectory information of the virtual camera matrix in the virtual space coordinate system, including:

[0117] The trajectory information of each frame of multi-degree-of-freedom video of the real camera matrix in the real space coordinate system is mapped to the trajectory information of each frame of video of the virtual camera matrix in the virtual space coordinate system.

[0118] Correspondingly, the association module 740 is used to associate the trajectory information of the virtual camera matrix in the virtual space coordinate system with the virtual object implanted with multi-degree-of-freedom video, so as to obtain the implantation information for implanting multi-degree-of-freedom video, including:

[0119] The trajectory information of each frame of video in the virtual space coordinate system is associated with the virtual object implanted in each frame of multi-degree-of-freedom video to obtain the implantation information for implanting each frame of multi-degree-of-freedom video.

[0120] In some embodiments, the trajectory reconstruction module 730 is used to map the trajectory information of each frame of multi-degree-of-freedom video of the real camera matrix in the real space coordinate system to the trajectory information of each frame of video of the virtual camera matrix in the virtual space coordinate system, including:

[0121] The translation and rotation matrices of the real camera matrix at each virtual viewpoint in each frame of multi-DOF video are mapped to the translation and rotation matrices of the virtual camera matrix at each virtual viewpoint in each frame of multi-DOF video. One virtual viewpoint corresponds to the shooting angle of one real camera, and one real camera corresponds to one virtual camera in virtual space.

[0122] In some embodiments, the trajectory reconstruction module 730 is used to map the translation and rotation matrices of the real camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video to the translation and rotation matrices of the virtual camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video, including:

[0123] The mapping matrix that maps the real space coordinate system to the virtual space coordinate system is multiplied by the translation and rotation matrix corresponding to the virtual viewpoint of each real camera in each frame of multi-degree-of-freedom video, to obtain the translation and rotation matrix corresponding to the virtual viewpoint of each virtual camera in each frame of multi-degree-of-freedom video.

[0124] In some embodiments, the association module 740 is configured to associate the trajectory information of the virtual camera matrix in the virtual space coordinate system with a virtual object embedded in multi-degree-of-freedom video, to obtain embedding information for embedding multi-degree-of-freedom video, including:

[0125] Extract at least one channel information of the virtual object, and aggregate the at least one channel information into a channel information sequence of the virtual object;

[0126] The implantation information is obtained by associating the channel information sequence of the virtual object with the trajectory information of the virtual camera matrix in the virtual space coordinate system.

[0127] Combination Figure 7 As shown, the apparatus provided in this application embodiment may further include:

[0128] The implantation module 750 is used to synchronize the intrinsic parameters of the real camera matrix with the virtual camera matrix; using the intrinsic parameters synchronized with the virtual camera matrix, the virtual objects in the implantation information are rendered into the multi-degree-of-freedom video, wherein the virtual objects rendered into the multi-degree-of-freedom video are associated with the trajectory information of the virtual camera matrix in the virtual space coordinate system carried in the implantation information.

[0129] In some embodiments, the channel information sequence includes: RGB channel information, Depth channel information, and Alpha channel information of the virtual object; the implantation module 750 is used to render the virtual object in the implanted information into the multi-degree-of-freedom video using the intrinsic parameters synchronized with the virtual camera matrix, including:

[0130] Based on the Depth and Alpha channel information of the virtual object, and utilizing the intrinsic parameters of the virtual camera matrix synchronization, depth occlusion and culling of RGB channel information are performed in multi-degree-of-freedom video.

[0131] This application also provides an electronic device that can implement the multi-degree-of-freedom video object implantation method provided in this application by loading the multi-degree-of-freedom video object implantation device described above. In some embodiments, Figure 8 A block diagram of an electronic device is shown, such as... Figure 8 As shown, the electronic device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4.

[0132] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4.

[0133] Optionally, communication interface 2 can be an interface for a communication module used for network communication.

[0134] Optionally, processor 1 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), NPU (Embedded Neural Network Processor), FPGA (Field Programmable Gate Array), TPU (Tensor Processing Unit), AI chip, ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0135] Memory 3 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0136] The memory 3 stores one or more computer-executable instructions, and the processor 1 calls the one or more computer-executable instructions to execute the object implantation method for multi-degree-of-freedom video provided in this application embodiment.

[0137] This application also provides a storage medium that can store one or more computer-executable instructions, which, when executed, implement the object implantation method for multi-degree-of-freedom video as provided in this application.

[0138] This application also provides a computer program that, when executed, implements the object implantation method for multi-degree-of-freedom video as provided in this application embodiment.

[0139] The foregoing describes multiple embodiment schemes provided by the embodiments of this application. The optional methods described in each embodiment scheme can be combined and cross-referenced with each other without conflict, thereby extending to a variety of possible embodiment schemes. These can all be considered as the embodiment schemes disclosed and published by the embodiments of this application.

[0140] While the embodiments disclosed above are described in this application, this application is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A method for object implantation in multi-degree-of-freedom video, wherein, include: Obtain the coordinates of the real camera matrix in the real space coordinate system, wherein the real camera matrix is ​​used to capture multi-degree-of-freedom video; Based on the coordinates of the real camera matrix in the real space coordinate system, a virtual space coordinate system is constructed. The virtual space coordinate system is the coordinate system corresponding to the virtual camera matrix in the virtual space. The virtual space coordinate system is used to realize the trajectory reproduction of the real camera matrix, so that the virtual object matches the shooting positions and angles of the real camera matrix based on the camera trajectory reproduced in the virtual space coordinate system. The trajectory information of the real camera matrix in the real space coordinate system is mapped to the trajectory information of the virtual camera matrix in the virtual space coordinate system; The trajectory information of the virtual camera matrix in the virtual space coordinate system is associated with the virtual object implanted with the multi-degree-of-freedom video to obtain the implantation information for implanting the multi-degree-of-freedom video. The real camera matrix includes multiple real cameras, and the construction of a virtual space coordinate system based on the coordinates of the real camera matrix in the real space coordinate system includes: Based on the coordinates of the first and last real cameras, construct the origin and x-axis basis vector of the virtual space coordinate system; Based on the coordinates of the intermediate real camera, the first real camera, and the last real camera, construct the z-axis basis vector of the virtual space coordinate system; Based on the x-axis basis vector and z-axis basis vector of the virtual space coordinate system, construct the y-axis basis vector of the virtual space coordinate system; A virtual spatial coordinate system is formed based on the origin, x-axis basis vector, y-axis basis vector, and z-axis basis vector of the virtual spatial coordinate system.

2. The method according to claim 1, wherein, The origin of the virtual space coordinate system, constructed based on the coordinates of the first and last real cameras, includes: The center of the line connecting the coordinates of the first and last real cameras is defined as the origin of the virtual space coordinate system. The x-axis basis vector of the virtual space coordinate system, constructed based on the coordinates of the first and last real cameras, includes: Based on the vector regularization result from the coordinates of the first real camera to the coordinates of the last real camera, the x-axis basis vector of the virtual space coordinate system is determined.

3. The method according to claim 1 or 2, wherein, The z-axis basis vector of the virtual space coordinate system, constructed based on the coordinates of the intermediate real camera, the first real camera, and the last real camera, includes: The cross product is performed between the coordinate vectors of the intermediate real camera and the coordinates of the first and last real cameras, and the regularization of the cross product result is used as the z-axis basis vector of the virtual space coordinate system. The construction of the y-axis basis vector of the virtual spatial coordinate system based on the x-axis basis vector and z-axis basis vector includes: The y-axis basis vector of the virtual space coordinate system is obtained by cross-product of the x-axis basis vector and the z-axis basis vector.

4. The method according to claim 1, wherein, The step of mapping the trajectory information of the real camera matrix in the real space coordinate system to the trajectory information of the virtual camera matrix in the virtual space coordinate system includes: The trajectory information of each frame of multi-degree-of-freedom video of the real camera matrix in the real space coordinate system is mapped to the trajectory information of each frame of video of the virtual camera matrix in the virtual space coordinate system. The step of associating the trajectory information of the virtual camera matrix in the virtual space coordinate system with the virtual object embedded in the multi-degree-of-freedom video to obtain the embedding information for embedding the multi-degree-of-freedom video includes: The trajectory information of each frame of video in the virtual space coordinate system is associated with the virtual object implanted in each frame of multi-degree-of-freedom video to obtain the implantation information for implanting each frame of multi-degree-of-freedom video.

5. The method according to claim 4, wherein, The process of mapping the trajectory information of each frame of multi-degree-of-freedom video of the real camera matrix in the real space coordinate system to the trajectory information of each frame of video of the virtual camera matrix in the virtual space coordinate system includes: The translation and rotation matrices of the real camera matrix at each virtual viewpoint in each frame of multi-DOF video are mapped to the translation and rotation matrices of the virtual camera matrix at each virtual viewpoint in each frame of multi-DOF video. One virtual viewpoint corresponds to the shooting angle of one real camera, and one real camera corresponds to one virtual camera in virtual space.

6. The method according to claim 5, wherein, The step of mapping the translation and rotation matrices of the real camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video to the translation and rotation matrices of the virtual camera matrix at each virtual viewpoint in each frame of multi-degree-of-freedom video includes: The mapping matrix that maps the real space coordinate system to the virtual space coordinate system is multiplied by the translation and rotation matrix corresponding to the virtual viewpoint of each real camera in each frame of multi-degree-of-freedom video, to obtain the translation and rotation matrix corresponding to the virtual viewpoint of each virtual camera in each frame of multi-degree-of-freedom video.

7. The method according to claim 1, wherein, The step of associating the trajectory information of the virtual camera matrix in the virtual space coordinate system with the virtual object embedded in the multi-degree-of-freedom video to obtain the embedding information for embedding the multi-degree-of-freedom video includes: Extract at least one channel information of the virtual object, and aggregate the at least one channel information into a channel information sequence of the virtual object; The implantation information is obtained by associating the channel information sequence of the virtual object with the trajectory information of the virtual camera matrix in the virtual space coordinate system.

8. The method according to claim 7, wherein, Also includes: Synchronize the intrinsic parameters of the real camera matrix with the virtual camera matrix; By utilizing the intrinsic parameters of the virtual camera matrix synchronization, virtual objects in the implanted information are rendered into multi-degree-of-freedom video. The virtual objects rendered into the multi-degree-of-freedom video are associated with the trajectory information of the virtual camera matrix in the virtual space coordinate system carried in the implanted information.

9. The method according to claim 8, wherein, The channel information sequence includes: RGB channel information, Depth channel information, and Alpha channel information of the virtual object; the process of rendering the virtual object in the implanted information into the multi-degree-of-freedom video using the intrinsic parameters synchronized by the virtual camera matrix includes: Based on the Depth and Alpha channel information of the virtual object, and utilizing the intrinsic parameters of the virtual camera matrix synchronization, depth occlusion and culling of RGB channel information are performed in multi-degree-of-freedom video.

10. An electronic device, wherein, include: At least one memory and at least one processor; The memory stores one or more computer-executable instructions, and the processor invokes the one or more computer-executable instructions to execute the object implantation method for multi-degree-of-freedom video as described in any one of claims 1-9.

11. A storage medium, wherein, The storage medium stores one or more computer-executable instructions, which, when executed, implement the object implantation method for multi-degree-of-freedom video as described in any one of claims 1-9.

Images