Video generation method and device, computer device, and holographic recording and broadcasting system

Abstract

The application discloses a video generation method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a plurality of image data obtained by a first image acquisition device in different poses for image acquisition of a target real environment and pose data corresponding to each frame of the image data; generating a three-dimensional model corresponding to the target real environment based on the plurality of image data and the pose data corresponding to each frame of the image data; acquiring depth image data of a target user and generating three-dimensional video data corresponding to the target user based on the depth image data; and superimposing the three-dimensional video data in the three-dimensional model to obtain a holographic video. The three-dimensional model corresponding to the target real environment is obtained through the plurality of image data of the target real environment, and then the three-dimensional video data of the target user is superimposed in the three-dimensional model to obtain a holographic video with six degrees of freedom, thereby improving the video watching experience of the user.

Description

Technical Field

[0001] This application relates to the field of augmented reality, and more specifically, to a video generation method, apparatus, computer equipment, and holographic recording and broadcasting system. Background Technology

[0002] In recent years, with the gradual development of mixed reality technology, holographic videos have been increasingly applied to more technological fields due to their unique user experience. For example, in the field of distance learning, teachers typically record three-dimensional teaching videos in advance using image acquisition devices. Students then watch these videos through head-mounted displays, seeing a three-dimensional image of the teacher. However, this teaching method lacks strong interaction between students and teachers, and students' viewing angles are still limited, which can easily affect their learning motivation. Summary of the Invention

[0003] In view of the above problems, this application proposes a video generation method, apparatus, computer equipment, and holographic recording and broadcasting system, which can obtain holographic video data with 6 degrees of freedom and improve the user's video viewing experience.

[0004] In a first aspect, embodiments of this application provide a video generation method, the method comprising: acquiring multiple frames of image data obtained by a first image acquisition device from images of a target real-world environment under different poses, and pose data corresponding to each frame of the image data; generating a three-dimensional model corresponding to the target real-world environment based on the multiple frames of image data and the pose data corresponding to each frame of the image data; acquiring depth image data of a target user, and generating three-dimensional video data corresponding to the target user based on the depth image data; and superimposing the three-dimensional video data on the three-dimensional model to obtain a holographic video.

[0005] Secondly, embodiments of this application provide a video generation apparatus, comprising: a data acquisition module, a model generation module, a video acquisition module, and a data overlay module. The data acquisition module acquires multiple frames of image data obtained by a first image acquisition device from image acquisition of a target real-world environment at different poses, and pose data corresponding to each frame of image data. The model generation module generates a three-dimensional model corresponding to the target real-world environment based on the multiple frames of image data and the pose data corresponding to each frame of image data. The video acquisition module acquires depth image data of a target user and generates three-dimensional video data corresponding to the target user based on the depth image data. The data overlay module overlays the three-dimensional video data onto the three-dimensional model to obtain a holographic video.

[0006] Thirdly, embodiments of this application provide a computer device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the video generation method provided in the first aspect above.

[0007] Fourthly, embodiments of this application provide a holographic recording and broadcasting system, which includes a video editing device and a video playback device. The video editing device is used to execute the video generation method provided in the first aspect above, and the video playback device is used to play the holographic video generated by the video editing device.

[0008] Fifthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the video generation method provided in the first aspect above.

[0009] The solution provided in this application acquires multiple frames of image data and pose data corresponding to each frame of the target real-world environment obtained by a first image acquisition device in different poses; generates a three-dimensional model corresponding to the target real-world environment based on the multiple frames of image data and the pose data corresponding to each frame of the image data; acquires depth image data of the target user and generates three-dimensional video data corresponding to the target user based on the depth image data; and superimposes the three-dimensional video data onto the three-dimensional model to obtain a holographic video. By obtaining a three-dimensional model corresponding to the target real-world environment through multiple frames of image data, and then superimposing the three-dimensional video data of the target user onto the three-dimensional model, a holographic video with 6 degrees of freedom is obtained, improving the user's video viewing experience. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a video generation method provided in one embodiment of this application is shown.

[0012] Figure 2 A schematic diagram of the depth camera in an embodiment of this application is shown.

[0013] Figure 3A flowchart illustrating a video generation method provided in another embodiment of this application is shown.

[0014] Figure 4 A schematic diagram illustrating the principle of generating holographic videos in an embodiment of this application is shown.

[0015] Figure 5 A schematic diagram of the structure of the video generation apparatus provided in an embodiment of this application is shown.

[0016] Figure 6 A structural block diagram of a computer device provided in an embodiment of this application is shown.

[0017] Figure 7 A structural block diagram of a holographic recording and broadcasting system provided in an embodiment of this application is shown.

[0018] Figure 8 A structural block diagram of a computer-readable storage medium provided in an embodiment of this application is shown. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0020] The inventors have proposed a video generation method, apparatus, computer equipment, and holographic recording and broadcasting system provided in the embodiments of this application. These systems can obtain a three-dimensional model of the target real environment through multi-frame image data corresponding to the target real environment, and then superimpose the target user's three-dimensional video data onto the three-dimensional model to obtain a holographic video with 6 degrees of freedom, thereby improving the user's video viewing experience.

[0021] The video generation method provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] Please see Figure 1 , Figure 1 This paper illustrates a flowchart of a video generation method provided in one embodiment of this application. The following will focus on... Figure 1 The process shown will be described in detail. The video generation method may specifically include the following steps:

[0023] Step S110: Obtain multiple frames of image data obtained by the first image acquisition device from image acquisition of the target real environment under different poses, as well as pose data corresponding to each frame of image data.

[0024] In this embodiment, a computer device can acquire multiple frames of image data corresponding to a target real-world environment using a first image acquisition device. The first image acquisition device can continuously change its position and pose data during the acquisition of multiple frames of image data; that is, the first image acquisition device can acquire multiple frames of image data of the target real-world environment with different pose data. Simultaneously, based on the first image acquisition device, the computer device can not only acquire multiple frames of image data of the target real-world environment, but also obtain the pose data corresponding to each frame of image data acquired by the first image acquisition device. This pose data is then associated with that frame of image data to subsequently obtain a three-dimensional model of the target real-world environment based on the multiple frames of image data corresponding to the target real-world environment and their respective pose data.

[0025] Specifically, the image data of the target real-world environment acquired by the computer device can include the color value and depth value corresponding to each pixel in the image data. The computer device can obtain three-dimensional image data of the target real-world environment from just one frame of image data, based on the position and viewpoint corresponding to that image data. Since the computer device acquires multiple frames of image data of the target real-world environment at different poses based on the first image acquisition device, it can stitch together these multiple frames of image data and the pose data corresponding to each frame to obtain a three-dimensional model of the target real-world environment.

[0026] In some implementations, the first image acquisition device may include a depth camera module and an inertial measurement unit (IMU) module. The depth camera module can not only acquire image data corresponding to the target's real-world environment at the current position and viewpoint—that is, determine the pixel value corresponding to each pixel—but also acquire the depth value corresponding to each pixel in the image data, which is the distance information between that pixel and the depth camera module in the target's real-world environment. Figure 2 As shown, it illustrates the principle of a depth camera. For point M in the real environment, the depth camera can obtain its imaging point Xm in the RGB image, as well as the distance from point M to the vertical plane (XY plane) where the depth camera is located. This distance is the depth value of point M.

[0027] Meanwhile, the IMU module can acquire the acceleration and angular velocity data corresponding to the three coordinate axes in three-dimensional space when the first image acquisition device acquires image data of the target real environment. The computer device can determine the position and attitude data of the first image acquisition device when acquiring each frame of image data based on the three-axis angular velocity and acceleration data obtained by the IMU module. That is, the position data of the first image acquisition device and the attitude data of the device's orientation angle when acquiring each frame of image data.

[0028] Step S120: Based on the multi-frame image data and the pose data corresponding to each frame of image data, generate a three-dimensional model corresponding to the target real environment.

[0029] In this embodiment, after obtaining multiple frames of image data of the target real-world environment and the pose data corresponding to each frame of image data based on the first image acquisition device, the computer device can directly generate a three-dimensional model of the target real-world environment based on the acquired image data and the corresponding pose data. In some embodiments, the computer device can determine the pose data when the first image acquisition device acquires each frame of image data based on the IMU data acquired by the first image acquisition device, and then stitch the multiple frames of image data together based on the pose data corresponding to each frame of image data to obtain a three-dimensional model of the target real-world environment.

[0030] Step S130: Obtain depth image data of the target user, and generate three-dimensional video data corresponding to the target user based on the depth image data.

[0031] In this embodiment, the computer device can also acquire depth image data of the target user and generate corresponding 3D video data based on the depth image data. This 3D video data is then superimposed onto the 3D model of the target real-world environment generated in the aforementioned steps to form a 6-DOF holographic video. Specifically, the computer device can acquire depth image data of the target user through an image acquisition device including a depth camera module. The depth camera module can directly acquire image data and depth data from the depth image data of the target user, enabling the computer device to generate corresponding 3D video data based on the depth image data. It is worth noting that the depth image data obtained by the computer device at this point is the target user's depth image data obtained when the image acquisition device is in a fixed position and posture. The computer device can acquire multiple depth image data of the same target user through multiple image acquisition devices located at different fixed positions and in different postures. However, these image acquisition devices are not continuous; that is, their relative positional relationships are unclear. The computer device can generate a separate 3D video data of the target user from a different perspective for each image acquisition device.

[0032] The depth image data of the target user collected by the computer device at this time is not the depth data and image data of the target user at a certain moment, but includes the uninterrupted depth image data of the target user within a continuous period of time. The computer device can process each frame of depth image data within the continuous period of time to obtain the corresponding three-dimensional video data of the target user within the continuous period of time.

[0033] In some implementations, the computer device can acquire not only the depth image data of the target user, but also the target user's audio data and time data. In the process of generating the three-dimensional video data corresponding to the target user based on the depth image data, the computer device can also fuse the audio data with the depth image data based on the time data. The resulting three-dimensional video data corresponding to the target user can include not only the target user's three-dimensional motion data, but also the audio data emitted by the target user that is synchronized with the three-dimensional motion data.

[0034] In some implementations, the target user can include at least two users. The computer device acquires depth image data of the target user, and may also acquire first depth image data corresponding to each of the at least two users. Based on the first depth image data corresponding to each user, the computer device generates reference 3D data for each user. This allows the reference 3D data corresponding to all users to be superimposed onto a 3D model, resulting in a holographic video including 3D video data from multiple users. Specifically, each of the target users can actually be located in a different spatial position. By synchronously superimposing the reference 3D data corresponding to each user onto the 3D model, the computer device can achieve the effect of multiple users in different spatial positions communicating or assisting in experiments within the same target real-world environment.

[0035] In some implementations, the computer device can also overlay reference 3D data for each user onto the 3D model in real time. This means that instead of waiting for the user to complete all actions before generating a long-term reference 3D data frame, it acquires a single frame of depth image data for each user, converts that frame into reference 3D data, and overlays it onto the 3D model in real time. The resulting holographic video can then display the user's current actions in real time. Clearly, if multiple users simultaneously generate a single frame of reference 3D data for their current moment, the computer device can time-calibrate these multiple single frames and overlay them simultaneously onto the 3D model. This allows for real-time interaction between multiple different users within a 3D model corresponding to the target real-world environment.

[0036] Simultaneously, the computer equipment can synchronously play each frame of the real-time generated holographic video on the head-mounted display devices corresponding to each user. This allows users to see the real-time actions of other users within the 3D model and provide feedback accordingly. This enables interactive assistance between multiple target users in different spatial locations within the 3D model. For example, the computer equipment can simultaneously acquire each frame of reference 3D data for both students and teachers, overlaying each frame onto the corresponding 3D model of the real-world environment in real time. Simultaneously, each frame of the generated holographic video is played synchronously on the head-mounted display devices of both students and teachers. Thus, both students and teachers can see each other's real-time actions and provide corresponding feedback, achieving student-teacher interaction and remote assistance.

[0037] Step S140: Superimpose the three-dimensional video data onto the three-dimensional model to obtain a holographic video.

[0038] In this embodiment, after obtaining the 3D model corresponding to the target real-world environment and the 3D video data corresponding to the target user, the computer device can overlay the 3D video data onto the 3D model. This yields 3D video data of the target user's actions within the target real-world environment constructed from the 3D model. The resulting holographic video, after overlay, includes not only the target user's continuous 3D motion data but also the 3D model of the target real-world environment in which the user resides. When the user views the overlaid holographic video through a relevant head-mounted display device, they can see not only the 3D video data corresponding to the target user but also the 3D data of the target real-world environment, and can freely switch perspectives to observe the 3D model corresponding to the target real-world environment.

[0039] In some implementations, the computer device may also acquire only the two-dimensional image data of the target user, that is, the image data of the target user from a fixed perspective. The computer device can still overlay this two-dimensional image data onto the three-dimensional stereo model corresponding to the target reality environment generated in the aforementioned steps. When the user watches the overlaid video data, they can still see 6 degrees of freedom video data. This is because although image data is overlaid, the computer device performs image data overlay on the basis of a three-dimensional stereo model corresponding to the target reality environment with 6 degrees of freedom. Therefore, according to the video data generated after the final overlay, the user can still see 6 degrees of freedom video data. However, since the computer device has not acquired the target user's three-dimensional stereo depth data, the user cannot see the target user's three-dimensional stereo depth data in the overlaid video data.

[0040] In some implementations, if the computer device can acquire reference 3D data corresponding to at least two users, the process of obtaining a holographic video by superimposing 3D video data onto a 3D model can also include simultaneously superimposing reference 3D data corresponding to each user onto the 3D model. Thus, the resulting holographic video can include 3D video data from multiple users. However, it should be understood that if there are no pre-defined operations between the multiple users, the 3D video data corresponding to each user in the holographic video will be independent and there will be no interactive actions. In some implementations, when acquiring reference 3D data, the computer device can also instruct users to perform specified operations at any time according to preset interactive instructions. Therefore, after superimposing multiple reference 3D data onto the 3D model, interactive actions can occur between the 3D video data of each user.

[0041] The video generation method provided in this application can be implemented by a holographic recording and broadcasting system comprising a front-end environment perception module and a back-end video generation module. This system generates 3D lecture videos delivered by teachers in specific training environments. The front-end environment perception module can be a device such as an MR camera capable of rapidly scanning and digitally reconstructing training scenes. The back-end video generation module can fuse the 3D video data containing the lecture content recorded by the teacher with the 3D environment model corresponding to the training scene acquired by the front-end environment perception module, resulting in a course resource package for teaching specific content. Students can view the system-generated holographic videos through devices such as MR headsets, personal computers, or mobile terminals, enhancing their learning experience and synchronizing theory with practice.

[0042] The video generation method provided in this application involves acquiring multiple frames of image data from a target real-world environment obtained by a first image acquisition device in different poses, along with pose data corresponding to each frame of image data; generating a three-dimensional model corresponding to the target real-world environment based on the multiple frames of image data and the pose data corresponding to each frame of image data; acquiring depth image data of a target user, and generating three-dimensional video data corresponding to the target user based on the depth image data; and superimposing the three-dimensional video data onto the three-dimensional model to obtain a holographic video. By obtaining a three-dimensional model corresponding to the target real-world environment through multiple frames of image data, and then superimposing the three-dimensional video data of the target user onto the three-dimensional model, a holographic video with 6 degrees of freedom is obtained, enhancing the user's video viewing experience.

[0043] Please see Figure 3 , Figure 3 A flowchart illustrating another embodiment of the video generation method provided in this application is shown below. Figure 3The process shown will be described in detail. The video generation method may specifically include the following steps:

[0044] Step S201: Obtain multiple frames of image data obtained by the first image acquisition device from image acquisition of the target real environment under different poses, as well as pose data corresponding to each frame of image data.

[0045] In this embodiment, step S201 can be referred to the content of other embodiments, and will not be repeated here.

[0046] Step S202: Based on the first depth data and the first pixel data in each frame of image data, determine the three-dimensional data corresponding to each frame of image data.

[0047] In this embodiment, each frame of image data of the target real-world environment acquired by the first image acquisition device includes first depth data and first pixel data. That is, the data recorded for each pixel in the image data acquired by the first image acquisition device includes not only the pixel value data representing the color of that pixel (i.e., the first pixel data), but also the distance data between that pixel and the first image acquisition device (i.e., the first depth data). Specifically, the first image acquisition device can simultaneously acquire the first depth data and the first pixel data of the target real-world environment using a depth camera. After the first image acquisition device acquires the first depth data and the first pixel data corresponding to each frame of image, the computer device can combine the first depth data and the first pixel data to generate three-dimensional data corresponding to each frame of image, so that a three-dimensional model corresponding to the target real-world environment can be obtained based on the three-dimensional data of each frame of image.

[0048] Among them, depth cameras can be structured light depth cameras, which use near-infrared lasers to project light with specific structural features onto the object being photographed, and then a dedicated infrared camera collects the data, converting the changes in the collected structured light into depth information, and sequentially acquiring the three-dimensional data corresponding to the image data; depth cameras can also be time-of-flight (TOF) depth cameras, which emit continuous laser pulses into the target environment, receive the reflected light through a sensor, and obtain the depth information of the target environment by measuring the round-trip time of the detection light pulses; depth cameras can also be binocular stereo vision depth cameras, which are based on the principle of parallax and use imaging equipment to acquire two images of the target environment from different positions, and determine the depth information of the target environment by calculating the positional deviation between corresponding points in the two images.

[0049] Step S203: Based on the three-dimensional data and pose data corresponding to the image data of each frame, generate a three-dimensional model corresponding to the target real environment.

[0050] In this embodiment, the pose data refers to the pose data corresponding to each frame of image data acquired by the first image acquisition device. The pose data of the first image acquisition device includes its position and orientation data in a three-dimensional coordinate system when acquiring each frame of image data. The first image acquisition device may be equipped with an IMU module to acquire its own pose data while acquiring image data. A computer device can generate a three-dimensional model of the target real-world environment based on the pose data corresponding to each frame of image data acquired by the first image acquisition device, and the three-dimensional data generated based on each frame of image data.

[0051] Specifically, the IMU module can record the angular velocity and acceleration of the first image acquisition device on the three coordinate axes of the three-dimensional spatial coordinate system when acquiring image data. The first image acquisition device can pre-acquire reference coordinate data in the world coordinate system at the start of image data acquisition. Subsequently, for each image data frame corresponding to the target real-world environment acquired, the device records its angular velocity and acceleration in three-dimensional space during the acquisition. The computer device can then obtain the coordinate data of the device when acquiring that frame of image data based on the reference coordinate data and the angular velocity and acceleration in three-dimensional space. If the device acquires another frame of image data, the computer device can obtain new coordinate data based on the coordinate data of the previous frame of image data and the new angular velocity and acceleration. This process is repeated, allowing the computer device to obtain the coordinate data corresponding to each frame of image data acquired by the first image acquisition device based on the IMU module. It can be understood that the pose data of the first image acquisition device includes its position and orientation data when acquiring each frame of image data, that is, its coordinate data in the world coordinate system and its angle information on the three coordinate axes. Clearly, the first image acquisition device can directly obtain the angle information, i.e., the orientation data, when acquiring image data based on the IMU module. Furthermore, as the foregoing analysis shows, the first image acquisition device can also obtain the coordinate data, i.e., the position data, when acquiring each frame of image data corresponding to the target's real-world environment based on the IMU module. Therefore, the first image acquisition device can obtain the pose data when acquiring each frame of image data through the IMU module.

[0052] Step S204: Based on the generation times corresponding to the same action of the target user in the depth image data under different viewpoints, calibrate the second depth data and the second pixel data corresponding to the target user's body under the different viewpoints.

[0053] In this embodiment, the depth image data includes second depth data and second pixel data acquired by the second image acquisition device from the target user's body at different viewpoints. To facilitate viewing the holographic video from different angles and perspectives, the computer device can use multiple second image acquisition devices to acquire depth image data corresponding to the target user's body at different viewpoints. This data is then used to generate corresponding 3D video data based on the depth image data from each viewpoint. The computer device also merges and superimposes the 3D video data from multiple viewpoints onto a 3D model to obtain a holographic video with multiple different viewpoints. However, it should be understood that because there are multiple different second image acquisition devices, and there is no data association between them during the subsequent 3D video data generation process, directly merging and superimposing the 3D video data corresponding to each viewpoint may result in the 3D video data from different viewpoints not being synchronized in the final holographic video. Therefore, to ensure that the 3D video data from different viewpoints are synchronized in the final generated holographic video, the computer device can perform time calibration on the depth image data from different viewpoints during the acquisition of depth image data from different viewpoints by each second image acquisition device. Specifically, the computer device can calibrate the second depth data and second pixel data of the target user's body from different perspectives based on the generation time corresponding to the same action of the target user in the depth image data at different viewpoints.

[0054] In some implementations, during the recording of 3D video data, the user can also manually calibrate the time between the various second image acquisition devices. Upon receiving the time calibration command sent by the user, each second image acquisition device can actively transmit time signals to each other to facilitate time calibration.

[0055] Step S205: Based on the second depth data and second pixel data corresponding to the target user's body from different perspectives within a preset time period, generate three-dimensional video data corresponding to the target user from different perspectives.

[0056] In this embodiment, a computer device can acquire depth image data (i.e., second depth data and second pixel data) of a target user's body from different perspectives using multiple second image acquisition devices. Based on the second depth data and second pixel data of the target user from the same perspective, it generates 3D video data of the target user from that perspective. Furthermore, based on the multiple second image acquisition devices, it obtains 3D video data of the target user from multiple perspectives. The second image acquisition devices can acquire the second depth data and second pixel data of the target user's body from a fixed perspective using a depth camera module. The depth camera module in the second image acquisition device can be the same as the depth camera module in the first image acquisition device mentioned earlier, and will not be described in detail here. Clearly, the computer device acquires 3D video data of the target user from different perspectives using multiple second image acquisition devices. This means that the perspective of the target user corresponding to each second image acquisition device is fixed, and the second image acquisition devices cannot be moved.

[0057] Step S206: Obtain the third depth data and third pixel data of the target user's hand from the target viewpoint, captured by the third image acquisition device worn by the target user.

[0058] Step S207: Based on the third depth data and the third pixel data, generate three-dimensional video data corresponding to the target user from the target viewpoint.

[0059] In this embodiment, the computer device can also acquire third depth data and third pixel data corresponding to the target user's hand from the target viewpoint via a third image acquisition device, and generate three-dimensional video data corresponding to the target user from the target viewpoint based on the third depth data and third pixel data. It is understood that, to facilitate a more detailed viewing of the target user's hand movements in the holographic video, the computer device can also acquire three-dimensional video data corresponding to the target user's hand via a third image acquisition device worn by the user, and then merge and superimpose the three-dimensional video data from the target viewpoint into a three-dimensional model. Thus, while watching the holographic video, the user can select the target viewpoint corresponding to the target user, and thus observe the target user's hand movements more closely.

[0060] In some implementations, the method by which a computer device generates 3D video data corresponding to a target user from a target perspective may further include the following steps:

[0061] The motion data corresponding to the target user's body joints is acquired through the motion capture sensor worn by the target user. The motion data includes the coordinate data of each body joint of the target user in the world coordinate system. Based on the motion data, the third depth data, and the third pixel data, three-dimensional video data of the target user corresponding to the target viewpoint is generated.

[0062] In other words, the computer device can not only acquire third depth data and third pixel data corresponding to the target user's hand from the target's perspective using a third image acquisition device worn by the target user, but also acquire motion data corresponding to the target user's body joints using motion capture sensors worn by the target user. Based on this motion data, third depth data, and third pixel data, it can then generate 3D video data of the target user from the target's perspective. Specifically, the motion capture sensors worn by the user can acquire the coordinate data of each joint of the target user's body in the world coordinate system in real time. This coordinate data is used to comprehensively determine the position coordinate data of each joint of the user's body at each frame of the image, based on the third depth data, thus forming the target user's 3D video data.

[0063] Step S208: Obtain spatial annotation data, which is the markings made by the target user collected by the fourth acquisition device.

[0064] In this embodiment, to facilitate understanding of the content explained by the target user in the 3D video data for users watching the holographic video, the target user can make some appropriate marks in space based on the fourth acquisition device during the recording of the 3D video data. The computer device can directly acquire the spatial annotation data corresponding to the marks made by the target user based on the fourth acquisition device, and use this spatial annotation data to superimpose it onto the 3D model.

[0065] In some implementations, the spatial annotation data acquired by the computer device may include time and location data of the target user's markings. The fourth acquisition device may include an IMU module. As can be seen from the foregoing steps of the embodiments, the computer device can obtain the pose data of the fourth acquisition device at any given time based on the angular velocity and acceleration data recorded by the IMU module corresponding to the fourth acquisition module. Simultaneously, the computer device can also acquire the target user's operation data on the fourth acquisition device, thereby comprehensively determining the spatial annotation data made by the target user in three-dimensional space.

[0066] Step S209: Overlay the spatial annotation data onto the three-dimensional model so that the three-dimensional model includes the markings made by the target user.

[0067] In this embodiment, after acquiring spatial annotation data made by a target user using a fourth acquisition device, the computer device can overlay the spatial annotation data onto a three-dimensional model to display the spatial markings made by the target user in the final holographic video. The spatial annotation data acquired by the computer device can include not only the spatial location where the target user made the marking, but also the time at which the target user made the marking. Therefore, the computer device can overlay the spatial annotation data onto the three-dimensional model at an appropriate time based on the time in the spatial annotation data.

[0068] Step S210: Obtain the first marker data in the three-dimensional model and the second marker data in the three-dimensional video data.

[0069] Step S211: Based on the first marker data and the second marker data, the three-dimensional video data is superimposed on the three-dimensional stereo model to obtain the holographic video.

[0070] In this embodiment, the first marker data is located in the target real-world environment. During the process of the first image acquisition device acquiring image data corresponding to the target real-world environment, the first marker data is also acquired into the image data. The first image acquisition device considers it as part of the target real-world environment during the acquisition process. The first marker data will also exist in the three-dimensional stereoscopic model corresponding to the target real-world environment finally generated by the computer device based on the image data. Similarly, during the process of the second and third image acquisition devices acquiring depth image data of the target user from their respective perspectives, the second marker data can be considered as part of the target user's body or hand data. The two-dimensional video data generated by the computer device from various perspectives based on the depth image data may also include the second marker data.

[0071] Understandably, the first and second marker data are set up to facilitate the computer device's overlay of 3D video data onto the 3D stereoscopic model. After obtaining the 3D stereoscopic model corresponding to the target real-world environment and the 3D video data corresponding to the target user from various perspectives through the above steps, the computer device needs to overlay the 3D video data onto the 3D stereoscopic model to form a holographic video. Since both are 3D data, the computer device needs a reference point that exists in both the 3D video data and the 3D stereoscopic model to successfully overlay the data. Therefore, after acquiring the 3D stereoscopic model and 3D video data, the computer device can search for the first and second marker data in the 3D stereoscopic model and 3D video data respectively, and then overlay the 3D video data onto the 3D stereoscopic model based on the first and second marker data. Specifically, the computer device can gradually overlay the 3D video data onto the 3D stereoscopic model, and obtain the degree of overlap between the first and second marker data during the overlay process. The computer device can complete the data overlay operation when the first and second marker data are completely overlapped.

[0072] like Figure 4 As shown, this diagram illustrates a holographic video provided in an embodiment of this application. A computer device acquires multiple frames of image data from the target real-world environment via a first image acquisition device and generates a three-dimensional model corresponding to the target real-world environment. A second image acquisition device (only one shown in the diagram) acquires depth image data of the target user from a fixed perspective and generates corresponding three-dimensional video data. The target user can generate three-dimensional video data corresponding to the target user's hand from a fixed perspective by wearing a third image acquisition device. Finally, the computer device can superimpose the three-dimensional video data from each perspective onto the three-dimensional model to obtain a holographic video.

[0073] Step S212: In response to the video query information input by the first user based on the holographic video being played, obtain the supplementary video data of the target user.

[0074] In this embodiment, after generating a holographic video, the computer device can send it to a designated user for viewing and learning. The target user receives explanations of knowledge points through the holographic video, enhancing their viewing experience and improving learning outcomes. Furthermore, to further enhance the viewing quality, the computer device can also acquire questions from the first user regarding the holographic video. This means the user can provide feedback on the content of the holographic video during viewing and input video inquiry information through the display device. Specifically, the video inquiry information from the first user can be either audio or text data, without limitation. The target user who captured the holographic video can receive and answer the video inquiry information from the first user in real time. Based on the video inquiry information from the first user, the target user can supplement the video data using the methods described above. The computer device can then stitch the supplementary video data with the originally recorded 3D video data to obtain supplementary 3D video data including answers to the video inquiry information.

[0075] Step S213: The supplemented video data is stitched together with the three-dimensional video data to obtain supplemented three-dimensional video data.

[0076] Step S214: Superimpose the supplementary video data onto the three-dimensional model to obtain a supplementary holographic video.

[0077] In this embodiment, after stitching together the supplementary video data recorded by the target user in response to video inquiries with the original 3D video data, the computer device can overlay the resulting supplementary 3D video data onto a 3D model to form a supplementary holographic video. Clearly, in this supplementary holographic video, the target user answers the questions raised by the first user regarding the content of the holographic video. The computer device can follow the same process to obtain different questions raised by multiple different users regarding the content of the holographic video, and obtain supplementary 3D video data recorded by the target user in response to these questions, continuously supplementing the content of the holographic video to aid subsequent users' understanding of the knowledge points when watching the supplementary holographic video.

[0078] The video generation method provided in this application involves acquiring image data of the target real-world environment and corresponding posture data of the first image acquisition device using a first image acquisition device to generate a three-dimensional model of the target real-world environment. Based on depth image data of the target user's body acquired from different viewpoints using a second image acquisition device, and depth image data of the target user's hand acquired from the target viewpoint using a third image acquisition device, three-dimensional video data of the target user is generated from each viewpoint. Based on first marker data in the three-dimensional model and second marker data in the three-dimensional video data, the three-dimensional video data is superimposed onto the three-dimensional model to obtain a holographic video. By obtaining a three-dimensional model of the target real-world environment through multiple frames of image data, and then superimposing the target user's three-dimensional video data onto the three-dimensional model, a holographic video with 6 degrees of freedom is obtained, which can enhance the user's experience of watching holographic videos.

[0079] Please see Figure 5 The diagram illustrates a structural block diagram of a video generation apparatus 200 according to an embodiment of this application. The video generation apparatus 200 includes: a data acquisition module 210, a model generation module 220, a video acquisition module 230, and a data overlay module 240. Specifically, the data acquisition module 210 acquires multiple frames of image data obtained by a first image acquisition device from image acquisition of a target real-world environment under different poses, as well as pose data corresponding to each frame of image data; the model generation module 220 generates a three-dimensional model corresponding to the target real-world environment based on the multiple frames of image data and the pose data corresponding to each frame of image data; the video acquisition module 230 acquires depth image data of a target user and generates three-dimensional video data corresponding to the target user based on the depth image data; and the data overlay module 240 overlays the three-dimensional video data onto the three-dimensional model to obtain a holographic video.

[0080] As one possible implementation, each frame of image data includes first depth data and first pixel data, and the pose data is the pose data corresponding to each frame of image data acquired by the acquisition device. The model generation module 220 is also used to determine the three-dimensional data corresponding to each frame of image data based on the first depth data and first pixel data in each frame of image data; and to generate a three-dimensional model corresponding to the target real environment based on the three-dimensional data and pose data corresponding to each frame of image data.

[0081] As one possible implementation, the depth image data includes second depth data and second pixel data acquired by the second image acquisition device from the target user's body at different viewpoints. The video acquisition module 230 is also used to generate three-dimensional video data corresponding to the target user at different viewpoints based on the second depth data and second pixel data corresponding to the target user's body at different viewpoints within a preset time period.

[0082] As one possible implementation, the video generation apparatus 200 also includes a data calibration module, which is used to calibrate the second depth data and the second pixel data corresponding to the same action of the target user in the depth image data at different viewpoints.

[0083] As one possible implementation, the video acquisition module 230 is also used to acquire third depth data and third pixel data corresponding to the hand of the target user captured by the third image acquisition device worn by the target user from the target viewpoint; and to generate three-dimensional video data corresponding to the target user from the target viewpoint based on the third depth data and the third pixel data.

[0084] As one possible implementation, the video acquisition module 230 is further configured to acquire motion data corresponding to the body joints of the target user through the motion capture sensor worn by the target user, the motion data including the coordinate data of each body joint of the target user in the world coordinate system; and generate three-dimensional video data of the target user corresponding to the target user in the target viewpoint based on the motion data, the third depth data and the third pixel data.

[0085] As one possible implementation, the data overlay module 240 is also used to acquire first marker data in the three-dimensional model and second marker data in the three-dimensional video data; based on the first marker data and the second marker data, the three-dimensional video data is overlaid in the three-dimensional model to obtain a holographic video.

[0086] As one possible implementation, the video generation device 200 further includes a marker acquisition module and a marker overlay module. The marker acquisition module is used to acquire spatial annotation data, which is the markers made by the target user and acquired by the fourth acquisition device. The marker overlay module is used to overlay the spatial annotation data into the three-dimensional model so that the three-dimensional model includes the markers made by the target user.

[0087] In one possible implementation, the target users include at least two users, and the video acquisition module 230 is further configured to acquire first depth image data corresponding to each of the at least two users; based on the first depth image data corresponding to each user, generate reference three-dimensional data corresponding to each user; the data overlay module 240 is further configured to simultaneously overlay the reference three-dimensional data corresponding to each user in the three-dimensional model to obtain a holographic video.

[0088] In one possible implementation, the video generation device 200 further includes a video supplementation module, a video stitching module, and a video overlay module. The video supplementation module is used to obtain supplementary video data from the target user in response to a video query input by a first user based on the holographic video being played. The video stitching module is used to stitch the supplementary video data with the three-dimensional video data to obtain supplementary three-dimensional video data. The video overlay module is used to overlay the supplementary video data onto the three-dimensional model to obtain a supplementary holographic video.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0090] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0091] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0092] In summary, the solution provided in this application acquires multiple frames of image data and pose data corresponding to each frame of the target real-world environment obtained by a first image acquisition device in different poses; generates a three-dimensional model corresponding to the target real-world environment based on the multiple frames of image data and the pose data corresponding to each frame of the image data; acquires depth image data of the target user and generates three-dimensional video data corresponding to the target user based on the depth image data; and superimposes the three-dimensional video data onto the three-dimensional model to obtain a holographic video. By obtaining a three-dimensional model corresponding to the target real-world environment through multiple frames of image data, and then superimposing the three-dimensional video data of the target user onto the three-dimensional model, a holographic video with 6 degrees of freedom is obtained, improving the user's video viewing experience.

[0093] Please refer to Figure 6This diagram illustrates a structural block diagram of a computer device 400 provided in an embodiment of this application. The computer device 400 can be a physical server, a cloud server, or the like. The computer device 400 in this application may include one or more of the following components: a processor 410, a memory 420, and one or more application programs. The one or more application programs may be stored in the memory 420 and configured to be executed by the one or more processors 410. The one or more programs are configured to perform the methods described in the foregoing method embodiments.

[0094] Processor 410 may include one or more processing cores. Processor 410 connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 420, and by calling data stored in memory 420. Optionally, processor 410 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 410 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 410 and may be implemented separately using a communication chip.

[0095] The memory 420 may include random access memory (RAM) or read-only memory (ROM). The memory 420 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 420 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created during the use of the computer device (such as phone books, audio and video data, chat log data, etc.).

[0096] Please refer to Figure 7This document illustrates a holographic recording and broadcasting system 600 provided in an embodiment of this application. The holographic recording and broadcasting system 600 includes a video editing device 610 and a video playback device 620. The video editing device 610 is used to execute the methods described in the above method embodiments, and the video playback device 620 is used to play the holographic video generated by the video editing device 610. The video playback device 620 can be installed in any terminal playback device such as a computer or television, or in any mobile playback device such as a mobile phone or tablet.

[0097] Please refer to Figure 8 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0098] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A video generation method, characterized in that, The method includes: Acquire multiple frames of image data obtained by the first image acquisition device from image acquisition of the target real environment under different poses, as well as pose data corresponding to each frame of image data; Based on the multi-frame image data and the pose data corresponding to each frame of image data, a three-dimensional model corresponding to the target real environment is generated. Acquire depth image data of the target user, and generate three-dimensional video data corresponding to the target user based on the depth image data; The three-dimensional video data is superimposed on the three-dimensional model to obtain a holographic video; The target users include at least two users, and the step of acquiring depth image data of the target users and generating 3D video data corresponding to the target users based on the depth image data includes: Obtain the first depth image data corresponding to each of the at least two users; Based on the first depth image data corresponding to each user, generate reference 3D data corresponding to each user; The 3D video data is superimposed on the 3D model to obtain a holographic video, including: The reference 3D data of the at least two users are time-calibrated and simultaneously superimposed onto the 3D stereoscopic model to obtain a holographic video; The depth image data includes second depth data and second pixel data acquired by the second image acquisition device from the target user's body at different viewpoints. Generating three-dimensional video data corresponding to the target user based on the depth image data includes: Based on the second depth data and second pixel data of the target user’s body from different perspectives within a preset time period, generate three-dimensional video data of the target user from different perspectives. Before generating the 3D video data corresponding to the target user based on the depth image data, the method further includes: Based on the generation times corresponding to the same action of the target user in the depth image data under different viewpoints, the second depth data and the second pixel data corresponding to the body of the target user under the different viewpoints are calibrated. The step of acquiring depth image data of the target user and generating 3D video data corresponding to the target user based on the depth image data includes: Acquire the third depth data and third pixel data of the target user's hand from the target viewpoint, obtained by the third image acquisition device worn by the target user. Based on the third depth data and the third pixel data, three-dimensional video data corresponding to the target user from the target viewpoint is generated.

2. The method according to claim 1, characterized in that, Each frame of image data includes first depth data and first pixel data. The pose data is the pose data corresponding to each frame of image data acquired by the first image acquisition device. Generating a three-dimensional model of the target real-world environment based on the multiple frames of image data and the pose data corresponding to each frame of image data includes: Based on the first depth data and the first pixel data in each frame of image data, determine the three-dimensional data corresponding to each frame of image data; Based on the three-dimensional data and pose data corresponding to each frame of the image data, a three-dimensional model corresponding to the target real environment is generated.

3. The method according to claim 1, characterized in that, The step of generating 3D video data corresponding to the target user from the target viewpoint based on the third depth data and the third pixel data includes: The motion data corresponding to the body joints of the target user is obtained by using motion capture sensors worn by the target user. The motion data includes the coordinate data of each body joint of the target user in the world coordinate system. Based on the motion data, the third depth data, and the third pixel data, three-dimensional video data corresponding to the target user from the target viewpoint is generated.

4. The method according to any one of claims 1-3, characterized in that, The process of superimposing the three-dimensional video data onto the three-dimensional model to obtain a holographic video includes: Obtain the first marker data in the three-dimensional model and the second marker data in the three-dimensional video data; Based on the first and second marker data, the three-dimensional video data is superimposed on the three-dimensional model to obtain the holographic video.

5. The method according to any one of claims 1-3, characterized in that, After acquiring the depth image data of the target user and generating the corresponding 3D video data based on the depth image data, the method further includes: Acquire spatial annotation data, which is the markings made by the target user collected by the fourth acquisition device; The spatial annotation data is superimposed on the three-dimensional model so that the three-dimensional model includes the markings made by the target user.

6. The method according to any one of claims 1-3, characterized in that, After superimposing the three-dimensional video data onto the three-dimensional model to obtain a holographic video, the method further includes: In response to a video query input by the first user based on the holographic video being played, supplementary video data of the target user is obtained; The supplemented video data is stitched together with the three-dimensional video data to obtain supplemented three-dimensional video data; The supplementary video data is superimposed on the three-dimensional model to obtain a supplementary holographic video.

7. A video generation apparatus, characterized in that, The device includes: The data acquisition module is used to acquire multiple frames of image data obtained by the first image acquisition device from image acquisition of the target real environment under different poses, as well as pose data corresponding to each frame of image data; The model generation module is used to generate a three-dimensional model of the target real environment based on the multi-frame image data and the pose data corresponding to each frame of image data. The video acquisition module is used to acquire depth image data of the target user and generate three-dimensional video data corresponding to the target user based on the depth image data; A data overlay module is used to overlay the three-dimensional video data onto the three-dimensional model to obtain a holographic video. The target users include at least two users. The video acquisition module is specifically used to acquire first depth image data corresponding to each of the at least two users; based on the first depth image data corresponding to each user, generate reference three-dimensional data corresponding to each user; the data overlay module is specifically used to perform time calibration on the reference three-dimensional data of the at least two users and simultaneously overlay it onto the three-dimensional stereo model to obtain a holographic video. The depth image data includes second depth data and second pixel data acquired by the second image acquisition device from the target user's body at different viewpoints. The video acquisition module is further used to generate three-dimensional video data corresponding to the target user at different viewpoints based on the second depth data and second pixel data corresponding to the target user's body at different viewpoints within a preset time period. The video generation device also includes a data calibration module, which is used to calibrate the second depth data and second pixel data corresponding to the target user's body at different viewpoints based on the generation time corresponding to the same action of the target user at different viewpoints in the depth image data before generating the three-dimensional video data corresponding to the target user based on the depth image data. The video acquisition module is further configured to acquire third depth data and third pixel data corresponding to the hand of the target user, acquired by the third image acquisition device worn by the target user from the target viewpoint; and generate three-dimensional video data of the target user from the target viewpoint based on the third depth data and the third pixel data.

8. A computer device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-6.

9. A holographic recording and broadcasting system, characterized in that, include: A video editing apparatus, the video editing apparatus being used to perform the method as described in any one of claims 1-6; A video playback device for playing holographic videos generated by the video editing device.

Images