Bullet time video generation method, device, electronic device and storage medium

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for generating a bullet time video, the method comprising obtaining an original pixel image collected by each of a plurality of real cameras and a first depth image corresponding to the original pixel image; determining a reference camera corresponding to each virtual camera from a plurality of real cameras according to the deployment position information corresponding to each of a plurality of virtual cameras and a plurality of real cameras; constructing a target pixel image corresponding to each virtual camera based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera; generating a bullet time video based on the target pixel images corresponding to the plurality of virtual cameras. The embodiments of the present disclosure can improve the fluency and richness of the bullet time video on the basis of effectively reducing the deployment cost of on-site equipment, the complexity of the system and the difficulty of deployment.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a bullet time video generation method, device, electronic device and storage medium. Background Art

[0002] Bullet time video is a special application form of six-degree-of-freedom video. It makes the target scene appear to be in a state of time stagnation or slow motion, and then can present a scene space picture with a large-scale continuous moving perspective, so as to present more information and more shocking effects.

[0003] In the related art, the generation of bullet time videos requires the use of a multi-camera array to synchronously shoot the target scene, first obtaining video data of the target scene at different perspectives, and then arranging the images obtained during the continuous large-scale movement of the lens in the target scene in a certain order to form a bullet time video. However, in the related art, the method of generating bullet time videos by using scene images captured by deployed real cameras requires the deployment of a large number of cameras, resulting in relatively high complexity, deployment difficulty and cost of the overall system. If the complexity, deployment difficulty and cost of the system need to be reduced, the number of deployed cameras needs to be reduced, which in turn affects the effect of the generated bullet time video, resulting in problems such as poor video fluency. Therefore, the related art cannot effectively balance the equipment deployment cost and video effects such as the fluency of the generated bullet time video during the generation of the bullet time video. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device and storage medium for generating a bullet time video, so as to at least solve the technical problem in the related art that it is impossible to effectively balance the equipment deployment cost and the video effects such as the smoothness of the generated bullet time video during the bullet time video generation process. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for generating a bullet time video is provided, comprising:

[0006] Acquire an original pixel image captured by each real camera among a plurality of real cameras and a first depth image corresponding to the original pixel image;

[0007] According to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras, determining a reference camera corresponding to each virtual camera from the plurality of real cameras;

[0008] Constructing a target pixel image corresponding to each virtual camera based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera;

[0009] A bullet time video is generated based on the target pixel images corresponding to the multiple virtual cameras.

[0010] In an optional embodiment, constructing the target pixel image corresponding to each virtual camera based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera includes:

[0011] Constructing a second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera;

[0012] Constructing a reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image;

[0013] The reference pixel images corresponding to each virtual camera are fused to obtain the target pixel images corresponding to each virtual camera.

[0014] In an optional embodiment, constructing the second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera includes:

[0015] Determine, according to each pixel in the first depth image, a first two-dimensional pixel coordinate in the first depth image and the depth data of each pixel, a first three-dimensional coordinate of each pixel in a corresponding reference camera coordinate system;

[0016] Determine, according to the first three-dimensional coordinates corresponding to each pixel point, a second three-dimensional coordinate of each pixel point in a corresponding virtual camera coordinate system;

[0017] Based on the second three-dimensional coordinates corresponding to each pixel point and the length of the position vector corresponding to each pixel point, a second depth image corresponding to each virtual camera is constructed, where the length of the position vector corresponding to each pixel point is the vector length between each pixel point in the corresponding virtual camera coordinate system and the origin of the corresponding virtual camera coordinate system.

[0018] In an optional embodiment, constructing the reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image includes:

[0019] Determine, according to each pixel point in the reference pixel image, the two-dimensional pixel coordinates in the reference pixel image and the depth data of each pixel point in the second depth image, a third three-dimensional coordinate of each pixel point in the corresponding virtual camera coordinate system;

[0020] Determine, according to the third three-dimensional coordinates corresponding to each pixel point, a fourth three-dimensional coordinate of each pixel point in a corresponding reference camera coordinate system;

[0021] Mapping the fourth three-dimensional coordinates corresponding to each pixel point onto an imaging plane of a corresponding reference camera to obtain a second two-dimensional pixel coordinate of each pixel point in a pixel coordinate system;

[0022] Extracting pixel data corresponding to each pixel point from the corresponding original pixel image based on the second two-dimensional pixel coordinates corresponding to each pixel point;

[0023] The reference pixel image corresponding to each virtual camera is generated according to the pixel data corresponding to each pixel point.

[0024] In an optional embodiment, the reference pixel image corresponding to each virtual camera includes a plurality of images; and the fusing process is performed on the reference pixel images corresponding to each virtual camera to obtain the target pixel image corresponding to each virtual camera, including:

[0025] Obtaining preset weight information corresponding to the multiple images;

[0026] The multiple images are weightedly fused based on the preset weight information to obtain the target pixel image corresponding to each virtual camera.

[0027] In an optional embodiment, the method is applied to a cloud server; the obtaining of the original pixel image captured by each real camera of the plurality of real cameras comprises:

[0028] Receive the original pixel image sent by the on-site server.

[0029] In an optional embodiment, the first depth image corresponding to the original pixel image is acquired in the following manner:

[0030] Upon receiving the bullet time video generation request sent by the on-site server, the first depth image is generated according to the original pixel image.

[0031] In an optional embodiment, determining a reference camera corresponding to each virtual camera from the plurality of real cameras according to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras comprises:

[0032] According to the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras, a real camera adjacent to each virtual camera in the multiple real cameras is used as a reference camera corresponding to each virtual camera.

[0033] In an optional embodiment, generating a bullet time video based on the target pixel images corresponding to the multiple virtual cameras includes:

[0034] Based on a preset virtual camera template, compressing the target pixel images corresponding to the multiple virtual cameras to obtain the bullet time video;

[0035] The preset virtual camera template represents the compression order of the target pixel images corresponding to the multiple virtual cameras.

[0036] According to a second aspect of an embodiment of the present disclosure, a bullet time video generating device is provided, comprising:

[0037] An image acquisition module is configured to acquire an original pixel image captured by each real camera of a plurality of real cameras and a first depth image corresponding to the original pixel image;

[0038] A reference camera determination module is configured to determine a reference camera corresponding to each virtual camera from the plurality of real cameras according to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras;

[0039] a target pixel image construction module, configured to construct a target pixel image corresponding to each virtual camera based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera;

[0040] The bullet time video generation module is configured to generate a bullet time video based on the target pixel images corresponding to the multiple virtual cameras.

[0041] In an optional embodiment, the target pixel image construction module includes:

[0042] A second depth image constructing unit is configured to construct a second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera;

[0043] a reference pixel image construction unit, configured to construct a reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image;

[0044] The image fusion processing unit is configured to perform fusion processing on the reference pixel image corresponding to each virtual camera to obtain the target pixel image corresponding to each virtual camera.

[0045] In an optional embodiment, the second depth image construction unit includes:

[0046] A first three-dimensional coordinate determining unit is configured to determine a first three-dimensional coordinate of each pixel point in a corresponding reference camera coordinate system according to each pixel point in the first depth image, a first two-dimensional pixel coordinate in the first depth image and depth data of each pixel point;

[0047] A second three-dimensional coordinate determining unit is configured to determine a second three-dimensional coordinate of each pixel point in a corresponding virtual camera coordinate system according to the first three-dimensional coordinate corresponding to each pixel point;

[0048] A second depth image construction unit is configured to construct a second depth image corresponding to each virtual camera based on the second three-dimensional coordinates corresponding to each pixel point and the length of the position vector corresponding to each pixel point, wherein the length of the position vector corresponding to each pixel point is the vector length between each pixel point in the corresponding virtual camera coordinate system and the origin of the corresponding virtual camera coordinate system.

[0049] In an optional embodiment, the reference pixel image construction unit includes:

[0050] A third three-dimensional coordinate determining unit is configured to determine a third three-dimensional coordinate of each pixel point in the corresponding virtual camera coordinate system according to the two-dimensional pixel coordinates of each pixel point in the reference pixel image and the depth data of each pixel point in the second depth image;

[0051] a fourth three-dimensional coordinate determining unit, configured to determine a fourth three-dimensional coordinate of each pixel point in a corresponding reference camera coordinate system according to the third three-dimensional coordinate corresponding to each pixel point;

[0052] A coordinate mapping unit is configured to map the fourth three-dimensional coordinate corresponding to each pixel point to an imaging plane of a corresponding reference camera to obtain a second two-dimensional pixel coordinate of each pixel point in a pixel coordinate system;

[0053] A pixel data extraction unit is configured to extract pixel data corresponding to each pixel point from the corresponding original pixel image based on the second two-dimensional pixel coordinates corresponding to each pixel point;

[0054] The reference pixel image generating unit is configured to generate the reference pixel image corresponding to each virtual camera according to the pixel data corresponding to each pixel point.

[0055] In an optional embodiment, the reference pixel image corresponding to each virtual camera includes multiple images; and the image fusion processing unit includes:

[0056] A preset weight information acquisition unit, configured to acquire preset weight information corresponding to the plurality of images;

[0057] The weighted fusion unit is configured to perform weighted fusion on the multiple images based on the preset weight information to obtain the target pixel image corresponding to each virtual camera.

[0058] In an optional embodiment, the device is arranged in a cloud server; the image acquisition module comprises:

[0059] The original pixel image receiving unit is configured to receive the original pixel image sent by the on-site server.

[0060] In an optional embodiment, the first depth image corresponding to the original pixel image is acquired by the following unit:

[0061] The first depth image generating unit is configured to generate the first depth image according to the original pixel image when receiving the bullet time video generating request sent by the on-site server.

[0062] In an optional embodiment, the reference camera determination module is specifically configured to execute, based on the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras, using the adjacent real camera of each virtual camera in the multiple real cameras as the reference camera corresponding to each virtual camera.

[0063] In an optional embodiment, the bullet time video generation module is specifically configured to perform compression processing on the target pixel images corresponding to the multiple virtual cameras based on a preset virtual camera movement template to obtain the bullet time video;

[0064] The preset virtual camera template represents the compression order of the target pixel images corresponding to the multiple virtual cameras.

[0065] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method as described in any one of the above-mentioned first aspects.

[0066] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the bullet-time video generation methods of the embodiments of the present disclosure.

[0067] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method as described in any one of the above-mentioned first aspects.

[0068] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0069] In the process of generating bullet time videos, the joint deployment of real cameras and virtual cameras can effectively reduce the on-site equipment deployment cost, system complexity and deployment difficulty. In addition, the target pixel image corresponding to each virtual camera can be constructed by combining the original pixel image collected by the real camera and the first depth image corresponding to the original pixel image. Then, based on the target pixel images corresponding to multiple virtual cameras, bullet time videos can be generated. The deployment of virtual cameras can be flexibly combined with actual needs. On the basis of greatly improving the degree of freedom of the lens, the smoothness and richness of the generated bullet time videos are effectively guaranteed.

[0070] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0072] Figure 1 is a schematic diagram of a bullet time video generation system according to an exemplary embodiment;

[0073] Figure 2 is a flowchart of a method for generating a bullet time video according to an exemplary embodiment;

[0074] Figure 3 is a flowchart showing a method of constructing a target pixel image corresponding to each virtual camera based on a first depth image corresponding to a reference camera and an original pixel image corresponding to the reference camera according to an exemplary embodiment;

[0075] Figure 4 is a flowchart showing a method of constructing a second depth image corresponding to each virtual camera according to a first depth image corresponding to a reference camera according to an exemplary embodiment;

[0076] Figure 5 is a flowchart showing a method of constructing a reference pixel image corresponding to each virtual camera according to a second depth image and an original pixel image corresponding to the second depth image according to an exemplary embodiment;

[0077] Figure 6 is a block diagram of a bullet time video generating device according to an exemplary embodiment;

[0078] Figure 7 The invention is a block diagram of an electronic device for generating bullet time video according to an exemplary embodiment. DETAILED DESCRIPTION

[0079] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0080] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0082] See also Figure 1 , Figure 1 is a schematic diagram of a bullet time video generation system according to an exemplary embodiment, which may include a plurality of real cameras 100 , an on-site server 200 and a server 300 .

[0083] In an optional embodiment, multiple real cameras 100 can be used to capture pixel images of the target scene from multiple perspectives. The on-site server 200 can run application software. The on-site server 200 is a server deployed on the site corresponding to the target scene. The application software can be used to perform camera control, data transmission and communication, and other processing. The server 300 can be used to generate pixel images corresponding to the virtual camera in combination with the pixel images captured by the real camera uploaded by the on-site server 200, and generate bullet time videos. Furthermore, the on-site server 200 can be provided with a display 201, and the display 201 can be used to perform on-site operations, such as performing operations for the application software, performing operations that trigger bullet time video generation requests, etc. Optionally, the above-mentioned display 201 can also be independent of the on-site server 200, and accordingly, the display 201 can be directly or indirectly connected to the on-site server 200 via wired or wireless communication ( Figure 1 What is shown in FIG. 2 is a display 201 disposed on the on-site server 200).

[0084] In a specific embodiment, the types of multiple real cameras 100 can be set according to actual application requirements, such as fisheye cameras, pinhole cameras, etc. The field server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers; the server 300 can be a cloud server that provides cloud computing services.

[0085] In addition, it should be noted that Figure 1 What is shown is only one application environment provided by the present disclosure. In actual application, other application environments may also be included, for example, more or fewer virtual cameras may be included.

[0086] In the embodiments of this specification, the above-mentioned multiple real cameras 100, the on-site server 200 and the server 300 can be directly or indirectly connected through wired or wireless communication, and the present disclosure does not limit this.

[0087] In practical applications, the bullet time video generation system can be deployed first. Specifically, the system deployment can include camera array installation (i.e., deploying multiple real cameras on site. The number and deployment location of the real cameras can be set in combination with actual application requirements), on-site server preparation, communication network construction between devices, camera frame adjustment (i.e., camera composition adjustment, such as focusing to make the lens closer or farther, or moving or rotating the camera to make the real-time image of the camera meet the requirements) and camera parameters (i.e., camera shooting related parameters, such as video resolution, frame rate, exposure, white balance, etc.) adjustment. Next, camera calibration can be performed (i.e., calibrating the camera parameters such as the internal and external parameters of the deployed multiple real cameras. The internal parameters of the camera include the focal length and principal point of the camera. The external parameters of the camera refer to the rotation matrix used to describe the camera posture and the translation vector used to describe the camera position; optionally, the camera parameters of the fisheye camera can also include distortion coefficients), virtual camera movement templates for producing bullet time videos (i.e., virtual camera movement data of pre-defined virtual cameras. Pre-editing virtual camera movement templates can improve the efficiency of subsequent bullet time video generation processes), and image format and storage location settings for camera acquisition.

[0088] Figure 2 is a flowchart of a method for generating a bullet time video according to an exemplary embodiment. The method can be applied to electronic devices such as servers, such as Figure 2 As shown, the method may include the following steps:

[0089] In step S201, an original pixel image captured by each real camera among a plurality of real cameras and a first depth image corresponding to the original pixel image are obtained.

[0090] In a specific embodiment, multiple real cameras are deployed in sequence at the scene of the target scene to capture images of the target scene under different visual perspectives. Specifically, the original pixel image captured by any real camera can be an image of the target scene under the visual perspective of the real camera. In an optional embodiment, the above method is applied to a cloud server; accordingly, the above acquisition of the original pixel image captured by each of the multiple real cameras may include:

[0091] Receives raw pixel images sent by the live server.

[0092] In a specific embodiment, the user of the on-site server uploads the original pixel images of the preset time captured by multiple real cameras to the cloud server according to actual application requirements, and then generates the bullet time video of the preset time. Optionally, the preset time can be a preset moment or a preset time period.

[0093] In an optional embodiment, the first depth image corresponding to the original pixel image is obtained in the following manner:

[0094] When a bullet time video generation request is received from a live server, a first depth image is generated according to the original pixel image.

[0095] In a specific embodiment, after the original pixel image (the screen image of the target scene captured by multiple real cameras in different visual conditions within a preset time) is uploaded to the cloud server, a bullet time video generation request can be sent to the cloud server. Accordingly, the cloud server can allocate computing resources for the bullet time video generation task and start a new bullet time video generation task (i.e., start the operation of generating a first depth image according to the original pixel image corresponding to the bullet time video generation request). Optionally, the depth estimation of the original pixel image captured by each real camera can be performed in combination with a stereo matching algorithm (PatchMatch) to obtain a first depth image. Specifically, the first depth image corresponding to any original pixel image can be the depth image of the original pixel image.

[0096] In an optional embodiment, after starting a video generation task, the cloud server can monitor the status of the video generation task, so that when it monitors that the bullet time video generation corresponding to the video generation task is completed, it notifies the on-site server that the bullet time production is completed and informs the video download address. In addition, since the cloud server itself has more computing resources, it can respond to new bullet time video generation requests in parallel while performing a certain video generation task, and thus can execute multiple video generation tasks in parallel.

[0097] In the above embodiment, the method of generating bullet time videos by combining an on-site server with a cloud server can effectively reduce the number of on-site devices and the amount of resources occupied by on-site devices, thereby greatly reducing equipment costs and deployment difficulties. At the same time, it can also be combined with a cloud server to greatly improve the stability, concurrency and production timeliness of the bullet video generation process.

[0098] In step S203, according to the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras, a reference camera corresponding to each virtual camera is determined from the multiple real cameras;

[0099] In an optional embodiment, the step of determining the reference camera corresponding to each virtual camera from the plurality of real cameras according to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras may include:

[0100] According to the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras, the adjacent real camera of each virtual camera in the multiple real cameras is used as the reference camera corresponding to each virtual camera.

[0101] In a specific embodiment, the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras can reflect the relative position relationship between each virtual camera and the multiple real cameras, and then the adjacent real cameras of each virtual camera can be determined in combination with the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras; the reference camera corresponding to each virtual camera can include multiple real cameras. Optionally, the nearest real camera to the left and the nearest real camera to the right of each virtual camera can be used as adjacent real cameras, or the two nearest real cameras to the left and the two nearest real cameras to the right of the virtual camera can be used as adjacent real cameras. The selection rules of adjacent real cameras can be pre-set in combination with actual application requirements. Optionally, one or more virtual cameras can be deployed between two real cameras.

[0102] In the above embodiment, the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras is combined, and the adjacent real camera of each virtual camera in the multiple real cameras is used as the reference camera corresponding to each virtual camera. This can ensure that the original pixel image used to construct the pixel image corresponding to the virtual camera can effectively include the visual image corresponding to the virtual camera, thereby better improving the richness and smoothness of the generated bullet time video.

[0103] In an optional embodiment, in order to avoid a large difference between the original pixel image vision captured by the adjacent real cameras and the corresponding vision of the virtual camera due to improper deployment of adjacent real cameras, the posture information of the virtual camera and the real camera can be combined to screen the adjacent real cameras to more accurately select the reference camera.

[0104] In step S205, a target pixel image corresponding to each virtual camera is constructed based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera;

[0105] In an optional embodiment, if Figure 3 As shown, the above-mentioned process of constructing a target pixel image corresponding to each virtual camera based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera may include the following steps:

[0106] In step S301, a second depth image corresponding to each virtual camera is constructed according to a first depth image corresponding to a reference camera;

[0107] In step S303, a reference pixel image corresponding to each virtual camera is constructed according to the second depth image and the original pixel image corresponding to the second depth image;

[0108] In step S305, a fusion process is performed on the reference pixel images corresponding to each virtual camera to obtain a target pixel image corresponding to each virtual camera.

[0109] In an optional embodiment, if Figure 4 As shown, the above-mentioned construction of the second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera may include the following steps:

[0110] In step S3011, according to each pixel in the first depth image, the first two-dimensional pixel coordinates in the first depth image and the depth data of each pixel, a first three-dimensional coordinate of each pixel in the corresponding reference camera coordinate system is determined;

[0111] In step S3013, according to the first three-dimensional coordinates corresponding to each pixel point, the second three-dimensional coordinates of each pixel point in the corresponding virtual camera coordinate system are determined;

[0112] In step S3015, a second depth image corresponding to each virtual camera is constructed based on the second three-dimensional coordinates corresponding to each pixel point and the length of the position vector corresponding to each pixel point.

[0113] In a specific embodiment, the first two pixel coordinates corresponding to each pixel in the first depth image may be the two-dimensional coordinates of each pixel in the first depth image. The depth data of each pixel may be the depth value of each pixel in the first depth image in the first depth image. In practical applications, in the process of converting the two-dimensional coordinates on the image into three-dimensional coordinates, the camera imaging principle may be combined to map each pixel in the first depth image to a reference camera coordinate system, and the depth information of each pixel in the first depth image may be converted into the position vector length of each pixel in the reference camera coordinate system, so as to obtain the first three-dimensional coordinates of each pixel in the corresponding reference camera coordinate system. Specifically, the position vector length of each pixel in the reference camera coordinate system may be the length of the vector between the pixel in the reference camera coordinate system and the origin of the reference camera coordinate system. Specifically, in the above process of mapping each pixel in the first depth image to the reference camera coordinate system and converting the depth information of each pixel in the first depth image into the position vector length of each pixel in the reference camera coordinate system, the corresponding transformation matrix may be determined in combination with the camera parameters.

[0114] In a specific embodiment, determining the second three-dimensional coordinates of each pixel point in the corresponding virtual camera coordinate system according to the first three-dimensional coordinates corresponding to each pixel point may include: combining the coordinate conversion matrix between the reference camera coordinate system and the virtual camera coordinate system to convert the first three-dimensional coordinates into the second three-dimensional coordinates. Specifically, the coordinate conversion matrix between the reference camera coordinate system and the virtual camera coordinate system may be pre-generated in combination with the camera parameters of the reference camera and the virtual camera.

[0115] In a specific embodiment, the position vector length corresponding to each pixel point can be the vector length between each pixel point in the corresponding virtual camera coordinate system and the origin of the corresponding virtual camera coordinate system; specifically, the above-mentioned construction of the second depth image corresponding to each virtual camera based on the second three-dimensional coordinates corresponding to each pixel point and the position vector length corresponding to each pixel point can include: through the camera imaging principle, mapping the second three-dimensional coordinates corresponding to each pixel point to the imaging plane of the virtual camera, obtaining the two-dimensional prime point coordinates corresponding to the second depth image, and converting the position vector length corresponding to each pixel point into a depth value and storing each pixel point at the two-dimensional prime point coordinates corresponding to the second depth image, to obtain the second depth image corresponding to each virtual camera.

[0116] In the above embodiment, three-dimensional reconstruction of pixel points in the virtual pixel coordinate system is performed in combination with the first depth image, and the second three-dimensional coordinates of each pixel point in the corresponding virtual camera coordinate system and the position vector length of each pixel point in the virtual camera coordinate system are combined to achieve reconstruction of the second depth image corresponding to each virtual camera, thereby achieving reconstruction of a large number of depth images of virtual cameras based on the depth images of a small number of real cameras, which greatly reduces the cost of on-site equipment deployment, the complexity of the system and the difficulty of deployment, and effectively guarantees the video effect of the subsequently generated bullet time video.

[0117] In an optional embodiment, if Figure 5 As shown, the construction of the reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image may include the following steps:

[0118] In step S3031, according to each pixel point in the reference pixel image, the two-dimensional pixel coordinates in the reference pixel image and the depth data of each pixel point in the second depth image, determine the third three-dimensional coordinates of each pixel point in the corresponding virtual camera coordinate system;

[0119] In step S3033, according to the third three-dimensional coordinates corresponding to each pixel point, the fourth three-dimensional coordinates of each pixel point in the corresponding reference camera coordinate system are determined;

[0120] In step S3035, the fourth three-dimensional coordinates corresponding to each pixel point are mapped to the imaging plane of the corresponding reference camera to obtain the second two-dimensional pixel coordinates of each pixel point in the pixel coordinate system;

[0121] In step S3037, based on the second two-dimensional pixel coordinates corresponding to each pixel point, pixel data corresponding to each pixel point is extracted from the corresponding original pixel image;

[0122] In step S3039, a reference pixel image corresponding to each virtual camera is generated according to the pixel data corresponding to each pixel point.

[0123] In a specific embodiment, the specific refinement of determining the third three-dimensional coordinates of each pixel point in the corresponding virtual camera coordinate system according to the reference pixel image, the two-dimensional pixel coordinates in the reference pixel image and the depth data of each pixel point in the second depth image can be referred to the above-mentioned specific refinement of determining the first three-dimensional coordinates of each pixel point in the corresponding reference camera coordinate system according to the first two-dimensional pixel coordinates in the first depth image and the depth data of each pixel point, which will not be repeated here.

[0124] In a specific embodiment, determining the fourth three-dimensional coordinates of each pixel point in the corresponding reference camera coordinate system based on the third three-dimensional coordinates corresponding to each pixel point may include combining the coordinate transformation matrix between the virtual camera coordinate system and the reference camera coordinate system to convert the third three-dimensional coordinates corresponding to each pixel point in the reference pixel image into the second two-dimensional pixel coordinates corresponding to each pixel point in the reference pixel image.

[0125] In a specific embodiment, the pixel data corresponding to each pixel point can be extracted from the corresponding original pixel image in combination with the second two-dimensional pixel coordinates corresponding to each pixel point in the reference pixel image; further, since each virtual camera corresponds to multiple reference cameras, a reference pixel image is constructed accordingly in combination with the pixel data in the original pixel image collected from each reference camera.

[0126] In the above embodiment, in the process of determining the pixel data of the reference pixel image corresponding to the virtual camera, the three-dimensional reconstruction of the virtual camera coordinate system can be performed by combining the two-dimensional pixel coordinates of each pixel in the reference pixel image and the depth data of each pixel in the second depth image, and the third three-dimensional coordinates of each pixel in the virtual camera coordinate system are combined to perform coordinate transformation to obtain the fourth three-dimensional coordinates of each pixel in the reference camera coordinate system, and then the fourth three-dimensional coordinates can be mapped to the imaging plane of the corresponding reference camera to obtain the second two-dimensional pixel coordinates of each pixel in the pixel coordinate system, and the pixel data corresponding to each pixel is extracted from the corresponding original pixel image in combination with the second two-dimensional pixel coordinates corresponding to each pixel, and the pixel image of the real camera can be accurately combined to determine the pixel data of the reference pixel image used to generate the virtual camera pixel image, and then the pixel images of a large number of virtual cameras can be reconstructed based on the pixel images of a small number of real cameras, which greatly reduces the cost of on-site equipment deployment, the complexity of the system and the difficulty of deployment, and effectively guarantees the video effect of the subsequently generated bullet time video.

[0127] In a specific embodiment, since each virtual camera corresponds to multiple reference cameras, the reference pixel image corresponding to each virtual camera includes multiple images; accordingly, the above-mentioned fusion processing of the reference pixel images corresponding to each virtual camera to obtain the target pixel image corresponding to each virtual camera may include:

[0128] Obtain preset weight information corresponding to multiple images;

[0129] Based on the preset weight information, multiple images are weightedly fused to obtain the target pixel image corresponding to each virtual camera.

[0130] In an optional embodiment, the above-mentioned preset weight information can be set in combination with actual application requirements. Optionally, the preset weight information corresponding to the image is negatively correlated with the distance between the reference camera corresponding to the image and the virtual camera, that is, the closer the distance between the reference camera corresponding to any image and the virtual camera, the larger the preset weight information corresponding to the image. Conversely, the farther the distance between the reference camera corresponding to any image and the virtual camera, the smaller the preset weight information corresponding to the image.

[0131] In the above embodiment, combined with preset weight information, weighted fusion is performed on multiple images containing pixel data extracted from original pixel images corresponding to multiple reference cameras, which can effectively ensure the validity of the target pixel image of the generated virtual camera, thereby improving the smoothness and richness of the subsequently generated bullet time video.

[0132] In addition, it should be noted that in actual applications, there may be some hole positions in the reference pixel image where pixel data is not obtained, resulting in some hole positions in the target pixel image where there is no pixel data; optionally, pixel data around the hole positions can be taken to fill them to make the target pixel image complete.

[0133] In the above embodiment, a second depth image corresponding to each virtual camera is constructed in combination with a first depth image corresponding to a real reference camera, and then a reference pixel image containing pixel data for reconstructing a target pixel image corresponding to each virtual camera is constructed in combination with the second depth image and the original pixel image corresponding to the second depth image. Then, the target pixel image corresponding to each virtual camera can be obtained by fusing the reference pixel image corresponding to each virtual camera, thereby reconstructing pixel images of a large number of virtual cameras based on pixel images and depth images of a small number of real cameras, thereby greatly reducing the cost of on-site equipment deployment, the complexity of the system, and the difficulty of deployment, and effectively ensuring the video effect of the subsequently generated bullet time video.

[0134] In step S207, a bullet time video is generated based on the target pixel images corresponding to the multiple virtual cameras.

[0135] In an optional embodiment, the generation of the bullet time video based on the target pixel images corresponding to the multiple virtual cameras may include:

[0136] Based on the preset virtual camera template, the target pixel images corresponding to multiple virtual cameras are compressed to obtain the bullet time video.

[0137] In a specific embodiment, the preset virtual camera template may represent the compression order of target pixel images corresponding to multiple virtual cameras. Accordingly, the target pixel images corresponding to multiple virtual cameras may be compressed in sequence in combination with the compression order of target pixel images corresponding to multiple virtual cameras represented in the preset virtual camera template to obtain a bullet time video.

[0138] In the above embodiment, the target pixel images corresponding to the multiple virtual cameras are compressed by combining a preset virtual camera movement template that represents the compression order of the target pixel images corresponding to the multiple virtual cameras. Different bullet time videos can be flexibly generated by configuring the preset virtual camera movement template, and the pre-setting of the virtual camera movement template can greatly improve the efficiency of the bullet time video generation process.

[0139] It can be seen from the technical solutions provided in the above embodiments of this specification that in the process of generating bullet time videos in the embodiments of this specification, the on-site equipment deployment cost, system complexity and deployment difficulty can be effectively reduced by jointly deploying real cameras and virtual cameras. In addition, by combining the original pixel image captured by the real camera and the first depth image corresponding to the original pixel image, a target pixel image corresponding to each virtual camera can be constructed, and then a bullet time video can be generated based on the target pixel images corresponding to multiple virtual cameras. The deployment of virtual cameras can be flexibly combined with actual needs, and the smoothness and richness of the generated bullet time video can be effectively guaranteed on the basis of greatly improving the degree of freedom of the lens.

[0140] Figure 6 is a block diagram of a device for generating a bullet time video according to an exemplary embodiment. Figure 6 , the device comprises:

[0141] The image acquisition module 610 is configured to acquire an original pixel image captured by each real camera of a plurality of real cameras and a first depth image corresponding to the original pixel image;

[0142] The reference camera determination module 620 is configured to determine a reference camera corresponding to each virtual camera from a plurality of real cameras according to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras;

[0143] The target pixel image construction module 630 is configured to construct a target pixel image corresponding to each virtual camera based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera;

[0144] The bullet time video generation module 640 is configured to generate a bullet time video based on target pixel images corresponding to multiple virtual cameras.

[0145] In an optional embodiment, the target pixel image construction module 630 includes:

[0146] A second depth image construction unit is configured to construct a second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera;

[0147] A reference pixel image construction unit is configured to construct a reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image;

[0148] The image fusion processing unit is configured to perform fusion processing on the reference pixel image corresponding to each virtual camera to obtain the target pixel image corresponding to each virtual camera.

[0149] In an optional embodiment, the second depth image construction unit includes:

[0150] A first three-dimensional coordinate determining unit is configured to determine a first three-dimensional coordinate of each pixel point in a corresponding reference camera coordinate system according to each pixel point in the first depth image, a first two-dimensional pixel coordinate in the first depth image and depth data of each pixel point;

[0151] A second three-dimensional coordinate determining unit is configured to determine a second three-dimensional coordinate of each pixel point in a corresponding virtual camera coordinate system according to the first three-dimensional coordinate corresponding to each pixel point;

[0152] The second depth image construction unit is configured to construct a second depth image corresponding to each virtual camera based on the second three-dimensional coordinates corresponding to each pixel point and the length of the position vector corresponding to each pixel point, wherein the length of the position vector corresponding to each pixel point is the vector length between each pixel point in the corresponding virtual camera coordinate system and the origin of the corresponding virtual camera coordinate system.

[0153] In an optional embodiment, the reference pixel image construction unit includes:

[0154] A third three-dimensional coordinate determining unit is configured to determine a third three-dimensional coordinate of each pixel point in the corresponding virtual camera coordinate system according to the two-dimensional pixel coordinate of each pixel point in the reference pixel image and the depth data of each pixel point in the second depth image;

[0155] a fourth three-dimensional coordinate determining unit, configured to determine a fourth three-dimensional coordinate of each pixel point in a corresponding reference camera coordinate system according to the third three-dimensional coordinate corresponding to each pixel point;

[0156] A coordinate mapping unit is configured to map the fourth three-dimensional coordinate corresponding to each pixel point to an imaging plane of the corresponding reference camera to obtain a second two-dimensional pixel coordinate of each pixel point in a pixel coordinate system;

[0157] A pixel data extraction unit is configured to extract pixel data corresponding to each pixel point from the corresponding original pixel image based on the second two-dimensional pixel coordinates corresponding to each pixel point;

[0158] The reference pixel image generation unit is configured to generate a reference pixel image corresponding to each virtual camera according to the pixel data corresponding to each pixel point.

[0159] In an optional embodiment, the reference pixel image corresponding to each virtual camera includes multiple images; and the image fusion processing unit includes:

[0160] A preset weight information acquisition unit is configured to acquire preset weight information corresponding to a plurality of images;

[0161] The weighted fusion unit is configured to perform weighted fusion on multiple images based on preset weight information to obtain a target pixel image corresponding to each virtual camera.

[0162] In an optional embodiment, the device is arranged in a cloud server; the image acquisition module comprises:

[0163] The original pixel image receiving unit is configured to receive the original pixel image sent by the on-site server.

[0164] In an optional embodiment, the first depth image corresponding to the original pixel image is acquired by the following unit:

[0165] The first depth image generating unit is configured to generate a first depth image according to the original pixel image when receiving a bullet time video generating request sent by the on-site server.

[0166] In an optional embodiment, the reference camera determination module 620 is specifically configured to execute, based on the deployment position information corresponding to each of the multiple virtual cameras and the multiple real cameras, using the adjacent real camera of each virtual camera in the multiple real cameras as the reference camera corresponding to each virtual camera.

[0167] In an optional embodiment, the bullet time video generation module 640 is specifically configured to perform compression processing on target pixel images corresponding to multiple virtual cameras based on a preset virtual camera template to obtain a bullet time video;

[0168] The preset virtual camera template represents the compression order of target pixel images corresponding to multiple virtual cameras.

[0169] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0170] The method for generating a bullet time video provided in the embodiment of the present application can be executed in a server or a similar computing device. Taking running on a server as an example, Figure 7 FIG. 1 is a block diagram of an electronic device for generating bullet time videos according to an exemplary embodiment. Figure 7As shown, the server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 710 (the processor 710 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 730 for storing data, and one or more storage media 720 (such as one or more mass storage devices) for storing application programs 723 or data 722. Among them, the memory 730 and the storage medium 720 can be short-term storage or permanent storage. The program stored in the storage medium 720 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 710 can be configured to communicate with the storage medium 720 and execute a series of instruction operations in the storage medium 720 on the server 700. The server 700 may also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input and output interfaces 740, and / or one or more operating systems 721, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0171] The input / output interface 740 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 700. In one example, the input / output interface 740 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 740 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0172] It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 7 Different configurations shown.

[0173] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0174] In an exemplary embodiment, an electronic device is also provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the bullet-time video generation method as in the embodiment of the present disclosure.

[0175] In an exemplary embodiment, a computer-readable storage medium is also provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the bullet-time video generation method in the embodiment of the present disclosure.

[0176] In an exemplary embodiment, a computer program product including instructions is also provided. When the computer program product is run on a computer, the computer is enabled to perform the bullet-time video generation method in the embodiment of the present disclosure.

[0177] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating a bullet time video, characterized in that: include: Acquire an original pixel image captured by each real camera among a plurality of real cameras and a first depth image corresponding to the original pixel image; According to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras, determining a reference camera corresponding to each virtual camera from the plurality of real cameras; Constructing a second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera; Constructing a reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image; Performing fusion processing on the reference pixel images corresponding to each virtual camera to obtain a target pixel image corresponding to each virtual camera; A bullet time video is generated based on the target pixel images corresponding to the multiple virtual cameras.

2. The method for generating a bullet time video according to claim 1, characterized in that: The constructing the second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera comprises: Determine, according to each pixel in the first depth image, a first two-dimensional pixel coordinate in the first depth image and the depth data of each pixel, a first three-dimensional coordinate of each pixel in a corresponding reference camera coordinate system; Determine, according to the first three-dimensional coordinates corresponding to each pixel point, a second three-dimensional coordinate of each pixel point in a corresponding virtual camera coordinate system; Based on the second three-dimensional coordinates corresponding to each pixel point and the length of the position vector corresponding to each pixel point, a second depth image corresponding to each virtual camera is constructed, where the length of the position vector corresponding to each pixel point is the vector length between each pixel point in the corresponding virtual camera coordinate system and the origin of the corresponding virtual camera coordinate system.

3. The method for generating a bullet time video according to claim 1, characterized in that: The constructing the reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image comprises: Determine, according to each pixel point in the reference pixel image, the two-dimensional pixel coordinates in the reference pixel image and the depth data of each pixel point in the second depth image, a third three-dimensional coordinate of each pixel point in the corresponding virtual camera coordinate system; Determine, according to the third three-dimensional coordinates corresponding to each pixel point, a fourth three-dimensional coordinate of each pixel point in a corresponding reference camera coordinate system; Mapping the fourth three-dimensional coordinates corresponding to each pixel point onto an imaging plane of a corresponding reference camera to obtain a second two-dimensional pixel coordinate of each pixel point in a pixel coordinate system; Extracting pixel data corresponding to each pixel point from the corresponding original pixel image based on the second two-dimensional pixel coordinates corresponding to each pixel point; The reference pixel image corresponding to each virtual camera is generated according to the pixel data corresponding to each pixel point.

4. The method for generating a bullet time video according to claim 1, characterized in that: The reference pixel image corresponding to each virtual camera includes a plurality of images; and the fusion processing is performed on the reference pixel images corresponding to each virtual camera to obtain the target pixel image corresponding to each virtual camera, including: Obtaining preset weight information corresponding to the multiple images; The multiple images are weightedly fused based on the preset weight information to obtain the target pixel image corresponding to each virtual camera.

5. The method for generating a bullet time video according to any one of claims 1 to 4, characterized in that: The method is applied to a cloud server; the step of obtaining an original pixel image captured by each of a plurality of real cameras comprises: Receive the original pixel image sent by the on-site server.

6. The method for generating a bullet time video according to claim 5, characterized in that: The first depth image corresponding to the original pixel image is obtained in the following manner: Upon receiving the bullet time video generation request sent by the on-site server, the first depth image is generated according to the original pixel image.

7. The method for generating a bullet time video according to any one of claims 1 to 4, characterized in that: The step of determining a reference camera corresponding to each virtual camera from the plurality of real cameras according to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras comprises: According to the deployment position information, a real camera adjacent to each virtual camera among the multiple real cameras is used as a reference camera corresponding to each virtual camera.

8. The method for generating a bullet time video according to any one of claims 1 to 4, characterized in that: The generating of the bullet time video based on the target pixel images corresponding to the multiple virtual cameras comprises: Based on a preset virtual camera template, compressing the target pixel images corresponding to the multiple virtual cameras to obtain the bullet time video; The preset virtual camera template represents the compression order of the target pixel images corresponding to the multiple virtual cameras.

9. A bullet time video generation device, characterized in that: include: An image acquisition module is configured to acquire an original pixel image captured by each real camera of a plurality of real cameras and a first depth image corresponding to the original pixel image; A reference camera determination module is configured to determine a reference camera corresponding to each virtual camera from the plurality of real cameras according to the deployment position information corresponding to each of the plurality of virtual cameras and the plurality of real cameras; a target pixel image construction module, configured to construct a target pixel image corresponding to each virtual camera based on the first depth image corresponding to the reference camera and the original pixel image corresponding to the reference camera; The target pixel image construction module includes: a second depth image construction unit, configured to construct a second depth image corresponding to each virtual camera according to the first depth image corresponding to the reference camera; a reference pixel image construction unit, configured to construct a reference pixel image corresponding to each virtual camera according to the second depth image and the original pixel image corresponding to the second depth image; an image fusion processing unit, configured to perform fusion processing on the reference pixel image corresponding to each virtual camera to obtain the target pixel image corresponding to each virtual camera; The bullet time video generation module is configured to generate a bullet time video based on the target pixel images corresponding to the multiple virtual cameras.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the bullet time video generation method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the bullet-time video generating method according to any one of claims 1 to 8.

12. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the bullet-time video generating method according to any one of claims 1 to 8.

Images