Video generation method and device, equipment, medium and product

Abstract

The invention discloses a video generation method and device, equipment, a medium and a product, and relates to the technical field of image processing. The method comprises the following steps: acquiring a two-dimensional video corresponding to a target scene; acquiring depth-of-field information of the target scene through depth-of-field acquisition equipment; for each frame of image in the two-dimensional video, fusing the two-dimensional image information and the depth-of-field information of each pixel point in the image to obtain initial three-dimensional image information; pixel points except a target pixel point in the initial three-dimensional image information are subjected to fuzzy processing, target three-dimensional image information is obtained, and the target pixel point is the viewpoint of the user; and rendering the target three-dimensional image information corresponding to each frame of image to generate a three-dimensional video image so as to realize the three-dimensional effect of the two-dimensional video. The image information and the depth-of-field information are subjected to pixel-level registration, so that the three-dimensional video is realized, the computing resources and time are saved, the time delay when the user watches the live video is reduced, the real watching effect of human eyes is simulated, and the user experience is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method, device, equipment, medium and product for generating a video. Background Art

[0002] With the popularization of the Internet, users have higher requirements for video viewing effects. For example, in the emerging live broadcast industry, if live broadcast videos are limited to two-dimensional images, it will affect the user's viewing experience. For example, when watching a sports game, users can only see a flat picture, which will affect the user's viewing experience. Therefore, how to make the video reflect a scene more realistically is an urgent problem to be solved.

[0003] In the prior art, based on 3D reconstruction technology, the three-dimensional effect of the video is achieved by scanning and modeling the objects in the target scene. Although this can make the video reflect the target scene more realistically, this method requires a lot of computing resources and time costs. When users watch live videos, the delay problem will be very serious. In addition, the three-dimensional effect achieved by scanning and modeling the objects in the target scene is just to make the two-dimensional picture three-dimensional, which has a certain gap with the picture actually viewed by the human eye, affecting the user experience. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device, medium and product for generating a video, which fuses the two-dimensional image information and depth of field information of each pixel point in each frame of the video, blurs the pixels other than the pixels where the user's viewpoint is located, obtains the target three-dimensional image information, renders the target three-dimensional image information corresponding to each frame of the image, and generates a three-dimensional video image to achieve three-dimensionalization of the two-dimensional video, that is, the three-dimensionalization of the video can be achieved by simply aligning the image information and the depth of field information at the pixel level, thereby saving computing resources and time, reducing the delay when the user watches the live video, simulating the real viewing effect of the human eye, and improving the user experience.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a video, comprising:

[0006] Acquire a two-dimensional video corresponding to a target scene, where the two-dimensional video includes multiple frames of images;

[0007] Acquire depth information of the target scene through a depth acquisition device;

[0008] For each frame of the two-dimensional video, the two-dimensional image information and the depth information of each pixel in the image are fused to obtain the initial three-dimensional image information;

[0009] Performing blur processing on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information, where the target pixel point is the pixel point where the user's viewpoint is located;

[0010] The target three-dimensional image information corresponding to each frame of the image is rendered to generate a three-dimensional video image to achieve stereoscopic transformation of the two-dimensional video.

[0011] In a possible implementation, blurring is performed on pixels other than target pixels in the initial three-dimensional image information to obtain target three-dimensional image information, including:

[0012] Identify the user's pupil location and determine the user's viewpoint;

[0013] Determine the target pixel point where the viewpoint is located;

[0014] Gaussian blur processing is performed on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information.

[0015] In a possible implementation, acquiring the depth information of the target scene by a depth acquisition device includes:

[0016] In the target scene, a first preset light is emitted by a depth of field acquisition device;

[0017] When receiving the reflected first preset light, calculating the difference between a first moment when the first preset light is emitted and a second moment when the first preset light is received;

[0018] Determining initial depth of field information of each object in the target scene according to the difference and the propagation speed of the first preset light;

[0019] According to the initial depth information of each object in the target scene, the depth information of each pixel in each frame of the two-dimensional video is generated.

[0020] In a possible implementation, acquiring the depth information of the target scene by a depth acquisition device includes:

[0021] In the target scene, emitting a second preset light beam through a depth of field acquisition device;

[0022] Acquire a light deformation image generated by irradiating the second preset light to the surface of each object in the target scene, wherein the light deformation image includes shape information and position information of the surface of each object in the target scene;

[0023] The depth information of each pixel in the light-deformed image is determined according to the shape information and the position information and a preset depth determination rule, and the pixel in each frame of the two-dimensional video is the same as the pixel of the light-deformed image.

[0024] In a possible implementation, for each frame of an image in a two-dimensional video, the two-dimensional image information and the depth information of each pixel in the image are fused to obtain initial three-dimensional image information, including:

[0025] Preprocessing the depth of field information to obtain target depth of field information, the preprocessing including Gaussian filtering;

[0026] For each frame of the two-dimensional video, the two-dimensional image information of each pixel in the image and the target depth information are fused to obtain the initial three-dimensional image information.

[0027] In a possible implementation, after rendering the target three-dimensional image information corresponding to each frame of image to generate a three-dimensional video image, the method further includes:

[0028] Displays three-dimensional video images.

[0029] In a second aspect, an embodiment of the present application provides a video generation device, including:

[0030] An acquisition module is used to acquire a two-dimensional video corresponding to a target scene, where the two-dimensional video includes multiple frames of images;

[0031] The acquisition module is also used to acquire the depth information of the target scene through the depth acquisition device;

[0032] A fusion module is used to fuse the two-dimensional image information and depth of field information of each pixel in each frame of the two-dimensional video to obtain initial three-dimensional image information;

[0033] A processing module, used for performing blur processing on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information, where the target pixel point is the pixel point where the user's viewpoint is located;

[0034] The rendering module is used to render the target three-dimensional image information corresponding to each frame of the image to generate a three-dimensional video image to achieve stereoscopic transformation of the two-dimensional video.

[0035] In one possible implementation, the processing module is specifically used for:

[0036] Identify the user's pupil location and determine the user's viewpoint;

[0037] Determine the target pixel point where the viewpoint is located;

[0038] Gaussian blur processing is performed on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information.

[0039] In a possible implementation, the acquisition module is specifically used for:

[0040] In the target scene, a first preset light is emitted by a depth of field acquisition device;

[0041] When receiving the reflected first preset light, calculating the difference between a first moment when the first preset light is emitted and a second moment when the first preset light is received;

[0042] Determining initial depth of field information of each object in the target scene according to the difference and the propagation speed of the first preset light;

[0043] According to the initial depth information of each object in the target scene, the depth information of each pixel in each frame of the two-dimensional video is generated.

[0044] In a possible implementation, the acquisition module is specifically used for:

[0045] In the target scene, emitting a second preset light beam through a depth of field acquisition device;

[0046] Acquire a light deformation image generated by irradiating the second preset light to the surface of each object in the target scene, wherein the light deformation image includes shape information and position information of the surface of each object in the target scene;

[0047] The depth information of each pixel in the light-deformed image is determined according to the shape information and the position information and a preset depth determination rule, and the pixel in each frame of the two-dimensional video is the same as the pixel of the light-deformed image.

[0048] In a possible implementation, the fusion module is specifically used for:

[0049] Preprocessing the depth of field information to obtain target depth of field information, the preprocessing including Gaussian filtering;

[0050] For each frame of the two-dimensional video, the two-dimensional image information of each pixel in the image and the target depth information are fused to obtain the initial three-dimensional image information.

[0051] In a possible embodiment, it is characterized in that the device further comprises:

[0052] The display module is used for rendering the target three-dimensional image information corresponding to each frame image to generate a three-dimensional video image and then displaying the three-dimensional video image.

[0053] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising:

[0054] a processor and a memory storing computer program instructions;

[0055] When the processor executes the computer program instructions, any of the above-mentioned methods for generating a video is implemented.

[0056] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, any of the above-mentioned methods for generating a video is implemented.

[0057] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the above-mentioned methods for generating a video.

[0058] The video generation method, device, equipment, medium and product of the embodiments of the present application include: obtaining a two-dimensional video corresponding to a target scene, the two-dimensional video including multiple frames of images; obtaining depth information of the target scene through a depth of field acquisition device; for each frame of the two-dimensional video, fusing the two-dimensional image information and depth information of each pixel in the image to obtain initial three-dimensional image information; blurring the pixels other than the target pixels in the initial three-dimensional image information to obtain target three-dimensional image information, the target pixels being the pixels at the user's viewpoint; rendering the target three-dimensional image information corresponding to each frame of the image to generate a three-dimensional video image to realize stereoscopicization of the two-dimensional video.

[0059] In this way, the two-dimensional image information and depth of field information of each pixel point in each frame of the video are merged, and the pixels other than the pixels where the user's viewpoint is located are blurred to obtain the target three-dimensional image information, and the target three-dimensional image information corresponding to each frame of the image is rendered to generate a three-dimensional video image to realize the three-dimensionalization of the two-dimensional video. That is, the three-dimensionalization of the video can be realized by simply aligning the image information and the depth of field information at the pixel level, which saves computing resources and time, reduces the delay when users watch live videos, simulates the real viewing effect of the human eye, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0061] Figure 1 It is a flowchart of a method for generating a video provided by an embodiment of the present application;

[0062] Figure 2 is a flowchart of a method for generating a video provided by another embodiment of the present application;

[0063] Figure 3 is a flowchart of a method for generating a video provided by another embodiment of the present application;

[0064] Figure 4 is a flowchart of a method for generating a video provided in yet another embodiment of the present application;

[0065] Figure 5 is a structural schematic diagram of a video generation device provided in yet another embodiment of the present application;

[0066] Figure 6 It is a structural diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0067] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0068] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0069] It should be noted that the acquisition, storage, use and processing of data in the embodiments of the present application are in compliance with the relevant provisions of national laws and regulations.

[0070] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0071] With the popularization of the Internet, users have higher requirements for video viewing effects. For example, in the emerging live broadcast industry, if live broadcast videos are limited to two-dimensional images, it will affect the user's viewing experience. For example, when watching a sports game, users can only see a flat picture, which will affect the user's viewing experience. Therefore, how to make the video reflect a scene more realistically is an urgent problem to be solved.

[0072] In the prior art, based on 3D reconstruction technology, the three-dimensional effect of the video is achieved by scanning and modeling the objects in the target scene. Although this can make the video reflect the target scene more realistically, this method requires a lot of computing resources and time costs. When users watch live videos, the delay problem will be very serious. In addition, the three-dimensional effect achieved by scanning and modeling the objects in the target scene is just to make the two-dimensional picture three-dimensional, which has a certain gap with the picture actually viewed by the human eye, affecting the user experience.

[0073] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment, medium and product for generating a video. The method for generating a video provided by the embodiments of the present application obtains target three-dimensional image information by fusing the two-dimensional image information and depth of field information of each pixel point of each frame image in the video, and blurring the pixel points other than the pixel points where the user's viewpoint is located, and rendering the target three-dimensional image information corresponding to each frame image to generate a three-dimensional video image to achieve three-dimensionalization of the two-dimensional video, that is, only by aligning the image information and the depth of field information at the pixel level, the three-dimensionalization of the video can be achieved, which saves computing resources and time, reduces the delay when the user watches the live video, and simulates the effect of real viewing by the human eye, thereby improving the user experience.

[0074] The following first introduces the method for generating a video provided in the embodiment of the present application. Figure 1 As shown, the video generation method provided in the embodiment of the present application includes the following steps.

[0075] S110: Acquire a two-dimensional video corresponding to a target scene, where the two-dimensional video includes multiple frames of images.

[0076] Here, the target scene includes sports games, etc. The two-dimensional video can be a live video or a pre-recorded video. The two-dimensional video can be a video captured by a camera. For example, the camera can be a video acquisition device, a camera on a terminal device, etc., and the specific type of the camera is not limited here.

[0077] In some embodiments, if the 2D video is a live video, the 2D video can be converted into 3D while being acquired. Specifically, each time a frame of the 2D video is acquired, the image is converted into 3D to obtain the final 3D video, thereby achieving the purpose of converting the 2D video into 3D.

[0078] S120: Acquire depth of field information of the target scene through a depth of field acquisition device.

[0079] Here, the depth of field acquisition device includes a depth of field sensor.

[0080] In some embodiments, the depth of field acquisition device is located in front of the camera head of the camera, and the depth of field information of the target scene is acquired through the depth of field acquisition device.

[0081] It should be noted that the depth information and the two-dimensional image of the target scene are collected synchronously, that is, the depth information and the two-dimensional image of the target scene are collected at the same time. Specifically, the image and depth information of the two-dimensional video at the target moment in the target scene are obtained, and the target moment can be any moment in the target time period.

[0082] It is understandable that the depth of field refers to the distance range that a camera can focus on in photography. The greater the depth of field, the clearer and more three-dimensional the scene looks. The depth of field acquisition device can obtain the depth of field data by measuring the depth information.

[0083] S130 . For each frame of the two-dimensional video, fuse the two-dimensional image information and the depth of field information of each pixel in the image to obtain initial three-dimensional image information.

[0084] Specifically, the pixel points of each frame of the two-dimensional video are determined, and the depth information corresponding to the determined pixel points is generated based on the depth information of the target scene obtained by the depth acquisition device. The two-dimensional image information and the depth information of each pixel point are registered and aligned at the pixel level, that is, the depth information of the target scene and the video picture are matched to ensure that each pixel point has corresponding depth information, and the depth effect of the two-dimensional video is adjusted.

[0085] In some embodiments, the depth of field information is the depth of field information obtained after the depth of field obtaining device receives the user's input for setting relevant parameters for obtaining the depth of field information.

[0086] S140 , performing blur processing on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information, where the target pixel point is the pixel point where the user's viewpoint is located.

[0087] In some embodiments, pixel points other than target pixel points in the initial three-dimensional image information are blurred to blur the image information corresponding to the background or foreground, thereby obtaining target three-dimensional image information.

[0088] S150: Render the target three-dimensional image information corresponding to each frame of image to generate a three-dimensional video image, so as to realize stereoscopic transformation of the two-dimensional video.

[0089] Specifically, a rendering engine is used to render the target three-dimensional image information to generate a three-dimensional video image corresponding to each frame image in the two-dimensional video, so as to realize stereoscopicization of the two-dimensional video.

[0090] In this way, the two-dimensional image information and depth of field information of each pixel point in each frame of the video are merged, and the pixels other than the pixels where the user's viewpoint is located are blurred to obtain the target three-dimensional image information, and the target three-dimensional image information corresponding to each frame of the image is rendered to generate a three-dimensional video image to realize the three-dimensionalization of the two-dimensional video. That is, the three-dimensionalization of the video can be realized by simply aligning the image information and the depth of field information at the pixel level, which saves computing resources and time, reduces the delay when users watch live videos, simulates the real viewing effect of the human eye, and improves the user experience.

[0091] Based on this, in some embodiments, Figure 2 As shown, the above S140 may specifically include:

[0092] S141, identifying the pupil position of the user and determining the viewpoint of the user;

[0093] S142, determining the target pixel point where the viewpoint is located;

[0094] S143 . Perform Gaussian blur processing on pixel points other than the target pixel points in the initial three-dimensional image information to obtain the target three-dimensional image information.

[0095] Here, the viewpoint represents the position where the user's pupil is looking directly. In the above S141, specifically, the user's viewpoint can be determined by identifying the user's pupil in real time. In the above S143, Gaussian blur processing usually uses a special function or library to generate a Gaussian kernel. The implementation principle of Gaussian blur is to assign weights to each pixel of the image. These weights are derived from the Gaussian function and are calculated based on the color values ​​of other pixels around the pixel. Specifically, Gaussian blur is the importance of the pixels involved in the blur (the middle pixel and the surrounding pixels), which conforms to the normal distribution, so the weighted average according to the law of normal distribution is the final blur value. The middle pixel has the largest weight, and the weight is smaller as it goes to the edge.

[0096] In some embodiments, for each frame of image in a two-dimensional video, before determining the target three-dimensional image information, the user's pupil position must be identified and the user's viewpoint must be determined, and Gaussian blur processing is performed on the pixel points other than the target pixel points where the viewpoint is located in the initial three-dimensional image information to obtain the target three-dimensional image information, so that after each frame of image is stereoscopicized, the obtained three-dimensional video image is consistent with the effect of being viewed by the human eye.

[0097] In this way, based on the actual situation of the human eye directly viewing the target scene, Gaussian blur processing is performed on the pixel points other than the target pixel points where the viewpoint is located in the initial three-dimensional image information, so that the user can feel a more real and three-dimensional picture, thereby improving the user's viewing experience.

[0098] Based on this, in some embodiments, such as Figure 3 As shown, the above S120 may specifically include:

[0099] S121, in the target scene, emitting a first preset light through a depth of field acquisition device;

[0100] S122, when the reflected first preset light is received, calculating a difference between a first moment when the first preset light is emitted and a second moment when the first preset light is received;

[0101] S123, determining initial depth of field information of each object in the target scene according to the difference and the propagation speed of the first preset light;

[0102] S124: Generate depth information of each pixel in each frame of the two-dimensional video according to the initial depth information of each object in the target scene.

[0103] Here, the first preset light is set in advance, such as a laser. The propagation speed of the first preset light is a fixed value, which refers to the propagation speed in the air.

[0104] Specifically, the above S121 to S124 refer to obtaining depth of field information based on TOF (Time of Flight) technology. TOF is a distance measurement technology based on the principle of time flight, which calculates the distance of the target object by emitting laser or light pulses and measuring the time required for these light rays to be reflected from the emission. This technology is not affected by the color, angle and material of the object.

[0105] It can be understood that the first preset light is emitted by the depth of field acquisition device, and when the first preset light hits an object in the target scene, it is reflected back to the depth of field acquisition device.

[0106] Specifically, the difference in the above S122 may be a time difference or a phase difference.

[0107] In this way, the first preset light is emitted by the depth of field acquisition device, and the depth of field information of each pixel is determined based on the reflection of light. The depth of field information is acquired by the depth of field acquisition device, so that the original two-dimensional video can be made three-dimensional, thereby improving the user experience.

[0108] Based on this, in some embodiments, such as Figure 4 As shown, the above S120 may specifically include:

[0109] S210, emitting a second preset light beam in the target scene through a depth of field acquisition device;

[0110] S220, obtaining a light deformation image generated when the second preset light irradiates the surface of each object in the target scene, where the light deformation image includes shape information and position information of the surface of each object in the target scene;

[0111] S230, determining depth information of each pixel in the light-deformed image according to the shape information and the position information and a preset depth-of-field determination rule, wherein the pixel in each frame of the two-dimensional video is the same as the pixel of the light-deformed image.

[0112] Here, the second preset light is set in advance, such as a laser. The propagation speed of the second preset light is a fixed value, which refers to the propagation speed in the air.

[0113] Specifically, the above S210 to S230 refer to obtaining depth of field information based on structured light technology. Structured light technology obtains depth of field by projecting specific coded patterns (such as grids or stripes) onto the object to be measured, and then capturing the deformation information of these patterns on the surface of the object. This process involves multiple steps and principles: using near-infrared laser as a light source to emit light with certain structural characteristics, these light rays carry coded information and can be patterns in the form of grids, stripes or other forms. Record the deformation caused by these coded light rays projected onto the surface of the object. By analyzing the distortion of the reflected coded pattern and combining the known positional relationship between the light source and the camera, the depth information of each point on the surface of the object is calculated.

[0114] In this way, a second preset light is emitted through the depth of field acquisition device, and the depth of field information of each pixel is determined based on the deformation of light on the surface of the object. The depth of field information is acquired through the depth of field acquisition device, so that the original two-dimensional video can be made three-dimensional, thereby improving the user experience.

[0115] Based on this, in some embodiments, the above S130 may specifically include:

[0116] Preprocessing the depth of field information to obtain target depth of field information, the preprocessing including Gaussian filtering;

[0117] For each frame of the two-dimensional video, the two-dimensional image information of each pixel in the image and the target depth information are fused to obtain the initial three-dimensional image information.

[0118] Among them, Gaussian filtering is a linear smoothing filtering technology widely used in the field of image processing. Its core idea is to use Gaussian function to perform convolution operation on the image, so as to achieve image smoothing and noise reduction. Gaussian filtering usually uses a two-dimensional matrix (i.e., convolution kernel) to replace the continuous Gaussian function, and applies it to each pixel in the image in a discretized manner. The weight distribution of this convolution kernel follows the Gaussian distribution, with the largest weight in the center and gradually decreasing outward. When the Gaussian filter is applied to an image, it replaces the value of each pixel with the weighted average of other pixels in its neighborhood, thereby achieving a smoothing effect.

[0119] In some embodiments, pre-processing may also include denoising processing.

[0120] In this way, the depth of field information is preprocessed and then fused with the two-dimensional image information, so as to improve the quality and accuracy of the depth of field information, thereby improving the accuracy of the fusion and the accuracy of the fusion result.

[0121] Based on this, in some embodiments, after the above S150, the method may further include:

[0122] Displays three-dimensional video images.

[0123] In some embodiments, the rendered image is output to a screen, a virtual reality device, or other media platform for display.

[0124] Specifically, a 3D video image application scenario input by a user is received, and a 3D video image corresponding to each frame image in a 2D video is displayed on a corresponding device or platform according to the 3D video image application scenario to achieve stereoscopic display of the 2D video.

[0125] In some embodiments, the user's pupils are identified in real time, the user's viewpoint is determined, and the viewing angle of the 3D video image is changed. In addition, the viewing angle of the 3D video image can also be changed by the user's input of the viewing angle change, such as by operating the handle of the wearable stereoscopic device to change the viewing angle of the 3D video image.

[0126] In some embodiments, a three-dimensional video image may also be displayed on a two-dimensional screen, and the viewing angle on the screen may be manipulated to watch the video from multiple angles.

[0127] In this way, when watching videos, users do not just stare at the screen to watch two-dimensional videos, but can wear three-dimensional devices such as virtual reality devices and use the handle to smoothly switch the video viewing angle. Users can watch the three-dimensional video corresponding to the two-dimensional video, allowing users to feel a more real and three-dimensional picture, improving the user's viewing experience.

[0128] In the embodiments provided in the present application, depth of field information is introduced to convert the originally two-dimensional live video into a three-dimensional effect during the real-time live broadcast process, allowing users to feel a more realistic and three-dimensional picture, thereby improving the user's viewing experience.

[0129] In the embodiments provided in the present application, the depth of field sensor can obtain depth of field data by measuring depth information, and then process the depth of field data into three-dimensional stereoscopic picture data in real time in combination with the dynamic effects captured by the camera to achieve a three-dimensional effect of the picture.

[0130] Based on the video generation method provided in the above embodiment, the present application also provides a specific implementation of a video generation device. Please refer to the following embodiment.

[0131] See also Figure 5 , the video generation device 300 provided in the embodiment of the present application includes:

[0132] An acquisition module 310 is used to acquire a two-dimensional video corresponding to a target scene, where the two-dimensional video includes multiple frames of images;

[0133] The acquisition module 310 is further used to acquire the depth information of the target scene through a depth acquisition device;

[0134] A fusion module 320 is used to fuse the two-dimensional image information and the depth of field information of each pixel in each frame of the two-dimensional video to obtain initial three-dimensional image information;

[0135] The processing module 330 is used to perform blur processing on the pixels other than the target pixel in the initial three-dimensional image information to obtain the target three-dimensional image information, where the target pixel is the pixel where the user's viewpoint is located;

[0136] The rendering module 340 is used to render the target three-dimensional image information corresponding to each frame of image to generate a three-dimensional video image to realize stereoscopic transformation of a two-dimensional video.

[0137] Based on this, in some embodiments, the processing module 330 may be specifically used to:

[0138] Identify the user's pupil location and determine the user's viewpoint;

[0139] Determine the target pixel point where the viewpoint is located;

[0140] Gaussian blur processing is performed on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information.

[0141] Based on this, in some embodiments, the acquisition module 310 may be specifically used for:

[0142] In the target scene, a first preset light is emitted by a depth of field acquisition device;

[0143] When receiving the reflected first preset light, calculating the difference between a first moment when the first preset light is emitted and a second moment when the first preset light is received;

[0144] Determining initial depth of field information of each object in the target scene according to the difference and the propagation speed of the first preset light;

[0145] According to the initial depth information of each object in the target scene, the depth information of each pixel in each frame of the two-dimensional video is generated.

[0146] Based on this, in some embodiments, the acquisition module 310 may be specifically used for:

[0147] In the target scene, emitting a second preset light beam through a depth of field acquisition device;

[0148] Acquire a light deformation image generated by irradiating the second preset light to the surface of each object in the target scene, wherein the light deformation image includes shape information and position information of the surface of each object in the target scene;

[0149] The depth information of each pixel in the light-deformed image is determined according to the shape information and the position information and a preset depth determination rule, and the pixel in each frame of the two-dimensional video is the same as the pixel of the light-deformed image.

[0150] Based on this, in some embodiments, the fusion module 320 may be specifically used for:

[0151] Preprocessing the depth of field information to obtain target depth of field information, the preprocessing including Gaussian filtering;

[0152] For each frame of the two-dimensional video, the two-dimensional image information of each pixel in the image and the target depth information are fused to obtain the initial three-dimensional image information.

[0153] Based on this, in some embodiments, the apparatus 300 may further include:

[0154] The display module is used for rendering the target three-dimensional image information corresponding to each frame image to generate a three-dimensional video image and then displaying the three-dimensional video image.

[0155] The various modules of the video generation device provided in the embodiment of the present application can realize the functions of the various steps of the video generation method provided above and achieve the corresponding technical effects, which will not be repeated here for the sake of concise description.

[0156] Based on the same inventive concept, an embodiment of the present application also provides an electronic device.

[0157] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0158] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0159] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0160] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0161] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0162] The processor 401 implements any one of the video generation methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0163] In one example, the electronic device may further include a communication interface 403 and a bus 410. Figure 6 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0164] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0165] The bus 410 includes hardware, software or both, coupling the components of the electronic device to each other. For example, but not limitation, the bus may include an accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, an enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, a front side bus (Front Side Bus, FSB), a hypertransport (Hyper Transport, HT) interconnection, an industry standard architecture (Industry Standard Architecture, ISA) bus, an infinite bandwidth interconnection, a low pin count (Linear Predictive Coding, LPC) bus, a memory bus, a micro channel architecture (MicroChannel Architecture, MCA) bus, a peripheral component interconnect (Peripheral Component Interconnect, PCI) bus, a PCI-Express (Peripheral Component Interconnect-X, PCI-X) bus, a serial advanced technology attachment (Serial Advanced Technology Attachment, SATA) bus, a video electronics standard association local (VESA Local Bus, VLB) bus or other suitable buses or a combination of two or more of these. Where appropriate, the bus 410 may include one or more buses. Although the embodiments of the present application describe and illustrate a specific bus, the present application considers any suitable bus or interconnection. The electronic device can execute the video generation method in the embodiments of the present invention, thereby realizing the above-mentioned video generation method.

[0166] In addition, in combination with the video generation method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the video generation methods in the above embodiment is implemented.

[0167] The present application also provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes each process of implementing any one of the above-mentioned video generation method embodiments.

[0168] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0169] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable read-only memory (Erasable ReadOnly Memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0170] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0171] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0172] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for generating a video, It is characterized in that include: Acquire a two-dimensional video corresponding to a target scene, wherein the two-dimensional video includes multiple frames of images; Acquiring depth information of the target scene through a depth acquisition device; For each frame of the image in the two-dimensional video, the two-dimensional image information of each pixel in the image and the depth of field information are fused to obtain initial three-dimensional image information; Performing blur processing on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information, wherein the target pixel point is the pixel point where the user's viewpoint is located; The target three-dimensional image information corresponding to each frame of the image is rendered to generate a three-dimensional video image, so as to realize the stereoscopicization of the two-dimensional video.

2. The method for generating a video according to claim 1, It is characterized in that The step of performing blurring processing on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information includes: Identifying a user's pupil location and determining the user's viewpoint; Determine a target pixel point where the viewpoint is located; Gaussian blur processing is performed on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information.

3. The method for generating a video according to claim 1, It is characterized in that The acquiring the depth information of the target scene by a depth acquisition device includes: In the target scene, emitting a first preset light beam through a depth of field acquisition device; When the reflected first preset light is received, calculating a difference between a first moment when the first preset light is emitted and a second moment when the first preset light is received; Determining initial depth of field information of each object in the target scene according to the difference and the propagation speed of the first preset light; The depth information of each pixel in each frame of the image of the two-dimensional video is generated according to the initial depth information of each object in the target scene.

4. The method for generating a video according to claim 1, It is characterized in that The acquiring the depth information of the target scene by a depth acquisition device includes: In the target scene, emitting a second preset light beam through a depth of field acquisition device; Acquire a light deformation image generated by the second preset light irradiating the surface of each object in the target scene, wherein the light deformation image includes shape information and position information of the surface of each object in the target scene; The depth information of each pixel in the light-deformed image is determined according to the shape information and the position information and a preset depth determination rule, and the pixel in the image of each frame of the two-dimensional video is the same as the pixel of the light-deformed image.

5. The method for generating a video according to claim 1, It is characterized in that For each frame of the image in the two-dimensional video, fusing the two-dimensional image information of each pixel in the image with the depth of field information to obtain initial three-dimensional image information includes: Preprocessing the depth of field information to obtain target depth of field information, wherein the preprocessing includes Gaussian filtering; For each frame of the image in the two-dimensional video, the two-dimensional image information of each pixel in the image and the target depth of field information are fused to obtain initial three-dimensional image information.

6. The method for generating a video according to claim 1, It is characterized in that After rendering the target three-dimensional image information corresponding to each frame of the image to generate a three-dimensional video image, the method further includes: The three-dimensional video image is displayed.

7. A video generation device, It is characterized in that include: An acquisition module, used to acquire a two-dimensional video corresponding to a target scene, wherein the two-dimensional video includes multiple frames of images; The acquisition module is further used to acquire the depth of field information of the target scene through a depth of field acquisition device; A fusion module, for fusing the two-dimensional image information of each pixel in each frame of the image in the two-dimensional video with the depth of field information to obtain initial three-dimensional image information; A processing module, used for performing blur processing on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information, wherein the target pixel point is the pixel point where the user's viewpoint is located; The rendering module is used to render the target three-dimensional image information corresponding to each frame of the image to generate a three-dimensional video image to achieve stereoscopicization of the two-dimensional video.

8. The video generation device according to claim 7, It is characterized in that The processing module is specifically used for: Identifying a user's pupil location and determining the user's viewpoint; Determine a target pixel point where the viewpoint is located; Gaussian blur processing is performed on pixel points other than target pixel points in the initial three-dimensional image information to obtain target three-dimensional image information.

9. The video generation device according to claim 7, It is characterized in that The acquisition module is specifically used for: In the target scene, emitting a first preset light beam through a depth of field acquisition device; When the reflected first preset light is received, calculating a difference between a first moment when the first preset light is emitted and a second moment when the first preset light is received; Determining initial depth of field information of each object in the target scene according to the difference and the propagation speed of the first preset light; The depth information of each pixel in each frame of the image of the two-dimensional video is generated according to the initial depth information of each object in the target scene.

10. The video generation device according to claim 7, It is characterized in that The acquisition module is specifically used for: In the target scene, emitting a second preset light beam through a depth of field acquisition device; Acquire a light deformation image generated by the second preset light irradiating the surface of each object in the target scene, wherein the light deformation image includes shape information and position information of the surface of each object in the target scene; The depth information of each pixel in the light-deformed image is determined according to the shape information and the position information and a preset depth determination rule, and the pixel in the image of each frame of the two-dimensional video is the same as the pixel of the light-deformed image.

11. The video generation device according to claim 7, It is characterized in that The fusion module is specifically used for: Preprocessing the depth of field information to obtain target depth of field information, wherein the preprocessing includes Gaussian filtering; For each frame of the image in the two-dimensional video, the two-dimensional image information of each pixel in the image and the target depth of field information are fused to obtain initial three-dimensional image information.

12. The video generation device according to claim 7, It is characterized in that The device also includes: The display module is used for rendering the target three-dimensional image information corresponding to each frame of the image to generate a three-dimensional video image and then displaying the three-dimensional video image.

13. An electronic device, It is characterized in that The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for generating a video as described in any one of claims 1 to 6 is implemented.

14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for generating a video according to any one of claims 1 to 6 is implemented.

15. A computer program product, It is characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the video generation method as described in any one of claims 1 to 6.

Images