Virtual shooting method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a virtual shooting method and device, equipment and a storage medium, and belongs to the technical field of image processing. The method comprises: based on camera parameters of a virtual camera in a sky ball model, determining a line-of-sight direction of a pixel point in a virtual shot image relative to the virtual camera, the virtual shot image being an image shot in the sky ball model at a shooting view angle of the virtual camera; based on the line-of-sight direction, extracting pixel values of the pixel points from a panoramic background image corresponding to the sky ball model, the panoramic background image being an image used for being mapped to the sky ball model as a virtual shooting background; and based on the pixel values of the pixel points, rendering through a shader to obtain a background image of the virtual shot image. By adopting the scheme provided by the embodiment of the invention, the problem of background distortion can be avoided while the image generation cost is reduced, and the generation quality of the background image is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a virtual shooting method, device, equipment and storage medium. Background Art

[0002] Virtual shooting refers to a video special effects technology that can run in real time or offline and can place real people into virtual scenes.

[0003] In the related art, a panoramic background image is mapped as a texture onto a sky sphere model, and a virtual camera is placed at the center of the sky sphere model. The picture obtained by observing the sky sphere model with the virtual camera is used as the background image, and the background image is used for virtual shooting to obtain a virtual shot image.

[0004] However, since the panoramic background image is texture-unwrapped and mapped to the sky sphere model, once the position of the virtual camera in the sky sphere model is offset, the background image will be distorted, thereby affecting the generation effect of the virtual shooting image. Summary of the invention

[0005] The present application provides a virtual shooting method, apparatus, device and storage medium, and the technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a virtual shooting method, the method comprising:

[0007] Based on camera parameters of a virtual camera in the sky sphere model, determining a line of sight direction of a pixel point in a virtual captured image relative to the virtual camera, wherein the virtual captured image refers to an image captured in the sky sphere model at a shooting angle of the virtual camera;

[0008] Based on the sight line direction, extracting the pixel value of the pixel point from the panoramic background image corresponding to the sky ball model, wherein the panoramic background image is an image mapped onto the sky ball model as a virtual shooting background;

[0009] Based on the pixel values ​​of the pixel points, a background image of the virtual captured image is obtained by shader rendering.

[0010] On the other hand, an embodiment of the present application provides a virtual shooting device, the device comprising:

[0011] A sight line determination module, used to determine the sight line direction of a pixel point in a virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model, wherein the virtual captured image refers to an image captured in the sky sphere model at a shooting angle of the virtual camera;

[0012] A pixel extraction module, used to extract the pixel value of the pixel point from the panoramic background image corresponding to the sky ball model based on the line of sight direction, wherein the panoramic background image is an image mapped onto the sky ball model as a virtual shooting background;

[0013] The first image generation module is used to obtain the background image of the virtual captured image through shader rendering based on the pixel value of the pixel point.

[0014] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the virtual shooting method as described in the above aspects.

[0015] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one computer instruction is stored. The at least one computer instruction is loaded and executed by a processor to implement the virtual shooting method as described in the above aspects.

[0016] On the other hand, an embodiment of the present application provides a computer program product, the computer program product includes at least one computer instruction, and the at least one computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer instruction from the computer-readable storage medium, and the processor executes the at least one computer instruction, so that the computer device executes the virtual shooting method described in the above aspect.

[0017] Different from the related art, in which the panoramic background image is mapped to the sky sphere model, the virtual camera needs to be placed at the center of the sky sphere model. Once the position of the virtual camera is displaced, the background image will be distorted. In the embodiment of the present application, the line of sight direction of the pixel point in the virtual captured image relative to the virtual camera is determined according to the camera parameters of the virtual camera, so that the pixel value of the pixel point is directly extracted from the panoramic background image corresponding to the sky sphere model according to the line of sight direction of the pixel point, without mapping the panoramic background image to the sky sphere model, thereby reducing the image generation cost while avoiding the background distortion problem and improving the generation quality of the background image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A structural block diagram of a computer system provided by an exemplary embodiment of the present application is shown;

[0020] Figure 2 A flowchart of a virtual shooting method provided by an exemplary embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of determining a sight line direction provided by an exemplary embodiment of the present application is shown;

[0022] Figure 4 A flowchart of a virtual shooting method provided by another exemplary embodiment of the present application is shown;

[0023] Figure 5 A flowchart of a process of generating a background image provided by an exemplary embodiment of the present application is shown;

[0024] Figure 6 A flowchart of image sky area rendering provided by an exemplary embodiment of the present application is shown;

[0025] Figure 7 A flowchart of image ground area rendering provided by an exemplary embodiment of the present application is shown;

[0026] Figure 8 A flowchart of calculating the self-luminous intensity provided by an exemplary embodiment of the present application is shown;

[0027] Fig. 9 A flowchart of a process of generating a virtual captured image provided by an exemplary embodiment of the present application is shown;

[0028] Fig.10 A schematic diagram showing a shadow area formed in a sky sphere model provided by an exemplary embodiment of the present application is shown;

[0029] Fig.11 A schematic diagram of a virtual captured image provided by an exemplary embodiment of the present application is shown;

[0030] Fig.12 A structural block diagram of a computer system for applying a virtual shooting method to a film and television production scene provided by an exemplary embodiment of the present application is shown;

[0031] Fig.13 A structural block diagram of a virtual shooting device provided by an exemplary embodiment of the present application is shown;

[0032] Fig.14 A schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that, although the terms first, second, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0037] First, the nouns involved in the embodiments of the present application are briefly introduced:

[0038] Virtual shooting: It is a film and television production method that combines computer graphics, image processing and virtual reality technologies. It simulates real scenes and objects through photography techniques performed in a computer graphics environment to achieve shooting in a virtual environment.

[0039] Sky sphere model: located in the world coordinate system, the center of the sky sphere model is the axis of the world coordinate system, and a virtual camera is set in the sky sphere model. Figure 3 As shown, the sky sphere model 301 is located in the world coordinate system, the center of the sky sphere model 301 is located at the axis of the world coordinate system, and from the front view, the virtual camera 302 is located in the sky sphere model 301.

[0040] Optionally, the sky sphere model has a mapping relationship with the panoramic background map, and in an embodiment of the present application, the sky sphere model includes a sky model area and a ground model area.

[0041] Virtual captured image: refers to an image obtained by capturing a virtual scene with a virtual camera's shooting angle. In the embodiment of the present application, the virtual scene is a sky sphere model. The shooting angle of the virtual camera is determined based on the camera external parameters of the virtual camera. Figure 3 As shown, by photographing the sky ball model 301 with the photographing angle of the virtual camera 302, a virtual photographed image 303 can be obtained (it should be noted that the virtual photographed image does not actually exist in the world coordinate system, and is only shown based on the imaging principle in the figure).

[0042] Panoramic background image: an image with a wide viewing angle, which shows a 360-degree viewing angle of a scene by stitching multiple individual photos into a continuous image. In the embodiment of the present application, the panoramic background image is used to map onto the sky ball model as a virtual shooting background.

[0043] Please refer to Figure 1 , which shows a structural block diagram of a computer system provided by an exemplary embodiment of the present application, and the computer system may include a terminal 110 and a server 120. The terminal 110 and the server 120 perform data communication via a communication network, and optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.

[0044] The terminal 110 is an electronic device installed with an application having an image generation function. The image generation function may be a function of a native application in the terminal 110 or a function of a third-party application; the terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a wearable device or a vehicle-mounted terminal, etc. Figure 1 The example of the terminal 110 being a desktop computer is merely used for illustration, but is not limited to this.

[0045] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In the embodiment of the present application, the server 120 may be a background server of an application having an image generation function.

[0046] In some embodiments, there is data interaction between the server and the terminal. Figure 1As shown, after receiving the panoramic background image uploaded by the user, the terminal 110 sends the panoramic background image, the model parameters of the sky sphere model and the camera parameters of the virtual camera to the server 120 in response to the user's setting operation on the model parameters of the sky sphere model and the camera parameters of the virtual camera, so that the server 120 determines the model coordinates of the sky sphere model and the camera coordinates of the virtual camera in the world coordinate system according to the model parameters of the sky sphere model and the camera parameters of the virtual camera, and determines the line of sight direction of the pixel point in the virtual captured image relative to the virtual camera according to the camera parameters of the virtual camera in the sky sphere model, so as to extract the pixel value of the pixel point from the panoramic background image corresponding to the sky sphere model according to the line of sight direction, and finally after obtaining the pixel value of each pixel point, obtains the background image of the virtual captured image through shader rendering, and returns the background image to the terminal 110 so that the user can use it to generate the virtual captured image later.

[0047] In combination with the above introduction, the virtual shooting method provided in the present application is described. The method can be executed by a server or a terminal, or can be executed by a server and a terminal together.

[0048] Please refer to Figure 2 , which shows a flowchart of a virtual shooting method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method used in a computer device (including a terminal and / or a server) as an example. The method includes the following steps:

[0049] Step 201, based on the camera parameters of the virtual camera in the sky sphere model, determine the sight direction of the pixel points in the virtual captured image relative to the virtual camera, where the virtual captured image refers to the image captured in the sky sphere model from the shooting angle of the virtual camera.

[0050] Optionally, the camera parameters of the virtual camera include camera internal parameters and camera external parameters. The camera internal parameters include the camera focal length and the principal point position; the camera external parameters include position coordinates and rotation matrix, wherein the position coordinates refer to the position of the virtual camera in the world coordinate system, and the rotation matrix refers to the direction of the virtual camera relative to the world coordinate system.

[0051] Optionally, the virtual captured image refers to an image captured in the sky sphere model using a shooting angle of a virtual camera, wherein the shooting angle of the virtual camera is determined based on external parameters of the camera.

[0052] Optionally, according to the imaging principle, starting from the viewpoint, along the line of sight, it will first pass through the virtual imaging plane and then reach the real object, that is, the line of sight connects the viewpoint, the point on the virtual imaging plane and the point on the real object into a line. Based on this, when the sky ball model is photographed by the virtual camera, the computer device can first determine the line of sight direction of each pixel point in the virtual photographed image relative to the virtual camera, and then determine the point coordinates on the sky ball model along the line of sight.

[0053] In some embodiments, the computer device may place the sky ball model in the world coordinate system and place the virtual camera in the sky ball model. Furthermore, the computer device first converts the points in the world coordinate system to the camera coordinate system according to the camera external parameters of the virtual camera, and then projects the points in the camera coordinate system to the image plane according to the internal parameters of the virtual camera, so that the three-dimensional coordinates of each pixel point on the image plane in the camera coordinate system can be obtained. Then, according to the three-dimensional coordinates of the pixel point in the camera coordinate system and the camera coordinates of the virtual camera, the computer device can determine the line of sight direction of the pixel point relative to the virtual camera.

[0054] Indicatively, Figure 3 As shown, the sky ball model 301 is located in the world coordinate system, and the center of the sky ball model 301 is located at the axis of the world coordinate system. From the front view, the virtual camera 302 is located in the sky ball model 301. The sky ball model 301 is photographed from the shooting angle of the virtual camera 302, and a virtual photographed image 303 can be obtained (it should be noted that the virtual photographed image does not actually exist in the world coordinate system, and is only shown based on the imaging principle in the figure). In the camera coordinate system corresponding to the virtual camera 302, the viewpoint of the virtual camera 302 is located at the axis of the camera coordinate system, the virtual photographed image 303 is located in the image UV coordinate system, and the object 305 is the object contained in the shooting angle of the virtual camera 302.

[0055] Among them, the computer device can convert the position coordinates of the object 305 in the world coordinate system into the shooting coordinates of the object 305 in the camera coordinate system according to the mapping relationship between the world coordinate system and the camera coordinate system, so that the shooting coordinates of the object 305 in the camera coordinate system, the display coordinates 304 of the object 305 in the virtual captured image 303 and the viewpoint of the virtual camera 302 are located in the same line of sight direction.

[0056] Step 202 , based on the sight line direction, extract the pixel value of the pixel point from the panoramic background image corresponding to the sky sphere model, the panoramic background image is an image used to be mapped onto the sky sphere model as a virtual shooting background.

[0057] Optionally, a panorama is an image with a wide viewing angle, and a panorama is formed by stitching multiple individual photos into a continuous image to show a 360-degree viewing angle of a scene. Optionally, a panoramic background image is used to map onto a sky ball model as a virtual shooting background. Optionally, a panoramic background image can be an image directly photographed from a real environment, or an image synthesized by artificial intelligence technology (e.g., AIGC (Artificial Intelligence Generated Content) technology), which is not limited in the embodiments of the present application.

[0058] Different from the related art, in which the panoramic background image is directly expanded and mapped onto the sky sphere model, which results in the need to limit the position of the virtual camera in the sky sphere model, in the embodiment of the present application, the panoramic background image is not directly mapped onto the sky sphere model, but the pixel values ​​of the pixel points are extracted from the panoramic background image only according to the mapping relationship between the sky sphere model and the panoramic background image.

[0059] In some embodiments, after obtaining the line of sight direction of the pixel point in the virtual captured image relative to the virtual camera, in order to determine the point coordinates on the sky sphere model along the line of sight direction, the computer device can also convert the line of sight direction into the world coordinate system according to the mapping relationship between the camera coordinate system and the world coordinate system, thereby determining the point coordinates (longitude and latitude) on the sky sphere model corresponding to each pixel point according to the line of sight direction in the world coordinate system, and thereby, based on the point coordinates and the mapping relationship between the sky sphere model and the panoramic background image, the computer device can extract the pixel value of the pixel point from the panoramic background image.

[0060] Step 203 : obtaining a background image of the virtual captured image through shader rendering based on the pixel values ​​of the pixels.

[0061] In some embodiments, after obtaining the pixel value corresponding to each pixel point, the computer device can perform image rendering through a shader to obtain a background image of the virtual captured image.

[0062] Furthermore, by combining the background image with the real photographed object, a virtual photographed image can be obtained, thereby reducing the image shooting cost.

[0063] To summarize, in the embodiments of the present application, by determining the line of sight direction of the pixel point in the virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera, the pixel value of the pixel point is directly extracted from the panoramic background image corresponding to the sky sphere model based on the line of sight direction of the pixel point, without the need to map the panoramic background image onto the sky sphere model. This achieves the goal of reducing the image generation cost while avoiding the background distortion problem and improving the generation quality of the background image.

[0064] In some embodiments, considering that the effect of image capture in a real scene is also affected by lighting (such as shadows formed by sunlight), in order to improve the authenticity of the virtual captured image, the computer device can also adjust the pixel value of each pixel point according to the self-luminous characteristics of different model areas in the sky ball model. The process will be described below through a specific embodiment.

[0065] Please refer to Figure 4 , which shows a flowchart of a virtual shooting method provided by another exemplary embodiment of the present application. This embodiment is described by taking the method used in a computer device (including a terminal and / or a server) as an example. The method includes the following steps:

[0066] Step 401, based on the camera parameters of the virtual camera in the sky sphere model, determine the sight direction of the pixel points in the virtual captured image relative to the virtual camera, where the virtual captured image refers to the image captured in the sky sphere model from the shooting angle of the virtual camera.

[0067] The implementation of this step may refer to step 201, and this embodiment will not be described in detail here.

[0068] Step 402, normalize the sight direction vector corresponding to the sight direction to obtain a sight direction unit vector.

[0069] In some embodiments, after obtaining the sight direction of the pixel point in the virtual captured image relative to the virtual camera, the computer device converts the sight direction vector corresponding to the sight direction to the world coordinate system according to the mapping relationship between the camera coordinate system and the world coordinate system. Furthermore, in order to facilitate pixel extraction, the computer device may first normalize the sight direction vector to obtain a sight direction unit vector.

[0070] Optionally, the gaze direction vector can be expressed as The process of normalizing the sight direction vector can be expressed as

[0071] Step 403: based on the mapping relationship between the sky sphere model and the panoramic background image, the sight direction unit vector is converted from the three-dimensional space to the texture space to obtain the texture coordinates of the sight direction unit vector in the panoramic background image.

[0072] Optionally, the panoramic background image corresponds to the image UV coordinate system, the lower left corner of the panoramic background image is the origin of the UV coordinate system (0,0), the U axis represents the horizontal direction, the V axis represents the vertical direction, and the upper right corner of the panoramic background image corresponds to the UV coordinate (1,1). Optionally, the mapping relationship between the sky sphere model and the panoramic background image is the mapping relationship between the sky sphere model and the UV coordinate system. Among them, the bottom center of the sky sphere model corresponds to the origin of the UV coordinate system.

[0073] In some embodiments, in order to determine the texture coordinates corresponding to the sight direction unit vector in the panoramic background image, the computer device can convert the sight direction unit vector from the three-dimensional space to the texture space according to the mapping relationship between the sky sphere and the panoramic background image.

[0074] Optionally, the sight direction unit vector is a three-dimensional vector, which can be expressed as (x, y, z), and its corresponding UV coordinate can be expressed as (u, v). The mapping relationship can be expressed as

[0075] Step 404: extracting pixel values ​​of the pixels from the panoramic background image based on the texture coordinates.

[0076] In some embodiments, after obtaining the texture coordinates corresponding to the unit vectors of each sight direction, the computer device can extract the pixel value of the pixel point from the panoramic background image according to the texture coordinates. That is, the pixel value of the coordinate point indicated by the texture coordinate can be extracted as the pixel value of the pixel point in the virtual captured image.

[0077] Optionally, the pixel value may be an RGB value. The computer device extracts the red, green and blue three-channel values ​​of the coordinate point indicated by the texture coordinate from the panoramic background image according to the texture coordinate, and uses them as the pixel value of the pixel point in the virtual captured image.

[0078] Step 405, determine the model area corresponding to the pixel point in the sky sphere model and the self-luminous intensity of the model area. The self-luminous intensity is used to characterize the lighting effect of the model area as a light source on the inside of the sky sphere model, and different model areas correspond to different self-luminous intensities.

[0079] In some embodiments, in order to simulate the lighting effects in real scenes, the computer device may first determine the self-luminous intensity corresponding to different model areas in the sky sphere model. The self-luminous intensity characterizes the lighting effect of the model area as a light source on the inside of the sky sphere model, and different model areas correspond to different self-luminous intensities.

[0080] In some embodiments, since the self-luminous intensity of different model areas is different, that is, when the virtual captured image contains multiple model areas, the brightness values ​​of pixels in different image areas will also be different. Therefore, in order to improve the accuracy of the pixel value, the computer device also needs to first determine the model area corresponding to each pixel point in the virtual captured image in the sky sphere model.

[0081] In one possible implementation, a computer device can determine the model area corresponding to the pixel point in the sky sphere model by line of sight calculation based on the line of sight direction corresponding to the pixel point, starting from the viewpoint of the virtual camera, and determine the self-luminous intensity of the model area based on the lighting parameters configured in the sky sphere model.

[0082] Optionally, the sky sphere model may include a sky model area and a ground model area. In a possible implementation, the computer device may construct a three-dimensional space coordinate system, i.e., a world coordinate system, so as to construct the sky sphere model with the axis of the three-dimensional space coordinate system as the sphere center and the input length as the sphere radius. The spherical surface of the sky sphere model is the sky model area, and the horizontal hemispherical surface of the sky sphere model is the ground model area.

[0083] Indicatively, Figure 5 As shown, the sky sphere model includes a sky model area 501 (i.e., the spherical surface of the sky sphere model) and a ground model area 502 (i.e., the horizontal hemispherical surface of the sky sphere model) in a front view.

[0084] In some embodiments, considering that the shooting effect in a real scene is usually affected by ambient light and luminous light sources, the luminous light sources are usually located in the sky area (such as the real sky outdoors, the top lights indoors, etc.), rather than the ground area. Based on this, the self-luminous intensity of the sky model area can be determined based on the ambient light and virtual light sources configured in the sky ball model; the self-luminous intensity of the ground model area can be determined based on the ambient light configured in the sky ball model.

[0085] Optionally, the ambient light configured in the sky sphere model refers to diffuse light, that is, the lighting that is ubiquitous and evenly distributed in the sky sphere model. Ambient light can simulate the light that is weakly shining on objects from all directions, which can increase the overall brightness within the sky sphere model and reduce the intensity of shadows.

[0086] Optionally, the virtual light source configured in the sky ball model can be directional light (used to simulate a light source at infinity, such as sunlight); it can also be a point light source (a light source that emits in all directions from a fixed point); it can also be a surface light source (a light source covering a certain area, used to simulate a large-area light-emitting panel); it can also be other types of light sources, which are not limited to the embodiments of the present application.

[0087] In a possible implementation, the computer device may determine a first self-luminous intensity corresponding to the sky model area according to a first illumination intensity of the ambient light configured in the sky sphere model, a second illumination intensity of the virtual light source, and an illumination direction.

[0088] Optionally, the illumination direction of the virtual light source can be determined based on the pitch angle and heading angle of the virtual light source. The pitch angle is used to indicate the degree of inclination of the virtual light source on the horizontal plane, that is, the angle of inclination upward or downward from the horizontal line. The heading angle is used to indicate the degree of rotation of the virtual light source on the vertical plane, that is, the angle of rotation clockwise or counterclockwise from the north direction.

[0089] Optionally, the self-illumination intensity of the sky model area is only affected by the pitch angle of the virtual light source, while the heading angle of the virtual light source affects the direction of the shadow.

[0090] Optionally, the first illumination intensity of the ambient light can be expressed as iambium (in lux), the second illumination intensity of the virtual light source can be expressed as ilightsource (in lux), and the pitch angle of the virtual light source can be expressed as a pitch (unit: degree (°)), the first self-luminous intensity corresponding to the sky model area can be expressed as e sky , the process of determining the first self-luminous intensity can be expressed as Where β is the conversion factor from illuminance to shader light intensity.

[0091] In a possible implementation, the computer device may determine a second self-luminous intensity corresponding to the ground model area according to the first illumination intensity of the ambient light.

[0092] Optionally, the second self-luminous intensity corresponding to the ground model area can be expressed as e floor , the process of determining the second self-luminous intensity can be expressed as e floor =β·iEnvironment, where β is the conversion factor from illuminance to shader light intensity.

[0093] Optionally, the first illumination intensity of the ambient light, the second illumination intensity of the virtual light source and the illumination direction configured in the sky sphere model can be set by the user, so that the computer device can automatically adjust the system through parameters to calculate the first self-luminous intensity corresponding to the sky model area and the second self-luminous intensity corresponding to the ground model area based on the illumination parameters input by the user.

[0094] Step 406, determining the self-luminous pixel value of the pixel point based on the self-luminous intensity and pixel value corresponding to the pixel point.

[0095] In some embodiments, after determining the model area corresponding to each pixel point in the virtual captured image in the sky sphere model and the self-luminous intensity corresponding to each model area, the computer device can adjust the pixel value according to the self-luminous intensity corresponding to the pixel point to obtain the self-luminous pixel value of the pixel point.

[0096] Optionally, the self-luminous pixel value is used to represent the color of the pixel under light, and the self-luminous pixel value is equal to the self-luminous intensity multiplied by the pixel value.

[0097] In a possible implementation, when the model area corresponding to the pixel point is a sky model area, the computer device can determine the first self-luminous pixel value of the pixel point based on the first self-luminous intensity and the pixel value; when the model area corresponding to the pixel point is a ground model area, the computer device can determine the second self-luminous pixel value of the pixel point based on the second self-luminous intensity and the pixel value.

[0098] Optionally, the first self-luminous pixel value is equal to e sky Multiply the pixel value, the second self-luminous pixel value is equal to e floor Multiplies the pixel value.

[0099] Optionally, when the first light intensity of the ambient light and the second light intensity of the virtual light source increase at the same time, the brightness of each pixel in the background image will increase, that is, the overall brightness of the image will increase; when the first light intensity of the ambient light and the second light intensity of the virtual light source decrease at the same time, the brightness of each pixel in the background image will decrease, that is, the overall brightness of the image will decrease.

[0100] Step 407 : Based on the self-luminous pixel value of the pixel point, a background image of the virtual captured image is obtained by shader rendering.

[0101] In some embodiments, after determining the self-luminous pixel value of each pixel point in the virtual captured image, the computer device may perform image rendering through a shader to obtain a background image corresponding to the virtual captured image.

[0102] Optionally, taking into account the different lighting properties of different model areas, for example, the sky model area has no collision and no projection (that is, the sky will not cast a shadow on the ground model area), and the ground model area has collision, other three-dimensional models can be allowed to form shadows on the ground model area. Therefore, in order to further improve the image quality, the computer device can also use different shaders to render the sky and ground areas in the background image.

[0103] In one possible implementation, the computer device may render the pixel points corresponding to the sky model area through a first shader based on the self-luminous pixel value of each pixel point, and render the pixel points corresponding to the ground model area through a second shader, thereby obtaining a background image of the virtual captured image.

[0104] The rendering mode of the first shader can be set to self-illumination mode (no base color, no light is emitted), and the rendering mode of the second shader can be set to self-illumination mode and reflection mode. It should be noted that since the second shader turns on the reflection mode, the parameters such as highlight, anisotropy, metalness, and clear coating need to be set to 0, and the roughness should be set to 1.

[0105] In the above embodiment, the sight direction unit vector is converted into UV coordinates according to the mapping relationship between the sky ball model and the panoramic background map, so that the pixel value of the pixel point is directly extracted from the panoramic background map according to the UV coordinates, thereby ensuring the accuracy of the pixel value acquisition. In addition, before coloring and rendering, the pixel value of the pixel point is adjusted by the self-luminous intensity of different model areas, and the self-luminous pixel value of the pixel point can be obtained, which increases the lighting effect in the background image and further improves the image quality of the background image.

[0106] Please refer to Figure 5 , which shows a flowchart of a process for generating a background image provided by an exemplary embodiment of the present application.

[0107] In some embodiments, after generating the background image, the computer device first obtains the model parameters of the sky sphere model, the camera parameters of the virtual camera, and the illumination parameters configured in the sky sphere model.

[0108] First, the computer device can calculate the self-luminous intensity S1 corresponding to the sky model area 501 and the self-luminous intensity F1 corresponding to the ground model area 502 through the parameter automatic adjustment system according to the illumination parameters configured by the user. At the same time, the computer device determines the sight direction of each pixel point in the virtual captured image relative to the virtual camera according to the camera parameters of the virtual camera in the sky ball model, and determines the model area corresponding to each pixel point based on the sight direction vector corresponding to the sight direction, and extracts the pixel value of each pixel point from the panoramic background image corresponding to the sky ball model.

[0109] Furthermore, the computer device calculates the self-luminous pixel value corresponding to each pixel point according to the model area and pixel value corresponding to each pixel point, the self-luminous intensity S1 and the self-luminous intensity F1. Then, the background image is obtained by shading and rendering using the first shader with the self-luminous mode turned on and the second shader with the self-luminous mode turned on and the reflection mode (setting the reflection intensity F2).

[0110] Indicatively, Figure 6As shown, it shows a flowchart of image sky area rendering provided by an exemplary embodiment of the present application. For the pixel points in the background image used to characterize the sky area, that is, the pixel points correspond to the sky model area in the sky ball model, the computer device first determines the sight direction of each pixel point in the virtual captured image relative to the virtual camera according to the camera parameters of the virtual camera in the sky ball model, so that when the sight direction vector corresponding to the sight direction points to the sky model area, the pixel point corresponding to the sight direction is determined to be used to characterize the sky area. Then, the computer device first normalizes the sight direction vector to obtain the sight direction unit vector, and extracts the pixel value of the pixel point from the panoramic background image according to the mapping relationship between the sky ball model and the panoramic background image, that is, the mapping relationship between the sky ball model and the UV coordinate system, so as to multiply the pixel value by the first self-luminous intensity corresponding to the sky model area, that is, to obtain the first self-luminous pixel value of the pixel point, and finally gives the first self-luminous pixel value to the first shader for coloring and rendering to generate the background image.

[0111] Indicatively, Figure 7 As shown, it shows a flowchart of image ground area rendering provided by an exemplary embodiment of the present application. For the pixel points in the background image used to characterize the ground area, that is, the pixel points correspond to the ground model area in the sky sphere model, the computer device first determines the sight direction of each pixel point in the virtual captured image relative to the virtual camera according to the camera parameters of the virtual camera in the sky sphere model, so that when the sight direction vector corresponding to the sight direction points to the ground model area, the pixel point corresponding to the sight direction is determined to be used to characterize the ground area. Then, the computer device first normalizes the sight direction vector to obtain the sight direction unit vector, and extracts the pixel value of the pixel point from the panoramic background image according to the mapping relationship between the sky sphere model and the panoramic background image, that is, the mapping relationship between the sky sphere model and the UV coordinate system, so as to multiply the pixel value by the second self-luminous intensity corresponding to the ground model area, that is, to obtain the second self-luminous pixel value of the pixel point, and finally gives the second self-luminous pixel value to the second shader for coloring and rendering to generate the background image.

[0112] In addition, since the ground allows other three-dimensional models to form shadows, the second shader also needs to turn on the reflection mode, that is, the computer device needs to set the virtual light source in the sky ball model according to the first light intensity of the virtual light source configured by the user.

[0113] Indicatively, Figure 8 As shown, it shows a flowchart of calculating the self-luminous intensity provided by an exemplary embodiment of the present application.

[0114] Optionally, the first self-luminous intensity corresponding to the sky model area is related to the first illumination intensity of the virtual light source, the pitch angle of the virtual light source, and the second illumination intensity of the ambient light, and the second self-luminous intensity corresponding to the ground model area is related to the second illumination intensity of the ambient light. Furthermore, the computer device can calculate the first self-luminous intensity corresponding to the sky model area and the second self-luminous intensity corresponding to the ground model area through the parameter automatic adjustment system according to the illumination parameters configured by the user.

[0115] In some embodiments, after obtaining the background image, the computer device can combine the real photographed object with the background image to obtain a complete virtual photographed image. The process will be described below through specific embodiments.

[0116] Please refer to Fig. 9 , which shows a flowchart of a process of generating a virtual captured image provided by an exemplary embodiment of the present application. This embodiment is described by taking the method used in a computer device (including a terminal and / or a server) as an example. The method includes the following steps:

[0117] Step 901: determine a virtual shooting position of a three-dimensional model corresponding to a shooting object in a sky sphere model.

[0118] In some embodiments, in the process of combining the virtual background with the real photographed object to obtain the virtual photographed image, considering that the photographed object in the actual photographed image is usually affected by lighting and produces shadows, in order to improve the authenticity of the virtual photographed image, the computer device can also add a shadow effect corresponding to the real photographed object in the virtual photographed image.

[0119] In a possible implementation, considering that the formation of shadows in an image is related to factors such as the position and intensity of the light source, the position and shape of the object, and the shooting angle, the computer device may first determine the virtual shooting position of the three-dimensional model corresponding to the shot object in the sky sphere model.

[0120] Optionally, when the photographed object is a regular object, the computer device can directly construct a three-dimensional model corresponding to the photographed object in the world coordinate system; when the photographed object is an irregular object (such as a person with different postures), the computer device can also use motion capture technology to collect marker point data installed on the photographed object, convert the marker point data into coordinate data in three-dimensional space, and apply the coordinate data to the three-dimensional human body model, thereby obtaining an accurate three-dimensional human body model.

[0121] Optionally, after determining the three-dimensional model corresponding to the photographed object, the computer device can place the three-dimensional model in the three-dimensional space coordinate system corresponding to the sky sphere model, thereby obtaining the model position coordinates corresponding to the three-dimensional model, that is, the virtual shooting position of the three-dimensional model in the sky sphere model.

[0122] Step 902 : determining an image shadow area corresponding to the photographed object in the background image based on the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model.

[0123] In some embodiments, since the formation of shadows in an image is related to factors such as the position and intensity of the light source, the position and shape of the object, and the shooting angle, in order to determine the image shadow area corresponding to the photographed object in the background image, the computer device needs to obtain the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model.

[0124] Optionally, the model parameters of the three-dimensional model include model shape and model size, the camera parameters include camera internal parameters and camera external parameters, and the lighting parameters configured in the sky sphere model include ambient light parameters and virtual light source parameters.

[0125] Optionally, the position, shape and size of the shadow formed in the three-dimensional space are related to the model parameters of the three-dimensional model, the virtual shooting position and the light source direction. For example, a low-angle light source will produce a long and shallow shadow, and a high-angle light source will produce a short and deep shadow. The color effect, shape and size of the shadow displayed in the two-dimensional image space are related to the shooting angle and lighting parameters. For example, the stronger the light intensity, the higher the contrast of the shadow; for another example, the closer the light source, the clearer the shadow and the higher the contrast, and the farther the light source, the blurrier the shadow and the lower the contrast.

[0126] In a possible implementation, the computer device may first determine the shadow area of ​​the three-dimensional model in the sky sphere model according to the model parameters of the three-dimensional model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model, and the shadow area is formed in the ground model area of ​​the sky sphere model. Further, the computer device renders the image shadow area corresponding to the photographed object in the background image according to the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, and the lighting parameters configured in the sky sphere model.

[0127] Indicatively, Fig.10 As shown, taking the virtual light source as sunlight as an example, the sky ball model includes a sky model area 1001 and a ground model area 1002, wherein the ground model area 1002 allows other three-dimensional models to form shadows. For the three-dimensional model 1004 of the photographed object, the computer device can determine the shadow area 1003 of the three-dimensional model 1004 in the sky ball model according to the model parameters of the three-dimensional model 1004, the virtual shooting position, and the light source direction of the sunlight configured in the sky ball model, and the shadow area 1003 is formed on the ground model area 1002 of the sky ball model.

[0128] Optionally, considering that the lighting configured in the sky sphere model includes both ambient light and virtual light sources, and ambient light and virtual light sources with different lighting intensities will produce different image shadow effects, the computer device also needs to specifically render and determine the image shadow area corresponding to the photographed object in the background image based on camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, the first lighting intensity of the ambient light configured in the sky sphere model, and the second lighting intensity of the virtual light source.

[0129] The brightness of the shadow area of ​​the image is positively correlated with the intensity difference between the second illumination intensity and the first illumination intensity. That is, the greater the intensity difference between the second illumination intensity and the first illumination intensity, the darker and more obvious the image shadow is; conversely, the smaller the intensity difference between the second illumination intensity and the first illumination intensity, the weaker and softer the image shadow area is.

[0130] Step 903: Generate a virtual captured image based on the real captured image, the shadow image area, and the background image corresponding to the captured object.

[0131] In some embodiments, after determining the image shadow area corresponding to the photographed object in the background image, the computer device can generate a virtual photographed image by combining the real photographed image (the image after cutout processing) corresponding to the photographed object, the shadow image area and the background image.

[0132] Indicatively, Fig.11 As shown, the computer device can generate a virtual captured image by combining the real captured image corresponding to the captured object 1102 , the shadow image area 1103 , and the background image 1101 .

[0133] In the above embodiment, by determining the shadow area of ​​the three-dimensional model corresponding to the photographed object in the sky sphere model, and then combining the model parameters, camera parameters and lighting parameters configured in the sky sphere model, the image shadow area corresponding to the photographed object in the background image is determined, thereby improving the authenticity of the shadow formed by the photographed object in the virtual photographed image and further optimizing the image quality of the virtual photographed image.

[0134] Optionally, the virtual shooting method provided in the embodiments of the present application can be applied to a variety of scenarios, such as film and television production scenarios (including film production, TV series production, advertising production, documentary production, etc.), game development scenarios, educational learning scenarios (simulated driving, flight training, etc.), etc.

[0135] For film and television production scenes:

[0136] In order to reduce the cost of film and television production while ensuring the quality of film and television works, the virtual shooting method proposed in the embodiment of the present application can be applied to the film and television production scene. In the film and television production scene, the virtual shooting image is the video image frame, the panoramic background image is the film and television scene image, and the computer device can construct a sky ball model and determine the mapping relationship between the film and television scene image and the sky ball model.

[0137] In some embodiments, the computer device first determines the sight direction of the pixel points in the video image frame relative to the virtual camera according to the camera parameters of the virtual camera in the sky sphere model, and the video image frame refers to the image captured in the sky sphere model from the shooting angle of the virtual camera. Further, based on the sight direction, the computer device extracts the pixel value of the pixel point from the film and television scene graph corresponding to the sky sphere model, and the film and television scene graph is an image used to be mapped to the sky sphere model as a virtual shooting background. Finally, based on the pixel value of the pixel point, the background image of the video image frame can be obtained by shader rendering.

[0138] In one possible implementation, a computer device may first determine the model area corresponding to the pixel point in the sky sphere model and the self-luminous intensity of the model area, where the self-luminous intensity is used to characterize the lighting effect of the model area as a light source on the interior of the sky sphere model, and different model areas correspond to different self-luminous intensities. Based on the self-luminous intensity and pixel value corresponding to the pixel point, the self-luminous pixel value of the pixel point is determined, and then based on the self-luminous pixel value of the pixel point, the background image of the video image frame is obtained through shader rendering.

[0139] Optionally, the computer device can determine the model area corresponding to the pixel point in the sky sphere model according to the line of sight direction corresponding to the pixel point, and determine the self-luminous intensity of the model area based on the lighting parameters configured in the sky sphere model.

[0140] Optionally, the sky sphere model includes a sky model area and a ground model area. The computer device may determine a first self-luminous intensity corresponding to the sky model area based on a first illumination intensity of the ambient light, a second illumination intensity of the virtual light source, and an illumination direction configured in the sky sphere model; and determine a second self-luminous intensity corresponding to the ground model area based on the first illumination intensity of the ambient light.

[0141] Optionally, when the model area corresponding to the pixel point is a sky model area, the computer device determines the first self-luminous pixel value of the pixel point based on the first self-luminous intensity and the pixel value; when the model area corresponding to the pixel point is a ground model area, the computer device determines the second self-luminous pixel value of the pixel point based on the second self-luminous intensity and the pixel value.

[0142] Optionally, the computer device renders the pixel points corresponding to the sky model area through a first shader based on the self-luminous pixel value of the pixel point, and renders the pixel points corresponding to the ground model area through a second shader to obtain a background image of the video image frame. The rendering mode of the first shader is set to the self-luminous mode, and the rendering mode of the second shader is set to the self-luminous mode and the reflection mode.

[0143] In some embodiments, the computer device also needs to first construct a three-dimensional space coordinate system, and construct a sky sphere model with the axis of the three-dimensional space coordinate system as the sphere center and the input length as the sphere radius. The spherical surface of the sky sphere model is the sky model area, and the horizontal hemispherical surface of the sky sphere model is the ground model area.

[0144] Regarding the process of extracting pixel values, in one possible implementation, the computer device can first normalize the line of sight direction vector corresponding to the line of sight direction to obtain the line of sight direction unit vector, and then based on the mapping relationship between the sky sphere model and the film and television scene graph, convert the line of sight direction unit vector from the three-dimensional space to the texture space to obtain the texture coordinates of the line of sight direction unit vector in the film and television scene graph, thereby extracting the pixel value of the pixel point from the film and television scene graph based on the texture coordinates.

[0145] Regarding the process of generating video image frames, in some embodiments, the computer device can first determine the virtual shooting position of the three-dimensional model corresponding to the photographed actor in the sky sphere model, and then determine the image shadow area corresponding to the photographed actor in the background image based on the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters and the lighting parameters configured in the sky sphere model, thereby generating a video image frame based on the real photographed image corresponding to the photographed actor, the shadow image area and the background image.

[0146] Optionally, the computer device can first determine the shadow area of ​​the three-dimensional model in the sky sphere model based on the model parameters of the three-dimensional model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model, and then render the image shadow area corresponding to the photographed actor in the background image based on the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, and the lighting parameters configured in the sky sphere model.

[0147] Optionally, the computer device can also render the image shadow area corresponding to the photographed actor in the background image based on camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, the first light intensity of the ambient light configured in the sky sphere model, and the second light intensity of the virtual light source.

[0148] Indicatively, Fig.12 As shown, it shows a structural block diagram of a computer system for applying a virtual shooting method to a film and television production scene provided by an exemplary embodiment of the present application.

[0149] In some embodiments, there is data interaction between the server 1202 and the terminal 1201. Schematically, as Fig.12 As shown, after receiving the film and television scene image uploaded by the user, the terminal 1201 responds to the user's setting operation on the model parameters of the sky sphere model and the camera parameters of the virtual camera, and sends the film and television scene image, the model parameters of the sky sphere model and the camera parameters of the virtual camera to the server 1202, so that the server 1202 determines the model coordinates of the sky sphere model and the camera coordinates of the virtual camera in the world coordinate system according to the model parameters of the sky sphere model and the camera parameters of the virtual camera, and determines the line of sight direction of the pixel point in the video image frame relative to the virtual camera according to the camera parameters of the virtual camera in the sky sphere model, so as to extract the pixel value of the pixel point from the film and television scene image corresponding to the sky sphere model according to the line of sight direction, and finally after obtaining the pixel value of each pixel point, obtains the background image of the video image frame through shader rendering, and returns the background image to the terminal 1201.

[0150] Furthermore, after receiving the real shot image uploaded by the user, the terminal 1201 performs cutout processing on the real shot image, and constructs a three-dimensional model according to the action posture of the actor in the real shot image, and then determines the virtual shooting position of the three-dimensional model in the sky sphere model, and determines the image shadow area corresponding to the actor in the background image based on the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters and the lighting parameters configured in the sky sphere model, thereby generating a video image frame based on the real shot image corresponding to the actor, the shadow image area and the background image.

[0151] For game development scenarios:

[0152] In order to reduce the cost of game production while ensuring the quality of the game screen, the virtual shooting method proposed in the embodiment of the present application can be applied to the game development scene. In the game development scene, the virtual shooting image is the virtual environment screen, the panoramic background image is the environment scene map, and the computer device can build a sky ball model and determine the mapping relationship between the environment scene map and the sky ball model.

[0153] In some embodiments, the computer device first determines the sight direction of the pixel points in the virtual environment screen relative to the virtual camera according to the camera parameters of the virtual camera in the sky sphere model, and the virtual environment screen refers to the image captured in the sky sphere model from the shooting angle of the virtual camera. Further, based on the sight direction, the computer device extracts the pixel value of the pixel point from the environment scene map corresponding to the sky sphere model, and the environment scene map is an image used to be mapped to the sky sphere model as a virtual shooting background. Finally, based on the pixel value of the pixel point, the background image of the virtual environment screen can be obtained through shader rendering.

[0154] In one possible implementation, a computer device may first determine the model area corresponding to the pixel point in the sky sphere model and the self-luminous intensity of the model area, where the self-luminous intensity is used to characterize the lighting effect of the model area as a light source on the interior of the sky sphere model, and different model areas correspond to different self-luminous intensities. Based on the self-luminous intensity and pixel value corresponding to the pixel point, the self-luminous pixel value of the pixel point is determined, and then based on the self-luminous pixel value of the pixel point, the background image of the virtual environment screen is obtained through shader rendering.

[0155] Optionally, the computer device can determine the model area corresponding to the pixel point in the sky sphere model according to the line of sight direction corresponding to the pixel point, and determine the self-luminous intensity of the model area based on the lighting parameters configured in the sky sphere model.

[0156] Optionally, the sky sphere model includes a sky model area and a ground model area. The computer device may determine a first self-luminous intensity corresponding to the sky model area based on a first illumination intensity of the ambient light, a second illumination intensity of the virtual light source, and an illumination direction configured in the sky sphere model; and determine a second self-luminous intensity corresponding to the ground model area based on the first illumination intensity of the ambient light.

[0157] Optionally, when the model area corresponding to the pixel point is a sky model area, the computer device determines the first self-luminous pixel value of the pixel point based on the first self-luminous intensity and the pixel value; when the model area corresponding to the pixel point is a ground model area, the computer device determines the second self-luminous pixel value of the pixel point based on the second self-luminous intensity and the pixel value.

[0158] Optionally, the computer device renders the pixel points corresponding to the sky model area through a first shader based on the self-luminous pixel value of the pixel point, and renders the pixel points corresponding to the ground model area through a second shader to obtain a background image of the virtual environment screen. The rendering mode of the first shader is set to the self-luminous mode, and the rendering mode of the second shader is set to the self-luminous mode and the reflection mode.

[0159] In some embodiments, the computer device also needs to first construct a three-dimensional space coordinate system, and construct a sky sphere model with the axis of the three-dimensional space coordinate system as the sphere center and the input length as the sphere radius. The spherical surface of the sky sphere model is the sky model area, and the horizontal hemispherical surface of the sky sphere model is the ground model area.

[0160] Regarding the process of extracting pixel values, in one possible implementation, the computer device can first normalize the line of sight direction vector corresponding to the line of sight direction to obtain the line of sight direction unit vector, and then based on the mapping relationship between the sky sphere model and the environment scene graph, convert the line of sight direction unit vector from the three-dimensional space to the texture space to obtain the texture coordinates of the line of sight direction unit vector in the environment scene graph, thereby extracting the pixel value of the pixel point from the environment scene graph based on the texture coordinates.

[0161] Regarding the process of generating a virtual environment screen, in some embodiments, the computer device can first determine the virtual shooting position of the three-dimensional model corresponding to the actor in the sky sphere model, and then determine the image shadow area corresponding to the actor in the background image based on the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters and the lighting parameters configured in the sky sphere model, thereby generating a virtual environment screen based on the real shooting image corresponding to the actor, the shadow image area and the background image.

[0162] Optionally, the computer device can first determine the shadow area of ​​the three-dimensional model in the sky sphere model based on the model parameters of the three-dimensional model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model, and then render the image shadow area corresponding to the photographed actor in the background image based on the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, and the lighting parameters configured in the sky sphere model.

[0163] Optionally, the computer device can also render the image shadow area corresponding to the photographed actor in the background image based on camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, the first light intensity of the ambient light configured in the sky sphere model, and the second light intensity of the virtual light source.

[0164] Please refer to Fig.13 , which shows a structural block diagram of a virtual shooting device provided by an exemplary embodiment of the present application, the device includes:

[0165] The sight line determination module 1301 is used to determine the sight line direction of a pixel point in a virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model, wherein the virtual captured image refers to an image captured in the sky sphere model at a shooting angle of the virtual camera;

[0166] A pixel extraction module 1302 is used to extract the pixel value of the pixel point from the panoramic background image corresponding to the sky ball model based on the line of sight direction, where the panoramic background image is an image mapped onto the sky ball model as a virtual shooting background;

[0167] The first image generation module 1303 is configured to obtain a background image of the virtual captured image through shader rendering based on the pixel values ​​of the pixel points.

[0168] Optionally, the first image generating module 1303 includes:

[0169] an intensity determination unit, used to determine a model area corresponding to the pixel point in the sky sphere model and a self-luminous intensity of the model area, wherein the self-luminous intensity is used to characterize the lighting effect of the model area as a light source on the inside of the sky sphere model, and different model areas correspond to different self-luminous intensities;

[0170] A pixel determination unit, configured to determine a self-luminous pixel value of the pixel point based on the self-luminous intensity corresponding to the pixel point and the pixel value;

[0171] An image generating unit is used to obtain the background image of the virtual captured image through shader rendering based on the self-luminous pixel value of the pixel point.

[0172] Optionally, the strength determination unit is used to:

[0173] Determining the model area corresponding to the pixel point in the sky sphere model according to the sight line direction corresponding to the pixel point;

[0174] The self-luminous intensity of the model area is determined based on the lighting parameters configured in the sky sphere model.

[0175] Optionally, the sky sphere model includes a sky model area and a ground model area; and the strength determination unit is used to:

[0176] Determine a first self-luminous intensity corresponding to the sky model area based on a first illumination intensity of the ambient light, a second illumination intensity of the virtual light source, and an illumination direction configured in the sky sphere model;

[0177] Based on the first illumination intensity of the ambient light, a second self-luminous intensity corresponding to the ground model area is determined.

[0178] Optionally, the pixel determination unit is used to:

[0179] When the model area corresponding to the pixel point is the sky model area, determining a first self-luminous pixel value of the pixel point according to the first self-luminous intensity and the pixel value;

[0180] When the model area corresponding to the pixel point is the ground model area, the second self-luminous pixel value of the pixel point is determined according to the second self-luminous intensity and the pixel value.

[0181] Optionally, the image generating unit is used to:

[0182] Based on the self-luminous pixel value of the pixel point, the pixel point corresponding to the sky model area is rendered by a first shader, and the pixel point corresponding to the ground model area is rendered by a second shader to obtain the background image of the virtual captured image;

[0183] The rendering mode of the first shader is set to the self-luminous mode, and the rendering mode of the second shader is set to the self-luminous mode and the reflection mode.

[0184] Optionally, the device further comprises:

[0185] Coordinate system construction module, used to construct a three-dimensional space coordinate system;

[0186] A model construction module is used to construct the sky sphere model with the axis center of the three-dimensional space coordinate system as the sphere center and the input length as the sphere radius; wherein the spherical surface of the sky sphere model is the sky model area, and the horizontal hemispherical surface of the sky sphere model is the ground model area.

[0187] Optionally, the pixel extraction module 1302 is used to:

[0188] Normalizing the sight direction vector corresponding to the sight direction to obtain a sight direction unit vector;

[0189] Based on the mapping relationship between the sky sphere model and the panoramic background image, the sight direction unit vector is converted from the three-dimensional space to the texture space to obtain the texture coordinates of the sight direction unit vector in the panoramic background image;

[0190] Based on the texture coordinates, the pixel value of the pixel point is extracted from the panoramic background image.

[0191] Optionally, the device further comprises:

[0192] A position determination module, used to determine a virtual shooting position of a three-dimensional model corresponding to a shooting object in the sky sphere model;

[0193] A region determination module, configured to determine an image shadow region corresponding to the photographed object in the background image based on model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and illumination parameters configured in the sky sphere model;

[0194] The second image generating module is used to generate the virtual captured image based on the real captured image corresponding to the captured object, the shadow image area and the background image.

[0195] Optionally, the region determination module is used to:

[0196] Determining a shadow area of ​​the three-dimensional model in the sky sphere model based on the model parameters of the three-dimensional model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model;

[0197] Based on the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, and the lighting parameters configured in the sky sphere model, an image shadow area corresponding to the photographed object in the background image is rendered.

[0198] Optionally, the region determination module is used to:

[0199] Rendering obtains an image shadow area corresponding to the photographed object in the background image based on the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, a first illumination intensity of the ambient light configured in the sky sphere model, and a second illumination intensity of the virtual light source;

[0200] There is a positive correlation between the brightness of the shadow area of ​​the image and the intensity difference between the second light intensity and the first light intensity.

[0201] To summarize, in the embodiments of the present application, by determining the line of sight direction of the pixel point in the virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera, the pixel value of the pixel point is directly extracted from the panoramic background image corresponding to the sky sphere model based on the line of sight direction of the pixel point, without the need to map the panoramic background image onto the sky sphere model. This achieves the goal of reducing the image generation cost while avoiding the background distortion problem and improving the generation quality of the background image.

[0202] It should be noted that: the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0203] Please refer to Fig.14, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. Specifically, the computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including a random access memory 1402 and a read-only memory 1403, and a system bus 1405 connecting the system memory 1404 and the central processing unit 1401. The computer device 1400 may also include a basic input / output system (I / O system) 1406 that helps transmit information between various devices in the computer, and a large-capacity storage device 1407 for storing an operating system 1413, application programs 1414 and other program modules 1415.

[0204] In some embodiments, the basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 such as a mouse and a keyboard for user inputting information. The display 1408 and the input device 1409 are connected to the central processing unit 1401 through an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include an input / output controller 1410 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, a printer, or other types of output devices.

[0205] The mass storage device 1407 is connected to the central processing unit 1401 through a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include a computer readable medium (not shown) such as a hard disk or drive.

[0206] Without loss of generality, the computer readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, cassette, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1404 and mass storage device 1407 can be collectively referred to as memory.

[0207] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1401. The one or more programs contain instructions for implementing the above-mentioned method. The central processing unit 1401 executes the one or more programs to implement the virtual shooting method provided by the above-mentioned various method embodiments.

[0208] According to various embodiments of the present application, the computer device 1400 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1400 can be connected to the network 1411 through the network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 can be used to connect to other types of networks or remote computer systems (not shown).

[0209] An embodiment of the present application further provides a computer-readable storage medium, in which at least one computer instruction is stored. The at least one computer instruction is loaded and executed by a processor to implement the virtual shooting method described in the above embodiment.

[0210] Optionally, the computer readable storage medium may include: ROM, RAM, solid state drives (SSD) or optical disks, etc. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0211] The embodiment of the present application provides a computer program product, which includes at least one computer instruction, and the at least one computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer instruction from the computer-readable storage medium, and the processor executes the at least one computer instruction, so that the computer device executes the virtual shooting method described in the above embodiment.

[0212] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0213] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A virtual shooting method, characterized in that: The method comprises: Based on camera parameters of a virtual camera in the sky sphere model, determining a line of sight direction of a pixel point in a virtual captured image relative to the virtual camera, wherein the virtual captured image refers to an image captured in the sky sphere model at a shooting angle of the virtual camera; Based on the sight line direction, extracting the pixel value of the pixel point from the panoramic background image corresponding to the sky sphere model, wherein the panoramic background image is an image used to be mapped onto the sky sphere model as a virtual shooting background; Based on the pixel values ​​of the pixel points, a background image of the virtual captured image is obtained by shader rendering.

2. The method according to claim 1, characterized in that The step of obtaining the background image of the virtual captured image by rendering using a shader based on the pixel value of the pixel point comprises: Determine a model area corresponding to the pixel point in the sky sphere model and the self-luminous intensity of the model area, wherein the self-luminous intensity is used to characterize the lighting effect of the model area as a light source on the inside of the sky sphere model, and different model areas correspond to different self-luminous intensities; Determine the self-luminous pixel value of the pixel point based on the self-luminous intensity corresponding to the pixel point and the pixel value; Based on the self-luminous pixel value of the pixel point, the background image of the virtual captured image is obtained through shader rendering.

3. The method according to claim 2, characterized in that The determining the model area corresponding to the pixel point in the sky sphere model and the self-luminous intensity of the model area includes: Determining the model area corresponding to the pixel point in the sky sphere model according to the sight line direction corresponding to the pixel point; The self-luminous intensity of the model area is determined based on the lighting parameters configured in the sky sphere model.

4. The method according to claim 3, characterized in that The sky sphere model includes a sky model area and a ground model area; and determining the self-luminous intensity of the model area based on the illumination parameters configured in the sky sphere model includes: Determine a first self-luminous intensity corresponding to the sky model area based on a first illumination intensity of the ambient light, a second illumination intensity of the virtual light source, and an illumination direction configured in the sky sphere model; Based on the first illumination intensity of the ambient light, a second self-luminous intensity corresponding to the ground model area is determined.

5. The method according to claim 4, characterized in that The determining the self-luminous pixel value of the pixel point based on the self-luminous intensity corresponding to the pixel point and the pixel value includes: When the model area corresponding to the pixel point is the sky model area, determining a first self-luminous pixel value of the pixel point according to the first self-luminous intensity and the pixel value; When the model area corresponding to the pixel point is the ground model area, the second self-luminous pixel value of the pixel point is determined according to the second self-luminous intensity and the pixel value.

6. The method according to claim 5, characterized in that The step of obtaining the background image of the virtual captured image by rendering through a shader based on the self-luminous pixel value of the pixel point includes: Based on the self-luminous pixel value of the pixel point, the pixel point corresponding to the sky model area is rendered by a first shader, and the pixel point corresponding to the ground model area is rendered by a second shader to obtain the background image of the virtual captured image; The rendering mode of the first shader is set to the self-luminous mode, and the rendering mode of the second shader is set to the self-luminous mode and the reflection mode.

7. The method according to claim 4, characterized in that The method further comprises: Construct a three-dimensional space coordinate system; The sky sphere model is constructed with the axis of the three-dimensional space coordinate system as the sphere center and the input length as the sphere radius; wherein the spherical surface of the sky sphere model is the sky model area, and the horizontal hemispherical surface of the sky sphere model is the ground model area.

8. The method according to claim 1, characterized in that The step of extracting the pixel value of the pixel point from the panoramic background image corresponding to the sky sphere model based on the line of sight direction includes: Normalizing the sight direction vector corresponding to the sight direction to obtain a sight direction unit vector; Based on the mapping relationship between the sky sphere model and the panoramic background image, the sight direction unit vector is converted from the three-dimensional space to the texture space to obtain the texture coordinates of the sight direction unit vector in the panoramic background image; Based on the texture coordinates, the pixel value of the pixel point is extracted from the panoramic background image.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Determine a virtual shooting position of the three-dimensional model corresponding to the shooting object in the sky sphere model; Determine an image shadow area corresponding to the photographed object in the background image based on model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and lighting parameters configured in the sky sphere model; The virtual captured image is generated based on the real captured image corresponding to the captured object, the shadow image area, and the background image.

10. The method according to claim 9, characterized in that The determining, based on the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and the illumination parameters configured in the sky sphere model, an image shadow area corresponding to the shot object in the background image includes: Determining a shadow area of ​​the three-dimensional model in the sky sphere model based on the model parameters of the three-dimensional model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model; Based on the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, and the lighting parameters configured in the sky sphere model, an image shadow area corresponding to the photographed object in the background image is rendered.

11. The method according to claim 10, characterized in that The rendering to obtain an image shadow area corresponding to the photographed object in the background image based on the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, and the illumination parameters configured in the sky sphere model includes: Rendering obtains an image shadow area corresponding to the photographed object in the background image based on the camera parameters, the shadow area of ​​the three-dimensional model in the sky sphere model, a first illumination intensity of the ambient light configured in the sky sphere model, and a second illumination intensity of the virtual light source; There is a positive correlation between the brightness of the shadow area of ​​the image and the intensity difference between the second light intensity and the first light intensity.

12. A virtual shooting device, characterized in that: The device comprises: A sight line determination module, used to determine the sight line direction of a pixel point in a virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model, wherein the virtual captured image refers to an image captured in the sky sphere model at a shooting angle of the virtual camera; A pixel extraction module, used to extract the pixel value of the pixel point from the panoramic background image corresponding to the sky ball model based on the line of sight direction, wherein the panoramic background image is an image mapped onto the sky ball model as a virtual shooting background; The first image generation module is used to obtain the background image of the virtual captured image through shader rendering based on the pixel value of the pixel point.

13. A computer device, characterized in that: The computer device includes a processor and a memory; the memory stores at least one computer instruction, and the at least one computer instruction is used to be executed by the processor to implement the virtual shooting method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one computer instruction, and the at least one computer instruction is used to be executed by a processor to implement the virtual shooting method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product includes at least one computer instruction, which is stored in a computer-readable storage medium; a processor of a computer device reads the at least one computer instruction from the computer-readable storage medium, and the processor executes the at least one computer instruction, so that the computer device implements the virtual shooting method as described in any one of claims 1 to 11.

Images