Virtual shooting method, device and equipment, and storage medium

Abstract

Embodiments of the present application disclose a virtual shooting method, device and equipment and a storage medium, and belong to the technical field of image processing. The method comprises: determining a line-of-sight direction of a pixel point in a virtual shooting image relative to a virtual camera based on camera parameters of the virtual camera in a sky sphere model, wherein the virtual shooting image is an image obtained by shooting in the sky sphere model at a shooting angle of the virtual camera; extracting a pixel value of the pixel point from a panoramic background image corresponding to the sky sphere model based on the line-of-sight direction, wherein the panoramic background image is an image used for mapping onto the sky sphere model as a virtual shooting background; and obtaining a background image of the virtual shooting image by rendering through a shader based on the pixel value of the pixel point. The scheme provided by the embodiments of the present application can reduce the cost of image generation, avoid the background distortion problem, and improve the generation quality of the background image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a virtual shooting method, apparatus, device, and storage medium. Background Technology

[0002] Virtual filming refers to a video special effects technology that can run in real time or offline, placing real-life people into virtual scenes.

[0003] In related technologies, a panoramic background image is applied as a texture map to a sky sphere model, and a virtual camera is placed at the center of the sky sphere model. The image obtained by observing the sky sphere model with the virtual camera is used as the background image, and this background image is used for virtual shooting to obtain a virtual shooting image.

[0004] However, since the panoramic background image is textured and mapped onto the sky sphere model, any shift in the virtual camera's position within the sky sphere model will cause distortion in the background image, thus affecting the generation of the virtual captured image. Summary of the Invention

[0005] This application provides a virtual shooting method, apparatus, device, and storage medium, the technical solution of which is as follows:

[0006] On one hand, embodiments of this application provide a virtual shooting method, the method comprising:

[0007] Based on the camera parameters of the virtual camera in the sky sphere model, the viewing direction of the pixels in the virtual captured image relative to the virtual camera is determined. The virtual captured image refers to the image captured within the sky sphere model from the shooting perspective of the virtual camera.

[0008] Based on the viewing direction, the pixel values ​​of the pixel points are extracted from the panoramic background image corresponding to the sky sphere model. The panoramic background image is an image used to be mapped onto the sky sphere model as a virtual shooting background.

[0009] Based on the pixel values ​​of the aforementioned pixels, a background image of the virtual captured image is obtained through shader rendering.

[0010] On the other hand, embodiments of this application provide a virtual shooting device, the device comprising:

[0011] The line-of-sight determination module is used to determine the line-of-sight direction of pixels in a virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. The virtual captured image refers to an image captured within the sky sphere model from the shooting perspective of the virtual camera.

[0012] The pixel extraction module is used to extract the pixel value of the pixel from the panoramic background image corresponding to the sky sphere model based on the viewing direction. The panoramic background image is an image used to be mapped onto the sky sphere model as a virtual shooting background.

[0013] The first image generation module is used to obtain the background image of the virtual captured image by rendering it through a shader based on the pixel values ​​of the pixel points.

[0014] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the virtual shooting method as described above.

[0015] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the virtual shooting method as described above.

[0016] On the other hand, embodiments of this application provide a computer program product comprising at least one computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer instruction from the computer-readable storage medium and executes the at least one computer instruction, causing the computer device to perform the virtual shooting method described above.

[0017] Unlike related technologies that use panoramic background images to overlay onto a sky sphere model, which requires placing the virtual camera at the center of the sky sphere model and causes distortion if the virtual camera moves, this embodiment determines the viewing direction of pixels in the virtual image relative to the virtual camera based on the camera parameters. Then, based on this viewing direction, the pixel values ​​are directly extracted from the panoramic background image corresponding to the sky sphere model, eliminating the need to overlay the panoramic background image onto the sky sphere model. This reduces image generation costs, avoids background distortion, and improves the quality of the generated background image. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This application shows a structural block diagram of a computer system provided in an exemplary embodiment;

[0020] Figure 2 A flowchart illustrating a virtual shooting method provided in an exemplary embodiment of this application is shown;

[0021] Figure 3 This illustration shows a schematic diagram of determining the direction of sight provided in an exemplary embodiment of this application;

[0022] Figure 4 A flowchart illustrating a virtual shooting method provided by another exemplary embodiment of this application is shown;

[0023] Figure 5 A flowchart illustrating the process of generating a background image provided in an exemplary embodiment of this application is shown;

[0024] Figure 6 A flowchart illustrating the rendering of an image sky region provided in an exemplary embodiment of this application is shown;

[0025] Figure 7 A flowchart illustrating image ground region rendering provided in an exemplary embodiment of this application is shown;

[0026] Figure 8 A flowchart for calculating the self-luminous intensity is shown in an exemplary embodiment of this application;

[0027] Figure 9 A flowchart illustrating the process of generating a virtual captured image provided in an exemplary embodiment of this application is shown;

[0028] Figure 10 This illustration shows a schematic diagram of a shadowed area formed in a sky sphere model provided in an exemplary embodiment of this application;

[0029] Figure 11 A schematic diagram illustrating a virtual captured image provided in an exemplary embodiment of this application is shown;

[0030] Figure 12 This illustration shows a structural block diagram of a computer system that applies a virtual shooting method to a film and television production scene, according to an exemplary embodiment of this application.

[0031] Figure 13 A structural block diagram of a virtual shooting device provided in an exemplary embodiment of this application is shown;

[0032] Figure 14 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0037] First, a brief introduction to the terms used in the embodiments of this application:

[0038] Virtual filming is a film production method that combines computer graphics, image processing, and virtual reality technologies. It simulates real scenes and objects using photographic techniques executed within a computer graphics environment, enabling filming in a virtual setting.

[0039] Sky Sphere Model: Located in the world coordinate system, the center of the sky sphere is the axis of the world coordinate system. A virtual camera is set within the sky sphere model. (Illustrative example, such as...) Figure 3 As shown, the sky sphere model 301 is located in the world coordinate system, and the center of the sky sphere model 301 is located at the axis of the world coordinate system. From the front view, the virtual camera 302 is located inside the sky sphere model 301.

[0040] Optionally, the sky sphere model and the panoramic background texture have a mapping relationship, and in this embodiment, the sky sphere model includes a sky model area and a ground model area.

[0041] Virtual captured image: refers to an image obtained by capturing a virtual scene from the shooting perspective of a virtual camera. In this embodiment, the virtual scene is a sky sphere model. The shooting perspective of the virtual camera is determined based on the camera's external parameters. (Illustrative example, such as...) Figure 3 As shown, by taking a picture of the sky sphere model 301 from the shooting perspective of the virtual camera 302, a virtual shooting image 303 can be obtained (it should be noted that the virtual shooting image does not actually exist in the world coordinate system, and is only shown in the figure based on the imaging principle).

[0042] A panoramic background image is an image with a wide field of view. A panoramic image is created by stitching together multiple individual photographs into a continuous image to display a 360-degree view of a scene. In this embodiment, the panoramic background image is used to map onto a sky sphere model as a virtual shooting background.

[0043] Please refer to Figure 1 This illustration shows a structural block diagram of a computer system provided in an exemplary embodiment of this application. The computer system may include a terminal 110 and a server 120. The terminal 110 and the server 120 communicate via a communication network. Optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0044] Terminal 110 is an electronic device with an application that has image generation capabilities installed. This image generation capability can be a native feature of the application on terminal 110, or a feature of a third-party application. Terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart TV, wearable device, or in-vehicle terminal, etc. Figure 1 The example of terminal 110 being a desktop computer is used for illustration only, but it is not a limitation.

[0045] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In this embodiment, server 120 can be a backend server for an application with image generation capabilities.

[0046] In some embodiments, data interaction exists between the server and the terminal. (Illustrative example, such as...) Figure 1As shown, after receiving the panoramic background image uploaded by the user, terminal 110, in response to the user's setting operations on the model parameters of the sky sphere model and the camera parameters of the virtual camera, sends the panoramic background image, the model parameters of the sky sphere model, and the camera parameters of the virtual camera to server 120. Server 120 then determines the model coordinates of the sky sphere model and the camera coordinates of the virtual camera in the world coordinate system based on the model parameters of the sky sphere model and the camera parameters of the virtual camera. Based on the camera parameters of the virtual camera in the sky sphere model, server 120 determines the viewing direction of the pixels in the virtual image relative to the virtual camera. Based on the viewing direction, server 120 extracts the pixel values ​​of the pixels from the panoramic background image corresponding to the sky sphere model. Finally, after obtaining the pixel values ​​of each pixel, server 120 renders the background image of the virtual image using a shader and returns the background image to terminal 110 for the user to use in subsequent virtual image generation.

[0047] Based on the above introduction, the virtual shooting method provided in this application will be described. This method can be executed by a server or a terminal, or by both a server and a terminal.

[0048] Please refer to Figure 2 This document illustrates a flowchart of a virtual shooting method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps:

[0049] Step 201: Based on the camera parameters of the virtual camera in the sky sphere model, determine the viewing direction of the pixels in the virtual captured image relative to the virtual camera. The virtual captured image refers to the image captured within the sky sphere model from the shooting perspective of the virtual camera.

[0050] Optionally, the camera parameters of the virtual camera include internal camera parameters and external camera parameters. Internal camera parameters include the camera focal length and principal point position; external camera parameters include position coordinates and rotation matrix. The position coordinates refer to the virtual camera's position in the world coordinate system, and the rotation matrix refers to the virtual camera's orientation relative to the world coordinate system.

[0051] Optionally, a virtual captured image refers to an image captured within the sky sphere model from the shooting perspective of a virtual camera. The shooting perspective of the virtual camera is determined based on external camera parameters.

[0052] Optionally, based on the imaging principle, starting from the viewpoint, the line of sight will first pass through the virtual imaging plane and then reach the real object. That is, the line of sight connects the viewpoint, the point on the virtual imaging plane, and the point on the real object into a line. Based on this, when photographing the sky sphere model with a virtual camera, the computer device can first determine the direction of each pixel in the virtual image relative to the virtual camera's line of sight, and then determine the coordinates of the points on the sky sphere model along the line of sight.

[0053] In some embodiments, the computer device can place a sky sphere model in a world coordinate system and place a virtual camera within the sky sphere model. Further, the computer device first transforms the points in the world coordinate system to the camera coordinate system based on the virtual camera's extrinsic parameters, and then projects the points in the camera coordinate system onto the image plane based on the virtual camera's intrinsic parameters, thereby obtaining the three-dimensional coordinates of each pixel on the image plane within the camera coordinate system. Then, based on the three-dimensional coordinates of the pixels in the camera coordinate system and the camera coordinates of the virtual camera, the computer device can determine the viewing direction of the pixel relative to the virtual camera.

[0054] Indicative, such as Figure 3 As shown, the sky sphere model 301 is located in the world coordinate system, and its center is located at the axis of the world coordinate system. From the front view, the virtual camera 302 is located within the sky sphere model 301. By taking a picture of the sky sphere model 301 from the shooting perspective of the virtual camera 302, a virtual image 303 can be obtained (it should be noted that the virtual image does not actually exist in the world coordinate system; it is only shown in the figure based on the imaging principle). In the camera coordinate system corresponding to the virtual camera 302, the viewpoint of the virtual camera 302 is located at the axis of the camera coordinate system, the virtual image 303 is located in the image UV coordinate system, and the object 305 is the object contained within the shooting perspective of the virtual camera 302.

[0055] The computer device can convert the position coordinates of object 305 in the world coordinate system into the shooting coordinates of object 305 in the camera coordinate system according to the mapping relationship between the world coordinate system and the camera coordinate system. As a result, the shooting coordinates of object 305 in the camera coordinate system, the display coordinates 304 of object 305 in the virtual shooting image 303, and the viewpoint of virtual camera 302 are located in the same line of sight.

[0056] Step 202: Based on the viewing direction, extract the pixel values ​​of the pixels from the panoramic background image corresponding to the sky sphere model. The panoramic background image is an image used to map onto the sky sphere model as a virtual shooting background.

[0057] Optionally, a panoramic image is an image with a wide field of view. A panoramic image is created by stitching together multiple individual photos into a continuous image to display a 360-degree view of a scene. Optionally, a panoramic background image is used to map onto a sky sphere model as a virtual shooting background. Optionally, the panoramic background image can be an image directly captured from the real environment, or it can be an image synthesized using artificial intelligence technology (such as AIGC (Artificial Intelligence Generated Content) technology). This application embodiment does not limit this.

[0058] Unlike related technologies that directly unfold and paste the panoramic background image onto the sky sphere model, which necessitates restricting the virtual camera's position within the sky sphere model, this embodiment does not directly paste the panoramic background image onto the sky sphere model. Instead, it extracts pixel values ​​from the panoramic background image based solely on the mapping relationship between the sky sphere model and the panoramic background image.

[0059] In some embodiments, after obtaining the viewing direction of pixels in the virtual captured image relative to the virtual camera, in order to determine the point coordinates on the sky sphere model along the viewing direction, the computer device can also transform the viewing direction to the world coordinate system according to the mapping relationship between the camera coordinate system and the world coordinate system. Thus, based on the viewing direction in the world coordinate system, the point coordinates (latitude and longitude) on the sky sphere model corresponding to each pixel can be determined. Based on the point coordinates and the mapping relationship between the sky sphere model and the panoramic background image, the computer device can extract the pixel values ​​of the pixels from the panoramic background image.

[0060] Step 203: Based on the pixel values ​​of the pixels, the background image of the virtual captured image is obtained by rendering through a shader.

[0061] In some embodiments, after obtaining the pixel values ​​corresponding to each pixel, the computer device can perform image rendering through a shader to obtain the background image of the virtual captured image.

[0062] Furthermore, by combining the background image with the real subject, a virtual image can be obtained, thereby reducing the cost of image capture.

[0063] In summary, in this embodiment, by determining the viewing direction of pixels in the virtual image relative to the virtual camera based on the camera parameters of the virtual camera, the pixel values ​​of the pixels can be directly extracted from the panoramic background image corresponding to the sky sphere model based on the viewing direction of the pixels, without having to paste the panoramic background image onto the sky sphere model. This reduces the image generation cost while avoiding background distortion problems and improving the generation quality of the background image.

[0064] In some embodiments, considering that the effect of image capture in a real scene is also affected by lighting (such as shadows formed by sunlight), in order to improve the realism of the virtual captured image, the computer device can also adjust the pixel value of each pixel according to the self-illumination characteristics of different model regions in the sky sphere model. This process will be described below through specific embodiments.

[0065] Please refer to Figure 4 This document illustrates a flowchart of a virtual shooting method provided in another exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps:

[0066] Step 401: Based on the camera parameters of the virtual camera in the sky sphere model, determine the viewing direction of the pixels in the virtual captured image relative to the virtual camera. The virtual captured image refers to the image captured within the sky sphere model from the shooting perspective of the virtual camera.

[0067] The implementation method of this step can be referred to step 201, and will not be repeated here in this embodiment.

[0068] Step 402: Standardize the line-of-sight vector corresponding to the line-of-sight direction to obtain the line-of-sight unit vector.

[0069] In some embodiments, after obtaining the gaze direction of a pixel in a virtual image relative to a virtual camera, the computer device transforms the gaze direction vector corresponding to the gaze direction into the world coordinate system based on the mapping relationship between the camera coordinate system and the world coordinate system. Furthermore, to facilitate pixel extraction, the computer device can first standardize the gaze direction vector to obtain a gaze direction unit vector.

[0070] Optionally, the line-of-sight vector can be represented as The process of standardizing the line-of-sight direction vector can be represented as follows:

[0071] Step 403: Based on the mapping relationship between the sky sphere model and the panoramic background image, the viewing direction unit vector is transformed from three-dimensional space to texture space to obtain the texture coordinates of the viewing direction unit vector in the panoramic background image.

[0072] Optionally, the panoramic background image corresponds to the image's UV coordinate system. The bottom left corner of the panoramic background image is the origin (0,0) of the UV coordinate system, the U-axis represents the horizontal direction, the V-axis represents the vertical direction, and the top right corner of the panoramic background image corresponds to the UV coordinate system (1,1). Optionally, the mapping relationship between the sky sphere model and the panoramic background image is the mapping relationship between the sky sphere model and the UV coordinate system. The bottom center of the sky sphere model corresponds to the origin of the UV coordinate system.

[0073] In some embodiments, in order to determine the texture coordinates corresponding to the gaze direction unit vector in the panoramic background image, the computer device can convert the gaze direction unit vector from three-dimensional space to texture space according to the mapping relationship between the sky sphere and the panoramic background image.

[0074] Optionally, the view direction unit vector is a three-dimensional vector, which can be represented as (x, y, z), and its corresponding UV coordinates can be represented as (u, v). The mapping relationship can be represented as follows:

[0075] Step 404: Extract pixel values ​​from the panoramic background image based on texture coordinates.

[0076] In some embodiments, after obtaining the texture coordinates corresponding to the unit vectors of each viewing direction, the computer device can extract the pixel values ​​of pixels from the panoramic background image based on the texture coordinates. That is, the pixel values ​​of the coordinate points indicated by the texture coordinates can be extracted as the pixel values ​​of pixels in the virtual captured image.

[0077] Optionally, the pixel value can be an RGB value. The computer device extracts the red, green, and blue channel values ​​of the coordinate points indicated by the texture coordinates from the panoramic background image, and uses these values ​​as the pixel values ​​of the pixels in the virtual captured image.

[0078] Step 405: Determine the model region corresponding to the pixel in the sky sphere model and the self-emission intensity of the model region. The self-emission intensity is used to characterize the effect of the model region as a light source on the illumination inside the sky sphere model, and different model regions correspond to different self-emission intensities.

[0079] In some embodiments, in order to simulate the lighting effects in a real scene, the computer device may first determine the self-emission intensity corresponding to different model regions in the sky sphere model. The self-emission intensity represents the lighting effect of the model region as a light source on the interior of the sky sphere model, and different model regions correspond to different self-emission intensities.

[0080] In some embodiments, since the self-emission intensity of different model regions is different, that is, when the virtual captured image contains multiple model regions, the brightness value of the pixels in different image regions will also be different. Therefore, in order to improve the accuracy of pixel values, the computer device also needs to first determine the model region corresponding to each pixel in the virtual captured image in the sky sphere model.

[0081] In one possible implementation, the computer device can determine the model area corresponding to the pixel in the sky sphere model by calculating the line of sight from the viewpoint of the virtual camera, based on the line of sight direction corresponding to the pixel, and determine the self-illumination intensity of the model area based on the lighting parameters configured in the sky sphere model.

[0082] Optionally, the sky sphere model may include a sky model region and a ground model region. In one possible implementation, the computer device can construct a three-dimensional spatial coordinate system, i.e., a world coordinate system, and then construct the sky sphere model with the axis of the three-dimensional spatial coordinate system as the center of the sphere and the input length as the radius of the sphere. The spherical surface of the sky sphere model is the sky model region, and the horizontal hemisphere of the sky sphere model is the ground model region.

[0083] Indicative, such as Figure 5 As shown in the front view, the sky sphere model includes a sky model area 501 (i.e., the sphere of the sky sphere model) and a ground model area 502 (i.e., the horizontal hemisphere of the sky sphere model).

[0084] In some embodiments, considering that the shooting effect in a real scene is usually affected by ambient light and light sources, the light sources are usually located in the sky area (such as the real sky outdoors, indoor ceiling lights, etc.) rather than the ground area. Based on this, the self-emission intensity of the sky model area can be determined based on the ambient light and virtual light sources configured in the sky sphere model; the self-emission intensity of the ground model area can be determined based on the ambient light configured in the sky sphere model.

[0085] Optionally, the ambient light configured in the sky sphere model refers to diffuse light, which is the ubiquitous and uniformly distributed illumination within the sky sphere model. Ambient light can simulate light that is weakened and falls on objects from various directions, increasing the overall brightness within the sky sphere model and reducing the intensity of shadows.

[0086] Optionally, the virtual light source configured in the sky sphere model can be a directional light (used to simulate a light source at an infinite distance, such as sunlight); it can also be a point light source (a light source emitted from a fixed point in all directions); it can also be a surface light source (a light source covering a certain area, used to simulate a large-area light-emitting panel); or it can be other types of light sources. This application embodiment does not limit this.

[0087] In one possible implementation, the computer device can determine the first self-illumination intensity corresponding to the sky model area based on the first illumination intensity of the ambient light configured in the sky sphere model, the second illumination intensity of the virtual light source, and the illumination direction.

[0088] Optionally, the illumination direction of the virtual light source can be determined based on the pitch and yaw angles of the virtual light source. The pitch angle indicates the degree of tilt of the virtual light source on the horizontal plane, i.e., the angle at which it tilts upwards or downwards from the horizontal line. The yaw angle indicates the degree of rotation of the virtual light source on the vertical plane, i.e., the angle at which it rotates clockwise or counterclockwise from true north.

[0089] Optionally, the self-illumination intensity of the sky model area is only affected by the pitch angle of the virtual light source, while the heading angle of the virtual light source will affect the direction of the shadow.

[0090] Optionally, the first illuminance of the ambient light can be represented as iambient (in illuminance (Lux)), the second illuminance of the virtual light source can be represented as ilight source (in illuminance (Lux)), and the pitch angle of the virtual light source can be represented as α. pitch (Unit: degrees (°)), the first self-luminous intensity corresponding to the sky model region can be expressed as e sky The process of determining the first self-luminous intensity can be expressed as follows: Where β is the conversion factor from illuminance to shader light intensity.

[0091] In one possible implementation, the computer device can determine the second self-luminous intensity corresponding to the ground model area based on the first illuminance of ambient light.

[0092] Optionally, the second self-luminous intensity corresponding to the ground model region can be expressed as e floor The process of determining the second self-luminous intensity can be expressed as e floor =β·ienvironment, where β is the conversion factor from illuminance to shader light intensity.

[0093] Optionally, the first illumination intensity of the ambient light, the second illumination intensity of the virtual light source, and the illumination direction configured in the sky sphere model can be set by the user. Thus, the computer device can automatically adjust the system through parameters and calculate the first self-illumination intensity corresponding to the sky model area and the second self-illumination intensity corresponding to the ground model area based on the illumination parameters input by the user.

[0094] Step 406: Determine the self-illuminating pixel value of the pixel based on the self-illumination intensity and pixel value corresponding to the pixel.

[0095] In some embodiments, after determining the model region corresponding to each pixel in the virtual captured image in the sky sphere model and the self-emission intensity corresponding to each model region, the computer device can adjust the pixel value according to the self-emission intensity corresponding to the pixel to obtain the self-emission pixel value of the pixel.

[0096] Optionally, the self-illuminating pixel value is used to characterize the color of a pixel under illumination. The self-illuminating pixel value is equal to the self-illumination intensity multiplied by the pixel value.

[0097] In one possible implementation, when the model area corresponding to the pixel is a sky model area, the computer device can determine the first self-illuminating pixel value of the pixel based on the first self-illuminating intensity and the pixel value; when the model area corresponding to the pixel is a ground model area, the computer device can determine the second self-illuminating pixel value of the pixel based on the second self-illuminating intensity and the pixel value.

[0098] Optionally, the first self-illuminating pixel value is equal to e. sky Multiply by the pixel value, the second self-illuminating pixel value equals e. floor Multiply by the pixel value.

[0099] Optionally, when both the first illumination intensity of the ambient light and the second illumination intensity of the virtual light source increase simultaneously, the brightness of each pixel in the background image will increase, i.e., the overall brightness of the image will increase; when both the first illumination intensity of the ambient light and the second illumination intensity of the virtual light source decrease simultaneously, the brightness of each pixel in the background image will decrease, i.e., the overall brightness of the image will decrease.

[0100] Step 407: Based on the self-illuminating pixel values ​​of the pixels, the background image of the virtual captured image is obtained by rendering through a shader.

[0101] In some embodiments, after determining the self-illuminating pixel values ​​of each pixel in the virtual captured image, the computer device can render the image using a shader to obtain the background image corresponding to the virtual captured image.

[0102] Optionally, considering the different lighting properties of different model areas, such as the sky model area having no collision and no projection (i.e., the sky will not cast shadows on the ground model area), while the ground model area has collision and allows other 3D models to cast shadows on the ground model area, computer devices can also use different shaders to render the sky and ground areas in the background image in order to further improve image quality.

[0103] In one possible implementation, the computer device can render the pixels corresponding to the sky model area through a first shader and the pixels corresponding to the ground model area through a second shader based on the self-illuminating pixel values ​​of each pixel, thereby obtaining the background image of the virtual captured image.

[0104] The first shader can be set to self-illuminating mode (no base color, no light emitted), while the second shader can be set to either self-illuminating mode or reflection mode. It should be noted that since the second shader is in reflection mode, parameters such as specular highlight, anisotropy, metallicity, and transparency should be set to 0, and roughness should be set to 1.

[0105] In the above embodiments, by converting the viewing direction unit vector into UV coordinates based on the mapping relationship between the sky sphere model and the panoramic background texture, the pixel values ​​of the pixels are directly extracted from the panoramic background texture based on the UV coordinates, ensuring the accuracy of pixel value acquisition. Furthermore, before shading and rendering, the pixel values ​​of the pixels are adjusted by the self-illumination intensity of different model regions, which increases the lighting effect in the background image and further improves the image quality of the background image.

[0106] Please refer to Figure 5 The diagram illustrates a flowchart of a process for generating a background image, as provided in an exemplary embodiment of this application.

[0107] In some embodiments, after generating the background image, the computer device first obtains the model parameters of the sky sphere model, the camera parameters of the virtual camera, and the lighting parameters configured in the sky sphere model.

[0108] First, the computer equipment can automatically adjust the system through parameters to calculate the self-illumination intensity S1 corresponding to the sky model region 501 and the self-illumination intensity F1 corresponding to the ground model region 502 based on the user-configured lighting parameters. Simultaneously, the computer equipment determines the viewing direction of each pixel in the virtual image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. Based on the viewing direction vector corresponding to the viewing direction, it determines the model region corresponding to each pixel and extracts the pixel values ​​of each pixel from the panoramic background image corresponding to the sky sphere model.

[0109] Furthermore, the computer device calculates the self-illuminating pixel value corresponding to each pixel based on the model region and pixel value corresponding to each pixel, the self-illumination intensity S1, and the self-illumination intensity F1. Then, the background image is obtained by shading and rendering using a first shader with self-illumination mode enabled and a second shader with both self-illumination mode and reflection mode enabled (reflection intensity F2 set).

[0110] Indicative, such as Figure 6The diagram illustrates a flowchart of image sky region rendering provided in an exemplary embodiment of this application. For pixels in the background image used to represent the sky region, i.e., pixels corresponding to the sky model region in the sky sphere model, the computer device first determines the viewing direction of each pixel in the virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. Thus, if the viewing direction vector points to the sky model region, the pixel corresponding to that viewing direction is determined to represent the sky region. Furthermore, the computer device first standardizes the viewing direction vector to obtain a viewing direction unit vector, and extracts the pixel value of the pixel from the panoramic background image based on the mapping relationship between the sky sphere model and the panoramic background image, i.e., the mapping relationship between the sky sphere model and the UV coordinate system. The pixel value is then multiplied by the first self-illumination intensity corresponding to the sky model region to obtain the first self-illuminating pixel value of the pixel. Finally, this first self-illuminating pixel value is given to the first shader for shading and rendering to generate the background image.

[0111] Indicative, such as Figure 7 The diagram illustrates a flowchart of image ground region rendering provided in an exemplary embodiment of this application. For pixels in the background image used to represent the ground region, i.e., pixels corresponding to the ground model region in the sky sphere model, the computer device first determines the viewing direction of each pixel in the virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. Thus, if the viewing direction vector points to the ground model region, the pixel corresponding to that viewing direction is determined to represent the ground region. Furthermore, the computer device first standardizes the viewing direction vector to obtain a viewing direction unit vector, and extracts the pixel value of the pixel from the panoramic background image based on the mapping relationship between the sky sphere model and the panoramic background image, i.e., the mapping relationship between the sky sphere model and the UV coordinate system. The pixel value is then multiplied by the second self-illumination intensity corresponding to the ground model region to obtain the second self-illuminating pixel value of the pixel. Finally, this second self-illuminating pixel value is given to the second shader for shading and rendering to generate the background image.

[0112] In addition, since the ground allows other 3D models to cast shadows, the second shader also needs to enable reflection mode, that is, the computer device needs to set the virtual light source in the sky sphere model according to the first light intensity of the virtual light source configured by the user.

[0113] Indicative, such as Figure 8 As shown, it illustrates a flowchart for calculating the self-luminous intensity provided in an exemplary embodiment of this application.

[0114] Optionally, the first self-illumination intensity of the sky model area is related to the first illumination intensity of the virtual light source, the pitch angle of the virtual light source, and the second illumination intensity of the ambient light, while the second self-illumination intensity of the ground model area is related to the second illumination intensity of the ambient light. Therefore, the computer device can automatically adjust the system to calculate the first self-illumination intensity of the sky model area and the second self-illumination intensity of the ground model area based on the illumination parameters configured by the user.

[0115] In some embodiments, after obtaining the background image, the computer device can combine the real subject with the background image to obtain a complete virtual image. This process will be described below through specific embodiments.

[0116] Please refer to Figure 9 This document illustrates a flowchart of a process for generating a virtual photographed image according to an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps:

[0117] Step 901: Determine the virtual shooting position of the 3D model corresponding to the shooting object in the sky sphere model.

[0118] In some embodiments, during the process of combining a virtual background with a real subject to obtain a virtual image, considering that the subject in the actual image is usually affected by lighting and will produce shadows, the computer device may also add shadow effects corresponding to the real subject to the virtual image in order to improve the realism of the virtual image.

[0119] In one possible implementation, considering that the formation of shadows in an image is related to factors such as the position and intensity of the light source, the position and shape of the object, and the shooting angle, the computer device can first determine the virtual shooting position of the three-dimensional model corresponding to the shooting object in the sky sphere model.

[0120] Optionally, when the subject is a regular object, the computer device can directly construct a 3D model of the subject in the world coordinate system; when the subject is an irregular object (such as a person with different postures), the computer device can also use motion capture technology to collect the data of the marker points mounted on the subject, convert the marker point data into coordinate data in 3D space, and apply the coordinate data to the 3D human body model to obtain an accurate 3D human body model.

[0121] Optionally, after determining the 3D model corresponding to the subject being photographed, the computer device can place the 3D model in the 3D spatial coordinate system corresponding to the sky sphere model, thereby obtaining the model position coordinates corresponding to the 3D model, that is, the virtual shooting position of the 3D model in the sky sphere model.

[0122] Step 902: Based on the model parameters of the 3D model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model, determine the image shadow area corresponding to the shooting object in the background image.

[0123] In some embodiments, since the formation of shadows in an image is related to factors such as the position and intensity of the light source, the position and shape of the object, and the shooting angle, in order to determine the image shadow area corresponding to the shooting object in the background image, the computer device needs to obtain the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model.

[0124] Optionally, the model parameters of the 3D model include the model shape and model size, the camera parameters include the internal camera parameters and the external camera parameters, and the lighting parameters configured in the sky sphere model include the ambient light parameters and the virtual light source parameters.

[0125] Optionally, the position, shape, and size of shadows formed in three-dimensional space are related to the model parameters of the 3D model, the virtual shooting position, and the direction of the light source. For example, low-angle light sources produce long and shallow shadows, while high-angle light sources produce short and deep shadows. In two-dimensional image space, the color effect, shape, and size of shadows are related to the shooting angle and lighting parameters. For example, the stronger the light intensity, the higher the contrast of the shadow; and the closer the light source, the clearer and higher the contrast of the shadow, while the farther the light source, the blurrier and lower the contrast of the shadow.

[0126] In one possible implementation, the computer device can first determine the shadow area of ​​the 3D model in the sky sphere model based on the model parameters of the 3D model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model. This shadow area is formed in the ground model area of ​​the sky sphere model. Further, the computer device then renders the corresponding image shadow area of ​​the photographed object in the background image based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, and the lighting parameters configured in the sky sphere model.

[0127] Indicative, such as Figure 10 As shown, taking sunlight as an example of a virtual light source, the sky sphere model includes a sky model region 1001 and a ground model region 1002. The ground model region 1002 allows other 3D models to cast shadows. For the 3D model 1004 of the subject being photographed, the computer device can determine the shadow region 1003 of the 3D model 1004 in the sky sphere model based on the model parameters of the 3D model 1004, the virtual shooting position, and the direction of the sunlight source configured in the sky sphere model. This shadow region 1003 is formed on the ground model region 1002 of the sky sphere model.

[0128] Optionally, considering that the lighting configured within the sky sphere model includes both ambient light and virtual light sources, and that different light intensities of ambient light and virtual light sources will produce different image shadow effects, the computer device also needs to specifically render and determine the image shadow area corresponding to the photographed object in the background image based on camera parameters, the shadow area of ​​the 3D model in the sky sphere model, the first light intensity of the ambient light configured in the sky sphere model, and the second light intensity of the virtual light source.

[0129] Specifically, the brightness of the image shadow area is positively correlated with the intensity difference between the second and first light intensities. That is, the greater the intensity difference between the second and first light intensities, the darker and more obvious the image shadows; conversely, the smaller the intensity difference between the second and first light intensities, the weaker and softer the image shadow areas.

[0130] Step 903: Generate a virtual image based on the real image of the subject, the shadow image area, and the background image.

[0131] In some embodiments, after determining the image shadow area corresponding to the subject in the background image, the computer device can generate a virtual image by combining the real captured image (the image after image matting), the shadow image area, and the background image.

[0132] Indicative, such as Figure 11 As shown, the computer device can generate a virtual image by combining the real image corresponding to the subject 1102, the shadow image area 1103, and the background image 1101.

[0133] In the above embodiments, by determining the shadow area of ​​the three-dimensional model corresponding to the shooting object in the sky sphere model, and then combining the model parameters, camera parameters, and the lighting parameters configured in the sky sphere model, the image shadow area corresponding to the shooting object in the background image is determined, which improves the realism of the shadow formation of the shooting object in the virtual shooting image and further optimizes the image quality of the virtual shooting image.

[0134] Optionally, the virtual shooting method provided in this application embodiment can be applied to various scenarios, such as film and television production scenarios (including film production, TV series production, advertising production, documentary production, etc.), game development scenarios, educational and learning scenarios (driving simulators, flight training, etc.), etc.

[0135] For film and television production scenarios:

[0136] To reduce film and television production costs while ensuring the quality of the works, the virtual shooting method proposed in this application can be applied to film and television production scenarios. In a film and television production scenario, the virtual shooting image is a video image frame, the panoramic background image is a film and television scene image, and the computer device can construct a sky sphere model and determine the mapping relationship between the film and television scene image and the sky sphere model.

[0137] In some embodiments, the computer device first determines the viewing direction of pixels in a video image frame relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. This video image frame refers to an image captured within the sky sphere model from the shooting perspective of the virtual camera. Further, based on the viewing direction, the computer device extracts the pixel values ​​of the pixels from a corresponding film / video scene image on the sky sphere model. This film / video scene image is used to map onto the sky sphere model as a virtual shooting background. Finally, based on the pixel values, the background image of the video image frame is obtained through shader rendering.

[0138] In one possible implementation, the computer device can first determine the model region corresponding to the pixel in the sky sphere model and the self-emission intensity of the model region. The self-emission intensity is used to characterize the illumination effect of the model region as a light source on the interior of the sky sphere model, and different model regions correspond to different self-emission intensities. Based on the self-emission intensity and pixel value corresponding to the pixel, the self-emission pixel value of the pixel is determined. Then, based on the self-emission pixel value of the pixel, the background image of the video image frame is rendered by the shader.

[0139] Optionally, the computer device can determine the model region corresponding to the pixel in the sky sphere model based on the viewing direction corresponding to the pixel, and determine the self-emission intensity of the model region based on the lighting parameters configured in the sky sphere model.

[0140] Optionally, the sky sphere model includes a sky model region and a ground model region. The computer device can determine the first self-emission intensity corresponding to the sky model region based on the first illumination intensity of the ambient light, the second illumination intensity of the virtual light source, and the illumination direction configured in the sky sphere model; and determine the second self-emission intensity corresponding to the ground model region based on the first illumination intensity of the ambient light.

[0141] Optionally, when the model area corresponding to the pixel is a sky model area, the computer device determines the first self-illuminating pixel value of the pixel based on the first self-illuminating intensity and the pixel value; when the model area corresponding to the pixel is a ground model area, the computer device determines the second self-illuminating pixel value of the pixel based on the second self-illuminating intensity and the pixel value.

[0142] Optionally, the computer device renders the pixels corresponding to the sky model area using a first shader and the pixels corresponding to the ground model area using a second shader, based on the self-illuminating pixel values ​​of the pixels, to obtain the background image of the video image frame. The rendering mode of the first shader is set to self-illuminating mode, and the rendering mode of the second shader is set to both self-illuminating mode and reflection mode.

[0143] In some embodiments, the computer device also needs to first construct a three-dimensional spatial coordinate system, and then construct a sky sphere model with the axis of the three-dimensional spatial coordinate system as the center of the sphere and the input length as the radius of the sphere. The sphere of the sky sphere model represents the sky model region, and the horizontal hemisphere of the sky sphere model represents the ground model region.

[0144] Regarding the process of extracting pixel values, in one possible implementation, the computer device can first standardize the gaze direction vector corresponding to the gaze direction to obtain a gaze direction unit vector. Then, based on the mapping relationship between the sky sphere model and the film and television scene image, the gaze direction unit vector is converted from three-dimensional space to texture space to obtain the texture coordinates of the gaze direction unit vector in the film and television scene image. Based on the texture coordinates, the pixel values ​​of the pixel points are extracted from the film and television scene image.

[0145] Regarding the process of generating video image frames, in some embodiments, the computer device can first determine the virtual shooting position of the 3D model corresponding to the actor in the sky sphere model, and then determine the image shadow area corresponding to the actor in the background image based on the model parameters of the 3D model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model. Thus, based on the real shooting image corresponding to the actor, the shadow image area, and the background image, a video image frame is generated.

[0146] Optionally, the computer device can first determine the shadow area of ​​the 3D model in the sky sphere model based on the model parameters of the 3D model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model. Then, based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, and the lighting parameters configured in the sky sphere model, it can render the image shadow area corresponding to the actor in the background image.

[0147] Optionally, the computer device can also render the image shadow area corresponding to the actor in the background image based on camera parameters, the shadow area of ​​the 3D model in the sky sphere model, the first illumination intensity of the ambient light configured in the sky sphere model, and the second illumination intensity of the virtual light source.

[0148] Indicative, such as Figure 12 As shown, it illustrates a structural block diagram of a computer system that applies a virtual shooting method to a film and television production scene, according to an exemplary embodiment of this application.

[0149] In some embodiments, data interaction exists between server 1202 and terminal 1201. (Illustrative example, such as...) Figure 12 As shown, after receiving the video scene image uploaded by the user, terminal 1201, in response to the user's setting operations on the model parameters of the sky sphere model and the camera parameters of the virtual camera, sends the video scene image, the model parameters of the sky sphere model, and the camera parameters of the virtual camera to server 1202. Server 1202 then determines the model coordinates of the sky sphere model and the camera coordinates of the virtual camera in the world coordinate system based on the model parameters of the sky sphere model and the camera parameters of the virtual camera. Based on the camera parameters of the virtual camera in the sky sphere model, it determines the viewing direction of the pixels in the video image frame relative to the virtual camera. Based on the viewing direction, it extracts the pixel values ​​of the pixels from the video scene image corresponding to the sky sphere model. Finally, after obtaining the pixel values ​​of each pixel, it renders the background image of the video image frame through a shader and returns the background image to terminal 1201.

[0150] Furthermore, after receiving the real-shot image uploaded by the user, the terminal 1201 performs image matting on the real-shot image and constructs a 3D model based on the actor's movements and postures in the real-shot image. Then, it determines the virtual shooting position of the 3D model in the sky sphere model. Based on the model parameters of the 3D model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model, it determines the image shadow area corresponding to the actor in the background image. Thus, based on the real-shot image corresponding to the actor, the shadow image area, and the background image, a video image frame is generated.

[0151] For game development scenarios:

[0152] To reduce game production costs while ensuring game graphics quality, the virtual shooting method proposed in this application can be applied to game development scenarios. In a game development scenario, the virtual shooting image is the virtual environment image, the panoramic background image is the environment scene image, and the computer device can construct a sky sphere model and determine the mapping relationship between the environment scene image and the sky sphere model.

[0153] In some embodiments, the computer device first determines the viewing direction of pixels in the virtual environment image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. This virtual environment image refers to an image captured within the sky sphere model from the shooting perspective of the virtual camera. Further, based on the viewing direction, the computer device extracts the pixel values ​​of the pixels from the corresponding environmental scene map of the sky sphere model. The environmental scene map is an image used to map onto the sky sphere model as a virtual shooting background. Finally, based on the pixel values, the background image of the virtual environment image is obtained through shader rendering.

[0154] In one possible implementation, the computer device can first determine the model area corresponding to the pixel in the sky sphere model and the self-emission intensity of the model area. The self-emission intensity is used to characterize the lighting effect of the model area as a light source on the interior of the sky sphere model, and different model areas correspond to different self-emission intensities. Based on the self-emission intensity and pixel value corresponding to the pixel, the self-emission pixel value of the pixel is determined. Then, based on the self-emission pixel value of the pixel, the background image of the virtual environment is rendered by the shader.

[0155] Optionally, the computer device can determine the model region corresponding to the pixel in the sky sphere model based on the viewing direction corresponding to the pixel, and determine the self-emission intensity of the model region based on the lighting parameters configured in the sky sphere model.

[0156] Optionally, the sky sphere model includes a sky model region and a ground model region. The computer device can determine the first self-emission intensity corresponding to the sky model region based on the first illumination intensity of the ambient light, the second illumination intensity of the virtual light source, and the illumination direction configured in the sky sphere model; and determine the second self-emission intensity corresponding to the ground model region based on the first illumination intensity of the ambient light.

[0157] Optionally, when the model area corresponding to the pixel is a sky model area, the computer device determines the first self-illuminating pixel value of the pixel based on the first self-illuminating intensity and the pixel value; when the model area corresponding to the pixel is a ground model area, the computer device determines the second self-illuminating pixel value of the pixel based on the second self-illuminating intensity and the pixel value.

[0158] Optionally, the computer device renders the pixels corresponding to the sky model area using a first shader and the pixels corresponding to the ground model area using a second shader, based on the self-illuminating pixel values ​​of the pixels, to obtain the background image of the virtual environment. The rendering mode of the first shader is set to self-illuminating mode, and the rendering mode of the second shader is set to both self-illuminating mode and reflection mode.

[0159] In some embodiments, the computer device also needs to first construct a three-dimensional spatial coordinate system, and then construct a sky sphere model with the axis of the three-dimensional spatial coordinate system as the center of the sphere and the input length as the radius of the sphere. The sphere of the sky sphere model represents the sky model region, and the horizontal hemisphere of the sky sphere model represents the ground model region.

[0160] Regarding the process of extracting pixel values, in one possible implementation, the computer device can first standardize the gaze direction vector corresponding to the gaze direction to obtain a gaze direction unit vector. Then, based on the mapping relationship between the sky sphere model and the environment scene map, the gaze direction unit vector is converted from three-dimensional space to texture space to obtain the texture coordinates of the gaze direction unit vector in the environment scene map. Based on the texture coordinates, the pixel values ​​of the pixel points are extracted from the environment scene map.

[0161] Regarding the process of generating virtual environment images, in some embodiments, the computer device can first determine the virtual shooting position of the 3D model corresponding to the actor in the sky sphere model, and then determine the image shadow area corresponding to the actor in the background image based on the model parameters of the 3D model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model. Thus, a virtual environment image is generated based on the real shooting image corresponding to the actor, the shadow image area, and the background image.

[0162] Optionally, the computer device can first determine the shadow area of ​​the 3D model in the sky sphere model based on the model parameters of the 3D model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model. Then, based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, and the lighting parameters configured in the sky sphere model, it can render the image shadow area corresponding to the actor in the background image.

[0163] Optionally, the computer device can also render the image shadow area corresponding to the actor in the background image based on camera parameters, the shadow area of ​​the 3D model in the sky sphere model, the first illumination intensity of the ambient light configured in the sky sphere model, and the second illumination intensity of the virtual light source.

[0164] Please refer to Figure 13 The diagram illustrates a structural block diagram of a virtual shooting device provided in an exemplary embodiment of this application, the device comprising:

[0165] The line-of-sight determination module 1301 is used to determine the line-of-sight direction of a pixel in a virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. The virtual captured image refers to an image captured within the sky sphere model from the shooting angle of the virtual camera.

[0166] The pixel extraction module 1302 is used to extract the pixel value of the pixel from the panoramic background image corresponding to the sky sphere model based on the viewing direction. The panoramic background image is an image used to be mapped onto the sky sphere model as a virtual shooting background.

[0167] The first image generation module 1303 is used to obtain the background image of the virtual captured image by rendering a shader based on the pixel values ​​of the pixel points.

[0168] Optionally, the first image generation module 1303 includes:

[0169] An intensity determination unit is used to determine the model region corresponding to the pixel in the sky sphere model and the self-emission intensity of the model region. The self-emission intensity is used to characterize the illumination effect of the model region as a light source on the interior of the sky sphere model, and different model regions correspond to different self-emission intensities.

[0170] A pixel determination unit is used to determine the self-emitting pixel value of the pixel based on the self-emitting intensity corresponding to the pixel and the pixel value.

[0171] An image generation unit is used to render the background image of the virtual captured image by a shader based on the self-illuminating pixel value of the pixel.

[0172] Optionally, the intensity determining unit is used for:

[0173] Based on the viewing direction corresponding to the pixel, the model region corresponding to the pixel in the sky sphere model is determined;

[0174] The self-illumination intensity of the model region is determined based on the illumination parameters configured in the sky sphere model.

[0175] Optionally, the sky sphere model includes a sky model region and a ground model region; the intensity determination unit is used for:

[0176] Based on the first illumination intensity of the ambient light, the second illumination intensity of the virtual light source, and the illumination direction configured in the sky sphere model, the first self-illumination intensity corresponding to the sky model region is determined.

[0177] Based on the first illumination intensity of the ambient light, the second self-luminous intensity corresponding to the ground model area is determined.

[0178] Optionally, the pixel determining unit is used for:

[0179] When the model region corresponding to the pixel is the sky model region, the first self-illuminating pixel value of the pixel is determined based on the first self-illuminating intensity and the pixel value.

[0180] When the model region corresponding to the pixel is the ground model region, the second self-illuminating pixel value of the pixel is determined based on the second self-illuminating intensity and the pixel value.

[0181] Optionally, the image generation unit is used for:

[0182] Based on the self-illuminating pixel value of the pixel, the pixel corresponding to the sky model area is rendered by the first shader, and the pixel corresponding to the ground model area is rendered by the second shader to obtain the background image of the virtual shooting image;

[0183] The first shader is set to self-illuminating mode, and the second shader is set to both self-illuminating mode and reflection mode.

[0184] Optionally, the device further includes:

[0185] The coordinate system construction module is used to construct a three-dimensional spatial coordinate system;

[0186] The model building module is used to construct the sky sphere model with the axis of the three-dimensional spatial coordinate system as the center of the sphere and the input length as the radius of the sphere; wherein, the spherical surface of the sky sphere model is the sky model region, and the horizontal hemisphere of the sky sphere model is the ground model region.

[0187] Optionally, the pixel extraction module 1302 is used for:

[0188] The line-of-sight vector corresponding to the line-of-sight direction is standardized to obtain a line-of-sight unit vector;

[0189] Based on the mapping relationship between the sky sphere model and the panoramic background image, the gaze direction unit vector is transformed from three-dimensional space to texture space to obtain the texture coordinates of the gaze direction unit vector in the panoramic background image.

[0190] Based on the texture coordinates, the pixel value of the pixel point is extracted from the panoramic background image.

[0191] Optionally, the device further includes:

[0192] The position determination module is used to determine the virtual shooting position of the 3D model corresponding to the shooting object in the sky sphere model;

[0193] The region determination module is used to determine the image shadow region corresponding to the shooting object in the background image based on the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model.

[0194] The second image generation module is used to generate the virtual image based on the real image of the subject being photographed, the shadow image area, and the background image.

[0195] Optionally, the region determination module is used for:

[0196] Based on the model parameters of the 3D model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model, the shadow area of ​​the 3D model in the sky sphere model is determined;

[0197] Based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, and the lighting parameters configured in the sky sphere model, the image shadow area corresponding to the photographed object in the background image is rendered.

[0198] Optionally, the region determination module is used for:

[0199] Based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, the first illumination intensity of the ambient light configured in the sky sphere model, and the second illumination intensity of the virtual light source, the image shadow area corresponding to the photographed object in the background image is rendered.

[0200] The brightness of the image shadow area is positively correlated with the intensity difference between the second illumination intensity and the first illumination intensity.

[0201] In summary, in this embodiment, by determining the viewing direction of pixels in the virtual image relative to the virtual camera based on the camera parameters of the virtual camera, the pixel values ​​of the pixels can be directly extracted from the panoramic background image corresponding to the sky sphere model based on the viewing direction of the pixels, without having to paste the panoramic background image onto the sky sphere model. This reduces the image generation cost while avoiding background distortion problems and improving the generation quality of the background image.

[0202] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0203] Please refer to Figure 14This illustration shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1400 includes a Central Processing Unit (CPU) 1401, a system memory 1404 including a random access memory 1402 and a read-only memory 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The computer device 1400 may also include a basic input / output system (I / O system) 1406 to facilitate the transfer of information between various devices within the computer, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0204] In some embodiments, the basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to a system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0205] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer-readable media (not shown) such as a hard disk or drive.

[0206] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0207] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1401. The one or more programs contain instructions for implementing the above methods, and the central processing unit 1401 executes the one or more programs to implement the virtual shooting method provided by the above method embodiments.

[0208] According to various embodiments of this application, the computer device 1400 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1400 can be connected to the network 1411 via the network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 can be used to connect to other types of networks or remote computer systems (not shown).

[0209] This application also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the virtual shooting method described in the above embodiments.

[0210] Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0211] This application provides a computer program product including at least one computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one computer instruction from the computer-readable storage medium and executes the at least one computer instruction, causing the computer device to perform the virtual shooting method described in the above embodiment.

[0212] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0213] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A virtual shooting method, characterized in that, The method includes: Based on the camera parameters of the virtual camera in the sky sphere model, the viewing direction of the pixels in the virtual captured image relative to the virtual camera is determined. The virtual captured image refers to the image captured within the sky sphere model from the shooting perspective of the virtual camera. The sky sphere model includes a sky model area and a ground model area. The spherical surface of the sky sphere model is the sky model area, and the horizontal hemisphere of the sky sphere model is the ground model area. The ground model area allows other 3D models to cast shadows, but the sky model area will not cast shadows in the ground model area. Based on the viewing direction, the pixel values ​​of the pixel points are extracted from the panoramic background image corresponding to the sky sphere model. The panoramic background image is an image used to be mapped onto the sky sphere model as a virtual shooting background. Based on the pixel values ​​of the aforementioned pixels, a background image of the virtual captured image is obtained by rendering using a shader. Determine the virtual shooting position of the 3D model corresponding to the subject within the sky sphere model; Based on the model parameters of the 3D model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model, the image shadow area corresponding to the shooting object in the background image is determined; The virtual image is generated based on the real image of the subject being photographed, the shadow area of ​​the image, and the background image.

2. The method according to claim 1, characterized in that, The background image of the virtual captured image, rendered by a shader based on the pixel values ​​of the aforementioned pixels, includes: The model region corresponding to the pixel in the sky sphere model and the self-emission intensity of the model region are determined. The self-emission intensity is used to characterize the illumination effect of the model region as a light source on the interior of the sky sphere model, and different model regions correspond to different self-emission intensities. The self-illuminating pixel value of the pixel is determined based on the self-illumination intensity corresponding to the pixel and the pixel value. The background image of the virtual captured image is obtained by rendering the background image through a shader based on the self-illuminating pixel value of the pixel.

3. The method according to claim 2, characterized in that, Determining the model region corresponding to the pixel in the sky sphere model and the self-emission intensity of the model region includes: Based on the viewing direction corresponding to the pixel, the model region corresponding to the pixel in the sky sphere model is determined; The self-illumination intensity of the model region is determined based on the illumination parameters configured in the sky sphere model.

4. The method according to claim 3, characterized in that, Determining the self-illumination intensity of the model region based on the illumination parameters configured in the sky sphere model includes: Based on the first illumination intensity of the ambient light, the second illumination intensity of the virtual light source, and the illumination direction configured in the sky sphere model, the first self-illumination intensity corresponding to the sky model region is determined. Based on the first illumination intensity of the ambient light, the second self-luminous intensity corresponding to the ground model area is determined.

5. The method according to claim 4, characterized in that, Determining the self-illuminating pixel value of a pixel based on its self-illumination intensity and pixel value includes: When the model region corresponding to the pixel is the sky model region, the first self-illuminating pixel value of the pixel is determined based on the first self-illuminating intensity and the pixel value. When the model region corresponding to the pixel is the ground model region, the second self-illuminating pixel value of the pixel is determined based on the second self-illuminating intensity and the pixel value.

6. The method according to claim 5, characterized in that, The background image of the virtual captured image is obtained by rendering the self-illuminating pixel value based on the pixel point through a shader, including: Based on the self-illuminating pixel value of the pixel, the pixel corresponding to the sky model area is rendered by the first shader, and the pixel corresponding to the ground model area is rendered by the second shader to obtain the background image of the virtual shooting image; The first shader is set to self-illuminating mode, and the second shader is set to both self-illuminating mode and reflection mode.

7. The method according to claim 4, characterized in that, The method further includes: Construct a three-dimensional spatial coordinate system; The sky sphere model is constructed with the axis of the three-dimensional spatial coordinate system as the center of the sphere and the input length as the radius of the sphere.

8. The method according to claim 1, characterized in that, The step of extracting the pixel value of the pixel from the panoramic background image corresponding to the sky sphere model based on the viewing direction includes: The line-of-sight vector corresponding to the line-of-sight direction is standardized to obtain a line-of-sight unit vector; Based on the mapping relationship between the sky sphere model and the panoramic background image, the gaze direction unit vector is transformed from three-dimensional space to texture space to obtain the texture coordinates of the gaze direction unit vector in the panoramic background image. Based on the texture coordinates, the pixel value of the pixel point is extracted from the panoramic background image.

9. The method according to claim 1, characterized in that, The determination of the image shadow region corresponding to the photographed object in the background image based on the model parameters of the 3D model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model includes: Based on the model parameters of the 3D model, the virtual shooting position, and the light source direction of the virtual light source configured in the sky sphere model, the shadow area of ​​the 3D model in the sky sphere model is determined; Based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, and the lighting parameters configured in the sky sphere model, the image shadow area corresponding to the photographed object in the background image is rendered.

10. The method according to claim 9, characterized in that, The process of rendering the image shadow area corresponding to the photographed object in the background image based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, and the lighting parameters configured in the sky sphere model includes: Based on the camera parameters, the shadow area of ​​the 3D model in the sky sphere model, the first illumination intensity of the ambient light configured in the sky sphere model, and the second illumination intensity of the virtual light source, the image shadow area corresponding to the photographed object in the background image is rendered. The brightness of the image shadow area is positively correlated with the intensity difference between the second illumination intensity and the first illumination intensity.

11. A virtual shooting device, characterized in that, The device includes: The line-of-sight determination module is used to determine the line-of-sight direction of pixels in a virtual captured image relative to the virtual camera based on the camera parameters of the virtual camera in the sky sphere model. The virtual captured image refers to an image captured within the sky sphere model from the shooting perspective of the virtual camera. The sky sphere model includes a sky model area and a ground model area. The spherical surface of the sky sphere model is the sky model area, and the horizontal hemisphere of the sky sphere model is the ground model area. The ground model area allows other 3D models to cast shadows, but the sky model area will not cast shadows in the ground model area. The pixel extraction module is used to extract the pixel value of the pixel from the panoramic background image corresponding to the sky sphere model based on the viewing direction. The panoramic background image is an image used to be mapped onto the sky sphere model as a virtual shooting background. The first image generation module is used to render the background image of the virtual captured image by a shader based on the pixel values ​​of the pixel points. The position determination module is used to determine the virtual shooting position of the 3D model corresponding to the shooting object in the sky sphere model; The region determination module is used to determine the image shadow region corresponding to the shooting object in the background image based on the model parameters of the three-dimensional model, the virtual shooting position, the camera parameters, and the lighting parameters configured in the sky sphere model. The second image generation module is used to generate the virtual image based on the real image corresponding to the subject being photographed, the shadow area of ​​the image, and the background image.

12. A computer device, characterized in that, The computer device includes a processor and a memory; the memory stores at least one computer instruction, which is executed by the processor to implement the virtual shooting method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer instruction, which is executed by a processor to implement the virtual shooting method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes at least one computer instruction stored in a computer-readable storage medium; the processor of the computer device reads the at least one computer instruction from the computer-readable storage medium and executes the at least one computer instruction, causing the computer device to implement the virtual shooting method as described in any one of claims 1 to 10.

Images