Image generation method and device based on virtual camera, equipment, medium and program product

Abstract

The invention relates to an image generation method and device based on a virtual camera, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: in response to a trigger operation on a target position on a video session screen, taking the target position as a virtual arrangement position of a virtual camera on the video session screen, and arranging a plurality of physical cameras around the video session screen; determining a target space parameter between the virtual camera and each physical camera under the target coordinate system based on the physical layout position and the virtual layout position of each physical camera; obtaining a plurality of scene images, wherein each scene image is obtained by shooting a target object by a corresponding physical camera; and based on the target space parameter corresponding to each physical camera, the plurality of scene images and the virtual camera parameter of the virtual camera, image rendering is performed to obtain a rendered image, the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for generating an image based on a virtual camera. Background Art

[0002] With the development of Internet technology, video conversation has become an indispensable part of people's daily communication and remote collaboration. At present, ordinary video conferencing systems generally have cameras, which can be roughly divided into two categories according to the type of camera the system has: one is a conference system with only a single camera, which transmits the image captured by the camera to the other party in real time; the other is a conference system with multiple cameras, which will transmit the images captured by different cameras to the other party in real time at the appropriate time.

[0003] However, these two types of conference systems transmit real images taken by physical cameras, and can only shoot at fixed angles of the physical camera. They cannot shoot flexibly according to needs, and it is difficult to meet the personalized needs of users. Summary of the invention

[0004] Based on this, it is necessary to provide an image generation method, device, computer equipment, computer-readable storage medium and computer program product based on a virtual camera, which can flexibly shoot based on a virtual camera to meet the personalized needs of users and improve user experience in response to the above technical problems.

[0005] In a first aspect, the present application provides an image generation method based on a virtual camera, comprising:

[0006] In response to a triggering operation on a target position on the video conversation screen, the target position is used as a virtual placement position of a virtual camera on the video conversation screen, and a plurality of physical cameras are placed around the video conversation screen;

[0007] Determine target space parameters between the virtual camera and each physical camera in a target coordinate system based on the physical layout position of each physical camera and the virtual layout position;

[0008] Acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera;

[0009] Based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera, image rendering is performed to obtain a rendered image, where the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0010] In a second aspect, the present application also provides an image generation device based on a virtual camera, comprising:

[0011] a position determination module, for responding to a trigger operation on a target position on the video conversation screen, using the target position as a virtual placement position of a virtual camera on the video conversation screen, wherein a plurality of physical cameras are placed around the video conversation screen;

[0012] A parameter determination module, used to determine the target space parameters between the virtual camera and each physical camera in the target coordinate system based on the physical layout position of each physical camera and the virtual layout position;

[0013] An image acquisition module is used to acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera;

[0014] The image rendering module is used to perform image rendering based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera to obtain a rendered image, wherein the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0015] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] In response to a triggering operation on a target position on the video conversation screen, the target position is used as a virtual placement position of a virtual camera on the video conversation screen, and a plurality of physical cameras are placed around the video conversation screen;

[0017] Determine target space parameters between the virtual camera and each physical camera in a target coordinate system based on the physical layout position of each physical camera and the virtual layout position;

[0018] Acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera;

[0019] Based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera, image rendering is performed to obtain a rendered image, where the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0021] In response to a triggering operation on a target position on the video conversation screen, the target position is used as a virtual placement position of a virtual camera on the video conversation screen, and a plurality of physical cameras are placed around the video conversation screen;

[0022] Determine target space parameters between the virtual camera and each physical camera in a target coordinate system based on the physical layout position of each physical camera and the virtual layout position;

[0023] Acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera;

[0024] Based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera, image rendering is performed to obtain a rendered image, where the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0025] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0026] In response to a triggering operation on a target position on the video conversation screen, the target position is used as a virtual placement position of a virtual camera on the video conversation screen, and a plurality of physical cameras are placed around the video conversation screen;

[0027] Determine target space parameters between the virtual camera and each physical camera in a target coordinate system based on the physical layout position of each physical camera and the virtual layout position;

[0028] Acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera;

[0029] Based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera, image rendering is performed to obtain a rendered image, where the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0030] The above-mentioned image generation method, device, computer equipment, computer-readable storage medium and computer program product based on virtual camera, by responding to the trigger operation of the target position on the video conversation screen, uses the target position as the virtual layout position of the virtual camera on the video conversation screen, and multiple physical cameras are arranged around the video conversation screen. That is, as the position selected by the user in the video conversation screen, the position is automatically set as the placement position of the virtual camera, so that the user can set the virtual camera at any position on the video conversation screen according to actual needs, flexibly set the virtual camera, and improve the flexibility of the virtual camera setting process. Based on the physical layout position and virtual layout position of each physical camera, the target space parameters between the virtual camera and each physical camera in the target coordinate system are determined to reflect the correlation between the virtual camera and the physical camera in the spatial position; multiple scene images are obtained, each scene image is obtained by shooting the target object by the corresponding physical camera; based on the target space parameters corresponding to each physical camera, the multiple scene images, and the virtual camera parameters of the virtual camera, the image is rendered to obtain a rendered image. That is, after shooting the target object with a physical camera, the detailed information of the target object at the angle can be captured at the angle of the physical camera at different positions. Then, the spatial correlation between the physical camera and the virtual camera and the virtual camera parameters are combined to simulate the virtual camera shooting the target object at the virtual camera position. The whole process can be flexibly shot based on the actual needs of users, meet the personalized needs of users, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] FIG1 is a diagram showing an application environment of an image generation method based on a virtual camera in one embodiment;

[0033] Figure 2 A diagram showing an application environment of an image generation method based on a virtual camera in another embodiment;

[0034] Figure 3 A schematic diagram of a flow chart of an image generation method based on a virtual camera in one embodiment;

[0035] Figure 4 A schematic diagram of a video conversation scenario in an embodiment;

[0036] Figure 5A schematic diagram of a target space parameter determination step in one embodiment;

[0037] Figure 6 is a schematic diagram of a video conversation screen in a rectangular coordinate system in one embodiment;

[0038] Figure 7 is a schematic diagram of a video conversation screen in a polar coordinate system in one embodiment;

[0039] Figure 8 is a focusing schematic diagram of a virtual camera in one embodiment;

[0040] Fig. 9 A schematic diagram of an image preview pop-up window in one embodiment;

[0041] Fig.10 is a schematic diagram of an image preview pop-up window in another embodiment;

[0042] Fig.11 A schematic diagram of rendering image generation in one embodiment;

[0043] Fig.12 is a structural block diagram of an image generating device based on a virtual camera in one embodiment;

[0044] FIG. 13 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The image generation method based on a virtual camera provided in the embodiment of the present application can be applied in the application environment as shown in FIG. 1. The terminal 102 communicates with multiple camera modules 104 deployed on the outer edge of the terminal 102. For example, at least two camera modules 104 are arranged around the terminal 102, such as a camera module 104 is arranged on each of the four borders of the terminal 102 as shown in FIG. 1. At least one physical camera is arranged in each camera module 104, such as Figure 2 In another application scenario shown, Figure 2 The numbers 1 to 4 represent four camera modules, and the numbers 5 to 8 represent physical cameras. A physical camera is a camera with an independent physical camera device. A virtual camera (VCAM) is a device or software that uses computer vision and graphics technology to simulate the working principle of a real camera. That is, it does not have a physical camera device, but can simulate the shooting function of a physical camera for simulated shooting.

[0047] In some embodiments, the terminal 102 responds to a trigger operation on a target position on a video conversation screen in the terminal 102, and uses the target position as a virtual layout position of a virtual camera on the video conversation screen. A plurality of camera modules 104 are respectively arranged around the video conversation screen, and at least one physical camera is arranged in each camera module 104; based on the physical layout position and the virtual layout position of each physical camera, the target space parameters between the virtual camera and each physical camera in the target coordinate system are determined; the terminal 102 obtains a corresponding scene image sent by each camera module 104, and each scene image is obtained by photographing the target object with the corresponding physical camera; the terminal 102 performs image rendering based on the target space parameters corresponding to each physical camera, the plurality of scene images, and the virtual camera parameters of the virtual camera to obtain a rendered image, and the rendered image is obtained by simulating the virtual camera photographing the target object at the virtual camera position.

[0048] Among them, the terminal 102 is a terminal with a real-time video call function. Exemplarily, the terminal 102 can be regarded as an initiating terminal for initiating virtual camera shooting. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, etc. The Internet of Things devices can be smart TVs, smart car devices, projection devices, etc. The camera module 104 is used to send the scene image captured by the physical camera in the camera module 104 to the terminal 102. The camera module 104 includes a physical camera, which can be an RGB (Red, Green, Blue; color camera) camera or an RGB-D (RedGreenBlue-Depth, color depth) camera. Among them, the RGB camera, also known as a color camera, can simultaneously capture the three main color channels of red, green and blue of the image, thereby generating a complete color image; the RGB-D camera is a camera that combines RGB color images and depth images with three-dimensional visual sensing technology. It obtains three-dimensional information of the scene by collecting color images and depth information in the scene.

[0049] In an exemplary embodiment, Figure 3 As shown, a method for generating an image based on a virtual camera is provided, and the method is applied to the terminal 102 in FIG. 1 as an example for explanation, including the following steps S302 to S308. Among them:

[0050] Step S302, in response to a triggering operation on a target position on the video conversation screen, the target position is used as a virtual placement position of a virtual camera on the video conversation screen, and a plurality of physical cameras are placed around the video conversation screen.

[0051] The video session screen is a screen where a video session is in progress, and the video session includes but is not limited to conference sessions, teaching sessions, training sessions, live broadcasts, and social sessions. The video session can be a session based on naked-eye 3D (3 Dimensions) or a non-3D session. For example, when the video session is a session based on naked-eye 3D, the video session screen is a screen equipped with a naked-eye 3D display. The naked-eye 3D display is a visual experience technology that does not require any glasses or helmets or other auxiliary equipment. It allows viewers to directly watch 3D images with a sense of stereo with their naked eyes. Trigger operations include but are not limited to mouse click operations, mouse sliding operations, touch screen operations, etc.; accordingly, the target position can be determined based on the mouse, or it can be determined based on the user's part touching the video session screen. Multiple physical cameras are arranged around the video session screen. Exemplarily, based on the shape of the video session screen, at least one physical camera is arranged in different directions to obtain detailed information of the target object from different perspectives. Exemplarily, the shape of the video session screen is a rectangle, and at least one physical camera is arranged in each border of the rectangle, and the participants Figure 2 Layout.

[0052] The virtual placement position is used to represent the placement position of the virtual camera on the video session screen.

[0053] Optionally, during a video conversation, when it is verified that the virtual camera startup conditions are met, the terminal (which can be regarded as an initiating terminal that initiates virtual camera shooting) responds to a trigger operation on a target position on the video conversation screen and uses the target position as a virtual layout position of the virtual camera on the video conversation screen.

[0054] Exemplarily, during a video conversation, a virtual camera shooting control is displayed on a video conversation screen of a terminal, and in response to a triggering operation on the virtual camera shooting control, it is determined that a virtual camera start condition is satisfied.

[0055] Exemplarily, during a video conversation, the terminal receives a voice command from any conversation member of the video conversation to start a virtual camera, and determines that a virtual camera start condition is met.

[0056] For example, Figure 4FIG. 1 is a schematic diagram of a video conversation scenario in an embodiment. Terminal 1 of video call party A and terminal 2 of video call party B conduct a video conversation via the Internet, and the screens of terminal 1 and terminal 2 respectively display the video conversation screens of A and B synchronously. Taking video call party A as the main body for explanation, after video call party A clicks the virtual camera start control in terminal 1, video call party A clicks the target position of the video conversation screen of terminal 1, and terminal 1 determines that the target position is the virtual layout position of the virtual camera, which is used to perform image simulation on video call party A.

[0057] Step S304 : determining target space parameters between the virtual camera and each physical camera in the target coordinate system based on the physical layout position and the virtual layout position of each physical camera.

[0058] Among them, for each physical camera, the target space parameters of the physical camera and the virtual camera are used to characterize the spatial position relationship between the physical camera and the virtual camera in the target coordinate system, and the target control parameters include but are not limited to distance, angle, etc. The target coordinate system can be at least one of a rectangular coordinate system and a polar coordinate system.

[0059] Exemplarily, the terminal obtains the physical layout position of each physical camera on the video session screen, and selects a target coordinate system from a plurality of preset coordinate systems. For each physical camera, the terminal determines the target space parameter corresponding to the physical camera according to the physical layout position and the virtual layout position of the physical camera, and the target space parameter refers to the target space parameter between the virtual camera and each physical camera in the target coordinate system. For example, for each physical camera, the terminal determines at least one of the distance and angle between the physical camera and the virtual camera in the target coordinate system according to the physical layout position and the virtual layout position of the physical camera.

[0060] Step S306, obtaining a plurality of scene images, each of which is obtained by photographing the target object with a corresponding physical camera.

[0061] The target object may be a target in the scene where the terminal is located. In some embodiments, the target object may be the party who initiated the video call with the virtual camera. In some embodiments, the target object may also be the environment where the terminal is located. In other embodiments, the target object may also be the party who did not initiate the video call with the virtual camera, which is not specifically limited.

[0062] Step S308, performing image rendering based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera to obtain a rendered image, wherein the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0063] The virtual camera parameters may be default parameters, which do not need to be set by the user, or may be parameters input by the user according to the requirements. The virtual camera may be the pitch angle, rotation angle, etc. of the virtual camera.

[0064] Exemplarily, the terminal generates a rendered image through an image rendering algorithm according to the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera. The rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0065] Among them, the image rendering algorithm can be NeRF (Neural Radiance Fields, an implicit field representation method), which can learn continuous volume representation of scenes through deep neural networks and be used for high-quality new perspective synthesis. The image rendering algorithm can also be Spacetime Gaussian Feature Splatting: This is a new perspective synthesis method for dynamic scenes, aiming to achieve high resolution, realistic rendering effects, real-time rendering and compact storage.

[0066] In the above-mentioned image generation method based on virtual camera, by responding to the trigger operation of the target position on the video conversation screen, the target position is used as the virtual layout position of the virtual camera on the video conversation screen, and multiple physical cameras are arranged around the video conversation screen. That is, as the position selected by the user in the video conversation screen, the position is automatically set as the placement position of the virtual camera, so that the user can set the virtual camera at any position on the video conversation screen according to actual needs, flexibly set the virtual camera, and improve the flexibility of the virtual camera setting process. Based on the physical layout position and virtual layout position of each physical camera, the target space parameters between the virtual camera and each physical camera in the target coordinate system are determined to reflect the correlation between the virtual camera and the physical camera in the spatial position; multiple scene images are obtained, each scene image is obtained by shooting the target object by the corresponding physical camera; based on the target space parameters corresponding to each physical camera, the multiple scene images, and the virtual camera parameters of the virtual camera, the image is rendered to obtain a rendered image. That is, after shooting the target object with a physical camera, the detailed information of the target object at the angle can be captured at the angle of the physical camera at different positions. Then, the spatial correlation between the physical camera and the virtual camera and the virtual camera parameters are combined to simulate the virtual camera shooting the target object at the virtual camera position. The whole process can be flexibly shot based on the actual needs of users, meet the personalized needs of users, and improve the user experience.

[0067] In some embodiments, Figure 5FIG. 1 is a schematic diagram of a target space parameter determination step in an embodiment. The target coordinate system includes a rectangular coordinate system and a polar coordinate system. Based on the physical layout position and the virtual layout position of each physical camera, the target space parameters between the virtual camera and each physical camera in the target coordinate system are determined, including:

[0068] Step S502 : Based on the virtual layout position, first virtual coordinates of the virtual camera in the rectangular coordinate system and second virtual coordinates in the polar coordinate system are determined respectively.

[0069] For example, Figure 6 As shown, it is a schematic diagram of a video conversation screen in a rectangular coordinate system in one embodiment. The center of the video conversation screen is taken as the origin of the rectangular coordinate system, and the x-axis and y-axis are determined based on the up and down, left and right of the video conversation screen to establish a rectangular coordinate system. That is, the x-axis is in the middle of the upper and lower parts of the video conversation screen, the right is the positive direction of the x-axis, and the left is the negative direction of the x-axis. The y-axis is in the middle of the left and right parts of the video conversation screen, the top is the positive direction of the y-axis, and the bottom is the negative direction of the y-axis. The coordinates of the center of the screen are (0,0), and the size of the video conversation screen is established: the length and width are 2Xc and 2Yc respectively; if the virtual camera is at point A of the video conversation screen, the first virtual coordinate in the rectangular coordinate system is (Xa, Ya).

[0070] like Figure 7 , which is a schematic diagram of a video conversation screen in a polar coordinate system in one embodiment. The center of the video conversation screen is taken as the pole of the polar coordinate system, and the ray Ox directed horizontally to the right from the pole is taken as the polar axis to establish the polar coordinate system. Then, the second virtual coordinate of the virtual camera in the polar coordinate system is (ra, θa), where ra = θa is the angle with the polar coordinates.

[0071] Step S504 : for each physical camera, based on the physical layout position of the physical camera, respectively determine the first physical coordinates of the physical camera in the rectangular coordinate system and the second physical coordinates of the physical camera in the polar coordinate system.

[0072] Reference Figure 2 The layout of the physical camera, combined with Figure 6 , it can be known that the first physical coordinates of the physical cameras 5-8 in the rectangular coordinate system are (-Xc, 0), (Xc, 0), (0, Yc), (0, -Yc). Figure 7 The second physical coordinates of the physical cameras 5-8 in the polar coordinate system are (Xc, π), (Xc, 0), (Yc, π / 2), and (Yc, 3π / 2).

[0073] Step S506 , for each physical camera, determine a first spatial parameter between the physical camera and the virtual camera according to the first physical coordinate and the first virtual coordinate, and determine a second spatial parameter between the physical camera and the virtual camera according to the second physical coordinate and the second virtual coordinate.

[0074] Exemplarily, for each physical camera, the angle between the physical camera and the virtual camera is calculated according to the first physical coordinate and the first virtual coordinate as the corresponding first spatial parameter. According to the second physical coordinate and the second virtual coordinate, the angle between the physical camera and the virtual camera is determined as the corresponding second spatial parameter.

[0075] Continuing with the above example, the first physical coordinates of the physical cameras 5-8 in the rectangular coordinate system are (-Xc, 0), (Xc, 0), (0, Yc), (0, -Yc), and the first virtual coordinates of the virtual camera are (Xa, Ya); the distances from the virtual cameras to the physical cameras 5-8 are recorded as the first spatial parameters, recorded as d1, d2, d3, and d4 respectively.

[0076] The second physical coordinates of the physical cameras 5-8 in the polar coordinate system are (Xc, π), (Xc, 0), (Yc, π / 2), and (Yc, 3π / 2). The second virtual coordinates of the virtual camera are (ra, a); then the angles between the virtual camera and the physical cameras 5-8 are recorded as the second space parameters, which are recorded as α1, α2, α3, and α4 respectively.

[0077] Step S508 : for each physical camera, taking the first spatial parameter and the second spatial parameter between the physical camera and the virtual camera as the target spatial parameter between the physical camera and the virtual camera.

[0078] In this embodiment, by determining the respective coordinates of the virtual camera and each physical camera in the rectangular coordinate system and the respective coordinates in the polar coordinate system, the spatial position relationship between the virtual camera and each physical camera is accurately and comprehensively obtained. Therefore, accurate simulation rendering can be performed based on this and the scene image taken by the physical camera to ensure the accuracy of the rendered image generated by the virtual camera.

[0079] In some embodiments, the virtual camera parameters of the virtual camera include at least a pitch angle, a rotation angle, and a focus point position parameter of the virtual camera for focusing on the target object.

[0080] like Figure 8 FIG. 1 is a schematic diagram of focusing of a virtual camera in an embodiment. Point O is the center of the video session screen, the focus point is the target object, and the distance between point O and the focus point is L.

[0081] Exemplarily, the pitch angle and rotation angle of the virtual camera may be set by default, for example, both are set to 0. The parameters of the virtual camera may be input and set according to the requirements.

[0082] Exemplarily, the virtual camera parameters also include field of view, resolution, and frame rate, which can be consistent with those of the physical camera.

[0083] In this embodiment, the virtual camera parameters of the virtual camera at least include the pitch angle, rotation angle, and focus point position parameters of the virtual camera for focusing on the target object. In this way, the shooting parameters of the virtual camera can be accurately reflected to ensure the shooting quality of the rendered image.

[0084] In some embodiments, image rendering is performed based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera to obtain a rendered image, including: calling a trained image processing model, inputting the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera into the image processing model to obtain a rendered image.

[0085] Among them, the trained image processing model is a model built based on a neural network, which can be built using a NeRF algorithm or a spatiotemporal Gaussian feature spraying algorithm.

[0086] Exemplarily, after the trained image processing model obtains the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera, the image processing model is used to determine the ray direction of each pixel in each scene image according to the scene image and the camera parameters of the physical camera, and the color and illumination information of each pixel in each scene image is determined. The image processing model is used to fit the ray direction of each pixel in the rendered image according to the target space parameters and virtual camera parameters corresponding to each physical camera; the image processing model is used to determine the intersection point corresponding to each pixel in each scene image (i.e., the intersection point of each pixel in the scene image and the scene in the scene image) according to the pose of the physical camera corresponding to each scene image. The image processing model is used to determine the color and illumination information of each intersection point corresponding to each pixel in the rendered image according to the ray direction, color and illumination information of each pixel in each scene image, the ray direction of each pixel in the rendered image, and the intersection point corresponding to each pixel in each scene image, thereby determining the rendering color of each pixel in the rendered image, and generating the rendered image according to the rendering color of each pixel.

[0087] In this embodiment, by calling the trained image processing model, image fitting (rendering) can be performed accurately and efficiently based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera to obtain an accurate rendered image.

[0088] In some embodiments, the method also includes: displaying an image preview pop-up window at a virtual layout position on the video session screen, the image preview pop-up window including a rendered image; the method also includes: in response to a trigger operation at other positions on the video session screen, displaying a new image preview pop-up window at other positions, the new preview pop-up window including a new rendered image, and the new rendered image is obtained by simulating a virtual camera shooting at other positions.

[0089] For example, Fig. 9 FIG. 1 is a schematic diagram of an image preview pop-up window in an embodiment. Fig. 9 , after the virtual camera is set at the virtual layout position A in the video session screen, an image preview pop-up window is displayed at the virtual layout position, and the obtained rendered image is displayed in the image preview pop-up window. Exemplarily, the corresponding image preview pop-up window can also be displayed at a position around the virtual layout position, for example, above the virtual layout position. Exemplarily, the virtual layout position can be displayed using a camera icon.

[0090] The triggering operation on other positions on the video session screen may be a moving operation, a dragging operation, etc. For example, in the rendered image preview state, the video call party who initiates the virtual camera shooting can hold down the virtual camera position and the image preview screen with the left mouse button, and drag the virtual camera to any position on the video session screen. During the dragging process, the virtual camera position and the rendered image in the image preview pop-up window are updated in real time, and the virtual camera position is always kept matching with the image preview pop-up window.

[0091] In the process of triggering the drag operation, it is necessary to ensure that the focus point position parameters of the virtual camera remain unchanged. For example, after the virtual camera takes the point 1.2 meters in front of the screen and at the same height as the center of the screen O as the focus point, it needs to remain unchanged during the drag operation. At the same time, the pitch angle and rotation angle of the virtual camera may need to be changed to ensure it.

[0092] For example, Fig.10 FIG. 1 is a schematic diagram of an image preview pop-up window in another embodiment. The virtual layout position of the current virtual camera is Fig.10 Position A in the figure shows the front of the target object. By dragging, change the virtual layout position from position A to Fig.10 After changing position A to position B, see the new rendered image corresponding to position B, showing the target object on the left. Or, by dragging, change from position A to Fig.10 After moving to position C in the image, see the new rendered image corresponding to position C, which shows the target object to the right.

[0093] In this embodiment, after generating a rendering image corresponding to a virtual layout position, the user can directly continue to select other positions to generate a new rendering image using the other positions as new virtual layout positions, thereby improving the user experience.

[0094] In some embodiments, the step of generating a new rendered image includes: updating the virtual camera parameters of the virtual camera to obtain updated virtual camera parameters; taking other positions as new virtual layout positions, and determining new spatial parameters between the virtual camera and each physical camera in the target coordinate system based on the new virtual layout positions and the physical layout positions of each physical camera; performing image rendering based on the new spatial parameters corresponding to each physical camera, multiple scene images, and updated virtual camera parameters to obtain a new rendered image.

[0095] Exemplarily, the terminal obtains a new pitch angle and a new rotation angle of the virtual camera, and obtains updated virtual camera parameters based on the focus point position parameters, the new pitch angle, and the new rotation angle for focusing on the target object. Based on the new virtual layout position and the physical layout position of each physical camera, the new spatial parameters between the virtual camera and each physical camera in the target coordinate system are determined; based on the new spatial parameters corresponding to each physical camera, multiple scene images, and the updated virtual camera parameters, image rendering is performed to obtain a new rendered image.

[0096] Exemplarily, after the new spatial parameters are obtained, the terminal can also reacquire a new scene image, perform image rendering based on the new spatial parameters, new scene image, and updated virtual camera parameters corresponding to each physical camera, and obtain a new rendered image.

[0097] In this embodiment, after the new virtual layout position is obtained, the virtual camera parameters of the virtual camera need to be updated in time, so that a new rendering image matching the new virtual layout position can be rendered.

[0098] In a specific embodiment, the specific implementation process refers to the following steps:

[0099] Step 1: In response to a trigger operation on a target position on a video conversation screen, the target position is used as a virtual placement position of a virtual camera on the video conversation screen, and a plurality of physical cameras are placed around the video conversation screen.

[0100] Step 2: Based on the virtual layout position, determine the first virtual coordinates of the virtual camera in the rectangular coordinate system and the second virtual coordinates in the polar coordinate system; for each physical camera, based on the physical layout position of the physical camera, determine the first physical coordinates of the physical camera in the rectangular coordinate system and the second physical coordinates in the polar coordinate system; for each physical camera, determine the first spatial parameters between the physical camera and the virtual camera according to the first physical coordinates and the first virtual coordinates, and determine the second spatial parameters between the physical camera and the virtual camera according to the second physical coordinates and the second virtual coordinates; for each physical camera, use the first spatial parameters and the second spatial parameters between the physical camera and the virtual camera as the target spatial parameters between the physical camera and the virtual camera.

[0101] Step 3: Acquire multiple scene images, each of which is obtained by photographing the target object with a corresponding physical camera. Determine virtual camera parameters of the virtual camera, which at least include the pitch angle, rotation angle, and focus point position parameters of the virtual camera for focusing on the target object.

[0102] Step 4: Call the trained image processing model, input the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera into the image processing model to obtain a rendered image. Display an image preview pop-up window at the virtual layout position on the video session screen, and the image preview pop-up window includes the rendered image.

[0103] Specifically, Fig.11 As shown, it is a schematic diagram of rendering image generation in one embodiment. When the virtual layout position is set to A, refer to the method mentioned above to obtain four first spatial parameters d1, d2, d3, d4 in the rectangular coordinate system; second spatial parameters α1, α2, α3, α4 in the polar coordinate system. The scene image (four-way real-time video image), d1, d2, d3, d4, α1, α2, α3, α4, virtual camera parameters (set virtual camera parameters, default values) are input into the image processing model (image processing algorithm model, such as NeRF / spatiotemporal Gaussian feature spraying), and the fitted rendering image (virtual camera image) is output, that is, the image taken by the virtual camera at position A.

[0104] Step 5: In response to the trigger operation of other positions on the video session screen, a new rendered image is generated, that is, the virtual camera parameters of the virtual camera are updated to obtain updated virtual camera parameters; other positions are used as new virtual layout positions, and according to the new virtual layout positions and the physical layout positions of each physical camera, the new spatial parameters between the virtual camera and each physical camera in the target coordinate system are determined; based on the new spatial parameters corresponding to each physical camera, multiple scene images, and updated virtual camera parameters, image rendering is performed to obtain a new rendered image. A new image preview pop-up window is displayed at other positions, and the new preview pop-up window includes the new rendered image.

[0105] In this embodiment, by responding to the trigger operation of the target position on the video conversation screen, the target position is used as the virtual layout position of the virtual camera on the video conversation screen, and multiple physical cameras are arranged around the video conversation screen. That is, as the position selected by the user in the video conversation screen, the position is automatically set as the placement position of the virtual camera. Therefore, the user can set the virtual camera at any position on the video conversation screen according to actual needs, flexibly set the virtual camera, and improve the flexibility of the virtual camera setting process. Based on the physical layout position and virtual layout position of each physical camera, the target space parameters between the virtual camera and each physical camera in the target coordinate system are determined to reflect the correlation between the virtual camera and the physical camera in spatial position; multiple scene images are obtained, each scene image is obtained by shooting the target object with the corresponding physical camera; based on the target space parameters corresponding to each physical camera, the multiple scene images, and the virtual camera parameters of the virtual camera, image rendering is performed to obtain a rendered image. That is, after shooting the target object with a physical camera, the detailed information of the target object at the angle can be captured at the angle of the physical camera at different positions. Then, the spatial correlation between the physical camera and the virtual camera and the virtual camera parameters are combined to simulate the virtual camera shooting the target object at the virtual camera position. The whole process can be flexibly shot based on the actual needs of users, meet the personalized needs of users, and improve the user experience.

[0106] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present application also provides a virtual camera-based image generation device for implementing the above-mentioned virtual camera-based image generation method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more virtual camera-based image generation device embodiments provided below can refer to the above-mentioned limitations on the virtual camera-based image generation method, and will not be repeated here.

[0108] In an exemplary embodiment, Fig.12 As shown, a virtual camera-based image generation device 1200 is provided, comprising: a position determination module 1202, a parameter determination module 1204, an image acquisition module 1206 and an image rendering module 1208, wherein:

[0109] A position determination module 1202 is used for responding to a trigger operation on a target position on the video conversation screen, using the target position as a virtual placement position of a virtual camera on the video conversation screen, and placing multiple physical cameras around the video conversation screen;

[0110] A parameter determination module 1204 is used to determine the target space parameters between the virtual camera and each physical camera in the target coordinate system based on the physical layout position and the virtual layout position of each physical camera;

[0111] The image acquisition module 1206 is used to acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera;

[0112] The image rendering module 1208 is used to perform image rendering based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera to obtain a rendered image. The rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

[0113] In some embodiments, the target coordinate system includes a rectangular coordinate system and a polar coordinate system. The parameter determination module 1204 is used to determine the first virtual coordinates of the virtual camera in the rectangular coordinate system and the second virtual coordinates in the polar coordinate system based on the virtual layout position; for each physical camera, based on the physical layout position of the physical camera, the first physical coordinates of the physical camera in the rectangular coordinate system and the second physical coordinates in the polar coordinate system are determined; for each physical camera, the first spatial parameters between the physical camera and the virtual camera are determined according to the first physical coordinates and the first virtual coordinates, and the second spatial parameters between the physical camera and the virtual camera are determined according to the second physical coordinates and the second virtual coordinates; for each physical camera, the first spatial parameters and the second spatial parameters between the physical camera and the virtual camera are used as the target spatial parameters between the physical camera and the virtual camera.

[0114] In some embodiments, the virtual camera parameters of the virtual camera include at least a pitch angle, a rotation angle, and a focus point position parameter of the virtual camera for focusing on the target object.

[0115] In some embodiments, the image rendering module 1208 is used to call a trained image processing model, input the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera into the image processing model to obtain a rendered image.

[0116] In some embodiments, the device also includes a display module, which is used to display an image preview pop-up window at a virtual layout position on the video session screen, and the image preview pop-up window includes a rendered image; the display module is used to respond to a trigger operation at other positions on the video session screen, and display a new image preview pop-up window at other positions, and the new preview pop-up window includes a new rendered image, and the new rendered image is obtained by simulating a virtual camera shooting at other positions.

[0117] In some embodiments, the device also includes an update module, which is used to update the virtual camera parameters of the virtual camera to obtain updated virtual camera parameters; a parameter determination module 1204 is used to take other positions as new virtual layout positions, and determine the new spatial parameters between the virtual camera and each physical camera in the target coordinate system according to the new virtual layout position and the physical layout position of each physical camera; an image rendering module is used to perform image rendering based on the new spatial parameters corresponding to each physical camera, multiple scene images, and updated virtual camera parameters to obtain a new rendered image.

[0118] Each module in the image generation device of the virtual camera can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0119] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and its internal structure diagram can be shown in Figure 13. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating an image of a virtual camera is implemented.

[0120] Those skilled in the art will understand that the structure shown in FIG. 13 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0122] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0123] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0125] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0127] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating an image based on a virtual camera, characterized in that: The method comprises: In response to a triggering operation on a target position on the video conversation screen, the target position is used as a virtual placement position of a virtual camera on the video conversation screen, and a plurality of physical cameras are placed around the video conversation screen; Determine target space parameters between the virtual camera and each physical camera in a target coordinate system based on the physical layout position of each physical camera and the virtual layout position; Acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera; Based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera, image rendering is performed to obtain a rendered image, where the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

2. The method according to claim 1, characterized in that The target coordinate system includes a rectangular coordinate system and a polar coordinate system, and the determining of the target space parameters between the virtual camera and each physical camera in the target coordinate system based on the physical layout position of each physical camera and the virtual layout position includes: Based on the virtual layout position, determining first virtual coordinates of the virtual camera in the rectangular coordinate system and second virtual coordinates in the polar coordinate system respectively; For each physical camera, based on the physical layout position of the physical camera, respectively determine a first physical coordinate of the physical camera in a rectangular coordinate system and a second physical coordinate of the physical camera in a polar coordinate system; For each physical camera, determine a first spatial parameter between the physical camera and the virtual camera according to the first physical coordinate and the first virtual coordinate, and determine a second spatial parameter between the physical camera and the virtual camera according to the second physical coordinate and the second virtual coordinate; For each physical camera, a first spatial parameter and a second spatial parameter between the physical camera and the virtual camera are used as a target spatial parameter between the physical camera and the virtual camera.

3. The method according to claim 1, characterized in that The virtual camera parameters of the virtual camera at least include a pitch angle, a rotation angle, and a focus point position parameter of the virtual camera for focusing on the target object.

4. The method according to claim 1, characterized in that: The method of performing image rendering based on the target space parameters corresponding to each physical camera, the multiple scene images, and the virtual camera parameters of the virtual camera to obtain a rendered image includes: The trained image processing model is called, and the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera are input into the image processing model to obtain the rendered image.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Displaying an image preview pop-up window at a virtual layout position on the video session screen, the image preview pop-up window including the rendered image; The method further comprises: In response to a trigger operation on other locations on the video session screen, a new image preview pop-up window is displayed at the other locations, and the new preview pop-up window includes a new rendered image, and the new rendered image is obtained by simulating the virtual camera shooting at the other locations.

6. The method according to claim 5, characterized in that The new rendering image generation step comprises: Updating virtual camera parameters of the virtual camera to obtain updated virtual camera parameters; Taking the other positions as new virtual layout positions, and determining new spatial parameters between the virtual camera and each physical camera in the target coordinate system according to the new virtual layout positions and the physical layout positions of each physical camera; Based on the new spatial parameters corresponding to each physical camera, multiple scene images, and updated virtual camera parameters, image rendering is performed to obtain a new rendered image.

7. An image generation device based on a virtual camera, characterized in that: The device comprises: a position determination module, for responding to a trigger operation on a target position on the video conversation screen, using the target position as a virtual placement position of a virtual camera on the video conversation screen, wherein a plurality of physical cameras are placed around the video conversation screen; A parameter determination module, used to determine the target space parameters between the virtual camera and each physical camera in the target coordinate system based on the physical layout position of each physical camera and the virtual layout position; An image acquisition module is used to acquire multiple scene images, each of which is obtained by photographing a target object with a corresponding physical camera; The image rendering module is used to perform image rendering based on the target space parameters corresponding to each physical camera, multiple scene images, and virtual camera parameters of the virtual camera to obtain a rendered image, wherein the rendered image is obtained by simulating the virtual camera to shoot the target object at the virtual camera position.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images