Image processing method and device for virtual shooting

Abstract

The invention relates to an image processing method and device for virtual shooting. The method comprises the following steps: receiving a latest image generated by rendering a current virtual scene from a rendering module and associated information of the latest image; generating a target number of frame insertion images according to the latest image, the associated information of the latest image and the previous image; performing projection transformation on the newest image and each frame insertion image based on the corresponding camera data to obtain a transformed newest image and each transformed frame insertion image so as to correspond to a camera; and in response to the refresh rate of the screen, sequentially sending the transformed frame insertion images and the transformed latest image to the screen, so that the screen sequentially displays the transformed frame insertion images and the transformed latest image after the previous image is displayed. The real-time frame insertion is realized while the high resolution of the virtual scene picture is ensured, the screen display picture is free of pause and smoother, the frame rate of the screen image is improved, the virtual shooting effect is improved, and the shooting requirement is met.

Description

Technical Field

[0001] The present disclosure relates to the technical field of virtual photography, and in particular to an image processing method and device for virtual photography. Background Art

[0002] During virtual filming, the crew needs to display a real-time rendered virtual scene image on the screen. To achieve a realistic effect, virtual scenes often contain a large number of hyper-realistic, high-precision models. Rendering these high-precision models requires significant hardware computing power. Given limited hardware computing power, the higher the model precision, the lower the rendering frame rate. This results in noticeable stuttering on the screen, failing to meet the requirements of virtual filming and becoming a pressing issue. Summary of the Invention

[0003] In view of this, the present disclosure proposes an image processing method and apparatus for virtual photography.

[0004] According to one aspect of the present disclosure, there is provided an image processing method for virtual photography, the method comprising:

[0005] Receive a latest image generated by rendering a current virtual scene from a rendering module and associated information of the latest image, the associated information including depth information corresponding to the latest image and motion information of the latest image relative to a previously rendered image;

[0006] generating a target number of interpolation images between the previous image and the latest image according to the latest image, associated information of the latest image, and the previous image;

[0007] Performing a projection transformation on the latest image and each of the interpolated frame images based on corresponding camera data to obtain a transformed latest image and each of the transformed interpolated frame images;

[0008] In response to the refresh rate of the screen, each of the transformed interpolated frame images and the transformed latest image is sent to the screen in sequence, so that the screen displays each of the transformed interpolated frame images and the transformed latest image in sequence after displaying the previous image.

[0009] In a possible implementation, generating a target number of interpolated images between the previous image and the latest image according to the latest image, associated information of the latest image, and the previous image includes:

[0010] transforming the latest image to generate first to-be-processed images corresponding to each of the interpolated frame images according to the frame interval between each interpolated frame image and the latest image and the association information of the latest image;

[0011] transforming the previous image to generate a second to-be-processed image corresponding to each interpolated frame image according to a frame interval between each interpolated frame image to be generated and the previous image, depth information of the previous image, and motion information of the previous image relative to the latest image;

[0012] The interpolated frame images are generated by weighted calculation based on each of the first images to be processed and the second images to be processed corresponding to the first images to be processed.

[0013] In a possible implementation, generating each of the interpolated images through weighted calculation based on each of the first images to be processed and a second image to be processed corresponding to the first image to be processed includes:

[0014] Each interpolated image is generated by weighted calculation based on each first pixel in each first image to be processed and its corresponding first weight, and each second pixel in a second image to be processed corresponding to the first image to be processed and its corresponding second weight.

[0015] In a possible implementation, generating each of the interpolated images through weighted calculation based on each of the first images to be processed and a second image to be processed corresponding to the first image to be processed further includes:

[0016] Determining, based on depth information corresponding to each first pixel in the first image to be processed and depth information corresponding to each second pixel in a second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel; or

[0017] Based on the depth information and color value corresponding to each first pixel in the first image to be processed and the depth information and color value corresponding to each second pixel in the second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel are determined.

[0018] In a possible implementation, the method further includes:

[0019] Based on the current virtual scene corresponding to the latest image, determining the target quantity corresponding to the current virtual scene from a pre-built quantity mapping table;

[0020] The quantity mapping table is used to record the target quantity corresponding to each virtual scene.

[0021] In a possible implementation, the target number is determined according to a refresh rate of the screen and an image rendering speed of the rendering module for the current virtual scene.

[0022] According to another aspect of the present disclosure, there is provided an image processing device for virtual photography, the device comprising:

[0023] an interpolation module configured to receive, from the rendering module, a latest image rendered for a current virtual scene and associated information of the latest image, the associated information including corresponding depth information and motion information of the latest image relative to a previously rendered image; and generate, based on the latest image, the associated information of the latest image, and the previous image, a target number of interpolation images between the previous image and the latest image;

[0024] A projection transformation module, configured to perform projection transformation on the latest image and each of the interpolated frame images based on corresponding camera data to obtain a transformed latest image and each of the transformed interpolated frame images;

[0025] The upper screen control module is used to send each of the transformed interpolated frame images and the transformed latest image to the screen in sequence in response to the refresh rate of the screen, so that the screen displays each of the transformed interpolated frame images and the transformed latest image in sequence after displaying the previous image.

[0026] In a possible implementation, the frame insertion module includes:

[0027] a first image generation submodule, configured to transform the latest image to generate first to-be-processed images corresponding to each interpolated frame image according to the frame interval between each interpolated frame image to be generated and the latest image and the association information of the latest image;

[0028] a second image generation submodule, configured to transform the previous image to generate a second to-be-processed image corresponding to each interpolated frame image to be generated based on a frame interval between each interpolated frame image to be generated and the previous image, depth information of the previous image, and motion information of the previous image relative to the latest image;

[0029] The interpolation frame image generation submodule is configured to generate each of the interpolation frame images through weighted calculation based on each of the first to-be-processed images and a second to-be-processed image corresponding to the first to-be-processed image.

[0030] In a possible implementation, generating each of the interpolated images through weighted calculation based on each of the first images to be processed and a second image to be processed corresponding to the first image to be processed includes:

[0031] Each interpolated image is generated by weighted calculation based on each first pixel in each first image to be processed and its corresponding first weight, and each second pixel in a second image to be processed corresponding to the first image to be processed and its corresponding second weight.

[0032] In a possible implementation, generating each of the interpolated images through weighted calculation based on each of the first images to be processed and a second image to be processed corresponding to the first image to be processed further includes:

[0033] Determining, based on depth information corresponding to each first pixel in the first image to be processed and depth information corresponding to each second pixel in a second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel; or

[0034] Based on the depth information and color value corresponding to each first pixel in the first image to be processed and the depth information and color value corresponding to each second pixel in the second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel are determined.

[0035] In a possible implementation, the apparatus further includes:

[0036] a target quantity determination module, configured to determine the target quantity corresponding to the current virtual scene from a pre-built quantity mapping table based on the current virtual scene corresponding to the latest image;

[0037] The quantity mapping table is used to record the target quantity corresponding to each virtual scene.

[0038] In a possible implementation, the target number is determined according to a refresh rate of the screen and an image rendering speed of the rendering module for the current virtual scene.

[0039] According to another aspect of the present disclosure, an image processing device for virtual photography is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0040] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0041] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0042] The image processing method and device for virtual photography provided by the embodiments of the present disclosure can generate a target number of interpolated images during the virtual photography process based on the image output by the rendering module, the depth information corresponding to the image, and the motion information of the image relative to the previous image. While ensuring the high resolution of the virtual scene displayed on the screen, real-time interpolation is achieved in the virtual photography scene, avoiding the sense of stuttering in the screen display, making the screen display smoother, increasing the frame rate of the image that can be displayed on the screen, and improving the effect of virtual photography. Moreover, after transforming the interpolated image and the image rendered by the rendering module based on the camera data, the screen display is controlled in sequence to ensure that each image displayed on the screen corresponds to the camera, ensuring that each image is displayed correctly to meet the photography requirements.

[0043] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0045] Figure 1 A schematic diagram of a virtual shooting scene is shown.

[0046] Figure 2 A flowchart of an image processing method for virtual photography according to an embodiment of the present disclosure is shown.

[0047] Figure 3 A schematic diagram illustrating an application of an image processing method for virtual photography according to an embodiment of the present disclosure is shown.

[0048] Figure 4 A schematic diagram of frame insertion of an image processing method for virtual shooting according to an embodiment of the present disclosure is shown.

[0049] Figure 5 FIG. 1 is a block diagram showing an apparatus 1900 for image processing for virtual photography according to an exemplary embodiment. DETAILED DESCRIPTION

[0050] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0051] As used herein, the terms "comprises," "comprising," "having," or variations thereof are open ended and include one or more stated features, integers, elements, steps, parts, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, parts, functions, or groups thereof.

[0052] When an element is referred to as being "connected," "coupled," "responsive" or variations thereof to another element, it can be directly connected, coupled or responsive to the other element or intervening elements may be present.

[0053] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teachings of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments.

[0054] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0055] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0056] like Figure 1 As shown in the figure, a basic virtual shooting scene includes a camera and a screen (such as an LED screen). The actor performs in front of the screen, and the camera simultaneously shoots the actor and the screen behind the actor. The designed 3D virtual scene displayed on the screen will be displayed in the area that the camera can capture (i.e., the area that the camera can capture) according to the relative position of the camera and the screen. Figure 1 The inner frustum (the inner frustum) is rendered in real time to generate images. Generating images within the inner frustum involves two steps: 1. A 2D image is rendered based on the camera's position, similar to traditional game rendering. 2. The generated 2D image is then projected and transformed based on the relative positions of the camera and screen to ensure the camera maintains the correct perspective relationship between the actor and the image on the screen.

[0057] In order to ensure the effect of virtual shooting, it is necessary to ensure that the 3D virtual scene displayed on the screen is both clear and smooth. It is usually required that the rendered image resolution is above 4K, while maintaining a frame rate of more than 25 frames per second. Otherwise, the picture on the screen will be noticeably stuttered, making it difficult to meet the shooting requirements. However, 3D virtual scenes usually use a large number of hyper-realistic high-precision models to build the scene to create a realistic virtual shooting environment. This results in a low rendering frame rate due to the high model accuracy when the hardware computing power is limited, which cannot meet the shooting requirements. In order to meet the shooting requirements, the hardware computing power can be increased, or the frame rate can be increased by simplifying the model or reducing the rendering quality. However, it is obvious that increasing the computing power is costly, and simplifying the model or reducing the rendering quality will affect the shooting effect.

[0058] To solve the above technical problems, the embodiments of the present disclosure provide an image processing method and device for virtual shooting. During the virtual shooting process, the method and device can generate a target number of interpolated images based on the image output by the rendering module, the depth information corresponding to the image, and the motion information of the image relative to the previous image. While ensuring the high resolution of the virtual scene displayed on the screen, the method realizes real-time interpolation in the virtual shooting scene, avoids the screen display from causing a sense of frustration, makes the screen display smoother, increases the frame rate of the image that can be displayed on the screen, and improves the effect of virtual shooting. Moreover, after the interpolated image and the image rendered by the rendering module are transformed based on the camera data, the screen display is controlled in sequence to ensure that each image displayed on the screen corresponds to the camera, ensuring the correct display of each image to meet the shooting requirements.

[0059] like Figure 2-Figure 4 As shown, an embodiment of the present disclosure provides an image processing method for virtual photography, which includes: steps S101 to S104.

[0060] In step S101, a rendering module receives a latest image rendered for a current virtual scene, along with associated information about the latest image. The associated information includes motion information (also known as optical flow information) of the latest image relative to a previously rendered image and depth information about the latest image. Both the previous image and the latest image are rendered output by the rendering module.

[0061] In this embodiment, during the actual virtual shooting process, the rendering module (such as a rendering engine such as Unreal Engine (UE)) will continuously receive the camera data returned by the camera, and use the camera data returned by the camera as the camera data of the virtual camera corresponding to the camera, and then perform image rendering for the current virtual scene based on the camera data, and continuously output the latest image generated by the rendering and the associated information corresponding to the latest image. The rendering engine can be functionally expanded in advance so that it can output the associated information corresponding to the image while outputting the rendered image. The camera data returned by the camera may include data reported by the camera itself (such as lens parameters, etc.) and data detected in real time by the device that tracks and monitors the camera (such as position information, posture information, etc.). These camera data may include the camera's position information, posture information, lens parameters such as focal length and aperture of the camera, time code and synchronization data, color and image data, tracking data, motion data, and other data related to the camera's movement and performance, and the present disclosure does not impose any restrictions on this.

[0062] The motion information of the latest image relative to the previous image may indicate the change in motion of the latest image relative to the previous image. The motion information of the latest image relative to the previous image may be a motion vector of the latest image relative to the previous image, or a motion vector of each pixel in the latest image relative to each pixel in the previous image. The depth information may be the depth position of each pixel in the latest image, or may be the distance information between each pixel and the camera.

[0063] In step S102, a target number of interpolated images are generated between the previous image and the latest image based on the latest image, its associated information, and the previous image. Generating interpolated images based on the previous image and the latest image improves the accuracy of the interpolated images. Furthermore, because interpolation can be performed directly using associated information, interpolation efficiency and speed are improved, enabling real-time interpolation of ultra-high resolution images (e.g., 4K and above).

[0064] In this embodiment, the target number can be determined based on the refresh rate of the screen (that is, the number of frames of images displayed on the screen per second) and the image rendering speed of the rendering module for the current virtual scene, so that the image displayed on the screen meets the virtual shooting requirements. Among them, based on the image rendering speed of the rendering module for the current virtual scene, the rendering frame rate of the rendering module for the current virtual scene can be determined, and then based on the rendering frame rate and the refresh rate of the screen, the target number of frames that need to be inserted between two adjacent rendered images can be determined, so that the number of images that can be displayed on the screen per second is greater than or equal to the refresh rate of the screen. When the rendering frame rate is A and the refresh rate of the screen is B, the target number For example, if A = 12.5 and B = 50, then

[0065] In some embodiments, the method may include: based on the current virtual scene corresponding to the latest image, determining the target quantity corresponding to the current virtual scene from a pre-constructed quantity mapping table. Wherein, the quantity mapping table is used to record the target quantity corresponding to each virtual scene. During the virtual shooting process, multiple virtual scenes may be shot, then the rendering frame rate of the rendering module for each virtual scene can be determined in advance, and then the target number of frames that need to be inserted between the rendered images for each virtual scene can be calculated in combination with the refresh rate of the screen, and finally a quantity mapping table is constructed. Wherein, the target quantity corresponding to each virtual scene is recorded in the quantity mapping table. In this way, during the actual virtual shooting process, the target quantity can be determined directly based on the quantity mapping table, which simplifies the target quantity determination process and improves overall efficiency.

[0066] In some embodiments, step S102 may include: transforming each interpolated image generated as needed based on the frame interval between the latest image and the latest image, and association information of the latest image, to generate a first to-be-processed image corresponding to each interpolated image; transforming each interpolated image generated as needed based on the frame interval between the previous image and the previous image, depth information of the previous image, and motion information relative to the latest image, to generate a second to-be-processed image corresponding to each interpolated image; and generating each interpolated image through weighted calculation based on each first to-be-processed image and the second to-be-processed image corresponding to the first to-be-processed image. In this way, transforming the latest image to generate the first to-be-processed image and transforming the previous image to generate the second to-be-processed image can ensure that the generated to-be-processed images (including the first to-be-processed image and the second to-be-processed image) are more accurate. Further generating interpolated images based on the generated first to-be-processed image and the second to-be-processed image can balance the errors in the first to-be-processed image and the second to-be-processed image, thereby forming a more accurate interpolated image and further improving the accuracy of the interpolated images.

[0067] The frame interval between each interpolated frame image and the latest image can be determined based on the target number, and the frame interval Then, assuming that the latest image is the N+1th frame image and the previous image is the Nth frame image, the process of generating the first image to be processed, which corresponds to the N+i*Δth frame interpolation image (i∈[1, Q], i is an integer) between the latest image and the previous image, can be as follows: based on the motion information of the latest image, a motion vector for each pixel in the latest image is obtained. According to the motion vector of the latest image relative to the previous image, each pixel in the latest image is reversely displaced to the position of the N+i*Δth frame by reverse projection to obtain a moved image. The depth information of the latest image is used to determine the depth information corresponding to the N+i*Δth frame, wherein the new position of each pixel in the latest image in the N+i*Δth frame can be calculated based on the depth information of the latest image, the frame interval between the latest image and the N+i*Δth frame, and the motion vector of each pixel in the latest image, and the depth information of the pixel is propagated to the new position to obtain the depth information corresponding to the N+i*Δth frame. Then, depth compensation is performed on the back-projected moved image based on the depth information corresponding to the N+i*Δth frame to correct the visual difference caused by the depth change and obtain a compensated image. Pixel interpolation and / or hole filling processing is performed on the compensated image to obtain a first image to be processed. Pixel interpolation can be performed on the compensated image to fill pixel holes caused by the motion vector. Hole filling can be performed by using the average or median value of the neighborhood pixels of the compensated image to fill the holes that still exist in the compensated image after pixel interpolation. Based on the generation process of the first image to be processed, the previous image can be transformed to generate a second image to be processed corresponding to each interpolated image according to the frame interval between each interpolated image to be generated and the previous image, and the motion information of the previous image relative to the latest image. For example, based on the motion information of the previous image relative to the latest image, a motion vector of the previous image relative to the latest image is obtained. Each pixel in the previous image is forward-shifted to the position of the N+i*Δth frame through forward projection to obtain a shifted image. Using the depth information of the previous image, the depth information of the N+i*Δth frame is determined. Depth compensation, pixel interpolation, hole filling, and other processing are then performed to obtain a second image to be processed. The motion information of the previous image relative to the latest image is determined based on the association information of the latest image.

[0068] In some embodiments, generating each interpolated image based on each first image to be processed and the second image to be processed corresponding to the first image to be processed by weighted calculation may include: generating each interpolated image based on each first pixel in each first image to be processed and its corresponding first weight, and each second pixel in the second image to be processed corresponding to the first image to be processed and its corresponding second weight. The position of the first pixel in the first image to be processed is the same as the position of the second pixel in the second image to be processed. For the pixel at position (x, y) in the interpolated image of the N+i*Δth frame, It can be calculated by the following formula: in, is the first pixel at (x, y) in the first image to be processed corresponding to the N+i*Δth frame, To correspond to The first weight of . is the second pixel at (x, y) in the second image to be processed corresponding to the N+i*Δth frame, To correspond to The first weight of .

[0069] In some embodiments, generating each of the interpolated images by weighted calculation based on each of the first to-be-processed images and the second to-be-processed images corresponding to the first to-be-processed images may further include: a weight determination step. The weight determination step may include the following two methods:

[0070] Weight determination method 1: Determine the first weight corresponding to each first pixel and the second weight corresponding to each second pixel based on the depth information corresponding to each first pixel in the first image to be processed and the depth information corresponding to each second pixel in the second image to be processed corresponding to the first image to be processed. The first weight corresponding to the second pixel The sum is 1, that is The smaller the depth of the pixel, the greater the corresponding weight. For example, for the N+i*Δ frame position (x, y), if the depth of the first pixel is The depth of the second pixel is and but and

[0071] The calculation method of the first weight and the second weight in the first weight determination method can be set according to actual needs, and the present disclosure does not limit this.

[0072] For example, the first weight and the second weight can be calculated by the following formula:

[0073]

[0074] like but

[0075] like but

[0076] in, and are the weights before normalization of the first weight and the second weight respectively.

[0077] final, Among them, α is the weight adjustment coefficient, which is between [0,1] and is used to control the depth difference The degree of influence on weight.

[0078] By determining the first weight and the second weight in the above manner, the interpolated image can be more naturally integrated with the depth information, thereby avoiding depth inconsistency or visual conflict and improving the visual effect of the interpolated image.

[0079] Method 2 for determining weights: Determine the first weight corresponding to each first pixel and the second weight corresponding to each second pixel based on the depth information and color value corresponding to each first pixel in the first image to be processed and the depth information and color value corresponding to each second pixel in the second image to be processed corresponding to the first image to be processed.

[0080] Among them, the depth sub-weight of the first pixel and the depth sub-weight corresponding to the second pixel can be calculated based on the depth information, and the color sub-weight of the first pixel and the color sub-weight corresponding to the second pixel can be calculated based on the color information, and then further weighted calculation is performed to calculate the first weight based on the depth sub-weight and color sub-weight of the first pixel, and the second weight based on the depth sub-weight and color sub-weight of the second pixel. Among them, the smaller the depth of the pixel, the larger the corresponding depth sub-weight. The greater the similarity and / or the greater the contrast between the pixel colors, the larger the corresponding color sub-weight. This application does not limit the specific relationship between the pixel color and the color sub-weight.

[0081] For example, the color sub-weight of the first pixel can be calculated by the following formula: The color sub-weight corresponding to the second pixel

[0082]

[0083] in, Represents the similarity between the pixel value of the first pixel at position (x, y) in the first to-be-processed image of the N+i*Δ th frame and the pixel value of the second pixel at position (x, y) in the second to-be-processed image of the N+i*Δ th frame. is the Euclidean distance between the pixel value of the first pixel at position (x, y) in the first to-be-processed image of the N+i*Δ th frame and the pixel value of the second pixel at position (x, y) in the second to-be-processed image of the N+i*Δ th frame.

[0084] and then, Among them, γ is the coefficient that balances depth and color weights and is between [0,1]. and are the weights before normalization of the first weight and the second weight respectively.

[0085] final,

[0086] By determining the first weight and the second weight in the above manner, the interpolated image can more naturally fuse depth information and color information, avoid depth inconsistency or visual conflict, and color conflict problems, and improve the visual effect of the interpolated image.

[0087] In step S103, the latest image and each interpolated image are projectively transformed based on the corresponding camera data to obtain a transformed latest image and each transformed interpolated image. This projective transformation ensures that each image to be displayed on the screen (the transformed latest image and each transformed interpolated image) corresponds to the camera, ensuring that each image is displayed correctly and meets the shooting requirements.

[0088] In this case, the corresponding camera position and orientation can be determined based on the camera data corresponding to the latest image, and then the latest image can be transformed corresponding to the camera position and orientation to obtain the transformed latest image. In some embodiments, the method for forming the transformed latest image can be: constructing a camera transformation matrix based on the camera position and orientation corresponding to the latest image (the matrix is ​​used to convert the camera position and orientation into a transformation in the image coordinate system, such as a translation transformation and a rotation transformation), and then performing a coordinate transformation on each pixel in the latest image based on the camera transformation matrix, and projecting the coordinate transformation results into the target coordinate system to obtain the new position corresponding to each pixel (the new position corresponds to the camera position and orientation). If the new position is not an integer coordinate or the image needs to be converted to a different resolution, the pixel values ​​of each pixel in the latest image can be interpolated and resampled for the new position to obtain the pixel values ​​corresponding to each new position, thereby obtaining the transformed latest image. In this way, the camera's posture information can be accurately mapped to the image coordinate system, ensuring that the transformation result conforms to the actual scene. It is applicable to images of different resolutions and perspectives and has strong versatility.

[0089] Among them, the camera position and direction corresponding to each interpolated image can be determined based on the camera data corresponding to the latest image, and then each interpolated image can be transformed corresponding to the camera position and direction by sampling the method of generating the transformed latest image as above to obtain the transformed interpolated image.

[0090] The camera position and direction corresponding to each interpolated image can be determined according to actual needs, and the present disclosure does not limit this. For example, the camera position corresponding to each interpolated image can be calculated by the following formula:N+i*Δ =P N *i*Δ+P N+1 *(1-i*Δ). Where, P N+i*Δ Indicates the camera position corresponding to the N+i*Δth frame interpolation image, P N Indicates the camera position corresponding to the Nth frame image (that is, the last image rendered by the rendering module), P N+1 Indicates the camera position corresponding to the N+1th frame image (that is, the latest image rendered by the rendering module).

[0091] In step S104, in response to the refresh rate of the screen, each of the transformed interpolated images and the transformed latest image is sent to the screen in sequence, so that the screen displays each of the transformed interpolated images and the transformed latest image in sequence after displaying the previous image.

[0092] In this embodiment, in response to the refresh rate of the screen, when the Vsync signal (vertical synchronization signal) returned by the screen is received, the next image to be displayed after the currently displayed image can be sent to the screen, wherein the next image to be displayed can be the transformed interpolated image or the latest image after the transformation.

[0093] Through the method provided by the embodiment of the present disclosure, Figure 4 As shown, after receiving the first frame image and its associated information rendered by the rendering module, Q frames of interpolated images before the first frame image are generated based on the first frame image and its associated information, and then the screen is controlled to display the first frame image in sequence. Frame interpolation image, Frame interpolation images... Furthermore, in the process of controlling the screen to display images, the second frame image and its associated information output by the rendering module are continuously received, and based on the second frame image and its associated information and the first frame image, Q frame interpolated images are generated between the first frame image and the second frame image, so that after the screen is determined to display the first frame image, the control screen is controlled to display the first frame image in sequence. Frame interpolation image, Frame interpolation images... By inserting frames of images, the second frame of images, and so on, the screen display images in virtual shooting can meet the shooting needs.

[0094] The present disclosure also provides an image processing device for virtual photography, the device comprising:

[0095] an interpolation module configured to receive, from the rendering module, a latest image generated for rendering the current virtual scene and associated information of the latest image, the associated information including motion information of the latest image relative to a previous rendered image and depth information corresponding to the latest image; and generate, based on the latest image, the associated information of the latest image, and the previous image, a target number of interpolation images between the previous image and the latest image;

[0096] A projection transformation module, configured to perform projection transformation on the latest image and each of the interpolated frame images based on corresponding camera data to obtain a transformed latest image and each of the transformed interpolated frame images;

[0097] The upper screen control module is used to send each of the transformed interpolated frame images and the transformed latest image to the screen in sequence in response to the refresh rate of the screen, so that the screen displays each of the transformed interpolated frame images and the transformed latest image in sequence after displaying the previous image.

[0098] In a possible implementation, the frame insertion module includes:

[0099] a first image generation submodule, configured to transform the latest image to generate first to-be-processed images corresponding to each interpolated frame image according to the frame interval between each interpolated frame image to be generated and the latest image and the association information of the latest image;

[0100] a second image generation submodule, configured to transform the previous image to generate a second to-be-processed image corresponding to each interpolated frame image to be generated based on a frame interval between each interpolated frame image to be generated and the previous image, depth information of the previous image, and motion information of the previous image relative to the latest image;

[0101] The interpolation frame image generation submodule is configured to generate each of the interpolation frame images through weighted calculation based on each of the first to-be-processed images and a second to-be-processed image corresponding to the first to-be-processed image.

[0102] In a possible implementation, generating each of the interpolated images through weighted calculation based on each of the first images to be processed and a second image to be processed corresponding to the first image to be processed includes:

[0103] Each interpolated image is generated by weighted calculation based on each first pixel in each first image to be processed and its corresponding first weight, and each second pixel in a second image to be processed corresponding to the first image to be processed and its corresponding second weight.

[0104] In a possible implementation, generating each of the interpolated images through weighted calculation based on each of the first images to be processed and a second image to be processed corresponding to the first image to be processed further includes:

[0105] Determining, based on depth information corresponding to each first pixel in the first image to be processed and depth information corresponding to each second pixel in a second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel; or

[0106] Based on the depth information and color value corresponding to each first pixel in the first image to be processed and the depth information and color value corresponding to each second pixel in the second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel are determined.

[0107] In a possible implementation, the apparatus further includes:

[0108] a target quantity determination module, configured to determine the target quantity corresponding to the current virtual scene from a pre-built quantity mapping table based on the current virtual scene corresponding to the latest image;

[0109] The quantity mapping table is used to record the target quantity corresponding to each virtual scene.

[0110] In a possible implementation, the target number is determined according to a refresh rate of the screen and an image rendering speed of the rendering module for the current virtual scene.

[0111] It should be noted that while the above embodiments are used as examples to describe the image processing method and apparatus for virtual photography, those skilled in the art will appreciate that the present disclosure is not limited thereto. In fact, users can flexibly configure the various steps and modules based on their personal preferences and / or actual application scenarios, as long as they comply with the technical solutions of the present disclosure.

[0112] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0113] An embodiment of the present disclosure further provides an image processing device for virtual photography, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0114] An embodiment of the present disclosure further provides a non-volatile computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0115] An embodiment of the present disclosure further provides a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0116] Figure 5 1 is a block diagram of an apparatus 1900 for image processing for virtual photography according to an exemplary embodiment. For example, the apparatus 1900 may be provided as a server or a terminal device. Figure 5 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0117] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0118] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0119] A computer-readable storage medium can be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0120] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0121] The computer program (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by utilizing state information of computer-readable program instructions to personalize and customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0122] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0123] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0124] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0125] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0126] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method for virtual photography, characterized in that: The method comprises: Receive a latest image generated by rendering a current virtual scene from a rendering module and associated information of the latest image, the associated information including depth information corresponding to the latest image and motion information of the latest image relative to a previously rendered image; generating a target number of interpolation images between the previous image and the latest image according to the latest image, associated information of the latest image, and the previous image; Performing a projection transformation on the latest image and each of the interpolated frame images based on corresponding camera data to obtain a transformed latest image and each of the transformed interpolated frame images; In response to the refresh rate of the screen, each of the transformed interpolated frame images and the transformed latest image is sent to the screen in sequence, so that the screen displays each of the transformed interpolated frame images and the transformed latest image in sequence after displaying the previous image.

2. The method according to claim 1, characterized in that Generating a target number of interpolated images between the previous image and the latest image according to the latest image, associated information of the latest image, and the previous image, comprising: transforming the latest image to generate first to-be-processed images corresponding to each of the interpolated frame images according to the frame interval between each interpolated frame image and the latest image and the association information of the latest image; transforming the previous image to generate second to-be-processed images corresponding to each interpolated frame image according to a frame interval between each interpolated frame image and the previous image, depth information of the previous image, and motion information of the previous image relative to the latest image; The interpolated frame images are generated by weighted calculation based on each of the first images to be processed and the second images to be processed corresponding to the first images to be processed.

3. The method according to claim 2, characterized in that Generating each of the interpolated frame images through weighted calculation based on each of the first to-be-processed images and a second to-be-processed image corresponding to the first to-be-processed image includes: Each interpolated image is generated by weighted calculation based on each first pixel in each first image to be processed and its corresponding first weight, and each second pixel in a second image to be processed corresponding to the first image to be processed and its corresponding second weight.

4. The method according to claim 3, characterized in that Generating each of the interpolated frame images through weighted calculation based on each of the first to-be-processed images and a second to-be-processed image corresponding to the first to-be-processed image, further comprising: determining, based on depth information corresponding to each first pixel in the first image to be processed and depth information corresponding to each second pixel in a second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel; or Based on the depth information and color value corresponding to each first pixel in the first image to be processed and the depth information and color value corresponding to each second pixel in the second image to be processed corresponding to the first image to be processed, a first weight corresponding to each first pixel and a second weight corresponding to each second pixel are determined.

5. The method according to claim 1, wherein The method further comprises: Based on the current virtual scene corresponding to the latest image, determining the target quantity corresponding to the current virtual scene from a pre-built quantity mapping table; The quantity mapping table is used to record the target quantity corresponding to each virtual scene.

6. The method according to claim 1 or 5, characterized in that The target number is determined according to the refresh rate of the screen and the image rendering speed of the rendering module for the current virtual scene.

7. An image processing device for virtual photography, characterized in that: The device comprises: an interpolation module configured to receive, from the rendering module, a latest image rendered for a current virtual scene and associated information of the latest image, the associated information including corresponding depth information and motion information of the latest image relative to a previously rendered image; and generate, based on the latest image, the associated information of the latest image, and the previous image, a target number of interpolation images between the previous image and the latest image; A projection transformation module, configured to perform projection transformation on the latest image and each of the interpolated frame images based on corresponding camera data to obtain a transformed latest image and each of the transformed interpolated frame images; The upper screen control module is used to send each of the transformed interpolated frame images and the transformed latest image to the screen in sequence in response to the refresh rate of the screen, so that the screen displays each of the transformed interpolated frame images and the transformed latest image in sequence after displaying the previous image.

8. An image processing device for virtual photography, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images