Image interpolation methods, devices, and storage media in virtual photography

Abstract

This disclosure relates to an image frame interpolation method, apparatus, and storage medium in virtual shooting. The method includes: acquiring at least two reference images generated by a rendering engine; acquiring motion information and depth information of the at least two reference images generated by the rendering engine; determining the pixel values ​​of an interpolated image between the at least two reference images based on the motion information and depth information of the at least two reference images; the interpolated image and the at least two reference images are sent to a display screen for display, with the display screen serving as the shooting background in virtual shooting. According to embodiments of this disclosure, smooth playback of on-screen content can be achieved without improving the performance of the rendering end, significantly improving the image quality of the final product, fully utilizing the data output capabilities of the rendering engine, avoiding redundant calculations, saving computing power, achieving a significant frame rate increase at a lower cost, meeting the needs of virtual shooting scenarios, and exhibiting good stability and robustness.

Description

Technical Field

[0001] This disclosure relates to the field of virtual photography technology, and in particular to an image frame interpolation method, apparatus and storage medium for virtual photography. Background Technology

[0002] With the rapid development of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies, virtual shooting, as an emerging film and television production method integrating multiple advanced technologies, has been widely used in film, games, advertising, and other fields. By combining real-time rendering engines, motion capture technology, and virtual scene construction, virtual shooting allows for real-time previewing and adjustment of shooting effects in a virtual environment, significantly improving production efficiency and creative freedom.

[0003] Among numerous real-time rendering engines, Unreal Engine (UE) has become one of the most mainstream rendering platforms in virtual shooting due to its powerful real-time rendering capabilities and high customizability. However, as the complexity of assets in virtual shooting scenarios continues to increase (such as multiple light sources), coupled with the objective factor of limited machine performance, the content displayed on screen often faces the problem of insufficient frame rate, resulting in stuttering playback and affecting the shooting experience and the quality of the final product. Therefore, in virtual shooting scenarios, improving the smoothness of the content displayed on screen, thereby directly improving the image quality of the final product and saving hardware resources, and increasing the real-time rendering frame rate has become an important problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, this disclosure proposes an image frame interpolation method, apparatus and storage medium for virtual shooting.

[0005] According to one aspect of this disclosure, an image frame interpolation method for virtual shooting is provided. The method includes:

[0006] Obtain at least two reference frames generated by the rendering engine;

[0007] Obtain motion and depth information from at least two reference frames generated by the rendering engine;

[0008] Based on the motion and depth information of at least two reference images, the pixel values ​​of an interpolated image between the at least two reference images are determined. The interpolated image and the at least two reference images are sent to a display screen for display, which serves as the shooting background in the virtual shooting.

[0009] In one possible implementation, the method further includes:

[0010] Correct abnormal motion information in at least two reference images, including the motion vector of pixels from the reference image to the previous frame of the reference image;

[0011] Abnormal depth information in at least two reference images is corrected. The depth information represents the distance from the object corresponding to the pixel to the virtual camera, which is used to capture the reference images.

[0012] In one possible implementation, the pixel values ​​of the interpolated frame image between at least two reference frames are determined based on motion and depth information from at least two reference frames, including:

[0013] Based on the motion and depth information of at least two reference images, the mapping relationship between the interpolated image and the pixels of the at least two reference images is determined; wherein, the at least two reference images include a first reference image of the frame preceding the interpolated image and a second reference image of the frame following the interpolated image, and the mapping relationship includes a first mapping relationship between the pixels of the interpolated image and the first reference image, and a second mapping relationship between the pixels of the interpolated image and the second reference image.

[0014] The pixel values ​​of the interpolated image are determined based on the first and second mapping relationships.

[0015] In one possible implementation, based on motion and depth information from at least two reference frames, the mapping relationship between the interpolated frame image and the pixels of the at least two reference frames is determined, including:

[0016] Based on the motion information of at least two reference images and a preset motion mode, when it is determined that there are multiple pixels in any reference image that are mapped to a certain pixel in the interpolated image, based on the depth information of the multiple pixels, it is determined to map the pixel of the multiple pixels that is closest to the virtual camera to the corresponding object in the interpolated image.

[0017] In one possible implementation, a preset motion mode is associated with the content of the image, and the preset motion mode includes any one or more of uniform motion, accelerated motion, decelerated motion, and variable motion.

[0018] In one possible implementation, the pixel values ​​of the interpolated image are determined based on the first mapping relationship and the second mapping relationship, including:

[0019] Based on the first mapping relationship, determine the first reference pixel in the first reference image corresponding to any pixel in the interpolated image;

[0020] Based on the second mapping relationship, determine the second reference pixel in the second reference image corresponding to the pixel in the interpolated image;

[0021] The pixel value in the interpolated frame image is calculated based on the first reference pixel value and the second reference pixel value.

[0022] In one possible implementation, determining the pixel values ​​of the interpolated image based on the first and second mapping relationships further includes:

[0023] Determine the regions in the interpolated image that are occluded by the first reference image and the regions that are occluded by the second reference image, wherein the pixels in the regions occluded by the first reference image do not have a first mapping relationship, and the pixels in the regions occluded by the second reference image do not have a second mapping relationship;

[0024] Based on the second mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the first reference image;

[0025] Based on the first mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the second reference image.

[0026] In one possible implementation, determining the pixel values ​​of the interpolated frame image between at least two reference frames based on motion and depth information of at least two reference frames further includes:

[0027] Identify the hole regions in the interpolated image, where the pixels in the hole regions do not have a first mapping relationship and do not have a second mapping relationship;

[0028] Determine the pixel value of the hole region based on at least one pixel value surrounding the hole region.

[0029] According to another aspect of this disclosure, an image frame interpolation device for virtual shooting is provided. The device includes:

[0030] The first acquisition module is used to acquire at least two reference images generated by the rendering engine;

[0031] The second acquisition module is used to acquire motion and depth information of at least two reference images generated by the rendering engine.

[0032] The determination module is used to determine the pixel values ​​of an interpolated image between at least two reference images based on motion and depth information of at least two reference images. The interpolated image and the at least two reference images are sent to a display screen for display, which serves as the shooting background in the virtual shooting.

[0033] In one possible implementation, the device further includes:

[0034] The first correction module is used to correct abnormal motion information in at least two reference images, the motion information including the motion vector of pixels from the reference image to the previous frame of the reference image;

[0035] The second correction module is used to correct abnormal depth information in at least two reference images. The depth information represents the distance from the object corresponding to the pixel to the virtual camera, which is used to capture the reference images.

[0036] In one possible implementation, a module is defined for:

[0037] Based on the motion and depth information of at least two reference images, the mapping relationship between the interpolated image and the pixels of the at least two reference images is determined; wherein, the at least two reference images include a first reference image of the frame preceding the interpolated image and a second reference image of the frame following the interpolated image, and the mapping relationship includes a first mapping relationship between the pixels of the interpolated image and the first reference image, and a second mapping relationship between the pixels of the interpolated image and the second reference image.

[0038] The pixel values ​​of the interpolated image are determined based on the first and second mapping relationships.

[0039] In one possible implementation, based on motion and depth information from at least two reference frames, the mapping relationship between the interpolated frame image and the pixels of the at least two reference frames is determined, including:

[0040] Based on the motion information of at least two reference images and a preset motion mode, when it is determined that there are multiple pixels in any reference image that are mapped to a certain pixel in the interpolated image, based on the depth information of the multiple pixels, it is determined to map the pixel of the multiple pixels that is closest to the virtual camera to the corresponding object in the interpolated image.

[0041] In one possible implementation, a preset motion mode is associated with the content of the image, and the preset motion mode includes any one or more of uniform motion, accelerated motion, decelerated motion, and variable motion.

[0042] In one possible implementation, the pixel values ​​of the interpolated image are determined based on the first mapping relationship and the second mapping relationship, including:

[0043] Based on the first mapping relationship, determine the first reference pixel in the first reference image corresponding to any pixel in the interpolated image;

[0044] Based on the second mapping relationship, determine the second reference pixel in the second reference image corresponding to the pixel in the interpolated image;

[0045] The pixel value in the interpolated frame image is calculated based on the first reference pixel value and the second reference pixel value.

[0046] In one possible implementation, determining the pixel values ​​of the interpolated image based on the first and second mapping relationships further includes:

[0047] Determine the regions in the interpolated image that are occluded by the first reference image and the regions that are occluded by the second reference image, wherein the pixels in the regions occluded by the first reference image do not have a first mapping relationship, and the pixels in the regions occluded by the second reference image do not have a second mapping relationship;

[0048] Based on the second mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the first reference image;

[0049] Based on the first mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the second reference image.

[0050] In one possible implementation, a module is defined for:

[0051] Identify the hole regions in the interpolated image, where the pixels in the hole regions do not have a first mapping relationship and do not have a second mapping relationship;

[0052] Determine the pixel value of the hole region based on at least one pixel value surrounding the hole region.

[0053] According to another aspect of this disclosure, an image interpolation device for virtual shooting is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0054] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0055] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0056] According to embodiments of this disclosure, by acquiring at least two reference images generated by the rendering engine, and the motion and depth information of the at least two reference images generated by the rendering engine, the pixel values ​​of an interpolated image between the at least two reference images are determined based on the motion and depth information of the at least two reference images. The interpolated image and the at least two reference images are sent to a display screen for display, with the display screen serving as the shooting background in virtual photography. This allows for smooth playback of the content displayed without improving the performance of the rendering end, significantly improving the final image quality. Compared to traditional methods that rely on increasing the complexity of the rendering algorithm or upgrading hardware configuration, embodiments of this disclosure do not require calculating the motion and depth information of the reference images from scratch. Instead, they can directly utilize the relevant data generated by the rendering engine to efficiently call its built-in motion and depth information, fully leveraging the data output capabilities of the rendering engine, avoiding redundant calculations, significantly reducing dependence on computing power, and improving the real-time processing capabilities of the system. Furthermore, by performing frame interpolation at the 2D image level, this method is simple, efficient, and easy to deploy, requiring no additional hardware costs and not altering the existing rendering pipeline. It achieves a significant frame rate increase at a relatively low cost, effectively meeting the demands for low latency and high image quality in virtual shooting scenarios, making it suitable for shooting applications with high real-time performance requirements. Simultaneously, the number of interpolated frames can be flexibly adjusted according to actual frame rate needs, adapting to different performance conditions and optimizing display effects based on specific shooting requirements, demonstrating good stability and robustness.

[0057] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0058] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0059] Figure 1 A schematic diagram illustrating an application scenario according to an embodiment of this disclosure is shown.

[0060] Figure 2 A flowchart illustrating an image frame interpolation method in virtual shooting according to an embodiment of the present disclosure is shown.

[0061] Figure 3 A schematic diagram illustrating image frame interpolation according to an embodiment of the present disclosure is shown.

[0062] Figure 4 A schematic diagram of the occlusion area according to an embodiment of the present disclosure is shown.

[0063] Figure 5 A schematic flowchart of an image interpolation method in virtual shooting according to an embodiment of the present disclosure is shown.

[0064] Figure 6 A structural diagram of an image interpolation device in virtual shooting according to an embodiment of the present disclosure is shown.

[0065] Figure 7 This is a block diagram illustrating an apparatus 1900 for image interpolation in virtual photography according to an exemplary embodiment. Detailed Implementation

[0066] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0067] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0068] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0069] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0070] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0071] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0072] With the rapid development of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies, virtual shooting, as an emerging film and television production method integrating multiple advanced technologies, has been widely used in film, games, advertising, and other fields. By combining real-time rendering engines, motion capture technology, and virtual scene construction, virtual shooting allows for real-time previewing and adjustment of shooting effects in a virtual environment, significantly improving production efficiency and creative freedom.

[0073] Among numerous real-time rendering engines, Unreal Engine (UE) has become one of the most mainstream rendering platforms in virtual shooting due to its powerful real-time rendering capabilities and high customizability. However, as the complexity of assets in virtual shooting scenarios continues to increase (such as multiple light sources), coupled with the objective factor of limited machine performance, the content displayed on screen often faces the problem of insufficient frame rate, resulting in stuttering playback and affecting the shooting experience and the quality of the final product. Therefore, in virtual shooting scenarios, improving the smoothness of the content displayed on screen, thereby directly improving the image quality of the final product and saving hardware resources, and increasing the real-time rendering frame rate has become an important problem that urgently needs to be solved.

[0074] In view of this, embodiments of this disclosure provide an image frame interpolation method, apparatus, and storage medium for virtual shooting. The method of this disclosure acquires at least two reference images generated by a rendering engine, along with motion and depth information of these reference images. Based on the motion and depth information of the at least two reference images, it determines the pixel values ​​of an interpolated image between the at least two reference images. The interpolated image and the at least two reference images are sent to a display screen for display, which serves as the shooting background in virtual shooting. This enables smooth playback of the displayed content without improving the performance of the rendering engine, significantly improving the final image quality. Compared to traditional methods that rely on increasing the complexity of rendering algorithms or upgrading hardware configurations, embodiments of this disclosure do not require calculating the motion and depth information of the reference images from scratch. Instead, they can directly utilize the relevant data generated by the rendering engine to efficiently call its built-in motion and depth information, fully leveraging the data output capabilities of the rendering engine, avoiding redundant calculations, significantly reducing reliance on computing power, and improving the real-time processing capabilities of the system. Furthermore, by performing frame interpolation at the 2D image level, this method is simple, efficient, and easy to deploy, requiring no additional hardware costs and not altering the existing rendering pipeline. It achieves a significant frame rate increase at a relatively low cost, effectively meeting the demands for low latency and high image quality in virtual shooting scenarios, making it suitable for shooting applications with high real-time performance requirements. Simultaneously, the number of interpolated frames can be flexibly adjusted according to actual frame rate needs, adapting to different performance conditions and optimizing display effects based on specific shooting requirements, demonstrating good stability and robustness.

[0075] Figure 1 The diagram illustrates an application scenario according to an embodiment of the present disclosure. The image interpolation system in virtual shooting according to this disclosure can be used in virtual shooting scenarios, such as... Figure 1 As shown, the image interpolation system can acquire reference images (reference frame 1 and reference frame 2 in the figure) and their related motion and depth information generated by the rendering engine, so as to perform interpolation processing on at least two reference images to obtain one or more interpolated images between at least two reference images (interpolated frame 1 and interpolated frame 2 in the figure).

[0076] The image interpolation system in virtual filming can send both a reference image and the interpolated image to a display device for display. This display device can be a light-emitting diode (LED) screen or other type of display terminal used in the virtual filming scene to present the virtual content to the filming system in real time. Subsequently, the actual camera in the filming system captures images of the screen and the real-world scene in front of it, generating the final video content.

[0077] The image frame interpolation system in the virtual shooting of this disclosure can be used in a terminal device or a server. The terminal device can be any one or more of the following: mobile phone, foldable electronic device, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), and in-vehicle device. This disclosure does not impose any special limitation on the specific type of terminal device, which can have wired or wireless communication capabilities.

[0078] Servers can be located locally or in the cloud, and can be physical devices or virtual devices such as virtual machines and containers. They possess wireless communication capabilities, which can be configured within the server's chip (system) or other components. Wireless communication capabilities can be implemented through mobile communication technologies such as 2G / 3G / 4G / 5G, as well as Wi-Fi, Bluetooth, frequency modulation (FM), data radio, and satellite communication. Wired connections can also be used to communicate and interact with other devices.

[0079] It should be noted that the image interpolation system in the virtual shooting of this disclosure can also be applied to other scenarios that may require image interpolation, and this disclosure does not limit this.

[0080] Figure 2 A flowchart illustrating an image interpolation method in virtual shooting according to an embodiment of the present disclosure is provided. This method can be used in image interpolation systems, such as... Figure 2 As shown, the method may include:

[0081] Step S201: Obtain at least two reference images generated by the rendering engine.

[0082] The reference image can be an image rendered by a rendering engine (such as UE). For example, in a virtual shooting scenario, a reference image can be captured by a virtual camera. The virtual camera can be an object set in the rendering engine to simulate the behavior of a real camera, and can have similar parameter configurations to a real camera.

[0083] The process of capturing data using a virtual camera refers to the process of rendering 3D elements such as models, materials, and lighting effects in a 3D virtual scene by controlling the camera's viewpoint and movement trajectory through a rendering engine. This 3D scene is then projected into 2D image frames, and the rendered image data can serve as a reference image. This reference image can be directly output to an LED screen or other display device in subsequent virtual shooting, serving as the final shooting material for the actual camera.

[0084] To ensure the effectiveness of subsequent frame interpolation and improve the display frame rate, at least two reference images are required. These can include one or more frames before and one or more frames after the interpolated image. The interpolated image is the result of the interpolation, i.e., the image to be obtained through interpolation. The interpolated image can be one or more frames, and its number can be determined according to the needs of virtual shooting. The number of interpolated frames can be freely selected based on the actual frame rate to achieve different on-screen frame rates, offering high flexibility.

[0085] Step S202: Obtain motion and depth information of at least two reference images generated by the rendering engine.

[0086] Motion and depth information can be obtained using the rendering engine's own features. Taking UE as the rendering engine as an example, motion information can be obtained by using motion vectors generated by UE's rendering pipeline, which can be achieved by enabling UE's motion vector rendering function. Depth information can be obtained by using depth values ​​generated by UE's rendering pipeline, which are generally automatically generated during UE's normal workflow.

[0087] For any pixel in a reference frame, the motion information generated by the rendering engine can include a motion vector of the pixel from the reference image to the previous frame of the reference image. This motion vector can be the displacement of the pixel from its position in the reference image to the previous frame of the reference image. For example, for a two-dimensional image, this displacement can include displacement along the horizontal and vertical axes.

[0088] In this embodiment of the disclosure, motion information can also be supplemented by a preset method (such as inverting existing motion vectors) so that the motion information includes the motion vector of the pixel from the previous frame of the reference image to the reference image, and the motion information includes dual-flow information (i.e., bidirectional optical flow information), making it more complete.

[0089] For any pixel in a reference frame, the depth information generated by the rendering engine can represent the distance (i.e., depth value) from the object corresponding to that pixel to the virtual camera. The object can be a point on the surface of the object corresponding to that pixel in a 3D virtual scene. The virtual camera can be used to capture the reference image.

[0090] When using a virtual camera to capture a reference image, if the object being captured is a semi-transparent object (which may simultaneously present foreground and background objects, causing confusion when the virtual camera calculates pixel position and depth) or an object of other special materials, there may be abnormalities in motion and depth information. In this case, motion and depth analysis can be performed on the reference image to correct the abnormal motion and depth information, thereby improving the accuracy of subsequent frame interpolation and image quality.

[0091] In step S202, the following can be done:

[0092] Correct any abnormal motion information in at least two reference frames.

[0093] Abnormal motion information can refer to any pixel in the reference image lacking corresponding motion information or having inaccurate motion information. For example, for any pixel corresponding to a preset object in the reference image, the known size, shape, and motion direction in consecutive frames of the object can be used to determine whether the motion information of the pixel conforms to physical laws. If there is a sudden change, discontinuity, or inconsistency with the overall motion direction of the object in the pixel corresponding to the object, then the motion information of the pixel can be considered inaccurate.

[0094] Motion information correction can be implemented based on existing technologies. It can be achieved by analyzing the temporal changes of the pixel, its similarity to surrounding pixels, and the motion information of those surrounding pixels. For example, the motion trend of the pixel can be determined by comparing the pixel values ​​in a reference image and the previous frame of the reference image (e.g., determining whether the pixel has moved; if not, the motion vector is 0). Alternatively, the similarity between the pixel values ​​of the pixel and its surrounding neighbors can be calculated. Based on the motion information of pixels whose similarity reaches a preset threshold (e.g., through weighted fusion of motion vectors, median averaging, etc.), the corrected motion information of the pixel can be determined.

[0095] In step S202, the following may also be done:

[0096] Correct any abnormal depth information in at least two reference images.

[0097] Abnormal depth information can refer to any pixel in the reference image lacking corresponding depth information or having inaccurate depth information. For example, for any pixel corresponding to a preset object in the reference image, the known geometric structure, size, and other information of the object can be used to make a judgment. If the depth value of the pixel has a sudden change compared with the depth value of the adjacent pixels on the object, or does not match the geometric structure of the object, the depth information of the pixel can be considered inaccurate.

[0098] Depth information correction can be implemented based on existing technologies. It can be achieved by analyzing the temporal changes of the pixel, its similarity to surrounding pixels, and the depth information of those surrounding pixels. For example, the motion trend of the pixel can be determined by comparing the pixel values ​​in a reference image and the previous frame of the reference image (e.g., determining whether the pixel has moved; if not, its depth information is consistent with the previous frame). Alternatively, the similarity between the pixel values ​​of the pixel and its surrounding neighbors can be calculated. The corrected depth information of the pixel can then be determined by calculating the depth information of pixels whose similarity reaches a preset threshold (e.g., through weighted fusion or median averaging of depth values).

[0099] Step S203: Based on the motion information and depth information of at least two reference images, determine the pixel values ​​of the interpolated frame image between at least two reference images.

[0100] Frame interpolation can be performed based on motion and depth information from at least two reference images to generate one or more interpolated images between adjacent reference images. The interpolated images and the at least two reference images can be sent to a display screen for display. For example, in a virtual shooting scenario, the interpolated images and the at least two reference images can be arranged in chronological order to form a continuous image sequence, which can then be sent to a display screen for playback. The display screen can serve as the shooting background in the virtual shooting, allowing the actual camera to capture images of the display screen and the real scene in front of it.

[0101] According to embodiments of this disclosure, by acquiring at least two reference images generated by the rendering engine, and the motion and depth information of the at least two reference images generated by the rendering engine, the pixel values ​​of an interpolated image between the at least two reference images are determined based on the motion and depth information of the at least two reference images. The interpolated image and the at least two reference images are sent to a display screen for display, with the display screen serving as the shooting background in virtual photography. This allows for smooth playback of the content displayed without improving the performance of the rendering end, significantly improving the final image quality. Compared to traditional methods that rely on increasing the complexity of the rendering algorithm or upgrading hardware configuration, embodiments of this disclosure do not require calculating the motion and depth information of the reference images from scratch. Instead, they can directly utilize the relevant data generated by the rendering engine to efficiently call its built-in motion and depth information, fully leveraging the data output capabilities of the rendering engine, avoiding redundant calculations, significantly reducing dependence on computing power, and improving the real-time processing capabilities of the system. Furthermore, by performing frame interpolation at the 2D image level, this method is simple, efficient, and easy to deploy, requiring no additional hardware costs and not altering the existing rendering pipeline. It achieves a significant frame rate increase at a relatively low cost, effectively meeting the demands for low latency and high image quality in virtual shooting scenarios, making it suitable for shooting applications with high real-time performance requirements. Simultaneously, the number of interpolated frames can be flexibly adjusted according to actual frame rate needs, adapting to different performance conditions and optimizing display effects based on specific shooting requirements, demonstrating good stability and robustness.

[0102] During frame interpolation, the pixel correspondence between the interpolated image and the reference image can be established first. In step S203, the following can be done:

[0103] Based on the motion and depth information of at least two reference images, determine the mapping relationship between the interpolated image and the pixels of at least two reference images.

[0104] The at least two reference images may include a first reference image of the frame preceding the interpolated image and a second reference image of the frame following the interpolated image. The mapping relationship may include a first mapping relationship between the pixels of the interpolated image and the first reference image, and a second mapping relationship between the pixels of the interpolated image and the second reference image.

[0105] Based on the motion information of pixels in the first and second reference images and a preset motion mode, the temporal trajectory of a pixel can be determined. The corresponding pixel in the interpolated image can then be determined based on the motion trajectory, thus establishing a mapping relationship between pixels. For example, taking the first reference image as an example, for any pixel in the first reference image, its position at one or more future moments (i.e., the time point corresponding to the interpolated image; for multiple interpolated images, this can be multiple time points) can be calculated based on its motion vector and preset motion mode. This position can point to a pixel in the interpolated image, thereby establishing a first mapping relationship between the pixel in the first reference image and the corresponding pixel in the interpolated image. The preset motion mode can be associated with the image content and can include any one or more of uniform motion, accelerated motion, decelerated motion, and variable motion (such as acceleration followed by deceleration, deceleration followed by acceleration, etc.). For example, regions in the image associated with various types of objects can be identified based on feature detection and other related technologies. Different types of objects can correspond to different preset motion modes. For regions in the image representing stable moving objects (such as normally moving vehicles), the motion mode of the corresponding pixels in that region can be set to uniform motion.

[0106] Similarly, the pixels in the second reference image can be reversed to obtain the position of the pixel at the corresponding time point in the interpolated image, thereby establishing a second mapping relationship between the pixels in the second reference image and the corresponding pixels in the interpolated image.

[0107] To ensure that the pixels in both the first and second mapping relationships are in a one-to-one correspondence, thereby improving the accuracy of determining the pixel values ​​of the interpolated image, in the process of determining the mapping relationship between the pixels of the interpolated image and the pixels of at least two reference images based on the motion and depth information of at least two reference images, the following can be done:

[0108] Based on the motion information of at least two reference images and a preset motion mode, when it is determined that there are multiple pixels in any reference image that are mapped to a certain pixel in the interpolated image, based on the depth information of the multiple pixels, it is determined to map the pixel of the multiple pixels that is closest to the virtual camera to the corresponding object in the interpolated image.

[0109] Figure 3 A schematic diagram illustrating image frame interpolation according to an embodiment of the present disclosure is shown. Figure 3As shown, taking the process of determining the first mapping relationship as an example, if it is determined that pixels A1, A2, and A3 in the first reference image have a mapping relationship with pixel B1 in the interpolated image, the pixel closest to the virtual camera (e.g., A2) can be selected from pixels A1, A2, and A3 to establish the first mapping relationship between pixel A2 in the first reference image and pixel B1 in the interpolated image. The determination of the second mapping relationship is similar.

[0110] After obtaining the first mapping relationship and the second mapping relationship, for pixels in the interpolated image that have the first mapping relationship and / or the second mapping relationship, the pixel value of the interpolated image can be determined based on the first mapping relationship and the second mapping relationship.

[0111] Specifically, for pixels in the interpolated image that have a first mapping relationship and a second mapping relationship, in the process of determining the pixel values ​​of the interpolated image based on the first mapping relationship and the second mapping relationship, the following can be done:

[0112] Based on the first mapping relationship, determine the first reference pixel in the first reference image corresponding to any pixel in the interpolated image;

[0113] Based on the second mapping relationship, determine the second reference pixel in the second reference image corresponding to the pixel in the interpolated image;

[0114] The pixel value in the interpolated frame image is calculated based on the first reference pixel value and the second reference pixel value.

[0115] The interpolation calculation can be performed using linear interpolation or nonlinear interpolation, and this disclosure does not limit the method. Taking linear interpolation as an example, for any pixel in the interpolated frame image, the interpolation weights corresponding to the first and second reference images can be determined based on the positional relationship between the time point corresponding to the interpolated frame image and the first and second reference images. If the time point corresponding to the interpolated frame image is closer to the first reference image, the interpolation weight corresponding to the first reference image is higher; conversely, the interpolation weight corresponding to the second reference image is higher. Based on the interpolation weights corresponding to the first and second reference images, the first reference pixel value and the second reference pixel value are weighted and calculated respectively to obtain the pixel value in the interpolated frame image.

[0116] Because multiple pixels in the reference image may point to the same pixel in the interpolated image when determining the first and second mapping relationships, some pixels in the interpolated image may not correspond to pixels in the reference image, or the objects to which some pixels in the reference image belong may have moved out of the viewpoint at the time corresponding to the interpolated image (i.e., no longer correspond to any pixel in the interpolated image). This could result in some pixels in the interpolated image lacking the first and / or second mapping relationships. For pixels in the interpolated image that only have the first mapping relationship or only have the second mapping relationship, in the process of determining the pixel values ​​of the interpolated image based on the first and second mapping relationships, the following can also be considered:

[0117] Determine the regions in the interpolated image that are occluded by the first reference image and the regions that are occluded by the second reference image;

[0118] Based on the second mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the first reference image;

[0119] Based on the first mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the second reference image.

[0120] In this context, the region occluded by the first reference image refers to an area in the interpolated image where, due to factors such as changes in the viewing angle of the first reference image, object movement, or occlusion, some pixels cannot find a corresponding mapping relationship under the first mapping relationship. Pixels in the interpolated image that do not have a first mapping relationship but have a second mapping relationship can be considered as regions occluded by the first reference image. Similarly, the region occluded by the second reference image refers to an area where a corresponding relationship cannot be established under the second mapping relationship. Pixels in the interpolated image that do not have a second mapping relationship but have a first mapping relationship can be considered as regions occluded by the second reference image. Pixels in the region occluded by the first reference image do not have a first mapping relationship, and pixels in the region occluded by the second reference image do not have a second mapping relationship. For regions occluded by the first reference image, the corresponding pixel value can be determined based on the second mapping relationship. For example, the pixel value of the second reference image corresponding to the second mapping relationship can be used as the pixel value of the corresponding pixel in the region occluded by the first reference image. Similarly, for regions occluded by the second reference image, the corresponding pixel value can be determined based on the first mapping relationship. For example, the pixel value of the first reference image corresponding to the first mapping relationship can be used as the pixel value of the corresponding pixel in the region occluded by the second reference image.

[0121] Figure 4 A schematic diagram of the obstructed area according to an embodiment of the present disclosure is shown. Figure 4As shown, taking pixel B2 in the region occluded by the first reference image in the interpolated image as an example, pixel B2 only has a second mapping relationship with pixel C1 in the second reference image, and no first mapping relationship with any pixel in the first reference image. In this case, the pixel value of pixel C1 can be used as the pixel value of pixel B2 according to the second mapping relationship. The same applies to pixels in the region occluded by the second reference image in the interpolated image.

[0122] For pixels in the interpolated image that lack both a first and a second mapping relationship, this is usually due to significant displacement, severe occlusion, or the appearance or disappearance of objects between the reference images, causing the pixel to be unable to find a valid correspondence in any reference image, thus forming a hole region. In step S203, the following can also be done:

[0123] Identify the hole regions in the interpolated image;

[0124] Determine the pixel value of the hole region based on at least one pixel value surrounding the hole region.

[0125] In this embodiment, pixels in the interpolated image that do not have a first mapping relationship and do not have a second mapping relationship can be designated as hole regions. For pixels in hole regions, at least one neighboring pixel whose depth value or color value differs from the pixel's depth value (or RGB color value) by less than a preset threshold can be found. Based on the determined neighboring pixel values, the pixel values ​​of each pixel in the hole region are determined through filling and repair methods such as interpolation filling, weighted averaging, and texture expansion. This embodiment does not limit the filling and repair method and can be implemented based on existing technologies.

[0126] For pixels in hole regions, in addition to filling and repairing them using other known pixel values ​​from the interpolated image, information from multiple historical frames can also be used to determine the pixel values. For example, the first reference image can be replaced with the Nth frame before the interpolated image (N is a positive integer greater than 1, such as 2), and the second reference image can be replaced with the Nth frame after the interpolated image. In this way, the first and second mapping relationships of the hole regions can be recalculated based on the new reference images, thereby determining the pixel values ​​of the hole regions more accurately.

[0127] Therefore, it can effectively repair missing parts in the image caused by occlusion or motion changes, and ensure the continuity and visual quality of the image after frame interpolation.

[0128] Figure 5 A schematic flowchart illustrating an image frame interpolation method in virtual shooting according to an embodiment of the present disclosure is shown. Figure 5As shown, in this embodiment of the present disclosure, after obtaining at least two reference images (reference frame 1 and reference frame 2 in the figure) and their corresponding motion and depth information generated by the rendering engine, motion analysis and depth analysis can be performed on the at least two reference images to correct the motion and depth information of the obtained at least two reference images (see the method of step S202 above); then, based on the motion and depth information of the at least two reference images, the mapping relationship between the reference images and the interpolated frame images can be determined (as shown in the "motion mapping" part in the figure, see the relevant method in step S203 above); based on the mapping relationship between the reference images and the interpolated frame images, the pixel values ​​in the interpolated frame images can be interpolated (corresponding to the "frame interpolation" part in the figure), and the pixel values ​​of the areas occluded by the reference images in the interpolated frame images can be completed (corresponding to "occluded area sampling" in the figure, see the relevant method in step S203 above) and the pixel values ​​of the hole areas in the interpolated frame images (corresponding to "hole area repair" in the figure, see the relevant method in step S203 above), thereby obtaining a complete interpolated frame image. For Figure 5 Any part of it can also be implemented using other methods besides those mentioned above. For example, a deep convolutional neural network can be used to achieve... Figure 5 The implementation effects of any part of the embodiments are not limited in this disclosure.

[0129] Figure 6 A structural diagram of an image interpolation device in virtual shooting according to an embodiment of the present disclosure is shown. Figure 6 As shown, the device may include:

[0130] The first acquisition module 601 is used to acquire at least two reference images generated by the rendering engine;

[0131] The second acquisition module 602 is used to acquire motion information and depth information of at least two reference images generated by the rendering engine.

[0132] The determining module 603 is used to determine the pixel values ​​of an interpolated image between at least two reference images based on motion information and depth information of at least two reference images. The interpolated image and the at least two reference images are used to send to the display screen for display, and the display screen serves as the shooting background in virtual shooting.

[0133] In one possible implementation, the device further includes:

[0134] The first correction module is used to correct abnormal motion information in at least two reference images, the motion information including the motion vector of pixels from the reference image to the previous frame of the reference image;

[0135] The second correction module is used to correct abnormal depth information in at least two reference images. The depth information represents the distance from the object corresponding to the pixel to the virtual camera, which is used to capture the reference images.

[0136] In one possible implementation, module 603 is defined for:

[0137] Based on the motion and depth information of at least two reference images, the mapping relationship between the interpolated image and the pixels of the at least two reference images is determined; wherein, the at least two reference images include a first reference image of the frame preceding the interpolated image and a second reference image of the frame following the interpolated image, and the mapping relationship includes a first mapping relationship between the pixels of the interpolated image and the first reference image, and a second mapping relationship between the pixels of the interpolated image and the second reference image.

[0138] The pixel values ​​of the interpolated image are determined based on the first and second mapping relationships.

[0139] In one possible implementation, based on motion and depth information from at least two reference frames, the mapping relationship between the interpolated frame image and the pixels of the at least two reference frames is determined, including:

[0140] Based on the motion information of at least two reference images and a preset motion mode, when it is determined that there are multiple pixels in any reference image that are mapped to a certain pixel in the interpolated image, based on the depth information of the multiple pixels, it is determined to map the pixel of the multiple pixels that is closest to the virtual camera to the corresponding object in the interpolated image.

[0141] In one possible implementation, a preset motion mode is associated with the content of the image, and the preset motion mode includes any one or more of uniform motion, accelerated motion, decelerated motion, and variable motion.

[0142] In one possible implementation, the pixel values ​​of the interpolated image are determined based on the first mapping relationship and the second mapping relationship, including:

[0143] Based on the first mapping relationship, determine the first reference pixel in the first reference image corresponding to any pixel in the interpolated image;

[0144] Based on the second mapping relationship, determine the second reference pixel in the second reference image corresponding to the pixel in the interpolated image;

[0145] The pixel value in the interpolated frame image is calculated based on the first reference pixel value and the second reference pixel value.

[0146] In one possible implementation, determining the pixel values ​​of the interpolated image based on the first and second mapping relationships further includes:

[0147] Determine the regions in the interpolated image that are occluded by the first reference image and the regions that are occluded by the second reference image, wherein the pixels in the regions occluded by the first reference image do not have a first mapping relationship, and the pixels in the regions occluded by the second reference image do not have a second mapping relationship;

[0148] Based on the second mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the first reference image;

[0149] Based on the first mapping relationship, determine the pixel values ​​of the region in the interpolated image that is occluded by the second reference image.

[0150] In one possible implementation, module 603 is defined for:

[0151] Identify the hole regions in the interpolated image, where the pixels in the hole regions do not have a first mapping relationship and do not have a second mapping relationship;

[0152] Determine the pixel value of the hole region based on at least one pixel value surrounding the hole region.

[0153] According to embodiments of this disclosure, by acquiring at least two reference images generated by the rendering engine, and the motion and depth information of the at least two reference images generated by the rendering engine, the pixel values ​​of an interpolated image between the at least two reference images are determined based on the motion and depth information of the at least two reference images. The interpolated image and the at least two reference images are sent to a display screen for display, with the display screen serving as the shooting background in virtual photography. This allows for smooth playback of the content displayed without improving the performance of the rendering end, significantly improving the final image quality. Compared to traditional methods that rely on increasing the complexity of the rendering algorithm or upgrading hardware configuration, embodiments of this disclosure do not require calculating the motion and depth information of the reference images from scratch. Instead, they can directly utilize the relevant data generated by the rendering engine to efficiently call its built-in motion and depth information, fully leveraging the data output capabilities of the rendering engine, avoiding redundant calculations, significantly reducing dependence on computing power, and improving the real-time processing capabilities of the system. Furthermore, by performing frame interpolation at the 2D image level, this method is simple, efficient, and easy to deploy, requiring no additional hardware costs and not altering the existing rendering pipeline. It achieves a significant frame rate increase at a relatively low cost, effectively meeting the demands for low latency and high image quality in virtual shooting scenarios, making it suitable for shooting applications with high real-time performance requirements. Simultaneously, the number of interpolated frames can be flexibly adjusted according to actual frame rate needs, adapting to different performance conditions and optimizing display effects based on specific shooting requirements, demonstrating good stability and robustness.

[0154] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0155] This disclosure also provides an image frame interpolation device for virtual shooting, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0156] This disclosure also provides a non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0157] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0158] Figure 7 This is a block diagram illustrating an apparatus 1900 for image frame interpolation in virtual photography, according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 7 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0159] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0160] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0161] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0162] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0163] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0164] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0165] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0166] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0167] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0168] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method of image interpolation in virtual photography, characterized in that, The method comprises: acquiring at least two frames of reference images generated by a rendering engine; acquiring motion information and depth information of the at least two frames of reference images generated by the rendering engine, the motion information comprising motion vectors generated by a rendering pipeline of the rendering engine, and the depth information comprising depth values generated by the rendering pipeline of the rendering engine; determining pixel values of an interpolated image between the at least two frames of reference images based on the motion information and the depth information of the at least two frames of reference images, the interpolated image and the at least two frames of reference images being used for sending to a display screen for display, the display screen serving as a shooting background in virtual shooting; The method further comprises: correcting abnormal motion information in the at least two frames of reference images, the motion information comprising motion vectors of pixels from a reference image to a previous frame of a reference image, the abnormal motion information indicating that any pixel in the reference image has no corresponding motion information or inaccurate motion information; correcting abnormal depth information in the at least two frames of reference images, the depth information indicating distances of objects corresponding to pixels to a virtual camera, the abnormal depth information indicating that any pixel in the reference image has no corresponding depth information or inaccurate depth information, the virtual camera being used for shooting to obtain the reference image.

2. The method of claim 1, wherein, The determination of the pixel values of the interpolated image between the at least two frames of reference images based on the motion information and the depth information of the at least two frames of reference images comprises: determining mapping relationships between pixels of the interpolated image and the at least two frames of reference images based on the motion information and the depth information of the at least two frames of reference images; wherein the at least two frames of reference images comprise a first reference image of a previous frame and a second reference image of a next frame of the interpolated image, the mapping relationships comprise a first mapping relationship between the pixels of the interpolated image and the first reference image, and a second mapping relationship between the pixels of the interpolated image and the second reference image; determining the pixel values of the interpolated image based on the first mapping relationship and the second mapping relationship.

3. The method of claim 2, wherein, The determination of the mapping relationships between the pixels of the interpolated image and the at least two frames of reference images based on the motion information and the depth information of the at least two frames of reference images comprises: when it is determined that a plurality of pixels in any frame of reference image are mapped to a certain pixel in the interpolated image based on the motion information of the at least two frames of reference images and a preset motion mode, determining that the pixel in the interpolated image is mapped to the pixel corresponding to an object closest to a virtual camera in the plurality of pixels based on the depth information of the plurality of pixels.

4. The method of claim 3, wherein, The preset motion mode is associated with the content of the image, and the preset motion mode comprises any one or more of uniform motion, accelerated motion, decelerated motion, and variable speed motion.

5. The method of claim 2, wherein, The determination of the pixel values of the interpolated image based on the first mapping relationship and the second mapping relationship comprises: determining a first reference pixel corresponding to any pixel in the interpolated image in the first reference image based on the first mapping relationship; determine, based on the second mapping relationship, a second reference pixel corresponding to the pixel in the second reference image in the inserted frame image; determine, based on the first reference pixel value and the second reference pixel value, the pixel value in the inserted frame image by interpolation.

6. The method of claim 2, wherein, The method further includes: determining a region in the inserted frame image that is occluded by the first reference image and a region in the inserted frame image that is occluded by the second reference image, wherein the first mapping relationship does not exist for a pixel in the region occluded by the first reference image, and the second mapping relationship does not exist for a pixel in the region occluded by the second reference image; determining, based on the second mapping relationship, a pixel value of the region in the inserted frame image that is occluded by the first reference image; determining, based on the first mapping relationship, a pixel value of the region in the inserted frame image that is occluded by the second reference image.

7. The method of claim 2, wherein, The method further includes: determining a hole region in the inserted frame image, wherein the first mapping relationship does not exist for a pixel in the hole region, and the second mapping relationship does not exist for the pixel in the hole region; determining, based on at least one pixel value around the hole region, a pixel value of the hole region.

8. An apparatus for interpolating images in virtual photography, characterized by The apparatus includes: a first obtaining module configured to obtain at least two reference images generated by a rendering engine; a second obtaining module configured to obtain motion information and depth information of the at least two reference images generated by the rendering engine, wherein the motion information includes motion vectors generated by a rendering pipeline of the rendering engine, and the depth information includes depth values generated by the rendering pipeline of the rendering engine; a determining module configured to determine, based on the motion information and the depth information of the at least two reference images, a pixel value of an inserted frame image between the at least two reference images, wherein the inserted frame image and the at least two reference images are used for sending to a display screen for display, and the display screen is used as a shooting background in virtual shooting. The apparatus further includes: a first correcting module configured to correct abnormal motion information in the at least two reference images, wherein the motion information includes a motion vector of a pixel from a reference image to a previous frame of the reference image, and the abnormal motion information indicates that any pixel in the reference image does not have corresponding motion information or the motion information is inaccurate; a second correcting module configured to correct abnormal depth information in the at least two reference images, wherein the depth information indicates a distance from an object corresponding to a pixel to a virtual camera, the abnormal depth information indicates that any pixel in the reference image does not have corresponding depth information or the depth information is inaccurate, and the virtual camera is used for shooting to obtain the reference image.

9. An apparatus for virtual image interpolation in a video camera, comprising a memory, a processor, and a computer program stored on the memory, wherein, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

11. A computer program product comprising a computer program or a non-transitory computer-readable storage medium bearing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images