Image rendering method and device

Abstract

The present application discloses an image rendering method and device thereof, belonging to the field of image technology. Based on a first reconstructed scene corresponding to N first images, a depth image of the N first images is obtained, where N is an integer greater than 1; based on the depth image, an equidistant surface where a rendering trajectory is located is determined from the first reconstructed scene; based on the shooting center position corresponding to the N first images and the equidistant surface, a first rendering trajectory is determined; based on the first rendering trajectory, the first reconstructed scene is rendered.

Description

Technical Field

[0001] The present application belongs to the field of image technology, and specifically relates to an image rendering method and a device thereof. Background Art

[0002] With the continuous development of 3D reconstruction technology, users have increasingly higher requirements for the quality of rendered images or rendered videos output based on 3D reconstruction technology.

[0003] In related technologies, electronic devices can use neural radiation field technology or three-dimensional Gaussian sputtering rendering technology to train scenes from captured images, such as the shape, color, and transparency of the scene; then, the electronic device can generate a rendered video based on the reconstructed scene and rendering trajectory. Among them, the electronic device can find a rendering trajectory close to the user's shooting trajectory by interpolating the trajectory of the training image that generates the radiation field.

[0004] However, in the above method, since the rendering trajectory is obtained based on the user's shooting trajectory, it cannot guarantee that the object will appear in the rendered image in a stable and proportional manner. For example, the size of the object in the rendered image may fluctuate, resulting in the inability to guarantee the quality of the rendered video. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image rendering method and device thereof, which can ensure the quality of rendered video.

[0006] In a first aspect, an embodiment of the present application provides an image rendering method, the method comprising: based on a first reconstructed scene corresponding to N first images, obtaining a depth image of the N first images, N being an integer greater than 1; based on the depth image, determining an equidistant surface where a rendering trajectory is located from the first reconstructed scene; based on the shooting center positions corresponding to the N first images and the equidistant surface, determining a first rendering trajectory; and based on the first rendering trajectory, rendering the first reconstructed scene.

[0007] In a second aspect, an embodiment of the present application provides an image rendering device, which includes: an acquisition module, a determination module and a processing module. The acquisition module is used to acquire a depth image of the N first images based on a first reconstructed scene corresponding to the N first images, where N is an integer greater than 1. The determination module is used to determine an equidistant surface where a rendering trajectory is located from the first reconstructed scene based on the depth image. The determination module is also used to determine a first rendering trajectory based on the shooting center positions corresponding to the N first images and the equidistant surface. The processing module is used to render the first reconstructed scene based on the first rendering trajectory.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the electronic device can obtain N depth images corresponding to the N images based on the first reconstructed scene corresponding to the N first images, and then the electronic device can determine the equidistant surface where the rendering trajectory is located from the first reconstructed scene based on the N depth images, and determine the first rendering trajectory according to the equidistant surface and the shooting center position corresponding to the N first images, and then the electronic device can use the first rendering trajectory to render the first reconstructed scene. Through this scheme, on the one hand, since the first rendering trajectory is determined based on the equidistant surface and the shooting center position corresponding to the N first images, and there is a distance between the equidistant surface and the surface of the object in the first reconstructed scene, the equidistant surface will not intersect with the object, so the phenomenon of the first rendering trajectory passing through the object can be avoided; on the other hand, since the above-mentioned equidistant surface is obtained by combining the depth information of the depth image, the obtained equidistant surface is continuous and smooth, that is, the first rendering trajectory moves continuously and smoothly on the equidistant surface, so that the electronic device uses the first rendering trajectory determined by the equidistant surface for rendering, which can ensure that the object appears in a stable and proportional manner on the rendered image, and avoid the phenomenon that the size of the object on the rendered image appears large and small. In this way, the quality of the video rendered by the electronic device can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flowcharts of an image rendering method provided in an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of an example of a camera posture for shooting provided in an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of an example of a depth image provided in an embodiment of the present application;

[0016] Figure 4 This is one of the schematic diagrams of a two-dimensional projection example of an equidistant surface provided in an embodiment of the present application;

[0017] Figure 5 This is a second schematic diagram of a two-dimensional projection example of an equidistant surface provided in an embodiment of the present application;

[0018] Figure 6 This is a second flow chart of an image rendering method provided in an embodiment of the present application;

[0019] Figure 7 is a schematic diagram of an example of a spatial grid provided in an embodiment of the present application;

[0020] Figure 8 This is a third flowchart of an image rendering method provided in an embodiment of the present application;

[0021] Fig. 9 It is one of the schematic diagrams of a spatial distance field provided in an embodiment of the present application;

[0022] Fig.10 This is a fourth flowchart of an image rendering method provided in an embodiment of the present application;

[0023] Fig.11 This is a fifth flowchart of an image rendering method provided in an embodiment of the present application;

[0024] Fig. 12A This is a second schematic diagram of a spatial distance field provided in an embodiment of the present application;

[0025] Fig. 12B This is a third schematic diagram of a spatial distance field provided in an embodiment of the present application;

[0026] Fig.13 is a schematic diagram of an equidistant surface provided in an embodiment of the present application;

[0027] Fig.14 This is a sixth flowchart of an image rendering method provided in an embodiment of the present application;

[0028] Fig.15A This is one of the schematic diagrams of a method for determining a rendering trajectory provided in an embodiment of the present application;

[0029] Fig. 15B This is a second schematic diagram of a method for determining a rendering trajectory provided in an embodiment of the present application;

[0030] Fig.16This is a third schematic diagram of a method for determining a rendering trajectory provided in an embodiment of the present application;

[0031] Fig.17A This is a fourth schematic diagram of a method for determining a rendering trajectory provided in an embodiment of the present application;

[0032] Fig. 17B This is a fifth schematic diagram of a method for determining a rendering trajectory provided in an embodiment of the present application;

[0033] Fig.18 This is the seventh flowchart of an image rendering method provided in an embodiment of the present application;

[0034] Fig.19 This is a sixth schematic diagram of a method for determining a rendering trajectory provided in an embodiment of the present application;

[0035] Fig. 20 This is a flowchart of an image rendering method provided by an embodiment of the present application.

[0036] Fig.21 It is one of the schematic diagrams of a method for determining a rendering camera pose provided in an embodiment of the present application;

[0037] Fig. 22 This is a second schematic diagram of a method for determining a rendering camera pose provided in an embodiment of the present application;

[0038] Fig.23 This is a third schematic diagram of a method for determining a rendering camera pose provided in an embodiment of the present application;

[0039] Fig.24 This is a tenth flowchart of an image rendering method provided in an embodiment of the present application;

[0040] Fig.25 This is a flowchart of an image rendering method provided by an embodiment of the present application;

[0041] Fig.26 This is a twelfth flowchart of an image rendering method provided in an embodiment of the present application;

[0042] Fig. 27 is a structural schematic diagram of an image rendering device provided in an embodiment of the present application;

[0043] Fig.28 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0044] Fig.29 It is a hardware schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0046] The nouns or terms in the embodiments of the present application are explained below.

[0047] Structure From Motion (SFM): used for scene reconstruction from sparse point clouds using photographs.

[0048] Image-Based Simultaneously Localization And Mapping (SLAM): Use photos or cameras for real-time reconstruction, extract features for data association and state estimation, and achieve timely updates of states and features.

[0049] Neural Radiance Fields (NeRF): Obtained through photo training, it is stored by a neural network and can express the color and transparency of any point in a certain space as it changes with the observation angle.

[0050] 3D Gaussian Splatting (3DGS): A large number of 3D Gaussian ellipsoids are used to represent the scene. Each ellipsoid has a certain degree of transparency and a color that changes with the viewing angle.

[0051] Signed Distance Fields (SDF): A distance field with positive and negative values.

[0052] Marching Cube: SDF-based equidistant surface and level set surface extraction algorithm.

[0053] Dijkastra's algorithm and Fast Marching algorithm: shortest path search algorithms based on graph theory.

[0054] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0055] The image rendering method and device provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0056] The embodiments of the present application can be applied to at least one of the following scenarios: rendering images in electronic devices to generate videos with good quality and visual effects; rendering synthesized digital virtual scenes to generate realistic images; and rendering to enable augmented reality (AR) applications to more realistically simulate light and shadow effects in real scenes. The embodiments of the present application are not limited.

[0057] Exemplarily, the embodiments of the present application may be applied to a scenario in which a captured image or video is rendered, or may be applied to a scenario in which an image or screen in a preview interface is rendered.

[0058] The following two examples are used to illustrate the application scenarios of the embodiments of the present application.

[0059] Example 1: Rendering an image in an electronic device to generate a rendered video scene:

[0060] Assume that the electronic device uses 10 images, that is, N first images for rendering processing, and each image includes a background wall, a floor, a makeup box, a desk on which the makeup box is placed, and a potted plant next to the makeup box at different angles. Specifically, the electronic device can reconstruct a scene based on the 10 images, that is, a first reconstructed scene, and the reconstructed scene includes a makeup box, a desk, a potted plant, a background wall, and a floor, and obtain a depth image of each of the 10 images. Then, the electronic device can determine the equidistant surfaces of the makeup box, desk, potted plant, background wall, and floor where the rendering trajectory is located in the reconstructed scene based on the 10 depth images, project the starting shooting center position and the ending shooting center position in the 10 images onto the equidistant surface, determine the rendering trajectory, and then the electronic device can render the reconstructed scene according to the rendering trajectory to generate a rendered video. In this way, since the electronic device can obtain the equidistant surfaces in the reconstructed scene where the rendering trajectory is located based on the depth image of the image, and perform rendering processing according to the rendering trajectory to generate a rendered video, each object in the rendered video can appear on each frame of the rendered image with a stable size, thereby improving the quality of the rendered video.

[0061] Example 2: Rendering a synthesized digital virtual scene to generate a realistic image:

[0062] Assume that the electronic device uses a 2-second video of a digital virtual scene, and the 2-second video has 60 frames of digital virtual scene images, that is, N first images are rendered, and there are digitized background walls, floors, makeup boxes, desks on which makeup boxes are placed, and potted plants next to the makeup boxes in the video. Specifically, the electronic device can reconstruct the scene based on the 2-second video, that is, the first reconstructed scene, and the reconstructed scene includes makeup boxes, desks, potted plants, background walls, and floors, and obtains a depth image of each image in the 60 frames of images. Then, based on the 60 frames of depth images, the electronic device can determine the equidistant surfaces of the makeup box, desk, potted plant, background wall, and floor where the rendering trajectory is located in the reconstructed scene, project the starting shooting center position and the ending shooting center position in the 60 frames of images onto the equidistant surface, determine the rendering trajectory, and then the electronic device can render the reconstructed scene according to the rendering trajectory and output a frame of rendered image required by the user. In this way, since the electronic device can obtain the equidistant surfaces in the reconstructed scene where the rendering trajectory is located based on the depth image of the image, and perform rendering processing according to the rendering trajectory to output the rendered image, the various objects in the rendered image can appear on the rendered image with stable size, thereby improving the quality of the rendered image.

[0063] The image rendering method provided in the embodiment of the present application can be executed by an electronic device or at least one of the functional modules and entity modules in the electronic device that can implement the image rendering method. The specific execution can be determined according to actual use requirements and is not limited in the embodiment of the present application. In the embodiment of the present application, an electronic device executing the image rendering method is taken as an example to illustrate the image rendering method provided in the embodiment of the present application.

[0064] The present application embodiment provides an image rendering method, Figure 1 A schematic diagram of the process of the image rendering method provided by the embodiment of the present application is shown. Figure 1 As shown, the image rendering method provided in the embodiment of the present application may include steps 101 to 104.

[0065] Step 101: The electronic device obtains depth images of N first images based on a first reconstructed scene corresponding to the N first images.

[0066] Wherein, N is an integer greater than 1.

[0067] In some embodiments of the present application, the N first images may be images of the same scene taken at different angles using the same camera, or images of the same scene taken at different angles using different cameras, or multiple frames of images in a video. The specific embodiments of the present application are not limited.

[0068] In some implementations of the present application, the image content of the above-mentioned first image can be content in a real scene, or can be content in a digital virtual scene, and the embodiments of the present application are not limited to this.

[0069] In some embodiments of the present application, the above-mentioned shooting scene may be a scene containing objects, such as an indoor scene or outdoor scene containing multiple objects with complex geometric structures.

[0070] For example, the electronic device uses the same camera to shoot a group of scenes at different angles, including a table, multiple packaging boxes on the table, a bouquet of flowers in a vase behind some packaging boxes, multiple wine bottles next to the table, and images of background parts such as curtains. For example, the overall position of the objects on the table is photographed from the front of packaging box A to capture the outlines of the multiple packaging boxes and the bouquet of flowers on the table, the distribution of the multiple packaging boxes and the bouquet of flowers on the table, and the positions of the multiple wine bottles next to the table. The packaging box A is photographed from the side to obtain the width information of the multiple packaging boxes and the specific radius of the bouquet of flowers, while showing the distribution of the multiple wine bottles and the texture details of the background.

[0071] As another example, the electronic device uses different cameras to shoot a group of scenes at different angles, including a table, multiple packaging boxes on the table, a bouquet of flowers in a vase behind some of the packaging boxes, multiple wine bottles next to the table, and background parts such as curtains. For example, a high-resolution camera is used to shoot at a 45-degree angle from packaging box A to obtain a clear image of the items placed on the table, including the arrangement of multiple packaging boxes and the morphological features of the bouquet. A wide-angle camera is used to shoot downward from the top of packaging box A to obtain a macro perspective of the entire scene, including the patterns of the table and the floor and the layout of the entire room.

[0072] In some embodiments of the present application, the first reconstructed scene is a scene generated by processing the N first images through three-dimensional reconstruction technology, and contains the precise spatial position and depth information of each object in the corresponding shooting scene.

[0073] Specifically, the electronic device may use a 3D reconstruction technology, such as SFM technology, to perform sparse scene reconstruction, and obtain the position and posture of the shooting camera (hereinafter referred to as posture), the shooting camera lens parameters and the sparse expression of the scene. Then, the electronic device may use another 3D reconstruction technology, such as NeRF technology or 3DGS technology, to use the shooting camera posture, the shooting camera lens parameters and N first images to perform dense scene reconstruction, and obtain a dense reconstructed scene, that is, the first reconstructed scene, and the electronic device may obtain the depth image of the N first images from the dense scene.

[0074] In some embodiments of the present application, the position and posture of the above-mentioned shooting camera can be understood as: the shooting position and posture of the shooting camera when the shooting camera shoots an object.

[0075] For example, Figure 2 As shown, the vertex where the minimum angle of the triangle is located represents the center position of the shooting camera, and the direction of the minimum angle of the triangle toward the object is the posture of the shooting camera. Figure 2 T1 in the figure indicates that the time when the camera captured the object is T1. n Indicates that the time when the camera photographs the object is time T1.

[0076] In some embodiments of the present application, the depth image acquired by the electronic device based on the first reconstructed scene refers to the depth information of the depth image, that is, the information of each pixel in the depth image. It should be noted that the information of each pixel in the depth image refers to the distance information from the surface of the object where each pixel in the depth image is located to the center of the shooting. For example, Figure 3 As shown, it is a depth image of a captured image. Figure 3 Medium-light colors represent that the object surface is close to the center of the camera, and dark colors represent that the object surface is far from the center of the camera.

[0077] In some embodiments of the present application, the electronic device may use NeRF technology or 3DGS technology to reconstruct the scene and obtain the depth information of each image. Specifically, the electronic device may reconstruct the scene by using the camera projection formula to obtain the depth information of each image. Among them, the camera projection formula can be expressed by projecting a point in three-dimensional space onto a two-dimensional image plane. In the classic pinhole camera model, it is represented by the intrinsic parameter matrix K and the extrinsic parameter matrix R[I|-C]=[R|t]. Where R is the rotation matrix, Rt is the transposed matrix of the rotation matrix R, I is a three-row and three-column unit matrix, C is the camera center, and t=-RC.

[0078] Assume that a 3D point X is X = [X, Y, Z, 1], and through the camera projection equation, the corresponding 2D point x on the image plane can be obtained as x = [x, y, 1]. The formula x = KR[I|-C]X = K[R|t]X is shown in the following formula (1):

[0079]

[0080] Where K is the camera's intrinsic parameter matrix, as shown in the following formula (2):

[0081]

[0082] Step 102: The electronic device determines an equidistant surface where the rendering trajectory is located from a first reconstructed scene based on the depth images of the N first images.

[0083] In some embodiments of the present application, the electronic device can obtain the position of the shooting camera based on the three-dimensional reconstruction technology, such as the SFM technology, to obtain the size information of the object in the first reconstructed scene, and through the above formula (1), it can estimate the size of the target object in the first reconstructed scene on the image to obtain a scale. Alternatively, the electronic device can infer the distance from the best rendering trajectory to the surface of the object in the first reconstructed scene based on the preset rendering size to obtain a scale. Thus, the electronic device can determine the equidistant surface where the rendering trajectory is located based on the obtained scale.

[0084] The scale can be used to determine the display information of the rendered image when the electronic device performs rendering processing. For example, the larger the scale during the rendering process, the more detailed information of the rendered object will be displayed by the electronic device in the rendered image, and the smaller the scale during the rendering process, the more overall information of the rendered object will be displayed by the electronic device in the rendered image.

[0085] It should be noted that, for the specific method for the electronic device to determine the equidistant surface from the first reconstructed scene, reference may be made to the description in the following embodiments, which will not be described in detail here.

[0086] Step 103: The electronic device determines a first rendering trajectory based on the shooting center positions and equidistant surfaces corresponding to the N first images.

[0087] In some embodiments of the present application, the electronic device may project the position of the shooting camera obtained according to the 3D reconstruction technology, such as the SFM technology, onto an equidistant surface to obtain the position information of the center position of the shooting camera on the equidistant surface. The electronic device may then determine the first rendering trajectory based on the position information using a software algorithm, such as a trajectory estimation algorithm.

[0088] In some embodiments of the present application, in three-dimensional space, the zero-distance surface is the outer surface of an object in the first reconstructed scene.

[0089] In some embodiments of the present application, the equidistant surface is a surface in the first reconstructed scene that wraps the object and has a fixed distance to the surface of the object.

[0090] For example, assuming that there are two spheres with overlapping areas in the scene, the equidistant surfaces of the two spheres can be gourd-shaped surfaces centered on the two spheres. Figure 4 As shown, Figure 4 It is a schematic diagram of the projection of the two spheres and their equidistant surfaces onto the XOY surface, and lines 401 to 404 are the contour lines formed by projecting the four equidistant surfaces of the two spheres onto the XOY surface. For example, line 401 is the contour line corresponding to the equidistant surface 1 cm away from the two sphere surfaces, line 402 is the contour line corresponding to the equidistant surface 2 cm away from the two sphere surfaces, line 403 is the contour line corresponding to the equidistant surface 3 cm away from the two sphere surfaces, and line 404 is the contour line corresponding to the equidistant surface 4 cm away from the two sphere surfaces.

[0091] For example, assuming that the objects in the scene are two cube boxes with a length, width and height of 4 cm each, and one face of the two cube boxes is completely fitted together, then the equidistant surfaces of the two cube boxes are the surfaces formed by the cuboids. Figure 5 As shown, Figure 5It is a schematic diagram of the two cube boxes and their equidistant surfaces projected onto the XOY surface. After the zero-distance surfaces of the two cube boxes are projected onto the XOY surface, they are the outline of the rectangle formed by the two squares in the figure. The length of the rectangle is 4 cm and the width is 8 cm. Lines 501 to 504 outside the rectangle are the contour lines of the equidistant surfaces of the two cube boxes projected onto the XOY surface. For example, line 501 is the contour line corresponding to the equidistant surface 1 cm away from the surfaces of the two cube boxes, line 502 is the contour line corresponding to the equidistant surface 2 cm away from the surfaces of the two cube boxes, line 503 is the contour line corresponding to the equidistant surface 3 cm away from the surfaces of the two cube boxes, and line 504 is the contour line corresponding to the equidistant surface 4 cm away from the surfaces of the two cube boxes.

[0092] Step 104: The electronic device renders the first reconstructed scene based on the first rendering trajectory.

[0093] In some embodiments of the present application, the electronic device may use the above formula (1) to sequentially synthesize the projection image of the first reconstructed scene, determine the duration of generating the rendered video through user input, and sequentially and continuously render the synthesized projection image of the first reconstructed scene to generate the rendered video.

[0094] It should be noted that, for the specific method in which the electronic device performs rendering processing on the first reconstructed scene based on the first rendering trajectory, reference may be made to the description in the following embodiments, which will not be described in detail here.

[0095] In some embodiments of the present application, the electronic device performs rendering processing on the first reconstructed scene based on the first rendering trajectory, which can achieve at least one of the following effects:

[0096] 1) Increase the realistic effect of 3D model display;

[0097] 2) Real-time rendering of 3D scenes to support virtual reality and augmented reality applications;

[0098] 3) Real-time rendering of game scenes to improve user experience;

[0099] 4) Assist in advertising production and marketing;

[0100] 5) Assist users to interact with virtual scenes;

[0101] 6) Assist in the production of animation and film and television special effects.

[0102] Some embodiments of the present application provide an image rendering method, in which an electronic device can obtain N depth images corresponding to the N images based on a first reconstructed scene corresponding to the N first images, and then the electronic device can determine an equidistant surface where the rendering trajectory is located from the first reconstructed scene based on the N depth images, and determine the first rendering trajectory according to the equidistant surface and the shooting center positions corresponding to the N first images, and then the electronic device can use the first rendering trajectory on the equidistant surface to render the first reconstructed scene. Therefore, on the one hand, since the first rendering trajectory is determined according to the equidistant surface and the shooting center positions corresponding to the N first images, and there is a distance between the equidistant surface and the surface of the object in the first reconstructed scene, so that the equidistant surface will not intersect with the object, the phenomenon of the first rendering trajectory passing through the object can be avoided; on the other hand, since the above-mentioned equidistant surface is obtained by combining the depth information of the depth image, the equidistant surface obtained is continuous and smooth, that is, the first rendering trajectory moves continuously and smoothly on the equidistant surface, and then the electronic device uses the first rendering trajectory determined by the equidistant surface for rendering, which can ensure that the object appears in a stable and proportional manner on the rendered image, and avoid the phenomenon that the size of the object on the rendered image appears large and small. In this way, the quality of video rendered by electronic devices can be improved.

[0103] In some embodiments of the present application, Figure 1 ,like Figure 6 As shown, the above step 102 can be specifically implemented through the following steps 105 to 107.

[0104] Step 105: The electronic device establishes a distance field of a space where a first reconstructed scene is located based on the depth images of the N first images.

[0105] In some embodiments of the present application, the electronic device may set a spatial grid with a certain resolution in the first reconstruction scene, the size of which can cover the scene. For example, the following may be set: Figure 7 As shown, the resolution is 7×7×7 grid space, Figure 7 Each point in is the center of a cube grid, that is, the center point of the grid.

[0106] It can be understood that the spatial grid covering the scene will not be particularly small, thus avoiding the problem that the spatial grid is unable to cover the entire scene due to being particularly small, and the electronic device is unable to fully display the scene in the rendered image; the spatial grid covering the scene will not be infinitely large, thus avoiding the problem that the spatial grid is extremely large and causes waste of related resources used by the electronic device to render the image.

[0107] In some embodiments of the present application, the electronic device can establish a distance field in the space where the first reconstructed scene is located by adding the distance from the grid center point to the nearest object surface at each grid center point based on the above-mentioned depth image, the shooting center position and the grid center point of the above-mentioned spatial grid.

[0108] Step 106: The electronic device extracts the surface information of the object from the distance field of the space where the first reconstructed scene is located.

[0109] In some embodiments of the present application, the electronic device may use a surface extraction algorithm, such as a marching cube algorithm, to extract surface information of objects in the first reconstructed scene.

[0110] For example, the electronic device may determine the location of the surface of the object according to the distance value of each grid center point in the distance field.

[0111] Step 107: The electronic device determines the equidistant surface where the rendering trajectory is located based on the surface information of the object.

[0112] In some embodiments of the present application, the electronic device may use a surface extraction algorithm, such as a marching cube algorithm, to determine the equidistant surface where the rendering trajectory is located in the first reconstructed scene according to the above-mentioned object surface information.

[0113] For example, the electronic device may adjust the distance value of each grid center point in the distance field to the distance value from the corresponding grid point to the equidistant surface based on the location of the object surface, and then determine the equidistant surface using the method described in step 106 according to the distance value obtained after the adjustment.

[0114] In this way, since the electronic device can establish a distance field in the space where the first reconstructed scene is located based on the depth images of the N first images, so as to extract the surface information of the object from the distance field, and then determine the equidistant surface where the rendering trajectory is located based on the surface information, that is, there is a distance between the equidistant surface and the surface of the object in the scene, it can be ensured that the rendering trajectory will not intersect with the object, thereby ensuring the quality of the rendered video.

[0115] In some embodiments of the present application, Figure 6 ,like Figure 8 As shown, the above step 105 can be specifically implemented through the following steps 108 and 109.

[0116] Step 108: The electronic device projects each grid center point of the first grid onto at least one corresponding depth image to obtain projection point coordinate information of each grid center point projected onto the corresponding depth image.

[0117] In some embodiments of the present application, the first grid may be a grid covering the first reconstructed scene. The at least one depth image may be a depth image closest to the center point of the corresponding grid among the depth images of the N first images.

[0118] It should be noted that it is assumed that the depth images of the above-mentioned N first images include H depth images. For each grid center point of the above-mentioned first grid, a grid center point is projected onto at least one depth image corresponding to the one grid center point, and the at least one depth image is part or all of the H depth images. Moreover, the at least one depth image corresponding to each grid center point of the above-mentioned first grid may be the same or different, or the at least one depth image corresponding to some grid center points of the above-mentioned first grid may be the same.

[0119] In some embodiments of the present application, the electronic device can project each grid center point of the above-mentioned spatial grid onto several depth images that are closest to it and can be observed, that is, the depth image closest to the corresponding grid center point.

[0120] For example, Fig. 9 As shown, the electronic device can place the grid center point p j Projected onto the depth image 51 where the front of the packaging box can be seen, the grid center point p is obtained j The coordinate information d of the projection point 55 projected onto the corresponding depth image 51 i (x,y).

[0121] The above “can be observed” can be understood as: using the above formula (1), the projection point of the grid center point projected onto the depth image falls within the effective field of view, and the effective range refers to the scene displayed in the depth image.

[0122] Step 109: The electronic device establishes a distance field in the space where the first reconstructed scene is located based on the first distance from each grid center point to the shooting center position, the second distance from each grid center point to the object surface on the depth image, and the depth map value corresponding to the projection point coordinate information corresponding to each grid center point.

[0123] In some embodiments of the present application, the electronic device can calculate the first distance from each grid center point to the shooting center position. like Fig. 9 As shown, the grid center point p j To the shooting center position c i The distance between is the first distance. The first distance can be calculated using formula (3).

[0124]

[0125] In some embodiments of the present application, the electronic device can obtain the depth map value d corresponding to the projection point in the depth image according to the coordinate information (x, y) of the projection point corresponding to each grid center point. i (x, y). The depth map value can represent the distance from the object surface to the shooting center in the depth image.

[0126] In some embodiments of the present application, the second distance may be calculated from the corresponding first distance and the depth map value of the corresponding coordinate point.

[0127] In this way, since the electronic device can establish a distance field based on the first distance from each grid center point to the shooting center position in the spatial grid, the second distance from each grid center point to the surface of the object on the depth image, and the depth map value corresponding to the projection point coordinate information corresponding to each grid center point, the accuracy of the electronic device in establishing the distance field in the space where the first reconstructed scene is located can be improved.

[0128] In some embodiments of the present application, Figure 8 ,like Fig.10 As shown, before the above step 109, the image rendering method provided by the embodiment of the present application may include the following step 110.

[0129] Step 110: The electronic device calculates, for each grid center point of the first grid, a second distance from the center point of the first grid to the surface of the object in each depth image based on the difference between the first distance corresponding to the center point of the first grid and the value of the first depth map, to obtain the second distance from each grid center point to the surface of the object on the depth image.

[0130] In some embodiments of the present application, the first depth map value is a depth map value corresponding to the projection point coordinate information corresponding to the center point of the first grid. The first grid center point is any grid center point in the first grid.

[0131] It can be understood that for each grid center point of the first grid, the second distance corresponding to each grid center point can be obtained by executing the above step 110.

[0132] In some embodiments of the present application, the second distance dis from the center point of the first grid to the surface of the object in each depth image is i,j It can be calculated by formula (4).

[0133]

[0134] Where (x, y) in formula (4) is the grid center point p j The projected point coordinates on the depth image, d i (x,y) is the depth map value of the projected point in the depth image.

[0135] In this way, since the electronic device can calculate the second distance from the first grid center point to the surface of the object in each depth image based on the difference between the first distance corresponding to the first grid center point and the first depth map value for each grid center point of the first grid, and obtain the second distance from each grid center point to the surface of the object on the depth image, the position information of each grid center point in the object can be obtained, thereby improving the accuracy of the distance field established by the electronic device in the space where the first reconstructed scene is located.

[0136] In some embodiments of the present application, Figure 8 ,like Fig.11 As shown, the above step 109 can be specifically implemented through the following steps 111 and 112.

[0137] Step 111: The electronic device marks M grid center points in the first grid whose first distance is greater than or equal to the depth map value as first values, and marks L grid center points in the first grid whose first distance is less than the depth map value as second values, based on the relationship between the first distance corresponding to each grid center point and the depth map value.

[0138] In some embodiments of the present application, the M grid center points are located inside the object in the depth image, and the L grid center points are located outside the object in the depth image, where M and L are both positive integers.

[0139] In some embodiments of the present application, for each depth image, the electronic device may project the grid center point P (x, y, z) to the depth image coordinates (x, y). If the first distance from the grid center point to the shooting center position is Less than the depth map value d at the depth image coordinate (x, y) of the grid center point i (x, y), that is, the second distance in formula (4) is a positive value, then the grid center point is outside the object; if the first distance is greater than or equal to the depth image value of the grid center point projected to the depth image coordinate (x, y), that is, the second distance is zero or a negative value, then the grid center point is inside the object. In this way, the electronic device can distinguish between the M grid center points located inside the object and the L grid center points located inside the object.

[0140] In some embodiments of the present application, the electronic device can calculate the distance from the center point of the grid to the surface of the object in the above-mentioned depth image according to formula (5), and the distance is: the average value of the distance from the center point of the grid to the surface of the object in the depth image obtained by the nearest several cameras that can observe the object.

[0141]

[0142] In some embodiments of the present application, according to formula (5), the electronic device may mark the distance recorded by the L grid center points located outside the object as a positive distance, that is, mark the L grid center points as the second value. The electronic device may mark the distance recorded by the M grid center points located inside the object as a zero distance or a negative distance, that is, mark the M grid center points as the first value. Among them, the negative distance represents that the grid center point is inside the object.

[0143] The sum of M and L may be less than or equal to the sum of all grid center points.

[0144] In some embodiments of the present application, the electronic device can set the negative value threshold of the grid center point to the size of a preset number of grids, for example, the distance between two grid center points is set to a negative value threshold. If a grid center point is deep inside the object and exceeds the set threshold, the distance is no longer recorded.

[0145] For example, in a spatial grid, the values ​​marked on several grid center points are +1, -1, -2, -3, and -4 respectively. If the distance between two grid center points is set to a negative threshold, the distance between grid center points marked with values ​​of -3, -4, and grid center points with values ​​less than -4 will no longer be recorded.

[0146] In some embodiments of the present application, a value is marked for each grid center point in order to estimate the surface position of a scene object. Therefore, the distance value of the grid center point that is very deep inside the object can be ignored.

[0147] Step 112: The electronic device establishes a distance field in a space where a first reconstructed scene is located based on the second distance from each grid center point to the object surface on the depth image, the first value, and the second value.

[0148] In some embodiments of the present application, the electronic device may determine the first value and the second value based on the second distance from each grid center point to the surface of the object on the depth image. Then, the electronic device may mark the first value or the second value for each point in the spatial grid, that is, establish a distance field in the space where the first reconstructed scene is located.

[0149] For example, Fig. 12A In the spatial distance field shown, each grid center point can represent a value. For example, if the value of the grid center point is outside the range 12, it is a positive value, and if the value of the grid center point is within the range 12, it is a negative value.

[0150] In some embodiments of the present application, the electronic device may extract object surface information from the spatial distance field using a zero level set algorithm to obtain the spatial surface of the object, that is, the position of the object surface in the spatial distance field.

[0151] For example, the electronic device can use a related algorithm, such as the Marching Cube algorithm, to connect a grid center point with a value of +1 in the spatial distance field with a grid center point with a value of -1 that is closest to the +1 grid center point in the spatial distance field to form a line segment, and then the electronic device can take the midpoint of these two grid center points as the location of the object surface, and make the above judgment on all the grid center points in the spatial distance field to finally determine the object surface. Fig. 12B As shown, Fig. 12B The middle ellipsoid is the object surface extracted from the spatial distance field, that is, the surface with a distance of zero. This example is a simple description of the method of extracting the object surface by the Marching Cube algorithm. In actual implementation, the electronic device can perform more complex calculations, which is not limited in the embodiments of the present application.

[0152] For example, the electronic device can use the grid center point a with a value of +1 and the grid center point b with a value of -0.3 which is closest to the grid center point a to determine the location of the surface of the object.

[0153] In some embodiments of the present application, the electronic device may use an algorithm, such as a Marching Cube algorithm, to extract equidistant surfaces from a spatial distance field.

[0154] Exemplarily, when the electronic device uses the Marching Cube algorithm to extract an equidistant surface whose distance from the object surface is D and D>0, the electronic device can use the difference between all grid point values ​​and D to calculate the zero-distance surface, so that the electronic device can extract an equidistant surface at any distance from the object surface.

[0155] For example, Fig.13 Schematic diagram of equidistant surfaces extracted using the Marching Cube algorithm for an electronic device. Fig.13 As shown, Fig.13 shows a zero distance surface, i.e. Fig.13 The object surface shown in Figure 1 and the three equidistant surfaces extracted at positive distances. Fig.13 In the figure, the distance D1 surface is an equidistant surface whose distance from the object surface is D1, the distance D2 surface is an equidistant surface whose distance from the object surface is D2, and the distance D3 surface is an equidistant surface whose distance from the object surface is D3.

[0156] In this way, since the electronic device can mark each grid point with a signed distance value based on the size relationship between the first distance corresponding to each grid center point and the depth map value, and establish a distance field in the space where the first reconstructed scene is located, the electronic device can accurately extract the object surface and the equidistant surface, so that the rendering trajectory will not pass through the object when it is on the equidistant surface, thereby improving the quality of the video rendered by the electronic device.

[0157] In some embodiments of the present application, Figure 1 ,like Fig.14 As shown, the above step 103 can be specifically implemented through the following steps 113 and 114.

[0158] Step 113: The electronic device maps the shooting center position to an equidistant surface to obtain a first position point and a second position point.

[0159] In some embodiments of the present application, the first position point represents a starting shooting center position corresponding to the N first images, and the second position point represents a terminating shooting center position corresponding to the N first images.

[0160] In some embodiments of the present application, the electronic device can determine the starting shooting center position and the ending shooting center position in the N first images based on the three-dimensional reconstruction technology. Then the electronic device can project the two shooting center positions onto the equidistant surface obtained above to obtain the starting point position (first position point) and the ending point position (second position point) of the rendering trajectory.

[0161] For example, Fig.15A As shown, point 81 is the starting shooting center position point, point 83 is the projection point of the starting shooting center position point on the equidistant surface, point 82 is the ending shooting center position point, and point 83 is the projection point of the ending shooting center position point on the equidistant surface. Point 83 can be used as the starting point of the rendering trajectory, and point 84 can be used as the ending point of the rendering trajectory.

[0162] In some embodiments of the present application, the electronic device can use formula (6) to calculate the gradient of each grid center point (hereinafter referred to as a sampling point) in the spatial distance field. The gradient direction is the direction in which the value increases most rapidly, which is approximately the direction of the normal vector of the object surface.

[0163]

[0164] In some embodiments of the present application, the gradient at a non-sampling point can be replaced by the normal vector of the triangle patch. Fig.16 As shown, the ellipsoid surface is a zero-distance surface, and each line segment on the ellipsoid represents the normal vector of the ellipsoid surface.

[0165] In some embodiments of the present application, the electronic device can move each center position point of the shooting camera forward along the gradient direction calculated by formula (6) to obtain a larger value, or move backward to obtain a smaller value. The electronic device can iterate multiple times to find the isosurface that needs to be projected by the shooting center position point, that is, the equidistant surface.

[0166] For example, Fig.15A As shown, Fig.15A are spatial point 81 and spatial point 82 and their projections on the equidistant surface, wherein point 81 and point 82 are the original positions, and point 83 and point 84 are the projections of the original positions along the normal vector on the equidistant surface.

[0167] In some embodiments of the present application, Fig. 15B As shown, the dotted line 85 is the position sampling point of the rendering trajectory on the equidistant surface. The rendering trajectory is composed of a plurality of position sampling points.

[0168] In some embodiments of the present application, the electronic device may use an algorithm to find the triangular facet where the projection point of the shooting center position is located on the equidistant surface, and project the shooting center position onto the equidistant surface.

[0169] For example, Fig.17A As shown, point 1001 and point 1002 are projection points corresponding to the projection of the shooting center position point onto the equidistant surface.

[0170] Step 114: The electronic device determines the shortest path from the first position point along the equidistant surface to the second position point as a first rendering trajectory.

[0171] In some embodiments of the present application, the electronic device may use an algorithm, such as a trajectory estimation algorithm, to extract a geodesic line between two points on an equidistant surface and estimate the position of the rendering trajectory (the first rendering trajectory). Fig. 15B As shown, the position of the surface where the dotted line 85 is located is the position of the rendering trajectory.

[0172] It is understood that a geodesic line is the shortest distance path between two points on an equidistant surface. For example, a geodesic line may be the shortest path from a first position point along the equidistant surface to a second position point.

[0173] In some embodiments of the present application, the electronic device can estimate the starting center position point and the ending center position point of the shooting camera, project the starting center position point and the ending center position point onto an equidistant surface, obtain the starting point position and the ending point position of the rendering trajectory, and then the electronic device can estimate the rendering trajectory based on the starting point position and the ending point position of the rendering trajectory.

[0174] In some embodiments of the present application, the electronic device may also project the starting position and the ending position input by the user onto an equidistant surface to obtain the starting point position and the ending point position of the rendering trajectory, and then the electronic device may estimate the rendering trajectory based on the starting point position and the ending point position. In actual implementation, the electronic device may determine the rendering trajectory based on user needs, which is not limited in the embodiments of the present application.

[0175] For example, Fig. 15B As shown, point 83 can be used as the starting point of the rendering trajectory, and point 84 can be used as the ending point of the rendering trajectory.

[0176] In some embodiments of the present application, the electronic device can use an algorithm to set the projection point of the shooting center position on the equidistant surface as a new triangle vertex, and can connect the vertex with the three vertices of the triangle to form a new triangular mesh, that is, adding a vertex and three boundaries to the original triangle.

[0177] For example, Fig. 17B As shown, point 1001 is a vertex added to triangle patch 1003. Point 1001 is connected with three vertices of triangle patch 1003 to obtain three new boundaries, thereby forming a new triangle mesh.

[0178] In some embodiments of the present application, the geodesic between any two vertices on the triangular mesh can be used by an algorithm, such as the Fast Marching algorithm, to obtain the initial position of the rendering trajectory, and then the optimal position of the rendering trajectory is obtained according to the difference algorithm on the triangular face.

[0179] In this way, since the electronic device can map the shooting center position to the equidistant surface to obtain the first position point and the second position point, and determine the shortest path from the first position point to the second position point as the first rendering trajectory, the equidistant surface where the first rendering trajectory is located will not intersect with the object, and the first rendering trajectory can use the shortest path as the rendering path. Therefore, on the one hand, it can be achieved that the rendered object will not be passed through the rendering trajectory, and on the other hand, it can be achieved that the rendered object can appear in the rendered image of each frame in the rendered video in equal proportion, that is, the object in the rendered video can appear in the rendered video at a fixed size, thereby improving the quality of the rendered video.

[0180] In some embodiments of the present application, Figure 1 ,like Fig.18 As shown, the above step 103 can be specifically implemented through the following steps 115 to 118.

[0181] Step 115: The electronic device maps the shooting center positions of the N first images to an equidistant surface to obtain N position points.

[0182] For example, Fig.19 As shown, Fig.19 The points in are projection points where the shooting center positions of the N first images are projected (mapped) onto the equidistant surface.

[0183] In some embodiments of the present application, when an electronic device needs to play a rendered video seamlessly, the electronic device needs to obtain a closed first rendering trajectory.

[0184] Step 116: The electronic device calculates the shortest distance from the first center position point of the N position points to each of the N position points to obtain N shortest distances.

[0185] In some embodiments of the present application, the electronic device may estimate a center point, namely, a first center position point, based on the calculated projection position of the shooting center position on the equidistant surface.

[0186] In some embodiments of the present application, Fig.17A and Fig. 17B In the manner shown, a vertex and a boundary line are added to the triangular face for each projection point of the shooting center position to form a new triangular mesh. The electronic device can use an algorithm, such as a fast marching algorithm, to calculate the geodesic line and the geodesic distance from the first center position point to each projection point of the shooting center position. The geodesic distance is the length of the geodesic line.

[0187] Step 117: The electronic device determines Q shortest distances that are smaller than a threshold distance from the N shortest distances.

[0188] Wherein, Q is a positive integer less than or equal to N.

[0189] In some embodiments of the present application, the electronic device can calculate a threshold distance dis_T so that the distance from most of the projection points at the shooting center position, such as 80% of the projection points, to the first center position point (Q shortest distances) is less than the threshold distance dis_T, thereby determining Q shortest distances that are less than the threshold distance.

[0190] Step 118: The electronic device determines a first rendering trajectory based on the position points corresponding to the Q shortest distances and the first central position point.

[0191] In some embodiments of the present application, the electronic device can find a closed curve on the equidistant surface based on the threshold distance dis_T and the first center position point. The closed curve contains the projection points of most of the shooting center positions. The curve is the rendering trajectory to be found. The electronic device can determine the first rendering trajectory by using this method. Fig.19 As shown, curve 1101 is the first rendering trajectory.

[0192] In some embodiments of the present application, when the electronic device is performing the first reconstruction of the scene, the area that has been photographed more times will be reconstructed with high quality by the electronic device; the area that has been photographed less times will have poor reconstruction quality by the electronic device. Therefore, the reconstruction quality of the scene in the shooting center position area is higher; the reconstruction quality of the scene in the shooting edge position area is generally poor because the number of shots is small. Therefore, when the electronic device determines the first rendering trajectory, it is necessary to avoid reconstructing the edge area in the scene, that is, the electronic device must filter out projection points greater than the threshold distance (projection points of the shooting center position on the equidistant surface) based on the threshold distance.

[0193] In this way, since the electronic device can map the shooting center position onto an equidistant surface to obtain N position points, the electronic device can then calculate the first center position point of the N position points based on the N position points, obtain the shortest distance from the first center position point to each position point, and determine Q shortest distances less than the threshold distance from the N shortest distances, and determine the first rendering trajectory based on the position points corresponding to the Q shortest distances and the first center position point. Therefore, the electronic device can use the threshold distance to filter out points on the equidistant surface from which the shooting center position is mapped with large deviations, avoid reconstructing edge areas of poor quality in the scene, thereby improving the accuracy of the rendering trajectory and further improving the quality of the rendered video.

[0194] In some embodiments of the present application, Figure 1 ,like Fig. 20 As shown, after the above step 103, the image rendering method provided in the embodiment of the present application may include the following steps 119 to 120, and the above step 104 may be specifically implemented by the following step 121.

[0195] Step 119: The electronic device determines distribution information and position information of rendering cameras on the first rendering trajectory according to the rendering duration.

[0196] In some embodiments of the present application, the above-mentioned rendering cameras can be distributed on a first rendering trajectory, and the electronic device can determine the distribution information and position information of the rendering cameras on the first rendering trajectory based on the rendering duration and rendering frequency and equal distance difference calculation.

[0197] For example, Fig. 15B As shown, assuming that the rendering time is 1 second and the rendering frequency is 30 frames, 30 rendering cameras can be distributed with equal distance differences on the rendering trajectory 85, so as to determine the distribution information and position information of the rendering cameras on the rendering trajectory.

[0198] In some embodiments of the present application, the above rendering duration may be a default of the electronic device or may be pre-set by the user.

[0199] In some embodiments of the present application, the above rendering frequency may be a default setting of the electronic device or may be pre-set by the user.

[0200] Step 120: The electronic device determines the pose information of the first rendering camera for each rendering camera on the first rendering trajectory based on the equidistant surface, the position information of the first rendering camera of the electronic device, and the position information of the second rendering camera, to obtain the pose information of each rendering camera.

[0201] In some embodiments of the present application, the first rendering camera is any rendering camera on the first rendering track. The second rendering camera is a rendering camera next to the first rendering camera on the first rendering track.

[0202] In some embodiments of the present application, the electronic device can render the objects in the reconstructed scene according to the camera projection formula to generate a projection image, and multiple projection images can be rendered continuously to generate a rendered video. The camera projection formula can be the above x=KR[I|-C]X=K[R|t]X.

[0203] In some embodiments of the present application, after obtaining the position C of the rendering camera, the electronic device can also determine the rendering camera posture of each rendering camera at the rendering position, that is, the rotation matrix R, and then use the formula x=KR[I|-C]X=K[R|t]X for calculation.

[0204] In some embodiments of the present application, Fig.21 As shown in FIG. 1 , the normal vector 1202 of each point on the equidistant plane always points in the direction where the distance increases fastest. Fig. 22 As shown, the positive direction of the rendering camera plane will be parallel and opposite to the normal vector direction of the point, that is, the Z axis of the rendering camera is opposite to the normal vector direction +N, that is, the positive direction of the Z axis of the rendering camera is -N. In this way, it can be ensured that the rendering camera always faces the direction where the distance decreases the fastest, that is, the rendering camera faces the direction with the shortest distance to the object surface.

[0205] In some embodiments of the present application, each rendering camera plane is parallel to a tangent plane of a rendering point on the equidistant surface, wherein each rendering point indicates a rendering position in a rendering trajectory of the rendering camera on the equidistant surface. For example, the rendering camera plane 2201 is parallel to the tangent plane 2202, wherein the tangent plane 2202 is the tangent plane of the rendering point 2203 on the equidistant surface 2204.

[0206] In some embodiments of the present application, the rendering camera is at position point C i =[c x ,c y ,c z The normal vector at ] is This normal vector can indicate Fig.23 The positive direction of the Z axis of the rendering camera shown, that is, the normal vector is opposite to the positive direction of the Z axis, where n x *n x +n y *n y +n z *n z =1.

[0207] In some embodiments of the present application, the over-rendering camera position point C i The normal vector is The expression of the spatial plane can be determined using formula (7).

[0208] n x *x+n y *y+n z *z-(n x *c x +n y *c y +n z *c z )=0 (7)

[0209] In some embodiments of the present application, Fig.23 As shown, position point C i+1 Along the normal vector Projecting onto the above space plane, we can get the position point C′ i+1 Among them, the position point C i+1 Indicates that the rendering camera is at position C i The position of the camera for the next rendering.

[0210] In some embodiments of the present application, the X axis of the rendering camera can be represented by X=C′ i+1 -C i OK, so that the X axis of the rendering camera and position point C can be guaranteed i The normal vector at is perpendicular.

[0211]

[0212] Y=Z norm ×X norm (10)

[0213]

[0214] Y=(Z y X z -Z z X y , Z z X x -Z x Xz ,Z x X y -Z y Y x ) (12)

[0215] In some embodiments of the present application, the electronic device can obtain the solution of the XYZ coordinate system through formula (8) to formula (12); the rotation matrix R = [X, Y, Z] can be calculated by formula (13).

[0216]

[0217] In some embodiments of the present application, the rotation matrix calculation principle and process of the rendering camera are as follows: Fig. 22 As shown. At each distribution point, the rendering camera in the electronic device calculates the rotation matrix R = [X, Y, Z] by the above formulas (8) to (13).

[0218] Step 121: The electronic device renders the first reconstructed scene based on the position information of each rendering camera.

[0219] It should be noted that the electronic device performs rendering processing on the first reconstructed scene based on the posture information of each rendering camera. Please refer to the relevant description of the following embodiments, which will not be repeated here.

[0220] In this way, since the electronic device can determine the pose information of the first rendering camera for each rendering camera on the first rendering trajectory based on the equidistant surface, the position information of the first rendering camera of the electronic device and the position information of the second rendering camera, the pose information of each rendering camera can be obtained. Then, the electronic device can use the pose information to make the rendering camera always face the direction opposite to the normal vector of the object surface, render the scene in the first reconstruction, and generate a rendered video, thereby improving the quality of the rendered video.

[0221] In some embodiments of the present application, the above-mentioned equidistant surface may include a first equidistant surface and a second equidistant surface. Figure 1 ,like Fig.24 As shown, the above step 103 can be specifically implemented through the following steps 122 to 124.

[0222] Step 122: The electronic device maps the shooting center position to the first equidistant surface to obtain a second rendering trajectory.

[0223] Step 123: The electronic device maps the shooting center position to a second equidistant surface to obtain a third rendering trajectory.

[0224] In some embodiments of the present application, the method for the electronic device to determine the second rendering trajectory and the third rendering trajectory is similar to the method for determining the first rendering trajectory mentioned above. For specific methods, please refer to the relevant descriptions in the above steps. In order to avoid repetition, they are not described here.

[0225] Step 124: The electronic device performs weighted averaging on the position and pose information of the rendering camera on the second rendering trajectory and the position and pose information of the rendering camera on the third rendering trajectory to obtain a first rendering trajectory.

[0226] In some embodiments of the present application, the electronic device can calculate the maximum distance and the minimum distance from the rendering camera to the object according to the preset size range of the object in the image, thereby extracting multiple equidistant surfaces in the spatial distance field, such as Fig.13 As shown, the electronic device can extract the surface at distance D1, the surface at distance D2, and the surface at distance D3.

[0227] In some embodiments of the present application, the electronic device can achieve a gradual visual effect of objects changing from far to near and from near to far in the rendered video according to user needs.

[0228] In some embodiments of the present application, when the electronic device needs to achieve the effect of an object moving from far to near in the rendered video, the starting point of the rendering trajectory of the rendering camera is distributed on equidistant surfaces with a larger distance from the object surface. In the process of the electronic device moving the rendering camera in each frame, in addition to the horizontal movement of the rendering camera, that is, the movement in the X-axis direction, the rendering camera also increases slow movement in the vertical direction of the camera plane, that is, the Z-axis direction, so that the rendering camera gradually approaches the projection point position on the equidistant surface with a smaller distance from the object surface, thereby achieving a gradual effect of the rendered object from small to large. Conversely, when the electronic device needs the rendering camera to be away from the object surface, it allows each rendering camera position to slowly retreat in the vertical direction of the rendering camera plane, that is, the Z-axis direction, along the equidistant surface normal direction.

[0229] In some embodiments of the present application, the electronic device can determine which rendering trajectory the rendering camera is closer to by taking a weighted average of the posture information of the rendering camera on the second rendering trajectory and the posture information of the rendering camera on the third rendering trajectory, thereby obtaining a synthetic trajectory that can achieve a gradient effect, namely, the first rendering trajectory.

[0230] In some embodiments of the present application, the electronic device can achieve a uniform change in the distance from the rendering camera to the object by performing weighted averaging on the posture information of the rendering camera on the second rendering trajectory and the posture information of the rendering camera on the third rendering trajectory, so that the electronic device can make the objects in the rendered video change uniformly, presenting a visual effect of the objects in the rendered video moving from far to near or from near to far.

[0231] In this way, since the electronic device can perform weighted averaging on the posture information of the rendering camera on the second rendering trajectory and the posture information of the rendering camera on the third rendering trajectory to obtain the first rendering trajectory, it is possible to achieve a stable enlargement or reduction in the size of objects in the rendered video, thereby improving the quality of the rendered video.

[0232] In some embodiments of the present application, Figure 1 ,like Fig.25 As shown, the above step 104 can be specifically implemented through the following step 125.

[0233] Step 125: The electronic device performs rendering processing at the distribution position of each rendering camera in sequence and continuously based on the position information of each rendering camera on the first rendering trajectory to obtain a rendered video.

[0234] In some embodiments of the present application, the electronic device may deliver a continuous and smooth first rendering trajectory to a rendering algorithm. The rendering algorithm then obtains Ri and Ci at each rendering camera pose. The electronic device may then use the camera projection formula to correspond to the projection image of the synthetic scene, and the electronic device may sequentially and continuously perform rendering processing at the distribution position of each rendering camera to obtain a rendered video at all sampling positions, and continuous rendering photos to form the final rendered video.

[0235] In this way, since the electronic device can perform rendering processing at the distribution position of each rendering camera in sequence and continuously based on the posture information of each rendering camera on the first rendering trajectory to obtain a rendered video, the quality of the rendered video can be improved.

[0236] In some embodiments of the present application, since the first rendering trajectory is calculated by the electronic device, the electronic device can obtain a camera movement track that ensures a stable display of the object size, as well as a camera posture of the camera movement.

[0237] In some embodiments of the present application, since the rendering cameras are distributed on the first rendering trajectory, and the first rendering trajectory may be on an equidistant surface in the first reconstructed scene, and the equidistant surface is the equidistant surface of the objects in the first reconstructed scene, therefore, on the one hand, the electronic device can ensure that the distance from the rendering scene to the rendering camera remains unchanged, thereby ensuring that the size of the objects in the scene in the rendered video is stable and will not fluctuate; on the other hand, the equidistant surface can ensure that the rendering trajectory will never intersect with the objects in the first reconstructed scene.

[0238] Combine the following Fig.26 , the image rendering method provided in the embodiment of the present application is exemplarily described. Fig.26As shown, the rectangular box represents the input required by the electronic device in the process of generating the rendered video, or the content obtained by the electronic device according to the algorithm, and the parallelogram represents the algorithm used by the electronic device in the process of generating the rendered video.

[0239] Illustratively, some embodiments of the present application may be implemented through the following steps 201 to 208 .

[0240] Step 201: The electronic device restores the position and posture of the shooting camera based on the video or a group of photos, performs sparse scene reconstruction of the scene, and obtains the shooting camera internal parameters and the shooting camera motion trajectory.

[0241] Among them, the electronic device can determine the movement trajectory of the shooting camera according to the shooting camera posture.

[0242] Step 202: The electronic device performs dense scene reconstruction based on the radiation field technology to obtain the radiation field and depth image of the scene, where the radiation field of the scene is the first reconstructed scene, and the depth image is the depth image of the N first images.

[0243] Step 203: The electronic device obtains a spatial distance field of the reconstructed scene based on the depth map fusion technology and the depth image, that is, the distance field of the space where the first reconstructed scene is located.

[0244] Step 204: The electronic device obtains the starting point and the ending point of the shooting camera based on the motion trajectory of the shooting camera, that is, the above-mentioned starting shooting center position and ending shooting center position.

[0245] Step 205: The electronic device obtains a rendering distance based on the internal parameters of the shooting camera.

[0246] Among them, the electronic device can obtain the size of the object after rendering on the rendered image according to the rendering distance, that is, the scale information in the above embodiment.

[0247] Step 206: The electronic device obtains the position of the equidistant surface, that is, the equidistant surface mentioned above, based on the rendering distance and the reconstructed spatial distance field.

[0248] Step 207: The electronic device projects the starting point and the end point of the shooting camera onto an equidistant surface, and uses a trajectory generation algorithm to obtain a rendering trajectory, namely, a first rendering trajectory.

[0249] Step 208: The electronic device obtains a rendered video using a rendering algorithm based on the radiation field of the scene, that is, the first reconstructed scene, and the rendering trajectory.

[0250] In some embodiments of the present application, for the detailed description of the above steps 201 to 208, reference may be made to the relevant descriptions in the above steps, and in order to avoid repetition, they are not described here.

[0251] The image rendering method provided in the embodiment of the present application may be executed by an image rendering device. In the embodiment of the present application, an image rendering device executing the image rendering method is taken as an example to illustrate the image rendering device provided in the embodiment of the present application.

[0252] like Fig. 27 As shown, the embodiment of the present application further provides an image rendering device 1500, which may include: an acquisition module 1501, a determination module 1502, and a processing module 1503. The acquisition module 1501 may be used to acquire depth images of N first images based on a first reconstructed scene corresponding to the N first images, where N is an integer greater than 1; the determination module 1502 may be used to determine the equidistant surface where the rendering trajectory is located from the first reconstructed scene based on the depth images of the N first images acquired by the acquisition module 1501, and may be used to determine the first rendering trajectory based on the shooting center position and the equidistant surface corresponding to the N first images; the processing module 1503 may be used to render the first reconstructed scene based on the first rendering trajectory determined by the determination module 1502.

[0253] In some embodiments of the present application, the above-mentioned determination module 1502 may be specifically used to:

[0254] Projecting each grid center point of the first grid onto at least one corresponding depth image, respectively, to obtain projection point coordinate information of each grid center point projected onto the corresponding depth image, wherein the first grid is a grid covering the first reconstructed scene, and the at least one depth image is a depth image that is closest to the corresponding grid center point among the depth images of the N first images;

[0255] Based on a first distance from each grid center point to the shooting center position, a second distance from each grid center point to the object surface on the depth image, and a depth map value corresponding to the projection point coordinate information corresponding to each grid center point, a distance field of the space where the first reconstructed scene is located is established;

[0256] From the distance field, extract the object surface information;

[0257] Based on the surface information of the object, the above-mentioned equidistant surface is determined.

[0258] In some embodiments of the present application, the above-mentioned processing module 1503 can also be used to calculate, for each grid center point of the first grid, a second distance from the first grid center point to the surface of the object in each depth image based on the difference between the first distance corresponding to the first grid center point and the first depth map value, to obtain the second distance from each grid center point to the surface of the object on the depth image; wherein the first depth map value is the depth map value corresponding to the projection point coordinate information corresponding to the first grid center point, and the first grid center point is any grid center point in the first grid.

[0259] In some embodiments of the present application, the above-mentioned determination module 1502 can be specifically used to: based on the size relationship between the first distance corresponding to each grid center point and the depth map value, mark the M grid center points in the first grid whose first distance is greater than or equal to the depth map value as the first value, and mark the L grid center points in the first grid whose first distance is less than the depth map value as the second value, the M grid center points are located on the inner side of the object in the depth image, and the L grid center points are located on the outer side of the object in the depth image, and M and L are both positive integers; based on the second distance from each grid center point to the surface of the object on the depth image, the first value and the second value, establish a distance field.

[0260] In some embodiments of the present application, the above-mentioned determination module 1502 can be specifically used to: map the shooting center position to the above-mentioned equidistant surface to obtain a first position point and a second position point, the first position point represents the starting shooting center position corresponding to the N first images, and the second position point represents the ending shooting center position corresponding to the N first images; and determine the shortest path from the first position point along the above-mentioned equidistant surface to the second position point as the first rendering trajectory.

[0261] In some embodiments of the present application, the above-mentioned determination module 1502 can be specifically used to: map the shooting center position to the above-mentioned equidistant surface to obtain N position points; calculate the shortest distance from the first center position point of the N position points to each position point of the N position points to obtain N shortest distances; from the N shortest distances, determine Q shortest distances that are less than a threshold distance, where Q is a positive integer less than or equal to N; based on the position points corresponding to the Q shortest distances and the first center position point, determine the first rendering trajectory.

[0262] In some embodiments of the present application, the determination module 1502 can also be used to determine the distribution information and position information of the rendering camera on the first rendering trajectory according to the rendering duration after determining the first rendering trajectory based on the shooting center positions corresponding to the N first images and the equidistant surface; and, for each rendering camera on the first rendering trajectory, determine the posture information of the first rendering camera based on the equidistant surface, the position information of the first rendering camera, and the position information of the second rendering camera, and obtain the posture information of each rendering camera; the processing module 1503 can be specifically used to render the first reconstructed scene based on the posture information of each rendering camera. The first rendering camera is any rendering camera on the first rendering trajectory, and the second rendering camera is the next rendering camera of the first rendering camera on the first rendering trajectory.

[0263] In some embodiments of the present application, the above-mentioned equidistant surface may include a first equidistant surface and a second equidistant surface; the above-mentioned determination module 1502 can be specifically used to: map the shooting center position to the first equidistant surface to obtain a second rendering trajectory; map the shooting center position to the second equidistant surface to obtain a third rendering trajectory; and perform weighted averaging on the posture information of the rendering camera on the second rendering trajectory and the posture information of the rendering camera on the third rendering trajectory to obtain the first rendering trajectory.

[0264] In some embodiments of the present application, the processing module 1503 may be specifically used to perform rendering processing at the distribution position of each rendering camera in sequence and continuously based on the posture information of each rendering camera on the first rendering trajectory to obtain a rendered video.

[0265] The embodiment of the present application provides an image rendering device, which can obtain N depth images corresponding to the N images based on a first reconstructed scene corresponding to the N first images, and then the image rendering device can determine the equidistant surface where the rendering trajectory is located from the first reconstructed scene based on the N depth images, and determine the first rendering trajectory according to the equidistant surface and the shooting center positions corresponding to the N first images, and then the image rendering device can use the first rendering trajectory to render the first reconstructed scene. Therefore, on the one hand, since the first rendering trajectory is determined according to the equidistant surface and the shooting center positions corresponding to the N first images, and there is a distance between the equidistant surface and the surface of the object in the first reconstructed scene, so that the equidistant surface will not intersect with the object, it is possible to avoid the phenomenon that the first rendering trajectory passes through the object; on the other hand, since the above-mentioned equidistant surface is obtained by combining the depth information of the depth image, the obtained equidistant surface is continuous and smooth, that is, the first rendering trajectory moves continuously and smoothly on the equidistant surface, and then the image rendering device uses the first rendering trajectory determined by the equidistant surface for rendering, which can ensure that the object appears in a stable and proportional manner on the rendered image, and avoid the phenomenon that the size of the object on the rendered image appears to be large and small. In this way, the quality of video rendered by the image rendering device can be improved.

[0266] The image rendering device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0267] The image rendering device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0268] The image rendering device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0269] Alternatively, if Fig.28 As shown, an embodiment of the present application also provides an electronic device 2600, including a processor 2601 and a memory 2602, wherein the memory 2602 stores programs or instructions that can be executed on the processor 2601, and when the program or instructions are executed by the processor 2601, the various steps of the above-mentioned image rendering method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0270] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0271] Fig.29 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0272] The electronic device 1700 includes but is not limited to: a radio frequency unit 1701, a network module 1702, an audio output unit 1703, an input unit 1704, a sensor 1705, a display unit 1706, a user input unit 1707, an interface unit 1708, a memory 1709, and a processor 1710 and other components.

[0273] Those skilled in the art will appreciate that the electronic device 1700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.29 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0274] Among them, the processor 1710 can be used to obtain depth images of N first images based on the first reconstructed scene corresponding to the N first images, where N is an integer greater than 1; the processor 1710 can be used to determine the equidistant surface where the rendering trajectory is located from the first reconstructed scene based on the depth images of the N first images obtained by the processor 1710, and can be used to determine the first rendering trajectory based on the shooting center position and the equidistant surface corresponding to the N first images; the processor 1710 can be used to render the first reconstructed scene based on the first rendering trajectory determined by the processor 1710.

[0275] In some embodiments of the present application, the processor 1710 may be specifically used to: project each grid center point of the first grid onto at least one corresponding depth image, and obtain projection point coordinate information of each grid center point projected onto the corresponding depth image, wherein the first grid is a grid covering the first reconstructed scene, and the at least one depth image is a depth image closest to the corresponding grid center point among the depth images of the N first images; establish a distance field of the space where the first reconstructed scene is located based on a first distance from each grid center point to the shooting center position, a second distance from each grid center point to the object surface on the depth image, and a depth map value corresponding to the projection point coordinate information corresponding to each grid center point; extract object surface information from the distance field; and determine the equidistant surface based on the object surface information.

[0276] In some embodiments of the present application, the above-mentioned processor 1710 can also be used to calculate, for each grid center point of the first grid, a second distance from the first grid center point to the surface of the object in each depth image based on the difference between the first distance corresponding to the first grid center point and the first depth map value, to obtain the second distance from each grid center point to the surface of the object on the depth image; wherein the first depth map value is the depth map value corresponding to the projection point coordinate information corresponding to the first grid center point, and the first grid center point is any grid center point in the first grid.

[0277] In some embodiments of the present application, the above-mentioned processor 1710 can be specifically used to: based on the size relationship between the first distance corresponding to each grid center point and the depth map value, mark the M grid center points in the first grid whose first distance is greater than or equal to the depth map value as the first value, and mark the L grid center points in the first grid whose first distance is less than the depth map value as the second value, the M grid center points are located on the inner side of the object in the depth image, and the L grid center points are located on the outer side of the object in the depth image, and M and L are both positive integers; based on the second distance from each grid center point to the surface of the object on the depth image, the first value and the second value, establish a distance field.

[0278] In some embodiments of the present application, the above-mentioned processor 1710 can be specifically used to: map the shooting center position to the above-mentioned equidistant surface to obtain a first position point and a second position point, the first position point represents the starting shooting center position corresponding to the N first images, and the second position point represents the ending shooting center position corresponding to the N first images; and determine the shortest path from the first position point along the above-mentioned equidistant surface to the second position point as the first rendering trajectory.

[0279] In some embodiments of the present application, the above-mentioned processor 1710 can be specifically used to: map the shooting center position to the above-mentioned equidistant surface to obtain N position points; calculate the shortest distance from the first center position point of the N position points to each position point of the N position points to obtain N shortest distances; from the N shortest distances, determine Q shortest distances that are less than a threshold distance, where Q is a positive integer less than or equal to N; based on the position points corresponding to the Q shortest distances and the first center position point, determine the first rendering trajectory.

[0280] In some embodiments of the present application, the processor 1710 can also be used to determine the distribution information and position information of the rendering cameras on the first rendering trajectory according to the rendering duration after determining the first rendering trajectory based on the shooting center positions corresponding to the N first images and the equidistant surface; for each rendering camera on the first rendering trajectory, determine the pose information of the first rendering camera based on the equidistant surface, the position information of the first rendering camera, and the position information of the second rendering camera, and obtain the pose information of each rendering camera; the processor 1710 can be specifically used to render the first reconstructed scene based on the pose information of each rendering camera. The first rendering camera is any rendering camera on the first rendering trajectory, and the second rendering camera is the next rendering camera of the first rendering camera on the first rendering trajectory.

[0281] In some embodiments of the present application, the above-mentioned equidistant surface may include a first equidistant surface and a second equidistant surface; the above-mentioned processor 1710 can be specifically used to: map the shooting center position to the first equidistant surface to obtain a second rendering trajectory; map the shooting center position to the second equidistant surface to obtain a third rendering trajectory; and perform weighted averaging on the posture information of the rendering camera on the second rendering trajectory and the posture information of the rendering camera on the third rendering trajectory to obtain the first rendering trajectory.

[0282] In some embodiments of the present application, the processor 1710 may be specifically configured to perform rendering processing at the distribution position of each rendering camera in sequence and continuously based on the posture information of each rendering camera on the first rendering trajectory to obtain a rendered video.

[0283] The embodiment of the present application provides an electronic device, which can obtain N depth images corresponding to the N images based on the first reconstructed scene corresponding to the N first images, and then the electronic device can determine the equidistant surface where the rendering trajectory is located from the first reconstructed scene based on the N depth images, and determine the first rendering trajectory according to the equidistant surface and the shooting center position corresponding to the N first images, and then the electronic device can use the first rendering trajectory to render the first reconstructed scene. Therefore, on the one hand, since the first rendering trajectory is determined according to the equidistant surface and the shooting center position corresponding to the N first images, and there is a distance between the equidistant surface and the surface of the object in the first reconstructed scene, so that the equidistant surface will not intersect with the object, it is possible to avoid the phenomenon that the first rendering trajectory passes through the object; on the other hand, since the above-mentioned equidistant surface is obtained by combining the depth information of the depth image, the equidistant surface obtained is continuous and smooth, that is, the first rendering trajectory moves continuously and smoothly on the equidistant surface, and then the electronic device uses the first rendering trajectory determined by the equidistant surface for rendering, which can ensure that the object appears in a stable and proportional manner on the rendered image, and avoid the phenomenon that the size of the object on the rendered image appears large and small. In this way, the quality of the video rendered by the electronic device can be improved.

[0284] It should be understood that in the embodiment of the present application, the input unit 1704 may include a graphics processor (Graphics Processing Unit, GPU) 17041 and a microphone 17042, and the graphics processor 17041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1706 may include a display panel 17061, and the display panel 17061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1707 includes a touch panel 17071 and at least one of other input devices 17072. The touch panel 17071 is also called a touch screen. The touch panel 17071 may include two parts: a touch detection device and a touch controller. Other input devices 17072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0285] The memory 1709 can be used to store software programs and various data. The memory 1709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1709 may include a volatile memory or a non-volatile memory, or the memory 1709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0286] The processor 1710 may include one or more processing units; optionally, the processor 1710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1710.

[0287] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image rendering method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0288] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0289] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image rendering method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0290] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0291] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image rendering method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0292] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0293] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0294] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image rendering method, characterized in that: The method comprises: Based on a first reconstructed scene corresponding to the N first images, acquiring depth images of the N first images, where N is an integer greater than 1; Based on the depth image, determining an equidistant surface where a rendering trajectory is located from the first reconstructed scene; Determine a first rendering trajectory based on the shooting center positions corresponding to the N first images and the equidistant surface; Based on the first rendering trajectory, rendering processing is performed on the first reconstructed scene.

2. The method according to claim 1, characterized in that The step of determining, based on the depth image, an equidistant surface on which the rendering trajectory is located from the first reconstructed scene comprises: Projecting each grid center point of the first grid onto at least one corresponding depth image, respectively, to obtain projection point coordinate information of each grid center point projected onto the corresponding depth image, wherein the first grid is a grid covering the first reconstructed scene, and the at least one depth image is a depth image that is closest to the corresponding grid center point among the depth images of the N first images; Establishing a distance field of a space where the first reconstructed scene is located based on a first distance from each grid center point to the shooting center position, a second distance from each grid center point to the surface of the object on the depth image, and a depth map value corresponding to the projection point coordinate information corresponding to each grid center point; Extracting object surface information from the distance field; The equidistant surface is determined based on the surface information of the object.

3. The method according to claim 2, characterized in that The method further comprises: For each grid center point of the first grid, based on the difference between the first distance corresponding to the first grid center point and the first depth map value, calculate the second distance from the first grid center point to the object surface in each depth image, and obtain the second distance from each grid center point to the object surface on the depth image; The first depth map value is a depth map value corresponding to the projection point coordinate information corresponding to the first grid center point, and the first grid center point is any grid center point in the first grid.

4. The method according to claim 2 or 3, characterized in that: The establishing of the distance field based on a first distance from each grid center point to the shooting center position, a second distance from each grid center point to the object surface on the depth image, and a depth map value corresponding to the projection point coordinate information corresponding to each grid center point includes: Based on the relationship between the first distance corresponding to each grid center point and the depth map value, M grid center points in the first grid whose first distance is greater than or equal to the depth map value are marked as first values, and L grid center points in the first grid whose first distance is less than the depth map value are marked as second values, the M grid center points are located on the inner side of the object in the depth image, and the L grid center points are located on the outer side of the object in the depth image, and M and L are both positive integers; The distance field is established based on the second distance from each grid center point to the surface of the object on the depth image, the first value, and the second value.

5. The method according to claim 1, characterized in that The determining a first rendering trajectory based on the shooting center positions corresponding to the N first images and the equidistant surface includes: Mapping the shooting center position to the equidistant surface to obtain a first position point and a second position point, wherein the first position point represents a starting shooting center position corresponding to the N first images, and the second position point represents an ending shooting center position corresponding to the N first images; The shortest path from the first position point along the equidistant surface to the second position point is determined as the first rendering trajectory.

6. An image rendering device, characterized in that: The device comprises: an acquisition module, a determination module and a processing module; The acquisition module is used to acquire depth images of the N first images based on a first reconstructed scene corresponding to the N first images, where N is an integer greater than 1; The determining module is used to determine the equidistant surface where the rendering trajectory is located from the first reconstructed scene based on the depth image acquired by the acquiring module; The determination module is further used to determine a first rendering trajectory based on the shooting center positions corresponding to the N first images and the equidistant surface; The processing module is used to perform rendering processing on the first reconstructed scene based on the first rendering trajectory determined by the determination module.

7. The device according to claim 6, characterized in that The determining module is specifically used for: Projecting each grid center point of the first grid onto at least one corresponding depth image, respectively, to obtain projection point coordinate information of each grid center point projected onto the corresponding depth image, wherein the first grid is a grid covering the first reconstructed scene, and the at least one depth image is a depth image that is closest to the corresponding grid center point among the depth images of the N first images; Establishing a distance field of a space where the first reconstructed scene is located based on a first distance from each grid center point to the shooting center position, a second distance from each grid center point to the surface of the object on the depth image, and a depth map value corresponding to the projection point coordinate information corresponding to each grid center point; Extracting object surface information from the distance field; The equidistant surface is determined based on the surface information of the object.

8. The device according to claim 7, characterized in that The processing module is further configured to calculate, for each grid center point of the first grid, a second distance from the center point of the first grid to the surface of the object in each depth image based on a difference between a first distance corresponding to the center point of the first grid and a value of the first depth image, so as to obtain a second distance from each grid center point to the surface of the object on the depth image; The first depth map value is a depth map value corresponding to the projection point coordinate information corresponding to the first grid center point, and the first grid center point is any grid center point in the first grid.

9. The device according to claim 7 or 8, characterized in that The determining module is specifically used for: Based on the relationship between the first distance corresponding to each grid center point and the depth map value, M grid center points in the first grid whose first distance is greater than or equal to the depth map value are marked as first values, and L grid center points in the first grid whose first distance is less than the depth map value are marked as second values, the M grid center points are located on the inner side of the object in the depth image, and the L grid center points are located on the outer side of the object in the depth image, and M and L are both positive integers; The distance field is established based on the second distance from each grid center point to the surface of the object on the depth image, the first value, and the second value.

10. The device according to claim 6, characterized in that The determining module is specifically used for: Mapping the shooting center position to the equidistant surface to obtain a first position point and a second position point, wherein the first position point represents a starting shooting center position corresponding to the N first images, and the second position point represents an ending shooting center position corresponding to the N first images; The shortest path from the first position point along the equidistant surface to the second position point is determined as the first rendering trajectory.

Images