Image rendering method and device, equipment and storage medium

Abstract

The invention discloses an image rendering method and device, equipment and a storage medium, and the method comprises the steps: obtaining a three-dimensional point cloud, the three-dimensional point cloud comprises a plurality of three-dimensional points, the attributes of the three-dimensional points comprise a position attribute and a color related attribute, and the color related attribute comprises color information and also comprises at least one of transparency and reflectivity; determining a point set corresponding to each pixel point in the rendered image under the target visual angle from the three-dimensional point cloud; and determining the pixel value of each pixel point based on the attribute of the three-dimensional point in the point set corresponding to each pixel point. According to the scheme, the rendering effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image rendering method, apparatus, device and storage medium. Background Art

[0002] Rendering, in computer graphics, refers to the process of converting a 3D model, animation, or scene into a two-dimensional image or animation through software algorithms.

[0003] The existing traditional rendering technology usually implements rendering based on position information and color information, but the rendering effect of this method is not very detailed. How to improve the image rendering effect has become an urgent problem to be solved. Summary of the invention

[0004] The present application at least provides an image rendering method, apparatus, device and storage medium.

[0005] The present application provides an image rendering method, including: obtaining a three-dimensional point cloud, the three-dimensional point cloud including multiple three-dimensional points, the attributes of the three-dimensional points including position attributes and color-related attributes, the color-related attributes including color information, and also including at least one of transparency and reflectivity; determining from the three-dimensional point cloud a point set corresponding to each pixel point in a rendered image under a target perspective; and determining a pixel value of each pixel point based on the attributes of the three-dimensional points in the point set corresponding to each pixel point.

[0006] The present application provides an image rendering device, including an acquisition module, a point selection module and a pixel value determination module. The acquisition module is used to acquire a three-dimensional point cloud, which includes a plurality of three-dimensional points, and the attributes of the three-dimensional points include position attributes and color-related attributes, and the color-related attributes include color information and at least one of transparency and reflectivity; the point selection module is used to determine the point set corresponding to each pixel point in the rendering image under the target perspective from the three-dimensional point cloud; the pixel value determination module is used to determine the pixel value of each pixel point based on the attributes of the three-dimensional points in the point set corresponding to each pixel point.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement any of the above methods.

[0008] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, any of the above methods is implemented.

[0009] In the above scheme, position attributes and color-related attributes are added to the three-dimensional points, wherein the color-related attributes include color information and at least one of transparency and reflectivity. At least one of transparency and reflectivity participates in rendering, providing richer representation information of the object to be rendered, making the rendering image closer to the real situation and improving the rendering effect.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 It is a flowchart of an embodiment of the image rendering method of the present application; Figure 2 is a schematic diagram of a three-dimensional point in an embodiment of the image rendering method of the present application; Figure 3 is a schematic diagram of sampling points in an embodiment of the image rendering method of the present application; Figure 4 This application Figure 1 A flowchart of another embodiment of step S110 in FIG. Figure 5 is a schematic diagram of another embodiment of the image rendering method of the present application; Figure 6 It is a schematic diagram of the framework of an embodiment of the image rendering device of the present application; Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0013] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0014] In the following description, for the purpose of explanation rather than limitation, specific details such as specific subsystem structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0015] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an image rendering method of the present application. The method may be executed by a processing device. Specifically, the method may include: Step S110: Acquire a three-dimensional point cloud.

[0017] It can be understood that the above three-dimensional point cloud can represent the object to be rendered. The three-dimensional point cloud includes a plurality of three-dimensional points. Each three-dimensional point has its attributes, and the attributes of the three-dimensional point include position attributes and color-related attributes. The position attribute can represent the position of the three-dimensional point, and the color-related attribute can represent the information related to the color of the three-dimensional point. The color-related attribute includes color information and at least one of transparency and reflectivity.

[0018] In some embodiments, the color information is expressed as color components under multiple color channels.

[0019] In some implementation scenarios, the color-related attributes include color information and transparency. In some implementation scenarios, the color-related attributes include color information and reflectivity. In some implementation scenarios, the color-related attributes include color information, transparency, and reflectivity.

[0020] Step S120: determining, from the three-dimensional point cloud, point sets corresponding to each pixel point in the rendering image at the target viewing angle.

[0021] It is understandable that the image rendering method provided in the embodiment of the present application can be used to obtain a rendering image at a certain viewing angle. The rendering image can be a two-dimensional image, and the rendering process can be understood as determining the pixel value of each pixel in the two-dimensional image. Each pixel can independently determine its pixel value.

[0022] The pixel value of a pixel in the rendering of the target perspective is affected by the 3D points in the imaging range corresponding to the pixel under the perspective. The imaging range may refer to the spatial range that affects the imaging of the pixel. The pixel values ​​of different pixels may be affected by the same or different 3D points. A 3D point may also affect the pixel values ​​of one or more different pixels. First, the corresponding point set may be determined for each pixel, which includes the 3D points that affect the pixel value of the pixel.

[0023] Step S130: Determine the pixel value of each pixel point based on the attribute of the three-dimensional point in the point set corresponding to each pixel point.

[0024] The pixel value of each pixel point is determined respectively, and the pixel value of each pixel point can be determined according to the attribute of the three-dimensional point in the corresponding point set.

[0025] In some embodiments, point sets corresponding to different pixel points may include the same three-dimensional point.

[0026] It is understandable that the pixel value of a pixel point is affected by the attributes of the three-dimensional points within its imaging range. After determining the point set corresponding to the pixel point, the pixel value of the pixel point can be determined using the attributes of all three-dimensional points in the point set.

[0027] A 3D point cloud is used to represent the object to be rendered. At a specific viewing angle, a 3D point within the imaging range of each pixel is selected from the 3D point cloud to obtain the pixel value of the pixel. The 3D points that affect the pixel value of the pixel are determined based on the imaging range of the pixel, so that the pixel value of each pixel can be closer to the object to be rendered, making the rendered image closer to the real situation and improving the rendering effect.

[0028] Furthermore, by adding position attributes, color information, and at least one of transparency and reflectivity to the three-dimensional points, at least one of transparency and reflectivity is used as an attribute to participate in rendering, providing richer representation information of the object to be rendered, making the rendering image closer to the real situation and improving the rendering effect.

[0029] The position attribute includes the spatial position information of the three-dimensional point. The spatial position information may represent the spatial position of the three-dimensional point. For example, the spatial position information may be represented by three-dimensional spatial coordinates.

[0030] In some embodiments, the position attribute may further include control range information of the three-dimensional point in space. The control range information may represent the control range of the three-dimensional point in space.

[0031] It should be noted that the control range of the 3D point in the image rendering method of the present application can be represented within the control range, and the rendering is related to the corresponding 3D point attributes. The control range can be a certain range centered on the spatial position of the point.

[0032] The control range can be used to determine the point set corresponding to the pixel. Specifically, if the imaging light of a pixel passes through the control range of a 3D point, the 3D point can be added to the point set corresponding to the pixel, affecting its pixel value. It is worth noting that at this time, the 3D point itself may not fall on the imaging light, and the properties of the 3D point can still act on the rendering of the pixel.

[0033] It is understandable that the specific shape, size, etc. of the control range can be set according to actual application needs. For example, the control range can be set to a sphere, an ellipsoid, a cube, etc.

[0034] In some implementation scenarios, the control range can be reflected by the posture information of the three-dimensional point coordinate system and the control distance in each coordinate axis direction in the three-dimensional point coordinate system. The representation form of the posture information can be set according to the actual application needs. For example, the posture angle of the three-dimensional point coordinate system relative to the world coordinate system, the rotation matrix, or the vector representation of the coordinate axis of the three-dimensional point coordinate system in the world coordinate system can be used.

[0035] In a specific application scenario, the control range information may include the attitude angle of each coordinate axis of the three-dimensional point coordinate system in the world coordinate system, and the control distance in the direction of each coordinate axis. The control range can be defined as an ellipsoid, and the control distance in the direction of each coordinate axis can be used as the length of the three semi-axes of the ellipsoid.

[0036] See also Figure 2 , Figure 2 It is a schematic diagram of three-dimensional points in an embodiment of the image rendering method of the present application.

[0037] The attributes of a 3D point include position attributes and color-related attributes. The position attributes include the spatial position information of the 3D point and the control range information of the 3D point in space. The color-related attributes include color information, transparency and reflectivity. The color information is the color components under multiple color channels.

[0038] Among them, the spatial location information can be expressed as = (x, y, z). The control range information can include the posture information of the three-dimensional point coordinate system and the control distance in each coordinate axis direction in the three-dimensional point coordinate system, which can be expressed as = (α, β, γ, δ α , δ β , δ γ ). Among them, α, β, γ represent the directions of the three axes of the three-dimensional point, δ α , δ β , δ γ Represents the control distance in the three axis directions.

[0039] Specifically, α, β, and γ can represent vectors of the three axes respectively. The control range of the three-dimensional point can be constructed as an ellipsoid, δ α , δ β , δ γ The three semi-axis lengths of the ellipsoid can be expressed as Figure 2 shown.

[0040] Color information can be expressed as = (r, g, b). Transparency can be expressed as , the reflectivity can be expressed as .

[0041] The meaning of the above attributes: Position represents the coordinates of this point in space; Transparency represents the ability of light to pass through this point, ranging from 0 to 1, and the larger the value, the more transparent the point; Reflectivity represents the ability of light to reflect color after encountering this point, ranging from 0 to 1, and the larger the value, the more the color will be enhanced after the light passes through this point; Color represents the inherent color attribute RGB of this point; Control range information mainly indicates the range shape of the spatial area controlled by the point.

[0042] In some embodiments, determining the pixel value of each pixel point based on the attributes of the three-dimensional point in the point set corresponding to each pixel point may include: accumulating the first sub-component and / or the second sub-component corresponding to each three-dimensional point in the point set to obtain the pixel value of the pixel point.

[0043] It should be noted that the color-related attributes may include at least one of transparency and reflectivity, both of which may affect the pixel value of the pixel point. The specific calculation of the pixel value may be related to the content contained in the color-related attributes.

[0044] Specifically, for each 3D point in the point set, a first subcomponent and / or a second subcomponent can be calculated. The former can be obtained based on transparency and color information, indicating the color affected by transparency, and the latter can be obtained based on reflectivity and color information, indicating the color affected by reflectivity. When the color-related attribute includes transparency, the first subcomponent can be calculated, and when the color-related attribute includes reflectivity, the second subcomponent can be calculated. All components of all 3D points are added together to obtain a pixel value.

[0045] In some implementation scenarios, color information can be represented as color components under multiple color channels, and the final pixel value of the pixel can also be represented as the target color component under each corresponding color channel. For example, three components under the RGB channel. Each color channel is independent, and the target color component under a color channel is calculated by the color component of the three-dimensional point in the point set under the channel.

[0046] Furthermore, transparency and reflectivity may act equally on the color components of each color channel, and the first subcomponent and / or the second subcomponent of each color channel may be calculated respectively to obtain the target color component of the channel.

[0047] In some implementation scenarios, for a color channel, the first subcomponent and / or the second subcomponent of each three-dimensional point in the point set in the color channel is accumulated to obtain a target color component corresponding to the pixel point.

[0048] The first subcomponent of the color channel is the product of the transparency weight of the three-dimensional point and the color component corresponding to the color channel, and the second subcomponent is the product of the reflectivity weight of the three-dimensional point and the color component corresponding to the color channel.

[0049] Furthermore, the transparency weight is inversely proportional to the transparency of the 3D point. The higher the transparency of the 3D point, the lower the weight of its color information, indicating that the point has less influence on the final pixel.

[0050] In some application scenarios, transparency can be expressed as Transparency represents the ability of light to pass through this point, ranging from 0 to 1. The larger the value, the more transparent the point. The transparency weight can be expressed as When the transparency is 1, the first subcomponent of the point is zero, indicating that it has no effect on the pixel value from the perspective of transparency.

[0051] Furthermore, the reflectivity weight is proportional to the reflectivity. Reflectivity represents the ability of light to reflect color after encountering this point, ranging from 0 to 1. The larger the value, the more the color will be enhanced after the light passes through this point.

[0052] In some embodiments, the reflectivity weight can also be associated with the positional relationship between the imaging light of the pixel point and the three-dimensional point. The imaging light of the pixel point can be understood as the light emitted by the camera and related to the imaging of a pixel point. Each pixel point can correspond to its imaging light, and the imaging light can represent the imaging range of the pixel point. The positional relationship between the imaging light of a pixel point and a three-dimensional point is associated with the reflectivity weight between the three-dimensional point and the pixel point.

[0053] Specifically, the distance between the imaging light corresponding to the pixel point and the three-dimensional point can be called the target distance, and the reflectivity weight can be proportional to the target distance.

[0054] That is to say, for the same 3D point, transparency is its fixed attribute, and the transparency weight is only related to the transparency. For any pixel point where the imaging light passes through the control range of the 3D point, the transparency weight is consistent.

[0055] Reflectivity is a fixed property, but the reflectivity weight is not only related to the reflectivity, but also to the position of the imaging light. Different imaging lights passing through the control range of the 3D point have different positions and different distances from the 3D point. Correspondingly, the reflectivity weight can also be different.

[0056] In some embodiments, the reflectivity weight may be the product of the ratio of the target distance to the target square root and the reflectivity of the three-dimensional point, wherein the target square root is the root of the sum of the squares of the control distances in the directions of the coordinate axes in the three-dimensional point coordinate system.

[0057] In some embodiments, the target distance may be determined based on the spatial position information of the imaging light and the three-dimensional point. For example, the minimum distance between the imaging light and the three-dimensional point may be selected.

[0058] In some implementation scenarios, the imaging light passes through the control range of the three-dimensional point, and the portion of the imaging light within the control range is a line segment in the three-dimensional space. Sampling is performed on the line segment to obtain a number of sampling points. The spatial distance between each sampling point and the three-dimensional point is counted respectively, and the target distance can be determined based on the spatial distances corresponding to all sampling points to represent the distance between the imaging light and the three-dimensional point. Exemplarily, the central tendency statistical value of the spatial distances corresponding to all sampling points can be used as the target distance.

[0059] See also Figure 3 , Figure 3 Schematic diagram of sampling points in an embodiment of the image rendering method of the present application.

[0060] Figure 3 A schematic diagram of a control range where an imaging ray passes through a three-dimensional point is shown in FIG. The portion where the imaging ray passes through the control range is uniformly sampled to obtain three sampling points S1, S2, and S3.

[0061] In this embodiment, the uniform sampling of part of the imaging light within the control range of the three-dimensional point is taken as an example. Assuming that the imaging light of a pixel point passes through the control range of K three-dimensional points, the rendering equation of the final color on the imaging light is as follows:

[0062] in, Represents the target color component of any of the three color channels, using It is also the color component of the corresponding channel. represents three sampling points, Represents the average distance from three sampling points to the 3D point. As shown in the above formula, the RGB channels are calculated separately.

[0063] The above scheme provides a differentiable spatial point rendering algorithm.

[0064] In some embodiments, when determining the point set corresponding to each pixel in the rendering image under the target perspective from the three-dimensional point cloud, each pixel can be processed independently. For a pixel, it is determined whether the imaging light of the pixel passes through the control range of the three-dimensional point, and all the three-dimensional points determined to be yes are used to form the point set corresponding to the pixel.

[0065] Specifically, the control range of the 3D point can be determined based on the spatial position information and control range information of the 3D point, so as to determine whether the imaging light of a certain pixel point under a specific viewing angle passes through the control range. All 3D points that the imaging light of the pixel point passes through constitute the corresponding point set.

[0066] In some implementation scenarios, the position information of the imaging light of a certain pixel point at a specific viewing angle is determined, and the position information of the control range of the three-dimensional point is also determined. Therefore, it is possible to solve whether the two intersect to determine whether the imaging light passes through the control range of the three-dimensional point.

[0067] It should be noted that the attributes of the 3D points used in the final rendering can be learned from the first captured image captured by the camera. First, the initial value of the attribute of each 3D point is determined, and then the current first predicted image is rendered using the current attribute, and the attributes of the 3D points are iteratively optimized based on the difference between the first captured image actually captured by the camera and the first predicted image.

[0068] The first captured image may be captured by one or more cameras, and all cameras capture the object to be rendered. Different cameras correspond to different posture information, that is, the captured viewing angles are different.

[0069] The first captured image can be considered as the expected rendering true value, while the first predicted image is the predicted result rendered using the current attributes. The attributes of the 3D points are optimized based on the difference between the two, so that the attributes of the 3D points can more accurately represent the object to be rendered.

[0070] In some cases, the camera's posture may have a certain deviation. The camera's posture information can also be used as an object of adjustment, and there is a certain tolerance for the deviation of the camera's posture.

[0071] Specifically, after one rendering, one of the pose information and the attribute is used as the object of this adjustment: the pose information is fixed and the attribute is adjusted, or the attribute is fixed and the pose information is adjusted.

[0072] It is understandable that the camera's posture information will affect the position information of the imaging light of the pixel point. After the camera's posture information is updated, the three-dimensional points in the point set used by the pixel point will change in the subsequent rendering process.

[0073] See also Figure 4 , Figure 4 This application Figure 1 Schematic diagram of another embodiment of step S110 in FIG. Specifically, this step may include: Step S411: respectively determine the initial values ​​of the attributes of each three-dimensional point to generate a three-dimensional point cloud.

[0074] Each 3D point has multiple attributes, and an initial value is determined for each attribute, thereby generating 3D points with attributes to form a 3D point cloud.

[0075] Step S412: Rendering the current first predicted image of each camera based on the current attributes of the multiple three-dimensional points and the current posture information of at least one camera.

[0076] Among them, one or more first acquired images may be used, and different first acquired images are acquired by cameras in different positions and postures.

[0077] Rendering is performed independently for each camera, and the current posture of the camera and the current attributes of multiple three-dimensional points are used to render a current first predicted image of the camera, which is compared with the first acquired image of the camera.

[0078] The step of rendering the first predicted image of the camera in step S412 can refer to the rendering step in the aforementioned embodiment. Specifically, under the current position information of the camera, the imaging light of each pixel point in the expected first predicted image is determined, and a point set corresponding to the pixel point is selected from all three-dimensional points, and the current pixel value is determined using the current attribute of the three-dimensional point in the point set to obtain the current first predicted image.

[0079] Step S413: Based on the difference between each current first predicted image and the first captured image of the corresponding camera, adjust one of the attributes of the multiple three-dimensional points and the position and posture information of each camera.

[0080] After step S413, the process may return to step S412 to repeat the iterative optimization. The adjustment object of one optimization is one of the attributes of the multiple three-dimensional points and the position and posture information of each camera, and the other is fixed.

[0081] Furthermore, the step of taking the attributes of the three-dimensional point as the adjustment object is repeated to a first execution number, and the step of taking the position information of the camera as the adjustment object is repeated to a second execution number. The first execution number is greater than a preset multiple of the second execution number.

[0082] In some embodiments, the first execution number is greater than ten times the second execution number.

[0083] In a specific application scenario, the camera pose is optimized 10 times every time the attributes of the 3D points are optimized 1000 times, thus achieving fine-tuning of the camera pose.

[0084] In some embodiments, the second execution number may be limited, for example, a threshold may be set for the second execution number. In some embodiments, it may also be determined to stop the optimization of the pose information according to the degree of optimization of the pose information of the camera. For example, if the difference between the pose information after optimization and before optimization is less than a certain threshold, the optimization may be stopped. The above conditions may also be applied together, and the optimization may be stopped if any one of them is met.

[0085] The initial value of the spatial position information may be obtained based on image extraction, and the initial values ​​of other attributes may be preset values ​​or randomly generated values. Specifically, the initial value of transparency may be a preset value, for example, it may be 0. The initial value of reflectivity may be a randomly generated value. The posture information of the three-dimensional point coordinate system may be a preset value, for example, it may indicate that the direction is consistent with the world coordinate system. The control distance may be a preset value, for example, δ α , δ β , δ γ It can be preset to 1. The color information can be a randomly generated value.

[0086] In some embodiments, the initial value of the spatial position information of the three-dimensional point can be extracted from the second acquired images acquired by multiple cameras. Specifically, point cloud extraction is performed based on the second acquired images of the multiple cameras to obtain the spatial position information of multiple initial three-dimensional points; the multiple initial three-dimensional points are clustered to obtain the initial value of the spatial position information of the multiple three-dimensional points.

[0087] The second acquired image is acquired from the object to be rendered. There are multiple second acquired images. The second acquired image and the first acquired image may be the same or different.

[0088] In a specific application scenario, a point cloud extraction algorithm can be used to obtain the spatial position information of the initial three-dimensional points. For example, an algorithm such as Patchmatch can be used. The input of the algorithm includes the camera poses of multiple cameras and the corresponding second acquired images, and the output is the spatial position information of multiple initial three-dimensional points.

[0089] In a specific application scenario, the postures of N cameras and the corresponding second acquired images are input, and the spatial position information of M initial three-dimensional points is output. The M initial three-dimensional points are divided into m categories through clustering algorithms such as K-means, and the centroids of the m categories are calculated as the m three-dimensional points, thereby obtaining the initial values ​​of the spatial position information of the m three-dimensional points.

[0090] It is understandable that the target perspective can be a new perspective that is different from the camera that captures the image. Before rendering the new perspective, the number of 3D points contained in the point set corresponding to each pixel can be determined. If the number of 3D points in the point set corresponding to a pixel is less than a preset number threshold, it is necessary to expand the number of 3D points based on the existing 3D points. Specifically, at least one 3D point in the point set is used as a 3D point to be split, and multiple new 3D points are derived based on a 3D point to be split.

[0091] In some embodiments, for a 3D point to be split, multiple spatial points are selected within the control range of the 3D point to be split as new 3D points, and the attributes of the new 3D point are determined based on the attributes of the 3D point to be split.

[0092] The number of new 3D points derived from a 3D point to be split can be set according to actual application needs. The position within the control range at which the new 3D point is selected can also be set according to actual application needs.

[0093] Furthermore, the spatial position information of the new 3D point can be determined according to actual conditions, the control range information includes the posture information and the control distance, and the posture information of the new 3D point is the same as the posture information of the 3D point to be split. The control distance of the new 3D point is less than the control distance of the 3D point to be split. The new color-related attributes are the same as the color-related attributes of the 3D point to be split.

[0094] In some embodiments, after deriving multiple new 3D points, the second predicted image of each camera may be rendered based on the attribute of the current 3D point and the pose information of at least one camera, and the attribute of the current 3D point may be adjusted based on the difference between each second predicted image and the third captured image of the corresponding camera. The above steps may be performed several times.

[0095] The third acquired image, the second acquired image and the first acquired image may be the same, and are acquired from cameras with the same posture and for the object to be rendered.

[0096] See also Figure 5 , Figure 5 It is a schematic diagram of another embodiment of the image rendering method of the present application.

[0097] In this embodiment, a point cloud generation module with attributes and a new perspective rendering module are provided to implement an image rendering method.

[0098] The input of the point cloud generation module with attributes may include a captured image with camera pose information, and the output is a three-dimensional point cloud with attributes.

[0099] The input of the new perspective rendering module includes a three-dimensional point cloud with attributes and the pose information of the target perspective, and the output is a rendering of the target perspective.

[0100] It can be understood that the pose information of the camera is different from the pose information of the target perspective, and the target perspective can be considered as a new perspective.

[0101] The point cloud generation module with attributes first generates a three-dimensional point cloud with initial attributes. Specifically, it uses the captured images of multiple cameras to extract point clouds, extracts initial three-dimensional points and clusters them to obtain a three-dimensional point cloud with initial values ​​of spatial position information, and then adds initial values ​​to other attributes of the three-dimensional points to obtain a three-dimensional point cloud with initial attributes.

[0102] Then, the acquired images of at least one camera are used to iteratively optimize the attributes and camera pose information to obtain the final three-dimensional point cloud with attributes.

[0103] Specifically, by emitting light from the center of the camera, the corresponding rendering of the camera is obtained , the real captured image corresponding to the known camera is represented as I, so an optimization objective is constructed: ,Through this optimization objective, the position, direction, transparency, reflectivity, and color of each point in the space can be obtained.

[0104] In addition, fix the inherent properties of the points obtained above, optimize the camera parameters R|t, and the optimization target is The above two optimization steps usually iterate 1000 attributes and 10 camera parameters.

[0105] The attributes and camera parameters are iterated cyclically until the set iteration stop condition is reached. The attributes of these points are recorded and saved to obtain the final 3D point cloud with attributes for use by the new perspective rendering module.

[0106] The new perspective rendering module first calculates the number of point clouds N that the light emitted by each pixel passes through based on the pose information of the target perspective. If N is less than 100, the points need to be split.

[0107] Assume that the attribute of the point to be split is position = (x, y, z), control range information = (α, β, γ, δ α , δ β , δ γ ),transparency , reflectivity ,color = (r, g, b). Three new three-dimensional points are derived from one point. Specifically, the midpoint of the point to be split on the three three-dimensional point coordinate axes is used as the new three-dimensional point. The midpoint refers to the midpoint between the point to be split and the point at the control distance on the three-dimensional point coordinate axis. For example, on the axis corresponding to α, The point at which the 3D point is split is used as a new 3D point, and the spatial position information of the new 3D point is re-determined accordingly. The color-related attributes of the new 3D point are consistent with those of the point to be split. The posture information of the coordinate system of the new 3D point is consistent with that of the point to be split, and the control distance in each axis direction becomes half of the original.

[0108] Then the positions of the three points is the midpoint of the three axes, facing ,transparency , reflectivity ,color = (r, g, b).

[0109] After splitting, the attributes can also be optimized. For details, please refer to the attribute optimization steps in the previous module and use the existing acquired images to optimize the attributes of the split 3D point cloud. The iterative optimization steps here can be much less than the iterative optimization steps in the previous module. After the splitting is completed, the new 3D point cloud is used for rendering to obtain a rendering image of the target perspective.

[0110] The above scheme designs the attributes of the point cloud to fit the rendering results of the scene. Each point is assigned position, direction, transparency, reflectivity and color attributes, which can better represent the spatial color information and take into account the changes after the light passes through each point, thus meeting the requirements for the representation of some translucent, reflective and other scenes.

[0111] The differentiable spatial point rendering algorithm is used to conveniently optimize point attributes. The view is rendered under each camera pose according to the proposed rendering algorithm, and the consistency of the rendered view and the original view is ensured through optimization. The attribute value of each point is obtained to accurately represent the information of the object to be rendered. On this basis, the spatial color information can be well characterized by using three-dimensional point rendering. And this information is used to optimize the camera pose in turn, reducing the impact of incorrect camera pose estimation on the final result.

[0112] Since this solution explicitly records the position of each point, it is convenient to re-edit the scene. Each point can be edited, and the device can respond to the user's editing instructions to delete, add, and change the 3D points and their attributes.

[0113] Adaptively adjust the number of points to ensure the precision and stability of rendering for new perspectives. Render directly after quickly obtaining points with attributes, and use the logic of splitting the range of sparse points during the rendering process to reduce the time for calculating attribute points and improve the precision of the final rendering result.

[0114] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of the image rendering device of the present application.

[0115] In this embodiment, the image rendering device 60 includes an acquisition module 61, a point selection module 62, and a pixel value determination module 63. The acquisition module 61 is used to acquire a three-dimensional point cloud, which includes a plurality of three-dimensional points. The attributes of the three-dimensional points include position attributes and color-related attributes. The color-related attributes include color information and at least one of transparency and reflectivity. The point selection module 62 is used to determine the point set corresponding to each pixel point in the rendering image under the target perspective from the three-dimensional point cloud, and the pixel value determination module 63 is used to determine the pixel value of each pixel point based on the attributes of the three-dimensional points in the point set corresponding to each pixel point.

[0116] See also Figure 7 , Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application.

[0117] The electronic device 70 includes a memory 71 and a processor 72, and the processor 72 is used to execute the program instructions stored in the memory 71 to implement the steps in any of the above-mentioned image rendering method embodiments. In a specific implementation scenario, the electronic device 70 may include, but is not limited to: computer equipment, electrical equipment, microcomputers, desktop computers, servers, and in addition, the electronic device 70 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0118] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above-mentioned image rendering method embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0119] See also Figure 8 , Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0120] The computer-readable storage medium 80 provided in this embodiment stores program instructions 81 that can be executed by a processor. When the program instructions 81 are executed by the processor, they are used to implement the steps in any of the above-mentioned image rendering method embodiments.

[0121] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another subsystem, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0123] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. An image rendering method, characterized in that: The method comprises: Acquire a three-dimensional point cloud, the three-dimensional point cloud comprising a plurality of three-dimensional points, the attributes of the three-dimensional points comprising position attributes and color-related attributes, the color-related attributes comprising color information and at least one of transparency and reflectivity; Determine, from the three-dimensional point cloud, point sets corresponding to each pixel point in the rendering image under the target perspective; Based on the attributes of the three-dimensional points in the point set corresponding to each of the pixel points, a pixel value of each of the pixel points is determined.

2. The method according to claim 1, characterized in that The color information includes color components under multiple color channels, and / or the position attribute includes spatial position information of the three-dimensional point and control range information of the three-dimensional point in space.

3. The method according to claim 2, characterized in that The control range information of the three-dimensional point in space includes the posture information of the three-dimensional point coordinate system and the control distance in each coordinate axis direction of the three-dimensional point coordinate system.

4. The method according to claim 2, characterized in that: The determining the pixel value of each pixel point based on the attribute of the three-dimensional point in the point set corresponding to each pixel point comprises: For each of the color channels, accumulating the first subcomponent and / or the second subcomponent of each of the three-dimensional points in the point set in the color channel to obtain a target color component corresponding to the pixel point in the color channel; Among them, the first subcomponent of the color channel is the product of the transparency weight of the three-dimensional point and the color component corresponding to the color channel, and the second subcomponent of the color channel is the product of the reflectivity weight of the three-dimensional point and the color component corresponding to the color channel; the transparency weight is inversely proportional to the transparency; the reflectivity weight is proportional to the reflectivity and is proportional to the target distance, and the target distance represents the distance between the imaging light corresponding to the pixel point and the three-dimensional point.

5. The method according to claim 4, characterized in that The transparency weight is a preset value minus the transparency of the three-dimensional point; And / or, the reflectivity weight is the product of the ratio of the target distance to the target square root and the reflectivity of the three-dimensional point, the control range information includes the control distance in each coordinate axis direction in the three-dimensional point coordinate system; the target square root is the root of the sum of the squares of the control distances in each coordinate axis direction in the three-dimensional point coordinate system; The target distance is a centralized tendency statistical value of the spatial distances corresponding to a number of sampling points, wherein the sampling points are obtained by sampling the portion of the imaging light within the control range of the three-dimensional point, and the spatial distance corresponding to the sampling points is the distance between the sampling points and the three-dimensional point.

6. The method according to claim 2, characterized in that The step of determining from the three-dimensional point cloud the point sets corresponding to the pixels in the rendering image at the target viewing angle comprises: For each of the pixel points, based on the spatial position information of the three-dimensional point and the control range information, determining whether the imaging light of the pixel point under the target viewing angle passes through the control range of the three-dimensional point; All the three-dimensional points determined to be yes are used to form a point set corresponding to the pixel point.

7. The method according to claim 1, characterized in that The obtaining of the three-dimensional point cloud comprises: Determining the initial value of the attribute of each of the three-dimensional points respectively to generate the three-dimensional point cloud; Rendering a current first predicted image of each camera based on current attributes of the plurality of three-dimensional points and current pose information of at least one camera; Adjusting one of the attributes of the plurality of three-dimensional points and the position information of each camera based on a difference between each current first predicted image and a corresponding first captured image of the camera; Return to the step of rendering the current first predicted image of each camera based on the current attributes of the multiple three-dimensional points and the current pose information of at least one camera.

8. The method according to claim 7, characterized in that The adjusting one of the attributes of the plurality of three-dimensional points and the pose information of each camera based on the difference between the current first predicted images and the corresponding first captured images of the camera comprises any one of the following steps: Fix the position information of the camera, and adjust the attributes of the plurality of three-dimensional points based on the difference between each current first predicted image and the corresponding first captured image of the camera, and repeat this step to a first execution number; Fix the attributes of the three-dimensional points, and adjust the posture information of each camera based on the difference between each first predicted image and the corresponding first captured image of the camera, and repeat this step to a second execution number; Wherein, the first execution number is greater than a preset multiple of the second execution number.

9. The method according to claim 7, characterized in that: The position attribute includes spatial position information of the three-dimensional point; and the step of respectively determining initial values ​​of the attributes of each of the three-dimensional points to generate the three-dimensional point cloud includes: Extracting point clouds based on the second acquired images of the multiple cameras to obtain spatial position information of multiple initial three-dimensional points; The multiple initial three-dimensional points are clustered to obtain initial values ​​of spatial position information of the multiple three-dimensional points.

10. The method according to claim 1, characterized in that Before determining the pixel value of each pixel point based on the attribute of the three-dimensional point in the point set corresponding to each pixel point, the method further includes: In response to the number of the three-dimensional points included in the point set being less than a preset number threshold, at least one of the three-dimensional points in the point set is used as a three-dimensional point to be split, and a plurality of new three-dimensional points are derived based on the three-dimensional point to be split.

11. The method according to claim 10, characterized in that The position attribute includes the spatial position information of the 3D point and the control range information of the 3D point in space; the deriving of a plurality of new 3D points based on the 3D point to be split includes: A plurality of spatial points are selected within the control range of the three-dimensional point to be split as new three-dimensional points, and the attributes of the new three-dimensional points are determined based on the attributes of the three-dimensional point to be split; wherein the color-related attributes of the new three-dimensional points are the same as the color-related attributes of the three-dimensional points to be split.

12. The method according to claim 10, characterized in that After taking at least one of the three-dimensional points in the point set as the three-dimensional point to be split, and deriving a plurality of new three-dimensional points based on the three-dimensional point to be split, the method further comprises: Rendering a second predicted image of each camera based on the attributes of the current three-dimensional point and the pose information of at least one camera; Based on the difference between each of the second predicted images and the corresponding third acquired image of the camera, the attribute of the current three-dimensional point is adjusted.

13. An image rendering device, characterized in that: The device comprises: An acquisition module, configured to acquire a three-dimensional point cloud, wherein the three-dimensional point cloud includes a plurality of three-dimensional points, and the attributes of the three-dimensional points include position attributes and color-related attributes, wherein the color-related attributes include color information and at least one of transparency and reflectivity; A point selection module, used to determine, from the three-dimensional point cloud, point sets corresponding to each pixel point in the rendering image under the target perspective; The pixel value determination module is used to determine the pixel value of each pixel point based on the attribute of the three-dimensional point in the point set corresponding to each pixel point.

14. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images