Image rendering method, apparatus and device

Abstract

The application provides an image rendering method, device and equipment, which can be applied to the technical field of image processing. The method comprises the following steps: performing feature fusion on a plurality of initial images to obtain image fusion features, wherein the plurality of initial images represent a target object based on different initial shooting angles; generating a voxel grid representing the target object according to the image fusion features, wherein a center point of the voxel grid is used as an anchor point feature; performing attention fusion on a specified pose of the plurality of initial images, the image fusion features and the anchor point feature based on an attention mechanism to obtain a target feature; performing image attribute detection on the target feature to obtain a target attribute; and performing image rendering by using the target attribute to obtain a target image representing the target object based on a target shooting angle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly to an image rendering method, device and equipment. BACKGROUND

[0002] New view synthesis technology infers the three-dimensional structure and shape of objects in a scene by using scene information observed from multiple viewpoints, and then generates a new view.

[0003] In the related art, a method based on neural rendering is used for new view synthesis, but in this process, point cloud features cannot be fully utilized for rendering, resulting in that the rendered image is not real enough in details and lacks obvious feature performance. SUMMARY

[0004] In view of the above problems, the present application provides an image rendering method, device and equipment.

[0005] According to a first aspect of the present application, an image rendering method is provided, comprising: performing feature fusion on a plurality of initial images to obtain image fusion features, the plurality of initial images representing a target object based on different initial shooting angles; generating a voxel grid representing the target object according to the image fusion features, wherein the center point of the voxel grid is used as an anchor point feature; performing attention fusion on a specified pose of the plurality of initial images, the image fusion features and the anchor point feature based on an attention mechanism to obtain a target feature; performing image attribute detection on the target feature to obtain a target attribute; and performing image rendering using the target attribute to obtain a target image representing the target object based on a target shooting angle.

[0006] According to an embodiment of the present application, the attention fusion on the specified pose of the plurality of initial images, the image fusion features and the anchor point feature based on the attention mechanism to obtain the target feature comprises: performing attention fusion on the specified pose and the image fusion features to obtain a first fusion feature; performing attention fusion on the specified pose and the anchor point feature to obtain a second fusion feature; and fusing the first fusion feature and the second fusion feature to obtain the target feature.

[0007] According to an embodiment of the present application, the attention fusion on the specified pose and the image fusion features to obtain the first fusion feature comprises: projecting the target object according to the specified pose to obtain a projection image; determining a plurality of initial associated pixels corresponding to a target projection pixel of the projection image from the plurality of initial images; determining an associated pixel similarity vector related to the target projection pixel according to a similarity value between the plurality of initial associated pixels; and performing weighted summation on the image fusion features according to the associated pixel similarity vector corresponding to each of the plurality of target projection pixels to determine the first fusion feature.

[0008] According to an embodiment of the present application, the attention fusion is performed on the specified pose and the anchor point feature to obtain a second fusion feature, including: performing weighted summation on the anchor point feature according to the correlation pixel similarity vectors corresponding to the plurality of target projection pixels respectively, to determine the second fusion feature.

[0009] According to an embodiment of the present application, the feature fusion is performed on the plurality of initial images to obtain an image fusion feature, including: performing feature extraction on the plurality of initial images to obtain color features and view distance deviation features; and performing feature fusion on the color features and the view distance deviation features to obtain the image fusion feature.

[0010] According to an embodiment of the present application, the image attribute detection is performed on the target feature to obtain a target attribute, including: determining a relative angle distance between a camera and an anchor point feature according to a specified pose; and performing image attribute detection on the relative angle distance and the target feature by using a multi-layer perception model to obtain the target attribute.

[0011] According to an embodiment of the present application, the multi-layer perception model is optimized based on the following operations: performing feature fusion on a plurality of initial sample images to obtain a sample image fusion feature, the plurality of initial sample images representing a target sample object based on different initial sample shooting angles; generating a sample voxel grid representing the target sample object according to the sample image fusion feature, wherein a center point of the sample voxel grid is used as a sample anchor point feature; performing attention fusion on a specified pose of the plurality of initial sample images, the sample image fusion feature and the sample anchor point feature based on an attention mechanism to obtain a target sample feature; performing image attribute detection on the target sample feature by using the multi-layer perception model to obtain a target sample attribute; performing image rendering by using the target sample attribute to obtain a target sample image representing the target sample object based on a target sample shooting angle; performing loss calculation on the target sample image and a label sample image related to the plurality of initial sample images to obtain a loss calculation result; and optimizing the multi-layer perception model according to the loss calculation result to obtain an optimized multi-layer perception model.

[0012] According to an embodiment of the present application, the loss calculation is performed on the target sample image and the label sample image related to the plurality of initial sample images to obtain a loss calculation result, including: determining a volume regularization loss according to a product of a plurality of target sample parameters associated with the target sample feature; determining a pixel loss according to a pixel color difference between the target sample image and the label sample image; determining a similarity loss according to a similarity between the target sample image and the label sample image; and determining the loss calculation result according to the volume regularization loss, the pixel loss and the similarity loss.

[0013] The second aspect of the present application provides an image rendering device, comprising: a first fusion module configured to perform feature fusion on a plurality of initial images to obtain image fusion features, the plurality of initial images representing a target object based on different initial shooting angles; a generation module configured to generate a voxel grid representing the target object according to the image fusion features, wherein a center point of the voxel grid is taken as an anchor point feature; a second fusion module configured to perform attention fusion on a specified pose of the plurality of initial images, the image fusion features and the anchor point feature based on an attention mechanism to obtain a target feature; a detection module configured to perform image attribute detection on the target feature to obtain a target attribute; and a rendering module configured to perform image rendering using the target attribute to obtain a target image representing the target object based on a target shooting angle.

[0014] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory configured to store one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0015] The fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method.

[0016] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method.

[0017] According to the embodiments of the present application, feature fusion is performed on a plurality of initial images to obtain image fusion features, the plurality of initial images representing a target object based on different initial shooting angles; a voxel grid representing the target object is generated according to the image fusion features, wherein a center point of the voxel grid is taken as an anchor point feature; attention fusion is performed on a specified pose of the plurality of initial images, the image fusion features and the anchor point feature based on an attention mechanism to obtain a target feature; image attribute detection is performed on the target feature to obtain a target attribute; and image rendering is performed using the target attribute to obtain a target image representing the target object based on a target shooting angle. Feature extraction is performed on the initial images to obtain the image fusion features, and image features related to point cloud features are extracted; attention fusion is performed on the specified pose of the plurality of initial images, the image fusion features and the anchor point feature, thereby enhancing the detail performance of the target image and improving the recognition accuracy of the target image. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above content and other purposes, features and advantages of the present application will be more apparent from the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0019] Figure 1An application scenario diagram of the image rendering method, device, equipment, medium and program product according to the embodiments of the present application is shown.

[0020] Figure 2 A flow chart of the image rendering method according to the embodiments of the present application is shown.

[0021] Figure 3 A schematic diagram of the image rendering method according to the embodiments of the present application is shown.

[0022] Figure 4 A structural block diagram of the image rendering device according to the embodiments of the present application is shown; and

[0023] Figure 5 A block diagram of the electronic equipment suitable for implementing the image rendering method according to the embodiments of the present application is shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present application, and is not intended to limit the scope of the present application. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it would be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known structures and techniques have not been described in detail in order to avoid obscuring aspects of the present application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, shall not be taken to exclude

[0026] All terms used herein including technical and scientific terms have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined herein. It should be noted that the terms used herein are merely specific examples for the described present application and should not be interpreted as limiting unless otherwise defined herein.

[0027] In the case where expressions similar to "at least one of A, B, and C, and the like" are used, it is generally to be interpreted as including one or more of the items enumerated in the expression (e.g., "a system having at least one of A, B, and C" should include, but not be limited to, a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0028] Embodiments of the present application provide an image rendering method, which performs feature fusion on a plurality of initial images to obtain image fusion features, the plurality of initial images representing a target object based on different initial shooting perspectives; generates a voxel grid representing the target object according to the image fusion features, wherein a center point of the voxel grid is taken as an anchor point feature; performs attention fusion on a specified pose of the plurality of initial images, the image fusion features, and the anchor point feature based on an attention mechanism to obtain a target feature; performs image attribute detection on the target feature to obtain a target attribute; and performs image rendering using the target attribute to obtain a target image representing the target object based on a target shooting perspective.

[0029] Figure 1 An application scenario diagram of the image rendering method, apparatus, device, medium, and program product according to embodiments of the present application is shown.

[0030] As shown in Figure 1 , the application scenario 100 according to this embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0031] A user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc.

[0033] The server 105 can be a server providing various services, such as a background management server supporting a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data generated according to user requests, etc.) to the terminal device.

[0034] It should be noted that the image rendering method provided by the embodiment of the present application can be generally executed by the server 105. Correspondingly, the image rendering apparatus provided by the embodiment of the present application can be generally arranged in the server 105. The image rendering method provided by the embodiment of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the image rendering apparatus provided by the embodiment of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0035] It should be understood that the number of terminal devices, networks and servers in the above-mentioned scenario is only an example. According to the implementation needs, there can be any number of terminal devices, networks and servers. Figure 1

[0036] The image rendering method of the embodiment of the present application will be described in detail based on the scenario described above. Figure 1

[0037] Figure 2 A flowchart of the image rendering method according to the embodiment of the present application is shown.

[0038] As shown in Figure 2 , the image rendering method of this embodiment includes operations S210-S250.

[0039] In operation S210, feature fusion is performed on a plurality of initial images to obtain image fusion features. The plurality of initial images represent a target object based on different initial shooting angles.

[0040] According to the embodiment of the present application, the plurality of initial images can be images obtained by shooting the target object at different initial shooting angles for a specified target object, wherein the plurality of initial images can also include a plurality of images of different image scales for the same initial shooting angle.

[0041] According to the embodiment of the present application, the feature fusion on the plurality of initial images can be the fusion of a plurality of different image features of one of the initial images, so that the detailed information of the image, such as color information, visual deviation information, etc., can be fully considered in the subsequent rendering process.

[0042] In operation S220, a voxel grid representing the target object is generated according to the image fusion features.

[0043] ​​According to an embodiment of the present application, the center point of the voxel grid is taken as an anchor point feature. The voxel grid is a kind of three-dimensional spatial data structure composed of uniformly distributed cubic units (i.e. voxels), which can represent the target object corresponding to multiple initial images.

[0044] According to an embodiment of the present application, the image fusion features are processed based on a motion recovery structure operation (Structure from Motion, SfM) to generate a sparse point cloud, the obtained point cloud scene is divided into a sparse voxel grid V, and each voxel center is defined as an anchor point v of the voxel, wherein the definition formula of the sparse voxel grid V is shown in formula (1),

[0045] (1).

[0046] wherein, is the voxel size, represents the coordinate position of any point cloud in the point cloud scene, after the 3D space is divided into a series of cubic grids, the center position of each voxel is specified as the anchor point feature of the voxel, and is represented by a variable, which will be used to extract and summarize the feature information of the surrounding point cloud.

[0047] In operation S230, the specified poses of the multiple initial images, the image fusion features and the anchor point features are attentionally fused based on an attention mechanism to obtain target features.

[0048] According to an embodiment of the present application, the specified pose of the multiple initial images can be one of the multiple poses corresponding to the multiple initial images.

[0049] According to an embodiment of the present application, the specified pose of the multiple initial images and the image fusion features are attentionally fused based on an image cross-attention mechanism; the specified pose of the multiple initial images and the anchor point features are attentionally fused based on a 3D structure cross-attention mechanism; and the target features are obtained based on the two kinds of attention fusion mechanisms.

[0050] According to an embodiment of the present application, the specified pose is processed in the self-attention mechanism and then transmitted to the image cross-attention mechanism and the 3D structure cross-attention mechanism respectively to realize matching operation. The core role of the two cross-attention mechanisms is to map the specified pose, the image fusion features and the anchor point features to two different spatial domains.

[0051] In operation S240, image attribute detection is performed on the target features to obtain target attributes.

[0052] According to an embodiment of the present application, image attribute detection is performed on the target features, and a series of linear transformations and activation functions are used to decode the image attributes from the target features, wherein the image attributes can include position opacity covariance related quaternion scaling and color .

[0053] In operation S250, image rendering is performed using the target attribute to obtain a target image representing the target object based on a target shooting angle.

[0054] According to an embodiment of the present application, image rendering is performed using the target attribute, and the generated target image can enhance the detail performance of the image, clearly display the color texture information of the image, and the edges and textures of the target object compared with the initial image.

[0055] According to an embodiment of the present application, feature fusion is performed on a plurality of initial images to obtain image fusion features, and the plurality of initial images represent the target object based on different initial shooting angles; a voxel grid representing the target object is generated according to the image fusion features, wherein the center point of the voxel grid is used as an anchor point feature; attention fusion is performed on the specified pose of the plurality of initial images, the image fusion features, and the anchor point feature based on an attention mechanism to obtain a target feature; image attribute detection is performed on the target feature to obtain a target attribute; and image rendering is performed using the target attribute to obtain a target image representing the target object based on a target shooting angle. Feature extraction is performed on the initial images to obtain image fusion features, and image features related to point cloud features are extracted; attention fusion is performed on the specified pose of the initial images, the image fusion features, and the anchor point feature, which enhances the detail performance of the target image and improves the recognition accuracy of the target image.

[0056] According to an embodiment of the present application, attention fusion is performed on the specified pose of the plurality of initial images, the image fusion features, and the anchor point feature based on an attention mechanism to obtain a target feature, including: attention fusion is performed on the specified pose and the image fusion features to obtain a first fusion feature; attention fusion is performed on the specified pose and the anchor point feature to obtain a second fusion feature; and the first fusion feature and the second fusion feature are fused to obtain the target feature.

[0057] According to an embodiment of the present application, attention fusion is performed on the specified pose and the image fusion features, which can be based on an image cross-attention mechanism, and the specified pose of the plurality of initial images and the image fusion features are fused based on the image cross-attention mechanism, wherein the image cross-attention mechanism establishes a similar association between different images based on the specified pose, enhances the understanding and fusion of image features by the model, and obtains the first fusion feature.

[0058] According to an embodiment of the present application, the attention fusion of the specified pose and the anchor point feature can be based on a 3D structure cross attention mechanism, which is an attention mechanism combined with 3D data (i.e., anchor point feature), further improving the understanding ability of the model to 3D structure, capturing local details and global structure, to obtain the second fusion feature.

[0059] According to an embodiment of the present application, the fusion of the first fusion feature and the second fusion feature obtains the target feature, which not only ensures the effective use of the specified pose, but also realizes the efficient matching of the image feature and the 3D anchor point feature through the cross attention mechanism.

[0060] According to an embodiment of the present application, the attention fusion of the specified pose and the image fusion feature obtains the first fusion feature, including: projecting the target object according to the specified pose to obtain a projection image; determining a plurality of initial associated pixels corresponding to a target projection pixel of the projection image from a plurality of initial images; determining an associated pixel similarity vector related to the target projection pixel according to the similarity values between the plurality of initial associated pixels; and weighting and summing the image fusion feature according to the associated pixel similarity vectors corresponding to the plurality of target projection pixels to determine the first fusion feature.

[0061] According to an embodiment of the present application, the target object is projected according to the specified pose, wherein the specified pose represents the position and direction of the target object in the initial image in the three-dimensional space, and the position and direction in the three-dimensional space are mapped to the two-dimensional plane through the specified projection mode (such as orthogonal projection or perspective projection, etc.) to obtain the projection image.

[0062] According to an embodiment of the present application, in the obtained projection image, different initial images corresponding to the same position can have the same pixel information, and a plurality of initial images having the same pixel information (i.e., target projection pixels) are determined, and the pixel information of each of the plurality of initial images is determined as a plurality of initial associated pixels.

[0063] According to an embodiment of the present application, the similarity values between the plurality of initial associated pixels are calculated based on a similarity distance calculation mode, and the obtained similarity values are determined as the associated pixel similarity vector related to the target projection pixel.

[0064] According to an embodiment of the present application, the image fusion feature is weighted and summed according to the associated pixel similarity vectors corresponding to the plurality of target projection pixels, so as to obtain a cross attention representation fused with the information of the two input sequences, effectively integrating the information of different sources while standardizing the cross attention representation, to ensure obtaining a reasonable and final weight distribution.

[0065] According to an embodiment of the present application, the specified pose and anchor point features are fused by attention to obtain a second fusion feature, including: weighting and summing the anchor point features according to the respective associated pixel similarity vectors of the plurality of target projection pixels to determine the second fusion feature.

[0066] According to an embodiment of the present application, similar to the above image cross-attention mechanism, in the 3D structure cross-attention mechanism, the respective associated pixel similarity vectors of the plurality of target projection pixels are weighted and summed to obtain the second fusion feature.

[0067] According to an embodiment of the present application, the first fusion feature and the second fusion feature are fused to obtain a target feature, which can be as shown in formula (2).

[0068] (2).

[0069] wherein, represents the target feature, represents the first fusion feature, represents the second fusion feature. , , , , , are the query, key and value of the 2D and 3D embedding respectively. is a projection matrix corresponding to the projection image, represents a coefficient for balancing the image condition and the 3D structure condition feature. represents the image fusion feature, represents the embedding of the image fusion feature, represents the embedding of the anchor point feature.

[0070] According to an embodiment of the present application, in terms of optimizing the pose, the anchor point feature refers to the key points selected in the point cloud, and their positions and orientations in the three-dimensional space have an important influence on the quality of the new view image. In order to obtain more accurate anchor point positions, a cross-attention mechanism operation is adopted to input the anchor point feature, the specified pose and the image fusion feature.

[0071] According to an embodiment of the present application, the feature fusion is performed on the plurality of initial images to obtain an image fusion feature, including: performing feature extraction on the plurality of initial images to obtain color features and view distance deviation features; and performing feature fusion on the color features and the view distance deviation features to obtain the image fusion feature.

[0072] According to an embodiment of the present application, the image fusion feature only considers the color information of the image itself , and also considers the view distance deviation The range of consideration is included. The perspective deviation refers to the difference in distance perception when an observer views an image due to the difference in viewing angle. By adding the perspective deviation to the image-based features, the rendering result can be more in line with human visual habits, thereby improving the realism of the rendering effect. Specifically, the image fusion features can be obtained as shown in formulas (3) and (4).

[0073] (3).

[0074] (4).

[0075] wherein, is a function of extracting features from an image, is a function of fusing color information and perspective deviation of an image. The image fusion features are represented as wherein is a combination of color information , perspective deviation .

[0076] According to the embodiment of the present application, by analyzing the texture information in the initial image, image features related to the point cloud features are extracted. These image features include color, brightness, texture, etc., which play an important role in describing the visual properties of the point cloud. In order to effectively combine the image features with the point cloud features, the above-mentioned feature fusion technology is adopted to obtain the image fusion features. This technology fuses the point cloud features with the image features to generate a feature representation with rich information. In this way, the point cloud features can better reflect the details and complexity of the scene in the rendering process.

[0077] According to the embodiment of the present application, the target feature is subjected to image attribute detection to obtain the target attribute, including: determining the relative angle distance between the camera and the anchor point feature according to the specified pose; using a multi-layer perception model to perform image attribute detection on the relative angle distance and the target feature to obtain the target attribute.

[0078] According to the embodiment of the present application, the multi-layer perception model decodes the target attribute from the target feature using a series of linear transformations and activation functions. Each transformation corresponds to an attribute of the neural Gaussian. Specifically, the target attribute generated from the target feature can be calculated by a specific formula (5).

[0079] (5).

[0080] wherein, represents the target attribute, represents an activation function, represents the relative angle distance between the camera and the i-th anchor point feature, represents the target feature.

[0081] According to an embodiment of the present application, the position, covariance related quaternion, scale and color of the neural Gaussian, i.e. the target attribute, are derived based on the above formula. The core idea of this process is to decompose the complex attribute into simple elements, and then learn the relationship between them through a neural network. It is worth noting that the prediction of the target attribute is dynamic, and this dynamic activation avoids unnecessary calculations, greatly improving the efficiency of the algorithm.

[0082] According to an embodiment of the present application, the multi-layer perception model is optimized based on the following operations: feature fusion is performed on a plurality of initial sample images to obtain sample image fusion features, the plurality of initial sample images representing a target sample object based on different initial sample shooting angles; a sample voxel grid representing the target sample object is generated according to the sample image fusion features, wherein the center point of the sample voxel grid is used as a sample anchor point feature; target sample features are obtained by performing attention fusion on a specified pose of the plurality of initial sample images, the sample image fusion features and the sample anchor point feature based on an attention mechanism; image attribute detection is performed on the target sample features using the multi-layer perception model to obtain target sample attributes; image rendering is performed using the target sample attributes to obtain a target sample image representing the target sample object based on a target sample shooting angle; loss calculation is performed on the target sample image and label sample images related to the plurality of initial sample images to obtain a loss calculation result; and the multi-layer perception model is optimized according to the loss calculation result to obtain an optimized multi-layer perception model.

[0083] According to an embodiment of the present application, in the training process of the multi-layer perception model, after the above operations are performed on the plurality of initial sample images to obtain the target sample image, loss calculation is performed on the target sample image and label sample images related to the plurality of initial sample images, wherein the label sample images related to the plurality of initial sample images can be images obtained by rendering processing on the plurality of initial sample images based on a related model or tool.

[0084] According to an embodiment of the present application, the loss calculation on the target sample image and the label sample images related to the plurality of initial sample images to obtain the loss calculation result includes: determining a volume regularization loss according to the product of a plurality of target sample parameters associated with the target sample features; determining a pixel loss according to the pixel color difference between the target sample image and the label sample image; determining a similarity loss according to the similarity between the target sample image and the label image; and determining the loss calculation result according to the volume regularization loss, the pixel loss and the similarity loss.

[0085] According to an embodiment of the present application, the calculation method of the volume regularization can be as shown in formula (6).

[0086] (6).

[0087] refers to the number of target sample features in the scene. represents the product of the vector values, that is, the scale of each target sample feature the product of the plurality of target sample parameters (associated with the target sample features).

[0088] According to an embodiment of the present application, the volume regularization term encourages the number of target sample features in the model to remain small, and reduces the overlap between them. Such regularization strategy helps to prevent overfitting, and improves the accuracy of evaluation.

[0089] According to an embodiment of the present application, the image similarity evaluation method (Structural Similarity Index, SSIM) is used to measure the similarity between two images, and the similarity between the target sample image and the label image is determined based on the SSIM to determine the similarity loss.

[0090] According to an embodiment of the present application, the loss calculation result is determined according to the volume regularization loss, the pixel loss and the similarity loss, which can be shown as formula (7).

[0091] (7).

[0092] According to an embodiment of the present application, represents the loss calculation result, represents the pixel loss, represents the similarity loss, represents the volume regularization loss.

[0093] Figure 3 shows a schematic diagram of an image rendering method according to an embodiment of the present application

[0094] According to an embodiment of the present application, as Figure 3The image fusion features are obtained by fusing features of multiple initial images. The voxel grid is generated based on the image fusion features, so as to obtain anchor point features of the voxel grid. The specified poses of the multiple initial images and the image fusion features are fused based on an image cross-attention mechanism, so as to obtain first fusion features. The specified poses of the multiple initial images and the anchor point features are fused based on a three-dimensional structure cross-attention mechanism, so as to obtain second fusion features. The specified poses can be processed based on a self-attention mechanism before being fused with the image fusion features and the anchor point features respectively. The first fusion features and the second fusion features are fused, so as to obtain target features. Image attribute detection is performed on the target features by using a feedforward neural network (FNN), so as to obtain target attributes. Image rendering is performed by using the target attributes, so as to obtain a target image representing a target object based on a target shooting angle.

[0095] According to the embodiments of the present application, the initial images contain rich color and texture information, which helps to represent the detailed features of the object surface. By combining the texture information of the image fusion features with the shape information of the point cloud (anchor point features), a more realistic and delicate three-dimensional model can be obtained, and then the target image can be obtained.

[0096] According to the embodiments of the present application, the pixel information in the image fusion features can be used to identify the object edges and textures, which helps to accurately determine the boundaries of the object and identify specific parts of the object in the point cloud data (voxel grid).

[0097] According to the embodiments of the present application, the point cloud data fused with the image features can better restore the geometric shape and surface structure of the object, thereby improving the quality of three-dimensional reconstruction.

[0098] According to the embodiments of the present application, by matching the anchor point features with the image fusion features, a one-to-many calibration advantage is obtained, and then the camera pose is optimized by using the specified pose, so that the data from different angles or different time points can be aligned to a common coordinate system, ensuring the consistency of the data and providing a basis for subsequent data processing and analysis. In real measurement, due to various factors, the collected data often has certain deviations. Pose correction can help to reduce these errors and improve the quality of reconstructed data.

[0099] Based on the above image rendering method, the present application further provides an image rendering device. The following will be combined with Figure 4 The device will be described in detail.

[0100] Figure 4 The structure block diagram of the image rendering device according to the embodiments of the present application is shown.

[0101] AsFigure 4 As shown, the image rendering apparatus 400 of this embodiment includes a first fusion module 410, a generation module 420, a second fusion module 430, a detection module 440, and a rendering module 450.

[0102] The first fusion module 410 is configured to perform feature fusion on a plurality of initial images to obtain image fusion features, the plurality of initial images representing a target object based on different initial shooting perspectives. In an embodiment, the first fusion module 410 can be configured to perform operation S210 described above, and details are not repeated here.

[0103] The generation module 420 is configured to generate a voxel grid representing the target object according to the image fusion features, wherein a center point of the voxel grid serves as an anchor point feature. In an embodiment, the generation module 420 can be configured to perform operation S220 described above, and details are not repeated here.

[0104] The second fusion module 430 is configured to perform attention fusion on a specified pose of the plurality of initial images, the image fusion features, and the anchor point feature based on an attention mechanism to obtain target features. In an embodiment, the second fusion module 430 can be configured to perform operation S230 described above, and details are not repeated here.

[0105] The detection module 440 is configured to perform image attribute detection on the target features to obtain target attributes. In an embodiment, the detection module 440 can be configured to perform operation S240 described above, and details are not repeated here.

[0106] The rendering module 450 is configured to perform image rendering using the target attributes to obtain a target image representing the target object based on a target shooting perspective. In an embodiment, the rendering module 450 can be configured to perform operation S250 described above, and details are not repeated here.

[0107] According to an embodiment of the present application, a plurality of initial images are subjected to feature fusion to obtain image fusion features, the plurality of initial images representing a target object based on different initial shooting perspectives; a voxel grid representing the target object is generated according to the image fusion features, wherein a center point of the voxel grid serves as an anchor point feature; attention fusion is performed on a specified pose of the plurality of initial images, the image fusion features, and the anchor point feature based on an attention mechanism to obtain target features; image attribute detection is performed on the target features to obtain target attributes; and image rendering is performed using the target attributes to obtain a target image representing the target object based on a target shooting perspective. Feature extraction is performed on the initial images to obtain the image fusion features, and image features related to point cloud features are extracted; attention fusion is performed on the specified pose of the initial images, the image fusion features, and the anchor point feature, which enhances the detail performance of the target image and improves the recognition accuracy of the target image.

[0108] According to an embodiment of the present application, any of the first fusion module 410, the generation module 420, the second fusion module 430, the detection module 440 and the rendering module 450 can be combined in one module, or any of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules and implemented in one module. According to an embodiment of the present application, at least one of the first fusion module 410, the generation module 420, the second fusion module 430, the detection module 440 and the rendering module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or in any one of software, hardware and firmware implementation or in a suitable combination of any of them. Alternatively, at least one of the first fusion module 410, the generation module 420, the second fusion module 430, the detection module 440 and the rendering module 450 can be at least partially implemented as a computer program module that can perform corresponding functions when executed.

[0109] According to an embodiment of the present application, the second fusion module 430 includes a first fusion sub-module, a second fusion sub-module and a third fusion sub-module.

[0110] The first fusion sub-module is configured to perform attention fusion on the specified pose and the image fusion feature to obtain a first fusion feature.

[0111] The second fusion sub-module is configured to perform attention fusion on the specified pose and the anchor point feature to obtain a second fusion feature.

[0112] The third fusion sub-module is configured to fuse the first fusion feature and the second fusion feature to obtain a target feature.

[0113] According to an embodiment of the present application, the first fusion sub-module includes a projection unit, a first determination unit, a second determination unit and a third determination unit.

[0114] The projection unit is configured to project the target object according to the specified pose to obtain a projection image.

[0115] The first determination unit is configured to determine a plurality of initial associated pixels corresponding to a target projection pixel of the projection image from a plurality of initial images.

[0116] The second determination unit is configured to determine an associated pixel similarity vector related to the target projection pixel according to a similarity value between the plurality of initial associated pixels.

[0117] The third determination unit is configured to determine the first fusion feature by performing weighted summation on the image fusion features according to the correlation pixel similarity vectors corresponding to the target projection pixels.

[0118] According to an embodiment of the present application, the second fusion sub-module comprises a fourth determination unit.

[0119] The fourth determination unit is configured to determine the second fusion feature by performing weighted summation on the anchor point features according to the correlation pixel similarity vectors corresponding to the target projection pixels.

[0120] According to an embodiment of the present application, the first fusion module 410 comprises a first extraction sub-module and a fourth fusion sub-module.

[0121] The first extraction sub-module is configured to perform feature extraction on the plurality of initial images to obtain color features and view distance deviation features.

[0122] The fourth fusion sub-module is configured to perform feature fusion on the color features and the view distance deviation features to obtain image fusion features.

[0123] According to an embodiment of the present application, the detection module 440 comprises a first determination sub-module and a first detection sub-module.

[0124] The first determination sub-module is configured to determine a relative view angle distance between the camera and the anchor point features according to the specified pose.

[0125] The first detection sub-module is configured to perform image attribute detection on the relative view angle distance and the target features by using a multi-layer perception model to obtain target attributes.

[0126] According to an embodiment of the present application, the first detection sub-module comprises a first fusion unit, a generation unit, a second fusion unit, a first detection unit, a first rendering unit, a first calculation unit and a first optimization unit.

[0127] The first fusion unit is configured to perform feature fusion on a plurality of initial sample images to obtain sample image fusion features, the plurality of initial sample images representing a target sample object from different initial sample shooting view angles.

[0128] The generation unit is configured to generate a sample voxel grid representing the target sample object according to the sample image fusion features, wherein a center point of the sample voxel grid is used as a sample anchor point feature.

[0129] The second fusion unit is configured to perform attention fusion on the specified pose, the sample image fusion features and the sample anchor point features of the plurality of initial sample images based on an attention mechanism to obtain target sample features.

[0130] The first detection unit is configured to perform image attribute detection on the target sample feature by using the multi-layer perception model, and obtain a target sample attribute.

[0131] The first rendering unit is configured to perform image rendering by using the target sample attribute, and obtain a target sample image representing the target sample object based on a target sample shooting angle.

[0132] The first calculation unit is configured to perform loss calculation on the target sample image and a label sample image related to the plurality of initial sample images, and obtain a loss calculation result.

[0133] The first optimization unit is configured to optimize the multi-layer perception model according to the loss calculation result, and obtain an optimized multi-layer perception model.

[0134] According to the embodiment of the present application, the first calculation unit comprises a first determination sub-unit, a second determination sub-unit, a third determination sub-unit and a fourth determination sub-unit.

[0135] The first determination sub-unit is configured to determine a volume regularization loss according to a product of a plurality of target sample parameters associated with the target sample feature.

[0136] The second determination sub-unit is configured to determine a pixel loss according to a pixel color difference between the target sample image and the label sample image.

[0137] The third determination sub-unit is configured to determine a similarity loss according to a similarity between the target sample image and the label image.

[0138] The fourth determination sub-unit is configured to determine the loss calculation result according to the volume regularization loss, the pixel loss and the similarity loss.

[0139] Figure 5 A block diagram of an electronic device suitable for implementing the image rendering method according to the embodiment of the present application is shown.

[0140] As shown in Figure 5 The electronic device 500 according to the embodiment of the present application comprises a processor 501 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or loaded from a storage portion 508 to a random access memory (RAM) 503. The processor 501 can comprise, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset, and / or a special-purpose microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 501 can further comprise an on-board memory for cache use. The processor 501 can comprise a single processing unit or a plurality of processing units for performing different actions of the method flow according to the embodiment of the present application.

[0141] In the RAM 503, various programs and data required for the operation of the electronic device 500 are stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other via the bus 504. The processor 501 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 502 and / or the RAM 503. It should be noted that the programs can also be stored in one or more memories other than the ROM 502 and the RAM 503. The processor 501 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0142] According to the embodiments of the present application, the electronic device 500 can further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 can further include one or more of the following components connected to the input / output (I / O) interface 505: an input part 506 including a keyboard, a mouse, etc.; an output part 507 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 508 including a hard disk, etc.; and a communication part 509 including a network interface card such as a LAN card, a modem, etc. The communication part 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as necessary. A removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 510 as necessary, so that a computer program read therefrom is installed in the storage part 508 as necessary.

[0143] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0144] According to an embodiment of the present application, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories of the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503.

[0145] Embodiments of the present application also include a computer program product, which comprises a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the image rendering method provided by the embodiments of the present application.

[0146] The above functions defined in the system / device of the embodiments of the present application are performed when the computer program is executed by the processor 501. According to an embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0147] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, downloaded and installed in the form of signals on a network medium, and downloaded and installed through the communication part 509 and / or installed from the detachable medium 511. The program codes contained in the computer program can be transmitted by any suitable network medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0148] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509 and / or installed from the detachable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiments of the present application are performed. According to an embodiment of the present application, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0149] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming language can include, but is not limited to, Java, C++, python, "C" language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0150] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0151] Those skilled in the art will appreciate that the features recited in the various embodiments of the present application can be combined and / or integrated in a variety of ways, even if such combinations or integrations are not expressly noted in the present application. In particular, the features recited in the various embodiments of the present application can be combined and / or integrated in a variety of ways without departing from the spirit and scope of the present application. All such combinations and / or integrations are within the scope of the present application.

[0152] The embodiments of the present application described above are merely intended to illustrate the present application. These embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present application. Although the above describes each embodiment separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Numerous alternatives and modifications of embodiments of the present application can be made by those skilled in the art without departing from the scope of the present application, and all such alternatives and modifications are to be included within the scope of the present application.

Claims

1. An image rendering method, characterized by, The method comprises: feature extraction is performed on a plurality of initial images to obtain color features and view distance deviation features; the color features and the view distance deviation features are fused to obtain image fusion features, and the plurality of initial images represent a target object based on different initial shooting angles; a voxel grid representing the target object is generated according to the image fusion features, wherein a center point of the voxel grid is used as an anchor point feature; attention fusion is performed on a specified pose of the plurality of initial images, the image fusion features and the anchor point feature based on an attention mechanism to obtain a target feature; image attribute detection is performed on the target feature to obtain a target attribute; image rendering is performed using the target attribute to obtain a target image representing the target object based on a target shooting angle.

2. The method of claim 1, wherein, The attention fusion performed on the specified pose of the plurality of initial images, the image fusion features and the anchor point feature based on the attention mechanism to obtain the target feature comprises: attention fusion is performed on the specified pose and the image fusion features to obtain a first fusion feature; attention fusion is performed on the specified pose and the anchor point feature to obtain a second fusion feature; the first fusion feature and the second fusion feature are fused to obtain the target feature.

3. The method of claim 2, wherein, The attention fusion performed on the specified pose and the image fusion features to obtain the first fusion feature comprises: projection is performed on the target object according to the specified pose to obtain a projection image; a plurality of initial associated pixels corresponding to target projection pixels of the projection image are determined from the plurality of initial images; an associated pixel similarity vector related to the target projection pixels is determined according to similarity values between the plurality of initial associated pixels; the image fusion features are weighted and summed according to the associated pixel similarity vectors corresponding to the plurality of target projection pixels to determine the first fusion feature.

4. The method of claim 3, wherein, The attention fusion performed on the specified pose and the anchor point feature to obtain the second fusion feature comprises: the anchor point features are weighted and summed according to the associated pixel similarity vectors corresponding to the plurality of target projection pixels to determine the second fusion feature.

5. The method of claim 1, wherein, The feature fusion performed on the plurality of initial images to obtain the image fusion features comprises: feature extraction is performed on the plurality of initial images to obtain color features and view distance deviation features; the color features and the view distance deviation features are fused to obtain the image fusion features.

6. The method of claim 1, wherein, The image attribute detection performed on the target feature to obtain the target attribute comprises: a relative angle distance between a camera and the anchor point feature is determined according to the specified pose; image attribute detection is performed on the relative angle distance and the target feature using a multi-layer perception model to obtain the target attribute.

7. The method of claim 6, wherein, The multi-layer perception model is optimized based on the following operation: sample image fusion features are obtained by performing feature fusion on a plurality of initial sample images, and the plurality of initial sample images represent a target sample object based on different initial sample shooting angles; generate a sample voxel grid representing the target sample object based on the sample image fusion feature, wherein a center point of the sample voxel grid is used as a sample anchor point feature; perform attention fusion on the specified poses of the plurality of initial sample images, the sample image fusion feature, and the sample anchor point feature based on an attention mechanism to obtain a target sample feature; perform image attribute detection on the target sample feature using the multi-layer perception model to obtain a target sample attribute; perform image rendering using the target sample attribute to obtain a target sample image representing the target sample object based on a target sample shooting angle; perform loss calculation on the target sample image and a label sample image related to the plurality of initial sample images to obtain a loss calculation result; and optimize the multi-layer perception model based on the loss calculation result to obtain an optimized multi-layer perception model.

8. The method of claim 7, wherein, The loss calculation on the target sample image and the label sample image related to the plurality of initial sample images to obtain a loss calculation result includes: determining a volume regularization loss based on a product of a plurality of target sample parameters associated with the target sample feature; determining a pixel loss based on a pixel color difference between the target sample image and the label sample image; determining a similarity loss based on a similarity between the target sample image and the label sample image; and determining the loss calculation result based on the volume regularization loss, the pixel loss, and the similarity loss.

9. An image rendering apparatus, characterized by comprising: The apparatus includes: a first fusion module configured to extract color features and view distance deviation features from a plurality of initial images representing a target object based on different initial shooting angles, perform feature fusion on the color features and the view distance deviation features to obtain image fusion features, and generate a voxel grid representing the target object based on the image fusion features, wherein a center point of the voxel grid is used as an anchor point feature; a second fusion module configured to perform attention fusion on specified poses of the plurality of initial images, the image fusion features, and the anchor point feature based on an attention mechanism to obtain a target feature; a detection module configured to perform image attribute detection on the target feature to obtain a target attribute; and a rendering module configured to perform image rendering using the target attribute to obtain a target image representing the target object based on a target shooting angle.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-8. ​

Images