Real-time image rendering generation system based on generative artificial intelligence

By using generative artificial intelligence technology to divide local areas and reconstruct pixel feature in the TSDF field, the problem of lattice point correlation in the rendering method is solved, and the rendering quality and reality are improved.

CN118963602BActive Publication Date: 2024-12-06SHIYOU (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411448712.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-10-17
Publication Date
2024-12-06
Estimated Expiration
2044-10-17

AI Technical Summary

Technical Problem

When the existing TSDF field rendering method deals with complex surfaces, rendering based on intersection information may ignore the correlation between grid points, resulting in insufficient realism and fineness of the rendering results.

Method used

Using processing and analysis technology based on generative artificial intelligence, local area division and pixel feature extraction are performed on the TSDF field implicit surface, and dimensional reconstruction and gradient correlation enhancement are performed on the local pixel features of lattice points to generate rendering results.

Benefits of technology

By capturing the connection between grid points, considering more local and global semantic information, improving rendering quality and achieving more intelligent image rendering.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963602B_ABST
    Figure CN118963602B_ABST
Patent Text Reader

Abstract

The present application provides a real-time image rendering generation system based on generative artificial intelligence, which relates to the field of image processing. It uses processing and analysis techniques based on generative artificial intelligence to perform local area division and pixel feature extraction on implicit surfaces of truncated signed distance function TSDF fields containing intersection points, and reconstruct the dimensions of local pixel features of each TSDF field grid point, so as to intelligently generate rendering results based on the associated enhanced features of each pixel gradient in the reconstructed features. In this way, the connection between each grid point can be better captured, so that more local and global semantic information can be taken into account during rendering, the rendering quality can be improved, and more intelligent image rendering can be achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to a real-time image rendering generation system based on generative artificial intelligence. Background Art

[0002] Image rendering technology refers to the process of converting a three-dimensional model into a two-dimensional image, which usually involves steps such as lighting calculation, texture mapping, and shadow processing to generate realistic visual effects. TSDF (truncated signed distance function) technology can fuse multiple frames of scan data into a fractional field. Through TSDF technology, complex three-dimensional scenes can be efficiently reconstructed and rendered, improving the realism and detail performance of the rendering results. In the existing technology, point rendering and triangle mesh rendering are two visualization methods of TSDF fields. However, point rendering lacks details at close distances and the color transition is unnatural at long distances. Although triangle mesh rendering can provide more complete details, it has a high computational cost, which affects the frame rate and real-time rendering.

[0003] In response to the above technical problems, patent CN115170715A proposes an image rendering method, device, electronic device and medium, which generates a ray set and calculates the intersection with the TSDF field by receiving display instructions and obtaining images and parameters, and then uses the intersection information to render the grid points. Finally, the results are displayed on the interface to achieve efficient image rendering, thereby solving the problems of poor rendering effect, low efficiency and impact on frame rate in the prior art, realizing real-time rendering of images and improving rendering quality.

[0004] In the above patent, rendering is performed based on the intersection information between the ray and the implicit surface of the TSDF field. However, in the TSDF field, each grid point not only stores distance information, but may also contain attributes such as color and normal vector. If rendering is performed based on intersection information, it mainly focuses on the attributes of a single grid point, while ignoring the correlation between grid points. For example, when calculating the color of a pixel, the method based on intersection information may only refer to the color information of the grid point at the intersection, without considering the color information and other attributes (such as texture, material, etc.) of the surrounding grid points. This may cause some details to be lost during the rendering process, especially when dealing with complex surfaces, which may affect the realism and refinement of the rendering results.

[0005] Therefore, an optimized real-time image rendering generation system is desired. Summary of the invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a real-time image rendering generation system based on generative artificial intelligence, which uses processing and analysis technology based on generative artificial intelligence to perform local area division and pixel feature extraction on the implicit surface of the truncated signed distance function TSDF field containing intersection points, and reconstruct the dimensions of the local pixel features of each TSDF field grid point, so as to intelligently generate rendering results based on the associated enhanced features of each pixel gradient in the reconstructed features. In this way, the connection between each grid point can be better captured, so that more local and global semantic information can be taken into account during rendering, the rendering quality can be improved, and more intelligent image rendering can be achieved.

[0007] According to one aspect of the present application, a real-time image rendering generation system based on generative artificial intelligence is provided, which includes: an image parameter acquisition module for rendering, which is used to obtain the image to be rendered and rendering parameters in the interactive interface in response to a display instruction received in the interactive interface; a ray set generation module, which is used to generate a ray set based on the image to be rendered and the rendering parameters; a TSDF field implicit surface pre-generation module, which is used to pre-generate a truncated signed distance function TSDF field implicit surface; an intersection calculation module, which is used to calculate the intersection of each ray in the ray set and the truncated signed distance function TSDF field implicit surface to obtain the truncated signed distance function TSDF field implicit surface containing the intersection; a rendering module, which is used to render the TSDF field grid points where the intersection is located based on the truncated signed distance function TSDF field implicit surface containing the intersection to generate a rendering result; and a rendering result display module, which is used to display the rendering result on the interactive interface.

[0008] Among them, the rendering module includes: a TSDF field implicit surface area division unit, which is used to perform local area division on the truncated signed distance function TSDF field implicit surface containing the intersection point to obtain a set of local areas of the TSDF field grid points; a TSDF field grid point pixel area extraction and reconstruction unit, which is used to extract pixel features from the set of local areas of the TSDF field grid points, and perform dimensional reconstruction on the set of extracted local pixel feature vectors of the TSDF field grid points to obtain local pixel features of the TSDF field grid points; a TSDF field grid point pixel association enhancement unit, which is used to perform association enhancement on the local pixel features of the TSDF field grid points based on pixel gradient distribution to obtain local pixel association enhancement features of the TSDF field grid points; and a rendering result generation unit, which is used to obtain the rendering result based on the local pixel association enhancement features of the TSDF field grid points.

[0009] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the TSDF field implicit surface area division unit is used to: divide the truncated signed distance function TSDF field implicit surface containing the intersection points into local areas according to the TSDF field grid points where the intersection points are located to obtain a set of local areas of the TSDF field grid points.

[0010] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the TSDF field grid point pixel area extraction and reconstruction unit includes: a TSDF field grid point local pixel feature extraction subunit, which is used to pass the set of TSDF field grid point local areas through a grid point local area pixel feature extractor based on a deep neural network model to obtain a set of TSDF field grid point local pixel feature vectors; a TSDF field grid point local pixel feature dimension reconstruction subunit, which is used to reconstruct the dimension of the set of TSDF field grid point local pixel feature vectors according to the local area division method to obtain a TSDF field grid point local pixel feature matrix as the TSDF field grid point local pixel feature.

[0011] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the grid point local area pixel feature extractor based on the deep neural network model is a grid point local area pixel feature extractor based on the convolutional neural network model.

[0012] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the TSDF field grid point pixel association enhancement unit includes: a TSDF field grid point local pixel gradient amplitude calculation subunit, which is used to calculate the multi-directional gradient value distribution of each position in the TSDF field grid point local pixel feature matrix, and determine the gradient amplitude value of each position in the TSDF field grid point local pixel feature matrix based on the multi-directional gradient value distribution of each position to obtain the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; a TSDF field grid point local pixel gradient masking subunit, Used to input the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix into the gated mask module for masking processing to obtain the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; TSDF field grid point local pixel enhancement subunit is used to use the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix as the weight matrix, and perform weighted enhancement on the TSDF field grid point local pixel feature matrix to obtain the TSDF field grid point local pixel association enhancement feature matrix as the TSDF field grid point local pixel association enhancement feature.

[0013] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the TSDF field grid point local pixel gradient amplitude calculation subunit is used to: calculate the gradient amplitude value of each eigenvalue in the TSDF field grid point local pixel feature matrix along the width direction to obtain the TSDF field grid point local pixel width direction characteristic gradient value feature matrix; calculate the gradient amplitude value of each eigenvalue in the TSDF field grid point local pixel feature matrix along the height direction to obtain the TSDF field grid point local pixel height direction characteristic gradient value feature matrix; calculate the square root of the sum of the squares of the eigenvalues ​​at corresponding positions in the TSDF field grid point local pixel width direction characteristic gradient value feature matrix and the TSDF field grid point local pixel height direction characteristic gradient value feature matrix to obtain the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix.

[0014] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the TSDF field grid point local pixel gradient masking subunit is used to: compare the eigenvalue of each position in the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix with a preset threshold to obtain the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; wherein, in response to the eigenvalue being greater than the preset threshold, the eigenvalue greater than the preset threshold is set to one; in response to the eigenvalue being less than or equal to the preset threshold, the eigenvalue less than or equal to the preset threshold is set to zero.

[0015] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the TSDF field grid point local pixel enhancement subunit is used to: perform positional dot multiplication on the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix and the TSDF field grid point local pixel feature matrix, and then input the feature matrix after the dot multiplication into the sigmoid function for enhancement processing to obtain the TSDF field grid point local pixel association enhancement feature matrix.

[0016] In the above-mentioned real-time image rendering generation system based on generative artificial intelligence, the rendering result generation unit is used to: pass the local pixel association enhancement feature matrix of the TSDF field grid point through a rendering module based on the AIGC model to obtain the rendering result.

[0017] Compared with the prior art, the real-time image rendering generation system based on generative artificial intelligence provided by the present application uses generative artificial intelligence-based processing and analysis technology to perform local area division and pixel feature extraction on the implicit surface of the truncated signed distance function TSDF field containing intersection points, and reconstructs the dimension of the local pixel features of each TSDF field grid point, so as to intelligently generate rendering results based on the associated enhanced features of each pixel gradient in the reconstructed features. In this way, the connection between each grid point can be better captured, so that more local and global semantic information can be taken into account during rendering, the rendering quality can be improved, and more intelligent image rendering can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] In the attached picture: Figure 1 It is a system block diagram of a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application.

[0020] Figure 2 It is a block diagram of a rendering module in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application.

[0021] Figure 3 Schematic diagram of data flow of a rendering module in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application.

[0022] Figure 4 It is a block diagram of a TSDF field grid point pixel area extraction and reconstruction unit in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application.

[0023] Figure 5 It is a block diagram of a TSDF field grid point pixel association enhancement unit in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0025] Image rendering technology involves converting a 3D model into a 2D image. This process includes multiple steps such as lighting calculation, texture mapping, and shadow processing, aiming to create realistic visual effects. TSDF (truncated signed distance function) technology can efficiently reconstruct and render complex 3D scenes by fusing multiple frames of scan data into a fractional field, thereby improving the realism and detail of the rendering results. Among the existing technologies, point rendering and triangle mesh rendering are two common TSDF field visualization methods. However, point rendering lacks details at close distances, and the color transition appears unnatural at long distances; while triangle mesh rendering can provide more complete details, its high computational cost will reduce the frame rate and affect the real-time performance of the rendering.

[0026] In response to the above technical problems, patent CN115170715A proposes an image rendering method, device, electronic device and medium, which generates a set of rays by receiving display instructions and obtaining images and their parameters, and calculates the intersection of these rays with the TSDF field. Based on these intersection information, grid points are further rendered, and the rendering results are finally displayed on the interface. This solution realizes efficient image rendering, solves the problems of poor rendering effect, low efficiency and negative impact on frame rate in the prior art, realizes real-time rendering of images, and improves rendering quality.

[0027] In the above patent, rendering is mainly based on the intersection information of rays and implicit surfaces in the TSDF field. In the TSDF field, each grid point not only stores distance information, but may also contain attributes such as color and normal vector. If rendering is based only on intersection information, excessive attention may be paid to the attributes of a single grid point, while ignoring the correlation between surrounding grid points. This may cause some details to be lost during the rendering process, especially when dealing with complex surfaces, which may affect the realism and refinement of the rendering results.

[0028] Based on this, this application proposes a real-time image rendering generation system based on generative artificial intelligence. Figure 1 1 is a system block diagram of a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application. Figure 1As shown, in a real-time image rendering generation system 100 based on generative artificial intelligence, it includes: an image parameter acquisition module 110 to be rendered, which is used to obtain the image to be rendered and the rendering parameters in the interactive interface in response to the display instruction received in the interactive interface; a ray set generation module 120, which is used to generate a ray set based on the image to be rendered and the rendering parameters; a TSDF field implicit surface pre-generation module 130, which is used to pre-generate a truncated signed distance function TSDF field implicit surface; an intersection calculation module 140, which is used to calculate the intersection of each ray in the ray set and the truncated signed distance function TSDF field implicit surface to obtain the truncated signed distance function TSDF field implicit surface containing the intersection; a rendering module 150, which is used to render the TSDF field grid points where the intersection is located based on the truncated signed distance function TSDF field implicit surface containing the intersection to generate a rendering result; a rendering result display module 160, which is used to display the rendering result on the interactive interface.

[0029] In an embodiment of the present application, the image to be rendered parameter acquisition module 110 is used to respond to the display instruction received in the interactive interface, and obtain the image to be rendered and rendering parameters in the interactive interface. Specifically, based on the user's triggering operation on the preset display control on the display page, the display instruction is received and responded to, and the image to be rendered and the rendering parameters are obtained. Here, the display page refers to a page for displaying a rendered image, and the display control refers to an anchor point set in the above-mentioned display page for display operation. The expression form of the display control is not limited, and it can be an icon or text information. It should be understood that the image to be rendered in the interactive interface in the present application refers to the current display screen pixel point in the interactive interface, that is, the area to be rendered, and can also be understood as the surface of the TSDF field under the current camera perspective, and the rendering parameters refer to the intrinsic parameters of the rendering camera (the position of the rendering camera in space, consisting of a rotation and a translation) and the extrinsic parameters of the rendering camera (the internal parameters corresponding to the pinhole rendering camera, which converts the three-dimensional coordinates into two-dimensional plane coordinates and depth). It is worth noting that the rendering camera refers to the simulated camera defined in the program, not the real camera.

[0030] In an embodiment of the present application, the ray set generation module 120 is used to generate a ray set based on the image to be rendered and the rendering parameters. In the present application, a ray refers to a straight line formed by infinitely extending one end of a line segment, which is composed of pixels corresponding to the image to be rendered and the intrinsic and extrinsic parameters of the rendering camera, and a ray set refers to a set including multiple rays. In particular, in a specific implementation method of an embodiment of the present application, the center of the rendering camera can be determined based on the rendering parameters, multiple pixels can be determined based on the image to be rendered, and multiple rays can be determined based on the center of the rendering camera and each pixel, thereby obtaining a ray set.

[0031] In an embodiment of the present application, the TSDF field implicit surface pre-generation module 130 is used to pre-generate a truncated signed distance function TSDF field implicit surface. It should be understood that the pre-generated TSDF field implicit surface constructs a continuous surface representation by storing information about the surface of an object on discrete grid points in three-dimensional space. Each grid point stores a TSDF value, which represents the signed distance from the point to the nearest surface, as well as information such as the surface normal vector and color. In other words, the truncated signed distance function TSDF field implicit surface is a surface containing rich content information. The pre-generated TSDF field implicit surface can completely preserve the geometric information of the object, thereby providing more accurate surface details during rendering, and by pre-generating the TSDF field, the required distance and direction information can be quickly accessed during the rendering process, avoiding performance degradation caused by real-time calculations.

[0032] In an embodiment of the present application, the intersection calculation module 140 is used to calculate the intersection of each ray in the ray set and the implicit surface of the truncated signed distance function TSDF field to obtain the implicit surface of the truncated signed distance function TSDF field containing the intersection. It should be noted that, for any point in a given space, if this point has a score in the TSDF field, and the score is 0, then this point is a point on the implicit surface of the TSDF field. In a specific implementation method of the embodiment of the present application, the grid point set can be calculated by screening the grid point set to obtain the target grid point, and the score value and weight of the target grid point are calculated to obtain the score value of each ray, and based on the score value, the intersection of each ray and the implicit surface of the TSDF field is determined, and then the truncated signed distance function TSDF field implicit surface containing the intersection point is obtained based on multiple intersection points.

[0033] In the embodiment of the present application, the rendering module 150 is used to render the TSDF field grid points where the intersection points are located based on the truncated signed distance function TSDF field implicit surface containing the intersection points to generate a rendering result.

[0034] In rendering the TSDF field grid points where the intersections are located based on the truncated signed distance function TSDF field implicit surface containing the intersections to generate the rendering results, the technical concept of the present application is to use generative artificial intelligence-based processing and analysis technology to perform local area division and pixel feature extraction on the truncated signed distance function TSDF field implicit surface containing the intersections, and to perform dimensional reconstruction of the local pixel features of each TSDF field grid point, so as to intelligently generate the rendering results based on the associated enhanced features of each pixel gradient in the reconstructed features. In this way, the connection between each grid point can be better captured, so that more local and global semantic information can be taken into account during rendering, the rendering quality can be improved, and more intelligent image rendering can be achieved.

[0035] Figure 2 It is a block diagram of a rendering module in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application. Figure 3 Schematic diagram of data flow of a rendering module in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application. Figure 2 and Figure 3 As shown, the rendering module 150 includes: a TSDF field implicit surface area division unit 151, which is used to perform local area division on the truncated signed distance function TSDF field implicit surface containing the intersection point to obtain a set of local areas of the TSDF field grid points; a TSDF field grid point pixel area extraction and reconstruction unit 152, which is used to perform pixel feature extraction on the set of local areas of the TSDF field grid points, and perform dimensional reconstruction on the set of extracted local pixel feature vectors of the TSDF field grid points to obtain local pixel features of the TSDF field grid points; a TSDF field grid point pixel association enhancement unit 153, which is used to perform association enhancement on the local pixel features of the TSDF field grid points based on pixel gradient distribution to obtain local pixel association enhancement features of the TSDF field grid points; and a rendering result generation unit 154, which is used to obtain the rendering result based on the local pixel association enhancement features of the TSDF field grid points.

[0036] In an embodiment of the present application, the TSDF field implicit surface area division unit 151 is used to perform local area division on the truncated signed distance function TSDF field implicit surface containing the intersection to obtain a set of local areas of the TSDF field grid points. Specifically, in an embodiment of the present application, the TSDF field implicit surface area division unit is used to: perform local area division on the truncated signed distance function TSDF field implicit surface containing the intersection according to the TSDF field grid points where the intersections are located to obtain a set of local areas of the TSDF field grid points. It should be understood that different intersections and the areas around them in the truncated signed distance function TSDF field implicit surface containing the intersections have different detail features. Therefore, in order to be able to more accurately process the features between each local area, while only focusing on the areas around the intersections, and avoiding complex calculations on large areas far away from the intersections, in the technical solution of the present application, the truncated signed distance function TSDF field implicit surface containing the intersections is divided into local areas according to the TSDF field grid points where the intersections are located to obtain a set of local areas of the TSDF field grid points. That is, the intersection points are usually where the light intersects the surface of the object. These places are the areas that need to be focused on during the rendering process. By treating these intersection points and the areas around them as local areas, these important details can be processed more finely, which helps to better extract the features in each area.

[0037] In the embodiment of the present application, the TSDF field grid point pixel area extraction and reconstruction unit 152 is used to extract pixel features from the set of local areas of the TSDF field grid points, and to reconstruct the dimension of the set of local pixel feature vectors of the extracted TSDF field grid points to obtain local pixel features of the TSDF field grid points. Specifically, Figure 4 FIG. 1 is a block diagram of a TSDF field grid point pixel region extraction and reconstruction unit in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application. Figure 4 As shown, the TSDF field grid point pixel area extraction and reconstruction unit 152 includes: a TSDF field grid point local pixel feature extraction subunit 1521, which is used to pass the set of the TSDF field grid point local areas through a grid point local area pixel feature extractor based on a deep neural network model to obtain a set of the TSDF field grid point local pixel feature vectors; a TSDF field grid point local pixel feature dimension reconstruction subunit 1522, which is used to reconstruct the dimension of the set of the TSDF field grid point local pixel feature vectors according to a local area division method to obtain a TSDF field grid point local pixel feature matrix as the TSDF field grid point local pixel feature.

[0038] It should be understood that pixel features can represent key information of each local area, including but not limited to color, texture, edge, curvature, etc. Therefore, in order to further capture and extract the pixel features contained in each TSDF field grid point local area to more effectively characterize the information in each local area, in the technical solution of the present application, the set of the TSDF field grid point local area is passed through a grid point local area pixel feature extractor based on a convolutional neural network model to obtain a set of TSDF field grid point local pixel feature vectors. Accordingly, considering that there are mutual correlations and influences between the local pixel feature features of each TSDF field grid point in the set of the TSDF field grid point local pixel feature vectors, and in order to better reflect the relationship between local areas in the future, in the technical solution of the present application, the set of the TSDF field grid point local pixel feature vectors is dimensionally reconstructed according to the local area division method to obtain the TSDF field grid point local pixel feature matrix. In this way, the interdependence between different features can be better captured, which is very important for the subsequent rendering process.

[0039] In an embodiment of the present application, the TSDF field grid point pixel association enhancement unit 153 is used to perform association enhancement based on pixel gradient distribution on the local pixel features of the TSDF field grid points to obtain the local pixel association enhancement features of the TSDF field grid points. Accordingly, considering the different importance and contribution of different positions in the TSDF field grid point local pixel association enhancement feature matrix, in order to more selectively highlight and highlight the key pixel feature information in the TSDF field grid point local pixel association enhancement feature matrix, so as to better capture the details and structure of the local area, in the technical solution of the present application, the TSDF field grid point local pixel feature matrix is ​​subjected to association enhancement based on pixel gradient distribution to obtain the TSDF field grid point local pixel association enhancement feature matrix as the TSDF field grid point local pixel association enhancement feature. It should be understood that the pixel gradient reflects the degree of change between pixels, and the gradient distribution can better capture the edge, texture and other detail information, and the correlation between features can be enhanced through association enhancement, so that the information in the feature matrix is ​​more concentrated and prominent, which helps to better reflect the real characteristics of the object surface in the subsequent rendering process.

[0040] Specifically, first, the gradient amplitude value of each position in the local pixel association enhancement feature matrix of the TSDF field grid point is calculated to obtain the gradient amplitude distribution feature matrix of the local pixel feature of the TSDF field grid point. It can be understood that the gradient amplitude value reflects the degree of change in intensity between pixels. That is, a larger gradient amplitude usually corresponds to a mutation at the edge or the surface of an object. Therefore, by calculating the gradient amplitude value, these edge and detail features can be captured more accurately, so as to correctly represent the object boundary in the rendering process, and then, in order to enhance or suppress certain features, so as to better highlight or ignore certain details, in order to optimize the final rendering result, the gradient amplitude distribution feature matrix of the local pixel feature of the TSDF field grid point is input into the gated mask module for masking to obtain the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix. In other words, the masking process can help select and enhance those features that are crucial to the rendering result. Specifically, through the mask, those features with large gradient amplitudes representing important details can be highlighted, while those with small gradient amplitudes representing noise or unimportant details can be suppressed or ignored. Finally, the masked features are used as weight matrices to weight the TSDF field grid point local pixel association enhancement feature matrix to assign different weights to different features in the original feature matrix. Afterwards, the weighted features are input into the sigmoid function for enhancement, so that the final generated TSDF field grid point local pixel association enhancement feature matrix can better reflect the real characteristics of the object surface and can more accurately capture important details during the rendering process.

[0041] Specifically, Figure 5 FIG. 1 is a block diagram of a TSDF field grid point pixel association enhancement unit in a real-time image rendering generation system based on generative artificial intelligence according to an embodiment of the present application. Figure 5 As shown, the TSDF field grid point pixel association enhancement unit 153 includes: a TSDF field grid point local pixel gradient amplitude calculation subunit 1531, which is used to calculate the multi-directional gradient value distribution of each position in the TSDF field grid point local pixel feature matrix, and determine the gradient amplitude value of each position in the TSDF field grid point local pixel feature matrix based on the multi-directional gradient value distribution of each position to obtain the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; a TSDF field grid point local pixel gradient masking subunit 1532, which is used to mask the TSD The F field grid point local pixel feature gradient amplitude distribution feature matrix is ​​input into the gated mask module for masking processing to obtain the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; the TSDF field grid point local pixel enhancement subunit 1533 is used to use the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix as the weight matrix, and perform weighted enhancement on the TSDF field grid point local pixel feature matrix to obtain the TSDF field grid point local pixel association enhancement feature matrix as the TSDF field grid point local pixel association enhancement feature.

[0042] More specifically, in an embodiment of the present application, the TSDF field grid point local pixel gradient amplitude calculation subunit is used to: calculate the gradient amplitude value along the width direction of each eigenvalue in the TSDF field grid point local pixel feature matrix to obtain the TSDF field grid point local pixel width direction characteristic gradient value feature matrix; calculate the gradient amplitude value along the height direction of each eigenvalue in the TSDF field grid point local pixel feature matrix to obtain the TSDF field grid point local pixel height direction characteristic gradient value feature matrix; calculate the square root of the sum of the squares of the eigenvalues ​​at corresponding positions in the TSDF field grid point local pixel width direction characteristic gradient value feature matrix and the TSDF field grid point local pixel height direction characteristic gradient value feature matrix to obtain the TSDF field grid point local pixel characteristic gradient amplitude distribution feature matrix.

[0043] More specifically, in an embodiment of the present application, the TSDF field grid point local pixel gradient masking subunit is used to: compare the eigenvalue of each position in the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix with a preset threshold to obtain the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; wherein, in response to the eigenvalue being greater than the preset threshold, the eigenvalue greater than the preset threshold is set to one; in response to the eigenvalue being less than or equal to the preset threshold, the eigenvalue less than or equal to the preset threshold is set to zero.

[0044] More specifically, in an embodiment of the present application, the TSDF field grid point local pixel enhancement subunit is used to: perform positional dot multiplication on the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix and the TSDF field grid point local pixel feature matrix, and then input the feature matrix after the dot multiplication into a sigmoid function for enhancement processing to obtain the TSDF field grid point local pixel association enhancement feature matrix.

[0045] In the embodiment of the present application, specifically, the TSDF field grid point pixel association enhancement unit is used to: perform association enhancement processing based on pixel gradient distribution on the TSDF field grid point local pixel feature matrix according to the following formula; wherein the formula is: ;in, , , and are the local pixel feature matrix of the TSDF field grid point. , , and The eigenvalues ​​at the positions, is the number of pixels along the width direction in the local pixel feature matrix of the TSDF field grid point The gradient value at each position, is the first pixel in the local pixel feature matrix of the TSDF field grid point along the height direction. The gradient value at each position, is the local pixel feature gradient amplitude distribution feature matrix of the TSDF field grid point. The eigenvalues ​​at the positions, For masking operation, It is the eigenvalue of each position in the characteristic matrix of the local pixel feature gradient amplitude distribution of the masked TSDF field grid point, is the preset threshold, is the local pixel feature gradient amplitude distribution feature matrix of the masked TSDF field grid point, is the local pixel feature matrix of the TSDF field grid points, It means point multiplication by position. yes function, It is the local pixel correlation enhancement feature matrix of the TSDF field grid points.

[0046] In an embodiment of the present application, the rendering result generation unit 154 is used to obtain the rendering result based on the local pixel association enhancement feature of the TSDF field grid point. Specifically, in an embodiment of the present application, the rendering result generation unit is used to: pass the local pixel association enhancement feature matrix of the TSDF field grid point through a rendering module based on the AIGC model to obtain the rendering result. The AIGC model in this application specifically refers to an image generation model based on a generative adversarial network. The generative adversarial network consists of two neural networks: a generator and a discriminator. The generator is responsible for generating new data samples, trying to create samples similar to real data. The discriminator is responsible for determining whether the input sample is real or generated. It receives real samples and generated samples, and outputs a probability value indicating the possibility that the sample is real. The generator and the discriminator compete with each other during the training process. The generator tries to generate more and more real samples, while the discriminator continuously improves its recognition ability. The goal of training is to make the samples generated by the generator real enough so that the discriminator cannot distinguish the difference between them and real samples. The present application generates the rendering result by inputting the local pixel association enhancement feature matrix of the TSDF field grid points into the generator in the AIGC model through deconvolution coding. That is, the local pixel association enhancement features of the TSDF field grid points obtained by performing pixel association enhancement using the local pixel feature matrix of the TSDF field grid points are generated and processed to intelligently generate the rendering result. In this way, the connection between each grid point can be better captured, so that more local and global semantic information can be taken into account during rendering, the rendering quality can be improved, and more intelligent image rendering can be achieved.

[0047] In particular, the present application takes into account that each local pixel feature vector of the TSDF field grid point in the set of local pixel feature vectors of the TSDF field grid points respectively represents the image semantic features in the local image semantic space domain of the implicit surface of the truncated signed distance function TSDF field containing the intersection point. When the dimensionality is reconstructed and the association is enhanced based on the pixel gradient distribution, the difference in the association of the pixel gradients of the image semantic features in the local image semantic space domain will also make the dimensional composite image semantic representation of the local pixel association enhancement feature matrix of the TSDF field grid points have a complex spatial structure based on different local spatial distribution enhancement structures, thereby expecting to improve the generative regression convergence and generalization effects under complex spatial structures.

[0048] Preferably, in an example of the present application, the TSDF field grid point local pixel association enhancement feature matrix is ​​passed through a rendering module based on an AIGC model to obtain the rendering result, including: calculating the sum of the absolute values ​​of each eigenvalue of the TSDF field grid point local pixel association enhancement feature matrix to obtain a first TSDF field grid point local pixel association enhancement spatial structure value, and calculating the square root of the sum of the squares of each eigenvalue of the TSDF field grid point local pixel association enhancement feature matrix to obtain a second TSDF field grid point local pixel association enhancement spatial structure value; the TSDF field grid point local pixel association enhancement feature matrix is ​​passed through a rendering module based on an AIGC model to obtain the rendering result, including: calculating the sum of the absolute values ​​of each eigenvalue of the TSDF field grid point local pixel association enhancement feature matrix to obtain a first TSDF field grid point local pixel association enhancement spatial structure value, and calculating the square root of the sum of the squares of each eigenvalue of the TSDF field grid point local pixel association enhancement feature matrix to obtain a second TSDF field grid point local pixel association enhancement spatial structure value. Each eigenvalue of the pixel association enhancement feature matrix is ​​multiplied with the first TSDF field grid point local pixel association enhancement spatial structure value and the second TSDF field grid point local pixel association enhancement spatial structure value to obtain the first TSDF field grid point local pixel association enhancement structure reference value and the second TSDF field grid point local pixel association enhancement structure reference value corresponding to each eigenvalue; each eigenvalue of the TSDF field grid point local pixel association enhancement feature matrix is ​​multiplied with the scale of the TSDF field grid point local pixel association enhancement feature matrix and the square root of the scale to obtain the first TSDF field grid point local pixel association enhancement structure reference value and the second TSDF field grid point local pixel association enhancement structure reference value corresponding to each eigenvalue. The first TSDF field grid point local pixel association enhancement scale transformation value and the second TSDF field grid point local pixel association enhancement scale transformation value corresponding to the value; the first TSDF field grid point local pixel association enhancement structure reference value is divided by the difference between the first TSDF field grid point local pixel association enhancement spatial structure value and the first TSDF field grid point local pixel association enhancement scale transformation value to obtain the first TSDF field grid point local pixel association enhancement transformation adjustment value; the second TSDF field grid point local pixel association enhancement structure reference value is divided by the second TSDF field grid point local pixel association enhancement scale transformation value The difference between the spatial structure value and the local pixel association enhancement scale transformation value of the second TSDF field grid point is calculated to obtain the second TSDF field grid point local pixel association enhancement transformation adjustment value; the weighted sum of the first TSDF field grid point local pixel association enhancement transformation adjustment value and the second TSDF field grid point local pixel association enhancement transformation adjustment value is calculated to obtain each eigenvalue of the optimized TSDF field grid point local pixel association enhancement feature matrix; the optimized TSDF field grid point local pixel association enhancement feature matrix is ​​passed through the rendering module based on the AIGC model to obtain the rendering result.

[0049] Here, the optimization expression of the local pixel correlation enhancement feature matrix of the TSDF field grid point is: ;in, represents the local pixel correlation enhancement feature matrix of the TSDF field grid point, represents the eigenvalues ​​of each position in the local pixel correlation enhancement feature matrix of the TSDF field grid point, represents the width of the local pixel correlation enhancement feature matrix of the TSDF field grid point, represents the height of the local pixel correlation enhancement feature matrix of the TSDF field grid point, represents the number of eigenvalues ​​in the local pixel correlation enhancement feature matrix of the TSDF field grid point, represents the spatial structure value of the local pixel association enhancement of the first TSDF field grid point, represents the local pixel association enhancement spatial structure value of the second TSDF field grid point, represents the first TSDF field grid point local pixel correlation enhancement transformation adjustment value of each position in the first TSDF field grid point local pixel correlation enhancement transformation adjustment matrix, represents the second TSDF field grid point local pixel correlation enhancement transformation adjustment value of each position in the second TSDF field grid point local pixel correlation enhancement transformation adjustment matrix, represents the local pixel correlation enhancement transformation adjustment matrix of the first TSDF field grid point, represents the local pixel correlation enhancement transformation adjustment matrix of the second TSDF field grid point, Indicates adding by position point. It means point multiplication by position. represents the weighted hyperparameter, Represents the optimized TSDF field grid point local pixel correlation enhancement feature matrix.

[0050] That is, in the preferred example, for the spatial structure information of the feature set of the local pixel association enhancement feature matrix of the TSDF field grid point in the high-dimensional space, the scale-based frame transformation of each eigenvalue of the local pixel association enhancement feature matrix of the TSDF field grid point is performed by using the class norm space structured representation of the local pixel association enhancement feature matrix of the TSDF field grid point as a reference window, and the spatial structure-based frame attention weight adjustment of each eigenvalue of the local pixel association enhancement feature matrix of the TSDF field grid point is realized to ensure the invariance of the spatial transformation (translation, scaling and rotation) of the local pixel association enhancement feature matrix of the TSDF field grid point under the interaction of feature space, thereby improving the convergence and generalization effect of generative regression of the feature set of the local pixel association enhancement feature matrix of the TSDF field grid point under the complex spatial structure representation, and improving the image quality of the rendering result obtained by passing it through the rendering module based on the AIGC model. In this way, the connection between each grid point can be better captured, so that more local and global semantic information can be considered during rendering, the rendering quality can be improved, and more intelligent image rendering can be achieved.

[0051] In the embodiment of the present application, the rendering result display module 160 is used to display the rendering result on the interactive interface. Specifically, the rendering results include a depth map, a color map and a normal map, and the final image is rendered and displayed according to the depth map, the color map and the normal map.

[0052] In summary, the real-time image rendering generation system 100 based on generative artificial intelligence according to the embodiment of the present application is explained, which uses generative artificial intelligence-based processing and analysis technology to perform local area division and pixel feature extraction on the implicit surface of the truncated signed distance function TSDF field containing intersection points, and reconstructs the dimension of the local pixel features of each TSDF field grid point, so as to intelligently generate rendering results based on the associated enhanced features of each pixel gradient in the reconstructed features. In this way, the connection between each grid point can be better captured, so that more local and global semantic information can be taken into account during rendering, the rendering quality can be improved, and more intelligent image rendering can be achieved.

[0053] As described above, the real-time image rendering generation system 100 based on generative artificial intelligence according to the embodiment of the present application can be implemented in various terminal devices, such as a server for real-time image rendering generation based on generative artificial intelligence, etc. In one example, the real-time image rendering generation system 100 based on generative artificial intelligence according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the real-time image rendering generation system 100 based on generative artificial intelligence can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the real-time image rendering generation system 100 based on generative artificial intelligence can also be one of the many hardware modules of the terminal device.

[0054] Alternatively, in another example, the real-time image rendering generation system 100 based on generative artificial intelligence and the terminal device may also be separate devices, and the real-time image rendering generation system 100 based on generative artificial intelligence may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

Claims

1. A real-time image rendering generation system based on generative artificial intelligence, characterized in that: include: A module for acquiring parameters of an image to be rendered, used to acquire the image to be rendered and rendering parameters in the interactive interface in response to a display instruction received in the interactive interface; A ray set generating module, used for generating a ray set based on the image to be rendered and the rendering parameters; TSDF field implicit surface pre-generation module, used to pre-generate truncated signed distance function TSDF field implicit surface; An intersection calculation module, used for calculating the intersection of each ray in the ray set and the truncated signed distance function TSDF field implicit surface to obtain the truncated signed distance function TSDF field implicit surface containing the intersection point; A rendering module, used for rendering the TSDF field grid points where the intersection points are located based on the truncated signed distance function TSDF field implicit surface containing the intersection points, and generating a rendering result; A rendering result display module, used to display the rendering result on the interactive interface; wherein the rendering module includes: a TSDF field implicit surface area division unit, used to perform local area division on the truncated signed distance function TSDF field implicit surface containing the intersection point to obtain a set of local areas of the TSDF field grid points; a TSDF field grid point pixel area extraction and reconstruction unit, used to perform pixel feature extraction on the set of local areas of the TSDF field grid points, and perform dimensional reconstruction on the set of extracted local pixel feature vectors of the TSDF field grid points to obtain local pixel features of the TSDF field grid points; a TSDF field grid point pixel association enhancement unit, used to perform association enhancement on the local pixel features of the TSDF field grid points based on pixel gradient distribution to obtain local pixel association enhancement features of the TSDF field grid points; a rendering result generation unit, used to obtain the rendering result based on the local pixel association enhancement features of the TSDF field grid points.

2. The real-time image rendering generation system based on generative artificial intelligence according to claim 1 is characterized in that: The TSDF field implicit surface area division unit is used to: divide the truncated signed distance function TSDF field implicit surface containing the intersection point into local areas according to the TSDF field grid points where the intersection points are located to obtain a set of local areas of the TSDF field grid points.

3. The real-time image rendering generation system based on generative artificial intelligence according to claim 2 is characterized in that: The TSDF field grid point pixel area extraction and reconstruction unit includes: a TSDF field grid point local pixel feature extraction subunit, which is used to pass the set of TSDF field grid point local areas through a grid point local area pixel feature extractor based on a deep neural network model to obtain a set of TSDF field grid point local pixel feature vectors; a TSDF field grid point local pixel feature dimension reconstruction subunit, which is used to reconstruct the dimension of the set of TSDF field grid point local pixel feature vectors according to a local area division method to obtain a TSDF field grid point local pixel feature matrix as the TSDF field grid point local pixel features.

4. The real-time image rendering generation system based on generative artificial intelligence according to claim 3 is characterized in that: The grid point local area pixel feature extractor based on the deep neural network model is a grid point local area pixel feature extractor based on the convolutional neural network model.

5. The real-time image rendering generation system based on generative artificial intelligence according to claim 4 is characterized in that: The TSDF field grid point pixel association enhancement unit includes: a TSDF field grid point local pixel gradient amplitude calculation subunit, which is used to calculate the multi-directional gradient value distribution of each position in the TSDF field grid point local pixel feature matrix, and determine the gradient amplitude value of each position in the TSDF field grid point local pixel feature matrix based on the multi-directional gradient value distribution of each position to obtain the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; a TSDF field grid point local pixel gradient masking subunit, which is used to input the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix into a gated mask module for masking processing to obtain a masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; a TSDF field grid point local pixel enhancement subunit, which is used to use the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix as a weight matrix to perform weighted enhancement on the TSDF field grid point local pixel feature matrix to obtain a TSDF field grid point local pixel association enhancement feature matrix as the TSDF field grid point local pixel association enhancement feature.

6. The real-time image rendering generation system based on generative artificial intelligence according to claim 5 is characterized in that: The TSDF field grid point local pixel gradient amplitude calculation subunit is used to: calculate the gradient amplitude value of each eigenvalue in the TSDF field grid point local pixel feature matrix along the width direction to obtain the TSDF field grid point local pixel width direction characteristic gradient value feature matrix; calculate the gradient amplitude value of each eigenvalue in the TSDF field grid point local pixel feature matrix along the height direction to obtain the TSDF field grid point local pixel height direction characteristic gradient value feature matrix; calculate the square root of the sum of the squares of the eigenvalues ​​at corresponding positions in the TSDF field grid point local pixel width direction characteristic gradient value feature matrix and the TSDF field grid point local pixel height direction characteristic gradient value feature matrix to obtain the TSDF field grid point local pixel characteristic gradient amplitude distribution feature matrix.

7. The real-time image rendering generation system based on generative artificial intelligence according to claim 6 is characterized in that: The TSDF field grid point local pixel gradient masking subunit is used to: compare the eigenvalue of each position in the TSDF field grid point local pixel feature gradient amplitude distribution feature matrix with a preset threshold to obtain the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix; wherein, in response to the eigenvalue being greater than the preset threshold, the eigenvalue greater than the preset threshold is set to one; in response to the eigenvalue being less than or equal to the preset threshold, the eigenvalue less than or equal to the preset threshold is set to zero.

8. The real-time image rendering generation system based on generative artificial intelligence according to claim 7 is characterized in that: The TSDF field grid point local pixel enhancement subunit is used to: perform positional dot multiplication on the masked TSDF field grid point local pixel feature gradient amplitude distribution feature matrix and the TSDF field grid point local pixel feature matrix, and then input the feature matrix after the dot multiplication into a sigmoid function for enhancement processing to obtain the TSDF field grid point local pixel association enhancement feature matrix.

9. The real-time image rendering generation system based on generative artificial intelligence according to claim 8, characterized in that: The rendering result generating unit is used to: pass the TSDF field grid point local pixel association enhancement feature matrix through a rendering module based on the AIGC model to obtain the rendering result.