A real-time video rendering optimization method and system based on artificial intelligence
Through high-frequency mapping and normal distribution explicit expression, the problem of poor performance of neural radiation fields in high-frequency transformation is solved, the efficiency and effect of video rendering is improved, and the training speed and rendering quality are achieved.
Patent Information
- Application Number
- CN202411295873.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-14
AI Technical Summary
The existing video rendering technology based on neural radiation fields is poor in color and geometry in high-frequency transformation, and the implicit expression is not intuitive, making it difficult to meet the requirements of real-time rendering.
High-frequency mapping function is used to map the three-dimensional coordinates and viewing direction of the pixels of the viewing image, combined with the explicit expression of normal distribution, and use a multi-layer perceptron MLP to represent the neural radiation field, and determine based on the peak signal-to-noise ratio maximization of the rendering Output scale, train the joint image rendering model.
Improves the color and geometric performance of video rendering in high-frequency transformation, improves training speed and rendering efficiency, and achieves better rendering effects.
Smart Images

Figure CN118828108B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and more specifically, to a real-time video rendering optimization method and system based on artificial intelligence. Background Art
[0002] Video rendering is an important technology in computer image processing, and is widely used in the fields of video production and multimedia processing. It converts the original video material into a smooth, high-quality video image that the audience finally sees through computer processing. With the development of technology, it is more and more widely used in application scenarios such as live broadcast, games, and virtual AR. Three-dimensional image reconstruction, texture mapping, and scene lighting are key steps in video rendering. Among the various image reconstruction technologies in the prior art, artificial intelligence reconstruction technology based on neural networks is being widely used due to its unique advantages. In particular, rendering technology based on neural radiance fields (NeRF) has achieved remarkable results in dynamic scene reconstruction, showing a very broad range of usage scenarios and huge development potential. NeRF uses a multi-layer perceptron (MLP) to take the coordinates of each point in space and its corresponding viewing direction as the input of the neural network to generate a set of output colors and densities, and then accumulates these colors and densities into a two-dimensional image based on volume rendering technology to realize image display. However, the existing technology for rendering new vision based on neural radiance fields is still lacking. Figure 1 In general, the coordinates (x, y, z) are directly input, which will lead to poor color and geometric performance of the rendering results in high-frequency transformations. In addition, the neural radiation field is an implicit expression, which has many problems such as unintuitive expression and training efficiency that does not meet the requirements of real-time rendering. Summary of the invention
[0003] In view of the above problems, the present invention proposes a real-time video rendering optimization method and system based on artificial intelligence. By realizing high-frequency mapping of input coordinates and viewing directions, the rendering results can better fit the data containing high-frequency transformations, and the color and geometry performance in high-frequency transformations is more excellent; and, on the basis of three-dimensional scene reconstruction based on neural radiance field (NeRF), the normal distribution explicit expression is combined to capture and display the three-dimensional scene information, thereby overcoming the shortcomings of the implicit expression of neural radiation field that is not intuitive and difficult to meet the requirements of real-time rendering; thereby, the video rendering including dynamic scene reconstruction has higher training speed and rendering efficiency, and achieves better rendering effect.
[0004] To achieve the above object, the present invention proposes the following technical solutions:
[0005] In a first aspect, the present invention provides a real-time video rendering optimization method based on artificial intelligence, comprising:
[0006] S1, through high frequency mapping function High-frequency mapping of the three-dimensional coordinates of the view pixel points and the viewing direction;
[0007] The mapping of the three-dimensional coordinates of the pixel point is: , through the high frequency mapping function Map the three-dimensional coordinates x, y, and z in the three-dimensional point coordinate x=(x,y,z) respectively;
[0008] The mapping for viewing direction is: , through the high frequency mapping function For 2D viewing direction Viewing angle in and view direction Map them separately;
[0009] Where k is the high-frequency transformation coefficient, which is positively correlated with the frequency of the high-frequency transformation;
[0010] S2, the neural radiation field obtains the color and density output of the view based on the three-dimensional coordinates and two-dimensional viewing direction after high-frequency mapping;
[0011] Based on the multi-layer perceptron MLP, the neural radiation field is represented. ;in, Represents the view color output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping, Represents the view density output of the neural radiation field based on the three-dimensional coordinates and viewing direction after high-frequency mapping;
[0012] S3, obtaining the color output of the view pixel based on the normal distribution of the view pixel;
[0013] The view pixel color output is expressed as: ;
[0014] Where N represents the number of normal distributions covered on the view pixels; , which represents the first pixel covered on the view The transmittance of a normal distribution; , represents the opacity of the view pixel, where and They represent the first The mean and covariance of a normal distribution; is the view pixel at Observed color values under normal distribution;
[0015] S4, determining a final color output L of the view pixel based on the color value output in step S2 and the color value output in step S3;
[0016] in ; Wherein w1 and w2 represent weight factors of the view color values obtained in steps S2 and S3, respectively; wherein w1 and w2 are determined based on maximizing the peak signal-to-noise ratio of the rendering;
[0017] S5, repeating the above steps S1-S4 for multiple view pixels in the space to complete the training of the joint image rendering model;
[0018] S6. Realize real-time rendering of the video image based on the joint image rendering model trained in step S5.
[0019] In a second aspect, the present invention provides a real-time video rendering optimization system based on artificial intelligence, comprising:
[0020] High frequency mapping module, through high frequency mapping function High-frequency mapping of the three-dimensional coordinates of the view pixel points and the viewing direction;
[0021] The mapping of the three-dimensional coordinates of the pixel point is: , through the high frequency mapping function Map the three-dimensional coordinates x, y, and z in the three-dimensional point coordinate x=(x,y,z) respectively;
[0022] The mapping for viewing direction is: , through the high frequency mapping function For 2D viewing direction Viewing angle in and view direction Map them separately;
[0023] Where k is the high-frequency transformation coefficient, which is positively correlated with the frequency of the high-frequency transformation;
[0024] In the first output module, the neural radiation field obtains the color and density output of the view based on the three-dimensional coordinates and two-dimensional viewing direction after high-frequency mapping;
[0025] Based on the multi-layer perceptron MLP, the neural radiation field is represented. ;in, Represents the view color output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping, Represents the view density output of the neural radiation field based on the three-dimensional coordinates and viewing direction after high-frequency mapping;
[0026] A second output module obtains the color output of the view pixel based on the normal distribution of the view pixel;
[0027] The view pixel color output is expressed as: ;
[0028] Where N represents the number of normal distributions covered on the view pixels; , which represents the first pixel covered on the view The transmittance of a normal distribution; , represents the opacity of the view pixel, where and They represent the first The mean and covariance of a normal distribution; is the view pixel at Color observations under normal distribution;
[0029] A third output module, which determines a final color output L of a view pixel based on a color value output by the first output module and a color value output by the second output module;
[0030] in ; Wherein w1 and w2 represent weight factors of the view color values obtained by the second output module and the third output module respectively; wherein w1 and w2 are determined based on maximizing the peak signal-to-noise ratio of the rendering;
[0031] A control module controls a plurality of view pixels in the space to repeatedly complete the processing of the first output module, the second output module and the third output module to complete the training of the joint image rendering model;
[0032] The rendering module realizes real-time rendering of video images based on the joint image rendering model trained in the control module.
[0033] In a third aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is executed by the processor to implement an artificial intelligence-based real-time video rendering optimization method provided in the first aspect of the present invention;
[0034] In a fourth aspect, the present invention provides a computer-readable storage medium, on which is stored a computer program executable by a processor, and the computer program is executed by the processor to implement an artificial intelligence-based real-time video rendering optimization method provided in the first aspect of the present invention.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] The present invention proposes an artificial intelligence-based real-time video rendering optimization method and system, which can realize high-frequency mapping of the three-dimensional coordinates and viewing direction of view pixels in space through high-frequency mapping functions. Compared with the three-dimensional dynamic modeling method in the prior art in which the neural radiation field directly uses the three-dimensional coordinates and viewing direction of view pixels as input, it can better adapt to the application in high-frequency transformation, and improve the performance of video rendering in color and geometry in high-frequency transformation applications; and taking into account the implicit expression defects of the existing neural radiation field, the normal distribution deep sorting superposition characteristics of the combined view pixels, and based on the peak signal-to-noise ratio of the rendering, the proportion of the neural radiation field output based on the mapped input and the normal distribution output based on the pixel points is determined, and a joint image rendering model is trained, so that real-time rendering processing of video images is realized based on the trained joint image rendering model, so that the rendering of the video image is more optimized and better assists video image processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A flow chart of a real-time video rendering optimization method based on artificial intelligence according to an embodiment of the present invention; and
[0038] Figure 2 It is a structural diagram of an artificial intelligence-based real-time video rendering optimization system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0039] Now, exemplary embodiments of the present invention are described with reference to the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely and to fully convey the scope of the present invention to those skilled in the art. The terms used in the exemplary embodiments shown in the accompanying drawings are not intended to limit the present invention. In the accompanying drawings, the same units / elements are marked with the same reference numerals.
[0040] Unless otherwise specified, the terms (including technical terms) used herein have the commonly understood meanings to those skilled in the art. In addition, it is understood that the terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.
[0041] Embodiment 1:
[0042] The present invention provides a real-time video rendering optimization method based on artificial intelligence. Figure 1 The flowchart of the real-time video rendering optimization method based on artificial intelligence according to an embodiment of the present invention comprises:
[0043] S1, through high frequency mapping function High-frequency mapping of the three-dimensional coordinates of the view pixel points and the viewing direction;
[0044] The mapping of the three-dimensional coordinates of the pixel point is: , through the high frequency mapping function Map the three-dimensional coordinates x, y, and z in the three-dimensional point coordinate x=(x,y,z) respectively;
[0045] The mapping for viewing direction is: , through the high frequency mapping function For 2D viewing direction Viewing angle in and view direction Map them separately;
[0046] Where k is the high-frequency transformation coefficient, which is positively correlated with the frequency of the high-frequency transformation;
[0047] S2, the neural radiation field obtains the color and density output of the view based on the three-dimensional coordinates and two-dimensional viewing direction after high-frequency mapping;
[0048] Based on the multi-layer perceptron MLP, the neural radiation field is represented. ;in, Represents the view color output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping, Represents the view density output of the neural radiation field based on the three-dimensional coordinates and viewing direction after high-frequency mapping;
[0049] S3, obtaining the color output of the view pixel based on the normal distribution of the view pixel;
[0050] The view pixel color output is expressed as: ;
[0051] Each pixel in the view can be considered as the result of multiple normal distributions sorted and superimposed in depth, where N represents the number of normal distributions covering the view pixel; , which represents the first pixel covered on the view The transmittance of a normal distribution; , represents the opacity of the view pixel, where and They represent the first The mean and covariance of a normal distribution; is the view pixel at Color observations under normal distribution;
[0052] S4, determining a final color output L of the view pixel based on the color value output in step S2 and the color value output in step S3;
[0053] in ; Wherein w1 and w2 represent weight factors of the view color values obtained in steps S2 and S3, respectively; wherein w1 and w2 are determined based on maximizing the peak signal-to-noise ratio of the rendering;
[0054] The peak signal-to-noise ratio is an indicator to measure the quality of the rendered image. The higher the value, the better the image quality.
[0055] S5, repeating the above steps S1-S4 for multiple view pixels in the space to complete the training of the joint image rendering model;
[0056] Specifically, the rendered image can be compared with the real image, and the rendering loss function can be defined by the difference between them. The model parameters can be continuously adjusted by the optimization algorithm to minimize the loss or determine that the training of the joint image rendering model is completed when a predetermined number of training times is reached. The method for determining the loss function can adopt a method known in the prior art and is not specifically limited here.
[0057] S6. Realize real-time rendering of the video image based on the joint image rendering model trained in step S5.
[0058] Embodiment 2:
[0059] The present invention proposes a real-time video rendering optimization system 200 based on artificial intelligence, such as Figure 2 As shown, including:
[0060] High frequency mapping module, through high frequency mapping function High-frequency mapping of the three-dimensional coordinates of the view pixel points and the viewing direction;
[0061] The mapping of the three-dimensional coordinates of the pixel point is: , through the high frequency mapping function Map the three-dimensional coordinates x, y, and z in the three-dimensional point coordinate x=(x,y,z) respectively;
[0062] The mapping for viewing direction is: , through the high frequency mapping function For 2D viewing direction Viewing angle in and view direction Map them separately;
[0063] Where k is the high-frequency transformation coefficient, which is positively correlated with the frequency of the high-frequency transformation;
[0064] In the first output module, the neural radiation field obtains the color and density output of the view based on the three-dimensional coordinates and two-dimensional viewing direction after high-frequency mapping;
[0065] Based on the multi-layer perceptron MLP, the neural radiation field is represented. ;in, Represents the view color output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping, Represents the view density output of the neural radiation field based on the three-dimensional coordinates and viewing direction after high-frequency mapping;
[0066] A second output module obtains the color output of the view pixel based on the normal distribution of the view pixel;
[0067] The view pixel color output is expressed as: ;
[0068] Each pixel in the view can be considered as the result of multiple normal distributions sorted and superimposed in depth, where N represents the number of normal distributions covering the view pixel; , which represents the first pixel covered on the view The transmittance of a normal distribution; , represents the opacity of the view pixel, where and They represent the first The mean and covariance of a normal distribution; is the view pixel at Color observations under normal distribution;
[0069] A third output module, which determines a final color output L of a view pixel based on a color value output by the first output module and a color value output by the second output module;
[0070] in ; Wherein w1 and w2 represent weight factors of the view color values obtained by the second output module and the third output module respectively; wherein w1 and w2 are determined based on maximizing the peak signal-to-noise ratio of the rendering;
[0071] The peak signal-to-noise ratio is an indicator to measure the quality of the rendered image. The higher the value, the better the image quality.
[0072] A control module controls a plurality of view pixels in the space to repeatedly complete the processing of the first output module, the second output module and the third output module to complete the training of the joint image rendering model;
[0073] Specifically, the rendered image can be compared with the real image, and the rendering loss function can be defined by the difference between them. The model parameters can be continuously adjusted by the optimization algorithm to minimize the loss or determine that the training of the joint image rendering model is completed when a predetermined number of training times is reached. The method for determining the loss function can adopt a method known in the prior art and is not specifically limited here.
[0074] The rendering module realizes real-time rendering of video images based on the joint image rendering model trained in the control module.
[0075] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.
[0076] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0077] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0079] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the present invention is intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0080] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the present invention and its equivalent technologies, the present invention is also intended to include these changes and variations.
Claims
1. A real-time video rendering optimization method based on artificial intelligence, It is characterized in that The steps include: S1, through high frequency mapping function High-frequency mapping of the three-dimensional coordinates of the view pixel points and the viewing direction; The mapping of the three-dimensional coordinates of the pixel point is: , through the high frequency mapping function Map the three-dimensional coordinates x, y, and z in the three-dimensional point coordinate x=(x,y,z) respectively; The mapping for viewing direction is: , through the high frequency mapping function For 2D viewing direction Viewing angle in and view direction Map them separately; Where k is the high-frequency transform coefficient; S2, the neural radiation field obtains the color and density output of the view based on the three-dimensional coordinates and two-dimensional viewing direction after high-frequency mapping; S3, obtaining the color output of the view pixel based on the normal distribution of the view pixel; The view pixel color output is expressed as: ; Where N represents the number of normal distributions covered on the view pixels; , which represents the first pixel covered on the view The transmittance of a normal distribution; , represents the opacity of the view pixel, where and They represent the first The mean and covariance of a normal distribution; is the view pixel at Observed color values under normal distribution; S4, determining a final color output L of the view pixel based on the color value output in step S2 and the color value output in step S3; S5, repeating the above steps S1-S4 for multiple view pixels in the space to complete the training of the joint image rendering model; S6. Realize real-time rendering of the video image based on the joint image rendering model trained in step S5; The neural radiation field obtains the color and density output of the view based on the three-dimensional coordinates and two-dimensional viewing direction after high-frequency mapping, specifically including: representing the neural radiation field based on a multi-layer perceptron MLP, ;in, Represents the view color output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping, Represents the view density output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping.
2. A real-time video rendering optimization method based on artificial intelligence as claimed in claim 1, It is characterized in that The k is positively correlated with the frequency of high-frequency transformation.
3. A real-time video rendering optimization method based on artificial intelligence as claimed in claim 1, It is characterized in that The step of determining the final color output L of the view pixel based on the color value outputted in step S2 and the color value outputted in step S3 specifically includes: ; Wherein w1 and w2 represent the weight factors of the view color values obtained in steps S2 and S3 respectively.
4. A real-time video rendering optimization method based on artificial intelligence as claimed in claim 3, It is characterized in that The values of w1 and w2 are determined based on maximizing the peak signal-to-noise ratio of rendering.
5. A real-time video rendering optimization system based on artificial intelligence, running the real-time video rendering optimization method based on artificial intelligence as claimed in any one of claims 1 to 4, It is characterized in that The system comprises: High frequency mapping module, through high frequency mapping function High-frequency mapping of the three-dimensional coordinates of the view pixel points and the viewing direction; The mapping of the three-dimensional coordinates of the pixel point is: , through the high frequency mapping function Map the three-dimensional coordinates x, y, and z in the three-dimensional point coordinate x=(x,y,z) respectively; The mapping for viewing direction is: , through the high frequency mapping function For 2D viewing direction Viewing angle in and view direction Map them separately; Where k is the high frequency transform coefficient; In the first output module, the neural radiation field obtains the color and density output of the view based on the three-dimensional coordinates and two-dimensional viewing direction after high-frequency mapping; A second output module obtains the color output of the view pixel based on the normal distribution of the view pixel; The view pixel color output is expressed as: ; Where N represents the number of normal distributions covered on the view pixels; , which represents the first pixel covered on the view The transmittance of a normal distribution; , represents the opacity of the view pixel, where and They represent the first The mean and covariance of a normal distribution; is the view pixel at Color observations under normal distribution; a third output module, which determines a final color output L of a view pixel based on a color value output by the first output module and a color value output by the second output module; A control module, controlling a plurality of view pixels in the space to repeatedly complete the processing of the first output module, the second output module and the third output module, so as to complete the training of the joint image rendering model; A rendering module, which realizes real-time rendering of video images based on the joint image rendering model trained in the control module; The neural radiation field of the first output module obtains the color and density output of the view based on the three-dimensional coordinates and the two-dimensional viewing direction after high-frequency mapping, specifically including: representing the neural radiation field based on a multi-layer perceptron MLP, ;in, Represents the view color output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping, Represents the view density output of the neural radiance field based on the three-dimensional coordinates and viewing direction after high-frequency mapping.
6. A real-time video rendering optimization system based on artificial intelligence as claimed in claim 5, It is characterized in that The k is positively correlated with the frequency of high-frequency transformation.
7. A real-time video rendering optimization system based on artificial intelligence as claimed in claim 5, It is characterized in that The third output module determines the final color output L of the view pixel based on the color value output by the first output module and the color value output by the second output module, specifically including: ; wherein w1 and w2 represent weight factors of the view color values obtained by the first output module and the second output module respectively.
8. A real-time video rendering optimization system based on artificial intelligence as claimed in claim 7, It is characterized in that The values of w1 and w2 are determined based on maximizing the peak signal-to-noise ratio of rendering.
