Object rendering method and apparatus, electronic device, storage medium, and program product
By encapsulating the effects of texture materials and lighting models within a neural network on mobile devices, saving only the model framework and weight parameters, the problem of insufficient storage space on mobile devices is solved, achieving high-quality image rendering effects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2022-12-20
- Publication Date
- 2026-04-21
AI Technical Summary
When achieving high-quality rendering on mobile devices with low-power graphics processors, how to reduce the storage space occupied by texture materials and lighting data has become an urgent problem to be solved.
The effects of texture materials and lighting models are encapsulated in the training results of the neural network, saving only the trained neural network model framework and weight parameters, rather than the original texture materials and lighting data.
It significantly reduces storage space usage while providing higher image rendering quality without increasing computing power requirements.
Smart Images
Figure CN115861510B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, specifically to artificial intelligence technologies such as metaverse, augmented reality, virtual reality, deep learning, and visualization, and particularly to an object rendering method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology
[0002] The Internet technology has now entered a new era, and the rapid development of fields such as the metaverse has put forward new technical requirements for graphics rendering.
[0003] Achieving high-quality rendering on mobile devices with low power consumption of graphics processing units (GPUs) has always been a challenge. High-quality rendering often involves texture materials and lighting data that occupy a lot of storage space. How to reduce the amount of this data occupied by the limited storage space of mobile devices is a problem that needs to be solved by those skilled in the art. Summary of the Invention
[0004] This disclosure provides an object rendering method, apparatus, electronic device, computer-readable storage medium, and computer program product.
[0005] In a first aspect, embodiments of this disclosure propose an object rendering method, comprising: determining an object to be rendered in a target scene; obtaining the three-dimensional point coordinates and direction vector of the object to be rendered; using a rendering model corresponding to the object to be rendered, determining the three-dimensional point color value corresponding to the three-dimensional point coordinates and direction vector, wherein the rendering model is used to characterize the correspondence between the three-dimensional points of the object with corresponding directions and the three-dimensional point color value presented under the rendering effect of texture material and lighting model, wherein the direction of the three-dimensional point is determined based on the three-dimensional point coordinates and direction vector; and completing the rendering of the object to be rendered based on the three-dimensional point color value.
[0006] Secondly, embodiments of this disclosure propose an object rendering apparatus, comprising: a renderable object determination unit configured to determine renderable objects in a target scene; an object parameter acquisition unit configured to acquire the three-dimensional point coordinates and direction vectors of the renderable object; a three-dimensional point color value determination unit configured to determine the three-dimensional point color values corresponding to the three-dimensional point coordinates and direction vectors using a rendering model corresponding to the renderable object, wherein the rendering model is used to characterize the correspondence between the three-dimensional points of the object with corresponding directions and the three-dimensional point color values presented under the rendering effect of texture materials and lighting models, and the direction of the three-dimensional points is determined based on the three-dimensional point coordinates and direction vectors; and a rendering unit configured to complete the rendering of the renderable object based on the three-dimensional point color values.
[0007] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the object rendering method as described in the first aspect.
[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the object rendering method as described in the first aspect when executed.
[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the steps of the object rendering method as described in the first aspect.
[0010] The object rendering solution disclosed herein encapsulates the effects of texture materials and lighting models on the object to be rendered in the training results of the neural network (i.e., the trained model framework and model weight parameters). This allows the system to save only the trained neural network without directly storing the texture materials and lighting data of the object to be rendered. Since the model framework and model weight parameters, as the training results, occupy only a small amount of storage space compared to the original texture materials and lighting data, the system can significantly reduce storage space usage without increasing computing power requirements, making it more suitable for providing higher image rendering quality on mobile devices.
[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0012] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0013] Figure 1 This is an exemplary system architecture to which this disclosure can be applied;
[0014] Figure 2 A flowchart of an object rendering method provided in an embodiment of this disclosure;
[0015] Figure 3 A flowchart of another object rendering method provided in an embodiment of this disclosure;
[0016] Figure 4 A flowchart illustrating a method for storing a minimum number of model parameters provided in this disclosure embodiment;
[0017] Figure 5 A flowchart illustrating a method for determining the object to be rendered in a target scene based on scene dwell time, provided in an embodiment of this disclosure;
[0018] Figure 6 A flowchart illustrating a method for determining the object to be rendered in a target scene based on scene required sharpness, provided in an embodiment of this disclosure;
[0019] Figure 7 A flowchart illustrating a method for determining the object to be rendered in a target scene based on the observation field of a virtual character, as provided in this embodiment of the disclosure;
[0020] Figure 8 This is a schematic diagram of a neural network structure provided in an embodiment of the present disclosure;
[0021] Figure 9 This disclosure provides a structural block diagram of an object rendering apparatus according to an embodiment;
[0022] Figure 10 The disclosed embodiments provide a schematic diagram of the structure of an electronic device suitable for performing an object rendering method. Detailed Implementation
[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.
[0024] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0025] Figure 1 An exemplary system architecture 100 is shown, to which embodiments of the object rendering methods, apparatuses, electronic devices and computer-readable storage media of the present disclosure may be applied.
[0026] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0027] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include image rendering applications, virtual reality applications, and instant messaging applications.
[0028] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.
[0029] Terminal devices 101, 102, 103 and server 105 can provide various services through built-in applications. Taking a virtual reality application that can provide object rendering services as an example, when running this virtual reality application, terminal devices 101, 102, 103 and server 105 can achieve the following effects: First, determine the object to be rendered in the target scene; then, obtain the three-dimensional point coordinates and direction vector of the object to be rendered; next, using the rendering model corresponding to the object to be rendered, determine the three-dimensional point color value corresponding to the three-dimensional point coordinates and direction vector. The rendering model is used to characterize the correspondence between the three-dimensional points of the object with corresponding directions and the three-dimensional point color value presented under the rendering effect of texture materials and lighting models. The direction of the three-dimensional point is determined based on the three-dimensional point coordinates and the direction vector; finally, the rendering of the object to be rendered is completed based on the three-dimensional point color value.
[0030] The object rendering methods provided in the subsequent embodiments of this disclosure can be executed by server 105. Accordingly, the object rendering device is generally also located in server 105. However, if terminal devices 101, 102, and 103 have sufficient computing power and resources, they can also complete the aforementioned calculations performed by server 105 through virtual reality applications installed on them, and thus directly output the same results as server 105.
[0031] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0032] Please refer to Figure 2 , Figure 2 A flowchart of an object rendering method provided in this disclosure embodiment, wherein process 200 includes the following steps:
[0033] Step 201: Determine the objects to be rendered in the target scene;
[0034] This step is intended for the execution body of the object rendering method (e.g., Figure 1 The terminal devices 101, 102, 103, or server 105 shown determine the objects to be rendered in the target scene. The target scene can be all or part of a specific interface to be presented within a target application, while the objects to be rendered are those objects in the target scene that need to be rendered before they can be correctly presented. For example, in a game application's task interface scene, the sky, ground, vegetation, and other user characters in this task interface scene are all objects to be rendered, so that the current user can smoothly complete the task by recognizing the rendered objects.
[0035] It's important to note that the objects to be rendered in the target scene are not fixed. This means that objects in a scene don't necessarily need to be distinguished as individual objects; multiple individual objects can even be merged into a single object, rendered as a whole. Therefore, the question arises: how to segment the objects in the target scene to determine the objects that best meet the actual rendering requirements? The granularity of object segmentation can be determined based on various factors, such as the scene duration, scene clarity requirements, and whether it's within the user's virtual character's observation range. For example, if the scene duration is short (meaning frequent high-speed switching between scenes), a higher granularity can be used, i.e., a coarser segmentation, eliminating the need for finer granularity to improve rendering clarity. Conversely, a lower granularity can be used to provide higher rendering clarity. This principle can also be applied when determining the granularity of object segmentation based on other influencing factors.
[0036] Step 202: Obtain the 3D point coordinates and direction vectors of the object to be rendered;
[0037] Building upon step 201, this step aims to obtain the 3D point coordinates and direction vectors of the object to be rendered from the aforementioned execution entity. It should be understood that each object to be rendered consists of multiple 3D points, and the direction vector corresponding to the coordinates of each 3D point is used to determine the angle at which texture materials and lighting models should act on each 3D point to achieve the correct rendering effect.
[0038] Therefore, this step aims to obtain the 3D point coordinates and orientation vectors of the object to be rendered as input data.
[0039] Step 203: Using the rendering model corresponding to the object to be rendered, determine the 3D point color value corresponding to the 3D point coordinates and direction vector;
[0040] Building upon step 202, this step aims to have the aforementioned executing entity determine the 3D point color values corresponding to the 3D point coordinates and direction vectors using a rendering model corresponding to the object to be rendered. This rendering model characterizes the correspondence between the 3D points of the object with corresponding directions and the 3D point color values rendered under the influence of texture materials and lighting models. The direction of the 3D point is determined based on the 3D point coordinates and direction vectors. To enable the rendering model to characterize this correspondence, it is trained using training samples pre-constructed based on the 3D point coordinates and direction vectors of the object to be rendered and the 3D point color values that should be rendered under the influence of texture materials and lighting models. Specifically, the input samples in this training sample are the 3D point coordinates and direction vectors of the object to be rendered, and the output samples are the color values that each 3D point should display under the influence of texture materials and lighting models.
[0041] In other words, the rendering model used in this step uses the above training samples to save the rendering effects of texture materials and lighting models on the objects to be rendered as model training results in the trained rendering model, specifically in the model framework and model weight parameters after training.
[0042] Step 204: Render the object to be rendered based on the 3D point color values.
[0043] Building upon step 203, this step aims to have the aforementioned execution entity render the object to be rendered based on the three-dimensional point color values.
[0044] The object rendering method provided in this disclosure encapsulates the effects of texture materials and lighting models on the object to be rendered in the training results of the neural network (i.e., the trained model framework and model weight parameters). This allows the method to save only the trained neural network without directly storing the texture materials and lighting data of the object to be rendered. Since the model framework and model weight parameters, as the training results, occupy only a small amount of storage space compared to the original texture materials and lighting data, the method significantly reduces the storage space required without increasing the computing power requirements, making it more suitable for providing higher image rendering quality on mobile devices.
[0045] Please refer to Figure 3 , Figure 3 A flowchart of another object rendering method provided in this disclosure embodiment, wherein process 300 includes the following steps:
[0046] Step 301: Determine the objects to be rendered in the target scene;
[0047] Step 302: Obtain the 3D point coordinates and direction vectors of the object to be rendered;
[0048] Steps 301-302 above and as follows Figure 2 The steps 201-202 shown are the same. For the same parts, please refer to the corresponding parts of the previous embodiment. They will not be repeated here.
[0049] Step 303: Determine the rendering model corresponding to the object to be rendered;
[0050] This step aims to have the aforementioned execution entity determine the rendering model corresponding to the object to be rendered. In other words, to determine the rendering model corresponding to the object to be rendered, a pre-established correspondence between different objects to be rendered and different rendering models is required, so that the target rendering model corresponding to the target object can be determined subsequently based on this correspondence.
[0051] Typically, to provide the highest rendering quality, a dedicated rendering model can be pre-built for each different object. Each rendering model is used only to output the rendering result for its corresponding object. However, considering that running too many rendering models simultaneously with a large number of objects can place a significant performance burden on the device, causing stuttering, it's advisable to pre-build a rendering model for each object type, as long as these minor differences can be ignored. In practical applications, it's only necessary to determine the object type to be rendered and then determine the corresponding rendering model. Alternatively, a many-to-one relationship between objects and rendering models can be established based on other standards to reuse the same rendering model as much as possible, thereby reducing performance requirements.
[0052] Step 304: Reconstruct the pre-recorded model framework and model weight parameters of the rendering model in memory to obtain a rendering model in a usable state;
[0053] Building upon step 303, this step aims to have the aforementioned executing entity reconstruct the pre-recorded model framework and model weight parameters of the rendering model in memory to obtain a usable rendering model. That is, the rendering model determined in step 303 is only a rendering model in a state of awaiting reconstruction and cannot be used normally. It only saves the model parameters to guide the reconstruction in order to save memory usage. Before actual use, it is necessary to reconstruct the model framework and model weight parameters in memory to obtain a usable rendering model. Here, the model framework and model weight parameters are model parameters after training with training samples; that is, the usable rendering model should be the rendering model after training is complete.
[0054] Step 305: Using the rendering model in an available state, determine the 3D point color values corresponding to the 3D point coordinates and direction vectors;
[0055] Building upon step 304, this step aims to have the aforementioned executing entity determine the 3D point color values corresponding to the 3D point coordinates and direction vectors using the rendering model that is in an available state.
[0056] Step 306: Render the object to be rendered based on the 3D point color values.
[0057] Based on the previous embodiment, this embodiment provides an additional implementation scheme through steps 303-304 for determining the rendering model corresponding to the object to be rendered and how to reconstruct it into a usable rendering model, so as to avoid continuous memory occupation due to the normal operation of the rendering model as much as possible, and is more friendly to mobile devices with limited memory resources.
[0058] Building upon the previous embodiment, and considering the possibility of multiple different rendering models, this embodiment further minimizes the storage space required to store these rendering models. Figure 4 A specific baseline-based offset storage scheme is provided to reduce storage space usage. The process 400 includes the following steps:
[0059] Step 401: For different rendering models with the same model framework but different model weight parameters, determine the baseline model weight parameters, and take the rendering model with the baseline model parameters as the baseline model;
[0060] That is, firstly, a certain model weight parameter among multiple rendering models with the same model framework but different model weight parameters is determined as the baseline model weight parameter, and the rendering model with the baseline model parameter is used as the baseline model.
[0061] Among them, the baseline model weight parameters can be the model weight parameters that have the most common parts with the other model weight parameters, so as to minimize the differences.
[0062] Step 402: Calculate the difference in model weight parameters of other rendering models that differ from the baseline model relative to the baseline model weight parameters;
[0063] Based on step 401, calculate the differences in model weight parameters of other rendering models that are different from the baseline model relative to the baseline model weight parameters.
[0064] Step 403: Store the model framework, baseline model weight parameters, and weight parameter differences.
[0065] Building upon step 402, this step aims to reduce storage space usage by having the aforementioned execution entity store only the model framework, baseline model weight parameters, and weight parameter differences.
[0066] To deepen the understanding of how to determine the object to be rendered in the target scene, the following will provide implementation schemes for determining the object to be rendered in the current target scene based on three different influencing factors, so as to meet the actual needs respectively:
[0067] Figure 5 A flowchart of a method for determining the object to be rendered in a target scene based on scene dwell time is shown. The flowchart 500 includes the following steps:
[0068] Step 501: Determine the scene dwell time of the target scene;
[0069] Scene dwell time refers to the duration of time spent in the target scene. The longer the average dwell time in the target scene, the higher the scene dwell time, which means that scene switching is infrequent.
[0070] Step 502: Determine the granularity of object segmentation in the target scene based on scene dwell time;
[0071] Building upon step 501, this step aims to have the aforementioned executing entity determine the object segmentation granularity constituting the target scene based on scene dwell time. The level of scene dwell time is inversely proportional to the size of the object segmentation granularity; that is, the higher the scene dwell time and the smaller the object segmentation granularity (i.e., the finer the object segmentation granularity), the more likely different objects in the scene will be treated as different objects to be rendered and rendered separately. This improves rendering quality and clarity, preventing users from perceiving poor rendering quality when they remain in the target scene for an extended period. Conversely, the lower the scene dwell time and the larger the object segmentation granularity (i.e., the coarser the object segmentation granularity), the more likely different objects in the scene will be treated as the same objects to be rendered and merged for rendering. This avoids unnecessary high-quality rendering within the short presentation duration of the scene (because there is not enough time for users to examine the high-quality rendering effects in detail).
[0072] Step 503: Determine the objects to be rendered in the target scene based on the object segmentation granularity.
[0073] Based on step 502, this step aims to have the aforementioned execution entity determine the objects to be rendered in the target scene according to the object segmentation granularity.
[0074] Different from Figure 5 , Figure 6 The flowchart 600 shows a method for determining the objects to be rendered in a target scene based on the required sharpness of the scene. The process includes the following steps:
[0075] Step 601: Determine the scene sharpness requirements for the target scene;
[0076] Step 602: Determine the granularity of object segmentation in the target scene based on scene clarity requirements;
[0077] Building upon step 601, this step aims to have the aforementioned executing entity determine the object segmentation granularity constituting the target scene based on scene clarity requirements. The clarity required by the scene is inversely proportional to the size of the object segmentation granularity. That is, the higher the clarity required by the scene and the smaller the object segmentation granularity (i.e., the finer the object segmentation granularity), the more likely different objects in the scene will be treated as different objects to be rendered and rendered separately, thereby improving rendering quality and clarity. Conversely, the lower the clarity required by the scene and the larger the object segmentation granularity (i.e., the coarser the object segmentation granularity), the more likely different objects in the scene will be treated as the same objects to be rendered and merged for rendering.
[0078] Step 603: Determine the objects to be rendered in the target scene based on the object segmentation granularity.
[0079] the difference Figure 5 and Figure 6 , Figure 7 A flowchart of a method for determining the object to be rendered in a target scene based on the viewing angle of a virtual character is also shown. The flowchart 700 includes the following steps:
[0080] Step 701: Determine the virtual character's field of view in the target scene;
[0081] This step aims to have the aforementioned executing entity determine the virtual character's field of view in the target scene, and based on the field of view, divide all areas in the target scene into two categories: those within the field of view and those not within the field of view.
[0082] Step 702: Determine the objects to be rendered within the viewing field of view according to the preset first object segmentation granularity;
[0083] Step 703: Determine the objects to be rendered that are outside the viewing field of view according to the preset second object segmentation granularity.
[0084] In other words, for objects to be rendered that are within the viewing field of view, a finer-grained first object segmentation granularity is used to ensure high rendering quality and sharpness for objects within the user's viewing field of view; conversely, for objects to be rendered that are not within the viewing field of view, a coarser-grained second object segmentation granularity is used to ensure lower rendering quality and sharpness for objects within the user's viewing field of view. That is, the first object segmentation granularity is smaller than the second object segmentation granularity.
[0085] Based on any of the above embodiments, after rendering and presenting the object to be rendered in the target scene, when switching to another scene, the rendering result of the target scene can be temporarily stored in memory so that when switching back to the target scene later, the rendering result of the object to be rendered in the target scene cached in memory can be directly presented, thereby improving presentation efficiency and avoiding repeated rendering.
[0086] To enhance understanding, this disclosure also provides a specific implementation scheme based on a particular application scenario. Please refer to the following steps and appendix. Figure 8 The network structure shown is as follows:
[0087] 1) Import the vertex data of the object to be rendered, along with all texture assets and the lighting shader, into the rendering engine;
[0088] 2) Collect training data: the engine-rendered images and the corresponding camera poses;
[0089] 3) Create a neural network with the following structure: Figure 8 As shown:
[0090] The input is the coordinates of a 3D point (x, y, z) and the direction vector (d_x, d_y, d_z), and the output is (r, g, b, \theta).
[0091] 4) For any training image (engine-rendered image), the direction (d_x, d_y, d_z) of the ray passing through each pixel is calculated sequentially based on the camera pose. N points are uniformly sampled along this ray. The coordinates (x, y, z) of these N points and the ray direction (d_x, d_y, d_z) are used as inputs to the neural network to obtain the network's output, which is then used as the predicted pixel value. The pixel value corresponding to the training image is used as the ground truth value, and the L2 loss (least squares error) of the two is compared. The Adam (Adaptive Moment Estimation) optimizer is used to train the neural network until convergence. Therefore, a large amount of texture material and lighting shader are encoded into the weights of the neural network.
[0092] 5) After the neural network is trained, save the weights of the neural network;
[0093] 6) In the rendering engine, import the weights of the neural network. When rendering each frame, calculate the camera angle of the current frame, input it into the neural network, and output the pixel value of the current frame. This avoids saving texture materials (approximately 100MB), replacing them with only the weights of the neural network (10MB).
[0094] Further reference Figure 9As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an object rendering apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0095] like Figure 9 As shown, the object rendering apparatus 900 of this embodiment may include: a renderable object determination unit 901, an object parameter acquisition unit 902, a three-dimensional point color value determination unit 903, and a rendering unit 904. The renderable object determination unit 901 is configured to determine the renderable object in the target scene; the object parameter acquisition unit 902 is configured to acquire the three-dimensional point coordinates and direction vector of the renderable object; the three-dimensional point color value determination unit 903 is configured to determine the three-dimensional point color value corresponding to the three-dimensional point coordinates and direction vector using a rendering model corresponding to the renderable object; the rendering model is used to characterize the correspondence between the three-dimensional points of the object with corresponding directions and the three-dimensional point color values presented under the rendering effect of texture materials and lighting models, wherein the direction of the three-dimensional points is determined based on the three-dimensional point coordinates and direction vector; and the rendering unit 904 is configured to complete the rendering of the renderable object based on the three-dimensional point color values.
[0096] In this embodiment, the specific processing of the object rendering apparatus 900, including the object determination unit 901, the object parameter acquisition unit 902, the three-dimensional point color value determination unit 903, and the rendering unit 904, and the resulting technical effects, can be found in reference to [reference needed]. Figure 2 The relevant descriptions of steps 201-204 in the corresponding embodiments will not be repeated here.
[0097] In some optional implementations of this embodiment, the object rendering apparatus 900 may further include:
[0098] The rendering model determination unit is configured to determine the rendering model corresponding to the object to be rendered.
[0099] The temporary reconstruction unit is configured to reconstruct the rendering model in memory to obtain a usable rendering model from the pre-recorded model frame and model weight parameters; wherein the model frame and model weight parameters are model parameters that have been trained with training samples.
[0100] In some optional implementations of this embodiment, the rendering model determination unit can be further configured as follows:
[0101] Determine the object type of the object to be rendered;
[0102] Determine the rendering model corresponding to the object type.
[0103] In some optional implementations of this embodiment, the object rendering apparatus 900 may further include:
[0104] The baseline model determination unit is configured to determine the baseline model weight parameters for different rendering models with the same model framework but different model weight parameters, and to use the rendering model with the baseline model parameters as the baseline model.
[0105] The weight parameter difference calculation unit is configured to calculate the weight parameter difference of other rendered models that are different from the benchmark model relative to the benchmark model's weight parameters.
[0106] The parameter storage unit is configured to store the model framework, the baseline model weight parameters, and the weight parameter differences.
[0107] In some optional implementations of this embodiment, the object to be rendered determination unit 901 can be further configured to:
[0108] Determine the scene dwell time of the target scene; where scene dwell time refers to the duration of time spent in the target scene.
[0109] The granularity of object segmentation in the target scene is determined based on scene dwell time; the level of scene dwell time is directly proportional to the level of object segmentation granularity.
[0110] The objects to be rendered in the target scene are determined based on the object segmentation granularity.
[0111] In some optional implementations of this embodiment, the object to be rendered determination unit 901 can be further configured to:
[0112] Determine the scene clarity requirements for the target scene;
[0113] The granularity of object segmentation in the target scene is determined based on the scene clarity requirements; the clarity of the scene requirements is directly proportional to the granularity of object segmentation.
[0114] The objects to be rendered in the target scene are determined based on the object segmentation granularity.
[0115] In some optional implementations of this embodiment, the object to be rendered determination unit 901 can be further configured to:
[0116] Determine the field of view of the virtual character in the target scene;
[0117] The objects to be rendered within the viewing field of view are determined according to the preset first object segmentation granularity;
[0118] The objects to be rendered that are outside the viewing field of view are determined according to the preset second object segmentation granularity; wherein, the first object segmentation granularity is smaller than the second object segmentation granularity.
[0119] In some optional implementations of this embodiment, the object rendering apparatus 900 may further include:
[0120] The same scene rendering result reproduction unit is configured to respond to the reflow to the target scene and present the rendering result of the object to be rendered in the target scene cached in memory.
[0121] This embodiment exists as a device embodiment corresponding to the above method embodiment. The object rendering device provided in this embodiment encapsulates the effects of texture materials and lighting models on the object to be rendered in the training results of the neural network (i.e., the trained model framework and model weight parameters). Thus, it is only necessary to save the trained neural network, without directly storing the texture materials and lighting data of the object to be rendered. Since the model framework and model weight parameters, as the training results, occupy only a small amount of storage space compared to the amount of data of the original texture materials and lighting data, the storage space occupied is significantly reduced without increasing the computing power requirements, making it more suitable for providing higher image rendering quality on mobile devices.
[0122] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the object rendering method described in any of the above embodiments when executed.
[0123] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the object rendering method described in any of the above embodiments when executed.
[0124] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when executed by a processor, can implement the object rendering method described in any of the above embodiments.
[0125] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is provided. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0126] like Figure 10 The device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0127] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0128] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as object rendering methods. For example, in some embodiments, the object rendering method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the object rendering method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform object rendering methods by any other suitable means (e.g., by means of firmware).
[0129] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0130] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0133] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0134] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.
[0135] According to the technical solution of this disclosure, by encapsulating the effects of texture materials and lighting models on the object to be rendered in the training results of the neural network (i.e., the trained model framework and model weight parameters), it is possible to save only the trained neural network without directly storing the texture materials and lighting data of the object to be rendered. Since the model framework and model weight parameters, as the training results, occupy only a small amount of storage space compared to the amount of data in the original texture materials and lighting data, the storage space occupied is significantly reduced without increasing the computing power requirements, making it more suitable for providing higher image rendering quality on mobile devices.
[0136] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. An object rendering method, comprising: Determine the objects to be rendered in the target scene; Determine the rendering model corresponding to the object to be rendered; The pre-recorded model framework and model weight parameters of the rendering model are reconstructed in memory to obtain a rendering model in a usable state; wherein, the model framework and the model weight parameters are model parameters after training with training samples. Obtain the 3D point coordinates and direction vector of the object to be rendered; Using the rendering model corresponding to the object to be rendered, the color value of the three-dimensional point corresponding to the three-dimensional point coordinates and the direction vector is determined. The rendering model is a neural network used to characterize the correspondence between the three-dimensional points of the object with corresponding directions and the color values of the three-dimensional points presented under the rendering effect of texture materials and lighting models. The direction of the three-dimensional point is determined based on the three-dimensional point coordinates and the direction vector. The rendering of the object to be rendered is completed based on the three-dimensional point color values.
2. The method according to claim 1, wherein, Determining the rendering model corresponding to the object to be rendered includes: Determine the object type of the object to be rendered; Determine the rendering model corresponding to the object type.
3. The method according to claim 1, further comprising: For different rendering models with the same model framework but different model weight parameters, determine the baseline model weight parameters, and use the rendering model with the baseline model parameters as the baseline model; Calculate the difference in model weight parameters of other rendering models that differ from the baseline model relative to the weight parameters of the baseline model; Store the model framework, the baseline model weight parameters, and the differences in the weight parameters.
4. The method according to claim 1, wherein, The determination of the object to be rendered in the target scene includes: Determine the scene dwell time of the target scene; wherein, the scene dwell time refers to the duration of time spent in the target scene; The granularity of object segmentation in the target scene is determined based on the scene dwell time; wherein, the level of scene dwell time is inversely proportional to the size of the object segmentation granularity. The objects to be rendered in the target scene are determined based on the object segmentation granularity.
5. The method according to claim 1, wherein, The determination of the object to be rendered in the target scene includes: Determine the scene sharpness requirements for the target scene; The granularity of object segmentation in the target scene is determined based on the scene clarity requirements; wherein, the clarity of the scene clarity requirements is directly proportional to the granularity of object segmentation. The objects to be rendered in the target scene are determined based on the object segmentation granularity.
6. The method according to claim 1, wherein, The determination of the object to be rendered in the target scene includes: Determine the field of view of the virtual character in the target scene; The objects to be rendered within the observation field of view are determined according to the preset first object segmentation granularity; The objects to be rendered that are outside the viewing field of view are determined according to a preset second object segmentation granularity; wherein, the first object segmentation granularity is smaller than the second object segmentation granularity.
7. The method according to any one of claims 1-6, further comprising: In response to the reflow to the target scene, the rendering results of the objects to be rendered in the target scene cached in memory are presented.
8. An object rendering apparatus, comprising: The unit for determining the object to be rendered is configured to determine the object to be rendered in the target scene. The rendering model determination unit is configured to determine the rendering model corresponding to the object to be rendered. The temporary reconstruction unit is configured to reconstruct the pre-recorded model framework and model weight parameters of the rendering model in memory to obtain a rendering model in a usable state. The object parameter acquisition unit is configured to acquire the three-dimensional point coordinates and direction vector of the object to be rendered; The three-dimensional point color value determination unit is configured to use a rendering model corresponding to the object to be rendered to determine the three-dimensional point color value corresponding to the three-dimensional point coordinates and the direction vector. The rendering model is a neural network used to characterize the correspondence between the three-dimensional points of the object with corresponding directions and the three-dimensional point color values presented under the rendering effect of texture material and lighting model. The direction of the three-dimensional point is determined based on the three-dimensional point coordinates and the direction vector. The rendering unit is configured to render the object to be rendered based on the three-dimensional point color values.
9. The apparatus according to claim 8, wherein, The rendering model determination unit is further configured to: Determine the object type of the object to be rendered; Determine the rendering model corresponding to the object type.
10. The apparatus according to claim 8, further comprising: The baseline model determination unit is configured to determine the baseline model weight parameters for different rendering models with the same model framework but different model weight parameters, and to use the rendering model with the baseline model parameters as the baseline model. The weight parameter difference calculation unit is configured to calculate the weight parameter difference of the model weight parameters of other rendering models that are different from the benchmark model relative to the weight parameters of the benchmark model; The parameter storage unit is configured to store the model framework, the baseline model weight parameters, and the weight parameter differences.
11. The apparatus according to claim 8, wherein, The object-to-render determination unit is further configured to: Determine the scene dwell time of the target scene; wherein, the scene dwell time refers to the duration of time spent in the target scene; The granularity of object segmentation in the target scene is determined based on the scene dwell time; wherein, the level of scene dwell time is directly proportional to the level of object segmentation granularity. The objects to be rendered in the target scene are determined based on the object segmentation granularity.
12. The apparatus according to claim 8, wherein, The object-to-render determination unit is further configured to: Determine the scene sharpness requirements for the target scene; The granularity of object segmentation in the target scene is determined based on the scene clarity requirements; wherein, the clarity of the scene clarity requirements is directly proportional to the granularity of object segmentation. The objects to be rendered in the target scene are determined based on the object segmentation granularity.
13. The apparatus according to claim 8, wherein, The object-to-render determination unit is further configured to: Determine the field of view of the virtual character in the target scene; The objects to be rendered within the observation field of view are determined according to the preset first object segmentation granularity; The objects to be rendered that are outside the viewing field of view are determined according to a preset second object segmentation granularity; wherein, the first object segmentation granularity is smaller than the second object segmentation granularity.
14. The apparatus according to any one of claims 8-13, further comprising: The same scene rendering result reproduction unit is configured to, in response to the reflow to the target scene, present the rendering result of the object to be rendered in the target scene cached in memory.
15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the object rendering method of any one of claims 1-7.
16. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the object rendering method of any one of claims 1-7.
17. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the object rendering method according to any one of claims 1-7.
Citation Information
Patent Citations
Graphics texture mapping apparatus, operating method thereof, and storage medium
CN109559367A
Voxel map construction method and device, computer readable medium and electronic equipment
CN112927363A