Model training method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of computer vision, and provides a model training method and device, electronic equipment and a storage medium. The method comprises the following steps: starting a plurality of processes of a plurality of video card loads at the beginning of training; when any process serves as a control process, control information is broadcasted to the rendering process, and the aliasing sequence is output to the summarizing process; when serving as a rendering process, driving the corresponding sub-model to finish rendering according to the instruction of the control information, and outputting a rendering result to the summarizing process; when serving as a summarizing process, aliasing rendering results according to an aliasing sequence, calculating gradient information of each sub-model according to a final rendering result, and outputting the gradient information to a rendering process; and when serving as a rendering process, updating parameters of the corresponding sub-model according to the gradient information. According to the method, the training amount of the large model can be reduced, the training efficiency of the large model is improved, and the probability that the training cannot be carried out is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a model training method, device, electronic device and storage medium. Background Art

[0002] 3D reconstruction is an important research direction in computer vision. Its goal is to reconstruct the geometric shape of a 3D scene or object in 3D space by capturing 2D images from different perspectives, and to support sampling of 3D space at certain sampling positions and sampling perspectives. This process involves knowledge in multiple fields such as image processing, machine learning, and optimization algorithms, and plays an important role in multiple application scenarios such as virtual reality, augmented reality, and unmanned driving.

[0003] Since large models have excellent performance in the field of computer vision, the use of large models has become one of the options for 3D reconstruction. Large models refer to large-scale neural network models, which usually require the use of distributed computing, data parallelism and other technologies to reduce training time and control training resource consumption during training. When training large models, the existing technology has the disadvantages of large training volume, low efficiency, and easy occurrence of problems that make training impossible. How to reduce the training volume of large models, improve the training efficiency of large models, and reduce the probability of problems that make training impossible has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] In view of this, the present disclosure proposes a model training method, device, electronic device and storage medium. The model training method of the present disclosure can reduce the training amount of large models, improve the training efficiency of large models, and reduce the probability of problems that cause training to be unable to proceed.

[0005] According to one aspect of the present disclosure, a model training method is provided, the method is used to train a large model, the method comprising: at the beginning of any training of the large model, starting multiple processes, the multiple processes are loaded by multiple graphics cards, and each graphics card loads at least one process; when any one of the processes is used as a control process, the control process determines whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling perspective, and broadcasts the control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and outputs the aliasing order to the summary process; when any one of the processes is used as a rendering process , when the control information indicates that the sub-model corresponding to the rendering process participates in rendering, the rendering process drives the corresponding sub-model to complete rendering and outputs the rendering result to the summary process; when any one process acts as a summary process, the summary process aliases the received rendering results according to the aliasing order to obtain the final rendering result, calculates the gradient information of each sub-model according to the final rendering result and outputs it to the rendering process corresponding to each sub-model; when any one process acts as a rendering process, the rendering process updates the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete rendering next time.

[0006] In a possible implementation, the method also includes: any one of the multiple processes is used as a control process, at least one of the multiple processes is used as a summary process, and two or more of the multiple processes are used as rendering processes; a process can be used as one or more of a control process, a rendering process, and a summary process.

[0007] In one possible implementation, the large model is used to sample the three-dimensional space. Before the first training of the large model begins, or after the large model changes, the method further includes: dividing the three-dimensional space into multiple subspaces, each subspace corresponding to a submodel, and determining the progress of each graphics card load according to the scale of each submodel.

[0008] In one possible implementation, the large model is a Gaussian splash model, and the three-dimensional space is divided into multiple subspaces, including: taking the three-dimensional space as a node to be divided, and searching for a division surface of the node that meets a first preset condition according to the weight of each Gaussian sphere included in the node; based on the searched division surface, dividing the space corresponding to the node into Y subspaces, where Y is an integer greater than 1; taking each subspace as a node to be divided, repeatedly executing the steps of searching for a division surface of the node that meets the first preset condition according to the weight of each Gaussian sphere included in the node and subsequent steps until the number of space divisions reaches a preset first threshold.

[0009] In a possible implementation, the first preset condition includes: the weights of the Y subspaces obtained after the nodes are split based on the splitting plane are closest, or the difference between the weights of every two subspaces obtained after the nodes are split based on the splitting plane is less than a preset second threshold.

[0010] In a possible implementation, the calculating the gradient information of each sub-model according to the final rendering result and outputting it to the rendering process corresponding to each sub-model includes: outputting the gradient information of each sub-model to the rendering process corresponding to each sub-model according to the aliasing order.

[0011] In a possible implementation, determining whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models based on the sampling position and the sampling angle includes: determining a light cone based on the sampling position and the sampling angle; judging whether the light cone illuminates each subspace, determining the sub-model corresponding to the subspace that is not illuminated as a sub-model that does not participate in rendering, and determining the sub-model corresponding to the illuminated subspace as a sub-model that participates in rendering; and determining the aliasing order based on the order in which the light cone illuminates the subspaces corresponding to the sub-models that participate in rendering.

[0012] In a possible implementation, the gradient information of each sub-model is calculated based on the final rendering result and outputted to the rendering process corresponding to each sub-model, including: calculating the loss of the large model in this training process based on the final rendering result and the image true value corresponding to the sampling position and the sampling perspective; and calculating the gradient information of the loss relative to the rendering result of each sub-model based on the loss and the rendering result of each sub-model as the gradient information corresponding to each sub-model.

[0013] According to another aspect of the present disclosure, a model training device is provided, the device is used to train a large model, the device includes multiple processes, the multiple processes are started at the beginning of any training of the large model, the multiple processes are loaded by multiple graphics cards, each graphics card loads at least one process, any one of the processes is used as a control process to determine whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling perspective, and broadcast the control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and output the aliasing order to the summary process; any one of the processes is used as a rendering process Used to, when the control information indicates that the sub-model corresponding to the rendering process participates in rendering, drive the sub-model corresponding to the rendering process to complete rendering, and output the rendering result to the summary process; when any one of the processes is used as the summary process, it is used to alias the received rendering results according to the aliasing order to obtain the final rendering result, calculate the gradient information of each sub-model according to the final rendering result and output it to the rendering process corresponding to each sub-model; when any one of the processes is used as the rendering process, it is also used to update the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete rendering next time.

[0014] In a possible implementation, any one of the multiple processes serves as a control process, at least one of the multiple processes serves as a summary process, and two or more of the multiple processes serve as rendering processes; a process can serve as one or more of a control process, a rendering process, and a summary process.

[0015] In one possible implementation, the large model is used to sample the three-dimensional space, and the device also includes a segmentation module, which is used to: before the first training of the large model begins, or after the large model changes, divide the three-dimensional space into multiple subspaces, each subspace corresponds to a sub-model, and determine the progress of each graphics card load according to the scale of each sub-model.

[0016] In one possible implementation, the large model is a Gaussian splash model, and the three-dimensional space is divided into multiple subspaces, including: taking the three-dimensional space as a node to be divided, and searching for a division surface of the node that meets a first preset condition according to the weight of each Gaussian sphere included in the node; based on the searched division surface, dividing the space corresponding to the node into Y subspaces, where Y is an integer greater than 1; taking each subspace as a node to be divided, repeatedly executing the steps of searching for a division surface of the node that meets the first preset condition according to the weight of each Gaussian sphere included in the node and subsequent steps until the number of space divisions reaches a preset first threshold.

[0017] In a possible implementation, the first preset condition includes: the weights of the Y subspaces obtained after the nodes are split based on the splitting plane are closest, or the difference between the weights of every two subspaces obtained after the nodes are split based on the splitting plane is less than a preset second threshold.

[0018] In a possible implementation, the calculating the gradient information of each sub-model according to the final rendering result and outputting it to the rendering process corresponding to each sub-model includes: outputting the gradient information of each sub-model to the rendering process corresponding to each sub-model according to the aliasing order.

[0019] In a possible implementation, determining whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models based on the sampling position and the sampling angle includes: determining a light cone based on the sampling position and the sampling angle; judging whether the light cone illuminates each subspace, determining the sub-model corresponding to the subspace that is not illuminated as a sub-model that does not participate in rendering, and determining the sub-model corresponding to the illuminated subspace as a sub-model that participates in rendering; and determining the aliasing order based on the order in which the light cone illuminates the subspaces corresponding to the sub-models that participate in rendering.

[0020] In a possible implementation, the gradient information of each sub-model is calculated based on the final rendering result and outputted to the rendering process corresponding to each sub-model, including: calculating the loss of the large model in this training process based on the final rendering result and the image true value corresponding to the sampling position and the sampling perspective; and calculating the gradient information of the loss relative to the rendering result of each sub-model based on the loss and the rendering result of each sub-model as the gradient information corresponding to each sub-model.

[0021] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0022] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0024] According to the model training method of the embodiment of the present disclosure, at the beginning of any training of the large model, multiple processes are started, and the multiple processes are loaded by multiple graphics cards. Each graphics card loads at least one process, so that the training of the large model is completed by multiple graphics cards in a distributed manner, which can reduce the physical implementation pressure of the large model training; when any one of the processes is used as a control process, the control process determines whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling perspective, and broadcasts the control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and outputs the aliasing order to the summary process ... When a rendering process is used, if the control information indicates that the sub-model corresponding to the rendering process participates in the rendering, the rendering process drives the corresponding sub-model to complete the rendering and outputs the rendering result to the summary process; when any one process is used as a summary process, the summary process aliases the received rendering result according to the aliasing order to obtain the final rendering result, calculates the gradient information of each sub-model according to the final rendering result and outputs it to the rendering process corresponding to each sub-model; when any one process is used as a rendering process, the rendering process updates the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete the rendering next time. Each sub-model corresponds to a rendering process, so that the rendering of different sub-models can be parallelized, improving the training efficiency of the model; since the sub-model does not need to be expanded, the model training method disclosed in the present invention can reduce the training amount of large models and further improve the training efficiency of large models; at the same time, the scale of the sub-model is more controllable, and the scale of the process loaded by each graphics card is also more controllable, reducing the probability of problems that cause training to fail. In summary, the model training method of the embodiment of the present disclosure can reduce the training amount of large models, improve the training efficiency of large models, and reduce the probability of problems that cause training to be unable to proceed.

[0025] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0027] Figure 1 An exemplary application scenario of the model training method according to an embodiment of the present disclosure is shown.

[0028] Figure 2 A schematic diagram showing the process of the model training method according to an embodiment of the present disclosure.

[0029] Figure 3 An example of an inter-process communication method according to an embodiment of the present disclosure is shown.

[0030] Figure 4 An example of a division method of a three-dimensional space according to an embodiment of the present disclosure is shown.

[0031] Figure 5 An exemplary structural diagram of a model training device according to an embodiment of the present disclosure is shown.

[0032] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0034] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0036] The following takes the training of the Gaussian-Splatting model as an example to introduce the large model training method of the existing technology.

[0037] When using this method to train the Gaussian splash model, the three-dimensional space is divided into different regions, each corresponding to a sub-model. Each region is progressively expanded, and the sub-model corresponding to the region is also expanded, while the training perspectives associated with the region corresponding to the sub-model are collected. An expanded region and its associated training perspectives constitute a training task, which is used to train the sub-model corresponding to the region, and the training task can be loaded by a single graphics card. Different training tasks are independent of each other, and there is no need for data exchange between different training tasks, that is, the entire large model is disassembled into several independent sub-models, and each sub-model is trained separately. After all sub-models are trained, the large model is also trained.

[0038] The disadvantage of this method is that the extent of regional expansion is difficult to predict, and the carrying capacity of a single graphics card is limited. Before a region expands, the sub-model corresponding to the region can be carried by the graphics card. After the region expands, the scale of the sub-model corresponding to the region may exceed the carrying capacity of the graphics card. In this case, the training of this sub-model will not be able to continue. In addition, the expansion of sub-models and the collection of training perspectives inevitably increase the degree of model redundancy and the degree of reuse of training perspectives, which will lead to an increase in the overall training volume and reduce training efficiency. How to reduce the training volume of large models, improve the training efficiency of large models, and reduce the probability of problems that make training impossible has become a technical problem that needs to be solved urgently in this field.

[0039] In view of this, the present disclosure proposes a model training method, device, electronic device and storage medium. The model training method of the present disclosure can reduce the training amount of large models, improve the training efficiency of large models, and reduce the probability of problems that cause training to be unable to proceed.

[0040] Figure 1 An exemplary application scenario of the model training method according to an embodiment of the present disclosure is shown.

[0041] like Figure 1 As shown, the model training method disclosed in the present invention is mainly used to train a large model so that the large model can realize the sampling function of the three-dimensional space at a certain sampling position and sampling angle in three-dimensional reconstruction. The application scenario may include a large model to be trained, multiple graphics cards, and a three-dimensional space including the geometric shape of the reconstructed scene or object.

[0042] By executing the model training method disclosed in the present invention, the large model to be trained can be divided into multiple sub-models and driven by multiple processes (not shown) distributedly deployed on the graphics card. Correspondingly, the three-dimensional space is also divided into multiple sub-spaces corresponding to the multiple sub-models. In a training process, the sub-model involved in rendering samples the sub-space corresponding to itself according to a certain sampling position and sampling perspective, and renders to obtain a rendering result in the form of an image. The rendering results are aliased to obtain the final rendering result, which is also the sampling result of the three-dimensional space at a certain sampling position and sampling perspective. The final rendering result can be used to adjust the parameters of the sub-model. Repeat the above training process until the model converges, and the training of the large model can be completed.

[0043] Figure 2 A schematic diagram showing the process of the model training method according to an embodiment of the present disclosure.

[0044] like Figure 2 In a possible implementation, the present disclosure proposes a model training method, which is used to train a large model, and the method includes:

[0045] Step S20, when any training of the large model starts, multiple processes are started, the multiple processes are loaded by multiple graphics cards, and each graphics card loads at least one process;

[0046] Step S21, when any process serves as a control process, the control process determines whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling perspective, and broadcasts the control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and outputs the aliasing order to the summary process;

[0047] Step S22, when any process is used as a rendering process, if the control information indicates that the sub-model corresponding to the rendering process participates in rendering, the rendering process drives the corresponding sub-model to complete rendering, and outputs the rendering result to the aggregation process;

[0048] Step S23, when any one of the processes is used as a summary process, the summary process performs aliasing on the received rendering results according to the aliasing order to obtain a final rendering result, calculates the gradient information of each sub-model according to the final rendering result, and outputs it to the rendering process corresponding to each sub-model;

[0049] Step S24, when any one of the processes is used as a rendering process, the rendering process updates the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete rendering next time.

[0050] For example, a large model may require multiple training iterations to reach convergence conditions. In any training process, the above steps S20 to S24 may be performed. The method uses multiple graphics cards to train a large model, and multiple graphics cards load multiple processes, and each graphics card loads at least one process.

[0051] In a possible implementation, the method also includes: any one of the multiple processes serves as a control process, at least one of the multiple processes serves as a summary process, and two or more of the multiple processes serve as rendering processes; a process can serve as one or more of a control process, a rendering process, and a summary process.

[0052] For example, when a process has the task distribution function, the process can be used as a control process. When a process has the sub-model driving function, the process can be used as a rendering process. When a process has the result aggregation function, the process can be used as an aggregation process. A single process can be used as at least one of the control process, rendering process, and aggregation process at the same time. There can be two or more rendering processes, only one control process, and at least one aggregation process.

[0053] The number of rendering processes can be the same as the number of sub-models included in the large model. Each rendering process corresponds to a sub-model and is used to drive its corresponding sub-model. In this case, the training of the large model is split into the training of multiple sub-models, and the training of each sub-model is completed by a graphics card, that is, multiple sub-models are distributed and deployed on the graphics card.

[0054] Those skilled in the art should understand that if the load capacity of a single graphics card is strong enough, a single graphics card can also load multiple rendering processes at the same time. As long as all the processes loaded by each graphics card do not exceed the load capacity of the graphics card, the embodiments of the present disclosure do not limit the specific way in which the graphics card loads the processes.

[0055] After determining the graphics card used for large model training and the processes loaded by each graphics card, the training of the large model can be started. In step S20, multiple processes can be started at the beginning of any training of the large model. Since a process can be used as a rendering process and also as a control process and / or a summary process, the number of threads started in a single training can be greater than or equal to the number of sub-models.

[0056] Among them, after the control process is started, step S21 can be directly started. In the embodiment of the present disclosure, the sub-model is mainly used to render the rendering result in the form of an image according to the sampling position and the sampling perspective. After the rendering results of each sub-model are aliased, it is the final rendering result of the large model. Aliasing refers to the process of superimposing the rendering results of multiple sub-models. Before the start of a training of the large model, the user can pre-specify the sampling position and the sampling perspective. It can be understood that when the sampling position and the sampling perspective are certain, not all sub-models must participate in the rendering. Therefore, in step S21, after the control process is started, it can determine whether each sub-model included in the large model participates in the rendering and the aliasing order of the rendering results of the sub-model according to the sampling position and the sampling perspective. The aliasing order refers to the order in which the rendering results of multiple sub-models participating in the rendering are superimposed. The control process can broadcast the control information indicating whether each sub-model participates in the rendering to the rendering process corresponding to each sub-model, and output the aliasing order to the summary process (equivalent to realizing the task distribution function, distributing the rendering task to the rendering process, and distributing the result summary task to the summary process). An exemplary determination method of the control information and the aliasing order is given later.

[0057] Those skilled in the art should understand that the purpose of each process can be predetermined before training and the purpose of each process can be kept unchanged during training, or the purpose of each process can be predetermined before training and the purpose of the process can be changed according to the load of the graphics card during training, or the purpose of the process can be determined according to the load of the graphics card during training. For example, assuming that process 1 and process 2 are loaded by different graphics cards, process 1 is set as a summary process before training. After the training starts, if it is detected that the occupancy rate of the graphics card loaded with process 1 is too high, while the occupancy rate of the graphics card loaded with process 2 is low, the summary process can be changed to process 2. The embodiments of the present disclosure do not limit the method of setting the purpose of each process.

[0058] Each rendering process executes step S22. When the control information indicates that the sub-model corresponding to the rendering process participates in the rendering, the corresponding sub-model is driven to complete the rendering (equivalent to realizing the sub-model driving function). The sub-model can use the updated parameters in the last training process when rendering. The rendering process can output the rendering result of the sub-model to the summary process. The rendering result can be in the form of an image. The summary process can execute step S23 to alias the rendering result according to the aliasing order to obtain the final rendering result.

[0059] Among them, when the rendering process transmits the rendering result, the identifier of the rendering process can be transmitted synchronously, so that after the summary process receives the rendering result, it can determine which sub-model the rendering result is, so that the aliasing of the rendering result can be completed in accordance with the aliasing order. At this time, multiple rendering processes can be carried out in parallel, which can improve the training efficiency of the model.

[0060] Those skilled in the art should understand that other methods can also be used to ensure that aliasing is performed in the aliasing order. For example, the summary process can also actively query the rendering processes in the aliasing order to see whether the rendering results are obtained. After querying that a certain rendering process has obtained the rendering result, the rendering result is obtained. The rendering result obtained is the rendering result of the sub-model corresponding to the rendering process. Then continue to query whether the next rendering process in the aliasing order has obtained the rendering result, until the rendering result of the sub-model corresponding to the rendering process at the end of the aliasing order is also obtained by the summary process. The disclosed embodiment does not limit what information is specifically transmitted between the rendering process and the summary process, as long as the summary process can obtain the rendering result and ensure that the rendering result is aliased in the aliasing order (equivalent to realizing the result summary function).

[0061] When there is only one summary process, the summary process is responsible for aliasing the rendering results of all sub-models participating in the rendering. Therefore, the aliasing order output by the control process to the summary process also includes the aliasing order of the rendering results of all sub-models participating in the rendering, and each rendering process transmits the rendering results to the summary process.

[0062] When there are multiple summary processes, the correspondence between each summary process and the sub-model can be set first, so that each sub-model participating in the rendering corresponds to a unique summary process, different summary processes correspond to different sub-models, and each summary process is responsible for aliasing the rendering results of the corresponding sub-model. Therefore, the aliasing order output by the control process to any summary process may include the aliasing order of the rendering results of the sub-model corresponding to the summary process. The control information can also indicate the correspondence between the sub-model and the summary process, and the rendering process can transmit the rendering results of the sub-model to the summary process corresponding to the sub-model. In this way, the data transmission cost of the model training process can be reduced.

[0063] At this point, the forward reasoning work of the model training has been completed, and the reverse parameter update work of the model training can be continued. In step S23, the summary process is also responsible for calculating the gradient information of each sub-model according to the final rendering result and transmitting it to the rendering process corresponding to each sub-model. For example, the gradient information of sub-model 1 can be transmitted to the rendering process 1 corresponding to sub-model 1, and each rendering process only needs to receive the gradient information of the sub-model corresponding to itself.

[0064] After receiving the gradient information, the rendering process can execute step S24 to update the parameters of the sub-model corresponding to the rendering process according to the received gradient information. The updated parameters are used when driving the sub-model to complete rendering next time. Each rendering process only needs to update the parameters of the sub-model corresponding to itself. After the parameters of a single sub-model are updated, the training of the sub-model ends; after the parameters of all sub-models are updated, the training of the large model ends.

[0065] If the large model has not reached the convergence condition after the current training, the next training can be started. Before the next training starts, the user can change the large model, adjust the sampling position and / or sampling angle, etc. If the user does not adjust the sampling position and sampling angle, and does not change the large model, the next training can skip steps S20 and S21 and start from step S22. If the user adjusts the sampling position or sampling angle, or changes the large model, the next training can start from step S21.

[0066] Figure 3 Table 1 shows an example of the process timing according to an embodiment of the present disclosure. Figure 3 Table 1 and Table 1 introduce an exemplary inter-process communication method of the present disclosure.

[0067] Table 1

[0068]

[0069] like Figure 3As shown, the execution model training method uses graphics cards 0 to 3. The large model may include sub-model 0, sub-model 1, sub-model 2, and sub-model 3. Assume that graphics card 0 loads process 0, which is the rendering process of sub-model 0 and also the control process. Graphics card 1 loads process 1, which is the rendering process corresponding to sub-model 1 and also the summary process. Graphics card 2 loads process 2, which is the rendering process corresponding to sub-model 2. Graphics card 3 loads process 3, which is the rendering process corresponding to sub-model 3.

[0070] At the beginning of training, first execute step S20 to start process 0 to process 3. After process 0 is started, execute step S21 to determine that sub-model 0, sub-model 1, and sub-model 3 participate in rendering. The aliasing order of the rendering result is sub-model 3, sub-model 1, and sub-model 0. The summary process is process 1. Process 0 can generate control information and aliasing order as shown in Table 1, and processes 1 to 3 wait temporarily. Process 0 can broadcast control information to the remaining processes (process 1, process 2, process 3) and output the aliasing order to the summary process (process 1).

[0071] Next, step S22 is executed. Process 0, as a rendering process, can determine that its corresponding sub-model 0 participates in rendering according to the instruction of the control information. Process 0 can drive sub-model 0 to render and send the rendering result to process 1. Similarly, after receiving the control information, process 1 and process 3 also drive sub-model 1 and sub-model 3 to render respectively and send the rendering result to process 1. Process 2, as a rendering process, can determine that its corresponding sub-model 2 does not participate in rendering according to the instruction of the control information after receiving the control information. Therefore, process 2 does not drive the sub-model to render and continues to wait.

[0072] Next, step S23 is executed, and process 1 receives rendering results as a summary process. After all rendering results indicated by the aliasing order have been obtained, aliasing rendering results is started. According to the aliasing order shown in Table 1, the aliasing order of process 3 is the earliest, the aliasing order of process 1 is the second, and the aliasing order of process 0 is the last. Process 1 can calculate the gradient information of sub-model 0, sub-model 1, and sub-model 3 according to the final rendering result obtained by aliasing, send the gradient information of sub-model 0 to process 0, and send the gradient information of sub-model 3 to process 3. Process 0, process 2, and process 3 wait temporarily.

[0073] Next, step S24 is executed. Process 1 can update the parameters of sub-model 1 according to the gradient information of sub-model 1. Process 0 receives the gradient information of sub-model 0 from process 1 and updates the parameters of sub-model 0 according to the gradient information of sub-model 0. Process 3 receives the gradient information of sub-model 3 from process 1 and updates the parameters of sub-model 3 according to the gradient information of sub-model 3. Process 2 does not receive the gradient information, so it continues to wait. After the parameters of the sub-models involved in rendering are updated, the next training can be started.

[0074] According to the model training method of the embodiment of the present disclosure, at the beginning of any training of the large model, multiple processes are started, and the multiple processes are loaded by multiple graphics cards. Each graphics card loads at least one process, so that the training of the large model is completed by multiple graphics cards in a distributed manner, which can reduce the physical implementation pressure of the large model training; when any one of the processes is used as a control process, the control process determines whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling perspective, and broadcasts the control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and outputs the aliasing order to the summary process ... When a rendering process is used, if the control information indicates that the sub-model corresponding to the rendering process participates in the rendering, the rendering process drives the corresponding sub-model to complete the rendering and outputs the rendering result to the summary process; when any one process is used as a summary process, the summary process aliases the received rendering result according to the aliasing order to obtain the final rendering result, calculates the gradient information of each sub-model according to the final rendering result and outputs it to the rendering process corresponding to each sub-model; when any one process is used as a rendering process, the rendering process updates the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete the rendering next time. Each sub-model corresponds to a rendering process, so that the rendering of different sub-models can be parallelized, improving the training efficiency of the model; since the sub-model does not need to be expanded, the model training method disclosed in the present invention can reduce the training amount of large models and further improve the training efficiency of large models; at the same time, the scale of the sub-model is more controllable, and the scale of the process loaded by each graphics card is also more controllable, reducing the probability of problems that cause training to fail. In summary, the model training method of the embodiment of the present disclosure can reduce the training amount of large models, improve the training efficiency of large models, and reduce the probability of problems that cause training to be unable to proceed.

[0075] In one possible implementation, the large model is a Gaussian splatter model.

[0076] Those skilled in the art should understand that the large model may also be of other types, as long as it can be divided into multiple sub-models and each sub-model can be rendered according to the sampling position and sampling perspective. The embodiments of the present disclosure do not limit the specific type of the large model.

[0077] In a possible implementation, the large model is used to sample the three-dimensional space. Before the first training of the large model begins, or after the large model changes, the method further includes:

[0078] The three-dimensional space is divided into multiple subspaces, each subspace corresponds to a submodel, and the progress of each graphics card load is determined according to the scale of each submodel.

[0079] For example, the large model to be trained in the present disclosure is used to sample the three-dimensional space to achieve the sampling function of the three-dimensional space at a certain sampling position and sampling angle in the three-dimensional reconstruction. The large model is divided into multiple sub-models for training respectively. Therefore, before the first training of the large model begins, the three-dimensional space can also be divided into multiple sub-spaces so that each sub-space corresponds to a sub-model.

[0080] After the macro model changes, the original three-dimensional space division method may no longer be applicable. Therefore, after the macro model changes, the three-dimensional space can be re-divided into multiple subspaces.

[0081] After segmentation, the volumes of the subspaces may be inconsistent, so the scales of the submodels corresponding to the subspaces may also be inconsistent. The process of each graphics card load can be determined according to the scale of each submodel and the load capacity of the graphics card. It should be noted that the process of each graphics card load should be within the load capacity of the graphics card. The example of the graphics card load process has been described above and will not be repeated here.

[0082] The segmentation of the three-dimensional space can be realized based on the existing technology, and the present disclosure does not limit the specific segmentation method of the three-dimensional space. The following takes the Gaussian splash model as an example to introduce an exemplary segmentation method of the three-dimensional space.

[0083] In one possible implementation, the large model is a Gaussian splash model that divides the three-dimensional space into multiple subspaces, including:

[0084] Taking the three-dimensional space as a node to be segmented, searching for a segmentation surface of the node that meets a first preset condition according to the weight of each Gaussian sphere included in the node;

[0085] Based on the searched splitting surface, the space corresponding to the node is split into Y subspaces, where Y is an integer greater than 1;

[0086] Each subspace is taken as a node to be segmented, and the steps of searching for a segmentation surface of the node that meets a first preset condition and subsequent steps are repeated according to the weight of each Gaussian ball included in the node, until the number of space segmentation times reaches a preset first threshold.

[0087] Figure 4 An example of a division method of a three-dimensional space according to an embodiment of the present disclosure is shown.

[0088] like Figure 4 As shown, the segmentation of the three-dimensional space can be completed with the help of a tree structure. The three-dimensional space ABCD-EFGH is used as the node 0 to be segmented, that is, the root node of the tree. When the large model is a Gaussian splash model, the three-dimensional space may include multiple Gaussian balls (not shown). The function of the Gaussian balls is similar to that of points in a point cloud. When the number of Gaussian balls is sufficient, the geometric shape of a three-dimensional scene or object can be reconstructed in the three-dimensional space.

[0089] The weight of each Gaussian ball can be preset by the user. According to the weight of each Gaussian ball included in the node, a splitting plane of the node that meets the first preset condition can be searched. After the splitting plane is searched, the space corresponding to the node is split into Y subspaces based on the splitting plane. Figure 4 In the example, Y=2, when the three-dimensional space ABCD-EFGH is used as a node, the searched splitting surface can be the plane IJKL, and the two subspaces obtained by splitting once can be the subspace ABCD-IJKL and the subspace EFGH-IJKL.

[0090] Next, each subspace can be used as a node to be segmented, and the steps of searching for a segmentation surface of the node that meets the first preset condition and the subsequent steps can be repeated according to the weight of each Gaussian ball included in the node, until the number of space segmentation reaches a preset first threshold. Assuming that the first threshold is equal to 3, the subspace ABCD-IJKL and the subspace EFGH-IJKL can be used as nodes to search for a segmentation surface that meets the first preset condition.

[0091] exist Figure 4 In the example, when subspace ABCD-IJKL is used as node 1, the searched splitting surface can be plane MNOP, and the two subspaces obtained by splitting once can be subspace ILDA-MNOP and subspace JKCB-MNOP, corresponding to nodes 3 and 4 respectively. When subspace EFGH-IJKL is used as node 2, the searched splitting surface can be plane STNM, and the two subspaces obtained by splitting once can be subspace EHLI-STNM and subspace FGKJ-STNM, corresponding to nodes 5 and 6 respectively.

[0092] At this time, exactly three divisions are completed, and a total of four subspaces are obtained (corresponding to nodes 3 to 6).

[0093] Those skilled in the art should understand that in order to divide the space corresponding to each node into Y subspaces, the number of searched dividing surfaces should be less than or equal to Y-1. For example, when Y=4, the number of searched dividing surfaces can be 3 or 2. The embodiments of the present disclosure do not limit the specific number of searched dividing surfaces.

[0094] Those skilled in the art should understand that there are more ways to set the stop condition of spatial segmentation, such as stopping the segmentation when the number of subspaces obtained by segmentation reaches a preset third threshold, etc. The embodiments of the present disclosure do not limit the stop condition of spatial segmentation. The first threshold and the third threshold can be set according to the requirements of the application scenario, and the present disclosure does not limit the specific values ​​of the first threshold and the third threshold.

[0095] By using a tree structure to divide the three-dimensional space, the size of the sub-model is more controllable. The depth of the tree structure can be increased (increasing the number of sub-models and the number of divisions), the size of the sub-model can be reduced, and a reasonable batch size can be set during training to ensure the safety of the single graphics card load.

[0096] In a possible implementation, the first preset condition includes:

[0097] The weights of the Y subspaces obtained after splitting the nodes based on the splitting surface are closest, or,

[0098] The difference between the weights of every two subspaces obtained after the nodes are split based on the splitting plane is less than a preset second threshold.

[0099] For example, the higher the weight of the Gaussian balls included in a space, the larger the size of the sub-model corresponding to the space. Since different sub-models can be rendered in parallel, the closer the size of the sub-models, the smaller the difference in rendering time, the smaller the maximum rendering time, and the higher the model training efficiency.

[0100] Therefore, the first preset condition can be set as: the weights of the Y subspaces obtained after the nodes are segmented based on the segmentation plane are closest. In this case, the closer the scales of all subspaces finally segmented are, the further the model training efficiency can be improved.

[0101] The first preset condition can also be set as: the difference in weights of each two subspaces obtained after the node is split based on the splitting surface is less than a preset second threshold. In this case, the splitting surface that meets the condition can be queried more quickly. The second threshold can be set according to the application scenario requirements, and the present disclosure does not limit the specific value of the second threshold.

[0102] Those skilled in the art should understand that there are more ways to set the first preset condition, as long as the dividing surface that meets the first preset condition can divide the space into Y subspaces. The embodiments of the present disclosure do not limit the specific setting method of the first preset condition.

[0103] In a possible implementation, the gradient information of each sub-model is calculated according to the final rendering result and output to the rendering process corresponding to each sub-model, including:

[0104] The gradient information of each sub-model is output to the rendering process corresponding to each sub-model in the aliasing order.

[0105] For example, the aliasing order can be used not only to determine the order in which the rendering results are aliased, but also to determine the order in which the gradient information is transmitted. When there are multiple copies of gradient information to be transmitted between two processes, if the transmission order is not controlled, it may cause delays. Therefore, the gradient information of each sub-model can be output to the rendering process corresponding to each sub-model in the aliasing order to reduce delays and improve training efficiency.

[0106] The following describes a method for determining whether each sub-model participates in rendering and the aliasing order of the rendering results of the sub-models.

[0107] In a possible implementation, determining whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling viewing angle includes:

[0108] Determine the light cone according to the sampling position and the sampling viewing angle;

[0109] Determine whether the light cone illuminates each subspace, determine the submodel corresponding to the subspace that is not illuminated as a submodel that does not participate in rendering, and determine the submodel corresponding to the subspace that is illuminated as a submodel that participates in rendering;

[0110] The aliasing order is determined according to the order in which the light cone illuminates the subspaces corresponding to the submodels involved in the rendering.

[0111] For example, at a certain sampling position and at a certain sampling angle, the content of the image sampled is associated with the illumination range of the light cone (simulated light) corresponding to the sampling position and the sampling angle. Figure 4 Taking the three-dimensional space shown as an example, assuming that the sampling position (optical center) is on plane IMNL, and the sampling angle of view is that the illumination direction is toward plane ABCD and perpendicular to plane ABCD, then subspace ILDA-MNOP and subspace JKCB-MNOP can be illuminated by the light cone, and subspace EHLI-STNM and subspace FGKJ-STNM will not be illuminated by the light cone. The sub-model corresponding to the sub-space that is not illuminated can be determined as a sub-model that does not participate in rendering, and the sub-model corresponding to the illuminated sub-space can be determined as a sub-model that participates in rendering.

[0112] After determining the sub-models participating in the rendering, the aliasing order can be determined according to the order of illumination of the sub-spaces corresponding to the sub-models participating in the rendering by the light cone. For example, if the sampling position is on the plane IMNL, and the sampling angle of view is such that the illumination direction is toward the plane ABCD and perpendicular to the plane ABCD, then the sub-space ILDA-MNOP is illuminated first, and the sub-space JKCB-MNOP is illuminated later. The aliasing order can be such that the sub-model corresponding to the sub-space ILDA-MNOP is later, and the sub-model corresponding to the sub-space JKCB-MNOP is earlier. When aliasing occurs, the rendering result of the sub-model with the later aliasing order will overwrite the rendering result of the sub-model with the earlier aliasing order, so that in the final rendering result, the rendering result of the sub-model corresponding to the sub-space ILDA-MNOP is above the rendering result of the sub-model corresponding to the sub-space JKCB-MNOP.

[0113] Exemplarily, the relationship between the final rendering result and the rendering results of each sub-model is shown in formula (1):

[0114]

[0115] In formula (1), image j represents the rendering result of sub-model j at a certain sampling position and sampling perspective, α i Represents the opacity coefficient of the rendering result of sub-model i, Image represents the final rendering result, num model Indicates the number of sub-models involved in rendering.

[0116] In a possible implementation, the gradient information of each sub-model is calculated according to the final rendering result and output to the rendering process corresponding to each sub-model, including:

[0117] According to the final rendering result and the true value of the image corresponding to the sampling position and sampling angle, the loss of the large model in this training process is calculated;

[0118] According to the loss and the rendering results of each sub-model, the gradient information of the loss relative to the rendering results of each sub-model is calculated as the gradient information corresponding to each sub-model.

[0119] For example, since it is still in the training stage, the true value of the image sampled in the three-dimensional space according to the sampling position and the sampling perspective can be determined in advance based on the sampling position and the sampling perspective. If the model converges, the loss of the final rendering result and the true value of the image should be less than the preset threshold. Therefore, after obtaining the final rendering result, the loss of the large model in this training process can be calculated based on the final rendering result and the true value of the image. Afterwards, based on the loss and the rendering results of each sub-model, the gradient information of the loss relative to the rendering results of each sub-model is calculated as the gradient information corresponding to each sub-model. The calculation of the loss and gradient information can be implemented based on the existing technology and will not be repeated here.

[0120] The present disclosure also proposes a model training device, Figure 5 An exemplary structural diagram of a model training device according to an embodiment of the present disclosure is shown.

[0121] like Figure 5 As shown, in a possible implementation, the device is used to train a large model, and the device includes multiple processes, which are started at the beginning of any training of the large model, and the multiple processes are loaded by multiple graphics cards, and each graphics card loads at least one process.

[0122] When any one of the processes is used as a control process, it is used to determine whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling angle, and broadcast the control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and output the aliasing order to the summary process;

[0123] When any one of the processes is used as a rendering process, when the control information indicates that the sub-model corresponding to the rendering process participates in rendering, the sub-model corresponding to the rendering process is driven to complete rendering, and the rendering result is output to the aggregation process;

[0124] When any one of the processes is used as a summary process, it is used to alias the received rendering results according to the aliasing order to obtain a final rendering result, calculate the gradient information of each sub-model according to the final rendering result, and output it to the rendering process corresponding to each sub-model;

[0125] When any process is used as a rendering process, it is also used to update the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete rendering next time.

[0126] In a possible implementation, any one of the multiple processes serves as a control process, at least one of the multiple processes serves as a summary process, and two or more of the multiple processes serve as rendering processes; a process can serve as one or more of a control process, a rendering process, and a summary process.

[0127] In a possible implementation, the large model is used to sample the three-dimensional space, and the device further includes a segmentation module, wherein the segmentation module is used to:

[0128] Before the first training of the large model begins, or after the large model changes, the three-dimensional space is divided into multiple subspaces, each subspace corresponds to a submodel, and the progress of each graphics card load is determined according to the scale of each submodel.

[0129] In one possible implementation, the large model is a Gaussian splash model, and the three-dimensional space is divided into multiple subspaces, including: taking the three-dimensional space as a node to be divided, and searching for a division surface of the node that meets a first preset condition according to the weight of each Gaussian sphere included in the node; based on the searched division surface, dividing the space corresponding to the node into Y subspaces, where Y is an integer greater than 1; taking each subspace as a node to be divided, repeatedly executing the steps of searching for a division surface of the node that meets the first preset condition according to the weight of each Gaussian sphere included in the node and subsequent steps until the number of space divisions reaches a preset first threshold.

[0130] In a possible implementation, the first preset condition includes: the weights of the Y subspaces obtained after the nodes are split based on the splitting plane are closest, or the difference between the weights of every two subspaces obtained after the nodes are split based on the splitting plane is less than a preset second threshold.

[0131] In a possible implementation, the calculating the gradient information of each sub-model according to the final rendering result and outputting it to the rendering process corresponding to each sub-model includes: outputting the gradient information of each sub-model to the rendering process corresponding to each sub-model according to the aliasing order.

[0132] In a possible implementation, determining whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models based on the sampling position and the sampling angle includes: determining a light cone based on the sampling position and the sampling angle; judging whether the light cone illuminates each subspace, determining the sub-model corresponding to the subspace that is not illuminated as a sub-model that does not participate in rendering, and determining the sub-model corresponding to the illuminated subspace as a sub-model that participates in rendering; and determining the aliasing order based on the order in which the light cone illuminates the subspaces corresponding to the sub-models that participate in rendering.

[0133] In a possible implementation, the gradient information of each sub-model is calculated based on the final rendering result and outputted to the rendering process corresponding to each sub-model, including: calculating the loss of the large model in this training process based on the final rendering result and the true value of the image corresponding to the sampling position and the sampling perspective; and calculating the gradient information of the loss relative to the rendering result of each sub-model based on the loss and the rendering result of each sub-model as the gradient information corresponding to each sub-model.

[0134] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0135] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0136] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0137] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0138] Figure 6 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 6 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0139] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0140] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0141] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0142] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0143] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0144] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0145] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0146] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0147] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0148] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0149] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, characterized in that: The method is used to train a large model, and the method comprises: When any training of the large model starts, multiple processes are started, and the multiple processes are loaded by multiple graphics cards, and each graphics card loads at least one process; When any one of the processes serves as a control process, the control process determines whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling perspective, and broadcasts control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and outputs the aliasing order to the summary process; When any one of the processes is used as a rendering process, when the control information indicates that the sub-model corresponding to the rendering process participates in rendering, the rendering process drives the corresponding sub-model to complete rendering and outputs the rendering result to the aggregation process; When any one of the processes is used as a summary process, the summary process performs aliasing on the received rendering results according to the aliasing order to obtain a final rendering result, calculates the gradient information of each sub-model according to the final rendering result, and outputs it to the rendering process corresponding to each sub-model; When any process is used as a rendering process, the rendering process updates the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete rendering next time.

2. The method according to claim 1, characterized in that The method further comprises: Any one of the multiple processes is used as a control process, at least one of the multiple processes is used as a summary process, and two or more of the multiple processes are used as rendering processes; A process can be one or more of a control process, a rendering process, and a summary process.

3. The method according to claim 1, characterized in that The large model is used to sample the three-dimensional space. Before the first training of the large model begins, or after the large model changes, the method further includes: The three-dimensional space is divided into a plurality of subspaces, each subspace corresponds to a submodel, and the progress of each graphics card load is determined according to the scale of each submodel.

4. The method according to claim 3, characterized in that The large model is a Gaussian splash model, and the three-dimensional space is divided into a plurality of subspaces, including: Taking the three-dimensional space as a node to be segmented, searching for a segmentation surface of the node that meets a first preset condition according to the weight of each Gaussian sphere included in the node; Based on the searched splitting surface, the space corresponding to the node is split into Y subspaces, where Y is an integer greater than 1; Each subspace is taken as a node to be segmented, and the steps of searching for a segmentation surface of the node that meets a first preset condition and subsequent steps are repeated according to the weight of each Gaussian ball included in the node, until the number of space segmentation times reaches a preset first threshold.

5. The method according to claim 4, characterized in that The first preset condition includes: The weights of the Y subspaces obtained after splitting the nodes based on the splitting surface are closest, or, The difference between the weights of every two subspaces obtained after the nodes are split based on the splitting plane is less than a preset second threshold.

6. The method according to claim 1, characterized in that The step of calculating the gradient information of each sub-model according to the final rendering result and outputting the gradient information to the rendering process corresponding to each sub-model includes: The gradient information of each sub-model is output to the rendering process corresponding to each sub-model according to the aliasing order.

7. The method according to claim 3, characterized in that The step of determining whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling viewing angle includes: Determine a light cone according to the sampling position and the sampling viewing angle; Determine whether the light cone illuminates each subspace, determine the submodel corresponding to the subspace that is not illuminated as a submodel that does not participate in rendering, and determine the submodel corresponding to the subspace that is illuminated as a submodel that participates in rendering; The aliasing order is determined according to the order in which the light cone illuminates the subspaces corresponding to the submodels participating in the rendering.

8. The method according to claim 1 or 6, characterized in that: The step of calculating the gradient information of each sub-model according to the final rendering result and outputting the gradient information to the rendering process corresponding to each sub-model includes: Calculate the loss of the large model during this training process according to the final rendering result and the true value of the image corresponding to the sampling position and the sampling perspective; According to the loss and the rendering results of each sub-model, the gradient information of the loss relative to the rendering results of each sub-model is calculated as the gradient information corresponding to each sub-model.

9. A model training device, characterized in that: The device is used to train a large model. The device includes multiple processes. The multiple processes are started when any training of the large model starts. The multiple processes are loaded by multiple graphics cards, and each graphics card loads at least one process. When any one of the processes is used as a control process, it is used to determine whether each sub-model included in the large model participates in rendering and the aliasing order of the rendering results of the sub-models according to the sampling position and the sampling angle, and broadcast the control information indicating whether each sub-model participates in rendering to the rendering process corresponding to each sub-model, and output the aliasing order to the summary process; When any one of the processes is used as a rendering process, when the control information indicates that the sub-model corresponding to the rendering process participates in rendering, the sub-model corresponding to the rendering process is driven to complete rendering, and the rendering result is output to the aggregation process; When any one of the processes is used as a summary process, it is used to alias the received rendering results according to the aliasing order to obtain a final rendering result, calculate the gradient information of each sub-model according to the final rendering result, and output it to the rendering process corresponding to each sub-model; When any process is used as a rendering process, it is also used to update the parameters of the sub-model corresponding to the rendering process according to the received gradient information, and the updated parameters are used when driving the sub-model to complete rendering next time.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images