Digital twin remote rendering method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of digital image rendering, in particular to a digital twin remote rendering method and device, electronic equipment and a storage medium. The method comprises the following steps: selecting to execute a call request from the local, adding the call request into a call request data packet, forwarding the call request to an agent engine or sending the call request data packet to the agent engine, and then forwarding data of the agent engine to a call request sending end in response to the response of the agent engine after forwarding the call request to the agent engine. And finally, after the call request data packet is forwarded to the proxy engine, a returned rendering result is decoded in response to the response of the proxy engine, and a rendered picture is obtained. According to the method, the rendering requests are classified into multiple classes and are processed in a classified manner, so that the data transmission quantity is reduced, the transmission efficiency is improved, and the problem of remote rendering delay caused by low network transmission efficiency is relieved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image rendering, and in particular, to a digital twin remote rendering method, device, electronic device, and storage medium. Background Art

[0002] Image rendering refers to the process of converting a 3D model or scene into a 2D image, which involves multiple steps and techniques in computer graphics and aims to generate realistic or stylized visual effects. A rendering engine is a software framework or tool for image rendering, which encapsulates various algorithms and functions in the rendering process and provides a series of APIs for developers to use.

[0003] Remote rendering is a technology that sends rendering tasks from a local computer to a remote server or computer for processing. Users can submit rendering tasks anywhere with a network connection, without being restricted by local hardware devices.

[0004] Currently, the existing remote rendering technology mainly has problems of data delay and frame stuttering. The data volume of rendering tasks is usually large, including information such as 3D models, textures, and materials. When uploading to a remote server and downloading the rendering result back to the local, a high network bandwidth is required to ensure the data transmission speed. If the bandwidth is insufficient, it will cause the task upload and result download times to be too long, seriously affecting work efficiency.

[0005] Based on this, it is necessary to develop and design a digital twin remote rendering method. Summary of the Invention

[0006] Embodiments of the present invention provide a digital twin remote rendering method, device, electronic device, and storage medium, which are used to solve the problem of remote rendering delay caused by low network transmission efficiency in the prior art.

[0007] In a first aspect, an embodiment of the present invention provides a digital twin remote rendering method, which is applied to a terminal. The terminal is connected to a server, and the server is provided with a proxy engine and a rendering engine. The digital twin remote rendering method includes:

[0008] Obtain an application programming interface call request;

[0009] According to the attribute of the call request, select to execute the call request locally, add the call request to a call request data packet, forward the call request to the proxy engine, or send a call request data packet to the proxy engine;

[0010] After forwarding the call request to the proxy engine, forward the data of the proxy engine to the call request sender in response to the response of the proxy engine;

[0011] After forwarding the call request data packet to the proxy engine, in response to the response of the proxy engine, decode the returned rendering result to obtain a rendered image.

[0012] In a possible implementation manner, the selecting to execute the call request locally, adding the call request to a call request data packet, forwarding the call request to the proxy engine, or sending a call request data packet to the proxy engine according to the attributes of the call request includes:

[0013] Obtain the request parameters of the call request;

[0014] If the call request is a rendering request, send a call request data packet to the proxy engine;

[0015] If the request parameters do not involve parameters provided by other terminal devices and the call request is not a rendering request, select to execute the call request locally;

[0016] If the request parameters involve parameters provided by other terminal devices and the call request is not a rendering request, forward the call request to the proxy engine.

[0017] In a possible implementation manner, the decoding the returned rendering result to obtain a rendered image includes:

[0018] Obtain the rendering result;

[0019] Use a decoder to decode the rendering result to obtain a frequency-domain transform code matrix;

[0020] Perform a transformation on the frequency-domain transform code matrix according to a first formula to obtain a rendered image, where the first formula is:

[0021]

[0022] In the formula, IMG(x,y) is the pixel value at the x-th row and y-th column of the rendered image, MIMG(m,n) is the element at the m-th row and n-th column of the frequency-domain transform code matrix, e is the natural constant, π is the pi, i is the imaginary unit, M is the number of rows of the frequency-domain transform code matrix, N is the number of columns of the frequency-domain transform code matrix, X is the total number of row pixels of the rendered image, and Y is the total number of column pixels of the rendered image.

[0023] In a second aspect, an embodiment of the present invention provides a digital twin remote rendering method, which is applied to a server. The server is connected to a terminal, and the server is provided with a proxy engine and a rendering engine. The server accepts a request from the terminal to complete the rendering of an image. The digital twin remote rendering method includes:

[0024] Obtain a call request data packet, where the call request data packet is obtained according to the digital twin remote rendering method described in any one of claims 1-3;

[0025] Parse the call request data packet and extract the sending timestamp of the call request data packet;

[0026] Generate a rendering parameter indicating the rendering precision of the rendering engine and a transformation precision exponent indicating the precision of frequency domain transformation based on the sending timestamp, the current time, and the size of the call request data packet, and send the data parsed from the call request data packet to the rendering engine to generate a rendering picture;

[0027] Perform frequency domain transformation on the rendering picture according to the transformation precision exponent to generate a frequency domain transformation code matrix, and encode the frequency domain transformation code matrix to obtain a rendering result.

[0028] In a possible implementation manner, the performing frequency domain transformation on the rendering picture to generate a frequency domain transformation code matrix includes:

[0029] Obtain the total number of row pixels and the total number of column pixels of the rendering picture;

[0030] Determine the number of rows and the number of columns of the frequency domain transformation code matrix according to the transformation precision exponent, the total number of row pixels, and the total number of column pixels of the rendering picture;

[0031] Perform frequency domain transformation according to the second formula, the number of rows of the frequency domain transformation code matrix, and the number of columns of the frequency domain transformation code matrix to obtain the frequency domain transformation code matrix, where the second formula is:

[0032]

[0033] In the formula, MIMG(m,n) is the element at the m-th row and n-th column of the frequency domain transformation code matrix, X is the total number of row pixels of the rendering picture, Y is the total number of column pixels of the rendering picture, IMG(x,y) is the pixel value at the x-th row and y-th column of the rendering picture, e is the natural constant, π is the pi, and i is the imaginary unit.

[0034] In a possible implementation manner, the encoding the frequency domain transformation code matrix to obtain a rendering result includes:

[0035] Split the frequency domain transformation code matrix into multiple first arrays row by row or column by column;

[0036] By means of offset and numerical scaling, map the multiple numerical values of each first array among the multiple first arrays to a first numerical range, to obtain multiple second arrays and multiple mapping parameter arrays, where the mapping parameter array includes the parameters used when mapping the first array, and each mapping parameter array corresponds to a second array;

[0037] Input the multiple second arrays into an encoder to obtain an encoded queue;

[0038] Construct a rendering result from the encoded queue and the multiple mapping parameter arrays.

[0039] In a possible implementation manner, the encoder is constructed according to the occurrence probabilities of multiple first numerical values, the multiple first numerical values cover the first numerical range, and the construction process of the encoder includes:

[0040] Obtain a first fitting model and a second fitting model, where the first fitting model generates multi-bit codes according to numerical values, and the second fitting model generates numerical values according to multi-bit codes;

[0041] Input the multiple first numerical values into the first fitting model in sequence to obtain multiple first codes;

[0042] Extract the valid bits of each first code among the multiple first codes respectively to obtain multiple valid bits;

[0043] Calculate the correlation coefficient between the multiple valid bits and multiple target probabilities to obtain a first correlation coefficient, where the target probability is the probability of the valid bit originating from the first numerical value;

[0044] If the first correlation coefficient is greater than a threshold, adjust multiple parameters of the first fitting model, and return to the step of inputting the multiple first numerical values into the first fitting model in sequence to obtain multiple first codes;

[0045] Otherwise, fix multiple parameters of the first fitting model, and use the first fitting model as the encoder;

[0046] Input the multiple first numerical values into the encoder in sequence;

[0047] Input the output of the encoder into the second fitting model in sequence to obtain multiple second numerical values;

[0048] Calculate the deviation between the multiple first numerical values and the multiple second numerical values as the decoding deviation;

[0049] If the decoding deviation is greater than a threshold, adjust multiple parameters of the second fitting model, and jump to the step of inputting the output of the encoder into the second fitting model in sequence to obtain multiple second numerical values;

[0050] Otherwise, fix multiple parameters of the second fitting model, and use the second fitting model as the decoder.

[0051] In a third aspect, an embodiment of the present invention provides a digital twin remote rendering device for implementing the digital twin remote rendering method described in the first aspect or any possible implementation manner of the first aspect above. The digital twin remote rendering device includes:

[0052] A call request acquisition module for acquiring an application programming interface call request;

[0053] A call request processing module for selecting to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine according to the attribute of the call request;

[0054] A first forwarding module for, after forwarding the call request to the proxy engine, forwarding the data of the proxy engine to the call request sender in response to the response of the proxy engine;

[0055] And,

[0056] A second forwarding module for, after forwarding a call request data packet to the proxy engine, decoding the returned rendering result in response to the response of the proxy engine to obtain a rendered image.

[0057] In a fourth aspect, an embodiment of the present invention provides an electronic device including a memory and a processor. A computer program executable on the processor is stored in the memory. When the processor executes the computer program, the steps of the method described in the first aspect, any possible implementation manner of the first aspect, the second aspect, or any possible implementation manner of the second aspect above are implemented.

[0058] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps of the method described in the first aspect, any possible implementation manner of the first aspect, the second aspect, or any possible implementation manner of the second aspect above are implemented.

[0059] The beneficial effects of the embodiments of the present invention compared with the prior art are:

[0060] An embodiment of the present invention discloses a digital twin remote rendering method. In the embodiment of the digital twin remote rendering method of the present invention, first, an application programming interface call request is obtained. Then, according to the attributes of the call request, it is selected to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine. Next, after forwarding the call request to the proxy engine, in response to the response of the proxy engine, the data of the proxy engine is forwarded to the call request sender. Finally, after forwarding the call request data packet to the proxy engine, in response to the response of the proxy engine, the returned rendering result is decoded to obtain a rendered image. The digital twin remote rendering method for a terminal provided by the present invention classifies rendering requests into multiple categories. One category is executed locally by the terminal. Since no data transmission is required, the occupancy rate of network transmission and the pressure on the server to process requests are reduced. For requests that do not need to be executed immediately, these requests are added to the request data packet. When the sending condition is met, multiple requests included in the request data packet are sent to the server together, and the server centrally processes these requests. Therefore, the process of communication between the terminal and the server is reduced, and the communication pressure is alleviated. For the rendered image, the server transforms and encodes the rendered image to form a rendering result. The amount of data of the rendering result is smaller than that of the rendered image, reducing the amount of data transmission, improving the transmission efficiency, and alleviating the problem of remote rendering delay caused by low network transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0062] Figure 1 is a flowchart of the digital twin remote rendering method applied to a terminal provided by an embodiment of the present invention;

[0063] Figure 2 is a schematic diagram of the digital twin remote rendering system provided by an embodiment of the present invention;

[0064] Figure 3 is a flowchart of the digital twin remote rendering method applied to a server provided by an embodiment of the present invention;

[0065] Figure 4 is a schematic diagram of the encoding process of the rendered image provided by an embodiment of the present invention;

[0066] Figure 5It is a functional block diagram of a digital twin remote rendering device provided by an embodiment of the present invention;

[0067] Figure 6 It is a functional block diagram of an electronic device provided by an embodiment of the present invention. Specific Embodiments

[0068] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0069] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will be described through specific embodiments in conjunction with the accompanying drawings.

[0070] The following details the embodiments of the present invention. This example is implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0071] Figure 1 It is a flowchart of a digital twin remote rendering method provided by an embodiment of the present invention.

[0072] As Figure 1 shown, it shows a flowchart of a digital twin remote rendering method for a terminal provided by an embodiment of the present invention, which is described in detail as follows:

[0073] In step 101, an application programming interface call request is obtained.

[0074] In step 102, according to the attributes of the call request, select to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine.

[0075] In some embodiments, the step of selecting to execute the call request locally, adding the call request to a call request data packet, forwarding the call request to a proxy engine, or sending a call request data packet to the proxy engine according to the attributes of the call request includes:

[0076] Obtain the request parameters of the call request;

[0077] If the call request is a rendering request, send a call request data packet to the proxy engine;

[0078] If the request parameters do not involve parameters provided by other terminal devices and the call request is not a rendering request, then select to execute the call request locally;

[0079] If the request parameters involve parameters provided by other terminal devices and the call request is not a rendering request, then forward the call request to the proxy engine.

[0080] In step 103, after forwarding the call request to the proxy engine, in response to the response of the proxy engine, forward the data of the proxy engine to the call request sender.

[0081] Exemplarily, as described above, remote rendering is affected by the network transmission speed, resulting in problems such as data delay and frame stuttering. The present invention aims to improve the efficiency of remote rendering and reduce the impact brought by data transmission in terms of data transmission scheme, data processing scheme, and rendering screen data processing.

[0082] As Figure 2 shown, the figure shows the schematic diagram of the digital twin remote rendering system provided by the embodiment of the present invention. In the figure, the terminal 201 generates various requests related to remote rendering. These requests may require the server 202 to process data or render the model of the object to be rendered, and may also need to call some data generated by other terminals. For example, other terminals have an impact on the model of the object to be rendered. Before generating the final rendering screen request for the server 202, it is necessary to call the data generated by other terminals that affect the model. On the server 202 side, a proxy engine and a rendering engine are set up. Among them, the rendering engine is used to render the model to generate a rendering screen, and the proxy engine is used to receive requests from the terminal and transform these requests into requests for the rendering screen, for encoding and sending the rendering screen, or for providing data generated by other terminals; that is, the two engines are respectively responsible for screen rendering and data interaction. In this way, the rendering tasks can be reasonably allocated, and the data transmission volume can be reduced through encoding means.

[0083] To achieve the above object, the present invention divides the remote rendering requests of the terminal into four types, namely requests that can be called locally, requests that need to be executed by the server but not immediately, call requests that need to be forwarded (for example, the server has the complete data and execution conditions for the call request, or requests that require the server to provide call data), and requests that need to be immediately executed by the server.

[0084] Among them, the requests that can be called locally will directly interrupt the operation of the application program interface itself. The requests that need to be executed by the server but not immediately will be added to the call request data packet. When the conditions for executing the request are met, the request data packet will be sent to the server together. The server will parse out multiple requests from the request data packet and execute the call request.

[0085] Requests that need to be forwarded are generally sent to the server immediately. The server usually has the data and application interfaces to execute the forwarded requests. At this time, the requests will be executed, and the execution results, that is, the data generated after execution, will be returned. These data will be returned to the calling request side again. However, in some scenarios, the server only provides the call data that can be provided.

[0086] For requests that need to be immediately executed by the server, they will be added to the call request data packet and then sent to the server for execution immediately. After the proxy engine of the server receives it, it will parse the call request data packet and parse the data into rendering requests and other requests. Among them, the other requests will be executed by the proxy engine, and the rendering requests will call the rendering engine for rendering. The rendered image will be converted and encoded to form a rendering result and returned to the terminal. It should be noted that in terms of conversion and encoding, it is a process of compressing the rendered image, aiming to reduce the data volume, thereby improving the efficiency of data transmission and achieving the effect of data encryption.

[0087] After the terminal receives the rendering result, it will decode and inverse-transform the rendering result to restore the rendered image.

[0088] In step 104, after forwarding the call request data packet to the proxy engine, in response to the response of the proxy engine, the returned rendering result is decoded to obtain the rendered image.

[0089] In some embodiments, the decoding the returned rendering result to obtain the rendered image includes:

[0090] Obtain the rendering result;

[0091] Use a decoder to decode the rendering result to obtain a frequency-domain transform code matrix;

[0092] Transform the frequency-domain transform code matrix according to the first formula to obtain the rendered image, where the first formula is:

[0093]

[0094] In the formula, IMG(x,y) is the pixel value of the x-th row and y-th column of the rendered image, MIMG(m,n) is the element of the m-th row and n-th column of the frequency-domain transform code matrix, e is the natural constant, π is the pi, i is the imaginary unit, M is the number of rows of the frequency-domain transform code matrix, N is the number of columns of the frequency-domain transform code matrix, X is the total number of row pixels of the rendered image, and Y is the total number of column pixels of the rendered image.

[0095] Exemplarily, in terms of decoding the rendering result, the embodiment of the present invention first uses a decoder to decode the rendering result to obtain a frequency-domain transform code matrix, and this matrix is then inversely transformed to obtain the rendered image. In one scenario, the following formula is used:

[0096]

[0097] In the formula, IMG(x, y) is the pixel value of the x-th row and y-th column of the rendered image, MIMG(m, n) is the element of the m-th row and n-th column of the frequency-domain transform code matrix, e is the natural constant, π is the pi, i is the imaginary unit, M is the number of rows of the frequency-domain transform code matrix, N is the number of columns of the frequency-domain transform code matrix, X is the total number of row pixels of the rendered image, and Y is the total number of column pixels of the rendered image.

[0098] The digital twin remote rendering method for a terminal provided by the present invention divides the rendering requests into multiple categories. One category is executed locally by the terminal. Since no data transmission is required, the occupancy rate of network transmission and the pressure on the server to process requests are reduced. For requests that do not need to be executed immediately, these requests are added to the request data packet. When the sending condition is met, multiple requests included in the request data packet are sent to the server together, and the server processes these requests centrally. Therefore, the process of communication between the terminal and the server is reduced, and the communication pressure is alleviated; for the rendered image, the server transforms and encodes the rendered image to form a rendering result. The data volume of the rendering result is smaller than that of the rendered image, reducing the data transmission volume and improving the transmission efficiency.

[0099] As Figure 3 shown, it shows the flowchart of the digital twin remote rendering method for a server provided by the embodiment of the present invention, which is described in detail as follows:

[0100] In step 301, a call request data packet is obtained, where the call request data packet is obtained according to the digital twin remote rendering method described in the first aspect.

[0101] In step 302, the call request data packet is parsed to extract the sending timestamp of the call request data packet.

[0102] In step 303, a rendering parameter indicating the rendering accuracy of the rendering engine and a transform accuracy index indicating the accuracy of the frequency-domain transform are generated according to the sending timestamp, the current time, and the size of the call request data packet, and the data parsed from the call request data packet is sent to the rendering engine to generate a rendered image.

[0103] Exemplarily, on the server side, after the server receives a call request data packet, it parses the packet and executes it sequentially according to the time stamp. Among them, the rendering request is a request to be executed immediately and is usually located at the request bit that is executed last in the request data packet. In addition to indicating the request execution order, the time stamp also has the ability to evaluate the data transmission between the terminal and the server.

[0104] That is to say, the server will determine the data transmission speed between the terminal and the server according to the data volume size of the request data packet, the sending time stamp of the terminal, and the current time. The present invention adjusts the rendering accuracy and the accuracy of frequency domain transformation according to this transmission speed. That is to say, when the data transmission speed is lower, the rendering accuracy and the transformation accuracy are synchronously reduced to ensure the real-time performance of the picture.

[0105] In step 304, perform a frequency domain transformation on the rendered picture according to the transformation accuracy index to generate a frequency domain transformation code matrix, and encode the frequency domain transformation code matrix to obtain a rendering result.

[0106] In some embodiments, the performing a frequency domain transformation on the rendered picture to generate a frequency domain transformation code matrix includes:

[0107] Obtain the total number of row pixels and the total number of column pixels of the rendered picture;

[0108] Determine the number of rows of the frequency domain transformation code matrix and the number of columns of the frequency domain transformation code matrix according to the transformation accuracy index, the total number of row pixels of the rendered picture, and the total number of column pixels;

[0109] Perform a frequency domain transformation according to the second formula, the number of rows of the frequency domain transformation code matrix, and the number of columns of the frequency domain transformation code matrix to obtain the frequency domain transformation code matrix, where the second formula is:

[0110]

[0111] In the formula, MIMG(m,n) is the element in the m-th row and n-th column of the frequency domain transformation code matrix, X is the total number of row pixels of the rendered picture, Y is the total number of column pixels of the rendered picture, IMG(x,y) is the pixel value of the x-th row and y-th column of the rendered picture, e is the natural constant, π is the pi, and i is the imaginary unit.

[0112] Exemplarily, such as Figure 4As shown in the figure, this is the schematic diagram of the rendering picture encoding process provided by the present invention. After the rendering engine generates the rendering picture 401 and returns the picture to the proxy engine, the proxy engine will perform a frequency domain transformation on the rendering picture 401 to generate a frequency domain transformation code matrix 402. This frequency domain transformation code matrix 402 will be split into arrays 403 row by row or column by column. These arrays 403 are respectively mapped to a predetermined numerical interval to form new arrays (the mapping conditions are retained), and the new arrays are sent to the encoder 404. The formed encoding and the mapping conditions are constructed into the rendering result and sent back to the terminal. It should be noted that the encoder 404 and the decoder 405 are obtained by fitting models in a paired manner.

[0113] In terms of frequency domain transformation, the present invention controls the number of rows and columns of the frequency domain change code matrix obtained after transformation according to the transformation accuracy index obtained in the foregoing steps. The present invention provides a transformation formula:

[0114]

[0115] In the formula, MIMG(m,n) is the element at the m-th row and n-th column of the frequency domain transformation code matrix, X is the total number of row pixels of the rendering picture, Y is the total number of column pixels of the rendering picture, IMG(x,y) is the pixel value at the x-th row and y-th column of the rendering picture, e is the natural constant, π is the pi, and i is the imaginary unit.

[0116] In some embodiments, encoding the frequency domain transformation code matrix to obtain a rendering result includes:

[0117] Splitting the frequency domain transformation code matrix into multiple first arrays row by row or column by column;

[0118] By means of offset and numerical scaling, mapping the multiple numerical values of each first array in the multiple first arrays to a first numerical interval to obtain multiple second arrays and multiple mapping parameter arrays, where the mapping parameter array includes the parameters used when mapping the first array, and each mapping parameter array corresponds to a second array;

[0119] Inputting the multiple second arrays into the encoder to obtain an encoding queue;

[0120] Constructing the encoding queue and the multiple mapping parameter arrays into a rendering result.

[0121] In some embodiments, the encoder is constructed according to the occurrence probabilities of multiple first numerical values, and the multiple first numerical values cover the first numerical interval. The construction process of the encoder includes:

[0122] Obtaining a first fitting model and a second fitting model, where the first fitting model generates multi-bit encoding according to numerical values, and the second fitting model generates numerical values according to multi-bit encoding;

[0123] Input the multiple first numerical values into the first fitting model in sequence to obtain multiple first codes;

[0124] Extract the significant digits of each first code among the multiple first codes respectively to obtain multiple significant digits;

[0125] Calculate the correlation coefficient between the multiple significant digits and multiple target probabilities to obtain a first correlation coefficient, where the target probability is the probability that the significant digit comes from the first numerical value;

[0126] If the first correlation coefficient is greater than the threshold, adjust multiple parameters of the first fitting model, and return to the step of inputting the multiple first numerical values into the first fitting model in sequence to obtain multiple first codes;

[0127] Otherwise, fix multiple parameters of the first fitting model, and use the first fitting model as the encoder;

[0128] Input the multiple first numerical values into the encoder in sequence;

[0129] Input the output of the encoder into the second fitting model in sequence to obtain multiple second numerical values;

[0130] Calculate the deviation between the multiple first numerical values and the multiple second numerical values as the decoding deviation;

[0131] If the decoding deviation is greater than the threshold, adjust multiple parameters of the second fitting model, and jump to the step of inputting the output of the encoder into the second fitting model in sequence to obtain multiple second numerical values;

[0132] Otherwise, fix multiple parameters of the second fitting model, and use the second fitting model as the decoder.

[0133] Exemplarily, as described above, the frequency-domain transform code matrix is split into multiple arrays row by row or column by column. Since the numerical values in these arrays are widely distributed, the present invention performs interval unification on the arrays, that is, mapping. Specifically, by offsetting and scaling the numerical values in the arrays, the numerical values in the arrays are mapped to the target interval. For example, the numerical value distribution of a split array is n390 - n7890, and it is mapped to n00 - n255 through offsetting and scaling, and the following formula is applied:

[0134]

[0135] In the above formula, n (1) is the numerical value obtained after mapping, is the maximum value of the mapping target interval (n255 in the above example), n(0) is the mapped value, is the minimum value of the mapped array (n390 in the above example), is the maximum value of the mapped array (n7890 in the above example).

[0136] After the above process, the frequency of the values in the frequency-domain transform code matrix is greatly increased. The advantage of this is that the coding efficiency can be greatly improved, and the total amount of data after coding can be reduced.

[0137] In terms of the construction of the encoder and decoder, the present invention is constructed in pairs based on the probabilities of the data after coding using two fitting models. The fitting models can be, for example, artificial neural network models or polynomial regression models.

[0138] First, an encoder is constructed. The encoder encodes the values with high frequencies into shorter codes and the values with low frequencies into longer codes. The integer values in the above interval are sequentially input into the fitting model (the fitting model for constructing the encoder). The fitting model outputs codes. The effective bits of the codes (for example, the effective bits of the code 000nnn are nnn) are calculated for the correlation coefficient with the probabilities of the integer values (such as the Pearson correlation coefficient). If the correlation coefficient is greater than the threshold, it indicates a strong positive correlation, and the parameters of the fitting model are adjusted, and the process of sequentially inputting the integer values into the fitting model is repeated until the effective bits of the code and the probability show a strong negative correlation. At this time, the parameters of the fitting model are fixed as the encoder.

[0139] Then, a decoder is constructed. The integer values in the above interval are sequentially input into the encoder, and the encoder outputs codes. These codes are sequentially sent into the fitting model (the fitting model for constructing the decoder). The decoder gives multiple values, and the deviation operation is performed between these multiple values and the values input into the encoder. If the deviation is greater than the threshold, the parameters of the fitting model are adjusted, and the process returns to the step of sequentially sending the codes into the fitting model. The above steps are repeated until the deviation is less than the threshold. At this time, the parameters of the fitting model are fixed as the decoder.

[0140] Embodiment of the digital twin remote rendering method of the present invention. First, it obtains an application programming interface call request. Then, according to the attributes of the call request, it selects to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine. Next, after forwarding the call request to the proxy engine, it forwards the data of the proxy engine to the call request sender in response to the response of the proxy engine. Finally, after forwarding the call request data packet to the proxy engine, it decodes the returned rendering result in response to the response of the proxy engine to obtain a rendered image. The digital twin remote rendering method provided by the present invention divides rendering requests into multiple categories. One category is executed locally by the terminal. Since no data transmission is required, the latency of network transmission and the pressure on the server to process requests are reduced. For requests that do not need to be executed immediately, these requests are added to the request data packet. When the sending condition is met, multiple requests included in the request data packet are sent to the server together, and the server processes these requests centrally. Therefore, the process of communication between the terminal and the server is reduced, and the communication pressure is alleviated. For the rendered image, the server transforms and encodes the rendered image to form a rendering result. The amount of data of the rendering result is smaller than that of the rendered image, reducing the data transmission volume, improving the transmission efficiency, and alleviating the problem of remote rendering latency caused by low network transmission efficiency.

[0141] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0142] The following is the device embodiment of the present invention. For the details not described in detail herein, reference can be made to the corresponding method embodiments above.

[0143] Figure 5 is a functional block diagram of the digital twin remote rendering device provided by the embodiments of the present invention. Referring to Figure 5 , the digital twin remote rendering device includes: a call request acquisition module 501, a call request processing module 502, a first forwarding module 503, and a second forwarding module 504, where:

[0144] The call request acquisition module 501 is used to obtain an application programming interface call request;

[0145] The call request processing module 502 is used to select to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine according to the attributes of the call request;

[0146] The first forwarding module 503 is configured to forward the data of the proxy engine to the call request sender in response to the response of the proxy engine after forwarding the call request to the proxy engine.

[0147] The second forwarding module 504 is configured to decode the returned rendering result in response to the response of the proxy engine after forwarding the call request packet to the proxy engine to obtain a rendered picture.

[0148] Figure 6 It is a functional block diagram of an electronic device provided by an embodiment of the present invention. As Figure 6 shown, the electronic device 6 of this embodiment includes: a processor 600 and a memory 601, and a computer program 602 that can run on the processor 600 is stored in the memory 601. When the processor 600 executes the computer program 602, the steps in the above-mentioned various digital twin remote rendering methods and embodiments are implemented, such as Figure 1 the steps 101 to 104 shown.

[0149] Exemplarily, the computer program 602 can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 601 and executed by the processor 600 to complete the present invention.

[0150] The electronic device 6 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device 6 may include, but is not limited to, a processor 600 and a memory 601. Those skilled in the art can understand that Figure 6 it is only an example of the electronic device 6 and does not constitute a limitation on the electronic device 6. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device 6 may further include input / output devices, network access devices, buses, etc.

[0151] The so-called processor 600 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0152] The memory 601 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 601 may also be an external storage device of the electronic device 6, such as a plug-in hard disk equipped on the electronic device 6, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 601 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 601 is used to store the computer program 602 and other programs and data required by the electronic device 6. The memory 601 may also be used to temporarily store data that has been output or is to be output.

[0153] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated here.

[0154] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0155] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0156] In the embodiments provided by the present invention, it should be understood that the disclosed device / electronic device and method can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method and device embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0160] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A digital twin remote rendering method, characterized in that Applied to a terminal, the terminal is connected to a server, the server is provided with a proxy engine and a rendering engine, and the digital twin remote rendering method includes: Obtain an application programming interface call request; According to the attributes of the call request, select to execute the call request locally, add the call request to a call request data packet, forward the call request to the proxy engine, or send a call request data packet to the proxy engine; After forwarding the call request to the proxy engine, forward the data of the proxy engine to the call request sender in response to the response of the proxy engine; After forwarding a call request data packet to the proxy engine, decode the returned rendering result in response to the response of the proxy engine to obtain a rendered image.

2. The digital twin remote rendering method according to claim 1, wherein The step of according to the attributes of the call request, selecting to execute the call request locally, adding the call request to a call request data packet, forwarding the call request to the proxy engine, or sending a call request data packet to the proxy engine includes: Obtain the request parameters of the call request; If the call request is a rendering request, send a call request data packet to the proxy engine; If the request parameters do not involve parameters provided by other terminal devices and the call request is not a rendering request, select to execute the call request locally; If the request parameters involve parameters provided by other terminal devices and the call request is not a rendering request, forward the call request to the proxy engine.

3. The digital twin remote rendering method according to any one of claims 1-2, characterized in that, The step of decoding the returned rendering result to obtain a rendered image includes: Obtain the rendering result; Use a decoder to decode the rendering result to obtain a frequency domain transform code matrix; Perform a transform on the frequency domain transform code matrix according to a first formula to obtain a rendered image, where the first formula is: In the formula, IMG(x,y) is the pixel value of the x-th row and y-th column of the rendered image, MIMG(m,n) is the element of the m-th row and n-th column of the frequency domain transform code matrix, e is the natural constant, π is the pi, i is the imaginary unit, M is the number of rows of the frequency domain transform code matrix, N is the number of columns of the frequency domain transform code matrix, X is the total number of row pixels of the rendered image, and Y is the total number of column pixels of the rendered image.

4. A digital twin remote rendering method, characterized in that, Applied to a server, the server is connected to a terminal, the server is provided with a proxy engine and a rendering engine, and the server accepts the request of the terminal to complete the rendering of an image. The digital twin remote rendering method includes: Obtain a call request data packet, where the call request data packet is obtained according to the digital twin remote rendering method according to any one of claims 1-3; Parse the call request data packet and extract the sending timestamp of the call request data packet; Generate a rendering parameter indicating the rendering accuracy of the rendering engine and a transform accuracy index indicating the accuracy of frequency domain transform according to the sending timestamp, the current time, and the size of the call request data packet, and send the data parsed from the call request data packet to the rendering engine to generate a rendered image; Perform a frequency domain transform on the rendered image according to the transform accuracy index to generate a frequency domain transform code matrix, and encode the frequency domain transform code matrix to obtain a rendering result.

5. The digital twin remote rendering method according to claim 4, wherein Performing frequency domain transformation on the rendered image to generate a frequency domain transformation code matrix includes: Obtaining the total number of row pixels and the total number of column pixels of the rendered image; Determining the number of rows and the number of columns of the frequency domain transformation code matrix according to the transformation precision exponent, the total number of row pixels, and the total number of column pixels of the rendered image; Performing frequency domain transformation according to the second formula, the number of rows of the frequency domain transformation code matrix, and the number of columns of the frequency domain transformation code matrix to obtain the frequency domain transformation code matrix, where the second formula is: In the formula, MIMG(m,n) is the element at the m-th row and n-th column of the frequency domain transformation code matrix, X is the total number of row pixels of the rendered image, Y is the total number of column pixels of the rendered image, IMG(x,y) is the pixel value at the x-th row and y-th column of the rendered image, e is the natural constant, π is the pi, and i is the imaginary unit.

6. The digital twin remote rendering method according to any one of claims 4-5, characterized in that Encoding the frequency domain transformation code matrix to obtain a rendering result includes: Splitting the frequency domain transformation code matrix into multiple first arrays row by row or column by column; Mapping the multiple values of each first array in the multiple first arrays to a first value range by means of offset and numerical scaling to obtain multiple second arrays and multiple mapping parameter arrays, where the mapping parameter array includes the parameters used for mapping the first array, and each mapping parameter array corresponds to a second array; Inputting the multiple second arrays into an encoder to obtain an encoding queue; Constructing the rendering result from the encoding queue and the multiple mapping parameter arrays.

7. The digital twin remote rendering method according to claim 6, wherein The encoder is constructed according to the occurrence probabilities of multiple first values, the multiple first values covering the first value range, and the construction process of the encoder includes: Obtaining a first fitting model and a second fitting model, where the first fitting model generates multi-bit codes according to values, and the second fitting model generates values according to multi-bit codes; Sequentially inputting the multiple first values into the first fitting model to obtain multiple first codes; Respectively extracting the effective bits of each first code in the multiple first codes to obtain multiple effective bits; Calculating the correlation coefficient between the multiple effective bits and multiple target probabilities to obtain a first correlation coefficient, where the target probability is the probability of the effective bit source first value; If the first correlation coefficient is greater than the threshold, adjusting multiple parameters of the first fitting model and returning to the step of sequentially inputting the multiple first values into the first fitting model to obtain multiple first codes; Otherwise, fixing multiple parameters of the first fitting model and using the first fitting model as the encoder; Sequentially inputting the multiple first values into the encoder; Sequentially inputting the output of the encoder into the second fitting model to obtain multiple second values; Calculating the deviation between the multiple first values and the multiple second values as the decoding deviation; If the decoding deviation is greater than the threshold, adjusting multiple parameters of the second fitting model and jumping to the step of sequentially inputting the output of the encoder into the second fitting model to obtain multiple second values; Otherwise, fix multiple parameters of the second fitting model and use the second fitting model as a decoder.

8. A digital twin remote rendering device, characterized in that, A digital twin remote rendering apparatus for implementing the digital twin remote rendering method according to any one of claims 1-7, the digital twin remote rendering apparatus comprising: A call request acquisition module, configured to acquire an application programming interface call request; A call request processing module, configured to select, according to the attribute of the call request, to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine; A first forwarding module, configured to, after forwarding the call request to the proxy engine, forward the data of the proxy engine to the call request sender in response to the response of the proxy engine; And A second forwarding module, configured to, after forwarding a call request data packet to the proxy engine, decode the returned rendering result in response to the response of the proxy engine to obtain a rendered image.

9. An electronic device, comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 above are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 above are implemented.

Images