Digital twin remote rendering method and device, electronic equipment and storage medium

Abstract

This invention relates to the field of digital image rendering technology, and more particularly to a digital twin remote rendering method, apparatus, electronic device, and storage medium. The method first obtains an application programming interface (API) call request. Then, based on the attributes of the call request, it selects whether to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine. Next, after forwarding the call request to the proxy engine, in response to the proxy engine's response, the proxy engine's data is forwarded to the call request sender. Finally, after forwarding the call request data packet to the proxy engine, in response to the proxy engine's response, the returned rendering result is decoded to obtain the rendered image. This invention categorizes rendering requests into multiple types for classified processing, reducing data transmission volume, improving transmission efficiency, and alleviating the problem of remote rendering latency caused by low network transmission efficiency.

Description

Technical Field

[0001] This invention relates to the field of digital image rendering technology, and in particular to a digital twin remote rendering method, apparatus, electronic device and storage medium. Background Technology

[0002] Image rendering refers to the process of converting a 3D model or scene into a 2D image. It involves multiple steps and techniques in computer graphics, aiming to generate realistic or stylized visual effects. A rendering engine is a software framework or tool that implements image rendering. It encapsulates various algorithms and functions in the rendering process and provides a series of APIs for developers to use.

[0003] Remote rendering is a technology that sends rendering tasks from the local computer to a remote server or computer for processing. Users can submit rendering tasks from anywhere with a network connection, without being limited by local hardware devices.

[0004] Current remote rendering technologies mainly suffer from data latency and stuttering issues. Rendering tasks typically involve large amounts of data, including 3D models, textures, and material information. Uploading to a remote server and transmitting the rendering results back to the local machine require high network bandwidth to ensure fast data transmission. Insufficient bandwidth can lead to excessively long upload and download times, severely impacting work efficiency.

[0005] Therefore, it is necessary to develop and design a digital twin remote rendering method. Summary of the Invention

[0006] The present invention provides a digital twin remote rendering method, apparatus, electronic device and storage medium to solve the problem of remote rendering delay caused by low network transmission efficiency in the prior art.

[0007] In a first aspect, embodiments of the present invention provide a digital twin remote rendering method applied to a terminal, wherein the terminal is connected to a server, and the server is equipped with a proxy engine and a rendering engine, and the digital twin remote rendering method includes:

[0008] Get the application interface call request;

[0009] Based on the attributes of the call request, the following options are selected: execute the call request locally, add the call request to the call request data packet, forward the call request to the proxy engine, or send the call request data packet to the proxy engine.

[0010] After forwarding the call request to the proxy engine, in response to the response from the proxy engine, the data from the proxy engine is forwarded to the call request sender.

[0011] After forwarding the call request data packet to the proxy engine, the returned rendering result is decoded in response to the proxy engine's reply to obtain the rendered screen.

[0012] In one possible implementation, the step of selecting to execute the call request locally, adding the call request to a call request data packet, forwarding the call request to the proxy engine, or sending the call request data packet to the proxy engine based on the attributes of the call request includes:

[0013] Obtain the request parameters of the call request;

[0014] If the call request is a rendering request, then a call request data packet is sent to the proxy engine;

[0015] If the request parameters do not involve parameters provided by other terminal devices and the call request is not a rendering request, then the call request is executed locally.

[0016] If the request parameters involve parameters provided by other terminal devices and the call request is not a rendering request, then the call request will be forwarded to the proxy engine.

[0017] In one possible implementation, decoding the returned rendering result to obtain the rendered image includes:

[0018] Obtain the rendering result;

[0019] The rendering result is decoded using a decoder to obtain the frequency domain transform code matrix;

[0020] The frequency domain transform code matrix is ​​transformed according to the first formula to obtain the rendered image, wherein the first formula is:

[0021]

[0022] In the formula, IMG(x,y) is the pixel value in the x-th row and y-th column of the rendered image, MIMG(m,n) is the element in the m-th row and n-th column of the frequency domain transform code matrix, e is the natural constant, π is pi, i is the imaginary unit, M is the number of rows of the frequency domain transform code matrix, N is the number of columns of the frequency domain transform code matrix, X is the total number of rows of the rendered image, and Y is the total number of columns of the rendered image.

[0023] Secondly, embodiments of the present invention provide a digital twin remote rendering method, which is applied to a server, the server being connected to a terminal, the server having a proxy engine and a rendering engine, the server accepting requests from the terminal to complete the rendering of the image, and the digital twin remote rendering method comprising:

[0024] Obtain a call request data packet, wherein the call request data packet is obtained according to any one of claims 1-3 of the digital twin remote rendering method;

[0025] The call request data packet is parsed to extract the sending timestamp of the call request data packet;

[0026] Based on the sending timestamp, the current time, and the size of the call request data packet, the rendering parameters indicating the rendering precision of the rendering engine and the transformation precision index indicating the precision of the frequency domain transformation are generated, and the data parsed from the call request data packet is sent to the rendering engine to generate the rendering screen;

[0027] The rendering image is subjected to frequency domain transformation according to the transformation precision index to generate a frequency domain transformation code matrix, and the frequency domain transformation code matrix is ​​encoded to obtain the rendering result.

[0028] In one possible implementation, performing a frequency domain transformation on the rendered image to generate a frequency domain transformation code matrix includes:

[0029] Obtain the total number of rows and columns of pixels in the rendered image;

[0030] The number of rows and columns of the frequency domain transform code matrix are determined based on the transform precision index, the total number of rows of pixels in the rendered image, and the total number of columns of pixels.

[0031] The frequency domain transform code matrix is ​​obtained by performing a frequency domain transform based on the second formula, the number of rows in the frequency domain transform code matrix, and the number of columns in the frequency domain transform code matrix, wherein the second formula is:

[0032]

[0033] In the formula, MIMG(m,n) is the element in the m-th row and n-th column of the frequency domain transform code matrix, X is the total number of row pixels of the rendered image, Y is the total number of column pixels of the rendered image, IMG(x,y) is the pixel value in the x-th row and y-th column of the rendered image, e is the natural constant, π is pi, and i is the imaginary unit.

[0034] In one possible implementation, encoding the frequency domain transform code matrix to obtain the rendering result includes:

[0035] The frequency domain transform code matrix is ​​split into multiple first arrays by row or column;

[0036] By offsetting and scaling values, multiple values ​​of each first array are mapped to a first value range to obtain multiple second arrays and multiple mapping parameter arrays. The mapping parameter arrays include the parameters used when mapping the first arrays, and each mapping parameter array corresponds to a second array.

[0037] The multiple second arrays are input into the encoder to obtain the encoding queue;

[0038] The encoding queue and the plurality of mapping parameter arrays are used to construct the rendering result.

[0039] In one possible implementation, the encoder is constructed based on the probability of occurrence of a plurality of first values, the plurality of first values ​​covering a range of first values, and the construction process of the encoder includes:

[0040] Obtain a first fitting model and a second fitting model, wherein the first fitting model generates a multi-bit code based on the numerical value, and the second fitting model generates a numerical value based on the multi-bit code;

[0041] The plurality of first values ​​are sequentially input into the first fitting model to obtain a plurality of first codes;

[0042] Extract the effective number of bits from each of the plurality of first codes to obtain a plurality of effective bits;

[0043] Calculate the correlation coefficients between the plurality of valid bits and the plurality of target probabilities to obtain the first correlation coefficient, wherein the target probability is the probability of the valid bits originating from the first value;

[0044] If the first correlation coefficient is greater than the threshold, then adjust multiple parameters of the first fitting model and return to the step of sequentially inputting the multiple first values ​​into the first fitting model to obtain multiple first codes;

[0045] Otherwise, fix multiple parameters of the first fitting model and use the first fitting model as the encoder;

[0046] The plurality of first values ​​are sequentially input into the encoder;

[0047] The output of the encoder is sequentially input into the second fitting model to obtain multiple second values;

[0048] The deviation between the plurality of first values ​​and the plurality of second values ​​is calculated as the decoding deviation;

[0049] If the decoding deviation is greater than the threshold, then multiple parameters of the second fitting model are adjusted, and the process jumps to the step of sequentially inputting the encoder output into the second fitting model to obtain multiple second values;

[0050] Otherwise, fix multiple parameters of the second fitting model and use the second fitting model as a decoder.

[0051] Thirdly, embodiments of the present invention provide a digital twin remote rendering apparatus for implementing the digital twin remote rendering method as described in the first aspect or any possible implementation thereof, the digital twin remote rendering apparatus comprising:

[0052] The request retrieval module is used to retrieve application interface call requests;

[0053] The call request processing module is used to select, based on the attributes of the call request, whether to execute the call request locally, add the call request to the call request data packet, forward the call request to the proxy engine, or send the call request data packet to the proxy engine.

[0054] The first forwarding module is used to forward the data of the proxy engine to the call request sender in response to the response of the proxy engine after forwarding the call request to the proxy engine.

[0055] as well as,

[0056] The second forwarding module is used to decode the returned rendering result in response to the response of the proxy engine after forwarding the call request data packet to the proxy engine, and obtain the rendered screen.

[0057] Fourthly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0058] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0059] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0060] This invention discloses a digital twin remote rendering method. The method first obtains an application programming interface (API) call request. Then, based on the attributes of the call request, it selects whether to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine. Next, after forwarding the call request to the proxy engine, in response to the proxy engine's response, it forwards the proxy engine's data to the call request sender. Finally, after forwarding the call request data packet to the proxy engine, in response to the proxy engine's response, it decodes the returned rendering result to obtain the rendered image. The digital twin remote rendering method for terminals provided by this invention categorizes rendering requests into multiple types. One type is executed locally on the terminal, which reduces network transmission latency and server processing pressure since no data transmission is required. For requests that do not require immediate execution, these requests are added to a request data packet. When sending conditions are met, multiple requests included in the request data packet are sent to the server together, where they are centrally processed. This reduces the communication process between the terminal and the server, alleviating communication pressure. For the rendered image, the server transforms and encodes the rendered image to form a rendering result. The data volume of the rendering result is smaller than that of the rendered image, reducing data transmission volume, improving transmission efficiency, and alleviating the problem of remote rendering latency caused by low network transmission efficiency. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of a digital twin remote rendering method for terminals provided by an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of the digital twin remote rendering system provided in an embodiment of the present invention;

[0064] Figure 3 This is a flowchart of a digital twin remote rendering method for servers provided by an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the rendering screen encoding process provided by an embodiment of the present invention;

[0066] Figure 5This is a functional block diagram of the digital twin remote rendering device provided in the embodiments of the present invention;

[0067] Figure 6 This is a functional block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0069] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0070] The embodiments of the present invention will be described in detail below. These examples are implemented based on the technical solutions of the present invention, and detailed implementation methods and specific operation processes are given. However, the scope of protection of the present invention is not limited to the following embodiments.

[0071] Figure 1 A flowchart of a digital twin remote rendering method provided for an embodiment of the present invention.

[0072] like Figure 1 As shown, a flowchart of a digital twin remote rendering method for a terminal provided by an embodiment of the present invention is illustrated, and is described in detail below:

[0073] In step 101, the application interface call request is obtained.

[0074] In step 102, based on the attributes of the call request, the following options are selected: execute the call request locally, add the call request to the call request data packet, forward the call request to the proxy engine, or send the call request data packet to the proxy engine.

[0075] In some implementations, the step of selecting to execute the call request locally, adding the call request to a call request data packet, forwarding the call request to the proxy engine, or sending a call request data packet to the proxy engine based on the attributes of the call request includes:

[0076] Obtain the request parameters of the call request;

[0077] If the call request is a rendering request, then a call request data packet is sent to the proxy engine;

[0078] If the request parameters do not involve parameters provided by other terminal devices and the call request is not a rendering request, then the call request is executed locally.

[0079] If the request parameters involve parameters provided by other terminal devices and the call request is not a rendering request, then the call request will be forwarded to the proxy engine.

[0080] In step 103, after forwarding the call request to the proxy engine, the proxy engine's data is forwarded to the call request sender in response to the proxy engine's response.

[0081] For example, as mentioned above, remote rendering is affected by network transmission speed, resulting in data latency and screen stuttering. The present invention aims to improve the efficiency of remote rendering and reduce the impact of data transmission issues by improving data transmission schemes, data processing schemes, and data processing of rendered screens.

[0082] like Figure 2 As shown in the figure, this diagram illustrates the principle of a digital twin remote rendering system provided by an embodiment of the present invention. In the figure, terminal 201 generates various requests related to remote rendering. These requests may require server 202 to process data or render the model of the object being rendered. They may also require accessing data generated by other terminals. For example, if other terminals have influenced the model of the object being rendered, the data from those other terminals that influenced the model needs to be accessed before the final rendering request is sent to server 202. Server 202 is equipped with a proxy engine and a rendering engine. The rendering engine is used to render the model and generate the rendered image, while the proxy engine is used to receive requests from terminals and transform these requests into rendering image requests. It is used to encode and send the rendered image, or to provide data generated by other terminals. In other words, the two engines are responsible for image rendering and data interaction respectively. This allows for the reasonable allocation of rendering tasks and reduces data transmission volume through encoding.

[0083] To achieve the above objectives, this invention categorizes remote rendering requests from the terminal into four types: requests that can be invoked locally, requests that require server execution but are not executed immediately, requests that require forwarding (e.g., requests where the server possesses complete data and execution conditions for the request, or requests that require the server to provide the request data), and requests that require immediate execution to the server.

[0084] In this process, requests made locally will have their application interface execution interrupted. Requests that need to be executed by the server but are not executed immediately will be added to the request data packet. When the conditions for executing the request are met, the request data packet will be sent to the server. The server will parse out multiple requests from the request data packet and execute the request.

[0085] Requests that need to be forwarded are typically sent immediately to the server. The server usually has the data and application programming interface (API) to execute the forwarded request. It will then execute the request and return the resulting data, which will be sent back to the requesting client. However, in some scenarios, the server may only provide the data it can handle.

[0086] For requests that require immediate server execution, they are added to a request packet and immediately sent to the server for execution. Upon receiving the request packet, the server's proxy engine parses it, separating the data into rendering requests and other requests. The other requests are executed by the proxy engine, while the rendering requests invoke the rendering engine. The rendered image is then converted and encoded to form the final rendering result, which is returned to the terminal. It's important to note that the conversion and encoding process compresses the rendered image to reduce data volume, thereby improving data transmission efficiency and achieving data encryption.

[0087] After receiving the rendering result, the terminal will decode and inversely transform the rendering result to restore the rendered image.

[0088] In step 104, after forwarding the call request data packet to the proxy engine, the returned rendering result is decoded in response to the proxy engine's response to obtain the rendered screen.

[0089] In some implementations, decoding the returned rendering result to obtain the rendered image includes:

[0090] Obtain the rendering result;

[0091] The rendering result is decoded using a decoder to obtain the frequency domain transform code matrix;

[0092] The frequency domain transform code matrix is ​​transformed according to the first formula to obtain the rendered image, wherein the first formula is:

[0093]

[0094] In the formula, IMG(x,y) is the pixel value in the x-th row and y-th column of the rendered image, MIMG(m,n) is the element in the m-th row and n-th column of the frequency domain transform code matrix, e is the natural constant, π is pi, i is the imaginary unit, M is the number of rows of the frequency domain transform code matrix, N is the number of columns of the frequency domain transform code matrix, X is the total number of rows of the rendered image, and Y is the total number of columns of the rendered image.

[0095] For example, in decoding the rendering result, the embodiments of the present invention first use a decoder to decode the rendering result to obtain a frequency domain transform code matrix. This matrix is ​​then inversely transformed to obtain the rendered image. In one scenario, the following formula is used:

[0096]

[0097] In the formula, IMG(x,y) is the pixel value in the x-th row and y-th column of the rendered image, MIMG(m,n) is the element in the m-th row and n-th column of the frequency domain transform code matrix, e is the natural constant, π is pi, i is the imaginary unit, M is the number of rows of the frequency domain transform code matrix, N is the number of columns of the frequency domain transform code matrix, X is the total number of rows of the rendered image, and Y is the total number of columns of the rendered image.

[0098] The digital twin remote rendering method for terminals provided by this invention categorizes rendering requests into multiple types. One type is executed locally on the terminal, which reduces network transmission latency and server processing pressure since no data transmission is required. For requests that do not require immediate execution, these requests are added to a request data packet. When sending conditions are met, multiple requests included in the request data packet are sent to the server together, where they are centrally processed. This reduces the communication process between the terminal and the server, alleviating communication pressure. For the rendered image, the server transforms and encodes the rendered image to form a rendering result. The data volume of the rendering result is smaller than that of the rendered image, reducing data transmission volume and improving transmission efficiency.

[0099] like Figure 3 As shown, a flowchart of a digital twin remote rendering method for a server provided by an embodiment of the present invention is illustrated, and is described in detail below:

[0100] In step 301, a call request data packet is obtained, wherein the call request data packet is obtained according to the digital twin remote rendering method described in the first aspect.

[0101] In step 302, the call request data packet is parsed to extract the sending timestamp of the call request data packet.

[0102] In step 303, rendering parameters indicating the rendering precision of the rendering engine and a transformation precision index indicating the precision of the frequency domain transformation are generated based on the sending timestamp, the current time, and the size of the call request data packet. The data parsed from the call request data packet is then sent to the rendering engine to generate the rendered image.

[0103] For example, on the server side, after receiving a call request data packet, the server parses it and executes the requests sequentially according to the timestamps. The rendering request, as an immediately executed request, is typically located at the end of the request data packet. Besides indicating the order of request execution, the timestamp also serves to evaluate the data transmission between the terminal and the server.

[0104] In other words, the server determines the data transmission speed between the terminal and the server based on the size of the request data packet, the terminal's sending timestamp, and the current time. This invention adjusts the rendering precision and frequency domain transformation precision based on this transmission speed. That is, the lower the data transmission speed, the lower the rendering precision and transformation precision are simultaneously reduced to ensure the real-time performance of the image.

[0105] In step 304, the rendered image is subjected to frequency domain transformation according to the transformation precision index to generate a frequency domain transformation code matrix, and the frequency domain transformation code matrix is ​​encoded to obtain the rendering result.

[0106] In some implementations, performing a frequency domain transformation on the rendered image to generate a frequency domain transform code matrix includes:

[0107] Obtain the total number of rows and columns of pixels in the rendered image;

[0108] The number of rows and columns of the frequency domain transform code matrix are determined based on the transform precision index, the total number of rows of pixels in the rendered image, and the total number of columns of pixels.

[0109] The frequency domain transform code matrix is ​​obtained by performing a frequency domain transform based on the second formula, the number of rows in the frequency domain transform code matrix, and the number of columns in the frequency domain transform code matrix, wherein the second formula is:

[0110]

[0111] In the formula, MIMG(m,n) is the element in the m-th row and n-th column of the frequency domain transform code matrix, X is the total number of row pixels of the rendered image, Y is the total number of column pixels of the rendered image, IMG(x,y) is the pixel value in the x-th row and y-th column of the rendered image, e is the natural constant, π is pi, and i is the imaginary unit.

[0112] For example, such as Figure 4As shown in the figure, this is a schematic diagram of the rendering image encoding process provided by the present invention. After the rendering engine generates the rendering image 401 and returns it to the proxy engine, the proxy engine performs a frequency domain transformation on the rendering image 401 to generate a frequency domain transformation code matrix 402. This frequency domain transformation code matrix 402 is split into arrays 403 by row or column. These arrays 403 are mapped to predetermined numerical ranges to form new arrays (the mapping conditions are preserved). The new arrays are sent to the encoder 404, and the resulting encoding and mapping conditions are used to construct the rendering result, which is then sent to the terminal. It should be noted that the encoder 404 and decoder 405 are obtained by fitting a model to a ground pair.

[0113] Regarding frequency domain transformation, this invention controls the number of rows and columns of the frequency domain transformed code matrix obtained after the transformation based on the transformation precision index obtained in the aforementioned steps. This invention provides a transformation formula:

[0114]

[0115] In the formula, MIMG(m,n) is the element in the m-th row and n-th column of the frequency domain transform code matrix, X is the total number of row pixels of the rendered image, Y is the total number of column pixels of the rendered image, IMG(x,y) is the pixel value in the x-th row and y-th column of the rendered image, e is the natural constant, π is pi, and i is the imaginary unit.

[0116] In some implementations, encoding the frequency domain transform code matrix to obtain the rendering result includes:

[0117] The frequency domain transform code matrix is ​​split into multiple first arrays by row or column;

[0118] By offsetting and scaling values, multiple values ​​of each first array are mapped to a first value range to obtain multiple second arrays and multiple mapping parameter arrays. The mapping parameter arrays include the parameters used when mapping the first arrays, and each mapping parameter array corresponds to a second array.

[0119] The multiple second arrays are input into the encoder to obtain the encoding queue;

[0120] The encoding queue and the plurality of mapping parameter arrays are used to construct the rendering result.

[0121] In some implementations, the encoder is constructed based on the occurrence probabilities of a plurality of first values, the plurality of first values ​​covering a range of first values, and the construction process of the encoder includes:

[0122] Obtain a first fitting model and a second fitting model, wherein the first fitting model generates a multi-bit code based on the numerical value, and the second fitting model generates a numerical value based on the multi-bit code;

[0123] The plurality of first values ​​are sequentially input into the first fitting model to obtain a plurality of first codes;

[0124] Extract the effective number of bits from each of the plurality of first codes to obtain a plurality of effective bits;

[0125] Calculate the correlation coefficients between the plurality of valid bits and the plurality of target probabilities to obtain the first correlation coefficient, wherein the target probability is the probability of the valid bits originating from the first value;

[0126] If the first correlation coefficient is greater than the threshold, then adjust multiple parameters of the first fitting model and return to the step of sequentially inputting the multiple first values ​​into the first fitting model to obtain multiple first codes;

[0127] Otherwise, fix multiple parameters of the first fitting model and use the first fitting model as the encoder;

[0128] The plurality of first values ​​are sequentially input into the encoder;

[0129] The output of the encoder is sequentially input into the second fitting model to obtain multiple second values;

[0130] The deviation between the plurality of first values ​​and the plurality of second values ​​is calculated as the decoding deviation;

[0131] If the decoding deviation is greater than the threshold, then multiple parameters of the second fitting model are adjusted, and the process jumps to the step of sequentially inputting the encoder output into the second fitting model to obtain multiple second values;

[0132] Otherwise, fix multiple parameters of the second fitting model and use the second fitting model as a decoder.

[0133] For example, as mentioned above, the frequency domain transform code matrix is ​​split into multiple arrays by row or column. Since the values ​​in these arrays are widely distributed, this invention performs interval unification, or mapping, on the arrays. Specifically, this is done by offsetting and scaling the values ​​in the arrays to map them to a target interval. For example, if the value distribution of an array obtained from a certain split is n390-n7890, it can be mapped to n00-n255 by offsetting and scaling, using the following formula:

[0134]

[0135] In the above formula, n (1) The value obtained after mapping. To map the maximum value of the target interval (n255 in the example above), n(0) The value being mapped. The minimum value of the mapped array (n390 in the example above). This is the maximum value of the mapped array (n7890 in the example above).

[0136] After the above process, the frequency of the values ​​in the frequency domain transform code matrix is ​​greatly increased. This has the advantage of significantly improving the coding efficiency and reducing the total amount of data after coding.

[0137] In terms of the construction of the encoder and decoder, the present invention constructs them in pairs using two fitting models based on the probability of the occurrence of the encoded data. The fitting models are, for example, artificial neural network models or multinomial regression models.

[0138] First, an encoder is constructed. The encoder encodes frequently occurring values ​​into shorter codes and infrequently occurring values ​​into longer codes. Integer values ​​within these ranges are sequentially input into a fitting model (used to construct the encoder). The fitting model outputs codes, and the significant bits of the codes (e.g., in the code 000nnn, the significant bits are nnn) are correlated with the probabilities of the integer values ​​(e.g., Pearson correlation coefficient). If the correlation coefficient is greater than a threshold, it indicates a strong positive correlation. The parameters of the fitting model are then adjusted, and the process of sequentially inputting integer values ​​into the fitting model is repeated until a strong negative correlation is observed between the significant bits of the codes and their probabilities. At this point, the parameters of the fitting model are fixed, and this model serves as the encoder.

[0139] Then, a decoder is constructed. The integer values ​​in the above range are sequentially input into the encoder, which outputs codes. These codes are sequentially fed into the fitting model (used to construct the fitting model for the decoder). The decoder provides multiple values, and the deviation between these values ​​and the values ​​input into the encoder is calculated. If the deviation is greater than a threshold, the parameters of the fitting model are adjusted, and the process returns to the step where the codes are sequentially fed into the fitting model. The above steps are repeated until the deviation is less than the threshold. At this point, the parameters of the fitting model are fixed and used as the decoder.

[0140] The implementation method of the digital twin remote rendering method of the present invention first obtains an application programming interface (API) call request, and then, according to the attributes of the call request, selects to execute the call request locally, add the call request to a call request data packet, forward the call request to a proxy engine, or send a call request data packet to the proxy engine. Next, after forwarding the call request to the proxy engine, in response to the response of the proxy engine, the data of the proxy engine is forwarded to the call request sender. Finally, after forwarding the call request data packet to the proxy engine, in response to the response of the proxy engine, the returned rendering result is decoded to obtain the rendered screen. The digital twin remote rendering method for terminals provided by this invention categorizes rendering requests into multiple types. One type is executed locally on the terminal, which reduces network transmission latency and server processing pressure since no data transmission is required. For requests that do not require immediate execution, these requests are added to a request data packet. When sending conditions are met, multiple requests included in the request data packet are sent to the server together, where they are centrally processed. This reduces the communication process between the terminal and the server, alleviating communication pressure. For the rendered image, the server transforms and encodes the rendered image to form a rendering result. The data volume of the rendering result is smaller than that of the rendered image, reducing data transmission volume, improving transmission efficiency, and alleviating the problem of remote rendering latency caused by low network transmission efficiency.

[0141] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0142] The following are embodiments of the apparatus of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0143] Figure 5 This is a functional block diagram of the digital twin remote rendering device provided in an embodiment of the present invention, with reference to... Figure 5 The digital twin remote rendering device includes: a call request acquisition module 501, a call request processing module 502, a first forwarding module 503, and a second forwarding module 504, wherein:

[0144] The request retrieval module 501 is used to retrieve application interface call requests;

[0145] The call request processing module 502 is used to select, based on the attributes of the call request, to execute the call request locally, add the call request to the call request data packet, forward the call request to the proxy engine, or send the call request data packet to the proxy engine;

[0146] The first forwarding module 503 is used to forward the data of the proxy engine to the call request sender in response to the response of the proxy engine after forwarding the call request to the proxy engine;

[0147] The second forwarding module 504 is used to decode the returned rendering result in response to the response of the proxy engine after forwarding the call request data packet to the proxy engine, and obtain the rendered screen.

[0148] Figure 6 This is a functional block diagram of the electronic device provided in an embodiment of the present invention. For example... Figure 6 As shown, the electronic device 6 of this embodiment includes a processor 600 and a memory 601, wherein the memory 601 stores a computer program 602 that can run on the processor 600. When the processor 600 executes the computer program 602, it implements the steps of the various digital twin remote rendering methods and embodiments described above, for example... Figure 1 Steps 101 to 104 are shown.

[0149] For example, the computer program 602 may be divided into one or more modules / units, which are stored in the memory 601 and executed by the processor 600 to complete the present invention.

[0150] The electronic device 6 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The electronic device 6 may include, but is not limited to, a processor 600 and a memory 601. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 6 may also include input / output devices, network access devices, buses, etc.

[0151] The processor 600 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0152] The memory 601 can be an internal storage unit of the electronic device 6, such as a hard disk or memory. The memory 601 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 601 can include both internal and external storage units of the electronic device 6. The memory 601 is used to store the computer program 602 and other programs and data required by the electronic device 6. The memory 601 can also be used to temporarily store data that has been output or will be output.

[0153] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0154] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0156] In the embodiments provided by this invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0158] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various methods and apparatus embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0160] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A digital twin remote rendering method, characterized in that, Applied to a server, the server is connected to a terminal. The server is equipped with a proxy engine and a rendering engine. The server accepts requests from the terminal and completes the rendering of the image. The digital twin remote rendering method includes: Obtain the call request data packet, wherein the call request data packet is obtained according to the digital twin remote rendering method of the terminal application; The call request data packet is parsed to extract the sending timestamp of the call request data packet; Based on the sending timestamp, the current time, and the size of the call request data packet, the rendering parameters indicating the rendering precision of the rendering engine and the transformation precision index indicating the precision of the frequency domain transformation are generated, and the data parsed from the call request data packet is sent to the rendering engine to generate the rendering screen; The rendered image is subjected to frequency domain transformation based on the transformation precision index to generate a frequency domain transformation code matrix. The frequency domain transformation code matrix is ​​then encoded to obtain the rendering result, including: The frequency domain transform code matrix is ​​split into multiple first arrays by row or column; By offsetting and scaling values, multiple values ​​of each first array are mapped to a first value range to obtain multiple second arrays and multiple mapping parameter arrays. The mapping parameter arrays include the parameters used when mapping the first arrays, and each mapping parameter array corresponds to a second array. The multiple second arrays are input into the encoder to obtain the encoding queue; The encoding queue and the plurality of mapping parameter arrays are used to construct the rendering result; The encoder is constructed based on the occurrence probabilities of multiple first values, which cover a range of first values. The construction process of the encoder includes: Obtain a first fitting model and a second fitting model, wherein the first fitting model generates a multi-bit code based on the numerical value, and the second fitting model generates a numerical value based on the multi-bit code; The plurality of first values ​​are sequentially input into the first fitting model to obtain a plurality of first codes; Extract the effective number of bits from each of the plurality of first codes to obtain a plurality of effective bits; Calculate the correlation coefficients between the plurality of valid bits and the plurality of target probabilities to obtain the first correlation coefficient, wherein the target probability is the probability of the valid bits originating from the first value; If the first correlation coefficient is greater than the threshold, then adjust multiple parameters of the first fitting model and return to the step of sequentially inputting the multiple first values ​​into the first fitting model to obtain multiple first codes; Otherwise, fix multiple parameters of the first fitting model and use the first fitting model as the encoder; The plurality of first values ​​are sequentially input into the encoder; The output of the encoder is sequentially input into the second fitting model to obtain multiple second values; The deviation between the plurality of first values ​​and the plurality of second values ​​is calculated as the decoding deviation; If the decoding deviation is greater than the threshold, then multiple parameters of the second fitting model are adjusted, and the process jumps to the step of sequentially inputting the encoder output into the second fitting model to obtain multiple second values; Otherwise, fix multiple parameters of the second fitting model and use the second fitting model as a decoder.

2. The digital twin remote rendering method according to claim 1, characterized in that, The step of performing a frequency domain transformation on the rendered image to generate a frequency domain transformation code matrix includes: Obtain the total number of rows and columns of pixels in the rendered image; The number of rows and columns of the frequency domain transform code matrix are determined based on the transform precision index, the total number of rows of pixels in the rendered image, and the total number of columns of pixels. The frequency domain transform code matrix is ​​obtained by performing a frequency domain transform based on the second formula, the number of rows in the frequency domain transform code matrix, and the number of columns in the frequency domain transform code matrix, wherein the second formula is: In the formula, For the frequency domain transform code matrix, the first... Line 1 Column elements, This represents the total number of rows of pixels in the rendered image. This represents the total number of column pixels in the rendered image. To render the image Line 1 Column pixel values, It is a natural constant. Pi It is the imaginary unit.

3. A digital twin remote rendering method, characterized in that, The method is applied to a terminal connected to a server, the server having a proxy engine and a rendering engine, and the server employing the digital twin remote rendering method as described in any one of claims 1-2, the digital twin remote rendering method comprising: Get the application interface call request; Based on the attributes of the call request, the following options are selected: execute the call request locally, add the call request to the call request data packet, forward the call request to the proxy engine, or send the call request data packet to the proxy engine. After forwarding the call request to the proxy engine, in response to the response from the proxy engine, the data from the proxy engine is forwarded to the call request sender. After forwarding the call request data packet to the proxy engine, the returned rendering result is decoded in response to the proxy engine's reply to obtain the rendered screen.

4. The digital twin remote rendering method according to claim 3, characterized in that, The step of selecting to execute the call request locally, adding the call request to the call request data packet, forwarding the call request to the proxy engine, or sending the call request data packet to the proxy engine based on the attributes of the call request includes: Obtain the request parameters of the call request; If the call request is a rendering request, then a call request data packet is sent to the proxy engine; If the request parameters do not involve parameters provided by other terminal devices and the call request is not a rendering request, then the call request is executed locally. If the request parameters involve parameters provided by other terminal devices and the call request is not a rendering request, then the call request will be forwarded to the proxy engine.

5. The digital twin remote rendering method according to any one of claims 3-4, characterized in that, Decoding the returned rendering result to obtain the rendered image includes: Obtain the rendering result; The rendering result is decoded using a decoder to obtain the frequency domain transform code matrix; The frequency domain transform code matrix is ​​transformed according to the first formula to obtain the rendered image, wherein the first formula is: In the formula, To render the image Line 1 Column pixel values, For the frequency domain transform code matrix, the first... Line 1 Column elements, It is a natural constant. Pi The imaginary unit, Let be the row number of the frequency domain transform code matrix. The column number of the frequency domain transform code matrix. This represents the total number of rows of pixels in the rendered image. This represents the total number of columns of pixels in the rendered image.

6. A digital twin remote rendering device, characterized in that, For implementing the digital twin remote rendering method as described in any one of claims 1-5, the digital twin remote rendering apparatus comprises: The request retrieval module is used to retrieve application interface call requests; The call request processing module is used to select, based on the attributes of the call request, whether to execute the call request locally, add the call request to the call request data packet, forward the call request to the proxy engine, or send the call request data packet to the proxy engine. The first forwarding module is used to forward the data of the proxy engine to the call request sender in response to the response of the proxy engine after forwarding the call request to the proxy engine. as well as, The second forwarding module is used to decode the returned rendering result in response to the response of the proxy engine after forwarding the call request data packet to the proxy engine, and obtain the rendered screen.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5 above.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5 above.

Images