Rendering method and apparatus, and device-cloud collaborative system

Abstract

Provided in the embodiments of the present application are a rendering method and apparatus, and a device-cloud collaborative system. The method comprises: first, receiving a first candidate simulation result of the current frame; next, fusing a second candidate simulation result of the current frame and the first candidate simulation result of the current frame, so as to obtain a target simulation result of the current frame; and then, on the basis of the target simulation result of the current frame, rendering a three-dimensional scene, so as to obtain a rendered image of the current frame, wherein the first candidate simulation result of the current frame is a simulation result corresponding to a first time point, the first time point is a predicted time point at which a terminal device receives the first candidate simulation result of the current frame, the second candidate simulation result of the current frame is a simulation result corresponding to a second time point, and the second time point is a time point at which the terminal device actually receives the first candidate simulation result of the current frame. Device-cloud collaborative physical simulation can reduce the delay in receiving a physical simulation result by a device side, thereby reducing the probability of frame stuttering.

Description

Rendering method, device and end-cloud collaborative system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on February 4, 2024, with application number 202410163509.1 and application name “Rendering Method, Device and End-Cloud Collaborative System”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of graphics processing, and in particular to a rendering method, device, and end-cloud collaborative system. Background Art

[0003] Currently, the implementation of applications such as cloud gaming, the metaverse, digital human live streaming, virtual reality (VR), and augmented reality (AR) involves not only rendering but also other graphics processing, such as physics simulation and character animation prediction. Typically, graphics processing engines other than the rendering engine perform these other graphics processing tasks and generate these results over time. The rendering engine then renders based on these results, creating realistic images for users.

[0004] Since rendering and other graphics processing have high requirements for devices, a terminal-cloud collaboration approach is usually adopted, that is, the cloud side performs other graphics processing and sends the other graphics processing results to the terminal side, and then the terminal side performs rendering based on the other graphics processing results.

[0005] However, in actual applications, network delays and device freezes often occur, which will cause delays in other graphics processing results received by the terminal side, resulting in screen freezes and other problems. Summary of the Invention

[0006] In view of this, the present application provides a rendering method, device and end-cloud collaborative system, which can reduce the delay of other graphics processing results (such as physical simulation results, character animation prediction results, etc.) received on the end side, thereby reducing the probability of screen freezes.

[0007] In a first aspect, the present application provides a device-cloud collaboration system, which includes a server and a terminal device, wherein:

[0008] Terminal device, used to send physical simulation data to the server;

[0009] The server is configured to perform physical simulation on the physical simulation object based on the physical simulation data to obtain a first candidate simulation result of the current frame; and send the first candidate simulation result of the current frame to the terminal device;

[0010] The terminal device is further configured to fuse the second candidate simulation result of the current frame with the first candidate simulation result of the current frame to obtain a target simulation result of the current frame; and render the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame;

[0011] In which, the three-dimensional scene includes a physical simulation object; the first candidate simulation result of the current frame is the simulation result corresponding to the first time point, and the first time point is the predicted time point when the terminal device receives the first candidate simulation result of the current frame; the second candidate simulation result of the current frame corresponds to the simulation result corresponding to the second time point, and the second time point is the time point when the terminal device actually receives the first candidate simulation result of the current frame.

[0012] First, since it takes a certain amount of time for the interactive data to be transmitted from the terminal device to the server (or from the server to the terminal device), the time point (i.e., the first time point) when the terminal device receives the first candidate simulation result of the current frame can be predicted first; then, the physical simulation result corresponding to the first time point is generated as the first candidate simulation result of the current frame; in this way, the time point corresponding to the first candidate simulation result received by the terminal device is relatively close to the actual rendering and display time point, which can reduce the motion state delay of the physical simulation object, and thereby reduce the probability of screen freezes.

[0013] Secondly, since the first time point and the second time point may deviate, the second time point is closer to the real time point when the terminal device plays the rendered picture of the current frame relative to the first time point; therefore, fusing the second candidate simulation result of the current frame with the first candidate simulation result of the current frame can make the motion state of the physical simulation object closer to the actual motion state of the current frame corresponding to the real time point. In addition, fusing the second candidate simulation result of the current frame with the first candidate simulation result of the current frame can be understood as compromising the motion state of the physical simulation object at the first time point with the motion state of the physical simulation object at the second time point; when the first time point is after the second time point, the first time point is far away from the real time point when the terminal device plays the previous frame of the rendered picture; by compromising the motion state of the physical simulation object at the first time point with the motion state of the physical simulation object at the second time point, the change between the motion state of the physical simulation object in the current frame and the motion state of the previous frame can be made smoother, so that the user can see the motion state of the physical simulation object in the two adjacent frames more smoothly. When the first time point is before the second time point, this will cause a delay in the motion state of the physical simulation object. By compromising the motion state of the physical simulation object at the first time point with the motion state of the physical simulation object at the second time point, the delay in the motion state of the physical simulation object can be reduced, thereby reducing the probability of screen freezes.

[0014] Exemplarily, the end-cloud collaborative system of the present application can be applied to at least one of the following scenarios: cloud gaming, AR / VR conferences, virtual concerts, digital human live broadcasts, etc., and the present application does not impose any restrictions on this.

[0015] Exemplarily, the server may also be referred to as a cloud or cloud server. The specific implementation form of the server in this application may be a cloud server, a physical (stand-alone) server, a station cluster server, etc., and this application does not limit this. When applied to different scenarios, the server in this application may also be referred to as a game server, a video server, a live broadcast server, a VR / AR conference server, a virtual server, etc., depending on the scenario applied, and this application does not limit this.

[0016] Exemplarily, terminal devices include but are not limited to: personal computers (PCs), computer workstations, smart phones, tablets, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices (such as AR helmets, VR glasses), set-top boxes, game consoles, etc., which are not limited in this application.

[0017] For example, physical simulation (also known as physics simulation) is a technology that uses computers to simulate real-world physical phenomena in order to create more realistic interactions with the world in animations and games. Physical simulation can include rigid body simulation (which can be used to simulate the rotation and translation of rigid objects), elastic body simulation (which can be used to simulate the movement and deformation of elastic objects), cloth simulation, and fluid simulation.

[0018] For example, rendering may refer to a process of displaying a three-dimensional model of an object (eg, a character, a character's clothes, a character's props, etc.) in a three-dimensional scene onto a two-dimensional plane.

[0019] Exemplarily, the physical simulation objects involved in this application are bound to the characters, and the motion state of the physical simulation objects can change as the motion state of the characters changes. The characters involved in this application may include game characters, meeting characters, virtual singers, and digital people and other virtual objects; virtual objects correspond to entities (such as people, animals, etc.) that can move autonomously and whose movements are affected by external forces. For example, in a cloud game scene, physical simulation objects include game character clothes, game character props, etc. For another example, in an AR / VR conference scene, physical simulation objects include clothes for meeting characters. For another example, in a virtual concert scene, physical simulation objects include clothes for virtual singers. For another example, in a digital human live broadcast scene, virtual simulation objects include digital human clothes.

[0020] For example, after receiving the user operation, the terminal device may send the physical simulation data to the server. For example, the user operation may include at least one of adjusting the position of the character, adjusting the orientation of the character, or adjusting the action of the character (e.g., walking, running, or jumping, etc.; which may also be understood as adjusting the animation sequence of the character).

[0021] Exemplarily, physical simulation data may include data required for physical simulation (or in other words, physical simulation data refers to data that affects the motion state of a physical simulation object). Exemplarily, physical simulation data may include, but is not limited to: physical simulation parameters, the data processed by physical simulation itself, etc., and this application does not impose any restrictions on this. Among them, physical simulation parameters may include at least one of the following: external force parameters (such as wind parameters, the position and orientation of external collision bodies, etc.), character animation data (which can be used to describe the character's animation state, such as the character's (adjusted) position and orientation, the playback position corresponding to the character's animation sequence, etc.) and physical simulation object state information (for example, the position and orientation of a rigid body (adjusted) (which can be understood as the position and orientation of the center of the rigid body)), etc., and this application does not impose any restrictions on this. The data processed by physical simulation itself may include cloth models, rigid body models, etc., and this application does not impose any restrictions on this.

[0022] For example, in a cloud gaming scenario, physical simulation data includes wind parameters, descriptions of objects that may collide with the game character's clothes, and the character's animation data.

[0023] For example, in a cloud gaming scenario, physical simulation data includes wind parameters, descriptions of objects that may collide with the game character's clothes, character animation data, and the fabric model corresponding to the game character's clothes.

[0024] For example, in a cloud gaming scenario, physical simulation data includes the animation data of the game character, and the position and orientation of the game character's props.

[0025] For example, in a cloud gaming scenario, physical simulation data includes the animation data of the game character, the position and orientation of the game character's props, and the rigid body model corresponding to the game character's props.

[0026] It should be noted that the first time point may refer to the predicted time point when the terminal device starts to receive the first candidate simulation result of the current frame, or it may refer to the predicted time point when the terminal device completes receiving the first candidate simulation result of the current frame (the time difference between these two situations is very small and can be ignored in some scenarios. Therefore, the first time point may also be any time point between the time points corresponding to these two situations). Alternatively, the first time point may refer to the predicted time point when the terminal device starts to receive the encoded code stream, or it may refer to the predicted time point when the terminal device completes receiving the encoded code stream, or it may be the predicted time point when the terminal device decodes the encoded code stream to obtain the reconstruction result of the first candidate simulation result of the current frame (the time difference between these three situations is very small and can be ignored in some scenarios. Therefore, the first time point may also be any time point between any two time points in the time points corresponding to these three situations). The encoded code stream is obtained by encoding the first candidate simulation result. A field in the first few fields of the encoded code stream carries the length of the encoded code stream, and the terminal device can parse the field from the encoded code stream to determine whether the encoded code stream has been received.

[0027] It should be noted that the second time point may refer to the time point when the terminal device actually starts to receive the first candidate simulation result of the current frame, or it may refer to the time point when the terminal device actually completes receiving the first candidate simulation result of the current frame (the time difference between these two situations is very small and can be ignored in some scenarios; therefore, the second time point may also be any time point between the time points corresponding to these two situations). Alternatively, the second time point may refer to the time point when the terminal device actually starts to receive the encoded code stream, or it may refer to the time point when the terminal device actually completes receiving the encoded code stream, or it may be the time point when the terminal device decodes the encoded code stream to obtain the reconstructed result of the first candidate simulation result of the current frame (the time difference between the three situations is very small and can be ignored in some scenarios; therefore, the second time point may also be any time point between any two time points corresponding to these three situations).

[0028] Exemplarily, the server may compress the first candidate simulation result of the current frame before sending it, or may directly send the uncompressed first candidate simulation result of the current frame, which is not limited in this application.

[0029] Exemplarily, a three-dimensional scene may include a character and a physical simulation object bound to the character; it should be understood that a three-dimensional scene may also include other objects (for example, a character whose position, orientation, and action have not been adjusted by the user, other physical simulation objects, etc.), and this application does not limit this.

[0030] In one possible way, the second candidate simulation result of the current frame is obtained by the terminal device performing physical simulation based on the physical simulation data. It should be noted that the complexity of the physical simulation (which can be called "simple physical simulation") performed by the terminal device to obtain the second candidate simulation result of the current frame is lower than the complexity of the physical simulation (which can be called "complex physical simulation") performed by the server to obtain the first candidate simulation result of the current frame; in other words, the computing power required for the physical simulation performed by the terminal device to obtain the second candidate simulation result of the current frame is lower than the computing power required for the physical simulation performed by the server to obtain the first candidate simulation result of the current frame. Correspondingly, the accuracy of the second candidate simulation result of the current frame is lower than the accuracy of the first candidate simulation result of the current frame.

[0031] In one possible way, the second candidate simulation result of the current frame is pre-baked and generated by the terminal device, or pre-baked and generated by the server (or third-party device) and sent to the terminal device. Among them, the terminal device (or server or third-party device) can perform N (N is a positive integer) physical simulations in advance (the physical simulation in the pre-baking process has the same complexity as the physical simulation performed by the server in S302, and can also be called "complex physical simulation") to obtain N third candidate simulation results; these N third candidate simulation results correspond one-to-one to N preset user operations. Then, from the N third candidate simulation results, the third candidate simulation result that matches the user operation received this time is selected as the second candidate simulation result of the current frame. Among them, the third candidate simulation result that matches the user operation received this time may refer to the third candidate simulation result whose corresponding preset user operation is the same as or closest to the user operation received this time.

[0032] According to the first aspect, the terminal device is further configured to fuse the target simulation result of the rendered frame with the first candidate simulation result of the current frame to obtain a fused result; and the terminal device is specifically configured to fuse the second candidate simulation result of the current frame with the fused result to obtain the target simulation result of the current frame. In this way, by fusing the target simulation result of the rendered frame with the first candidate simulation result of the current frame, the change between the motion state of the physical simulation object in the current frame and the motion state of the previous frame can be made smoother, allowing the user to view the motion state of the physical simulation object in two adjacent frames more smoothly.

[0033] Exemplarily, the target simulation result of the rendered frame is generated by the terminal device. The generation method of the target simulation result of the rendered frame is similar to the generation method of the target simulation result of the current frame, and will not be repeated here.

[0034] For example, the rendered frame may be a previous frame of the current frame.

[0035] According to the first aspect, or any implementation of the first aspect above, the first candidate simulation result of the current frame includes the first cloth model of the current frame, the second candidate simulation result of the current frame includes the second cloth model of the current frame, and the target simulation result of the current frame includes the target cloth model of the current frame;

[0036] The server is further configured to encode the first fabric model of the current frame to obtain an encoded code stream, and send the encoded code stream to the terminal device;

[0037] The terminal device is also used to decode the encoded code stream to obtain the reconstruction result of the first fabric model of the current frame; and fuse the reconstruction results of the second fabric model of the current frame and the first fabric model of the current frame to obtain the target fabric model of the current frame.

[0038] Since the data volume of the fabric model is relatively large, compressing the fabric model before transmission can reduce bandwidth requirements, bit rate overhead and server load.

[0039] It should be noted that the reconstruction result of the first fabric model of the current frame may be the same as or different from the first fabric model of the current frame; it depends on whether the server encodes the first fabric model of the current frame using lossy encoding or lossless encoding.

[0040] Exemplarily, the terminal device is specifically used to fuse the target simulation result of the rendered frame and the reconstruction result of the first candidate simulation result of the current frame to obtain a fusion result; and fuse the second candidate simulation result of the current frame and the fusion result to obtain the target simulation result of the current frame.

[0041] According to the first aspect, or any implementation of the first aspect above, the physical simulation data includes first animation data; the first animation data is used to describe the animation state of the character at a third time point, and the third time point is the time point when the terminal device sends the first animation data;

[0042] The server is specifically configured to predict second animation data based on the first animation data; wherein the second animation data is used to describe the animation state of the character at the first time point;

[0043] The server is further configured to perform physical simulation on the physical simulation object based on the physical simulation data and the second animation data to obtain a first candidate simulation result of the current frame; wherein the three-dimensional scene includes the physical simulation object and the character; and the character is bound to the physical simulation object;

[0044] The server is further configured to send the second animation data;

[0045] The terminal device is further configured to receive second animation data.

[0046] Since it takes a certain amount of time for the interactive data to be transmitted from the terminal device to the server (or from the server to the terminal device), the server can predict the animation data of the character at a time point after the third time point (i.e., the first time point) based on the first animation data in the physical simulation data, so as to reduce the delay in the character's action; and the first candidate simulation result of the current frame is generated based on the second animation data, thereby reducing the delay in the motion state of the physical simulation object.

[0047] Exemplarily, the first animation data includes at least one of the following:

[0048] The character's animation sequence identifier;

[0049] Animation sequences for characters;

[0050] The character's encoded animation sequence;

[0051] The playback position of the character's animation sequence at a third point in time, as well as the character's position and orientation at the third point in time.

[0052] Exemplarily, the second animation data includes the playback position of the character's animation sequence at a first time point, and the character's skeletal state, position, and orientation at the first time point.

[0053] According to the first aspect, or any implementation of the first aspect above, the physical simulation data includes second animation data, and the second animation data is used to describe the animation state of the character at the first time point;

[0054] The server is specifically configured to perform physical simulation on the physical simulation object based on the second animation data to obtain a first candidate simulation result of the current frame; wherein the three-dimensional scene includes the physical simulation object and the character; and the character is bound to the physical simulation object.

[0055] According to the first aspect, or any implementation of the first aspect above,

[0056] The terminal device is further configured to fuse the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character;

[0057] The terminal device is specifically used to render the three-dimensional scene based on the target simulation result of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

[0058] In other words, by fusing the predicted character animation data at a first point in time with the character animation data determined by the terminal device at a second point in time, the character's motion between two adjacent frames is made smoother. The terminal device also fuses the first candidate simulation result for the current frame with the second candidate simulation result for the current frame; this ensures that the fabric's motion matches the character's motion, preventing clipping.

[0059] According to the first aspect, or any implementation of the first aspect above, the terminal device is specifically used to perform weighted calculation on the playback position of the character's animation sequence at the first time point and the playback position of the character's animation sequence at the second time point based on the first weight and the second weight to obtain the target playback position of the character.

[0060] According to the first aspect, or any implementation of the first aspect above, the first weight corresponds to the playback position of the character's animation sequence at a first time point, and the second weight corresponds to the playback position of the character's animation sequence at a second time point;

[0061] The first weight is inversely proportional to the difference between the first time point and the second time point, and the second weight is directly proportional to the difference between the first time point and the second time point.

[0062] In other words, the greater the difference between the first and second time points, the more reliable the character's animation sequence's playback position at the second time point is; the smaller the difference between the first and second time points, the more reliable the character's animation sequence's playback position at the first time point is. This prevents sudden changes or teleportation in the character's movements on the client side, making the character's movements between adjacent frames more natural and smooth.

[0063] Exemplarily, the second weight is the difference between 1 and the first weight.

[0064] According to the first aspect, or any implementation of the first aspect above, the terminal device is specifically used to perform weighted calculation on the target simulation result of the rendered frame and the first candidate simulation result of the current frame based on the third weight and the fourth weight to obtain a fusion result.

[0065] According to the first aspect, or any implementation method of the first aspect above, the third weight corresponds to the target simulation result of the rendered frame, and the fourth weight corresponds to the first candidate simulation result of the current frame; the third weight is inversely proportional to the difference between the fourth time point and the fifth time point, and the fourth weight is directly proportional to the difference between the fourth time point and the fifth time point; the fourth time point is the time point when the terminal device actually receives the first candidate simulation result of the rendered frame, and the fifth time point is the time point when the terminal device fuses the second candidate simulation result of the current frame and the first candidate simulation result of the current frame.

[0066] That is to say, the greater the difference between the fourth time point and the fifth time point, the smaller the weight of the target cloth model of the rendered frame; the smaller the difference between the fourth time point and the fifth time point, the greater the weight of the target cloth model of the rendered frame; in this way, the motion state of the cloth on the end side can be prevented from jumping or teleporting, making the motion state of the cloth in two adjacent frames more natural and smooth.

[0067] It should be noted that the fourth time point may refer to the time point when the terminal device actually starts to receive the first fabric model of the rendered frame, or the time point when the terminal device actually completes receiving the first fabric model of the rendered frame (the time difference between these two situations is very small and can be ignored in some scenarios; therefore, the fourth time point may also be any time point between the time points corresponding to these two situations). Alternatively, the fourth time point may refer to the time point when the terminal device actually starts to receive the encoded code stream, or the time point when the terminal device actually completes receiving the encoded code stream, or the time point when the terminal device decodes the encoded code stream to obtain the reconstruction result of the first fabric model of the rendered frame (the time difference between these three situations is very small and can be ignored in some scenarios; therefore, the fourth time point may also be any time point between the time points in these three situations).

[0068] It should be noted that the fifth time point can be the time point when the terminal device starts to fuse the target fabric model of the rendered frame and the reconstruction result of the first fabric model of the current frame; it can also be the time point when the fusion of the target fabric model of the rendered frame and the reconstruction result of the first fabric model of the current frame is completed (the time difference between these two situations is very small and can be ignored in some scenes; therefore, the fifth time point can also be any time point between the time points corresponding to these two situations).

[0069] According to the first aspect, or any implementation of the first aspect above, the terminal device is specifically used to perform weighted calculation on the second candidate simulation result and the fusion result of the current frame based on the fifth weight and the sixth weight to obtain the target simulation result of the current frame.

[0070] According to the first aspect, or any implementation of the first aspect above, the fifth weight corresponds to the second candidate simulation result of the current frame, and the sixth weight corresponds to the fusion result;

[0071] The fifth weight is proportional to the difference between the second time point and the sixth time point, and the sixth weight is inversely proportional to the difference between the second time point and the sixth time point;

[0072] The sixth time point is the time point when the terminal device sends physical simulation data.

[0073] This allows you to combine the low-profile results from the local device (the second candidate simulation result for the current frame) with the high-profile results from the cloud (the first candidate simulation result for the current frame). If the clock cycle between the device sending and receiving data is too long, the results from the cloud will be less reliable. This reduces the impact of cloud-based fabric simulation on the final result.

[0074] Illustratively, the sixth time point may be the same as or different from the first time point, and this application does not impose any limitation on this.

[0075] It should be noted that the sixth time point can be the time point when the terminal device starts sending physical simulation data, or it can be the predicted time point when the terminal device completes sending physical simulation data (the time difference between these two situations is very small and can be ignored in some scenarios; therefore, the sixth time point can also be any time point between the time points corresponding to these two situations), and this application does not impose any restrictions on this.

[0076] According to the first aspect, or any implementation of the first aspect above, the physical simulation data also includes switching information, and the switching information includes at least one of the following: an identifier of the physical simulation object, a model of the physical simulation object, or an encoded model of the physical simulation object; the identifier of the physical simulation object indicates the model of the physical simulation object.

[0077] In this way, it is possible to switch physical simulation objects according to user operations.

[0078] According to the first aspect, or any implementation of the first aspect above,

[0079] The terminal device is further configured to send a first timestamp;

[0080] The server is further configured to verify the first timestamp;

[0081] The server is specifically configured to perform physical simulation on the physical simulation object based on the physical simulation data after the first timestamp verification succeeds, so as to obtain a first candidate simulation result of the current frame.

[0082] In this way, the timeliness of the interactive data transmitted between the terminal device and the server can be guaranteed; the action of the character animation and the motion state of the physical simulation object can be prevented from being delayed too long, thereby improving the user viewing experience.

[0083] For example, the server can calculate the difference between the current time and the first timestamp; when the difference between the current time and the first timestamp is greater than a first verification threshold, it can be determined that the verification of the first timestamp has failed, and the process can be terminated, that is, character animation prediction and physical simulation, as well as other subsequent steps, are not performed. When the difference between the current time and the first timestamp is less than or equal to the first verification threshold, it can be determined that the verification of the first timestamp has succeeded. The first timestamp can be set as required, and this application does not impose any restrictions on this.

[0084] According to the first aspect, or any implementation of the first aspect above,

[0085] The server is further configured to send a second timestamp;

[0086] The terminal device is further configured to receive a second timestamp and verify the second timestamp;

[0087] The terminal device is specifically configured to, when the second timestamp is successfully verified, execute the step of fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain the target simulation result of the current frame.

[0088] In this way, the timeliness of the interactive data transmitted between the terminal device and the server can be guaranteed; the action of the character animation and the motion state of the physical simulation object can be prevented from being delayed too long, thereby improving the user viewing experience.

[0089] Exemplarily, the terminal device can calculate the difference between the current time and the second timestamp; when the difference between the current time and the second timestamp is greater than a second verification threshold, it can be determined that the verification of the second timestamp has failed, and the second fabric model of the current frame can be used as the target fabric model of the current frame; and the playback position corresponding to the character's animation sequence at the second moment is used as the character's target playback position. When the difference between the current time and the second timestamp is less than or equal to the second verification threshold, it can be determined that the verification of the second timestamp has succeeded. The second verification threshold can be set as required, and this application does not impose any restrictions on this.

[0090] According to the first aspect, or any implementation of the first aspect above, the first candidate simulation result of the current frame includes at least one of the following: the position and orientation of the rigid body, or a cloth model.

[0091] In the second aspect, the present application provides a rendering method applied to a terminal device, the method comprising: first, receiving a first candidate simulation result of the current frame; then, fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain a target simulation result of the current frame; thereafter, rendering the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame; wherein the first candidate simulation result of the current frame is a simulation result corresponding to a first time point, and the first time point is the predicted time point when the terminal device receives the first candidate simulation result of the current frame; the second candidate simulation result of the current frame corresponds to a simulation result corresponding to a second time point, and the second time point is the time point when the terminal device actually receives the first candidate simulation result of the current frame.

[0092] According to the second aspect, the method also includes: fusing the target simulation result of the rendered frame and the first candidate simulation result of the current frame to obtain a fusion result; fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain the target simulation result of the current frame, including: fusing the second candidate simulation result of the current frame and the fusion result to obtain the target simulation result of the current frame.

[0093] According to the second aspect, or any implementation of the second aspect above, the method also includes: sending physical simulation data; wherein the physical simulation data includes first animation data, the first animation data is used to describe the animation state of the character at a third time point, and the third time point is the time point when the terminal device sends the first animation data; the first candidate simulation result of the current frame is obtained by physically simulating the physical simulation object based on the second animation data, and the second animation data is predicted based on the first animation data, and the second animation data is used to describe the animation state of the character at the first time point; the three-dimensional scene includes a physical simulation object and a character, and the character is bound to the physical simulation object.

[0094] According to the second aspect, or any implementation of the second aspect, the first animation data includes at least one of the following:

[0095] The character's animation sequence identifier;

[0096] Animation sequences for characters;

[0097] The character's encoded animation sequence;

[0098] The playback position of the character's animation sequence at a third point in time, as well as the character's position and orientation at the third point in time.

[0099] According to the second aspect, or any implementation of the second aspect above, the method also includes: sending physical simulation data; wherein the physical simulation data includes second animation data, and the second animation data is used to describe the animation state of the character at a first time point; the first candidate simulation result of the current frame is obtained by physically simulating the physical simulation object based on the second animation data; the three-dimensional scene includes the physical simulation object and the character, and the character is bound to the physical simulation object.

[0100] According to the second aspect, or any implementation of the second aspect above, the second animation data includes the playback position of the character's animation sequence at a first time point, and the character's skeletal state, position, and orientation at the first time point.

[0101] According to the second aspect, or any implementation of the second aspect above, the method further includes: fusing the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character;

[0102] Rendering the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame, including: rendering the three-dimensional scene based on the target simulation result of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

[0103] For example, the playback position of the character's animation sequence at the first time point may be generated by the terminal device itself, or may be carried in the second animation data received from the server.

[0104] According to the second aspect, or any implementation method of the second aspect above, the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the first time point are integrated to obtain the character's target playback position, including: based on the first weight and the second weight, performing a weighted calculation on the playback position of the character's animation sequence at the first time point and the playback position of the character's animation sequence at the second time point to obtain the character's target playback position.

[0105] According to the second aspect, or any implementation of the second aspect above, the first weight corresponds to the playback position of the character's animation sequence at the first time point, and the second weight corresponds to the playback position of the character's animation sequence at the second time point; the first weight is inversely proportional to the difference between the first time point and the second time point, and the second weight is directly proportional to the difference between the first time point and the second time point.

[0106] According to the second aspect, or any implementation method of the above second aspect, the target simulation result of the rendered frame and the first candidate simulation result of the current frame are fused to obtain a fusion result; including: based on the third weight and the fourth weight, the target simulation result of the rendered frame and the first candidate simulation result of the current frame are weighted calculated to obtain a fusion result.

[0107] According to the second aspect, or any implementation method of the above second aspect, the third weight corresponds to the target simulation result of the rendered frame, and the fourth weight corresponds to the first candidate simulation result of the current frame; the third weight is inversely proportional to the difference between the fourth time point and the fifth time point, and the fourth weight is directly proportional to the difference between the fourth time point and the fifth time point; the fourth time point is the time point when the terminal device actually receives the first candidate simulation result of the rendered frame, and the fifth time point is the time point when the terminal device fuses the second candidate simulation result of the current frame and the first candidate simulation result of the current frame.

[0108] According to the second aspect, or any implementation method of the above second aspect, the second candidate simulation result and the fusion result of the current frame are fused to obtain the target simulation result of the current frame; including: based on the fifth weight and the sixth weight, the second candidate simulation result and the fusion result of the current frame are weighted calculated to obtain the target simulation result of the current frame.

[0109] According to the second aspect, or any implementation method of the above second aspect, the fifth weight corresponds to the second candidate simulation result of the current frame, and the sixth weight corresponds to the fusion result; the fifth weight is proportional to the difference between the second time point and the sixth time point, and the sixth weight is inversely proportional to the difference between the second time point and the sixth time point; the sixth time point is the time point when the terminal device sends physical simulation data, and the first candidate simulation result of the current frame is obtained by physically simulating the physical simulation object based on the physical simulation data.

[0110] According to the second aspect, or any implementation of the second aspect above, the first candidate simulation result of the current frame includes the first cloth model of the current frame, the second candidate simulation result of the current frame includes the second cloth model of the current frame, and the target simulation result of the current frame includes the target cloth model of the current frame;

[0111] Receiving a first candidate simulation result of a current frame includes: receiving an encoded bitstream, the encoded bitstream including an encoded first cloth model of the current frame;

[0112] The method also includes: decoding the encoded code stream to obtain a reconstruction result of the first fabric model of the current frame; fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain a target simulation result of the current frame, including: fusing the reconstruction result of the second fabric model of the current frame and the first fabric model of the current frame to obtain the target fabric model of the current frame.

[0113] According to the second aspect, or any implementation of the second aspect above, the physical simulation data also includes switching information, and the switching information includes at least one of the following: an identifier of the physical simulation object, a model of the physical simulation object, or an encoded model of the physical simulation object; the identifier of the physical simulation object indicates the model of the physical simulation object.

[0114] According to the second aspect, or any implementation of the second aspect above, the method further includes: sending a first timestamp; wherein the first candidate simulation result of the current frame is obtained by performing physical simulation on the physical simulation object based on the physical simulation data after the first timestamp verification is successful.

[0115] According to the second aspect, or any implementation of the second aspect above, the method also includes: receiving a second timestamp; when the second timestamp is verified successfully, executing the step of fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain the target simulation result of the current frame.

[0116] According to the second aspect, or any implementation of the second aspect above, the first candidate simulation result of the current frame includes at least one of the following: the position and orientation of the rigid body, or a cloth model.

[0117] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0118] In a third aspect, the present application provides a physical simulation method applied to a server, the method comprising: first, receiving physical simulation data; then, performing physical simulation on a physical simulation object based on the physical simulation data to obtain a first candidate simulation result of the current frame; then, encoding the first candidate simulation result of the current frame to obtain an encoded code stream; thereafter, sending the encoded code stream.

[0119] According to the third aspect, the first candidate simulation result of the current frame includes a first cloth model of the current frame.

[0120] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0121] In a fourth aspect, the present application provides a device-cloud collaborative system, which includes a server and a terminal device, wherein:

[0122] The terminal device is configured to send first animation data to the server; wherein the first animation data is configured to describe an animation state of a character at a first time point, where the first time point is the time point at which the terminal device sends the first animation data to the server;

[0123] The server is configured to predict second animation data based on the first animation data, wherein the second animation data is used to describe the animation state of the character at a second time point, the second time point being the predicted time point when the terminal device receives the second animation data; and send the second animation data to the terminal device;

[0124] A terminal device is used to fuse the playback position of the character's animation sequence at a second time point with the playback position of the character's animation sequence at a third time point to obtain a target playback position of the character; and render a three-dimensional scene based on the target playback position of the character to obtain a rendered image; wherein the three-dimensional scene includes the character, and the third time point is the time point when the terminal device actually receives the second animation data.

[0125] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0126] In a fifth aspect, an embodiment of the present application provides a rendering method applied to a terminal device, the method comprising: first, sending first animation data; wherein the first animation data is used to describe the animation state of the character at a first time point, and the first time point is the time point when the terminal device sends the first animation data to the server; then, receiving second animation data; wherein the second animation data is used to describe the animation state of the character at a second time point, and the second time point is the predicted time point when the terminal device receives the second animation data; thereafter, fusing the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at a third time point to obtain the target playback position of the character; wherein the third time point is the time point when the terminal device actually receives the second animation data; then, rendering the three-dimensional scene based on the target playback position of the character to obtain a rendered image; wherein the three-dimensional scene includes the character.

[0127] According to the fifth aspect, the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point are integrated to obtain the character's target playback position, including: based on the first weight and the second weight, performing a weighted calculation on the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point to obtain the character's target playback position.

[0128] According to the fifth aspect, or any implementation of the above fifth aspect, the first weight corresponds to the playback position of the character's animation sequence at the second time point, and the second weight corresponds to the playback position of the character's animation sequence at the third time point; the first weight is inversely proportional to the difference between the second time point and the third time point, and the second weight is directly proportional to the difference between the second time point and the third time point.

[0129] According to the fifth aspect, or any implementation of the fifth aspect, the first animation data includes at least one of the following:

[0130] The character's animation sequence identifier;

[0131] Animation sequences for characters;

[0132] The character's encoded animation sequence;

[0133] The playback position of the character's animation sequence at the first point in time, as well as the character's position and orientation at the first point in time.

[0134] According to the fifth aspect, or any implementation of the fifth aspect above, the second animation data includes the playback position of the character's animation sequence at the second time point, and the character's skeletal state, position and orientation at the second time point.

[0135] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0136] In a sixth aspect, an embodiment of the present application provides a rendering device, the rendering device comprising:

[0137] A first receiving module, configured to receive a first candidate simulation result of a current frame;

[0138] A first fusion module is used to fuse the second candidate simulation result of the current frame with the first candidate simulation result of the current frame to obtain a target simulation result of the current frame;

[0139] A first rendering module, configured to render the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame;

[0140] Among them, the first candidate simulation result of the current frame is the simulation result corresponding to the first time point, and the first time point is the predicted time point when the terminal device receives the first candidate simulation result of the current frame; the second candidate simulation result of the current frame is the simulation result corresponding to the second time point, and the second time point is the time point when the terminal device actually receives the first candidate simulation result of the current frame.

[0141] Exemplarily, the first fusion module is specifically used to fuse the target simulation result of the rendered frame and the first candidate simulation result of the current frame to obtain a fusion result; and fuse the second candidate simulation result of the current frame and the fusion result to obtain the target simulation result of the current frame.

[0142] It should be understood that the rendering device of the sixth aspect can execute the method steps in the above-mentioned second aspect and any implementation manner of the second aspect, and will not be repeated here.

[0143] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0144] In a seventh aspect, the present application provides a physical simulation device, the physical simulation device comprising:

[0145] A second receiving module, configured to receive physical simulation data;

[0146] A physical simulation module, configured to perform physical simulation on a physical simulation object based on the physical simulation data to obtain a first candidate simulation result of a current frame;

[0147] an encoding module, configured to encode the first candidate simulation result of the current frame to obtain an encoded bitstream;

[0148] The first sending module is used to send the encoded code stream.

[0149] Exemplarily, the first candidate simulation result of the current frame includes a first cloth model of the current frame.

[0150] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0151] In an eighth aspect, the present application provides a rendering device, the rendering device comprising:

[0152] A second sending module, configured to send the first animation data; wherein the first animation data is used to describe the animation state of the character at a first time point, and the first time point is the time point when the terminal device sends the first animation data to the server;

[0153] A third receiving module is configured to receive second animation data, wherein the second animation data is used to describe the animation state of the character at a second time point, and the second time point is a predicted time point when the terminal device receives the second animation data;

[0154] a second fusion module, configured to fuse the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at a third time point to obtain a target playback position of the character; wherein the third time point is the time point at which the terminal device actually receives the second animation data;

[0155] The second rendering module is used to render the three-dimensional scene based on the target playback position of the character to obtain a rendered image; wherein the three-dimensional scene includes the character.

[0156] It should be understood that the rendering device of the eighth aspect can execute the method steps in the fifth aspect and any implementation manner of the fifth aspect, which will not be repeated here.

[0157] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0158] In the ninth aspect, the present application provides a terminal device comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the terminal device to execute the method in the second aspect or any possible implementation of the second aspect, or to execute the method in the fifth aspect or any possible implementation of the fifth aspect.

[0159] The ninth aspect and any implementation of the ninth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the ninth aspect and any implementation of the ninth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0160] In the tenth aspect, the present application provides a server comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the server to execute the method in the third aspect or any possible implementation of the third aspect.

[0161] The tenth aspect and any implementation of the tenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the tenth aspect and any implementation of the tenth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0162] In the eleventh aspect, the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the rendering method in the second aspect or any possible implementation of the second aspect is executed; or, the method in the fifth aspect or any possible implementation of the fifth aspect is executed.

[0163] The eleventh aspect and any implementation of the eleventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eleventh aspect and any implementation of the eleventh aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0164] In the twelfth aspect, the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the method in the third aspect or any possible implementation of the third aspect is executed.

[0165] The twelfth aspect and any implementation of the twelfth aspect respectively correspond to the first aspect and any implementation of the first aspect. The technical effects corresponding to the twelfth aspect and any implementation of the twelfth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0166] In the thirteenth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the method in the second aspect or any possible implementation of the second aspect; or, enables the method in the fifth aspect or any possible implementation of the fifth aspect to be executed.

[0167] The thirteenth aspect and any implementation of the thirteenth aspect respectively correspond to the first aspect and any implementation of the first aspect. The technical effects corresponding to the thirteenth aspect and any implementation of the thirteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0168] In the fourteenth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, it enables the computer or the processor to execute the method in the third aspect or any possible implementation of the third aspect.

[0169] The fourteenth aspect and any implementation of the fourteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourteenth aspect and any implementation of the fourteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0170] In the fifteenth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the method in the second aspect or any possible implementation of the second aspect, or executes the method in the fifth aspect or any possible implementation of the fifth aspect.

[0171] The fifteenth aspect and any implementation of the fifteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifteenth aspect and any implementation of the fifteenth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0172] In the sixteenth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the method in the third aspect or any possible implementation of the third aspect.

[0173] The sixteenth aspect and any implementation of the sixteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixteenth aspect and any implementation of the sixteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0174] In a seventeenth aspect, the present application provides a device for storing an encoded code stream, the device comprising: a receiver and at least one storage medium, the receiver being used to receive the encoded code stream; the at least one storage medium being used to store the encoded code stream; the encoded code stream is generated according to the first aspect and any one of the implementations of the first aspect, or generated according to the second aspect and any one of the implementations of the second aspect, or generated according to the third aspect and any one of the implementations of the third aspect.

[0175] The seventeenth aspect and any implementation of the seventeenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventeenth aspect and any implementation of the seventeenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0176] In an eighteenth aspect, the present application provides a device for transmitting an encoded code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the encoded code stream, the encoded code stream being generated by the first aspect and any one of the implementation methods of the first aspect, or generated according to the second aspect and any one of the implementation methods of the second aspect, or generated according to the third aspect and any one of the implementation methods of the third aspect; the transmitter being used to obtain the encoded code stream from the storage medium and send the encoded code stream to the end-side device through the transmission medium.

[0177] The eighteenth aspect and any implementation of the eighteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighteenth aspect and any implementation of the eighteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0178] In a nineteenth aspect, the present application provides a system for distributing encoded code streams, the system comprising: at least one storage medium for storing at least one encoded code stream, the at least one encoded code stream being generated according to the first aspect and any one of the implementation methods of the first aspect, or generated according to the second aspect and any one of the implementation methods of the second aspect, or generated according to the third aspect and any one of the implementation methods of the third aspect, a streaming media device for obtaining a target encoded code stream from the at least one storage medium and sending the target encoded code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.

[0179] The nineteenth aspect and any implementation of the nineteenth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the nineteenth aspect and any implementation of the nineteenth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0180] In the twentieth aspect, the present application provides an encoded code stream, which is generated according to the first aspect and any one of the implementation methods of the first aspect, or generated according to the second aspect and any one of the implementation methods of the second aspect, or generated according to the third aspect and any one of the implementation methods of the third aspect.

[0181] The twentieth aspect and any implementation of the twentieth aspect respectively correspond to the first aspect and any implementation of the first aspect. The technical effects corresponding to the twentieth aspect and any implementation of the twentieth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, and will not be repeated here.

[0182] In a twenty-first aspect, the present application provides an encoded code stream, which includes a first candidate simulation result encoded for a current frame.

[0183] Exemplarily, the first candidate simulation result may be a first fabric model, the encoded first fabric model may be obtained by the server encoding the first fabric model, and the first fabric model may be obtained by the server performing physical simulation on the fabric based on physical simulation data uploaded by the terminal device.

[0184] According to the twenty-first aspect, the encoded code stream may further include a second timestamp.

[0185] The second timestamp may refer to a timestamp of the server, which is used to indicate the time when the server sends the first cloth model (or the time when the server sends the encoded code stream).

[0186] According to the twenty-first aspect, or any one of the implementations of the twenty-first aspect above, the encoded code stream further includes encoded second animation data.

[0187] The second animation data is obtained by the server performing animation prediction based on the first animation data.

[0188] The first animation data includes at least one of the following:

[0189] The character's animation sequence identifier;

[0190] Animation sequences for characters;

[0191] The character's encoded animation sequence;

[0192] The playback position of the character's animation sequence at a third point in time, as well as the character's position and orientation at the third point in time.

[0193] The second animation data includes the playback position of the character's animation sequence at the first time point, and the character's skeletal state, position, and orientation at the first time point. The twenty-first aspect and any one of the implementations of the twenty-first aspect correspond to the first aspect and any one of the implementations of the first aspect, respectively. The technical effects corresponding to the twenty-first aspect and any one of the implementations of the twenty-first aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any one of the implementations of the first aspect, and will not be repeated here.

[0194] In a twenty-second aspect, the present application provides an encoded code stream comprising physical simulation data.

[0195] According to the twenty-second aspect, the encoded code stream may further include a first timestamp.

[0196] The twenty-second aspect and any implementation of the twenty-second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the twenty-second aspect and any implementation of the twenty-second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0197] In a twenty-third aspect, the present application provides a computer storage medium comprising an encoded code stream, which is the encoded code stream of the twenty-first aspect or the encoded code stream of the twenty-second aspect.

[0198] The twenty-third aspect and any implementation of the twenty-third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the twenty-third aspect and any implementation of the twenty-third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0199] FIG1A is a schematic diagram illustrating an exemplary end-cloud collaborative application scenario;

[0200] FIG1B is a schematic diagram illustrating an exemplary three-dimensional scene;

[0201] FIG1C is a schematic diagram illustrating an exemplary end-cloud collaboration framework;

[0202] FIG2A is a schematic diagram illustrating the structure of an exemplary end-cloud collaborative system;

[0203] FIG2B is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 200;

[0204] FIG3A is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 300;

[0205] FIG3B is a schematic diagram showing a timing relationship of exemplary time points;

[0206] FIG3C is a schematic diagram illustrating an exemplary animation prediction process;

[0207] FIG3D is a schematic diagram illustrating an exemplary interpolation function;

[0208] FIG4A is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 400;

[0209] FIG4B is a schematic diagram illustrating an exemplary physical simulation process;

[0210] FIG5 is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 500;

[0211] FIG6 is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 600;

[0212] FIG7 is a schematic diagram illustrating an exemplary end-cloud collaborative application scenario;

[0213] FIG8 is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process;

[0214] FIG9 is a schematic diagram of an exemplary rendering device 900;

[0215] FIG10 is a schematic diagram of a physical simulation device 1000 shown as an example;

[0216] FIG11 is a schematic diagram of an exemplary rendering device 1100;

[0217] FIG12 is a schematic structural diagram of an exemplary device;

[0218] FIG13 is a schematic structural diagram of an exemplary electronic device. DETAILED DESCRIPTION

[0219] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0220] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0221] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0222] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0223] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0224] Figure 1A is a schematic diagram illustrating an exemplary end-cloud collaborative application scenario. The network in Figure 1A includes, but is not limited to, wired networks and wireless networks (such as cellular networks and WIFI (Wireless Fidelity) networks, etc.), which are not limited in this application.

[0225] 1A , exemplarily, after receiving a user operation, a terminal device (also referred to as an end or end-side device) can send physical simulation data to a server (also referred to as a cloud or cloud server) through a network; then, the server can perform physical simulation on a physical simulation object based on the physical simulation data to obtain a physical simulation result; subsequently, the server can send the physical simulation result to the terminal device through the network; thereafter, the terminal device can perform rendering based on the physical simulation result to obtain and display a rendered image.

[0226] For example, in the scenario of using a mobile phone for cloud gaming in Figure 1A, the mobile phone can send the physical simulation data of the game character's clothes and props to the server through the network, and the server can send the physical simulation results of the game character's clothes and props to the mobile phone through the network; in this way, the mobile phone renders based on the physical simulation results of the game character's clothes and props, and can obtain and display the game screen (that is, the rendered game screen).

[0227] For example, in the scenario of using AR / VR glasses for an AR / VR conference in FIG1A , the AR / VR glasses can send the physical simulation data of the clothes of the participating characters to the server via the network, and the server can send the physical simulation results of the clothes of the participating characters to the AR / VR glasses via the network; in this way, the AR / VR glasses perform rendering based on the physical simulation results of the clothes of the participating characters, and can obtain the AR / VR conference screen (that is, the rendered AR / VR conference screen) and display it.

[0228] For example, in the scene of watching a virtual concert using a personal computer in Figure 1A, the personal computer can send the physical simulation data of the virtual singer's clothes to the server through the network, and the server can send the physical simulation results of the virtual singer's clothes to the personal computer through the network; in this way, the personal computer performs rendering based on the physical simulation results of the virtual singer's clothes, and can obtain and display the concert picture (that is, the rendered concert picture).

[0229] For example, in the scenario of using a tablet computer to watch a live broadcast of a digital human in FIG1A , the tablet computer can send the physical simulation data of the digital human's clothes to the server via the network, and the server can send the physical simulation results of the digital human's clothes to the tablet computer via the network; in this way, the tablet computer performs rendering based on the physical simulation results of the digital human's clothes, and can obtain and display the live broadcast picture of the digital human (that is, the rendered live broadcast picture).

[0230] It should be understood that the present application can also be applied in other scenarios, and the present application does not limit this.

[0231] Exemplarily, the characters involved in this application may include game characters, meeting characters, virtual singers, and virtual objects such as digital humans; virtual objects correspond to entities (such as people, animals, etc.) that can move autonomously and whose movements are affected by external forces. The physical simulation objects involved in this application are bound to the characters, and the motion state of the physical simulation objects can change as the motion state of the characters changes. For example, in a cloud game scene, physical simulation objects include game character clothes, game character props, etc. For another example, in an AR / VR conference scene, physical simulation objects include clothes for meeting characters. For another example, in a virtual concert scene, physical simulation objects include clothes for virtual singers. For another example, in a digital human live broadcast scene, virtual simulation objects include digital human clothes.

[0232] For example, physical simulation (also known as physics simulation) is a technology that uses computers to simulate real-world physical phenomena in order to create more realistic interactions with the world in animations and games. Physical simulation can include rigid body simulation (which can be used to simulate the rotation and translation of rigid objects), elastic body simulation (which can be used to simulate the movement and deformation of elastic objects), cloth simulation, and fluid simulation.

[0233] For example, rendering may refer to a process of displaying a three-dimensional model of an object (eg, a character, a character's clothes, a character's props, etc.) in a three-dimensional scene onto a two-dimensional plane.

[0234] FIG1B is a schematic diagram illustrating an exemplary three-dimensional scene.

[0235] The three-dimensional scene shown in FIG1B (1) and (2) is a virtual fitting room, which includes a character and the character's clothes (ie, physical simulation objects).

[0236] For example, a user can use a virtual fitting room to try on clothes. For example, the clothes currently being tried on by the character are clothes 1, as shown in FIG1B (1); when the user wants to try on clothes 2, he can click on the clothes 2 option, and the terminal device can respond to the user's operation and send the physical simulation data of clothes 2 to the server. The server can perform physical simulation based on the physical simulation data of clothes 2, and generate a physical simulation result, that is, the fabric model of clothes 2; and send the fabric model of clothes 2 to the terminal device, which renders it based on the fabric model of clothes 2 to obtain a rendered picture; and can display the rendered picture, that is, the effect of the character trying on clothes 2, as shown in FIG1B (2).

[0237] Figure 1C is a schematic diagram of an exemplary end-cloud collaboration framework. The end-cloud collaboration system shown in Figure 1C may include a central server, an edge server, and a terminal device.

[0238] Exemplarily, the central server in Figure 1C is a type of server in Figure 1A. The central server can be a single server or a server cluster, and this application does not limit this. The specific implementation form of the server in this application can be a cloud server, a physical (independent) server, a station cluster server, etc., and this application does not limit this. When applied to different scenarios, the server in this application can also be called a game server, a video server, a live broadcast server, a VR / AR conference server, a virtual server, etc. according to the scenario applied, and this application does not limit this.

[0239] Referring to Figure 1C, by way of example, the terminal devices in Figure 1C include but are not limited to: personal computers (PCs), computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices (such as AR helmets, VR glasses), set-top boxes, game consoles, etc., and this application does not impose any restrictions on this.

[0240] 1C , illustratively, physical simulation data can be sent to a central server (such as a game server) via an edge server by a terminal device such as a mobile phone, a personal computer (PC), VR glasses, an AR helmet, a tablet computer, or other terminal devices. Then, the central server can perform physical simulation on the physical simulation object based on the physical simulation data to obtain a physical simulation result. Afterwards, the central server distributes the physical simulation result to each terminal device via the edge server. Subsequently, the terminal device can render the three-dimensional scene based on the physical simulation result to obtain a rendered image and display it.

[0241] FIG2A is a schematic diagram illustrating the structure of an exemplary end-cloud collaboration system. In FIG2A , the end-cloud collaboration system may include a server 210 and a terminal device 220 .

[0242] 2A , illustratively, the server 210 may include a physical simulation module 211, an encoding module 212, and a first communication module 213. It should be noted that the encoding module 212 is an optional module.

[0243] It should be understood that the server 210 in FIG. 2A is merely an example of the present application, and the server 210 of the present application may include more or fewer modules / units than those shown in FIG. 2A , and the present application does not impose any limitation on this.

[0244] 2A , illustratively, terminal device 220 may include a second communication module 221, a decoding module 222, a fusion module 223, a rendering module 224, a display module 225, and a physical simulation data generation module 226. It should be noted that decoding module 222 is an optional module.

[0245] It should be understood that the terminal device 220 in FIG. 2A is merely an example of the present application, and the terminal device 220 of the present application may include more or fewer modules / units than those shown in FIG. 2A , and the present application does not impose any limitation on this.

[0246] Exemplarily, the physical simulation module 211 can be implemented by a central processing unit (CPU) or a graphics processing unit (GPU); it should be understood that the physical simulation module 211 can also be implemented by other general-purpose processors or proprietary chips, and this application does not limit this.

[0247] Exemplarily, the encoding module 212 can be implemented by a CPU, or can be implemented by both a CPU and a GPU; it should be understood that the encoding module 212 can also be implemented by other general-purpose processors or proprietary chips, and this application does not limit this.

[0248] Exemplarily, the decoding module 222 can be implemented by a CPU, or by both a CPU and a GPU; it should be understood that the decoding module 222 can also be implemented by other general-purpose processors or proprietary chips, and this application does not impose any restrictions on this.

[0249] Exemplarily, the physical simulation data generation module 226 may be implemented by a CPU; it should be understood that the physical simulation data generation module 226 may also be implemented by other general-purpose processors or proprietary chips, and this application does not impose any limitation on this.

[0250] Exemplarily, the fusion module 223 may be implemented by a CPU; it should be understood that the fusion module 223 may also be implemented by other general-purpose processors or proprietary chips, and this application does not impose any limitation on this.

[0251] Exemplarily, the rendering module 224 may be implemented by a GPU, for example, the rendering module 224 may be a rendering engine in the GPU; it should be understood that the rendering module 224 may also be implemented by other general-purpose processors or proprietary chips, and this application does not impose any limitation on this.

[0252] The following uses a cloud gaming scenario as an example and describes the end-cloud collaborative rendering process in conjunction with Figure 2A.

[0253] FIG2B is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 200 .

[0254] S201: The terminal device sends physical simulation data to the server.

[0255] For example, after the terminal device 220 receives the user operation, the physical simulation data generation module 226 may generate physical simulation data and output the physical simulation data to the second communication module 221. Subsequently, the second communication module 221 may send the physical simulation data to the first communication module 213.

[0256] For example, the user operation may include at least one of adjusting the character's position, adjusting the character's orientation, or adjusting the character's movements (e.g., walking, running, or jumping, etc.; this may also be understood as adjusting the character's animation sequence). In other words, when the user performs any one or more of the above user operations, the terminal device may be triggered to send physical simulation data to the server.

[0257] For example, in a cloud gaming scenario, physical simulation objects can be game character clothes (also called fabrics), game character props (also called rigid bodies), etc.

[0258] Exemplarily, physical simulation data may include data required for physical simulation (or in other words, physical simulation data refers to data that affects the motion state of a physical simulation object). Exemplarily, physical simulation data may include, but is not limited to: physical simulation parameters, the data processed by physical simulation itself, etc., and this application does not impose any restrictions on this. Among them, physical simulation parameters may include at least one of the following: external force parameters (such as wind parameters, the position and orientation of external collision bodies, etc.), character animation data (which can be used to describe the character's animation state, such as the character's (adjusted) position and orientation, the playback position corresponding to the character's animation sequence, etc.) and physical simulation object state information (for example, the position and orientation of a rigid body (adjusted) (which can be understood as the position and orientation of the center of the rigid body)), etc., and this application does not impose any restrictions on this. The data processed by physical simulation itself may include cloth models, rigid body models, etc., and this application does not impose any restrictions on this.

[0259] For example, in a cloud gaming scenario, physical simulation data includes wind parameters, descriptions of objects that may collide with the game character's clothes, and the character's animation data.

[0260] For example, in a cloud gaming scenario, physical simulation data includes wind parameters, descriptions of objects that may collide with the game character's clothes, character animation data, and the fabric model corresponding to the game character's clothes.

[0261] For example, in a cloud gaming scenario, physical simulation data includes the animation data of the game character, and the position and orientation of the game character's props.

[0262] For example, in a cloud gaming scenario, physical simulation data includes the animation data of the game character, the position and orientation of the game character's props, and the rigid body model corresponding to the game character's props.

[0263] S202: The server performs physical simulation on the physical simulation object based on the physical simulation data to obtain a first candidate simulation result of the current frame.

[0264] For example, after receiving the physical simulation data, the first communication module 213 may output the physical simulation data to the physical simulation module 211. Then, the physical simulation module 211 may perform physical simulation on the physical simulation object based on the physical simulation data to obtain a physical simulation result of the current frame.

[0265] For the convenience of subsequent description, the physical simulation result of the current frame generated by the server may be referred to as the first candidate simulation result of the current frame. For example, the first candidate simulation result of the current frame is the simulation result corresponding to the first time point, which may be the predicted time point when the terminal device receives the first candidate simulation result of the current frame.

[0266] For example, if the physical simulation object is cloth, the first candidate simulation result for the current frame can be the first cloth model for the current frame, i.e., the cloth model corresponding to the first time point (wherein, the cloth model corresponding to the first time point can be used to describe the motion state of the cloth at the first time point). For another example, if the physical simulation object is a rigid body, the first candidate simulation result for the current frame can be the position and orientation of a prop in the current frame, i.e., the position and orientation of the prop at the first time point.

[0267] Exemplarily, the physical simulation module 211 may include a physical simulation engine, and physical simulation data may be input into the physical simulation engine. The physical simulation engine may perform physical simulation on the physical simulation object based on the physical simulation data to obtain a first candidate simulation result of the current frame.

[0268] Since it takes a certain amount of time for interactive data to be transmitted from the terminal device to the server (or from the server to the terminal device), we can first predict the time point (i.e., the first time point) when the terminal device receives the first candidate simulation result of the current frame based on the bandwidth and the data volume of the first candidate simulation result of the current frame; then, generate the physical simulation result corresponding to the first time point as the first candidate simulation result of the current frame; in this way, the delay in the motion state of the physical simulation object can be reduced, thereby reducing the probability of screen freezes.

[0269] S203: The server sends the first candidate simulation result of the current frame to the terminal device.

[0270] Exemplarily, it can be determined whether to encode the first candidate simulation result of the current frame based on the data amount of the first candidate simulation result of the current frame or the type of the physical simulation object. For example, when the data amount of the first candidate simulation result of the current frame is greater than the data amount threshold, it can be determined that the first candidate simulation result of the current frame is encoded; when the data amount of the first candidate simulation result of the current frame is less than or equal to the data amount threshold, it can be determined that the first candidate simulation result of the current frame is not encoded. For another example, when the physical simulation object is cloth, it can be determined that the first candidate simulation result of the current frame is encoded; when the physical simulation object is a rigid body, it can be determined that the first candidate simulation result of the current frame is not encoded; and so on, and this application does not impose any restrictions on this. Among them, the data amount threshold can be set as required, and this application does not impose any restrictions on this.

[0271] For example, in one possible approach, when the first candidate simulation result of the current frame is the first cloth model of the current frame, the physical simulation module 211 may output the first cloth model of the current frame to the encoding module 212, which encodes the first cloth model of the current frame to obtain an encoded code stream (the encoded code stream includes the encoded first cloth model of the current frame); thereafter, the encoding module 212 may output the encoded code stream to the first communication module 213. Subsequently, the first communication module 213 may send the encoded code stream to the second communication module 221.

[0272] For example, in one possible approach, when the first candidate simulation result of the current frame is the position and orientation of the rigid body of the current frame, the physics simulation module 211 may output the position and orientation of the rigid body of the current frame to the first communication module 213. Subsequently, the first communication module 213 may send the position and orientation of the rigid body of the current frame to the second communication module 221.

[0273] S204: The terminal device fuses the second candidate simulation result of the current frame with the first candidate simulation result of the current frame to obtain a target simulation result of the current frame.

[0274] For example, in one possible approach, the second communication module 221 receives the encoded code stream. Thereafter, the second communication module 221 may output the encoded code stream to the decoding module 222. Subsequently, the decoding module 222 may decode the encoded code stream to obtain a reconstruction result of the first candidate simulation result of the current frame and output the reconstruction result of the first candidate simulation result of the current frame to the fusion module 223.

[0275] For example, in one possible manner, the second communication module 221 receives the first candidate simulation result of the current frame. Thereafter, the second communication module 221 may output the first candidate simulation result of the current frame to the fusion module 223 .

[0276] Afterwards, the fusion module 223 may fuse the second candidate simulation result of the current frame with the reconstruction result of the first candidate simulation result of the current frame (or the first candidate simulation result of the current frame) to obtain the target simulation result of the current frame.

[0277] The second candidate simulation result of the current frame is a simulation result corresponding to the second time point, and the second time point may be a time point when the terminal device actually receives the second candidate simulation result of the current frame.

[0278] Since the first time point and the second time point may deviate, the second time point is closer to the real time point when the terminal device plays the rendered image of the current frame relative to the first time point; therefore, fusing the second candidate simulation result of the current frame with the first candidate simulation result of the current frame can make the motion state of the physical simulation object closer to the actual motion state of the current frame corresponding to the real time point. In addition, fusing the second candidate simulation result of the current frame with the first candidate simulation result of the current frame can be understood as compromising the motion state of the physical simulation object at the first time point with the motion state of the physical simulation object at the second time point; when the first time point is after the second time point, the first time point is far away from the real time point when the terminal device plays the previous frame of the rendered image; by compromising the motion state of the physical simulation object at the first time point with the motion state of the physical simulation object at the second time point, the change between the motion state of the physical simulation object in the current frame and the motion state of the previous frame can be made smoother, so that the user can see the motion state of the physical simulation object in the two adjacent frames more smoothly. When the first time point is before the second time point, this will cause a delay in the motion state of the physical simulation object. By compromising the motion state of the physical simulation object at the first time point with the motion state of the physical simulation object at the second time point, the delay in the motion state of the physical simulation object can be reduced, thereby reducing the probability of screen freezes.

[0279] It should be noted that the first time point may refer to the predicted time point when the terminal device starts to receive the first candidate simulation result of the current frame, or it may refer to the predicted time point when the terminal device completes receiving the first candidate simulation result of the current frame (the time difference between these two situations is very small and can be ignored in some scenarios. Therefore, the first time point may also be any time point between the time points corresponding to these two situations). Alternatively, the first time point may refer to the predicted time point when the terminal device starts to receive the encoded code stream, or it may refer to the predicted time point when the terminal device completes receiving the encoded code stream, or it may be the predicted time point when the terminal device decodes the encoded code stream to obtain the reconstructed result of the first candidate simulation result of the current frame (the time difference between these three situations is very small and can be ignored in some scenarios. Therefore, the first time point may also be any time point between any two time points corresponding to these three situations).

[0280] It should be noted that the second time point may refer to the time point when the terminal device actually starts to receive the first candidate simulation result of the current frame, or it may refer to the time point when the terminal device actually completes receiving the first candidate simulation result of the current frame (the time difference between these two situations is very small and can be ignored in some scenarios; therefore, the second time point may also be any time point between the time points corresponding to these two situations). Alternatively, the second time point may refer to the time point when the terminal device actually starts to receive the encoded code stream, or it may refer to the time point when the terminal device actually completes receiving the encoded code stream, or it may be the time point when the terminal device decodes the encoded code stream to obtain the reconstructed result of the first candidate simulation result of the current frame (the time difference between the three situations is very small and can be ignored in some scenarios; therefore, the second time point may also be any time point between any two time points corresponding to these three situations).

[0281] In one possible way, the second candidate simulation result of the current frame is obtained by the terminal device performing physical simulation based on the physical simulation data. It should be noted that the complexity of the physical simulation (which can be called "simple physical simulation") performed by the terminal device to obtain the second candidate simulation result of the current frame is lower than the complexity of the physical simulation (which can be called "complex physical simulation") performed by the server to obtain the first candidate simulation result of the current frame; in other words, the computing power required for the physical simulation performed by the terminal device to obtain the second candidate simulation result of the current frame is lower than the computing power required for the physical simulation performed by the server to obtain the first candidate simulation result of the current frame. Correspondingly, the accuracy of the second candidate simulation result of the current frame is lower than the accuracy of the first candidate simulation result of the current frame.

[0282] In one possible way, the second candidate simulation result of the current frame is pre-baked and generated by the terminal device, or pre-baked and generated by the server (or third-party device) and sent to the terminal device. Among them, the terminal device (or server or third-party device) can perform N (N is a positive integer) physical simulations in advance (the physical simulation in the pre-baking process has the same complexity as the physical simulation performed by the server in S302, and can also be called "complex physical simulation") to obtain N third candidate simulation results; these N third candidate simulation results correspond one-to-one to N preset user operations. Then, from the N third candidate simulation results, the third candidate simulation result that matches the user operation received this time is selected as the second candidate simulation result of the current frame. Among them, the third candidate simulation result that matches the user operation received this time may refer to the third candidate simulation result whose corresponding preset user operation is the same as or closest to the user operation received this time.

[0283] S205 , rendering the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame.

[0284] Exemplarily, the rendering module 224 may render the physical simulation object based on the physical simulation result to obtain a rendered image and output the rendered image to the display module 225 , which then displays the rendered image.

[0285] For example, the rendering module 224 may include a rendering engine. For example, vertex data and material data of the character model, as well as the target simulation result of the current frame, may be input into the rendering engine. The rendering engine then activates a rendering pipeline to process the vertex data and material data of the character model, as well as the target simulation result of the current frame, to obtain a rendered image of the current frame.

[0286] Exemplarily, a three-dimensional scene may include a character and a physical simulation object bound to the character; it should be understood that a three-dimensional scene may also include other objects (for example, a character whose position, orientation, and action have not been adjusted by the user, other physical simulation objects, etc.), and this application does not limit this.

[0287] Exemplarily, the animation data of the character in the physical simulation data may be the animation data of the character at a third time point (hereinafter referred to as the first animation data, which may be used to describe the animation state of the character at the third time point). Exemplarily, the third time point may be the time point when the terminal device sends the first animation data, and the third time point is before the first time point and the second time point. Since it takes a certain amount of time for the interactive data to be transmitted from the terminal device to the server (or from the server to the terminal device), the server may predict the animation data of the character at a time point after the third time point (i.e., the first time point) based on the first animation data in the physical simulation data, so as to reduce the delay in the character's action; and the first candidate simulation result of the current frame is generated based on the second animation data, thereby reducing the delay in the motion state of the physical simulation object.

[0288] Furthermore, the server can fuse the predicted character animation data at the first time point with the character animation data determined by the terminal device at the second time point to achieve smoother character motion between adjacent frames. The terminal device also fuses the first candidate simulation result for the current frame with the second candidate simulation result for the current frame. This ensures that the fabric's motion matches the character's motion, preventing clipping.

[0289] Figure 3A is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 300. In rendering process 300, the physical simulation object is cloth, and the physical simulation is cloth simulation. Furthermore, user operations received by the terminal device may include any one or more of adjusting the character's position, adjusting the character's orientation, and adjusting the character's movements (e.g., walking, running, or jumping).

[0290] S301, the terminal device sends physical simulation data to the server; wherein the physical simulation data includes: external force parameters and first animation data.

[0291] Exemplarily, the first animation data is used to describe the animation state of the character at the third time point.

[0292] Exemplarily, when the user operation includes adjusting a character's motion, the first animation data includes at least one of: an animation sequence identifier for the character, an animation sequence for the character, or an encoded animation sequence for the character. In this case, the animation sequence identifier for the character is an identifier of the adjusted animation sequence for the character, and the animation sequence for the character is the adjusted animation sequence for the character.

[0293] Exemplarily, when the user operation does not include adjusting the action of the character, in one possible manner, the first animation data includes the playback position of the character's animation sequence at the third time point, and the position and orientation of the character at the third time point.

[0294] For example, when the user operation does not include adjusting the character's action, in one possible embodiment, the first data includes: the playback position of the character's animation sequence at the third time point, as well as the position and orientation of the character at the third time point, and at least one of the character's animation sequence identifier, the character's animation sequence, or the character's encoded animation sequence. In this case, the character's animation sequence identifier is the identifier of the unadjusted character's animation sequence, and the character's animation sequence is the unadjusted character's animation sequence.

[0295] Exemplarily, the animation sequence identifier can be used to uniquely identify an animation sequence. For example, a running animation sequence corresponds to animation sequence identifier 1, a jumping animation sequence corresponds to animation sequence identifier 2, a walking animation sequence corresponds to animation sequence identifier 3, and so on.

[0296] Exemplarily, the playback position of the character's animation sequence at the third time point may refer to the frame identifier of the animation frame played by the character's animation sequence at the third time point, or the timestamp of the animation frame played corresponding to the character's animation sequence at the third time point.

[0297] Exemplarily, the position and orientation of the character at the third time point may be understood as the position and orientation of the character in the animation frame played corresponding to the animation sequence of the character at the third time point.

[0298] It should be noted that the third time point can be the time point when the terminal device starts sending the first animation data, or it can be the predicted time point when the terminal device completes sending the first animation data (the time difference between these two situations is very small and can be ignored in some scenarios; therefore, the third time point can also be any time point between the time points corresponding to these two situations), and this application does not impose any restrictions on this.

[0299] S302: The server predicts second animation data based on the first animation data in the physical simulation data.

[0300] Exemplarily, the second animation data is used to describe the animation state of the character at a first time point.

[0301] Exemplarily, the second animation data includes the playback position of the character's animation sequence at a first time point, and the character's skeletal state, position, and orientation at the first time point.

[0302] FIG3B is a schematic diagram showing exemplary timing relationships of time points.

[0303] 3B , for example, one rectangle represents one frame, and the time taken for one frame can be expressed as t c Indicates, for example, t c =30ms.

[0304] 3B , exemplarily, the terminal device sends the first animation data to the server at a certain time point (i.e., the third time point) in the i-n1-n2 frame. After the server completes the prediction of the second animation data (i.e., the animation data of the i-th frame) in the i-n1 frame, it can send the second animation data to the terminal device. The terminal device receives the second animation data at a certain time point (i.e., the second time point) in the i-th frame. The i-th frame can be regarded as the current frame. i, n1, and n2 are all positive integers. That is, the third time point is before the first time point and the second time point.

[0305] Continuing to refer to FIG3B , exemplarily, the second animation data predicted by the server is the animation data of the first time point of the i-th frame. The first time point may be before or after the second time point, and this application does not impose any limitation on this.

[0306] It should be noted that FIG3B shows the temporal relationship among the first time point, the second time point, and the third time point, and does not represent the actual interval among the first time point, the second time point, and the third time point.

[0307] Figure 3C is a schematic diagram illustrating an exemplary animation prediction process, wherein the first animation data includes the playback position of the character's animation sequence at a third time point, the position and orientation of the character at the third time point, and the character's animation sequence identifier.

[0308] 3C , illustratively, S302 may refer to the following steps S3021 to S3025:

[0309] S3021: The server determines whether the animation sequence needs to be switched based on the animation sequence identifier of the character in the first animation data.

[0310] For example, the server may determine whether the animation sequence identifier of the character received this time is the same as the animation sequence identifier of the character received last time. If the animation sequence identifier of the character received this time is different from the animation sequence identifier of the character received last time, it may be determined that the animation sequence needs to be switched, and S3022 may be executed. If the animation sequence identifier of the character received this time is the same as the animation sequence identifier of the character received last time, it may be determined that the animation sequence does not need to be switched, and S3024 may be executed.

[0311] For example, if the character's animation sequence identifier received last time is identifier 1 (corresponding to a running animation sequence), and the color animation sequence identifier received this time is identifier 2 (corresponding to a jumping animation sequence), it can be determined that the animation sequence needs to be switched.

[0312] For example, if the character's animation sequence identifier received last time is identifier 2 (corresponding to a jumping animation sequence), and the color animation sequence identifier received this time is identifier 2 (corresponding to a jumping animation sequence), it can be determined that the animation sequence does not need to be switched.

[0313] S3022: The server switches the original animation sequence to a target animation sequence; wherein the target animation sequence corresponds to the animation sequence identifier of the character in the first animation data.

[0314] For example, the original animation sequence may refer to the animation sequence corresponding to the animation sequence identifier of the character received last time.

[0315] Exemplarily, the target animation sequence may refer to the animation sequence corresponding to the animation sequence identifier of the character received this time.

[0316] For example, the animation sequence of the character received last time is identified as identification 1, that is, the original animation sequence is a running animation sequence, and the animation sequence of the character received this time is identified as identification 2, that is, the target animation sequence is a jumping animation sequence; that is, the server switches the running animation sequence to a jumping animation sequence.

[0317] S3023: The server predicts the character's skeletal state at the first time point and the playback position of the character's animation sequence at the first time point based on the target animation sequence.

[0318] For example, the target animation sequence, the current time point, and the first time point can be input into the animation algorithm, and the animation algorithm predicts the character's skeletal state at the first time point and the playback position of the character's animation sequence at the first time point. The skeletal state includes the position and orientation of the skeletal nodes.

[0319] Exemplarily, the target animation sequence may include multiple key frames, each key frame including the skeletal state of the character. Exemplarily, the process of the animation algorithm predicting the skeletal state of the character at the first time point may be as follows: the target animation sequence may be interpolated first, that is, one or more animation frames may be inserted between two adjacent key frames. Thereafter, from the interpolated target animation sequence, the playback position of the character's animation sequence at the first time point is predicted based on the difference between the current time point and the first time point. Thereafter, from the interpolated target animation sequence, an animation frame corresponding to the playback position of the character's animation sequence at the first time point is searched, and the animation frame includes the skeletal state of the character at the first time point.

[0320] S3024, the server predicts the character's skeletal state at the first time point and the playback position of the character's animation sequence at the first time point based on the original animation sequence and the playback position of the character's animation sequence at the third time point.

[0321] Exemplarily, the original animation sequence, the playback position of the character's animation sequence at the third time point, the third time point and the first time point can be input into the animation algorithm, and the animation algorithm predicts the character's skeletal state at the first time point and the playback position of the character's animation sequence at the first time point.

[0322] Specifically, the playback position p of the character's animation sequence at the first time point can be predicted according to the following formula (1):

[0323] In formula (1), p0 represents the playback position of the character's animation sequence at the third time point; The predicted duration per frame on the client side is usually the average duration of the first few frames on the client side. α represents the coefficient (which can be set to a value between 1.5 and 2). v represents the animation playback rate. T1 represents the first time point, and T3 represents the third time point.

[0324] Afterwards, the animation frame corresponding to the playback position p of the character's animation sequence at the first time point can be searched from the original animation sequence after the interpolation, and the animation frame includes the skeletal state of the character at the first time point.

[0325] S3025: The server predicts the position and orientation of the character at the first time point based on the position and orientation of the character at the third time point.

[0326] For example, the position and orientation of the character at the third time point and the first time point can be input into the animation algorithm, and the animation algorithm predicts the position and orientation of the character at the first time point based on the position and orientation of the character at the third time point.

[0327] For example, the present application does not limit the execution order of S3025 and any of steps S3021 to S3024.

[0328] S303: The server performs physical simulation on the physical simulation object based on the external force parameter and the second animation data to obtain a first cloth model of the current frame.

[0329] For example, the external force parameters and the second animation data can be input into a physics simulation engine, which then performs cloth simulation on the cloth based on the external force parameters and the second animation data to obtain a first cloth model for the current frame. The first cloth model for the current frame can be used to represent the motion state of the cloth at the first point in time.

[0330] S304: The server encodes the first cloth model of the current frame to obtain an encoded bitstream.

[0331] Exemplarily, the encoding module 212 of the server may encode the first cloth model of the current frame to obtain the encoded first cloth model of the current frame and encapsulate it into an encoded code stream.

[0332] S305: The server sends the encoded code stream.

[0333] For example, the server may send an encoded bitstream carrying the encoded first cloth model of the current frame.

[0334] S306: The server sends the second animation data.

[0335] Exemplarily, the server may further send second animation data.

[0336] It should be noted that this application does not limit the execution order of S305 and S306.

[0337] S307: The terminal device decodes the encoded code stream to obtain a reconstruction result of the first cloth model of the current frame.

[0338] Exemplarily, the terminal device may receive an encoded code stream sent by the server and carrying the encoded first cloth model of the current frame.

[0339] Exemplarily, the terminal device may decode the first cloth model encoded in the current frame in the encoded code stream to obtain a reconstruction result of the first cloth model of the current frame.

[0340] It should be noted that the reconstruction result of the first fabric model of the current frame may be the same as or different from the first fabric model of the current frame; it depends on whether the server encodes the first fabric model of the current frame using lossy encoding or lossless encoding.

[0341] S308: The terminal device merges the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character.

[0342] Exemplarily, the terminal device may perform a weighted calculation on the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the first time point to obtain the target playback position of the character.

[0343] Exemplarily, the terminal device may perform a weighted calculation on the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the first time point based on the first weight and the second weight to obtain the target playback position of the character.

[0344] For example, the first weight corresponds to the playback position of the character's animation sequence at the first time point, and the second weight corresponds to the playback position of the character's animation sequence at the second time point. The first weight is inversely proportional to the difference between the first time point and the second time point, and the second weight is directly proportional to the difference between the first time point and the second time point. In other words, the greater the difference between the first time point and the second time point, the more reliable the playback position of the character's animation sequence at the second time point; and the smaller the difference between the first time point and the second time point, the more reliable the playback position of the character's animation sequence at the first time point. This prevents jumps or teleportation in the character's movements on the client side, making the character's movements in two adjacent frames more natural and smooth.

[0345] Exemplarily, the second weight is the difference between 1 and the first weight.

[0346] For example, the target playback position p2 of the character can be determined by referring to the following formulas (2) to (4): p2 = f mix (p,p1,m) (2) f mix (a,b,c)=c*a+(1-c)*b (3) f clamp (a,b,c)=min(max(a,b),c) (5)

[0347] Among them, p1 is the playback position of the character's animation sequence at the second time point; p is the playback position of the character's animation sequence at the first time point; t c Indicates the predicted time consumption per frame (previous frame) on the end side. All constants in formula (4) can be empirical values.

[0348] For example, the position and orientation of the character at the second time point and the position and orientation of the character at the first time point may be fused to obtain the target position and target orientation of the character; for details, please refer to the description of S308 and will not be repeated here.

[0349] S309: The terminal device fuses the target cloth model of the rendered frame and the reconstruction result of the first cloth model of the current frame to obtain a fusion result.

[0350] Exemplarily, the terminal device may perform weighted calculation on the reconstruction results of the target cloth model of the rendered frame and the first cloth model of the current frame to obtain a fusion result. Specifically, the terminal device may perform weighted calculation on the reconstruction results of the target cloth model of the rendered frame and the first cloth model of the current frame based on the third weight and the fourth weight to obtain a fusion result.

[0351] Exemplarily, the third weight corresponds to the target fabric model of the rendered frame, and the fourth weight corresponds to the reconstruction result of the first fabric model of the current frame; the third weight is inversely proportional to the difference between the fourth time point and the fifth time point, and the fourth weight is proportional to the difference between the fourth time point and the fifth time point. Among them, the fourth time point is the time point when the terminal device actually receives the first fabric model of the rendered frame, and the fifth time point is the time point when the terminal device fuses the second fabric model of the current frame with the first fabric model of the current frame. In other words, the greater the difference between the fourth time point and the fifth time point, the smaller the weight of the target fabric model of the rendered frame; the smaller the difference between the fourth time point and the fifth time point, the greater the weight of the target fabric model of the rendered frame; in this way, the motion state of the fabric on the terminal side can be prevented from jumping or teleporting, making the motion state of the fabric in two adjacent frames more natural and smooth.

[0352] For example, the rendered frame may be a previous frame of the current frame.

[0353] It should be noted that the fourth time point may refer to the time point when the terminal device actually starts to receive the first fabric model of the rendered frame, or the time point when the terminal device actually completes receiving the first fabric model of the rendered frame (the time difference between these two situations is very small and can be ignored in some scenarios; therefore, the fourth time point may also be any time point between the time points corresponding to these two situations). Alternatively, the fourth time point may refer to the time point when the terminal device actually starts to receive the encoded code stream, or the time point when the terminal device actually completes receiving the encoded code stream, or the time point when the terminal device decodes the encoded code stream to obtain the reconstruction result of the first fabric model of the rendered frame (the time difference between these three situations is very small and can be ignored in some scenarios; therefore, the fourth time point may also be any time point between the time points in these three situations).

[0354] It should be noted that the fifth time point can be the time point when the terminal device starts to fuse the target fabric model of the rendered frame and the reconstruction result of the first fabric model of the current frame; it can also be the time point when the fusion of the target fabric model of the rendered frame and the reconstruction result of the first fabric model of the current frame is completed (the time difference between these two situations is very small and can be ignored in some scenes; therefore, the fifth time point can also be any time point between the time points corresponding to these two situations).

[0355] 3B again, the fourth time point may be a time point in the (i-1)th frame. The fifth time point may be a time point in the (i)th frame.

[0356] It should be noted that Figure 3B shows the temporal relationship between the fourth and fifth time points, and does not represent the actual interval between the fourth and fifth time points. Similarly, Figure 3B shows the temporal relationship between the five time points from the first to the fifth time point, and does not represent the actual interval between the five time points from the first to the fifth time point.

[0357] Exemplarily, the fourth weight is the difference between 1 and the third weight.

[0358] For example, the fusion result can be determined by referring to the following formula (6):

[0359] Among them, T4 is the fourth time point, T5 is the fifth time point, V i n is the position of the i-th vertex in the target cloth model of the current frame, V n-1 is the target cloth model of the previous frame, V d n The first cloth model of the current frame.

[0360] For another example, the fusion result can be determined by referring to the following formula (7):

[0361] In formula (7), k is a preset constant, and its value range can be any positive number; it can even be an infinite number, which means that no interpolation is performed.

[0362] FIG3D is a schematic diagram illustrating an exemplary interpolation function.

[0363] The dotted line in Figure 3D is the interpolation function in formula (6) Schematic diagram, the solid line in Figure 3D is the interpolation function in formula (7) As can be seen from Figure 3D, the function values ​​of the interpolation functions of formula (6) and formula (7) are limited to the interval 0 to 1. The design idea is that at a time point close to the time when the latest data (i.e., the first cloth model of the current frame) is received, the interpolation result is close to the target cloth model of the rendered frame, and vice versa.

[0364] The difference between formula (6) and formula (7) is that the interpolation function of formula (6) is is an exponential function, and the interpolation function in formula (7) is a linear function.

[0365] S310: The terminal device fuses the second fabric model of the current frame and the fusion result to obtain a target fabric model of the current frame.

[0366] Exemplarily, the terminal device may perform a weighted calculation on the second fabric model of the current frame and the fusion result to obtain the target fabric model of the current frame. Specifically, the terminal device may perform a weighted calculation on the second fabric model of the current frame and the fusion result based on the fifth weight and the sixth weight to obtain the target fabric model of the current frame.

[0367] Exemplarily, the fifth weight corresponds to the second fabric model of the current frame, and the sixth weight corresponds to the fusion result; the fifth weight is proportional to the difference between the second time point and the sixth time point, and the sixth weight is inversely proportional to the difference between the second time point and the sixth time point. Among them, the sixth time point can be the time point when the terminal device sends the physical simulation data. In this way, the local low-configuration results of the end-side device and the high-configuration results sent by the cloud side can be mixed. If the clock cycle from the end-side device sending data to receiving data is too long, in this case, the credibility of the result sent by the cloud side is not high, which can make the final result less affected by the cloud-side fabric simulation.

[0368] Illustratively, the sixth time point may be the same as or different from the first time point, and this application does not impose any limitation on this.

[0369] It should be noted that the sixth time point can be the time point when the terminal device starts sending physical simulation data, or it can be the predicted time point when the terminal device completes sending physical simulation data (the time difference between these two situations is very small and can be ignored in some scenarios; therefore, the sixth time point can also be any time point between the time points corresponding to these two situations), and this application does not impose any restrictions on this.

[0370] Exemplarily, the sixth weight is the difference between 1 and the fifth weight.

[0371] For example, the target cloth model V of the current frame can be determined by referring to the following formula (8): n :

[0372] Among them, T6 is the sixth time point, T2 is the second time point, V s n The second cloth model of the current frame.

[0373] For another example, the target cloth model V of the current frame can be determined by referring to the following formula (9): n :

[0374] Among them, in the extreme case of severe lag or frame loss, the target fabric model of the current frame determined according to formula (6) and formula (8) is more accurate than the target fabric model of the current frame determined according to formula (7) and formula (9). In the case of slight lag or frame loss, the target fabric model of the current frame determined according to formula (6) and formula (8) is equivalent to the target fabric model of the current frame determined according to formula (7) and formula (9).

[0375] S311 , the terminal device renders the three-dimensional scene based on the target cloth model of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

[0376] For example, the terminal device can determine the target skeletal state of the character based on the target playback position of the character and the character's animation sequence. Then, the skeletal state of the character model in the 3D scene is adjusted according to the target skeletal state. The 3D scene then renders the adjusted skeletal state character model and the target fabric model for the current frame to produce a rendered image for the current frame.

[0377] 4A is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 400. Compared to the rendering process 300, the user operation received by the terminal device in the rendering process 400 also includes switching physical simulation objects.

[0378] S401, the terminal device sends physical simulation data to the server; wherein the physical simulation data includes: external force parameters, first animation data and switching information.

[0379] Exemplarily, the switching information includes at least one of the following: an identifier of the physical simulation object, a model of the physical simulation object, or an encoded model of the physical simulation object; the identifier of the physical simulation object indicates the model of the physical simulation object.

[0380] Exemplarily, when the model of the physical simulation object after switching exists in the server, the switching information may only include the identifier of the physical simulation object. When the model of the physical simulation object after switching is newly generated by the terminal device, the switching information may include the identifier of the physical simulation object and the model of the physical simulation object (or the encoded model of the physical simulation object).

[0381] S402: The server predicts second animation data based on the first animation data in the physical simulation data.

[0382] For example, S402 may refer to the description of S302 above, which will not be repeated here.

[0383] S403: The server performs physical simulation on the physical simulation object based on the second animation data, the external force parameter and the switching information to obtain a first cloth model of the current frame.

[0384] FIG4B is a schematic diagram illustrating an exemplary physical simulation process.

[0385] Exemplarily, S403 may include the following S4031 to S4033:

[0386] S4031: Determine whether the cloth model needs to be switched based on the switching information.

[0387] For example, the server may determine whether the physical simulation object identifier in the switching information received this time is the same as the physical simulation object identifier in the switching information received last time. If the physical simulation object identifier in the switching information received this time is different from the physical simulation object identifier in the switching information received last time, it may be determined that the fabric model needs to be switched, and S4032 is executed. If the physical simulation object identifier in the switching information received this time is the same as the physical simulation object identifier in the switching information received last time, it may be determined that the animation sequence does not need to be switched, and S4024 is executed.

[0388] S4032: The server switches the original fabric model to a third fabric model of the current frame; wherein the third fabric model is determined based on the switching information received this time.

[0389] Exemplarily, the original fabric model is determined based on the last received switching information. For example, the original fabric model can be determined based on the identifier of the physical simulation object in the last received switching information. In another example, the fabric model in the last received switching information can be used as the original fabric model. In another example, the fabric model of the physical simulation object encoded in the last received switching information can be decoded to obtain the original fabric model.

[0390] Exemplarily, the original fabric model is stored in a cache of the rendering pipeline, and the server may switch the original fabric model stored in the cache of the rendering pipeline to a third fabric model. For example, the third fabric model may be determined based on an identifier of the physical simulation object in the currently received switching information. For another example, the fabric model in the currently received switching information may be used as the third fabric model. For another example, the third fabric model may be obtained by decoding the fabric model of the physically simulated object encoded in the currently received switching information.

[0391] S4033: Perform cloth simulation on the cloth based on the external force parameter, the second animation data, and the third cloth model of the current frame to obtain the first cloth model of the current frame.

[0392] Exemplarily, the external force parameters, the second animation data, and the third cloth model of the current frame may be input into a physical simulation engine to obtain the first cloth model of the current frame.

[0393] S4034: Perform cloth simulation on the cloth based on the external force parameter, the second animation data, and the original cloth model to obtain a first cloth model of the current frame.

[0394] Exemplarily, the external force parameters, the second animation data, and the original cloth model are input into the physical simulation engine to obtain the first cloth model of the current frame.

[0395] S404: The server encodes the first cloth model of the current frame to obtain an encoded bitstream.

[0396] S405: The server sends the encoded code stream.

[0397] S406: The server sends the second animation data.

[0398] S407: The terminal device decodes the encoded code stream to obtain a reconstruction result of the first cloth model of the current frame.

[0399] S408: The terminal device merges the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character.

[0400] For example, if the reconstruction result of the first fabric model of the current frame obtained by decoding in S407 corresponds to the physical simulation identifier in the switching information uploaded by the terminal device, S409 and S410 can be executed. If the reconstruction result of the first fabric model of the current frame obtained by decoding in S407 does not correspond to the physical simulation identifier in the switching information uploaded by the terminal device, it means that the server did not use the third fabric model for physical simulation. Therefore, in order to be able to respond to user operations in a timely manner and ensure that the final displayed image includes the switched fabric, the second fabric model of the current frame can be used as the target fabric model of the current frame.

[0401] It should be understood that when the user operation includes switching the physical simulation object, the terminal device can also execute S4032 and S4033 to obtain the second cloth model of the current frame. It should be noted that the physical simulation involved in the terminal device executing S4032 and S4033 is "simple physical simulation".

[0402] S409: The terminal device fuses the target cloth model of the rendered frame and the reconstruction result of the first cloth model of the current frame to obtain a fusion result.

[0403] S410: The terminal device fuses the second fabric model of the current frame and the fusion result to obtain a target fabric model of the current frame.

[0404] S411: The terminal device renders the three-dimensional scene based on the target cloth model of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

[0405] For example, S408 to S411 may refer to the description of S308 to S311 above, which will not be repeated here.

[0406] It should be understood that the switching information is optional in the physical simulation data.

[0407] Figure 5 is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 500. Rendering process 500 is shown based on rendering process 400. In rendering process 500, timestamps can be transmitted between the terminal device and the server to ensure the timeliness of the interactive data transmitted between the terminal device and the server.

[0408] S501, the terminal device sends physical simulation data to the server; wherein the physical simulation data includes: external force parameters, first animation data and switching information.

[0409] S502: The terminal device sends a first timestamp to the server.

[0410] For example, when receiving the user operation, the terminal device may also send a first timestamp to the server. This application does not show the execution order of S501 and S502.

[0411] S503: The server verifies the first timestamp.

[0412] Exemplarily, the server can calculate the difference between the current time and the first timestamp; when the difference between the current time and the first timestamp is greater than the first verification threshold, it can be determined that the verification of the first timestamp has failed, and the process can be terminated, that is, character animation prediction (i.e., S504) and physical simulation, as well as other subsequent steps, are not performed. When the difference between the current time and the first timestamp is less than or equal to the first verification threshold, it can be determined that the verification of the first timestamp is successful, and S504 can be executed. The first verification threshold can be set as required, and this application does not impose any restrictions on this.

[0413] S504: When the first timestamp is verified, the server predicts second animation data based on the first animation data in the physical simulation data.

[0414] S505: The server performs physical simulation on the physical simulation object based on the second animation data, the external force parameter, and the switching information to obtain a first cloth model of the current frame.

[0415] For example, S504 to S505 may refer to the description of S402 to S403 above, which will not be repeated here.

[0416] S506: The server encodes the first cloth model of the current frame to obtain an encoded bitstream.

[0417] S507: The server sends the encoded code stream.

[0418] S508: The server sends the second animation data.

[0419] S509: The server sends a second timestamp.

[0420] Illustratively, the present application does not limit the execution order of S507 , S508 , and S509 .

[0421] S510: The terminal verifies the second timestamp.

[0422] Exemplarily, the terminal device may calculate the difference between the current time and the second timestamp; when the difference between the current time and the second timestamp is greater than the second verification threshold, it may be determined that the verification of the second timestamp has failed, and the second fabric model of the current frame may be used as the target fabric model of the current frame; and the playback position corresponding to the animation sequence of the character at the second moment may be used as the target playback position of the character; and then S515 may be executed. When the difference between the current time and the second timestamp is less than or equal to the second verification threshold, it may be determined that the verification of the second timestamp has succeeded, and S511 to S515 may be executed. The second verification threshold may be set as required, and this application does not impose any restrictions thereto.

[0423] S511: The terminal device decodes the encoded code stream to obtain a reconstruction result of the first cloth model of the current frame.

[0424] S512, when the second timestamp is successfully verified, the terminal device merges the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character.

[0425] S513 , when the second timestamp is successfully verified, the terminal device fuses the target cloth model of the rendered frame and the reconstruction result of the first cloth model of the current frame to obtain a fusion result.

[0426] S514: The terminal device fuses the second fabric model of the current frame and the fusion result to obtain a target fabric model of the current frame.

[0427] S515 , the terminal device renders the three-dimensional scene based on the target cloth model of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

[0428] For example, S512 to S515 may refer to the description of S408 to S411 above, which will not be repeated here.

[0429] It should be understood that, based on the rendering process 300, the terminal device and the server can also transmit timestamps to each other to ensure the timeliness of the interactive data transmitted between the terminal device and the server, and this application does not impose any restrictions on this.

[0430] Figure 6 is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process 600. The difference between rendering process 600 and rendering process 300 is that the physical simulation data in rendering process 600 includes external force parameters and second animation data; and the server does not need to send the second animation data.

[0431] S601, the terminal device sends physical simulation data to the server; wherein the physical simulation data may include: external force parameters and second animation data.

[0432] Exemplarily, when receiving a user operation, the terminal device may further generate second animation data; the specific process may refer to the description of S302 above, which will not be repeated here.

[0433] S602: The server performs physical simulation on the physical simulation object based on the external force parameter and the second animation data to obtain a first cloth model of the current frame.

[0434] S603: The server encodes the first cloth model of the current frame to obtain an encoded bitstream.

[0435] S604: The server sends the encoded code stream.

[0436] S605: The terminal device decodes the encoded code stream to obtain a reconstruction result of the first cloth model of the current frame.

[0437] S606: The terminal device merges the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character.

[0438] S607: The terminal device fuses the target cloth model of the rendered frame and the reconstruction result of the first cloth model of the current frame to obtain a fusion result.

[0439] S608: The terminal device fuses the second fabric model of the current frame and the fusion result to obtain a target fabric model of the current frame.

[0440] S609: The terminal device renders the three-dimensional scene based on the target cloth model of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

[0441] It should be understood that in the rendering process 400 and the rendering process 500, the physical simulation data may also include external force parameters and the second animation data, and the server does not need to send the second animation data; it will not be repeated here.

[0442] The following statistics are based on the number of subjective delay evaluations by different users under different delays, as shown in Table 1.

[0443] Table 1

[0444] Referring to Table 1, when the latency was less than 60ms, five users experienced no lag, and no user experienced noticeable or severe lag. When the latency was between 60 and 70ms, four users experienced no lag, and no user experienced noticeable or severe lag. This analysis is repeated here.

[0445] This application uses a fabric model with 10,000 vertices as an example. At a 30 FPS frame rate on the client side, the data transmission bitrate of this solution is 0.98 Mbps. Ignoring network latency (upload and download latency), the average latency is 55 ms. The experimental results show that this invention can significantly reduce the bitrate, and the latency is acceptable to users. Even considering the current network latency, the perceived latency is still acceptable to users.

[0446] Furthermore, existing solutions, where a server renders and then compresses images or videos and sends them to a terminal device, typically require a downlink bandwidth of 10-20 Mbps. However, for cloth models commonly found in mobile games, this application reduces the downlink bandwidth to 0.5-5 Mbps, while still providing lossless video on the terminal device.

[0447] Figure 7 is a schematic diagram illustrating an exemplary end-cloud collaborative application scenario. The networks in Figure 7 include, but are not limited to, wired networks and wireless networks (such as cellular networks and WIFI (Wireless Fidelity) networks, etc.), which are not limited in this application.

[0448] 7 , exemplarily, after receiving a user operation, the terminal device (also referred to as an end or end-side device) can send first animation data to a server (also referred to as a cloud or cloud server) via a network; then, the server can predict the animation playback position based on the first animation data to obtain second animation data; subsequently, the server can send the second animation data to the terminal device via the network; thereafter, the terminal device can render based on the second animation data to obtain and display a rendered image.

[0449] For example, in the scenario of using a mobile phone for cloud gaming in Figure 7, the mobile phone can send the first animation data of the game character to the server through the network, and the server can send the second animation data of the game character to the mobile phone through the network; in this way, the mobile phone renders based on the second animation data of the game character, and can obtain and display the game screen (that is, the rendered game screen).

[0450] For example, in the scenario of using AR / VR glasses for an AR / VR conference in Figure 7, the AR / VR glasses can send the first animation data of the participating characters to the server via the network, and the server can send the second animation data of the participating characters to the AR / VR glasses via the network; in this way, the AR / VR glasses render based on the second animation data of the participating characters, and can obtain the AR / VR conference screen (that is, the rendered AR / VR conference screen) and display it.

[0451] For example, in the scene of watching a virtual concert using a personal computer in Figure 7, the personal computer can send the first animation data of the virtual singer to the server via the network, and the server can send the second animation data of the virtual singer to the personal computer via the network; in this way, the personal computer performs rendering based on the second animation data of the virtual singer, and can obtain and display the concert picture (that is, the rendered concert picture).

[0452] For example, in the scenario of using a tablet computer to watch the live broadcast of a cloud digital human in FIG7 , the tablet computer can send the first animation data of the digital human to the server via the network, and the server can send the second animation data of the digital human to the tablet computer via the network; in this way, the tablet computer renders based on the second animation data of the digital human, and can obtain and display the live broadcast picture of the digital human (that is, the rendered live broadcast picture).

[0453] It should be understood that the present application can also be applied in other scenarios, and the present application does not limit this.

[0454] The following uses a cloud gaming scenario as an example to illustrate the end-cloud collaborative rendering process.

[0455] FIG8 is a schematic diagram illustrating an exemplary end-cloud collaborative rendering process.

[0456] S801, the terminal device sends first animation data to the server; wherein the first animation data is used to describe the animation state of the character at a first time point, and the first time point may be the time point when the terminal device sends the first animation data to the server.

[0457] Exemplarily, the first animation data includes at least one of the following:

[0458] The character's animation sequence identifier;

[0459] Animation sequences for characters;

[0460] The character's encoded animation sequence;

[0461] The playback position of the character's animation sequence at the first point in time, as well as the character's position and orientation at the first point in time.

[0462] S802, the server predicts second animation data based on the first animation data; wherein the second animation data is used to describe the animation state of the character at a second time point, and the second time point may be the predicted time point when the terminal device receives the second animation data.

[0463] Exemplarily, the second animation data includes the playback position of the character's animation sequence at the second time point, and the character's skeletal state, position, and orientation at the second time point.

[0464] S803, the server sends the second animation data to the terminal device;

[0465] S804, the terminal device merges the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point to obtain the target playback position of the character; the third time point is the time point when the terminal device actually receives the second animation data.

[0466] Exemplarily, S804 may include performing a weighted calculation on the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point based on the first weight and the second weight to obtain the target playback position of the character.

[0467] Exemplarily, the first weight corresponds to the playback position of the character's animation sequence at the second time point, and the second weight corresponds to the playback position of the character's animation sequence at the third time point; the first weight is inversely proportional to the difference between the second time point and the third time point, and the second weight is directly proportional to the difference between the second time point and the third time point.

[0468] Exemplarily, the second weight may be the difference between 1 and the first weight.

[0469] For example, S801 to S804 may refer to the above description and will not be repeated here.

[0470] S805: The terminal device renders the three-dimensional scene based on the target playback position of the character to obtain a rendered image.

[0471] For example, the terminal device can determine the target skeletal state of the character based on the target playback position of the character and the character's animation sequence. Then, the skeletal state of the character model in the 3D scene is adjusted according to the target skeletal state. The 3D scene is then rendered with the adjusted skeletal state to produce a rendered image of the current frame.

[0472] In this way, by fusing the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point, it is possible to prevent the end-side character action from jumping or teleporting, making the character action in two adjacent frames more natural and smooth.

[0473] 9 is a schematic diagram of an exemplary rendering device 900. The rendering device 900 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0474] A first receiving module 901 is configured to receive a first candidate simulation result of a current frame;

[0475] A first fusion module 902 is configured to fuse the second candidate simulation result of the current frame with the first candidate simulation result of the current frame to obtain a target simulation result of the current frame;

[0476] A first rendering module 903 is configured to render the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame;

[0477] Among them, the first candidate simulation result of the current frame is the simulation result corresponding to the first time point, and the first time point is the predicted time point when the terminal device receives the first candidate simulation result of the current frame; the second candidate simulation result of the current frame is the simulation result corresponding to the second time point, and the second time point is the time point when the terminal device actually receives the first candidate simulation result of the current frame.

[0478] Exemplarily, the first fusion module 902 is further configured to fuse the target simulation result of the rendered frame and the first candidate simulation result of the current frame to obtain a fusion result;

[0479] The first fusion module 902 is specifically configured to fuse the second candidate simulation result of the current frame and the fusion result to obtain the target simulation result of the current frame.

[0480] Exemplarily, the rendering apparatus 900 further includes:

[0481] The third sending module is used to send physical simulation data; wherein the physical simulation data includes first animation data, the first animation data is used to describe the animation state of the character at a third time point, and the third time point is the time point when the terminal device sends the first animation data; the first candidate simulation result of the current frame is obtained by physically simulating the physical simulation object based on the second animation data, and the second animation data is predicted based on the first animation data, and the second animation data is used to describe the animation state of the character at the first time point; the three-dimensional scene includes a physical simulation object and a character, and the character is bound to the physical simulation object.

[0482] Exemplarily, the first animation data includes at least one of the following:

[0483] The character's animation sequence identifier;

[0484] Animation sequences for characters;

[0485] The character's encoded animation sequence;

[0486] The playback position of the character's animation sequence at a third point in time, as well as the character's position and orientation at the third point in time.

[0487] Exemplarily, the third sending module is used to send physical simulation data; wherein the physical simulation data includes second animation data, and the second animation data is used to describe the animation state of the character at a first time point; the first candidate simulation result of the current frame is obtained by physically simulating the physical simulation object based on the second animation data; the three-dimensional scene includes the physical simulation object and the character, and the character is bound to the physical simulation object.

[0488] Exemplarily, the second animation data includes the playback position of the character's animation sequence at a first time point, and the character's skeletal state, position, and orientation at the first time point.

[0489] Exemplarily, the rendering apparatus 900 further includes:

[0490] a third fusion module, configured to fuse the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character;

[0491] The first rendering module 903 is specifically configured to render the three-dimensional scene based on the target simulation result of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

[0492] Exemplarily, the third fusion module is specifically used to perform weighted calculation on the playback position of the character's animation sequence at the first time point and the playback position of the character's animation sequence at the second time point based on the first weight and the second weight to obtain the target playback position of the character.

[0493] Exemplarily, the first weight corresponds to the playback position of the character's animation sequence at the first time point, and the second weight corresponds to the playback position of the character's animation sequence at the second time point; the first weight is inversely proportional to the difference between the first time point and the second time point, and the second weight is directly proportional to the difference between the first time point and the second time point.

[0494] Exemplarily, the first fusion module 902 is specifically configured to perform weighted calculation on the target simulation result of the rendered frame and the first candidate simulation result of the current frame based on the third weight and the fourth weight to obtain a fusion result.

[0495] Exemplarily, the third weight corresponds to the target simulation result of the rendered frame, and the fourth weight corresponds to the first candidate simulation result of the current frame; the third weight is inversely proportional to the difference between the fourth time point and the fifth time point, and the fourth weight is directly proportional to the difference between the fourth time point and the fifth time point; the fourth time point is the time point when the terminal device actually receives the first candidate simulation result of the rendered frame, and the fifth time point is the time point when the terminal device fuses the second candidate simulation result of the current frame and the first candidate simulation result of the current frame.

[0496] Exemplarily, the first fusion module 902 is specifically configured to perform weighted calculation on the second candidate simulation result and the fusion result of the current frame based on the fifth weight and the sixth weight to obtain the target simulation result of the current frame.

[0497] Exemplarily, the fifth weight corresponds to the second candidate simulation result of the current frame, and the sixth weight corresponds to the fusion result; the fifth weight is proportional to the difference between the second time point and the sixth time point, and the sixth weight is inversely proportional to the difference between the second time point and the sixth time point; the sixth time point is the time point when the terminal device sends physical simulation data.

[0498] Exemplarily, the first candidate simulation result of the current frame includes the first cloth model of the current frame, the second candidate simulation result of the current frame includes the second cloth model of the current frame, and the target simulation result of the current frame includes the target cloth model of the current frame;

[0499] A first receiving module 901 is specifically configured to receive an encoded code stream, where the encoded code stream includes an encoded first cloth model of a current frame;

[0500] The rendering device 900 further includes a decoding module, the decoding module being configured to decode the encoded bitstream to obtain a reconstruction result of the first cloth model of the current frame;

[0501] The first fusion module 902 is specifically configured to fuse the reconstruction results of the second cloth model of the current frame and the first cloth model of the current frame to obtain the target cloth model of the current frame.

[0502] Exemplarily, the physical simulation data further includes switching information, and the switching information includes at least one of the following: an identifier of the physical simulation object, a model of the physical simulation object, or an encoded model of the physical simulation object; the identifier of the physical simulation object indicates the model of the physical simulation object.

[0503] Exemplarily, the third sending module is further configured to send a first timestamp; wherein the first candidate simulation result of the current frame is obtained by performing physical simulation on the physical simulation object based on the physical simulation data after the first timestamp verification is successful.

[0504] Exemplarily, the first receiving module 901 is further configured to receive a second timestamp;

[0505] Exemplarily, the first fusion module 902 is specifically configured to, when the second timestamp is successfully verified, execute the step of fusing the second candidate simulation result of the current frame with the first candidate simulation result of the current frame to obtain the target simulation result of the current frame.

[0506] Exemplarily, the first candidate simulation result of the current frame includes at least one of the following: the position and orientation of the rigid body, or a cloth model.

[0507] Figure 10 is a schematic diagram of an exemplary physical simulation device 1000. The physical simulation device 1000 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0508] The second receiving module 1001 is used to receive physical simulation data;

[0509] A physical simulation module 1002 is configured to perform physical simulation on a physical simulation object based on the physical simulation data to obtain a first candidate simulation result of a current frame;

[0510] The encoding module 1003 is configured to encode the first candidate simulation result of the current frame to obtain an encoded bitstream;

[0511] The first sending module 1004 is configured to send the encoded code stream.

[0512] Exemplarily, the first candidate simulation result of the current frame includes a first cloth model of the current frame.

[0513] 11 is a schematic diagram of an exemplary rendering device 1100. The rendering device 1100 can be used to execute the method of the aforementioned embodiment. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0514] The second sending module 1101 is configured to send first animation data; wherein the first animation data is used to describe the animation state of the character at a first time point, and the first time point is the time point when the terminal device sends the first animation data to the server;

[0515] A third receiving module 1102 is configured to receive second animation data, wherein the second animation data is used to describe the animation state of the character at a second time point, and the second time point is a predicted time point when the terminal device receives the second animation data;

[0516] A second fusion module 1103 is configured to fuse the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at a third time point to obtain a target playback position of the character; wherein the third time point is the time point when the terminal device actually receives the second animation data;

[0517] The second rendering module 1104 is configured to render the three-dimensional scene based on the target playback position of the character to obtain a rendered image; wherein the three-dimensional scene includes the character.

[0518] Exemplarily, the second fusion module 1103 is specifically used to perform weighted calculation on the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point based on the first weight and the second weight to obtain the target playback position of the character.

[0519] Exemplarily, the first weight corresponds to the playback position of the character's animation sequence at the second time point, and the second weight corresponds to the playback position of the character's animation sequence at the third time point; the first weight is inversely proportional to the difference between the second time point and the third time point, and the second weight is directly proportional to the difference between the second time point and the third time point.

[0520] Exemplarily, the first animation data includes at least one of the following:

[0521] The character's animation sequence identifier;

[0522] Animation sequences for characters;

[0523] The character's encoded animation sequence;

[0524] The playback position of the character's animation sequence at the first point in time, as well as the character's position and orientation at the first point in time.

[0525] Exemplarily, the second animation data includes the playback position of the character's animation sequence at the second time point, and the character's skeletal state, position, and orientation at the second time point.

[0526] In an example, FIG12 shows a schematic block diagram of a device 1200 according to an embodiment of the present application. The device 1200 may include: a processor 1201 and a transceiver / transceiver pin 1202 , and optionally, a memory 1203 .

[0527] The various components of the device 1200 are coupled together via a bus 1204, wherein the bus 1204 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 1204 in the figure.

[0528] Optionally, the memory 1203 may be used to store instructions in the aforementioned method embodiment. The processor 1201 may be used to execute the instructions in the memory 1203 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0529] The apparatus 1200 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0530] The electronic device may be a server or a terminal device.

[0531] Fig. 13 shows an exemplary structural diagram of an electronic device 120. The electronic device 120 may be a terminal device.

[0532] 13 , the electronic device 120 includes components such as an application processor 1201, a memory 1202, a wireless communication module 1203, a graphics processing unit (GPU) 1204, and an input / output (I / O) device 1205. Those skilled in the art will appreciate that the hardware structure shown in FIG13 does not limit the electronic device 120 , and the electronic device 120 may include more or fewer components than shown, or may combine certain components or arrange the components differently.

[0533] The following is a detailed introduction to the various components of the electronic device 120 with reference to FIG13:

[0534] The application processor 1201 is the control center of the electronic device 120, and connects various components of the electronic device 120 using various interfaces and buses. In some embodiments, the application processor 1201 may include one or more processing modules.

[0535] Memory 1202 stores computer programs, such as the operating system 1222 and application programs 1221 shown in FIG. 13 . Application processor 1201 is configured to execute the computer programs in memory 1202 to implement the functions defined by the computer programs. For example, application processor 1201 executes operating system 1222 to implement various operating system functions on electronic device 120 . Memory 1202 also stores other data besides computer programs, such as data generated during the operation of operating system 1222 and application programs 1221. Memory 1202 is a non-volatile storage medium, generally including internal memory and external storage. Internal memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), or cache. External storage includes, but is not limited to, flash memory, hard disks, optical disks, and universal serial bus (USB) drives. Computer programs are typically stored in external storage, and the processor loads the program from external storage into internal memory before executing the computer program.

[0536] The memory 1202 may be independent and connected to the application processor 1201 via a bus; the memory 1202 may also be integrated with the application processor 1201 into a chip subsystem.

[0537] The wireless communication module 1203 is used to provide wireless communication solutions applied to the electronic device 120, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 1203 can be one or more devices that integrate at least one communication processing module. The wireless communication module 1203 receives electromagnetic waves via an antenna, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the application processor 1201. The wireless communication module 1203 can also receive the signal to be sent from the application processor 1201, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0538] GPU 1204: Performs drawing and rendering calculations on image data to generate images for display. Also known as a display core or visual processor, this is a microprocessor that performs image processing and may include 2D and / or 3D processing capabilities. Electronic device 120 may include one or more GPUs that execute program instructions to generate or modify display information. For example, a GPU can be used to perform operations such as physics simulation, rendering, and predicting second animation data based on first animation data.

[0539] The input / output device 1205 includes but is not limited to a display 1251 , a touch screen 1253 , and an audio circuit 1255 .

[0540] The touch screen 1253 can collect touch events on or near the electronic device 120 by a user (e.g., a user's operation on or near the touch screen 1253 using a finger, stylus, or any other suitable object) and transmit the collected touch events to other devices (e.g., the application processor 1201). The user's operation near the touch screen 1253 is referred to as a hover touch; with hover touch, the user can select, move, or drag an object (e.g., an icon) without directly contacting the touch screen 1253. Furthermore, the touch screen 1253 can be implemented using various types, including resistive, capacitive, infrared, and surface acoustic wave.

[0541] The display (also known as a display screen) 1251 is used to display information input by the user or information presented to the user. The display can be configured in the form of a liquid crystal display (LCD), an organic light emitting diode (OLED), or the like. A touch screen 1253 can be overlaid on the display 1251. When the touch screen 1253 detects a touch event, it transmits the information to the application processor 1201 to determine the type of touch event. The application processor 1201 can then provide a corresponding visual output on the display 1251 based on the type of touch event. Although in Figure 13, the touch screen 1253 and the display 1251 are two separate components to implement the input and output functions of the electronic device 120, in some embodiments, the touch screen 1253 and the display 1251 can be integrated to implement the input and output functions of the electronic device 120. In addition, the touch screen 1253 and the display 1251 can be configured in a full-panel manner on the front of the electronic device 120 to achieve a borderless structure. For example, the display can be used to display the rendered image of the current frame.

[0542] The audio circuit 1255, speaker 1256, and microphone 1257 provide an audio interface between the user and the electronic device 120. The audio circuit 1255 can convert received audio data into electrical signals, transmit them to the speaker 1256, and the speaker 1256 converts them into sound signals for output. Meanwhile, the microphone 1257 converts collected sound signals into electrical signals, which are received by the audio circuit 1255 and converted into audio data. The audio data is then transmitted via a modem processor and a radio frequency module to, for example, another electronic device, or the audio data is output to the memory 1202 for further processing.

[0543] Optionally, the electronic device 120 may further include a microcontroller unit (MCU) 1206, which is a coprocessor for acquiring and processing data from the sensor 1261. The MCU 1206 has lower processing power and power consumption than the application processor 1201, but is "always on." This allows it to continuously collect and process sensor data even when the application processor 1201 is in sleep mode, ensuring normal sensor operation with extremely low power consumption. In one embodiment, the MCU 1206 may be a sensor hub chip. The sensor 1261 may include a light sensor and a gyroscope sensor. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display 1251 based on ambient light levels, and the proximity sensor can power down the display screen when the electronic device 120 is brought to the ear. The gyroscope sensor can be used to determine the motion posture of the electronic device 120. In some embodiments, the gyroscope sensor can be used to determine the angular velocity of the electronic device 120 around three axes (i.e., the x, y, and z axes). The angular velocity information obtained by the gyroscope sensor is converted into a rotation matrix that can describe the rotation of the object, which can be used to align the image. The gyroscope sensor can also be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor detects the angle of the electronic device 120 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 120 through reverse movement to achieve anti-shake. The gyroscope sensor can also be used for navigation and somatosensory game scenes. Sensor 1261 can also include other sensors such as accelerometers, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here. MCU 1206 and sensor 1261 can be integrated into the same chip, or they can be separate components connected through a bus.

[0544] Optionally, the electronic device 120 may further include an image signal processor (ISP) 1207. The ISP 1207 is connected to the camera 1271 to capture images and perform image processing (such as exposure control, white balance, color calibration, or noise removal) to generate image data. The ISP 1207 may include a processor core that performs necessary software processing or a pure hardware implementation.

[0545] Optionally, the electronic device 120 may further include a mobile communication module 1208. The mobile communication module 1208 may provide solutions for wireless communications, including 2G / 3G / 4G / 5G, applied to the electronic device 120. The mobile communication module 1208 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 1208 may receive electromagnetic waves from an antenna, filter, amplify, and perform other processing on the received electromagnetic waves, and transmit them to a modem processor for demodulation. The mobile communication module 1208 may also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some of the functional modules of the mobile communication module 1208 may be provided in the application processor 1201. In some embodiments, at least some of the functional modules of the mobile communication module 1208 may be provided in the same device as at least some of the modules of the application processor 1201.

[0546] Furthermore, the operating system 1222 carried by the electronic device 120 may be Or other operating systems, the embodiments of the present application do not impose any restrictions on this.

[0547] Those skilled in the art may understand that the electronic device 120 may include fewer or more components than those shown in FIG13 , and the electronic device shown in FIG13 only includes components that are more relevant to the multiple implementations disclosed in the embodiments of the present application.

[0548] In one possible implementation, the rendering method of the embodiment of the present application can be executed by the electronic device 120. For example, when a user uses the electronic device 120 to play a mobile game and tries on clothes in a virtual fitting room, the rendering method provided in the embodiment of the present application is used to perform rendering locally on the electronic device 120.

[0549] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0550] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 1202.

[0551] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0552] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0553] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0554] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0555] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0556] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0557] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0558] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0559] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0560] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0561] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0562] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0563] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A device-cloud collaborative system, characterized in that: The end-cloud collaborative system includes a server and a terminal device, wherein: The terminal device is used to send physical simulation data to the server; The server is configured to perform physical simulation on the physical simulation object based on the physical simulation data to obtain a first candidate simulation result of the current frame; and send the first candidate simulation result of the current frame to the terminal device; The terminal device is further configured to fuse the second candidate simulation result of the current frame with the first candidate simulation result of the current frame to obtain a target simulation result of the current frame; and render the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame; In which, the three-dimensional scene includes the physical simulation object; the first candidate simulation result of the current frame is the simulation result corresponding to the first time point, and the first time point is the predicted time point when the terminal device receives the first candidate simulation result of the current frame; the second candidate simulation result of the current frame is the simulation result corresponding to the second time point, and the second time point is the time point when the terminal device actually receives the first candidate simulation result of the current frame.

2. The system according to claim 1, wherein: The terminal device is further configured to fuse the target simulation result of the rendered frame and the first candidate simulation result of the current frame to obtain a fusion result; The terminal device is specifically configured to fuse the second candidate simulation result of the current frame and the fusion result to obtain the target simulation result of the current frame.

3. The system according to claim 1, wherein: The first candidate simulation result of the current frame includes the first cloth model of the current frame, the second candidate simulation result of the current frame includes the second cloth model of the current frame, and the target simulation result of the current frame includes the target cloth model of the current frame; The server is further configured to encode the first cloth model of the current frame to obtain an encoded code stream; and send the encoded code stream to the terminal device; The terminal device is further configured to decode the encoded code stream to obtain a reconstruction result of the first cloth model of the current frame; And fusing the reconstruction results of the second cloth model of the current frame and the first cloth model of the current frame to obtain the target cloth model of the current frame.

4. A rendering method applied to a terminal device, characterized in that: The method comprises: receiving a first candidate simulation result of a current frame; fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain a target simulation result of the current frame; Rendering the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame; Among them, the first candidate simulation result of the current frame is the simulation result corresponding to the first time point, and the first time point is the predicted time point when the terminal device receives the first candidate simulation result of the current frame; the second candidate simulation result of the current frame is the simulation result corresponding to the second time point, and the second time point is the time point when the terminal device actually receives the first candidate simulation result of the current frame.

5. The method according to claim 4, characterized in that The method further comprises: fusing the target simulation result of the rendered frame and the first candidate simulation result of the current frame to obtain a fusion result; The fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain the target simulation result of the current frame includes: The second candidate simulation result of the current frame and the fusion result are fused to obtain a target simulation result of the current frame.

6. The method according to claim 4 or 5, characterized in that The method further comprises: Send physical simulation data; wherein, the physical simulation data includes first animation data, the first animation data is used to describe the animation state of the character at a third time point, and the third time point is the time point when the terminal device sends the first animation data; the first candidate simulation result of the current frame is obtained by physically simulating the physical simulation object based on the second animation data, and the second animation data is predicted based on the first animation data, and the second animation data is used to describe the animation state of the character at the first time point; the three-dimensional scene includes the physical simulation object and the character, and the character is bound to the physical simulation object.

7. The method according to claim 6, characterized in that The first animation data includes at least one of the following: An animation sequence identifier of the character; animation sequences of said characters; an encoded animation sequence of the character; The playback position of the character's animation sequence at the third time point, and the position and orientation of the character at the third time point.

8. The method according to claim 4 or 5, characterized in that The method further comprises: Send physical simulation data; wherein the physical simulation data includes second animation data, and the second animation data is used to describe the animation state of the character at a first time point; the first candidate simulation result of the current frame is obtained by physically simulating the physical simulation object based on the second animation data; the three-dimensional scene includes the physical simulation object and the character, and the character is bound to the physical simulation object.

9. The method according to any one of claims 6 to 8, characterized in that The second animation data includes the playback position of the character's animation sequence at a first time point, and the skeletal state, position, and orientation of the character at the first time point.

10. The method according to claim 9, characterized in that The method further comprises: fusing the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character; The rendering of the three-dimensional scene based on the target simulation result of the current frame to obtain a rendered image of the current frame includes: The three-dimensional scene is rendered based on the target simulation result of the current frame and the target playback position of the character to obtain a rendered image of the current frame.

11. The method according to claim 10, characterized in that The step of fusing the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the first time point to obtain a target playback position of the character includes: Based on the first weight and the second weight, a weighted calculation is performed on the playback position of the character's animation sequence at the first time point and the playback position of the character's animation sequence at the second time point to obtain the target playback position of the character.

12. The method according to claim 11, characterized in that The first weight corresponds to the playback position of the character's animation sequence at a first time point, and the second weight corresponds to the playback position of the character's animation sequence at a second time point; The first weight is inversely proportional to the difference between the first time point and the second time point, and the second weight is proportional to the difference between the first time point and the second time point.

13. The method according to any one of claims 5 to 12, characterized in that The step of fusing the target simulation result of the rendered frame and the first candidate simulation result of the current frame to obtain a fusion result comprises: Based on the third weight and the fourth weight, a weighted calculation is performed on the target simulation result of the rendered frame and the first candidate simulation result of the current frame to obtain the fusion result.

14. The method according to claim 13, wherein: The third weight corresponds to the target simulation result of the rendered frame, and the fourth weight corresponds to the first candidate simulation result of the current frame; The third weight is inversely proportional to the difference between the fourth time point and the fifth time point, and the fourth weight is directly proportional to the difference between the fourth time point and the fifth time point; The fourth time point is the time point when the terminal device actually receives the first candidate simulation result of the rendered frame, and the fifth time point is the time point when the terminal device fuses the second candidate simulation result of the current frame and the first candidate simulation result of the current frame.

15. The method according to any one of claims 5 to 14, characterized in that The step of fusing the second candidate simulation result of the current frame and the fusion result to obtain the target simulation result of the current frame comprises: Based on the fifth weight and the sixth weight, a weighted calculation is performed on the second candidate simulation result of the current frame and the fusion result to obtain the target simulation result of the current frame.

16. The method according to claim 15, characterized in that The fifth weight corresponds to the second candidate simulation result of the current frame, and the sixth weight corresponds to the fusion result; The fifth weight is proportional to the difference between the second time point and the sixth time point, and the sixth weight is inversely proportional to the difference between the second time point and the sixth time point; The sixth time point is the time point when the terminal device sends physical simulation data, and the first candidate simulation result of the current frame is obtained by performing physical simulation on the physical simulation object based on the physical simulation data.

17. The method according to any one of claims 4 to 16, characterized in that The first candidate simulation result of the current frame includes the first cloth model of the current frame, the second candidate simulation result of the current frame includes the second cloth model of the current frame, and the target simulation result of the current frame includes the target cloth model of the current frame; The receiving the first candidate simulation result of the current frame includes: receiving an encoded bitstream, wherein the encoded bitstream includes an encoded first cloth model of the current frame; The method further comprises: Decoding the encoded bitstream to obtain a reconstruction result of the first cloth model of the current frame; The fusing the second candidate simulation result of the current frame and the first candidate simulation result of the current frame to obtain the target simulation result of the current frame includes: The reconstruction results of the second cloth model of the current frame and the first cloth model of the current frame are fused to obtain the target cloth model of the current frame.

18. The method according to any one of claims 6 to 12, characterized in that The physical simulation data further includes switching information, and the switching information includes at least one of the following: an identifier of the physical simulation object, a model of the physical simulation object, or an encoded model of the physical simulation object; the identifier of the physical simulation object indicates the model of the physical simulation object.

19. The method according to any one of claims 4 to 18, characterized in that The method further comprises: Sending a first timestamp; wherein, the first candidate simulation result of the current frame is obtained by performing physical simulation on the physical simulation object based on physical simulation data after the first timestamp verification is successful.

20. The method according to any one of claims 4 to 19, characterized in that The method further comprises: receiving a second timestamp; When the second timestamp is successfully verified, the step of fusing the second candidate simulation result of the current frame with the first candidate simulation result of the current frame to obtain the target simulation result of the current frame is performed.

21. The method according to any one of claims 4 to 20, characterized in that The first candidate simulation result of the current frame includes at least one of the following: a position and orientation of a rigid body, or a cloth model.

22. A physical simulation method applied to a server, characterized in that: The method comprises: receiving physical simulation data; Performing physical simulation on the physical simulation object based on the physical simulation data to obtain a first candidate simulation result of the current frame; Encoding the first candidate simulation result of the current frame to obtain an encoded bitstream; The encoded code stream is sent.

23. The method according to claim 20, characterized in that The first candidate simulation result of the current frame includes a first cloth model of the current frame.

24. A terminal-cloud collaborative system, characterized in that: The end-cloud collaborative system includes a server and a terminal device, wherein: The terminal device is configured to send first animation data to the server; wherein the first animation data is configured to describe the animation state of the character at a first time point, and the first time point is the time point at which the terminal device sends the first animation data to the server; The server is configured to predict second animation data based on the first animation data, wherein the second animation data is used to describe the animation state of the character at a second time point, the second time point being the predicted time point at which the terminal device receives the second animation data; and send the second animation data to the terminal device; The terminal device is used to fuse the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point to obtain the target playback position of the character; render the three-dimensional scene based on the target playback position of the character to obtain a rendered image; wherein the three-dimensional scene includes the character, and the third time point is the time point when the terminal device actually receives the second animation data.

25. A rendering method applied to a terminal device, characterized in that: The method comprises: Sending first animation data; wherein the first animation data is used to describe the animation state of the character at a first time point, and the first time point is the time point when the terminal device sends the first animation data to the server; Receiving second animation data; wherein the second animation data is used to describe the animation state of the character at a second time point, and the second time point is a predicted time point when the terminal device receives the second animation data; fusing the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at a third time point to obtain a target playback position of the character; wherein the third time point is the time point at which the terminal device actually receives the second animation data; A three-dimensional scene is rendered based on the target playback position of the character to obtain a rendered image; wherein the three-dimensional scene includes the character.

26. The method according to claim 25, characterized in that The step of fusing the playback position of the character's animation sequence at the second time point with the playback position of the character's animation sequence at the third time point to obtain a target playback position of the character includes: Based on the first weight and the second weight, a weighted calculation is performed on the playback position of the character's animation sequence at the second time point and the playback position of the character's animation sequence at the third time point to obtain the target playback position of the character.

27. The method according to claim 26, characterized in that The first weight corresponds to the playback position of the character's animation sequence at a second time point, and the second weight corresponds to the playback position of the character's animation sequence at a third time point; The first weight is inversely proportional to the difference between the second time point and the third time point, and the second weight is proportional to the difference between the second time point and the third time point.

28. The method according to any one of claims 25 to 27, characterized in that The first animation data includes at least one of the following: An animation sequence identifier of the character; animation sequences of said characters; an encoded animation sequence of the character; The playback position of the character's animation sequence at a first point in time, and the position and orientation of the character at the first point in time.

29. The method according to any one of claims 25 to 28, characterized in that The second animation data includes the playback position of the character's animation sequence at a second time point, and the skeletal state, position, and orientation of the character at the second time point.

30. A terminal device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the method according to any one of claims 4 to 21.

31. A server, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the server executes the method according to any one of claims 22 to 23.

32. A terminal device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the method according to any one of claims 25 to 29.

33. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when run on a computer or a processor, enables the computer or the processor to execute the method according to any one of claims 4 to 23, or enables the computer or the processor to execute the method according to any one of claims 25 to 29.

34. A computer program product, characterized in that The computer program product comprises computer instructions, which, when executed by a computer or a processor, cause the steps of the method according to any one of claims 4 to 23 to be performed, or cause the steps of the method according to any one of claims 25 to 29 to be performed.

Images