Digital twin display method, training method and computing device

Abstract

The embodiment of the invention discloses a digital twin display method, a training method and computing equipment, relates to the technical field of image processing, and at least can reduce the requirement on the performance of terminal equipment. The method comprises the following steps: displaying a two-dimensional image in a display page of a digital twin; the two-dimensional image is used for displaying a display effect and / or a display view angle of the digital twin; wherein the second image displayed in the current frame is obtained by predicting according to the first image displayed in the previous frame and the prediction prompt information; the prediction prompt information is used for indicating the change of the display effect and / or the display view angle of the second image relative to the first image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technologies, and in particular, to a display method, a training method, and a computing device for digital twins. Background Art

[0002] With the continuous development of computer simulation technologies, in order to visually display computing devices including servers, switches, and cabinets on a terminal device, a three-dimensional (3D) model of a physical device can be created, and the 3D model of the simulated physical device can be displayed on a computer, such as a digital twin.

[0003] Currently, the display of a digital twin is to transmit a 3D model to a terminal device (client), and the terminal device loads, renders, and displays it. This display method has relatively high requirements for the performance of the terminal device (especially the graphics card performance). In addition, when the terminal device changes, it is necessary to re-model the digital twin so that the digital twin can adapt to the new terminal device. Therefore, the use scenarios of the current display method are limited, and thus the digital twin cannot be well promoted and popularized. Summary of the Invention

[0004] The embodiments of the present application provide a display method, a training method, and a computing device for digital twins, which can at least reduce the requirements for the performance of terminal devices.

[0005] In a first aspect, the embodiments of the present application provide a display method for a digital twin, the method including: displaying a two-dimensional image in a display page of the digital twin; the two-dimensional image is used to display the display effect and / or display perspective of the digital twin; wherein, the second image displayed in the current frame is predicted based on the first image displayed in the previous frame and prediction hint information; the prediction hint information is used to indicate the change in the display effect and / or display perspective of the second image relative to the first image.

[0006] The display method for a digital twin provided by the embodiments of the present application simulates the change in the display effect and / or display perspective during the display process of a 3D model by switching between the two-dimensional first image and the second image. Compared with the traditional method of directly rendering a 3D model, it does not require the terminal device to load or render the 3D model, thus effectively reducing the requirements for the performance of the terminal device. It can also display a high-precision digital twin on a terminal device with low performance without updating the digital twin, which helps to promote and popularize the digital twin.

[0007] In a possible implementation, displaying a two-dimensional image of a digital twin includes: when displaying a first image, in response to meeting a switching condition, obtaining a second image; a prediction prompt message is determined based on the switching condition; updating the first image in the display page to the second image.

[0008] It should be noted that in the related art, when a terminal device displays a digital twin, it obtains the model rendering data of the digital twin, and then the terminal device renders the model rendering data and displays the digital twin on the screen. This display method has relatively high requirements for the performance of the terminal device (especially the graphics card performance). In the embodiments of the present application, when displaying a digital twin, a two-dimensional image including the digital twin is obtained, and then the digital twin is displayed on the screen in the form of displaying the two-dimensional image, thereby effectively reducing the performance requirements for the terminal device.

[0009] In a possible implementation, meeting the switching condition includes: receiving a switching operation from the user; the switching operation is used to indicate switching the rendering effect of the digital twin; the prediction prompt message includes the operation information of the switching operation.

[0010] In a possible implementation, obtaining the second image includes: inputting the first image into a first model to obtain the second image; the first model is used to predict the display effect and / or the change of the display perspective of the digital twin in the two-dimensional image.

[0011] In a possible implementation, before inputting the first image into the first model, the method further includes: obtaining the performance parameters of the terminal device; when the performance parameters meet the model operation requirements, obtaining the first model from the server.

[0012] It can be understood that the first model is deployed on the terminal device side, and the terminal device runs the first model, which can save the process of transmitting the two-dimensional image through the network. The terminal device can obtain and display the two-dimensional image more efficiently, ensure smooth switching between images, realize the change of the simulated 3D model, and enable the user to have a more immersive experience. In addition, the performance requirements for running the first model are much smaller than those for traditional 3D model rendering, thereby reducing the performance requirements for the terminal device during the display of the digital twin.

[0013] In a possible implementation, obtaining the second image includes: sending a first request to the server; the first request is used to request to obtain the second image; receiving the second image sent by the server.

[0014] It can be understood that after predicting the second image, the server can send it to the terminal device through the network. This is equivalent to handing over the pressure of running the first model to the server, further reducing the performance requirements for the terminal device.

[0015] In a possible implementation, before displaying the two-dimensional image, the method further includes: displaying a first UV map of the digital twin; the first UV map is used to display the surface details of the digital twin; in response to an adjustment operation on the first UV map, sending a second request to the server; the second request is used to request to obtain a second UV map; the adjustment operation is used to indicate the change in the surface details of the digital twin with respect to the first UV map for the second UV map; in response to a confirmation operation on the second UV map, sending a third request to the server; the third request is used to request the server to construct a two-dimensional image of the digital twin based on the second UV map; obtaining the two-dimensional image.

[0016] In a possible implementation, the server deploys a second model; the second model is used to generate a second UV map according to the first UV map and the adjustment operation.

[0017] It should be noted that in the embodiments of the present application, the user's personalized requirements for the digital twin are met through the adjustment operation of the UV map by the user. By using model generation to replace the process of the user directly adjusting the 3D model, the construction efficiency of the digital twin can be effectively improved, and the user's usage threshold can be reduced.

[0018] In a second aspect, an embodiment of the present application provides a training method, and the method includes: obtaining a training data set; each pair of training data in the training data set includes: a first training image before the switching operation, prediction hint information of the switching operation, and a second training image obtained by operating on the first training image based on the switching operation; the first training image is an image of the digital twin displayed according to a first display effect and / or a first display perspective, and the second training image is an image of the digital twin displayed according to a second display effect and / or a second display perspective; using the first training image and the prediction hint information as sample data, and the second training image as label data of the sample data, training an initial model to obtain a first model; the first model is used to predict the change in the display effect and / or display perspective of the digital twin in the two-dimensional image.

[0019] In a possible implementation, the method further includes: during the process of running the digital twin, simulating a switching operation on the digital twin to obtain a training data set.

[0020] In a possible implementation, the initial model is a diffusion model.

[0021] In a third aspect, an embodiment of the present application provides a training method, which includes: obtaining a second training data set. Each pair of training data in the second training data set includes: an initial UV map, adjustment information, and a target UV map obtained by adjusting the initial UV map according to the adjustment information. Using the initial UV map and the prompt information as sample data, and using the target UV map as the label data of the sample data to train the initial model, a second model is obtained. The second model is used to adjust the surface details of the UV map according to the adjustment information.

[0022] In a fourth aspect, an embodiment of the present application provides a display device for a digital twin, and the display device for the digital twin is used to execute any one of the digital twin display device methods provided in the first aspect above.

[0023] In a fifth aspect, an embodiment of the present application provides a training device, and the training device is used to execute any one of the training methods provided in the second aspect or the third aspect above.

[0024] In a sixth aspect, an embodiment of the present application provides a computing device, which includes a processor and a memory, and the processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor so that the computing device implements the digital twin display method or training method as described in the above aspects.

[0025] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which at least one computer program instruction is stored, and the computer program instruction is loaded and executed by the processor to implement the digital twin display method or training method as described in the above aspects.

[0026] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes the digital twin display method or training method provided in the various optional implementations of the first aspect or the second aspect above.

[0027] For the specific descriptions of the second aspect to the eighth aspect and their various implementation manners in the embodiments of the present application, reference may be made to the detailed descriptions in the first aspect and its various implementation manners; and, for the beneficial effects of the second aspect to the eighth aspect and their various implementation manners, reference may be made to the beneficial effect analysis in the first aspect and its various implementation manners, which will not be elaborated here.

[0028] These aspects or other aspects of the embodiments of the present application will be more clearly understood in the following description. Description of the Drawings

[0029] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the hardware structure of a server provided by an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the composition of a digital twin system provided by an embodiment of the present application;

[0032] Figure 4 A schematic flowchart of a method for displaying a digital twin provided by an embodiment of the present application;

[0033] Figure 5 A schematic diagram of a change in the display perspective provided by an embodiment of the present application;

[0034] Figure 6 A schematic diagram of a change in the display effect provided by an embodiment of the present application;

[0035] Figure 7 A schematic flowchart of another method for displaying a digital twin provided by an embodiment of the present application;

[0036] Figure 8 A schematic flowchart of yet another method for displaying a digital twin provided by an embodiment of the present application;

[0037] Figure 9 A schematic flowchart of yet another method for displaying a digital twin provided by an embodiment of the present application;

[0038] Figure 10 A schematic flowchart of a training method provided by an embodiment of the present application;

[0039] Figure 11 A schematic flowchart of the new construction process of a digital twin provided by an embodiment of the present application;

[0040] Figure 12 A schematic flowchart of the training process of a first model provided by an embodiment of the present application;

[0041] Figure 13 A schematic flowchart of an inference process provided by an embodiment of the present application;

[0042] Figure 14 A schematic diagram of the structure of a display device for a digital twin provided by an embodiment of the present application;

[0043] Figure 15 A schematic diagram of the structure of a training device provided by an embodiment of the present application. Detailed implementation manners

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings.

[0045] As used herein, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0046] Moreover, in the description of this application, unless otherwise specified, "a plurality of" means two or more than two. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item(s) or plural item(s). For example, at least one (item) of a, b, or c may represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c may be single or multiple.

[0047] In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, in the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and roles. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different. At the same time, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner for easy understanding.

[0048] First, an exemplary introduction to the technical terms involved in the embodiments of this application is provided.

[0049] 1. Digital twin: It can be a simulation process that makes full use of data such as physical models, sensor updates, and operation history, integrates multiple disciplines, multiple physical quantities, multiple scales, and multiple probabilities, and completes mapping in the virtual space, thereby reflecting the entire life cycle process of the corresponding physical equipment. Digital twin is a concept that goes beyond reality and can be regarded as a digital mapping system of one or more important and interdependent equipment systems.

[0050] 2. Digital twin: It can be a digital version of a "clone" created on the basis of a device or system, and can include virtual representations of the real world including physical objects, processes, relationships and behaviors. The digital twin skeleton can refer to the geometric body of each digital twin in the digital twin system. In geometry, a finite shape surrounded by several geometric faces (planes or curved surfaces) can be called a geometric body, the faces that surround the geometric body are called the interfaces or surfaces of the geometric body, the intersections of different interfaces are called the edges of the geometric body, and the intersections of different edges are called the vertices of the geometric body. The geometric body can also be regarded as a finite space area divided by several geometric faces in space.

[0051] 3. 3D model: It is a polygonal representation of an object, usually displayed by a computer or other display device. The displayed object can be a real-world entity or a fictional object.

[0052] 4. UV mapping: It is a technology that maps 2D texture images to the surface of 3D models. "UV" stands for texture mapping coordinates, which are similar to the X, Y, and Z axes of a spatial model. UV mapping accurately maps each point on the image to the surface of the 3D model, so that the texture can be correctly mapped to each part of the model and ensure that the texture presents the correct shape and details when rendered.

[0053] 5. Diffusion model: Among them, the diffusion model is a generative model based on probability theory. Its core idea is to simulate a gradual "diffusion" process from data distribution to simple noise distribution, and then reconstruct high-quality data samples from noise by learning the inverse process. This process can be divided into two main stages: 1. Forward process: gradually add noise to the data to generate a series of data samples that gradually become noise. Each step in this process is conditionally independent and gradually approaches the Gaussian noise distribution. 2. Reverse process: Starting from pure noise, gradually "denoising" through a series of inverse steps, and finally generating samples close to the original data distribution.

[0054] Among them, in the reverse process, the UNet network architecture can be used. The UNet network is an architecture specially designed for image segmentation tasks. In the image denoising task, the UNet network is used to learn to restore the original image from the noisy image. In addition, UNet is also a conditional denoising network, that is, it can not only remove the noise in the image, but also guide the denoising process according to the given label to generate an image that matches the label.

[0055] 6. Display perspective: The display perspective of the digital twin can refer to the angle at which the digital twin is displayed on the display page (such as front view, top view), or the distance at which the digital twin is displayed on the display page (such as an enlarged digital twin or a reduced digital twin).

[0056] 7. Display effect: The display effect of the digital twin can refer to the overall lighting intensity, color, surface gloss, roughness, environmental elements, etc. of the digital twin.

[0057] In the related art, the display of the digital twin is to transmit the 3D model to the terminal device, and the terminal device loads, renders, and displays it based on technologies such as Web Graphics Library (WebGL) technology or Open Graphics Library (OpenGL) technology. This display solution has certain requirements for the performance of the terminal device (especially the graphics card performance). However, the performance of different terminal devices often varies greatly, making the display solution of the digital twin need to ensure both the basic display requirements for the digital twin to be displayed on low-performance terminal devices (such as by reducing the accuracy of the digital twin) and the high-precision display requirements for the digital twin on high-performance terminal devices. These two situations are difficult to reconcile, resulting in limited usage scenarios for the digital twin and unable to be well promoted and popularized. In addition, for the same digital twin, when the terminal device used for display changes, it is also necessary to re-model the digital twin to adjust the accuracy of the digital twin so that the digital twin can adapt to the new terminal device.

[0058] In view of this, the embodiments of the present application provide a display method for a digital twin. This method is based on two-dimensional image loading and display, replacing the rendering and display of the 3D model, thereby effectively reducing the performance requirements for the terminal device and being able to display the high-precision digital twin on a terminal device with lower performance without updating the digital twin.

[0059] In one implementation, the terminal device displays a two-dimensional image on the display page of the digital twin. The two-dimensional image is used to display the display effect and / or display perspective of the digital twin; wherein, the second image displayed in the current frame is predicted based on the first image displayed in the previous frame and the prediction prompt information; the prediction prompt information is used to indicate the change in the display effect and / or display perspective of the second image relative to the first image. By switching between the two-dimensional first image and the second image, the change in the display effect and / or display perspective of the 3D model during the 3D model display process is simulated. Compared with the traditional method of directly rendering the 3D model, it does not require the terminal device to perform loading or rendering operations on the 3D model, thereby effectively reducing the performance requirements for the terminal device and being able to display the high-precision digital twin on a terminal device with lower performance, which helps to promote and popularize the digital twin.

[0060] In one implementation, the terminal device can implement obtaining a second image based on a first image prediction and prediction prompt information based on a first model. The first model is used to predict the display effect of the digital twin and / or the change in the display perspective in a two-dimensional image. In the embodiments of the present application, the first model can also be referred to as a digital twin engine.

[0061] Figure 1 FIG. shows a schematic diagram of an application scenario provided by the embodiments of the present application. As Figure 1 shown, it includes a terminal device 100 and a server 110. The embodiments of the present application can be executed by the terminal device 100.

[0062] Among them, the terminal device 100 can interact with the server 110 through a communication network. The communication network can be an Ethernet, a radio access network, a wireless local area network (WLAN), etc.

[0063] Among them, the terminal device 100 is a device with an interface display function.

[0064] Optionally, the terminal device 100 can be a personal computer, a smart phone, a tablet computer, an e-book reader, a portable computer, or other devices.

[0065] The digital twin system is deployed in the server 110. The digital twin system is used to construct a digital twin, provide the first model for the terminal device 100, and interact with the terminal device 100 to display the digital twin in the form of a two-dimensional image on the terminal device 100. For example, the terminal device 100 can access the digital twin system through a browser to display the display page of the digital twin. Or, the client of the digital twin system is deployed on the terminal device 100, and the terminal device 100 runs the client to display the display page of the digital twin.

[0066] Among them, from the perspective of form, the server can be a blade server, a high-density server, a rack server, or an all-in-one cabinet server; from the perspective of function, the server can be a general server, a graphics processing unit (GPU) server, an artificial intelligence (AI) server, etc.

[0067] In the embodiments of the present application, the server 110 is used to provide a first model for the terminal device 100 to support the terminal device 110 in executing the digital twin display method provided in the embodiments of the present application. One implementation is that the terminal device 100 obtains the first model from the server 110, and the terminal device runs the first model to implement the prediction of the second image based on the first image. In another implementation, the server 110 runs the first model, and the terminal device 100 sends a request to the server 110 through communication interaction with the server 110 and receives the second image returned by the server 110.

[0068] It should be noted that the server used to train the above first model may be the above server 110 or other servers. The embodiments of the present application do not make specific restrictions on this.

[0069] Figure 2 It is a schematic diagram of the hardware structure of a server (which may be the above server 110) provided in the embodiments of the present application. As Figure 2 shown, the server may include: a processor and a memory; the memory stores the program code of the above digital twin system; the processor is configured to run the digital twin system so that the server implements the method described in the corresponding Figure 1 embodiment.

[0070] Figure 3 It is a schematic diagram of the composition of a digital twin system provided in the embodiments of the present application. As Figure 3 shown, it includes a migration module, an agent module, and a construction module.

[0071] Among them, the migration module is mainly responsible for the new process of the digital twin. Specifically, when the user logs in to the digital twin system through the terminal device and creates a new digital twin, the migration module is responsible for displaying multiple preset digital twin templates and corresponding UV maps for the user (where the UV map is an image obtained by two-dimensional unfolding of the surface of the 3D model and is used to reflect the texture information of the 3D model surface, etc.). Among the multiple templates mentioned here, different templates correspond to different types of physical devices, such as the digital twin template corresponding to the server and the digital twin template corresponding to the switch. In addition, since the appearance of physical devices of the same type but different models may also vary, different templates can also correspond to different models of physical devices, such as the digital twin template corresponding to model A server and the digital twin template corresponding to model B server, etc.

[0072] The user can select the most suitable one from the templates of multiple digital twins (for example, if the user creates a digital twin of a server according to the current requirements, then select the template corresponding to the server). Then, through operations such as the user's text description and brush strokes on the UV map, the migration (transformation) of the display style of the UV map is realized. The user can repeatedly view and modify the display style until the requirements are met. Finally, the migration module generates the stylized digital twin desired by the user according to the new UV map, completing the addition / creation of the digital twin. The specific process of the display style transformation can be seen in the following Figure 9 description of the corresponding embodiment, which will not be elaborated in detail here.

[0073] Among them, the proxy module is mainly used to generate a training data set and send the training data set to the construction module. Specifically, the proxy module can be regarded as an agent, which can simulate the switching operation of the user and perform switching operations on the digital twin (display perspective switching and / or display effect switching). In this process, the proxy module can record the two-dimensional image before the change of the digital twin as the first training image, the operation information of the switching operation as the prediction prompt information, and the two-dimensional image after the change of the digital twin as the second training image. By repeating the above process, a training data set can be obtained. The specific training process can be seen in the following Figure 10 description of the corresponding embodiment, which will not be elaborated in detail here.

[0074] Among them, the construction module is mainly used to train the first model based on the training data set. The first model is used to predict the change of the display perspective and / or display effect of the digital twin in the two-dimensional image. The first model can be deployed on the terminal device or the server for inference use.

[0075] For the convenience of understanding, the following provides an exemplary introduction to the digital twin display method provided by the embodiments of the present application in conjunction with the accompanying drawings.

[0076] Figure 4 The flowchart of the digital twin display method provided by an exemplary embodiment of the present application is shown. The digital twin display method is executed by the terminal device and includes the following steps:

[0077] S101. In the display page of the digital twin, display a two-dimensional image.

[0078] Among them, the two-dimensional image is used to display the display effect and / or display perspective of the digital twin. Among them, the second image displayed in the current frame is predicted according to the first image displayed in the previous frame and the prediction prompt information. The prediction prompt information is used to indicate the change of the display effect and / or display perspective of the second image relative to the first image.

[0079] The display perspective mentioned here can refer to the angle at which the digital twin is displayed on the display page (such as the front view, top view), or the distance at which the digital twin is displayed on the display page (such as an enlarged digital twin or a reduced digital twin). The display effect mentioned here can refer to the overall illumination intensity, color, surface gloss, environmental elements, etc. of the digital twin.

[0080] As an example, Figure 5 FIG. is a schematic diagram showing a change in the display perspective provided by an embodiment of the present application. Figure 5 The first image is on the left side in the figure, and the second image is on the right side. Compared with the first image, the display perspective of the digital twin in the figure has changed.

[0081] As another example, Figure 6 FIG. is a schematic diagram showing a change in the display effect provided by an embodiment of the present application. Figure 6 The first image is on the left side in the figure, and the second image is on the right side. Compared with the first image, the display effect (the surface gloss of the digital twin) of the digital twin in the figure has changed.

[0082] In the embodiment of the present application, the user can access the digital twin system through a terminal device. After login verification, the digital twin system displays a two-dimensional image through the terminal device. The two-dimensional image can be a two-dimensional image of the digital twin displayed with a certain display effect and / or display perspective by default settings of the digital twin system when the user accesses the digital twin system. Or, the two-dimensional image can be a two-dimensional image of the digital twin displayed with a certain display effect and / or display perspective after the digital twin changes according to the user's operations during the process of the user operating the digital twin.

[0083] The digital twin mentioned here can be newly constructed when the user first accesses the digital twin system, or can be pre-configured and saved by the user.

[0084] In a possible implementation manner, the above-mentioned display of the two-dimensional image, such as Figure 7 shown, can be specifically implemented as:

[0085] S1011. When the first image is displayed, in response to meeting the switching condition, obtain the second image.

[0086] Among them, the prediction prompt information is determined based on the switching condition.

[0087] In some scenarios, since the user can rotate or scale a 3D model during the process of viewing the 3D model to understand the 3D model from multiple perspectives. To simulate this function, the display method of the digital twin provided by the embodiments of the present application can predict a second image based on user operations. Therefore, in some embodiments, meeting the switching condition may include: receiving a switching operation from the user. The switching operation is used to indicate switching the rendering effect of the digital twin. The predicted prompt information includes the operation information of the switching operation.

[0088] Exemplarily, when the terminal device displays the first image, the user can perform a swipe or click on the terminal device through virtual reality (VR) operations, keyboard operations, mouse operations, or touch screen operations, etc., to generate a switching operation.

[0089] For example, when the user rotates through VR or swipes the mouse to generate a switching operation for rotating the digital twin, the predicted prompt information corresponding to the switching operation may include information such as the direction of rotation and the angle of rotation. When the user clicks on a certain control in the page (multiple controls can be displayed in the digital twin page, and different controls correspond to different display effects of the data twin), or inputs information through the keyboard to change the display effect of the digital twin, the corresponding predicted prompt information may include the modified display effect, such as what the color is and what the transparency is, etc.

[0090] In some other embodiments, meeting the switching condition may further include: the display time of the first image exceeds a certain threshold, and the stereoscopic rendering effect of the digital twin in the first image conforms to a preset change rule (the preset change rule is, for example, the color changes in a preset order, the display perspective changes in a preset order, etc.). The predicted prompt information includes the prompt information generated according to the preset change rule.

[0091] Exemplarily, taking the preset change rule that the color of the digital twin changes in the order of left view - front view - right view as an example, if the display perspective of the digital twin in the current first image is the left view and the display time of the first image reaches the threshold, then according to the preset change rule, it is determined that the next display perspective of the digital twin is the front view. Therefore, "the display perspective is the front view" can be used as the predicted prompt information to guide the generation of the second image.

[0092] Another example is that taking the preset rule that the color of the digital twin changes in the order of red - green - blue as an example, if the color of the digital twin in the current first image is red and the display time of the first image reaches the threshold, then according to the preset change rule, it is determined that the next color to be adopted by the digital twin is green. Therefore, "the color is set to green" can be used as the predicted prompt information to guide the generation of the second image.

[0093] S1012. Update the first image in the display page to the second image.

[0094] After the terminal device obtains the second image, it replaces the first image in the display page with the second image and repeats the above process, that is, takes the second image as the new first image and obtains a new second image. Moreover, the switching speed between the first image and the second image can be at the frame-to-frame switching speed, so as to achieve smooth switching of the images, produce an animation effect, and further achieve the effect of simulating the change of the 3D model, ensuring the immersive experience of the user.

[0095] It should be noted that in the related art, when the terminal device displays the digital twin, it obtains the model rendering data of the digital twin, and then the terminal device renders the model rendering data and displays the digital twin on the screen. This display method has relatively high requirements for the performance of the terminal device (especially the graphics card performance). In the embodiment of the present application, when displaying the digital twin, a two-dimensional image containing the digital twin is obtained, and then the digital twin is displayed on the screen in the form of displaying the two-dimensional image, thereby effectively reducing the performance requirements for the terminal device.

[0096] It should be noted that in the embodiment of the present application, the next frame image of the first image is predicted based on the first model to obtain the second image. Before the terminal device displays the second image, it also needs to obtain the second image. The following describes two specific implementation methods for obtaining the second image:

[0097] In a possible implementation manner, the first model is deployed in the terminal device, and the first model is used to predict the display effect and / or the change of the display perspective of the digital twin in the two-dimensional image. The above S1011 can be specifically implemented as: inputting the first image and the prediction prompt information into the first model to obtain the second image.

[0098] In an implementation manner, the terminal device can obtain the first model from the server. That is, the method provided in the embodiment of the present application further includes: Step a. Obtain the performance parameters of the terminal device. The performance parameters can be, for example, the video memory size of the graphics card, the computing power of the processor, etc. Step b. When the performance parameters meet the model operation requirements, obtain the first model from the server. Here, the server is the server storing the trained first model. The model operation requirements can be determined by the server based on factors such as the size of the first model and the operation environment requirements. For example, by testing the video memory and computing power occupied when the first model runs at the maximum level, the model operation requirements are determined. The terminal device can obtain the model operation requirements from the server in advance and use them when needed.

[0099] It can be understood that the first model is deployed on the terminal device side and run by the terminal device, which can save the process of transmitting the two-dimensional image through the network. The terminal device can obtain and display the two-dimensional image more efficiently, ensure smooth switching between images, realize the change of the simulated 3D model, and provide users with a more immersive experience. In addition, the performance requirements for running the first model are much lower than those for traditional 3D model rendering, thus reducing the performance requirements for the terminal device during the display of the digital twin.

[0100] In another possible implementation, the first model is deployed in the server. As Figure 8 shown, the above S1011 can be specifically implemented as:

[0101] S1011 a. The terminal device sends a first request to the server. Correspondingly, the server receives the first request from the terminal device.

[0102] Among them, the first request is used to request to obtain a second image.

[0103] S1011 b. The server predicts the second image based on the first image and the prediction prompt information in response to the first request.

[0104] In one implementation, the first request may include the first image and the prediction prompt information. The first model is deployed on the server, and the first model is used to predict the display effect and / or the change of the display perspective of the digital twin in the two-dimensional image. After receiving the first request, the server can parse the first image and the prediction prompt information from the first request and input them into the first model to obtain the predicted second image.

[0105] In another implementation, the digital twin can run on the server. After receiving the first request, the server can adjust the digital twin according to the prediction prompt information (such as changing the display perspective, changing the display effect, etc.) to obtain the adjusted digital twin, and then the server takes a screenshot of the display page where the adjusted digital twin is located to obtain the second image.

[0106] S1011c. The server sends the second image to the terminal device. Correspondingly, the terminal device receives the second image returned by the server.

[0107] That is to say, after predicting the second image, the server can send it to the terminal device through the network. This is equivalent to transferring the pressure of running the first model to the server, further reducing the performance requirements for the terminal device.

[0108] The process of newly constructing a digital twin will be described below. During the user's access to the digital twin system, the digital twin can create a new 3D scene and provide the user with templates of multiple digital twins. The user can select a most suitable template according to their own needs (such as constructing a digital twin of a server according to user requirements) and add it to the 3D scene, and then modify its display style, repeatedly view the modification until the requirements are met, and generate the final digital twin. Then the terminal device displays the digital twin in the form of a two-dimensional image. As Figure 9 shown, the process of modifying the display style of the digital twin includes S201 - S203:

[0109] S201. Display the first UV map of the digital twin.

[0110] Among them, the first UV map is used to display the surface details of the digital twin.

[0111] It should be noted that the digital twin system can perform two-dimensional unfolding on the digital twin to obtain the corresponding UV map. The specific two-dimensional unfolding can be implemented using 3D model tools, and the embodiments of the present application do not make specific limitations on this. After the user selects a digital twin, the digital twin system can determine the first UV map corresponding to the digital twin and display it to the user through the terminal device.

[0112] S202. In response to the adjustment operation on the first UV map, send a second request to the server.

[0113] Among them, the second request is used to request to obtain the second UV map. The adjustment operation is used to indicate the change in the surface details of the digital twin with respect to the first UV map for the second UV map.

[0114] The adjustment operation here can include forms such as text description and editing operations on the first UV map (such as smearing a part of the first UV map to require that the details of this part are not displayed). The server can perform operations on the first UV map according to the information corresponding to the adjustment operation to obtain the second UV map that meets the user's requirements.

[0115] In some embodiments, a second model is deployed on the server. The second model is used to generate the second UV map according to the first UV map and the adjustment operation. The second request may include the first UV map and the adjustment information of the adjustment operation. After receiving the second request, the server can input the first UV map and the adjustment information of the adjustment operation into the second model, obtain the output second UV map, and send it to the terminal device.

[0116] S203. In response to the confirmation operation on the second UV map, send a third request to the server. The third request is used to request the server for a two-dimensional image of the digital twin constructed based on the second UV map.

[0117] The above S201 - S202 can be repeatedly executed, that is, the user can repeatedly adjust the UV map until the user's requirements are met. The terminal device can receive the user's confirmation operation and send a third request to the server. The server can generate a digital twin that meets the user's requirements according to the second UV map. Specifically, the process of generating a digital twin from the UV map can be implemented using 3D model tools, and the embodiments of the present application do not make specific limitations on this.

[0118] It should be noted that in the embodiments of the present application, the user's personalized requirements for the digital twin are met through the user's adjustment operation on the UV map. By generating the model instead of the user directly adjusting the 3D model, the construction efficiency of the digital twin can be effectively improved, and the user's usage threshold can be reduced.

[0119] The training process of the first model will be described below.

[0120] Figure 10 The flowchart of the training method provided by an exemplary embodiment of the present application is shown. This training method can be executed by the server and includes the following steps:

[0121] S301. Obtain a training data set.

[0122] Among them, each pair of training data in the training data set includes: a first training image before the switching operation, prediction hint information corresponding to the switching operation, and a second training image obtained by operating on the first training image based on the switching operation; the first training image is an image of the digital twin displayed according to the first display effect and / or the first display perspective, and the second training image is an image of the digital twin displayed according to the second display effect and / or the second display perspective.

[0123] The training data set can be obtained from the network or other places in advance, or can be generated by the server itself. That is, before S301, the following is also executed:

[0124] S300. During the process of running the digital twin, simulate the switching operation on the digital twin to obtain a training data set.

[0125] That is to say, the server can run a digital twin and set up an agent that can simulate the switching operations of users to switch the digital twin (display perspective switching and / or presentation effect switching). In this process, the server can record the two-dimensional image of the digital twin before the change as the first training image, the operation information of the switching operation as the prediction prompt information, and the two-dimensional image of the digital twin after the change as the second training image. Repeating the above process multiple times can obtain multiple sets of training data on the changes between different display perspectives and the changes between different presentation effects, so as to generate a training dataset.

[0126] It can be seen that by means of the server simulating user operations, a large number of training datasets can be automatically generated without excessive manual participation, effectively reducing the consumption of human resources.

[0127] It should be noted that the above S300 can be implemented by the Figure 3 preceding proxy module.

[0128] S302: Use the first training image and the prediction prompt information as sample data, and the second training image as the label data of the sample data to train the initial model to obtain the first model.

[0129] Among them, the first model is used to predict the presentation effect and / or the change of the display perspective of the digital twin in the two-dimensional image. The initial model refers to a basic model architecture selected before the start of training (which can be called the starting point of training). The parameters in this initial model are all preset initial values, and various parameters will be continuously adjusted during the training process until a model that can meet the user's needs is finally obtained.

[0130] During the training process, the first training image and the prediction prompt information in the training dataset are used as sample data and input into the initial model to obtain the output result, which is compared with the second training image used as the label data to determine the loss value, and then the model parameters of the initial model are adjusted based on algorithms such as the gradient descent algorithm or the backpropagation algorithm. Repeat the above process until the obtained loss value is the smallest, or the number of repetitions reaches a certain threshold, and the training is completed to obtain the first model.

[0131] It should be noted that the above S302 can be responsible for the Figure 3 preceding construction module.

[0132] In a possible implementation, the initial model is a diffusion model. Optionally, the initial model can also be other models for image prediction, such as a convolutional neural network model, etc.

[0133] As described above, the principle of the diffusion model is to gradually add noise to the image during the forward process, and then denoise and restore the clear image during the reverse process, so as to generate a new image relatively close to the original image. In the embodiments of the present application, the diffusion model is selected to train the first model. On the one hand, by adding noise to the image, the model can have a certain anti-noise ability during the training process, improving the robustness of the model. On the other hand, compared with traditional first models such as convolutional neural networks, the diffusion model has a smaller scale and a faster inference speed, and can be effectively applied to terminal devices to achieve image prediction. On the other hand, the diffusion model supports guiding the generation of images through labels. In the embodiments of the present application, by using the prediction prompt information as a label to train the first model, the first model is obtained to have the function of generating corresponding images according to the prediction, so that users can experience the process of simulating the operation of a 3D model during the viewing of the first image, with a more immersive experience.

[0134] The training process of the second model will be described below. It includes the following: 1. Obtain the second training dataset. Each pair of training data in the second training dataset includes: an initial UV map, adjustment information, and a target UV map obtained by adjusting the initial UV map according to the adjustment information.

[0135] 2. Use the initial UV map and the prompt information as sample data, and use the target UV map as the label data of the sample data to train the initial model to obtain the second model. The second model is used to adjust the surface details of the UV map according to the adjustment information.

[0136] Among them, the adjustment information can be the modification opinions of project personnel on the UV map in historical projects, which can be text descriptions, or the coordinate positions of editing operations on the UV map, etc. During the training process, the initial UV map and the adjustment information are used as sample data and input into the initial model to obtain the output result, which is compared with the target UV map as the label data to determine the loss value, and then the model parameters of the initial model are adjusted based on algorithms such as gradient descent algorithm or backpropagation algorithm. Repeat the above process until the obtained loss value is the smallest, or the number of repetitions reaches a certain threshold, and the training is completed to obtain the second model.

[0137] As an example, the initial model used to train the second model can also be an extended model.

[0138] Figure 11 It is a schematic diagram of the new process of a digital twin body provided by the embodiments of the present application. As Figure 11As shown, first, the user logs in to the system to create a new digital twin, and multiple preset digital twin templates and corresponding UV maps are displayed. The user can select the most suitable one from multiple templates. Then, through operations such as the user's text description and brush strokes on the UV map, the migration (transformation) of the display style of the UV map is achieved. The user can repeatedly view and modify until the requirements are met. Finally, the migration module generates the stylized digital twin desired by the user based on the new UV map, completing the addition / creation of the digital twin.

[0139] Figure 12 This is a schematic diagram of the training process of a first model provided by an embodiment of the present application. As Figure 12 shown, first, a training data set is obtained. During the training process, the first training image and the prediction prompt information in the training data set are used as sample data to be input into the initial model, and the output result is obtained. The loss value is determined by comparing it with the second training image used as the label data. Then, the model parameters of the initial model are adjusted based on algorithms such as the gradient descent algorithm or the backpropagation algorithm. The above process is repeatedly executed until the obtained loss value is minimized, and the training is completed to obtain the first model. The trained first model can generate a second image based on the first image and the prediction prompt information to achieve seamless switching of the screen scene, enabling the user to perceive that they are exploring in a 3D space.

[0140] Figure 13 This is a schematic diagram of an inference process provided by an embodiment of the present application. As Figure 13 shown, first, the user accesses the digital twin system, and the digital twin system loads the digital twin and displays the first image through the terminal device. Then, the user performs a switching operation to generate prediction prompt information. The digital twin system predicts the second image of the next frame in real time based on the first image and the user's operation. By switching between the images to display the digital twin, there is not much difference from the process of the user directly viewing the 3D model, and the user can experience the process of observing the digital twin from different angles without perception.

[0141] The digital twin display method provided by the embodiment of the present application simulates the display effect and / or the change of the display perspective during the 3D model display process by switching between the two-dimensional first image and the second image. Compared with the traditional method of directly rendering the 3D model, it does not require the terminal device to load or render the 3D model, thus effectively reducing the performance requirements for the terminal device. It can also display a high-precision digital twin on a terminal device with low performance without updating the digital twin, which helps to promote and popularize the digital twin.

[0142] The above mainly introduced the solution of the embodiment of the present application from the perspective of the method. It can be understood that in order for the page display device of the digital twin system to implement the above functions, it includes at least one of the corresponding hardware structures and software modules for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this article, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0143] The embodiments of the present application can divide the functional units of the display device of the digital twin body according to the above method examples. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0144] Exemplarily, Figure 14 FIG. shows a schematic structural diagram of a display device of a digital twin body provided by an exemplary embodiment of the embodiments of the present application. The display device of the digital twin body includes: a display module 1401.

[0145] The display module 1401 is configured to display a two-dimensional image in the display page of the digital twin body; the two-dimensional image is used to display the display effect and / or display perspective of the digital twin body; wherein, the second image displayed in the current frame is predicted based on the first image displayed in the previous frame and prediction hint information; the prediction hint information is used to indicate the change in the display effect and / or display perspective of the second image relative to the first image.

[0146] For the specific description of the above optional manner, reference can be made to the foregoing method embodiments, which will not be elaborated here. In addition, the explanations and descriptions of the beneficial effects of any of the above-provided display devices of the digital twin body can refer to the corresponding method embodiments above, and will not be elaborated.

[0147] Exemplarily, Figure 15 FIG. shows a schematic structural diagram of a training device provided by an exemplary embodiment of the embodiments of the present application. The training device includes: an acquisition module 1501 and a training module 1502.

[0148] The acquisition module 1501 is configured to acquire a training data set; each pair of training data in the training data set includes: a first training image before a switching operation, prediction prompt information corresponding to the switching operation, and a second training image obtained by operating on the first training image based on the switching operation; the first training image is an image displayed by the digital twin according to a first display effect and / or a first display perspective, and the second training image is an image displayed by the digital twin according to a second display effect and / or a second display perspective;

[0149] The training module 1502 is configured to use the first training image and the prediction prompt information as sample data, and the second training image as label data of the sample data to train an initial model to obtain a first model; the first model is used to predict changes in the display effect and / or display perspective of the digital twin in a two-dimensional image.

[0150] For the specific description of the above optional methods, reference may be made to the foregoing method embodiments, which will not be elaborated here. In addition, the explanations and descriptions of the beneficial effects of any of the above-provided training devices may refer to the corresponding method embodiments above and will not be elaborated.

[0151] In an exemplary embodiment, a computer-readable storage medium is further provided, which is used to store at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement all or part of the steps in the above memory fault prediction method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0152] In an exemplary embodiment, a computer program product or a computer program is further provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes all or part of the steps of the method shown in any of the above embodiments.

[0153] In some embodiments, the method shown in the embodiments of the present application may be implemented as computer program instructions encoded in a computer-readable storage medium in a machine-readable format or encoded on other non-transitory media or articles.

[0154] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed. For example, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0155] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0156] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units. For example, they can be located in one place, or they can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, in each embodiment of the embodiments of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0158] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks or optical disks, and other media that can store program codes.

[0159] The foregoing are only alternative embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the embodiments of the present application shall be included within the protection scope of the embodiments of the present application.

Claims

1. A display method for a digital twin, characterized in that, Applied to a terminal device, the method includes: In a display page of the digital twin, a two-dimensional image is displayed; the two-dimensional image is used to show the display effect and / or display perspective of the digital twin; wherein, the second image displayed in the current frame is predicted based on the first image displayed in the previous frame and prediction prompt information; the prediction prompt information is used to indicate the change in the display effect and / or the display perspective of the second image relative to the first image.

2. The display method according to claim 1, wherein The displaying of the two-dimensional image of the digital twin includes: When the first image is being displayed, in response to meeting a switching condition, obtain the second image; the prediction prompt information is determined based on the switching condition; Update the first image in the display page to the second image.

3. The display method according to claim 2, characterized in that, Meeting the switching condition includes: receiving a switching operation from the user; the switching operation is used to indicate switching the rendering effect of the digital twin; the prediction prompt information includes the operation information of the switching operation.

4. The display method according to claim 2, wherein The obtaining of the second image includes: Input the first image into a first model to obtain the second image; the first model is used to predict the change in the display effect and / or display perspective of the digital twin in the two-dimensional image.

5. The display method according to claim 4, wherein Before inputting the first image into the first model, the method further includes: Obtain the performance parameters of the terminal device; When the performance parameters meet the model operation requirements, obtain the first model from the server.

6. The display method according to claim 2, wherein The obtaining of the second image includes: Send a first request to the server; the first request is used to request to obtain the second image; Receive the second image sent by the server.

7. The display method according to claim 1, wherein Before displaying the two-dimensional image, the method further includes: Display the first UV map of the digital twin; the first UV map is used to show the surface details of the digital twin; In response to an adjustment operation on the first UV map, send a second request to the server; the second request is used to request to obtain a second UV map; the adjustment operation is used to indicate the change in the surface details of the digital twin in the second UV map relative to the first UV map; In response to a confirmation operation on the second UV map, send a third request to the server; the third request is used to request the digital twin constructed by the server based on the second UV map.

8. The display method according to claim 7, wherein The server deploys a second model; the second model is used to generate the second UV map according to the first UV map and the adjustment operation.

9. A training method, characterized in that, The method includes: Obtain a training data set; each pair of training data in the training data set includes: a first training image before a switching operation, prediction prompt information corresponding to the switching operation, and a second training image obtained by operating on the first training image based on the switching operation; the first training image is an image of the digital twin displayed in a first display effect and / or a first display perspective, and the second training image is an image of the digital twin displayed in a second display effect and / or a second display perspective; Using the first training image and the prediction prompt information as sample data, and the second training image as the label data of the sample data, train an initial model to obtain a first model; the first model is used to predict the display effect and / or the change of the display perspective of the digital twin in the two-dimensional image.

10. A computing device, characterized in that, The computing device includes a processor and a memory; the processor is coupled to the memory; The memory is used to store computer instructions; The computer instructions are loaded and executed by the processor to enable the computing device to implement the method according to any one of claims 1-9.

Images