Method and device for realizing twin multi-modal characterization

Abstract

The invention relates to a twin multi-modal representation implementation method and device. The method comprises the steps that model nodes of an entity model are improved to be twin objects; generating corresponding rendering data according to the business requirements; driving the rendering data access service data; and constructing display logic for rendering the model nodes according to business requirements. According to the method, the technical problem of how to realize efficient fusion display of multi-modal information of the digital twin in different business scenes is solved, and one-twin multi-mode intelligent visual innovation is realized.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for implementing multimodal representation of twin bodies. Background Art

[0002] In actual business scenarios, the same twin may have different modal representation requirements due to different business needs. For example, in the energy storage station scenario, the same energy storage cabinet needs to be presented as a temperature field in the temperature management business, as a physical model in the asset management business, and as an energy flow in the power management business.

[0003] At present, the mainstream approach to the multimodal representation of twins is to keep the physical model and process it by changing the model texture and color according to the business, or by adding a top card. Although the above processing method can achieve multimodal expression of twins to a certain extent, there are still major deficiencies. First, by modifying the model texture and color, it is impossible to effectively display business information; and although adding a top card and attaching text to the top card can display business data, it cannot significantly express the particularity of twins in different modes; secondly, whether it is modifying the texture color or adding a top card, it will destroy the unity of the scene style. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a method and device for realizing multimodal representation of twins in view of the deficiencies in the prior art, so as to realize intelligent visualization display of multiple different business scenarios.

[0005] The technical solution of this application to solve the above technical problems is as follows:

[0006] In a first aspect, a method for implementing a twin multimodal representation is provided, comprising:

[0007] Promote the model nodes of the entity model to twin objects;

[0008] Generate corresponding rendering data according to business needs;

[0009] By accessing the rendering data to the business data for driving;

[0010] Construct display logic for rendering the model nodes according to business requirements.

[0011] Optionally, the model node of the promoted entity model is a twin object, including:

[0012] Create a child twin object, and set the parent object of the child twin object to the twin object;

[0013] Find the corresponding mesh information in the twin object according to the model node name of the entity model; Split the mesh information from the twin object as the rendering node of the sub-twin object;

[0014] Sort out the remaining part of the twin object after removing the mesh information and use it as the rendering node of the twin object.

[0015] Optionally, the generating corresponding rendering data according to business requirements includes:

[0016] Create the rendering data of the twin object and the rendering data of the sub-twin object respectively according to business requirements;

[0017] Establish a root node for each rendering data, and use other data as the child nodes of this node to obtain the rendering trees of the twin object and the sub-twin object for this business requirement;

[0018] Mount the rendering tree.

[0019] Optionally, the mounting the rendering tree includes:

[0020] Create a rendering root node for the twin object and the sub-twin object respectively;

[0021] Mount the root nodes of the rendering trees of the twin object and the sub-twin object as child nodes to their respective rendering root nodes and name them;

[0022] Mount the rendering data of the twin object and the rendering data of the sub-twin object to the rendering root node and name them.

[0023] Optionally, the accessing the rendering data to business data driving includes:

[0024] When the change of business data causes a simple transformation of the data, update the corresponding matrix information of the child nodes;

[0025] When the change of business data causes a non-simple transformation of the rendering data, regenerate the rendering tree to replace the mounted nodes.

[0026] Optionally, the simple transformation includes: translation, rotation, scaling.

[0027] Optionally, the constructing the display logic for rendering the model node according to business requirements includes:

[0028] For the current modality, set the visibility of the nodes under the rendering root node;

[0029] Before the twin object switches levels, record the current modality of the twin object, and after the level switching is completed, update according to the modality information.

[0030] In a second aspect, an implementation device for twin multi-modal representation is provided, including:

[0031] A model node promotion module, configured to promote the model nodes of the entity model to twin objects;

[0032] A rendering data generation module, configured to generate corresponding rendering data according to service requirements;

[0033] A service data driving module, configured to drive by accessing service data to the rendering data;

[0034] A display logic processing module, configured to construct a display logic for rendering the model nodes according to service requirements.

[0035] In a third aspect, a computer storage medium is proposed, on which a computer program is stored. When the program is executed by a processor, the steps of the implementation method of twin multi-modal representation described in any one of the first aspects are implemented.

[0036] In a fourth aspect, a computer program product is provided, including a computer program or instruction. When the computer program or instruction is executed by a processor, the steps in the implementation method of twin multi-modal representation described in the first aspect are implemented.

[0037] Advantageous Effects

[0038] In this application, for different service requirements, based on the physical structure of twin objects, by dynamically constructing custom rendering data, adding it to the rendering layer, and constructing a display logic for node rendering according to service requirements, display management is performed based on service requirements, and finally, the multi-modal representation of twins is realized. It solves the technical problem of how to efficiently fuse and display multi-modal information of digital twins in different service scenarios, and realizes the intelligent visualization innovation of "one twin, multiple modes".

[0039] The advantages of the additional aspects of this application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of this application. Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions of the embodiments of this application, the drawings required to be used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the following described drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a schematic flowchart of the implementation method of twin multi-modal representation described in the embodiments of this application;

[0042] Figure 2Schematic diagram of an energy storage cabinet model example in an embodiment of the present application;

[0043] Figure 3 Schematic diagram of an example showing the power of the energy storage cabinet in an embodiment of the present application;

[0044] Figure 4 Principle block diagram of the implementation device for the twin multi-modal representation described in the embodiments of the present application. Detailed implementation manners

[0045] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0046] The implementation principle of the multi-modal representation of the twin is to create custom rendering data based on the physical structure of the twin according to business requirements, perform dynamic display in combination with business data, add it to the rendering layer, and rely on the display and hiding logic based on business requirements to perform unified management in combination with the entity model to achieve the multi-modal representation of the twin.

[0047] Embodiment 1

[0048] As Figure 1 shown, a method for implementing the multi-modal representation of a twin includes the following steps:

[0049] Step S1: Promote the model nodes of the entity model to twin objects;

[0050] In the physical world, twin objects often include several components. Therefore, although the entity model of a twin object is usually a complete model, the entity model is composed of different model nodes. Since special display and separate display and hiding are usually required for the components during rendering based on business logic, it is necessary to promote the model nodes of the entity model to twin objects.

[0051] The process of promoting the model nodes is as follows:

[0052] Step S1.1: Create a sub-twin object, and set the parent object of the sub-twin object to the twin object;

[0053] Specifically, create a twin object subObj; the attribute information, style information, and matrix information of subObj are all the same as those of the twin object; set the parent object of subObj to the twin object.

[0054] Step S1.2: Find the corresponding mesh information in the twin object according to the model node name of the entity model; Split the mesh information from the twin object as the rendering node of the sub-twin object.

[0055] Specifically, find the corresponding mesh information mesh in the twin object according to the model node name of the entity model; Split mesh from the object model as the rendering node of subObj.

[0056] Step S1.3: Organize the remaining part of the twin object after removing the mesh information and use it as the rendering node of the twin object.

[0057] Specifically, organize the remaining part of the twin object after removing mesh as the rendering node of the twin object.

[0058] As Figure 2 shown, the model data given by the energy storage cabinet object is a complete energy storage cabinet model, which includes several battery clusters. Due to specific business requirements, different battery clusters need to have different display performances, so the battery cluster object needs to be promoted to a sub-object of the energy storage cabinet object, and the rendering data of the battery clusters in the energy storage cabinet are respectively assigned to the corresponding sub-objects, and the remaining rendering data is left for the energy storage cabinet object.

[0059] Step S2: Generate corresponding rendering data according to business requirements;

[0060] Under different business requirements, the twin object is in different modes and has different forms of rendering requirements. Specifically, organize the data required for rendering under the current business requirements according to the business requirements.

[0061] The specific implementation process includes the following steps:

[0062] Step S2.1: Create the rendering data of the twin object and the rendering data of the sub-twin object respectively according to the business requirements;

[0063] Under the specified business requirements, the twin object and the promoted sub-twin object both have special display needs and have independent display logics with each other. Therefore, it is necessary to divide which generated rendering data belongs to the twin object or the sub-twin object.

[0064] Step S2.2: Create a root node for each rendering data, and use other data as the child nodes of this node to obtain the rendering trees of the twin object and the sub-twin object for this business requirement.

[0065] According to business requirements, render data of the twin object and render data of the sub-twin object are created respectively; it should be noted that due to the different complexities of business requirements, the render data that may be created may require multiple models, top cards, and text information to be jointly constructed. In this case, a root node needs to be established for each render data, and other data serves as the child nodes of this node. In this way, the final obtained is the render tree of the twin object and the sub-twin object for this business requirement.

[0066] Step S2.3: Mount the render tree;

[0067] Specifically, in the previous step, the twin object and the sub-twin object respectively obtained the corresponding render trees under the current requirements; in this step, the render trees need to be mounted well. Since the processing of the twin object and the sub-twin object is the same, the following will not distinguish them.

[0068] The mounting process is as follows:

[0069] Step S2.3.1: Create a render root node for the twin object and the sub-twin object respectively;

[0070] Specifically, create a render root node root for the object;

[0071] Step S2.3.2: Mount the root nodes of the render trees of the twin object and the sub-twin object as child nodes to their respective render root nodes and name them;

[0072] Specifically, mount the root node of the render tree as a child node to root and name it;

[0073] Step S2.3.3: Mount the render data of the twin object and the render data of the sub-twin object to the render root node and name them.

[0074] Specifically, after pointing the final render data of the twin object to be rendered to the remaining mesh in the twin object model excluding the enhanced part, mount it to root and name it.

[0075] Specific example illustration:

[0076] Such as Figure 3As shown, for the power display of the energy storage cabinet, it is necessary to display the transparent shell of the energy storage cabinet, the business data of the energy storage cabinet, and the battery cluster power box information. First, perform the division and rendering attribution. Among them, the transparent shell and business data belong to the display information of the energy storage cabinet body, and the battery cluster box belongs to the battery cluster display information. Then, draw the rendering tree. Taking the energy storage cabinet body as an example, first create a root node, and then create a transparent box object and a top sign object respectively, and add the two to the root node to obtain the rendering tree. Finally, mount the rendering tree. Create a rendering root node root on the energy storage cabinet object, and name and add the rendering data of the energy storage cabinet body and the rendering data of the rendering tree to root respectively.

[0077] Step S3: Drive by accessing the business data to the rendering data;

[0078] In a specified business scenario, the data of the twin object usually needs to be dynamically displayed. Therefore, the rendering data also needs to be driven based on the business data. However, since the rendering data has been added to the rendering layer, it is inconvenient to perform dynamic response. For this reason, we handle it in two cases. Specifically, it includes the following steps:

[0079] Step S3.1: When the rendering data undergoes a simple transformation after the business data changes, update the corresponding matrix information of the child nodes;

[0080] Under the rendering root node of the twin object, there is corresponding matrix information for different child nodes respectively, which can assist the child nodes in transformation; a simple transformation means only translation, rotation, and scaling occur.

[0081] Step S3.2: When the rendering data undergoes a non-simple transformation after the business data changes, regenerate the rendering tree to replace the mounted node.

[0082] After steps S1~S3, the rendering preparation work of the twin object based on the specified business requirements is completed. Further, we need to handle the display processing logic under specific business requirements.

[0083] Step S4: Construct the display logic for rendering the model node according to the business requirements. It includes:

[0084] Step S4.1: For the current modality, set the visibility of the nodes under the rendering root node;

[0085] Specifically, this step belongs to multi-modal switching. Since when the twin mounts the rendering tree, the rendering data nodes of various modalities under root are all named, when performing multi-modal switching, only for the current modality, it is necessary to set the visibility of the nodes under root, that is, only display the nodes of the current modality.

[0086] Step S4.2: Record the current modality of the twin object before the hierarchical switch, and update it according to the modality information after the hierarchical switch is completed.

[0087] Since the hierarchical switch will also set the visibility of the twin object, which will interfere with the multi-modal of the twin object, it is necessary to record the current modality of the twin object before the hierarchical switch and update it according to the modality information after the hierarchical switch is completed. At the same time, it should be noted that the above processing is completed in the same frame to avoid the flickering problem.

[0088] Through specific examples, it shows that this method dynamically creates twin rendering nodes according to the business logic and performs display management, so as to realize the intelligent visualization innovation of the multi-modal representation of twins.

[0089] Embodiment II

[0090] This application proposes an implementation device for the multi-modal representation of twins, as Figure 4 shown, including:

[0091] The model node promotion module is used to promote the model node of the entity model to a twin object;

[0092] The rendering data generation module is used to generate corresponding rendering data according to business requirements;

[0093] The business data driving module is used to drive by accessing business data to the rendering data;

[0094] The display logic processing module is used to construct the display logic for rendering the model node according to business requirements.

[0095] The implementation device for the multi-modal representation of twins provided in this embodiment is implemented by the implementation method for the multi-modal representation of twins provided in Embodiment I, and has the same technical features as the implementation method for the multi-modal representation of twins provided in Embodiment I, so it can also solve the same technical problems and achieve the same technical effects.

[0096] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0097] Embodiment III

[0098] This application proposes a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps of the implementation method for the multi-modal representation of twins in any one of Embodiments I.

[0099] It can be understood that the storage medium includes: flash memory, hard disk, multimedia card, card-type memory (such as SD (Secure Digital Memory Card), or DX (abbreviation for Memory Data Register, MDR, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, server, APP (abbreviation for Application, application software) application mall, and other various media that can store program verification codes.

[0100] Embodiment 4

[0101] The present application provides a computer program product, including a computer program or instruction, which, when executed by a processor, implements the steps in the method for realizing twin multi-modal representation described in the first aspect.

[0102] Based on such understanding, all or part of the processes in the methods of the above embodiments of the present application can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0103] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the protection scope of the present application.

[0104] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for realizing twin multi-modal representations, characterized in that, Including: Promote the model nodes of the entity model to twin objects; Generate corresponding rendering data according to business requirements; Drive by accessing business data to the rendering data; Construct the display logic for rendering the model nodes according to business requirements.

2. The implementation method of twin multi-modal representation according to claim 1, wherein The model nodes that promote the entity model to twin objects include: Create sub-twin objects, and set the parent object of the sub-twin objects as the twin object; According to the model node name of the entity model, find the corresponding mesh information in the twin object; Split the mesh information from the twin object as the rendering node of the sub-twin object; Sort out the remaining part of the twin object after removing the mesh information, and use it as the rendering node of the twin object.

3. The implementation method of twin multi-modal representation according to claim 1, wherein The generating corresponding rendering data according to business requirements includes: According to business requirements, create the rendering data of the twin object and the rendering data of the sub-twin object respectively; Establish a root node for each rendering data, and use other data as the sub-nodes of this node to obtain the rendering trees of the twin object and the sub-twin object for this business requirement; Mount the rendering tree.

4. The implementation method of the twin multi-modal representation according to claim 3, characterized in that The mounting the rendering tree includes: Create a rendering root node for the twin object and the sub-twin object respectively; Use the root nodes of the rendering trees of the twin object and the sub-twin object as sub-nodes and mount them to their respective rendering root nodes and name them; Mount the rendering data of the twin object and the rendering data of the sub-twin object to the rendering root node and name them.

5. The implementation method of the twin multi-modal representation according to claim 4, wherein The driving by accessing business data to the rendering data includes: When the business data changes and causes a simple transformation of the data, update the corresponding matrix information of the sub-node; When the business data changes and causes a non-simple transformation of the rendering data, regenerate the rendering tree to replace the mounted node.

6. The implementation method of the twin multi-modal representation according to claim 5, characterized in that, The simple transformation includes: translation, rotation, scaling.

7. The implementation method of the twin multi-modal representation according to claim 1, characterized in that The constructing the display logic for rendering the model nodes according to business requirements includes: For the current modality, set the visibility of the nodes under the rendering root node; Before the twin object switches levels, record the current modality of the twin object, and after the level switch is completed, update according to the modality information.

8. An implementation device for twin multi-modal representations, characterized in that, Including: A model node promotion module for promoting the model nodes of the entity model to twin objects; A rendering data generation module for generating corresponding rendering data according to business requirements; A business data driving module for driving by accessing business data to the rendering data; A display logic processing module for constructing the display logic for rendering the model nodes according to business requirements.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the implementation method of twin multi-modal representation as described in any one of claims 1-7.

10. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by the processor, it implements the steps in the implementation method of twin multi-modal representation as described in any one of claims 1-7.

Images