Multi-mode intelligent twin operation implementation method and computer program product

Abstract

The invention provides an operation implementation method of a multi-modal intelligent twin and a computer program product. Dynamic construction and flexible loading of twin instances are realized by loading a twin template for scene scanning; multi-modal sensing is performed based on motion data loaded by twin instances, a sensing result is fused to trigger an operation logic domain, so that the twin instances have an adaptive response capability, and then business behavior logic deduction corresponding to an operation state business flow is driven through a process engine in the operation logic domain, so that the operation state of the twin instances is realized. Intelligent operation of twin instances is realized; and finally, after the current deduction of the twin instance is finished, writing back an execution result and context data mapped by a running state of the twin instance to runtime data to form a self-updating running closed loop, thereby realizing the organic fusion of logic deduction and closed loop feedback of the twin instance. The intellectualization, self-evolution and dynamic adaptability of the twin instance and the scene to which the twin instance belongs are improved, and the multi-modal intelligent twin instance can be realized.

Description

Technical Field

[0001] This application relates to the field of digital twin technology, specifically to a method for implementing the operation of a multimodal intelligent twin and a computer program product. Background Technology

[0002] With the development of technologies such as artificial intelligence, Internet of Things (IoT), and digital twins, physical entities in the real physical space, such as various devices and facilities, are gradually establishing corresponding digital twins, thereby enabling operation in various scenarios.

[0003] By collecting real-time data on physical entities and using digital twins as carriers to replicate them, the behavior of physical entities in physical space can be reproduced in the scene, thus realizing the mapping of physical space and being widely applied in many industries.

[0004] Existing digital twins are implemented using a static approach, where all twin definitions, behaviors, and business rules are hardcoded. This means that adding a new digital twin requires code modification, adjusting any business rule requires recompiling, and changing the algorithm supporting each behavior requires restarting and redeploying. This is unacceptable for scenarios and systems where digital twins deployed in those scenarios are constantly changing.

[0005] In addition, since the operation of existing digital twins depends on the hard-coded behaviors and business rules, the operation of digital twins will be limited to mechanical execution in order to describe the physical state, but will not have the ability to operate autonomously.

[0006] In summary, existing digital twins are limited by static hard-coded implementations and lack scalability and autonomous operation capabilities. Summary of the Invention

[0007] One objective of this application is to eliminate the need for statically hardcoding twins, thereby enabling twin implementations to have scalability and autonomous operation capabilities.

[0008] According to one aspect of the embodiments of this application, a method for implementing the operation of a multimodal intelligent twin is disclosed, the method comprising: By scanning and loading a twin template for a scene, a twin instance of the scene is created. The twin template is used to describe the business behavior logic and definition information of a twin of a certain type. The runtime data loaded by the twin instance is used to trigger the multimodal awareness runtime logic domain of the twin instance. The triggering of the runtime logic domain includes the runtime business flow triggered by the process engine based on multimodal awareness. Triggered by the running logic domain, the business behavior logic configured in the running business flow is deduced, and the deduction is executed by the process engine by calling the corresponding function and / or plugin for the twin instance; After the current inference of the twin instance is completed, the obtained execution result and the context data mapped by its own running state are updated back to the runtime data loaded by the twin instance. The update of the runtime data drives the formation of the running state closed loop of the twin instance, and the twin instance that drives the formation of the running state closed loop constitutes a multimodal intelligent twin instance.

[0009] According to one aspect of the embodiments of this application, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0010] According to one aspect of the embodiments of this application, a computer program product is disclosed, including a computer program that, when executed by a processor, implements the steps of the method as described above.

[0011] According to one aspect of the embodiments of this application, a computer-readable storage medium is disclosed having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described above.

[0012] This application embodiment achieves dynamic construction and flexible loading of twin instances by loading twin templates for scene scanning; multimodal perception is performed based on the motion data loaded by the twin instance, and the perception results are fused to trigger the running logic domain, enabling the twin instance to have adaptive response capabilities. Then, in the running logic domain, the process engine drives the business behavior logic deduction corresponding to the running business flow, realizing the intelligent operation of the twin instance; finally, after the current deduction of the twin instance is completed, the execution result and the context data mapped by its own running state are written back to the running data, forming a self-updating running closed loop, thereby realizing the organic integration of twin instance logic deduction and closed loop feedback, improving the intelligence, self-evolution and dynamic adaptability of the twin instance and its scene, and enabling multimodal intelligent twin instances.

[0013] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0014] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0015] The above and other objectives, features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0016] Figure 1 This is a flowchart illustrating a method for implementing a multimodal intelligent twin according to an exemplary embodiment.

[0017] Figure 2 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of creating a twin instance of a scene by scanning and loading a twin template for the scene.

[0018] Figure 3 It is based on Figure 1 The method flowchart for creating a twin instance of a scene by scanning and loading a twin template for the scene, as shown in the corresponding embodiment, is described in another embodiment.

[0019] Figure 4 This is a schematic diagram illustrating the implementation of dynamic definition and loading of an intelligent twin according to one embodiment.

[0020] Figure 5 This is a schematic diagram illustrating the architecture of obtaining a twin instance based on an external twin plugin extension, as shown in an example.

[0021] Figure 6 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the method for triggering the runtime logic domain of multimodal awareness of twin instances based on runtime data loaded from twin instances.

[0022] Figure 7 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the method for triggering the runtime logic domain of multimodal awareness of twin instances based on runtime data loaded from twin instances.

[0023] Figure 8 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of deducing the business behavior logic configured in the runtime business flow through the triggering of the runtime logic domain. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0025] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0026] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] This application provides an intelligent agent platform that connects to a process engine. The intelligent agent platform can implement scenarios and multimodal intelligent twin instances for various industry scenarios.

[0028] Specifically, the twin instances created by the intelligent agent platform in a scenario can rely on the low-code process orchestration capabilities provided by the process engine to visually configure and customize the workflow of the twin instances, enabling the twin instances to automatically trigger and execute the corresponding runtime business flow based on the defined workflow.

[0029] Meanwhile, the process engine also provides a plug-in extension mechanism for the intelligent agent platform. With the support of the process engine, the twin instance can trigger the corresponding plug-in to realize the deduction process of specific business behavior logic under the running business flow, thereby making the twin instance's operation configurable, scalable and intelligent.

[0030] It should be noted that the implementation of this application utilizes an intelligent agent platform and the connected process engine to achieve a twin instance with intelligent operation characteristics. For example, the twin instance can perform multimodal perception based on runtime data, and then adaptively trigger runtime business flows to enter the corresponding running state, thereby having the ability to respond to external changes in real time and dynamically schedule business processes.

[0031] Based on this, the realized twin instances constitute multimodal intelligent twin instances, that is, intelligent agents with self-perception, self-decision-making, and self-execution capabilities.

[0032] The implementation of multimodal intelligent twins, their adaptive operation in scenarios, and even the implementation of intelligent agent platforms and process engines will be specifically elaborated through the multimodal intelligent twin operation implementation method provided in this application.

[0033] See Figure 1 , Figure 1 This is a flowchart illustrating a method for implementing a multimodal intelligent twin according to an exemplary embodiment.

[0034] This application provides a method for implementing a multimodal intelligent twin, including: Step S110: By scanning and loading the twin template for the scene, a twin instance of the scene is created. The twin template is used to describe the business behavior logic and definition information of the twin belonging to a certain type. Step S120: Trigger the runtime logic domain of the twin instance based on the runtime data loaded by the twin instance. The triggering of the runtime logic domain includes the runtime business flow triggered by the process engine based on multimodal perception. Step S130: By triggering the running logic domain, the business behavior logic configured in the running business flow is deduced. This deduction is executed by the process engine calling the corresponding function and / or plugin for the twin instance. Step S140: After the current inference of the twin instance is completed, the obtained execution result and the context data mapped by its own running state are updated back to the running state closed loop loaded by the twin instance, driving the twin instance that forms the running state closed loop to constitute a multimodal intelligent twin instance.

[0035] These steps are explained in detail below.

[0036] In step S110, it should first be specifically explained that scenarios belonging to various industries and twin instances running within those scenarios will be implemented based on industry applications. The scenario refers to the logical container implemented and run by the intelligent agent platform, used to support the collaborative operation of several twins. For example, for a business domain of an industry application, such as a smart park or a subway station, the corresponding scenario will be run. Through the collaborative operation between twin instances within the scenario, the business domain's operational status will be responded to, and the corresponding business behavior logic will be deduced, thereby achieving a virtual-real mapping of the real business domain.

[0037] The twin template includes a twin definition template and a twin instance template. Twin instances that can be deployed in a scenario include basic twin instances and intelligent twin instances. For example, a basic twin instance corresponds to a standardized physical entity, such as standard equipment with a fixed structure (air turbines, lighting fixtures, and other infrastructure). An intelligent twin instance, on the other hand, is a special setting adapted to a scenario and is dynamically generated for that scenario. Therefore, for the creation of basic twin instances in a scenario, the scanned and loaded twin templates include both the twin definition template and the twin instance template. For intelligent twin instances, the scanned and loaded template is a general twin definition template. In other embodiments, all twin instances in a scenario can also be created by dynamically loading the general twin definition template. The general twin definition template used to dynamically generate intelligent twin instances encapsulates a dynamic attribute definition loader, a dynamic event message definition loader, and an intelligent twin runtime instance mapping definition. Therefore, it can dynamically load definition information and inject business behavior logic, thereby creating a twin with autonomous behavior.

[0038] During step S110, the twin template to be loaded is determined based on the running scenario, and a context environment is provided for the running of the twin instance. As the scenario runs, the available twin templates registered by the twin management container are scanned to obtain twin templates associated with the scenario identifier. The scanned twin templates are loaded to parse their definition information. Then, based on the scenario configuration and / or snapshot data, runtime data is loaded and entity objects are bound. The corresponding twin instance of the definition information is created, thereby obtaining a twin instance loaded with runtime data and business flow triggers.

[0039] To further explain, the business process trigger is used to perform event detection and processing on the self-updated runtime data to determine the events triggered by the self-updated runtime data. In an exemplary embodiment, the business process trigger includes an initialization runtime business flow trigger and an event processing business flow trigger. The initialization runtime business flow trigger is used to control the runtime business flow triggering of the twin instance at the beginning of its creation, thereby initializing the deduction and implementation of the corresponding business behavior logic. Compared to the initialization runtime business flow trigger, the event processing business flow trigger is used to implement runtime business flow triggering in response to subsequent events for the twin instance entering the runtime state after the initialization of the runtime business flow.

[0040] In an exemplary embodiment, the created twin instance is also configured with a business operation domain. Under the influence of the loaded runtime data and the business operation domain, the twin instance achieves a dynamic closed loop, that is, it realizes a closed loop system of triggered event-decision (triggering runtime business flow)-response execution and status feedback, so that the twin instance no longer only describes the physical state of the corresponding physical entity, but has the ability to respond autonomously and adjust while executing according to status feedback.

[0041] The business flow triggers in the created twin instance are used for runtime awareness listening to trigger events. Specifically, based on the configured and associated runtime data, such as runtime data received from the Internet of Things, the twin instance obtains the events sensed and received in the current runtime state through the execution of event detection and processing, and then triggers the twin's runtime logic domain to respond to the event, as shown in step S120.

[0042] In summary, the twin template describes the business behavior logic and definition information of a particular type of twin. This business behavior logic is injected into the runtime logic domain of the twin instance to execute a specific deduction process. In other words, the business behavior logic described by the twin template provides the foundation for the self-running of the twin instance.

[0043] In step S120, the runtime logic domain is triggered based on the loaded runtime data. It should be noted that the runtime logic domain injects business behavior logic corresponding to each runtime business flow. This business behavior logic is used to implement deductions in response to runtime data. A runtime business flow is an execution chain triggered by runtime data in response to the workflow orchestrated and configured by the behavior tree. Based on the process node matched by the runtime data in the behavior tree, the runtime business flow mapped to that process node can be obtained. Therefore, runtime execution within the runtime logic domain is performed by loading and triggering the runtime business flow within the built-in runtime logic domain of the twin instance. This involves calling functions and / or plugins based on the runtime business flow, then executing actions—that is, deducing the business behavior logic—through the called functions and / or plugins, and finally updating the context.

[0044] Each twin instance is bound to a behavior tree. The process nodes distributed on the behavior tree represent a business behavior logic, such as the action of "device restart", which corresponds to a workflow of the twin instance. Similarly, the sequential execution of nodes in the behavior tree realizes the sequential execution of the twin instance's workflow. In short, through the process engine's interpretation and matching of the behavior tree, and the orderly selection of the distributed process nodes, the twin instance can run in a controlled and independent manner.

[0045] For example, for a twin instance of a forklift, its workflow sequence corresponds to the workflow node that instructs the forklift to move goods, the workflow node that moves to the unloading area, and the workflow node that unloads goods in the unloading area. The triggering of the running business flow associated with the workflow node can realize the execution of the business behavior logic under the current workflow node.

[0046] The runtime logic domain of each twin instance is the runtime environment within its respective twin instance that carries and interprets business behavior logic. In other words, in the runtime state, the runtime logic domain provides the runtime environment for the business behavior logic configured to execute the runtime business flow. Under the action of the runtime logic domain, the runtime business flow triggered by the behavior tree bound to the twin instance executes the corresponding business behavior logic through scheduling plugins, function calls, and other methods, thereby realizing the deduction process of the twin instance and maintaining the execution state of the runtime state.

[0047] In a specific instance, when the process engine triggers a runtime business flow on the behavior tree bound to the twin instance (the process engine matches the process nodes distributed in the behavior tree based on runtime data, and then finds the runtime business flow based on the mapping of the process nodes), as the runtime business flow is triggered, the runtime business flow is loaded into the runtime logic domain. The runtime logic domain calls functions to parse the execution chain represented by the runtime business flow, such as the behavior path for implementing wind turbine control, and follows this behavior path to implement function calls and / or plugin calls, thereby achieving the triggering of the runtime logic domain and the deduction of the business behavior logic in step S130.

[0048] The execution results and running status of each process node (e.g., current state variables, input and output parameters, and other context data) will be aggregated to update the runtime context, i.e., the twin instance self-update process implemented in subsequent step S140.

[0049] Here, the runtime business flow involved in step S120 will be further elaborated in conjunction with the behavior tree. The runtime business flow originates from the behavior tree bound to the twin instance. On the content side, the runtime business flow describes the behavior path that the twin instance is about to dynamically execute, i.e., the execution chain as mentioned above.

[0050] The runtime business flow is mapped to process nodes on the behavior tree. The business behavior logic of the process node is realized through the process definition of the runtime business flow, enabling the runtime execution of the twin instance on this process node. For step S120, the runtime logic domain triggering based on the runtime data loaded by the twin instance is the execution process of matching the runtime data with the runtime business flow under the action of the behavior tree bound to the twin instance. This process is specifically implemented through the matching of runtime data to process nodes and the mapping of process nodes to runtime business flows.

[0051] The runtime data loaded by the twin instance can be matched to specific process nodes through the process engine, thereby triggering the runtime business flow mapped to that process node. This is the basic implementation of twin instance operation. Retaining the basic implementation of twin instance operation, in the execution of step S120, other runtime business flows will also be matched based on the runtime data loaded by the twin through multimodal perception, thereby triggering the other matched runtime business flows in parallel.

[0052] Multimodal perception is the execution process of runtime data fusion and reasoning with runtime data as input, so that twin instances can move from passively responding to events to intelligent decision-making and automatic inference.

[0053] In other words, the triggering of the twin's operational logic domain will also be achieved through multimodal perception triggering, thereby leveraging multimodal perception capabilities to achieve adaptive responses to complex scenarios and enhance the intelligent decision-making capabilities of the twin instance, i.e., the multimodal intelligent twin.

[0054] In an exemplary embodiment, the multimodal awareness runtime logic domain triggering is based on the model service deployed by the process engine calling the deployed large model interface to achieve context semantic understanding and reasoning, so as to dynamically adjust the execution strategy for the twin instance.

[0055] As data from the physical environment mapped by the connected IoT is transmitted, such as sound waveform data and camera footage data, the runtime data loaded on the twin instance includes the received sound waveform data, camera footage data, and other similar data. This type of data is unstructured and therefore difficult to identify and process for matching process nodes on the bound behavior tree, making it difficult for the twin instance to respond to complex environments.

[0056] The model service deployed through the process engine will use the large model interface to perform real-time fusion and understanding of the full runtime data, namely multimodal data composed of sound waveform data, camera image data, sensor signals, text, etc., thereby breaking through the limitation of single-modal perception in process node matching.

[0057] By calling the large model interface through the model service deployed by the process engine, the system performs environment perception, semantic recognition, and decision-making with runtime data as input to obtain inference results. Then, based on the inference results, it matches the behavior tree bound to the twin instance, thereby driving the twin instance to run based on multimodal perception.

[0058] In summary, in step S120, the process engine is used to deploy the model for the operation of the scenario and the twin instance, enabling the twin instance to have the ability to reason about multimodal perception. This allows the twin instance to understand the complex environmental state for the triggering of the running business flow and dynamically match the optimal execution path, thus realizing an intelligent operation mode that integrates perception, decision-making and execution of the twin instance.

[0059] In step S130, the triggering of the running logic domain immediately performs the deduction of the business behavior logic of the running business flow configuration, which is the execution of the action under the corresponding process node. In an exemplary embodiment, the execution of the action under the process node will construct the plugin tool mechanism in the running twin instance through the plugin tool trigger deployed by the process engine, thereby triggering the plugin for the action execution of the business behavior logic, calling the corresponding plugin API interface, so that the twin instance can obtain the plugin capabilities of the twin plugin pool through plugin calls, so as to support the action execution capability of the twin instance.

[0060] In one exemplary embodiment, the plugin calls implemented for the twin instance are based on the process engine. The twin instance's twin plugin pool includes twin data read / write plugins, twin event publishing plugins, scene path lookup plugins, and twin relationship query plugins, etc., which will not be listed one by one here.

[0061] Therefore, it should be understood that the process engine, on the one hand, implements the orchestration of business logic for the twin instance, that is, implements the behavior tree bound to each twin instance, and on the other hand, it connects external functional modules and algorithm services, such as the deployment of model services to call the large model interface and plugin calls, so as to enable the twin instance to have high scalability and the ability to execute multiple business behavior logics.

[0062] To further explain, a plug-in trigger is a configurable execution unit deployed in the process engine. It is used to dynamically trigger corresponding plug-ins based on the matching process node definition, i.e., the runtime business flow mapped by the process node, so as to complete specific business behavior logic by means of the called plug-ins.

[0063] With the help of plugin triggers, different plugins can be dynamically loaded to adapt to process nodes, and the functionality can be expanded without modifying the code.

[0064] In summary, during the execution of step S130, automated deduction and closed-loop control of business behavior are achieved through function calls and / or plugin calls.

[0065] As the deduction of business behavior logic implemented by function calls and / or plugin calls is completed, the corresponding execution results and the context data mapped by the running state of the twin instance itself will be updated back to the running state closed loop loaded by the twin instance, as shown in step S140.

[0066] In step S140, the twin instance completes the execution of a business behavior logic, that is, after a simulation ends, the execution result and the context data describing the state changes of the twin instance are written back to the twin instance's own data structure, that is, the loaded runtime data is updated. Based on the data update, the twin instance is driven to trigger the perception of the corresponding physical entity and the physical environment again, and then the decision made by the multimodal perception provided by the model service is integrated. Finally, the corresponding runtime business flow is executed again. In this way, the twin instance forms an adaptive decision-updating runtime closed loop, which can self-feedback, self-update and run continuously.

[0067] Specifically, after the twin instance completes a runtime business flow execution, the process engine returns the execution result to the twin instance. At this time, the intelligent agent platform maps the execution result to the current runtime state of the twin instance, obtains context data describing the current runtime state of the twin instance, such as state variable changes, inputs, and outputs in the scene, and updates the runtime data loaded in the twin instance with the execution result and context data. This ensures that the runtime data is not limited to the data transmitted by the current physical entity and physical environment, but also includes context data describing the current state of the twin instance and the scene's runtime state, as well as the execution result of the previous round of simulation. All types of data will be synchronously updated back to the runtime data container of the twin instance after a round of simulation is completed.

[0068] After the update is completed, the new runtime data will be used as the input for the next round of matching and mapping of runtime business flows, as well as multimodal awareness and matching and mapping of other runtime business flows under multimodal awareness. Then, the updated state will continue to be scheduled to the next process node through the re-matching of process nodes and the mapping of runtime business flows.

[0069] The implementation of the runtime closed loop enables the twin instance to automatically receive new data and adjust business behavior logic in each business simulation, run continuously, update itself after each simulation and trigger the run again, truly realizing a multimodal intelligent twin instance, that is, a multimodal intelligent agent.

[0070] See Figure 2 , Figure 2 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of creating a twin instance of a scene by scanning and loading a twin template for the scene.

[0071] The step S110 provided in this application embodiment, which involves scanning and loading a twin template for a scene to create a twin instance of the scene, includes: In step S111a, when the scene is started, the scene runtime manager automatically scans and indexes the twin templates of the scene. The twin templates include twin definition templates and twin instance templates. The twin instances created by the twin instance templates run in the scene. Step S112a: Load the definition information in the twin instance template through the twin definition template. The twin instance template has built-in business triggers. Step S113a: Create a base twin instance for the scene based on the twin instance template that loads definition information.

[0072] These steps are explained in detail below.

[0073] First, it should be noted that this exemplary embodiment is used for the creation of basic twin instances. Basic twin instances correspond to infrastructure, standardized equipment, or other physical entities with fixed structures and relatively stable attributes, such as power transformers, pipeline nodes, and computer room air conditioners. These physical entities are static and ubiquitous, and are enumerable and relatively few in number.

[0074] For this type of physical entity, its corresponding twin definition template and twin instance template are pre-configured to describe the twin instance's attributes, events, and other definition information. Based on this, the intelligent agent platform can directly create the corresponding basic twin instance according to the pre-configured twin definition template and twin instance template.

[0075] This approach reduces the complexity and repetitive workload of creating twin instances by using templated configuration. Furthermore, since the basic twin instance changes relatively little, it ensures that the loading and deployment of the basic twin can be completed quickly, stably, and accurately during scenario startup.

[0076] Specifically, the twin definition template is used to define twins. Compared to twin instances, twins are virtualized objects of a type of physical entity that are not instantiated and deployed to run in a specific scenario. The twin definition template loads definition information such as dynamic attribute definitions and event definitions of twins through a unified configuration method.

[0077] The twin instance template has a built-in business flow trigger, enabling the created twin instance to schedule and execute the corresponding runtime business flow. In an exemplary embodiment, the twin instance template configures and describes business behavior logic and definition information, so that business behavior logic can be injected during the creation of the twin instance. The deduction process of the business behavior logic has the ability to call plugins, thereby increasing the capabilities of the created twin instance.

[0078] See Figure 3 , Figure 3 It is based on Figure 1The method flowchart for creating a twin instance of a scene by scanning and loading a twin template for the scene, as shown in the corresponding embodiment, is described in another embodiment.

[0079] The step S110 provided in this application embodiment, which involves scanning and loading a twin template for a scene to create a twin instance of the scene, includes: Step S111b: Load the runtime dynamic definition of the twin instance for the scene, and obtain the definition information of the twin instance for dynamic attribute definition and dynamic event definition; Step S112b: By scanning the mapping of information carriers and business behavior logic defined in the loaded general twin template, a smart twin instance of the scenario is dynamically created, and the runtime business flow corresponding to the defined information is bound to the smart twin instance.

[0080] As mentioned earlier, the general twin template encapsulates a dynamic attribute definition loader, a dynamic event message definition loader, and a smart twin runtime instance mapping definition. The dynamic attribute definition loader is used to load dynamically defined attributes through the twin dynamic definition configuration service, and the dynamic event message definition loader is used to load dynamically defined event message definitions through the twin dynamic definition configuration service. The loading of definition information is achieved through the actions of the dynamic attribute definition loader and the dynamic event message definition loader.

[0081] Based on this, it should be further explained that the dynamic definition of attributes and event messages referred to can refer to the dynamic definition configuration operation based on the front-end client user, so as to load the configuration in real time during the creation of the twin instance; or it can refer to reading numerous configurations in the database to obtain variable definition information relative to the general twin template, and then dynamically defining attributes and event messages in the general twin template.

[0082] The general twin template encapsulates the intelligent twin runtime instance mapping definition, which is used to map to business behavior logic and dynamically bind business behavior logic to the intelligent twin instance. In other words, the intelligent twin runtime instance mapping definition enables the definition information to have corresponding business behavior logic, so that the business behavior logic can be injected into the created twin instance, thereby enabling the intelligent twin instance to have intelligent behavior.

[0083] See Figure 4 , Figure 4 This is a schematic diagram illustrating the implementation of dynamic definition and loading of an intelligent twin according to one embodiment.

[0084] In one embodiment, the intelligent agent platform initiates the running of the twin under the action of the scene manager. At this time, the twin instance manager will be triggered to load the runnable definition in order to obtain the intelligent twin definition information, namely the aforementioned attribute and event message definition.

[0085] By running and loading the intelligent extension implementation class built for the twin definition, the running and loaded intelligent extension implementation class reads the configuration from the web client user's dynamic definition configuration operation and / or database through the twin dynamic definition configuration service to load attribute configuration and event information configuration, obtain the definition information of dynamic attributes and event messages, and bind business behavior logic to the twin under the intelligent twin runtime instance mapping definition of the intelligent extension implementation class. This enables the intelligent twin instance created and deployed to the scene to trigger the call of corresponding functions and / or plugins through the dynamically bound business behavior logic.

[0086] Therefore, based on the database configuration, the web client can dynamically define the triggering of configuration operations, dynamically create twins, dynamically register attributes and event messages, construct business behavior logic and establish mappings, that is, realize behavior mapping. The definition information such as attributes and event messages, as well as business behavior logic, are no longer hardcoded into the source code. In other words, there is no longer a need for code-level static twin definitions. The intelligent twin structure is configurable, and adding new intelligent twin types no longer requires recompiling the code. It has the ability to extend intelligent twins at runtime, thereby extending new intelligent twin instances.

[0087] In one exemplary embodiment, the twin instance includes a twin instance that extends the external twin.

[0088] Correspondingly, before step S110 of scanning and loading the twin template for the scene, the multimodal intelligent twin operation implementation method provided in this application embodiment further includes: The corresponding twin template is loaded and registered to take effect through external twin loading, and the twin instance is injected by dependency of the effective twin template.

[0089] In other words, the loading and registration of the external twin will obtain a twin template that is effective and hosted in the twin instance management container. Then, the twin instance management container can automatically provide the twin template it needs to the twin instance manager running on the intelligent agent platform, thereby creating a twin instance for the running scenario.

[0090] It should be noted that the external twin refers to the twin defined by loading and registering the twin definition provided by the external system, adding the twin template to the twin instance management container, that is, the twin template is hosted by the container, so that the twin template is called and uniformly managed by the twin instance manager running the intelligent agent platform, and finally the corresponding twin instance is created under the control of the twin instance manager.

[0091] To further explain, in an exemplary embodiment, the agent platform deploys a twin plugin loader and a container auxiliary component. Correspondingly, the loading and registration of external twins refers to loading the twin definition from the external twin plugin into the agent platform, then instantiating it and registering it with the twin instance management container through the container auxiliary component. At this time, the instantiated twin template takes effect, thereby enabling the components of the agent platform, such as the twin instance manager, to inject twin instances into the runtime scenario dependencies through the twin template in the twin instance management container. This provides a mechanism for the twin template and the implementation of the twin instance to inject the dependent objects into it by the twin instance management container. For example, it injects the required dependencies, including but not limited to services, such as the aforementioned business flow triggers, runtime logic domains and the business behavior logic used for deduction in the runtime logic domains, and initial runtime data, in order to realize the creation of twin instances. By analogy, for the agent platform, a platform architecture that enables twin instances to be scalable, maintainable and pluggable can be realized.

[0092] Therefore, by loading and registering through the intelligent agent platform, twins existing in the form of plugins can create twin instances in the scene within the intelligent agent platform, realizing dependency injection of twin instances. This eliminates the need to create various dependencies to extend external twins, and also eliminates the need to deploy a large amount of code to extend external twins. It is easy to extend and maintain, and avoids the defects of rigid and inflexible code, making the extension and loading of twin templates form a pluggable structure.

[0093] See Figure 5 , Figure 5 This is a schematic diagram illustrating the architecture of obtaining a twin instance based on an external twin plugin extension, as shown in an example.

[0094] The built-in twins of intelligent agent platforms are limited and cannot fully cover all industries. For industry-specific twins, such as those applicable to water conservancy and emergency response, external twins can be used. This involves a twin library designed for plug-in architecture, using its own twins—external twins relative to the intelligent agent platform—to create the twins missing from the platform, thus enabling the creation of the required twin instances.

[0095] For specific industries, industry business developers develop industry-specific twin plugins based on the constructed twin plugin abstraction layer, and obtain twin plugins corresponding to specific industries, that is, external twin plugins relative to the intelligent agent platform.

[0096] The constructed twin plugin abstraction layer defines twin definition templates and twin instance templates to develop industry-specific twin plugins, that is, external twin plugins relative to the intelligent agent platform.

[0097] With the development of industry-specific twin plugins, namely external twin plugins outside the intelligent agent platform, the external twin plugins themselves have been defined, that is, they have twin definitions. The twin definition is used to describe the static information, behavioral capabilities and event interaction mechanisms of the twin implemented by the external twin plugin in a templated way. The twin definition is used by the intelligent agent platform to create twin instances.

[0098] Specifically, the twin definition includes attributes, publishable and listenable events, the implementation logic of the twin instance, and the twin instance creation information. Based on the attributes, publishable and listenable events defined in the twin definition, the communication between twin instances is standardized, including the business behavior logic of the twin instance itself and the event publishing mechanism.

[0099] Therefore, a twin definition corresponds to a twin template, which can be instantiated as a twin template. The twin instance creation information includes at least three components: services, runtime logic domains, and runtime data. Services can be, at least, the aforementioned business flow triggers and implementation plugins, etc., to support the basic capabilities that the created twin instance can invoke at runtime. The runtime logic domain expresses the business behavior logic of the created twin instance in a specific industry. It maps to specific behavior functions and / or tool plugins, and also to the runtime business flow on the behavior tree bound to the created twin instance, thus inheriting the business capabilities from event triggering to behavior execution during the twin instance's operation, thereby achieving the deduction of business behavior logic. Runtime data is the state and context data of the created twin instance during runtime. It is the carrier for tracking the operation and driving closed loop of the created twin instance, enabling the twin instance to maintain continuity, memory, and state awareness.

[0100] In one exemplary embodiment, runtime data includes the current state of the created twin instance, context data, multimodal perception results, and inference results.

[0101] By defining a twin, a twin template can be obtained by loading the twin plugin loader in the intelligent agent platform and registering the container auxiliary component with the twin instance management container. This allows the twin instance manager to scan the twin instance when any scenario starts and create the required twin instance for that scenario.

[0102] In another exemplary embodiment, see Figure 6 , Figure 6 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the method for triggering the runtime logic domain of multimodal awareness of twin instances based on runtime data loaded from twin instances.

[0103] The step S120 of triggering the runtime logic domain for multimodal awareness of twin instances based on runtime data loaded by the twin instance provided in this application embodiment includes: Step S121a: Perform event detection and processing on the runtime data loaded by the twin instance to determine the events triggered by the runtime data; Step S122a: Based on the triggered events, determine the runtime business flow from the twin behavior interpreted by the process engine. Step S123a: The execution of the runtime logic domain is triggered by the execution call of the runtime business flow on the twin instance.

[0104] These steps are explained in detail below.

[0105] The runtime data loaded by the twin instance describes, on the one hand, the current state of the twin implementation, such as the execution results of completed business behavior logic deductions and the involved state variables, input and output parameters; on the other hand, it also describes the state of the corresponding physical entity. In other words, the runtime data package loaded by the twin instance contains real-time data transmitted by the physical entity through the Internet of Things (IoT). The input of real-time data enables the twin instance to continuously update its runtime data during operation, ensuring that the determination and execution of the running business flow correspond to real-time data and are tailored to a specific physical entity.

[0106] For the runtime data loaded in the twin instance, services within the twin instance, such as business flow triggers, perform runtime-aware listening based on the runtime data to determine the currently triggered event. At this time, the access process engine interprets and runs the twin behavior tree bound to the twin instance, matches the process node in the behavior tree, and the runtime business flow mapped to that process node is triggered for execution.

[0107] The process engine triggers the runtime business flow for the execution twin instance. Through function and / or plugin calls, it triggers the execution of the runtime logic domain and then executes the business behavior logic corresponding to the runtime business flow to complete the assigned deduction process.

[0108] This is a runtime logic domain response based directly on runtime data. However, with the process engine performing multimodal perception on the runtime data, it can match other process nodes in the behavior tree based on the obtained multimodal perception results, thereby realizing the execution of a multimodal perception twin instance. See details. Figure 7 The implementation process of step S120 is shown.

[0109] See Figure 7 , Figure 7 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the method for triggering the runtime logic domain of multimodal awareness of twin instances based on runtime data loaded from twin instances.

[0110] The step S120 of triggering the runtime logic domain for multimodal awareness of twin instances based on runtime data loaded by the twin instance provided in this application embodiment includes: Step S121b: After the twin instance loads runtime data, it triggers an event through multimodal perception of the runtime data; Step S122b: The behavior tree is interpreted in the process engine to run the response to the event, driving the running business flow; Step S123b: The runtime logic domain of the twin instance is triggered to respond by the runtime business flow.

[0111] In this exemplary embodiment, intelligent decision-making based on runtime data is realized, and then the operation of the twin instance is controlled based on the intelligent decision-making, thereby realizing an intelligent operation mode that integrates perception, decision-making and execution of the twin instance.

[0112] See Figure 8 , Figure 8 It is based on Figure 1 The corresponding embodiment shows a flowchart describing the steps of deducing the business behavior logic configured in the runtime business flow through the triggering of the runtime logic domain.

[0113] The S130 step provided in this application embodiment, which involves deducing the business behavior logic configured in the runtime business flow through the triggering of the runtime logic domain, includes: Step S131: As the running logic domain is triggered, determine the business behavior logic corresponding to the running state business flow under the running logic domain; Step S132: The deduction process of business behavior logic is executed through function calls of the twin instance itself and / or plugin calls triggered by the process engine.

[0114] In this exemplary embodiment, the triggering of the running logic domain is the triggering of the running business flow matched and mapped by the process engine side. As the running business flow is loaded into the running logic domain, the behavior path represented by the running business flow will be parsed, and the behavior path corresponds to the business behavior logic injected into the running logic domain.

[0115] For the required business logic, the corresponding functions and / or plugins are called to execute the deduction process of the business logic. At this time, as shown in step S132, the process engine causes the plugin tool trigger to call the corresponding plugin API interface for the twin instance based on the triggered running business flow, and then calls the plugins in the twin plugin pool to execute the behavior of the twin instance.

[0116] In summary, through the aforementioned exemplary embodiments, a twin instance with intelligent features is implemented for the running scenario, namely, the multimodal intelligent twin instance as mentioned above, thereby enabling the scenario and the twin instance running in the scenario to be embedded with multimodal perception and reasoning, ensuring the intelligent operation of the multimodal intelligent twin instance.

[0117] As the scenario and the multimodal intelligent twin instances in the scenario run, in an exemplary embodiment, a set of twin instances will be constructed for all multimodal intelligent twin instances in the scenario, and snapshot data corresponding to the current moment will be obtained for the execution of business behavior logic of each multimodal intelligent twin instance at the current moment, and the existing static data will be filtered based on the snapshot data.

[0118] Therefore, by integrating the filtered data, snapshot data of the relevant scenario can be batch-stored into the database, realizing the current snapshot storage of multimodal intelligent twin instances.

[0119] To adapt to the operation of the multimodal intelligent twin instance and its subsequent write-back, the corresponding snapshot data is obtained from the execution of its business behavior logic. In an exemplary embodiment, snapshot data is generated for the multimodal intelligent twin instance based on the executed business behavior logic, corresponding to the execution result, running state, and dynamically changing runtime data, in order to reconstruct the running environment (i.e., the scenario) of the multimodal intelligent twin instance. The running state of the multimodal intelligent twin instance returns to the executed business behavior logic, and the reconstructed scenario and the operation of the multimodal intelligent twin instance continue based on the written-back execution result, running state, and dynamically changing runtime data.

[0120] This enables the write-back of snapshot data to trigger the recovery of the multimodal intelligent twin instance, allowing for seamless deduction of subsequent business logic and achieving consistent recovery of data and state.

[0121] In addition, since the stored snapshot data is filtered, differential write-back is performed based on the filtered snapshot data during scene restoration, and the static data is adapted to fill it in for use in the entire reconstruction process.

[0122] To further explain, the runtime status in the snapshot data indicates the context data such as state variables and input / output parameters of function calls and / or plugin calls under the executed business behavior logic; the dynamically changing runtime data is adapted to the multimodal perception being performed, including but not limited to multimodal data such as sound waveform data and camera image data, and thus, together with the event stream obtained from detection and processing, it serves as the decision basis for the multimodal intelligent twin instance, and is applied to the subsequent execution of business behavior logic. In this way, the runtime status restoration of the scene and the multimodal intelligent twin instance in the scene is realized.

[0123] Furthermore, in an exemplary embodiment, the implementation of obtaining snapshot data corresponding to the current moment for the execution of business behavior logic of each multimodal intelligent twin instance includes freezing the runtime data, execution results, and running status of each multimodal intelligent twin instance, and obtaining the currently executed process node and the running state business flow mapped to the process node, so as to encapsulate it as snapshot data.

[0124] Based on this snapshot data, the twin instances of the corresponding scenario can be batch-added into the database to obtain a multimodal runtime state snapshot of the scenario, which can be used for subsequent scenario recovery, simulation rollback or cross-scenario migration.

[0125] In another exemplary embodiment, this application also provides an implementation of an intelligent agent platform and the connected process engine.

[0126] To enable the creation of multimodal intelligent twin instances and the scenarios in which they run, an intelligent agent platform was built, and a process engine was integrated into the intelligent agent platform.

[0127] Among them, the intelligent agent platform will be designed for industry application scenarios. In other words, it can realize multimodal intelligent twin instances belonging to any specific industry application, as well as intelligent platforms that run multimodal intelligent twin instances.

[0128] Thus, the intelligent agent platform, which enables multimodal intelligent twin instances and scenarios for any industry application, will also obtain multimodal intelligent twin instances extended by external twins through loading and registration of external twins, so as to widely realize the rapid and accurate creation and intelligent operation of multimodal intelligent twin instances for applications across all industries.

[0129] The process engine provides a low-code process orchestration platform and plug-in tools and triggers for the intelligent agent platform, and obtains multimodal perception capabilities by connecting to a multimodal large model.

[0130] The low-code workflow orchestration platform is used to construct the behavior tree of twin instances and to perform visual orchestration of the runtime business flows mapped to the process nodes on the behavior tree. Through the low-code workflow orchestration platform, users can graphically design and configure the executable actions and runtime business flows of a specific twin instance without having to worry about the underlying code implementation, thereby significantly improving the efficiency of behavior modeling and business adaptability of twin instances.

[0131] For the graphically configured runtime business flow, the low-code process orchestration platform of the process engine maps the corresponding function calls and / or plugin calls. Then, with the help of the plugin tool, the corresponding plugin calls are triggered for the twin instance based on the mapping, and finally the deduction process of business behavior logic under the runtime business flow is executed.

[0132] In addition, the process engine also gains multimodal perception capabilities by accessing a large multimodal model.

[0133] After completing the graphical configuration, the intelligent agent platform and process engine enable the operation and collaboration of various multimodal intelligent twin instances in the scene, and then control the corresponding physical entities based on the output running status and execution results.

[0134] In one exemplary embodiment, this application also provides a computer device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method as described above.

[0135] In one exemplary embodiment, this application also provides a computer program product including a computer program that, when executed by a processor, implements the steps of the method as described above.

[0136] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described above.

[0137] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0138] In an exemplary embodiment of this application, a computer program medium is also provided, on which computer-readable instructions are stored, which, when executed by a computer's processor, cause the computer to perform the methods described in the above method embodiments.

[0139] According to one embodiment of this application, a program product for implementing the methods in the above-described method embodiments is also provided. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0140] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0143] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0144] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0145] Furthermore, although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0146] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0147] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A method for operational implementation of a multi-modal intelligent twin, the method comprising: The method comprises: ​ creating a twin instance of a scene by loading a twin template for the scene, the twin template being used to describe the business behavior logic and definition information of a type of twin; triggering the running logic domain of the multi-modal perception of the twin instance based on the runtime data loaded by the twin instance, the triggering of the running logic domain comprising a process engine triggering a running state business flow based on multi-modal perception; deriving the configured business behavior logic of the running state business flow through the triggering of the running logic domain, the derivation being performed by the process engine calling corresponding functions and / or plug-ins for the twin instance; after the current derivation of the twin instance ends, updating the obtained execution result and the context data mapped from the running state of the twin instance back to the runtime data loaded by the twin instance, the updating of the runtime data driving the formation of a running state closed loop of the twin instance, the twin instance driving the formation of the running state closed loop constituting a multi-modal intelligent twin instance.

2. The method of claim 1, wherein, The creating of the twin instance of the scene by loading the twin template for the scene comprises: when starting a scene, a scene running manager automatically scans and indexes a twin template of the scene, the twin template comprising a twin definition template and a twin instance template, the twin instance template creating a twin instance running in the scene; loading definition information in the twin instance template through the twin definition template, the twin instance template having a business flow triggerer built-in; creating a basic twin instance of the scene according to the twin instance template loaded with the definition information.

3. The method of claim 1, wherein, The creating of the twin instance of the scene by loading the twin template for the scene comprises: loading a dynamic definition of a twin instance runtime of a scene, obtaining a dynamic attribute definition and a dynamic event definition as definition information of the twin instance; dynamically creating an intelligent twin instance of the scene through the mapping of the definition information and the business behavior logic in the definition information carried in the scanned and loaded general twin template, the running state business flow corresponding to the definition information being bound to the intelligent twin instance.

4. The method of claim 1, wherein, The twin instance comprises an externally extended twin instance.

5. The method of claim 4, wherein, Before the loading of the twin template for the scene, the method further comprises: registering and activating the corresponding twin template by loading an external twin, and injecting the twin instance by the activated twin template.

6. The method of claim 1, wherein, The triggering of the running logic domain of the multi-modal perception of the twin instance based on the runtime data loaded by the twin instance comprises: detecting and processing events of the runtime data loaded by the twin instance to determine events triggered by the runtime data; determining a running state business flow based on the triggered events in a twin behavior tree interpreted and run by a process engine; triggering the running of the running logic domain by the execution of the running state business flow of the twin instance.

7. The method according to claim 1 or 6, characterized in that, The triggering of the running logic domain of the multi-modal perception of the twin instance based on the runtime data loaded by the twin instance comprises: The twin instance loads runtime data, and then triggers an event through multi-modal perception of the runtime data; The event is responded to by an interpretive run of a behavior tree in a process engine, driving a runtime business flow; The runtime logic domain of the twin instance is triggered by the runtime business flow to respond.

8. The method of claim 1, wherein, The triggering of the runtime logic domain is used to perform the deduction of the business behavior logic configured by the runtime business flow, including: As the runtime logic domain is triggered, the business behavior logic corresponding to the runtime business flow under the runtime logic domain is determined; The deduction process of the business behavior logic is executed through function calls of the twin instance itself and / or plug-in calls triggered by the process engine.

9. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1-8.

10. A computer program product, comprising an agent platform for industry application implementation scenarios and multi-modal intelligent twins, and a process engine accessed by the agent platform; The process engine provides a low-code process orchestration platform and a plug-in tool trigger, and the process engine obtains multi-modal perception capabilities by accessing multi-modal large models; The plug-in tool is used to trigger the call of plug-ins for the twin instance running on the agent platform to execute the deduction process of the business behavior logic under the runtime business flow; The steps of the method of claims 1-8 are implemented through cooperation of the agent platform and the process engine.

Images