System and method for optimizing one or more computer simulated environments used for testing of an autonomous system

Abstract

Presented is a system (100) and a method (200) for optimizing computer simulated environment(s) used for testing of an autonomous system (404). The method (200) comprises generating a virtual representation of each failed test case in a computer simulated environment (102) based on the received one or more failed test results of the failed test cases and simulating the execution of the failed test case within the virtual representation of the autonomous system. The method (200) analyzes the simulated execution to identify deviation data from an expected behavior of the autonomous system and generates a feedback-based learning algorithm to train the computer simulated environment (102) based on the deviation data and expert inputs received from a second source. Furthermore, the method (200) comprises deploying, upon training, the feedback- based learning algorithm onto a test execution platform for testing of a system- under-test, wherein the system-under-test is associated with the autonomous system.

Description

[0001] SYSTEM AND METHOD FOR OPTIMIZING ONE OR MORE COMPUTER SIMULATED ENVIRONMENTS USED FOR TESTING OF AN AUTONOMOUS SYSTEM

[0002] The present invention relates generally to the field of computer simulation environment, and specifically to methods and systems for optimizing one or more computer simulated environments used for testing of an autonomous system.

[0003] Testing a complex autonomous or semi- autonomous system presents significant challenges due to the myriad of complexities inherent in its environment, including high-dimensionality, nonlinearity, stochasticity, and nonstationary- ness. Direct testing in the real world is deemed unsafe due to the complex and unpredictable nature of the outcomes, as well as the potential high costs associated with creating and replacing the system in case of failures. Therefore, the industry widely accepts the practice of utilizing simulations for expert-in-the- loop testing before real-world deployment.

[0004] In the event of failed test cases, thorough investigation is necessary to uncover the root cause of the failure. However, relying solely on simulations may limit the fidelity of root cause identification, as simulations may not always capture all underlying phenomena. Additionally, there is no direct method to diagnose failed test cases beyond the recorded signals or data from experiments, and there is no guarantee that the absolute root cause of a failure can be identified based on this limited dataset.

[0005] The emergence of the metaverse offers a promising solution to address these challenges. By providing a higher fidelity representation of system physics and enabling human immersion, the metaverse facilitates access to deeper data points. Furthermore, humans immersed in the metaverse can explore a wide range of potential solutions, generating valuable data to retrain or retune autonomous systems effectively.

[0006] Therefore, the object of the present invention is achieved by a method for for optimizing one or more computer simulated environments used for testing of an autonomous system, as disclosed herein. The method comprises of receiving one or more failed test results of one or more failed test cases from one or more first sources. The method further comprises of generating a virtual representation of each failed test case of the one or more failed test cases in a computer simulated environment based on the received one or more failed test results of the failed test cases. Furthermore, the method comprises of simulating the execution of the failed test case within the virtual representation of the autonomous system, wherein the simulated execution replicates the behavior of the autonomous system during an original test case. The method comprises of analyzing the simulated execution to identify deviation data from an expected behavior of the autonomous system and generating a feedback-based learning algorithm to train the computer simulated environment based on the deviation data and expert inputs received from a second source. Finally, the method comprises with deploying the feedback based learning algorithm onto a test execution platform for testing of a system-under-test, wherein the system-under-test is associated with the autonomous system. Advantageously, the present invention is computationally more efficient than a purely metaverse-based testing approach. Only the failed test cases are required to be simulated and experienced in the metaverse as it is more (computationally) expensive to do metaverse than it is to just do the simulation.

[0007] The step of the virtual representation of the autonomous system within the metaverse environment includes graphical and behavioural elements that accurately depict a hybrid model of the autonomous system with its surroundings, wherein the hybrid model comprises a sensor data model, a physics model and a data driven model. Advantageously, combining physicsbased and data-driven Al approaches results in more robust and comprehensive models, leveraging the strengths of both methodologies for enhanced simulation fidelity and predictive accuracy.

[0008] The step of the generation of the failed test cases data associated with the failed test case is based on the sensor data, control signals, system logs, and external stimuli encountered by the autonomous system during the test. Advantageously, the ability to choose which physical phenomena to replicate in the metaverse based on industry and domain -specific requirements ensures tailored and relevant testing for different applications.

[0009] The step of the simulated execution of the failed test case includes dynamic interactions with virtual entities, objects, and environmental factors present within the one or more computer simulated environments. Advantageously, dedicated modules relating to the dynamic interactions ensure that the simulated environment closely mirrors real-world physics, enhancing the accuracy and reliability of the testing outcomes.

[0010] The step of analysing the simulated execution includes comparing behaviour of the autonomous system during the simulated execution with at least one of predefined performance metrics and expected outcomes. Additionally, the step further comprises of analysing the simulated execution further comprises performing sensitivity analysis on the deviation data to identify influential variables and parameters affecting the outcome of the failed test case. Advantageously, the present invention captures and analyses the interaction effects between different sensing systems, aiding in comprehensive diagnostics and root cause analysis of observed anomalies.

[0011] The feedback-based learning algorithm comprises one of Reinforcement Learning (RL) algorithm, Reinforcement Learning using Human Feedback (RLHF) algorithm, and imitation learning algorithm. Advantageously, the integration of multiple learning methods improves the reliability and robustness of the system, making it better equipped to handle real-world challenges and unexpected situations. By leveraging the strengths of RL, RLHF, and imitation learning, the system can achieve faster convergence and higher performance, accelerating the overall training process.

[0012] Throughout the present disclosure, the term “computed simulated environment” as used herein refers to three-dimensional (3D) representation of a real or physical world. It can be understood as a virtual world. The computer-simulated environment is accessible by a user, i.e., it is accessible from the real / physical world. This comprises data exchange between the computer-simulated environment and the real / physical world. In particular, the computer-simulated environment can be understood as the “metaverse” and “industrial metaverse”, “digital twin” or “computer simulated environment” may be interchangeably used in the present invention. It is also possible to interact with the computer- simulated environment, i.e., to influence or use processes, components and / or functions in the computer-simulated environment. Therefore, processes in the computer-simulated environment may have direct influence on processes in the real / physical world, e.g., by modelling control processes virtually.

[0013] Throughout the present disclosure, the term “one or more sources” as used herein refers to the origins from which specific data, inputs, or information are obtained. To clarify and elaborate on this term, the sources may include, but not limited to, Internal Testing Logs (data collected from internal automated tests conducted within the organization), External Testing Facilities (data acquired from third- party testing services that specialize in evaluating autonomous systems), automated testing frameworks or platforms, User Feedback and Reports, Sensor Data Logs.

[0014] Throughout the present invention, it is to be understood that the terms "user", or "expert-in-the-loop" are utilized interchangeably to depict a user involved in the loop to either provide an input or provide supervision for the data.

[0015] Throughout the present invention, it is to be understood that the terms "processor" and "processing units" are utilized interchangeably to depict a processor of the system configured to perform processing according to an embodiment of the present invention.

[0016] For example, it is possible that a user can access the computer-simulated environment via a user device, e.g., a virtual reality (VR) or augmented reality (AR) device. The counterpart of the computer-simulated environment does not necessarily have to exist but can be for example a 3D model. It is also possible that physical forces and phenomena, e.g., gravity, are represented in a different way in the computer-simulated environment than in the real world, e.g., gravitational acceleration. For the purpose of this invention, the metaverse is comprised of a plurality of digital twins corresponding to one or more real-world assets in the industrial environment. The object of the invention is also achieved by a computer program product comprising machine readable instructions, that when executed by one or more processing units, cause the one or more processing units to perform the aforementioned method steps. The computer program product further comprises a storage unit communicatively coupled to the one or more processing units. The storage unit comprises a module stored in the form of machine-readable instructions executable by the one or more processing units. The module is configured to perform method steps as described above. The execution of the module may also be performed using co-processors such as Graphical Processing Unit (GPU), Field Programmable Gate Array (FPGA) or Neural Processing / Compute Engines.

[0017] The object of the present invention is further achieved by a computer readable medium on which program code sections of a computer program are saved, the program code sections being loadable into and / or executable in a system to make the system execute the method steps described above when the program code sections are executed in the system. This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the following description. It is not intended to identify features or essential features of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this invention.

[0018] The realization of the invention by a computer program product and / or a non- transitory computer-readable storage medium has the advantage that computer systems can be easily adopted by installing computer program in order to work as proposed by the present invention.

[0019] The computer program product can be, for example, a computer program or comprise another element apart from the computer program. This other element can be hardware, for example a memory device, on which the computer program is stored, a hardware key for using the computer program and the like, and / or software, for example a documentation or a software key for using the computer program. The above-mentioned attributes, features, and advantages of the present invention and the manner of achieving them, will become more apparent and understandable (clear) with the following description of embodiments of the invention in conjunction with the corresponding drawings. The illustrated embodiments are intended to illustrate, but not limit the invention.

[0020] The present invention is further described hereinafter with reference to illustrated embodiments shown in the accompanying drawings, in which:

[0021] FIG 1A is a block diagram representation of a system for optimizing one or more computer simulated environments used for testing of an autonomous system, in accordance with one or more embodiments of the present invention!

[0022] FIG IB is block diagram representation of an apparatus for optimizing one or more computer simulated environments used for testing of an autonomous system, in accordance with one or more embodiments of the present invention!

[0023] FIG 2 is a flowchart of a method for optimizing one or more computer simulated environments used for testing of an autonomous system, in accordance with one or more embodiments of the present invention!

[0024] FIG 3 is an architecture of data driven methods for creating physically realistic systems or objects in the metaverse! and

[0025] FIG 4A & FIG 4B are a depiction of an exemplary embodiment of diagnosing and learning from failed test cases using metaverse, in accordance with one or more embodiments of the present invention.

[0026] Hereinafter, embodiments for carrying out the present invention are described in detail. The various embodiments are described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It may be evident that such embodiments may be practiced without these specific details.

[0027] The present invention discloses a novel approach to optimize interaction steps by incorporating intelligent guidance mechanisms. The present invention addresses this challenge by introducing a solution that optimizes both system capabilities and user involvement. By leveraging system intelligence, the invention facilitates a balanced approach that minimizes energy consumption while ensuring effective problem - solving.

[0028] FIG 1A is a block diagram representation of a system 100 for optimizing one or more computer simulated environments 102 used for testing of an autonomous system, in accordance with one or more embodiments of the present invention. It may be appreciated that the system 100 described herein may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. One or more of the present embodiments may take a form of a computer program product comprising program modules accessible from computer-usable or computer-readable medium storing program code for use by or in connection with one or more computers, processors, or instruction execution system. For the purpose of this description, a computer-usable or computer- readable medium may be any apparatus that may contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium may be electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation mediums in and of themselves as signal carriers are not included in the definition of physical computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, random access memory (RAM), a read only memory (ROM), a rigid magnetic disk and optical disk such as compact disk read-only memory (CD- ROM), compact disk read / write, and digital versatile disc (DVD). Both processors and program code for implementing each aspect of the technology may be centralized or distributed (or a combination thereof) as known to those skilled in the art. The system 100 comprises a computer simulated environment 102, one or more data acquisition devices 104-1 to 104-N, a knowledge database 106, and an apparatus 110 communicating over a network 108. The computer simulated environment 102 is virtual representation of a real or physical world. It can be understood as a virtual world representing one or more elements / entities of the real-world such as assets, machines, robots, operators, workers, operators, objects, etc. The computer-simulated environment 102 is accessible by a user, i.e., it is accessible from the real / physical world. In particular, the computer- simulated environment 102 can be understood as the “metaverse”. It is also possible to interact with the computer-simulated environment 102, i.e., to influence or use processes, components and / or functions in the computer- simulated environment 102. The user or the avatar may interact with the objects rendered in the metaverse.

[0029] For example, it is possible that a user can access the computer-simulated environment 102 via an interface, e.g., a virtual reality (VR) or augmented reality (AR) interface. For the purpose of this invention, the metaverse is comprised of one or more animated scenes being rendered corresponding to the plurality of entities interacting in the industrial environment. The metaverse may comprise a plurality of computer-simulated components. The computer simulated components can for example be understood as a representation, in particular a 3D or virtual representation, of a real or physical component. A component can for example be a room, a building, an item, or an object. The computer-simulated component can have different functionalities / features, e.g., an access interface. The metaverse can be realized by a hosting environment. The hosting environment can be for example be implemented as a cloud environment, an edge-cloud environment and / or on specific devices, e.g., mobile devices.

[0030] Throughout the present disclosure, the terms similar to “acquisition device” 104- 1 and “one or more data acquisition devices” 104-1, 104-2, ... 104-N refers to any electronic device configured for capturing, acquiring, and transmitting data from a user to the apparatus. The acquisition device may take various forms depending on the specific application of the system 100. In an example, the acquisition device 104-1 includes one or more of a keyboard, a mouse, a touchscreen display, a microphone, a camera, or any other hardware component that enables the user to interact with the system 200. In another example, the acquisition device 104-1 may comprise of multiple modes of acquisition of the data in addition to a user input and therefore the acquisition devices 104-1, 104- 2, ... 104-N may comprises of a combination of voice assistants, specialized input devices, data entry devices, touch screen interfaces, camera modules, digital cameras, surveillance cameras, aerial imaging devices, 3D imaging devices, wearable devices and the like. For the simplicity of this invention, we may refer to a single acquisition device 104-1.

[0031] Throughout the present disclosure, the term “knowledge database” (interchangeably used as “database”) 106 refers to a knowledge database having metaverse ontologies, a plurality of generic metaverses, platform configuration and a preconfigured knowledge graphs derived from a knowledge graph repository comprising information pertaining to one or more real-world scenes. The knowledge base 106 may be a structured or non-structured collection of data or information stored in a computer-readable format. The knowledge base 106 comprises one or more tables, each consisting of rows and columns, where data is organized, stored, and managed. The data within the database can include text, numbers, images, audio, video, or any other form of electronic data. The knowledge base 106 is designed to efficiently store, retrieve, and manipulate large volumes of data, enabling rapid access and retrieval of information for various applications. It provides a centralized repository for organizing and managing data, facilitating data integrity, consistency, and security. The data within the knowledge base 106 can be accessed, modified, or queried through a database management system (DBMS), which provides a set of software tools and interfaces to interact with the database. The DBMS allows users or applications to perform operations such as inserting, updating, deleting, or searching for data within the knowledge base 106. The knowledge base 106 is a database which may be of various types, including but not limited to relational databases, NoSQL databases, distributed databases, in-memory databases.

[0032] Throughout the present the term “knowledge graph repository” refers to a data storage system designed to manage and organize structured data in the form of a knowledge graph. The repository comprises a collection of interconnected nodes and edges, where nodes represent entities or objects, and edges represent relationships or associations between these entities. The knowledge graph repository in the know is built to efficiently store, retrieve, and query large volumes of interconnected data, facilitating the representation and exploration of complex relationships and dependencies among entities. The repository is equipped with query and traversal capabilities that allow users or applications to perform operations such as searching, navigating, or reasoning over the interconnected data. It enables the retrieval of relevant information, inferencing of new relationships, and identification of patterns or insights within the knowledge graph. The knowledge graph repository in the knowledge database 106 may support various data models, including but not limited to property graph model, RDF (Resource Description Framework) model, hybrid models and so forth. In the context of the present invention, the knowledge graph repository stores the first knowledge graph developed based on the semantic binding and the knowledge graphs generated in all the next iterations.

[0033] In one embodiment, the apparatus 110 is deployed in a cloud computing environment. As used herein, “cloud computing environment” refers to a processing environment comprising configurable computing physical and logical resources, for example, networks, servers, storage, applications, services, etc., and data distributed over the network 108, for example, the internet. The cloud computing environment provides on-demand network access to a shared pool of the configurable computing physical and logical resources. The network 108 is referred as a distributed network throughout the invention. The apparatus 110 may include a module for managing access control for a plurality of metaverses or digital twins interacting in a computer simulated collaborative environment over a distributed network.

[0034] Particularly, the system 100 comprises a cloud computing device configured for providing cloud services for managing access control for a plurality of data acquisition devices 104-1 to 104-N interacting in a computer simulated collaborative environment over a distributed network. The cloud computing device comprises a cloud communication interface, a cloud computing hardware and OS, and a cloud computing platform. The cloud computing hardware and OS may include one or more servers on which an operating system (OS) is installed and includes one or more processing units, one or more storage devices for storing data, and other peripherals required for providing cloud computing functionality. The cloud computing platform is a platform which implements functionalities such as data storage, data analysis, data visuahzation, data communication on the cloud hardware and OS via APIs and algorithms! and dehvers the aforementioned cloud services using cloud-based applications.

[0035] FIG IB is a block diagram of an exemplary apparatus 110 for optimizing one or more computer simulated environments 102 used for testing of an autonomous system, according to an embodiment of the present invention. In an exemplary embodiment, the apparatus 110 is communicatively coupled, using a distributed network 108, to one or more data acquisition devices 104-1 to 104-N, the knowledge database 106, and the apparatus 110 to render the computer simulated environment 102.

[0036] The apparatus 110 may be a personal computer, a laptop computer, a tablet, a server, a virtual machine, and the like. The apparatus 110 includes a processing unit 112, a storage unit 114 comprising a database 116, a bus 118, a memory unit 120 comprising a module 124, and an input / output (I / O) unit 122.

[0037] The processing unit 112 as used herein, means any type of computational circuit, such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing microprocessor, reduced instruction set computing microprocessor, very long instruction word microprocessor, explicitly parallel instruction computing microprocessor, graphics processor, digital signal processor, or any other type of processing circuit. The processing unit 112 may also include embedded controllers, such as generic or programmable logic devices or arrays, application specific integrated circuits, single-chip computers, and the like.

[0038] The storage unit 114 comprises the database 116 for storing feedback data. The storage unit 114 and / or database 116 may be provided using various types of storage technologies, such as solid-state drives, hard disk drives, flash memory, and may be stored in various formats, such as relational databases, nonrelational databases, flat files, spreadsheets, and extended markup files, etc. The memory unit 120 may be non-transitory volatile memory and / or non-volatile memory. The memory unit 120 may be coupled for communication with the processing unit 112, such as being a computer-readable storage medium. The processing unit 112 may execute instructions and / or code stored in the memory unit 120. A variety of computer-readable instructions may be stored in and accessed from the memory unit 120. The memory unit 120 may include any suitable elements for storing data and machine-readable instructions, such as read only memory, random access memory, erasable programmable read only memory, electrically erasable programmable read only memory, a hard drive, a removable media drive for handling compact disks, digital video disks, diskettes, magnetic tape cartridges, memory cards, and the like.

[0039] In the present embodiment, the memory unit 120 includes the module 124 stored in the form of machine-readable instructions on any of the above-mentioned storage media and may be in communication to and executed by the processing unit 112. When the machine-readable instructions are executed by the processing unit 112, the module 124 causes the processing unit 112 to efficiently render one or more scenes in a computer simulated environment.

[0040] The module 124 comprises of a virtual representation (VR) generation module 124-1, a simulation module 124-2, an identification module 124-3, a learning module 124-4, a deployment module 124-5 and an analysis module 124-6.

[0041] The VR generation module 124-1 is configured to generate realistic digital replicas of failed test cases within a computer-simulated environment by leveraging real-world data to model and replicate physical phenomena accurately. The process begins with the module receiving failed test results from various sources, such as SIL, HIL, or physical tests, which include sensor readings, error logs, and environmental conditions. To enhance the virtual representation, data is collected from multiple sensors, including vibration, touch, temperature, cameras, LiDAR, force, fluid, taste, odor, GPS, and IMU sensors. Using a processing unit, the module integrates these failed test results with the sensor data, employing joint distribution modeling to capture trends and interaction effects, and physics-based modeling to ensure the virtual environment mirrors real-world physics. Statistical similarity is maintained through techniques like Copula, Generative Al, and Feature Embeddings, with evaluations using multivariate tests, deep learning, and correlation analyses. The VR generation module 124-1 may also include an analysis to validate the joint effects of different physical phenomena, ensuring accurate representation of real-world correlations, such as increased motor vibrations leading to higher temperatures. Therefore, resulting in a highly realistic and interactive virtual representation of each failed test case, thus improving the understanding and reliability of autonomous systems.

[0042] The simulation module 124-2 is configured to simulate the execution of a failed test case within a virtual representation of an autonomous system, accurately replicating the system's behavior as observed during the original test case. The processing unit simulates the execution of the failed test case within the virtual representation. The simulation module 124-2 further comprises enabling experts to interact with the virtual model, identify the root cause of the sensor anomalies, adjust environmental parameters, and test different braking scenarios to develop corrective measures, ultimately enhancing the vehicle's reliability and safety. Furthermore, the simulation module 124-2 enables realistic object manipulation behaviours, including physics-based interactions and animations.

[0043] The identification module 124-3 is configured to analyze the simulated execution of failed test cases within a virtual representation of an autonomous system, identifying deviations from expected behavior to diagnose the root causes of failures. The analysis process begins by receiving detailed execution data from the simulation module 124-2, which includes sensor readings, system responses, and environmental interactions. The identification module 124-3 establishes a baseline of expected behavior derived from performance standards, historical data, and theoretical models. Using a processing unit, the identification module 124-3 analyzes the simulated data through comparative analysis and pattern recognition to detect discrepancies. Specific deviation data points, such as sensor anomalies, system response irregularities, and environmental interaction discrepancies, are identified as potential indicators of underlying issues. Root cause analysis is conducted to determine the factors contributing to these deviations, involving correlation analysis and impact assessment. The findings are compiled into diagnostic reports, providing valuable insights for experts to develop corrective measures and improve system reliability. For instance, if an autonomous drone exhibits unexpected altitude changes during a simulated navigation test, the module identifies anomalies in IMU and GPS sensor readings, correlating them with irregular force sensor readings, and generates a report to address the mechanical faults, enhancing navigation reliability.

[0044] The learning module 124-4 is configured to enhance the accuracy and reliability of the computer-simulated environment by using deviation data and expert inputs to generate a feedback-based learning algorithm. The learning module 124-4 receives deviation data from the identification module, pinpointing where the simulated execution deviated from expected behavior, and incorporates expert inputs that provide insights and recommendations. The learning module 124-4 further comprises developing a feedback-based learning algorithm that integrates this data to adjust simulation parameters and update models, thereby improving the virtual representation. Through iterative refinement, the algorithm continuously trains the simulated environment, ensuring each cycle brings the virtual representation closer to the expected behavior. The updated simulation undergoes rigorous validation and testing, including comparative testing and performance metrics evaluation, to measure improvements. The learning module 124'4 maintains a feedback loop for continuous improvement, adapting the simulation as new test cases and expert feedback are received. For instance, in an autonomous vehicle simulation, deviations in obstacle detection are corrected by integrating expert feedback on LiDAR sensor models and response algorithms, resulting in a more accurate and reliable simulation that continuously improves with ongoing feedback and data.

[0045] The deployment module 124-5 is configured to deploy a trained computer- simulated environments onto a test execution platform for testing an autonomous system. It starts by receiving the trained models from the learning module 124'4, refined using deviation data and expert inputs. The deployment module 124-5 prepares these models by packaging necessary components and configuring test parameters, then deploys them onto a test execution platform, such as a HIL, SIL system, or physical testbed. The deployment module 124'5 further involves seamless integration and initialization of the simulation environment. The deployed environment is used to execute predefined test cases, monitoring the system-under-test associated with the autonomous system, and recording performance data. The deployment module 124-5 evaluates the test results through comparative analysis and performance metrics, identifying any deviations and ensuring accurate behavior. It establishes a feedback loop with the learning module 124-4, feeding test results back into the simulation refinement process for continuous improvement. For example, in testing an autonomous drone's navigation system, the module deploys trained models onto a HIL platform, subjects the drone to various scenarios, monitors its performance, and feeds back results for ongoing enhancement, validating improved obstacle detection and avoidance behaviors.

[0046] The analysis module 124-6 is configured to perform a comprehensive analysis of the simulated executions of failed test cases to identify root causes of failures and provide actionable insights for improvement. The analysis module 124-6 comprises receiving detailed simulation data, including various input parameters, system settings, and external factors. The analysis module 124-6 is further configured to conduct counterfactual analysis by simulating hypothetical changes to these variables and evaluating their effects on the test outcomes, helping identify critical factors and potential preventive measures. Simultaneously, causality analysis is performed using statistical techniques to determine relationships between different variables and events leading to failures. The analysis identifies key variables and their interactions that contributed to the failure. The findings from both analyses are compiled into detailed reports, highlighting identified causes of failure, the impact of hypothetical changes, and the relationships between variables. These reports provide actionable recommendations for system adjustments, design changes, and training enhancements to improve the autonomous system's performance and reliability. For example, in an autonomous vehicle simulation, the analysis module 124-6 identifies that improving LiDAR sensor calibration and addressing low visibility conditions can significantly reduce obstacle detection errors, providing targeted recommendations for system improvement.

[0047] The processing unit 112 is configured for performing all the functionality of the module 124. The processing unit 112 is configured to aggregate real-world data from diverse sensors to construct comprehensive virtual representations of the system and its environment. The processing unit 112 then executes test scenarios, including failure cases, to replicate system behavior and detect deviations from expected performance. Through root cause analysis, it scrutinizes these deviations to unveil underlying issues. Additionally, the processing unit 112 conducts counterfactual and causality analyses to assess hypothetical changes and comprehend variable interactions. The processing unit 112 additionally subsequently formulates feedback-based learning algorithms to refine and enhance the simulation models, ensuring a process of continuous improvement. Finally, the processing unit 112 readies and deploys these models onto a test execution platform, meticulously monitoring and evaluating test results to validate system enhancements and sustain an iterative feedback loop. For instance, in the testing of an autonomous drone, the processing unit 112 orchestrates navigation failure simulations, identifies sensor misalignments, refines the models, and validates improvements through hardware-in-the-loop testing, guaranteeing precise and dependable performance enhancements.

[0048] The I / O unit 122 may provide ports to receive input from input devices such as monitor, keypad, touch -sensitive display, camera (such as a camera receiving gesture-based inputs), head mounted devices, etc. capable of receiving a primary input based on user's expressed or inferred intent, which may be provided through direct input (e.g., text input, voice commands) or inferred from user behavior and context. Additionally, the input unit is configured to acquire information about the user, including preferences, past interactions, demographic data, and other relevant attributes, helps personalize the options provided by the system. The I / O unit 122 may provide ports to output data via output device with a graphical user interface for displaying one or more options for actions in the computer simulated virtual environment. The bus 118 acts as interconnect between the processing unit 112, the storage unit 114, the memory unit 120, and thel / O unit 122.

[0049] Those of ordinary skilled in the art will appreciate that the hardware depicted in FIG IB may vary for implementations. For example, other peripheral devices such as an optical disk drive and the like, Local Area Network (LAN) / Wide Area Network (WAN) / Wireless (e.g., Wi-Fi) adapter, graphics adapter, disk controller, input / output (I / O) adapter also may be used in addition to or in place of the hardware depicted. The depicted example is provided for the purpose of explanation only and is not meant to imply architectural hmitations with respect to the present disclosure.

[0050] The present invention is not limited to a particular computer system platform, processing unit, operating system, or network. One or more aspects of the present invention is distributed among one or more computer systems, for example, servers configured to provide one or more services to one or more client computers, or to perform a complete task in a distributed system. For example, one or more aspects of the present invention is performed on a client-server system that includes components distributed among one or more server systems that perform multiple functions according to various embodiments. These components comprise, for example, executable, intermediate, or interpreted code, which communicate over a network using a communication protocol. The present invention is not limited to be executable on any particular system or group of system, and is not limited to any particular distributed architecture, network, or communication protocol.

[0051] Referring now to FIG 2, illustrated is a flowchart of a method 200 as described hereinafter may be executed in the apparatus 110, or specifically in the processing unit 112 of the apparatus 110, for optimizing one or more computer simulated environments 102 used for testing of an autonomous system, in an industrial metaverse increasing sustainability.

[0052] In one or more embodiments of the present invention, at step 202, the method 200 includes receiving one or more failed test results originating from various first sources. The one or more failed results serve as critical indicators of potential deficiencies or malfunctions within the autonomous system under examination. Based on the received data, the processing unit 112 lays a groundwork for a thorough analysis aimed at enhancing the system's performance and reliability. Leveraging the received failed test results, the processing unit 112 proceeds to construct a virtual representation of each failed test case within a computer simulated environment. This virtual environment serves as a controlled space where the behaviour and responses of the autonomous system can be meticulously observed and evaluated. In one or more embodiments of the present invention, at step 204 of the method 200, the processing unit 112 then proceeds to simulate the execution of the failed test case within the virtual representation of the autonomous system. During this step, the processing unit 112 replicates the behaviour of the autonomous system as it was observed during the original test case. This simulation process involves executing the same sequences of actions and responses, thereby providing a controlled and repeatable environment for detailed examination and analysis.

[0053] In one or more embodiments of the present invention, at step 206 of the method 200, the processing unit 112 analyses the simulated execution to identify deviation data from the expected behaviour of the autonomous system. The analysis involves comparing the simulated behaviour with the predefined expected outcomes to detect any discrepancies or anomalies. The identification of these deviations is critical for understanding the underlying causes of the system's failures and for informing the subsequent optimization processes.

[0054] In one or more embodiments of the present invention, at step 204 of the method 200, the processing unit 112 generates a feedback-based learning algorithm based on the identified deviation data and expert inputs received from a second source. This algorithm is designed to iteratively improve the computer simulated environment by incorporating the deviation data and expert insights. The learning process adjusts the simulation parameters and models to reflect real- world conditions more accurately and to mitigate the identified issues, thereby enhancing the overall performance and reliability of the autonomous system.

[0055] In one or more embodiments of the present invention, at step 205 of the method 200, the processing unit 112 deploys the optimized environment onto a test execution platform. This platform is used for testing the system-under-test, which is associated with the autonomous system. The deployment ensures that the refined models and simulations are applied in real-world testing scenarios, allowing for rigorous validation of the system's capabilities and performance. The step 205 finalizes the method by enabling the autonomous system to be tested under enhanced and optimized conditions, thereby ensuring greater reliability and effectiveness in real-world applications.

[0056] Therefore, the proposed solution provides improved mechanism of optimization of interaction steps in industrial metaverse environments, thereby reducing energy consumption and carbon footprint. In the existing solution, heavy processing and huge amount of storage of data that results in extensive energy consumption. The options provided to the user / avatar based on the determination of least number of steps required to complete a specific task or complete one or more action aids in the reduced energy consumption and carbon footprint, increased sustainability, enhanced efficiency and resource optimization, improved user experience, and adaptability to changing conditions. The method provided in the proposed disclosure reduces the energy consumption in a big way and the manual mistakes happening during training are drastically minimized. As a result, the proposed solution is simple and can be easily, effectively implemented with enhanced efficiency.

[0057] Illustrated in FIG 3 is an architecture of data driven methods for creating physically realistic systems or objects in the metaverse. The workflow begins with extensive data collection using a variety of sensors to monitor different physical phenomena. The figure showcases a workflow that starts with a large number of sensors 300-1 to 300'10 collecting data related to different physical phenomena from the system / object and its environment. These sensors may include, but are not limited to:

[0058] Vibration Sensors 300-1 configured to capture infrasonic, audible, and ultrasonic frequencies to monitor vibrations.

[0059] Touch Sensors 300-2 configured to detect contact between material surfaces through electromagnetic interaction.

[0060] Temperature Sensors 300-3 configured to measure heat transfer rates through conduction, convection, and radiation.

[0061] Cameras 300-4 configured to perceiving colour and images. LiDAR 300'5 configured to provide depth information.

[0062] Force Sensors 300'6 configured to measure stress and strain.

[0063] Fluid Sensors 300'7 configured to monitor properties of air, water, and other fluids.

[0064] Taste Sensors 300'8 configured to perceive human taste.

[0065] Odor Sensors 300'9 configured to detect smell in the environment.

[0066] GPS and IMU 300'10 Providing spatial location and orientation data.

[0067] Additional sensors and data collection devices may be employed to capture other physical phenomena as required, and the framework can be adapted to incorporate these additional inputs. Using the highly heterogeneous data from these sensors through a data-driven approach, a computer simulated model is created to demonstrate the complex physical phenomena of an autonomous system. A joint distribution modeler is employed to capture trends, patterns, and interaction effects between different physical phenomena, thereby enabling computational use of the joint mathematical models in the digital world.

[0068] A fundamental criterion for training an Artificial Intelligent (Al) model is based on novel quantitative difference modules 314 between the real-world system / object and its generated digital twin. Another module involves the representativeness of the digital twin to the physics 312 expressed by the real- world system / object and its environment. The physics model data 302, comprising of a module to model physics interaction with environment, and data driven model 304, comprising of a module for Al based modelling, is subsequently modelled using sensor data in a joint fashion to create a realistic replication of system properties and behaviours. The key to achieving realistic replication lies in modules that capture the physical representativeness and statistical similarity between the real-world data and the digital twin model. Additionally, a phenomena factorial analysis module 306 is incorporated to capture and model the joint effects of the physics model data 302 and data driven model 304 into a joint hybrid model 308. Correlations between these phenomena validate the physics of the digital model. For example, if vibrations in a motor increase, there should be a corresponding increase in heat dissipation and recorded temperature. Thereafter, the processing unit 112 instantiates a digital twin of the autonomous system. This digital twin 310 is a highly detailed and interactive virtual model that mirrors the physical system and its environment, forming the basis of an industrial metaverse. The digital twin 310 incorporates the comprehensive data captured during the phenomena factorial analysis, ensuring that all relevant physical phenomena and their interactions are accurately represented. The instantiated digital twin is then prepared for integration into an industrial metaverse 316. This preparation involves enhancing the digital twin 310 with interactive capabilities, allowing human experts to immerse themselves in the virtual environment.

[0069] Illustrated in FIG 4A and FIG 4B are depictions of an exemplary embodiment of diagnosing and learning from failed test cases using metaverse, in accordance with one or more embodiments of the present invention, as outlined in one or more embodiments of the present invention. To ensure the safety, reliability, and efficiency of autonomous systems, they are rigorously tested using X-in-thedoop (XIL) methodologies, where X can be software, hardware, system, etc. Among these, the software-in-thedoop (SIL) methodology is the most cost-effective and safe. However, identifying the root cause of system failures in some SIL tests is challenging due to the complex nature of these systems, their operating environments, and the limited fidelity and immersiveness of simulations. The industrial metaverse, which is an immersive, interactive digital twin of the cyberphysical system and its environment, offers a cost-effective, safe, detailed, and repeatable way to investigate the root cause of a failed test case.

[0070] Fig 4A illustrates a typical SIL-based testing setup where the autonomous system is subjected to various test cases in a simulated environment. First, a plurality of test case scenarios 402 is meticulously defined to cover a range of operational conditions, including normal operations, edge cases, and stress conditions. The test cases are meticulously designed to cover a wide range of operational scenarios the autonomous system 404 might encounter. These scenarios have specific objectives, such as testing the system's response to obstacles, navigation accuracy, or decision-making in complex environments. Each test case is run on the autonomous system and generate a list of failed test case results 408.

[0071] Illustrated in FIG 4B is a method to diagnose and learn from failed test cases using metaverse. Using the method described in FIG 2, a metaverse is created with a digital twin of the autonomous system 404 in question. Using advanced simulation tools, a virtual environment 406 that closely mimics real-world conditions is created using user input via an input device 400, for example a 3D glasses. This virtual environment includes various elements like roads, buildings, pedestrians, weather conditions, and other vehicles. The autonomous system's digital twin or software model is loaded into the simulation environment. The model replicates the autonomous system's hardware and software components. The defined test scenarios are executed in the simulated environment. During these simulations, the autonomous system operates as it would in the real world, using its sensors, algorithms, and actuators to navigate and make decisions. Throughout the simulations, the autonomous system's behavior is closely monitored. This includes tracking sensor inputs, decision-making processes, actuator responses, and overall system performance. Detailed logs are maintained for each test case, capturing data such as sensor readings, system states, decision paths, and actions taken. The outcomes of each test case are recorded, noting whether the system successfully completed the scenario or encountered issues. The performance data and recorded outcomes are analyzed to identify any failures. Failures could include collisions, navigation errors, unexpected behaviors, or any deviations from the expected outcomes. For each identified failure, a detailed test report 410 is generated comprising a failure mode and a set of design changes required to be applied for the test case not to fail. Based on the received detailed test report, the autonomous system is redesigned 412. The redesigned autonomous system 412 is tested 414 with the same sample test case. Based on the results the failed test reports 408 is updated. The failed test reports 408 are compiled into a comprehensive set that provides an overview of the autonomous system's performance across all test cases. An exemplary example of a test case workflow begins with the scenario definition, such as testing a self-driving car's response to a pedestrian suddenly crossing the road. The environment setup involves creating a virtual city street with pedestrians, vehicles, and traffic signals. During the simulation run, the autonomous car navigates the street and encounters the pedestrian. Sensors track the pedestrian, and the car's decision-making process is logged. If the car fails to stop in time, resulting in a virtual collision, all sensor data, decision points, and actions taken are logged. Failure analysis is then conducted to determine why the car failed to stop. Finally, a test report 410 is generated, detailing the scenario, behavior, and identified issue and the steps required to redesign the autonomous system.

[0072] The present invention discloses a system and method for optimizing interactions through user in the loop by objective journey mapping, in an Industrial metaverse scenario. The present invention relates to a system and method for optimizing interaction steps within industrial metaverse environments to reduce energy consumption, minimize carbon footprint, and increase sustain ability. In industrial metaverse scenarios characterized by continuous activity and multiple users, each interaction step contributes to energy consumption and carbon emissions. The invention addresses this challenge by intelligently guiding users through an objective journey, thereby optimizing interaction steps and reducing environmental impact. By streamlining processes, enhancing efficiency, and promoting sustainable practices, the invention improves user experience and supports long-term viability within industrial metaverse ecosystems.

[0073] The proposed system and method for diagnosing and learning from failed test case scenarios in a metaverse offer numerous advantages for training autonomous systems. By leveraging expert knowledge and a sophisticated virtual environment, this approach enhances the accuracy, reliability, and efficiency of autonomous system training. The system allows for detailed analysis of failed test cases within a virtual environment that accurately replicates real-world conditions. The hands-on approach enables experts to manipulate various parameters and observe the outcomes, leading to deeper insights and more effective problem-solving. Simulating failure scenarios in the Metaverse eliminates the risks and costs associated with real-world testing. Therefore, providing safe, virtual environment allows for extensive testing and experimentation without the potential for damage to physical systems or harm to operators. By incorporating expert inputs into the feedback-based learning algorithms, the system leverages human expertise to enhance the training process. Further, the system supports an iterative learning process, where each round of diagnosis and feedback leads to incremental improvements in the autonomous system. The continuous refinement ensures that the system evolves and adapts to new challenges and scenarios over time. The virtual environment and data-driven approach are highly scalable, allowing for the simulation and analysis of a wide range of failure scenarios. The system can be adapted to different types of autonomous systems and varying conditions, making it a versatile tool for training. Furthermore, the system performs counterfactual and causality analyses, enabling experts to evaluate hypothetical changes and understand the relationships between different variables and events. This deep analysis helps in identifying the most critical factors influencing system failures and developing targeted solutions.

[0074] While the present invention has been described in detail with reference to certain embodiments, it should be appreciated that the present invention is not limited to those embodiments. In view of the present invention, many modifications and variations would be present themselves, to those skilled in the art without departing from the scope of the various embodiments of the present invention, as described herein. The scope of the present invention is, therefore, indicated by the following claims rather than by the foregoing description. All changes, modifications, and variations coming within the meaning and range of equivalency of the claims are to be considered within their scope.

[0075] List of references

[0076] 100 a system

[0077] 102 a computer simulated environment

[0078] 104-1 to 104-N data acquisition devices

[0079] 106 a knowledge database

[0080] 108 a communication network

[0081] 110 an apparatus

[0082] 112 processing unit

[0083] 114 a storage unit

[0084] 116 a database

[0085] 118 a bus

[0086] 120 a memory unit

[0087] 122 an input unit

[0088] 124 a module

[0089] 124-1 a virtual representation (VR) generation module

[0090] 124-2 a simulation module

[0091] 124-3 an identification module

[0092] 124-4 a learning module

[0093] 124-5 a deployment module

[0094] 124-6 an analysis module

[0095] 200 method

[0096] 300 an architecture of data driven methods for creating physically realistic systems or objects in the metaverse

[0097] 300-1 to 300-10 sensors

[0098] 302 physics model data

[0099] 304 data driven model

[0100] 306 phenomena factorial analysis module

[0101] 308 joint hybrid model

[0102] 310 interactive digital twin

[0103] 312 module for representativeness of physics

[0104] 314 quantitative difference modules

[0105] 316 metaverse

[0106] 400 user / expert

[0107] 402 test case scenarios 404 autonomous systems

[0108] 406 a virtual environment

[0109] 408 a list of failed test case results

[0110] 410 a detailed test report 412 redesigned autonomous system

[0111] 414 testing of the redesigned autonomous system

Claims

CLAIMS1. A computer-implemented method (200) for optimizing one or more computer simulated environments (102) used for testing of an autonomous system (404), the method comprising: receiving, by a processing unit (112), one or more failed test results of one or more failed test cases from one or more first sources! generating, by the processing unit (112), a virtual representation of each failed test case of the one or more failed test cases in a computer simulated environment based on the received one or more failed test results of the failed test cases! simulating, by the processing unit (112), the execution of the failed test case within the virtual representation of the autonomous system (404), wherein the simulated execution replicates the behavior of the autonomous system (404) during an original test case! analyzing, by the processing unit (112), the simulated execution to identify deviation data from an expected behavior of the autonomous system (404); generating, by the processing unit (112), a feedback-based learning algorithm to train the computer simulated environment based on the deviation data and expert inputs received from a second source! and deploying, upon training, by the processing unit (112), the feedback based learning algorithm onto a test execution platform for testing of a system-under- test, wherein the system-under-test is associated with the autonomous system (404).

2. The method of claim 1, wherein the virtual representation of the autonomous system (404) within the one or more computer simulated environments (102) includes graphical and behavioral elements that accurately depict a hybrid model of the autonomous system (404) with its surroundings, wherein the hybrid model comprises a sensor data model, a physics model and a data driven model.

3. The method of claim 1 or 2, wherein the generation of the failed test cases data associated with the failed test case is based on the sensor data, controlsignals, system logs, and external stimuli encountered by the autonomous system (404) during the test.

4. The method of any one of claims 1 - 3, wherein the simulated execution of the failed test case includes dynamic interactions with virtual entities, objects, and environmental factors present within the one or more computer simulated environments (102).

5. The method of any one of claims 1 - 4, wherein analysing the simulated execution includes comparing behaviour of the autonomous system (404) during the simulated execution with at least one of predefined performance metrics and expected outcomes.

6. The method of any one of claims 1 - 5, wherein analyzing the simulated execution further comprises performing sensitivity analysis on the deviation data to identify influential variables and parameters affecting the outcome of the failed test case.

7. The method of any one of claims 1 - 6, wherein analyzing the simulated execution further comprises performing counterfactual analysis on the simulated executions of the one or more failed test cases, wherein the counterfactual analysis comprises of simulating hypothetical changes to the input parameters, system settings, or external factors to evaluate their effects on the outcome of the test case.

8. The method of any one of claims 1 - 7, wherein analyzing the simulated execution further comprises performing causality analysis on the one or more failed test results of the failed test cases to determine relationships between different variables and events leading to the failure using statistical techniques.

9. The method of any one of claims 1 - 8, wherein the feedback-based learning algorithm comprises one of Reinforcement Learning (RL) algorithm, Reinforcement Learning using Human Feedback (RLHF) algorithm, and imitation learning algorithm.

10. A simulation apparatus (110) for testing of an autonomous system (404), the simulation apparatus (110) comprising: one or more processing units (112); and a memory unit communicatively coupleOd to the one or more processing units, wherein the memory unit comprises a module (124) stored in the form of machine-readable instructions executable by the one or more processing units (112), wherein the module is configured to perform method steps according to the claims 1 to 9.

11. A system (100) for optimizing one or more computer simulated environments used for testing of an autonomous system (404), the system (100) comprising: at least one acquisition device (104-N); and a simulation apparatus (110) according to claim 10, communicatively coupled to the at least one client, wherein the simulation apparatus (110) is configured for optimizing the computer simulated environments used for testing of autonomous vehicles based on inputs received from the at least one acquisition device (104-N), according to the claims 1 to 9.

12. A computer-program product having machine-readable instructions stored therein, which when executed by one or more processing units (112), cause the processing units to perform a method according to the claims 1 to 9.

Images