Mixed reality laboratory system for interactive simulation and control of electromechanical systems

Abstract

The present invention relates to a mixed reality laboratory system that combines virtual elements with real electronic devices to interactively simulate and control electromechanical systems. The innovative system according to the present invention guarantees accurate sampling times, simplifying communication with electronic devices. Furthermore, the system of the present invention can be advantageously implemented on different platforms, which makes it a comprehensive solution for control engineering education and training, offering a practical and versatile approach to learning and experimenting with complex control systems.

Description

[0001] Mixed Reality Laboratory System for Interactive Control and Simulation of Electromechanical Systems

[0002] Field of Invention

[0003] The present invention is in the field of education and training in control engineering, focusing on the simulation and control of electromechanical and robotic systems. Specifically, the present invention relates to a "Mixed Reality Laboratory System" that combines virtual elements and real-life electronic devices to enable interaction and hands-on learning in an educational and professional environment.

[0004] Furthermore, the present invention addresses the need to ensure accurate sampling times in simulation and simplify communication with electronic devices, which is essential for the fidelity and effectiveness of teaching and experimentation in control systems. The system disclosed herein adapts to a variety of platforms, including laptops, virtual reality headsets, and tablets, expanding its accessibility and utility in different contexts.

[0005] Background of the Invention

[0006] In the context of education and training in the field of control engineering, significant deficiencies have been identified in existing technologies and prior art. These fundamental limitations have hindered the achievement of effective and comprehensive learning in this crucial area.

[0007] One of the key challenges currently faced is the lack of a control laboratory accessible to all students. Current technologies, both in traditional, remote, and virtual laboratories, have not provided a comprehensive solution to address this fundamental issue. Traditional and remote laboratory systems require physical access to laboratory equipment, which limits its availability and accessibility.

[0008] This poses a significant technical challenge, as it creates a barrier for students who cannot access in-person labs due to geographic, time, or resource constraints.

[0009] Additionally, current technologies, and particularly remote and virtual laboratories, have not satisfactorily addressed the need to develop competencies in implementing control algorithms in different real-world devices. The lack of effective interaction between students and physical control devices impedes practical understanding of theoretical concepts. Virtual laboratories focus on pure simulation, without allowing users to interact with physical electronic devices, such as microcontrollers, PLCs, or operational amplifiers, which are essential components in practical applications.

[0010] Based on the above, the technologies known within the state of the art, in general, cannot guarantee the complete learning of a wide variety of theoretical and practical concepts in the field of control engineering.

[0011] Furthermore, the lack of versatile platforms that allow for the simulation of systems with diverse characteristics and complexities has been a critical limitation. Most existing technologies offer some capability to simulate electromechanical, robotic, manufacturing, process, and other systems common in professional practice.However, even though real systems present different complexities such as nonlinearity, MIMO systems, tracking control, perturbations, parametric variations, dead-zone, and bounding, most simulation platforms do not consider these aspects; moreover, it is evident that currently known simulation platforms do not allow or offer a simulated system that can, simultaneously, be altered by the alteration / manipulation of electronic devices physically and functionally connected to the computing system that hosts the simulated system, preventing the user from correctly adjusting the transient behavior of the system under control, thus limiting their learning on the subject.

[0012] The lack of a comprehensive platform for simulating these diverse applications limits the full acquisition of the knowledge and skills needed in this field. Furthermore, the lack of a simulation or simulation system that can be modified in response to changes in specific real-world control components (i.e., via electrical signals)—primarily in real time or essentially simultaneously—results in these simulations lacking realism, potentially undermining user immersion.

[0013] In this sense, it is clear that if the user, when manipulating and / or altering the state of the physical control components to which he has access in the real plane, does not reflect simultaneously - that is, with a considerable delay -, the simulation does not fulfill its main objective of providing an alternative means that faithfully reflects the behavior of the controlled system.

[0014] The scientific disclosure entitled "VIRTUAL AND REMOTE LABORATORIES IN EDUCATION PROCESS AT FCFT STU" published by the IEEE (Institute of Electrical and Electronic Engineers), and published on September 21, 2011, is known within the state of the art, which discloses a virtual and / or remote laboratory implemented as a technical solution for online education.

[0015] The virtual laboratory, according to the teachings of the previously referenced document, describes software technologies based on the most commonly used systems in the design of virtual and remote laboratories, such as MATLAB Web Server, Java, C / C++, and Adobe Flash, as online web applications; clearly, its purpose is to provide mathematical calculation functions and simulations of technological plants.

[0016] Likewise, the previously cited scientific outreach document describes a remote-controlled laboratory that operates or uses real experimental devices.

[0017] However, after reading the material disclosed in this scientific document, it is evident that it does not effectively address the deficiencies identified in the field of control engineering education and training. While this document presents a virtual and remote laboratory approach, a person with ordinary knowledge of the subject will notice that this scientific disclosure does not address the lack of universal accessibility to control laboratories or effective interaction with real electronic devices.

[0018] Furthermore, it does not offer a solution for simulating a wide range of systems with diverse characteristics and complexities. Therefore, the present invention is proposed as a solution to these limitations by effectively addressing the shortcomings existing in current technologies and prior art. The invention focuses on providing a truly accessible control laboratory, the ability to interact with real electronic devices, and the simulation of diverse and complex systems, thus overcoming the deficiencies observed in document D1 and the prior art as a whole.

[0019] In addition to the above, the document "A remote measurement laboratory for educational experiments" published by Elsevier and released on April 20, 1999, is known. It describes a website-based Remote Laboratory (RemLab) designed for remote measurement teaching. This virtual laboratory, called "RemLab", aims to provide remote access to the instrumentation through a conventional web browser.

[0020] In this context, users of the technology described in the aforementioned document use remote instrumentation and download results files for subsequent local processing. In other words, the briefly cited disclosure employs remote instrumentation to modify the state of a virtual laboratory for experimental purposes.

[0021] However, a closer look at the information presented in the scientific paper reveals that it does not effectively address the identified shortcomings in control engineering education and training. While the publication, "A Remote Measurement Laboratory for Educational Experiments," introduces a virtual and remote laboratory approach, it does not address the lack of universal accessibility to control laboratories or effective interaction with real electronic devices. Furthermore, it does not provide a solution for simulating a wide range of systems with diverse characteristics and complexities. This limitation stems from the nature of "RemLab," which does not offer the necessary openness to simulate systems of interest to the diverse needs of users.In this sense, the document does not meet the identified needs since it does not comprehensively address the technical problem of providing a truly accessible control laboratory, the ability to interact with real electronic devices, and the simulation of diverse and complex systems.

[0022] Additionally, the document “a flexible remote laboratory with programmable device under test”, published by Elsevier and disclosed on February 10, 2020, is known within the state of the art, which describes a remote laboratory that incorporates remotely configurable devices, such as a field-programmable analog array (FPAA).

[0023] The FPAA can emulate the behavior of a variety of analog circuits, allowing users to select the desired test circuit and conduct the experiment remotely, for example from their homes.

[0024] However, a closer look at the content of the aforementioned document reveals that, like the documents and technology previously identified, it does not effectively address the identified shortcomings in control engineering education and training.

[0025] Although the document "A Flexible Remote Laboratory with a Programmable Device Under Test" presents a virtual and remote laboratory approach, it clearly does not address the lack of universal accessibility to control laboratories or the lack of effective user interaction with real electronic devices that allow them to alter the state of a virtual environment. Furthermore, it does not provide a comprehensive solution for simulating a wide range of systems with diverse characteristics and complexities.

[0026] It will be clear that, while the aforementioned disclosure recognizes the importance of experiments with real-world instrumentation in engineering education, its proposal for remotely configurable devices does not overcome the limitations identified in the technical problem discussed above. The lack of a comprehensive solution for accessibility, interaction, and simulation of complex systems exposes persistent shortcomings in current technologies and prior art.

[0027] Finally, the document “developing a remote laboratory for engineering education” is known, published by Elsevier and released on March 24, 2011, which refers to a remote control laboratory, also called a virtual laboratory, designed for control systems students.

[0028] This interactive remote laboratory (RL), as its inventors have named it, is based on two main software tools: Simulink and Easy Java Simulations (EJS).

[0029] A person with ordinary knowledge in the field to which this technology belongs will evidently know that the former is a tool widely used by the control engineering community, while the latter is an authoring tool designed to create interactive applications in Java without special programming knowledge. The RL described in the cited document allows users to perform experiments with real equipment from anywhere, anytime and at their own pace, in other words, it provides a virtual laboratory that operates with the aid of real equipment to perform experiments within the same virtual environment, allowing users to carry out said experiments remotely.

[0030] However, after a thorough reading of the disclosure, it becomes clear that it does not effectively address the deficiencies identified in the field of control engineering education and training. Despite presenting a virtual and remote laboratory approach, this document does not address the lack of universal accessibility to control laboratories or effective interaction with real electronic devices.

[0031] Furthermore, it does not offer a comprehensive solution for simulating a wide range of systems with diverse characteristics and complexities.

[0032] Based on the lessons learned from the state of the art, it is clear that, to date, current technology remains a significant challenge, hindering access to various forms of experimentation.

[0033] To date, existing limitations, such as the high cost of the equipment, components, and systems required to conduct experiments in a conventional in-person laboratory setting, have created a significant gap in equitable access to practical training. This financial barrier not only restricts access to experimentation to well-resourced educational institutions but also limits the diversity of experiments that can be conducted, impacting the quality and breadth of practical training.

[0034] Another critical challenge lies in the difficulty of ensuring the active participation of each student in experiments. In common and / or well-known in-person laboratory environments, factors such as insufficient supplies or a large number of students can generate competition for available resources, resulting in a fragmented and exclusionary educational experience.

[0035] Furthermore, geographical limitations constitute a significant obstacle. The reliance on in-person instruction imposes restrictions on students who, for various reasons, cannot be physically present in the laboratory. Factors such as distance, time constraints, or personal restrictions can exclude some students from the practical learning process, depriving them of experiences essential to their development.

[0036] In this context, and based on the above, there is a clear need for a proactive and realistic solution to the technical problem that current technology has not been able to effectively address. That is, there is a need for a system that mitigates the high cost of equipment and overcomes the geographic and access barriers in conventional in-person laboratories, thereby overcoming a considerable obstacle in practical education in engineering and related disciplines. Furthermore, in addition to the existing need for a system as previously mentioned, there is an evident need to implement a methodology that not only addresses the previously mentioned deficiencies in practical engineering education, but also offers innovative and effective solutions to enhance remote learning with the support of the aforementioned system.

[0037] The current state of the art reveals the inherent difficulty in accessing different types of experimentation, highlighting obstacles such as the high cost of equipment, components, and systems required for experimentation in a conventional on-site laboratory.

[0038] Summary of the Invention

[0039] Therefore, one objective of the present invention is to provide a comprehensive solution to the evident limitations in practical engineering education, precisely addressing the deficiencies identified in current technology and prior art. In this context, the invention pursues multiple objectives that are broken down into two crucial dimensions: the system and the methodology.

[0040] More specifically, the present invention's main objective is to offer a remote-controlled laboratory system that overcomes the economic and geographic restrictions associated with traditional in-person laboratories. The system, according to the present invention, seeks to provide students with the ability to interact with real electronic devices from any location, eliminating the barriers imposed by the limited availability of equipment and resources. By overcoming the lack of universal accessibility to remote-controlled laboratories, it establishes itself as an innovative alternative that democratizes access to practical experimentation in engineering.

[0041] Particularly, the present invention aims to provide an innovative mixed reality laboratory, designed to visualize and control elements present in a virtual environment by manipulating controllers existing in the real environment, thus impacting the state of an element and / or in general, of a system represented and / or modeled in the virtual environment, achieving said alteration of the elements or system represented and / or modeled in the virtual environment simultaneously.

[0042] Brief Description of the Figures

[0043] Figure 1 shows a schematic diagram of the system according to the present invention, showing in a simple and practical way the form of interaction between the control elements that the user will be able to manipulate, the computing device - auxiliary means to contain and carry out the simulation of the virtual laboratory according to the present invention and the component called embedded, which establishes communication between the virtual environment and external electronic devices.

[0044] Figure 2 illustrates a series of examples of use of the virtual laboratory according to the present invention, in which the embedded system (hardware and software) is shown functioning as an intermediate element between the user (using virtual reality glasses in this example of use) and, on the other hand, an example of a possible simulation of a virtual laboratory, in which a virtual control element is simulated, generated according to the present invention.

[0045] Figure 3 shows a flowchart associated with the method for generating the virtual laboratory according to the present invention, where in addition, the control logic is shown once the laboratory is in operation and once the user alters or modifies the real control devices.

[0046] Detailed Description of the Invention

[0047] Some aspects of the present invention will now be described in more detail using further reference to the accompanying drawings in which some embodiments and advantages of the present invention are shown.

[0048] It will be apparent to one skilled in the art that various embodiments of the invention may be expressed in different ways and should not be construed as limited to the embodiments described herein; rather, these exemplary embodiments are provided to make this invention clear and complete, and to fully convey the scope of the invention to those skilled in the art. For example, unless otherwise indicated, something described as first, second, or the like should not be construed as implying a particular order. As used in the description and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0049] The different aspects of the present invention relate to a virtual laboratory system which provides users with the ability to interact with real electronic devices and, in turn, view and control elements present in a virtual environment.

[0050] Specifically, the virtual reality system according to the present invention allows, by manipulating controllers existing in the real environment, to modify and / or alter the state of an element and / or in general, of a system represented and / or modeled in the virtual environment, achieving said alteration of the elements or system represented and / or modeled in the virtual environment simultaneously.

[0051] Exemplifying this concept, users can interact with various controllers, such as potentiometers, microcontrollers, switches and drives, thus impacting the state of an element and / or in general, of a system represented and / or modeled in the virtual environment, achieving said alteration of the elements or system represented and / or modeled in the virtual environment simultaneously.

[0052] In the context of the present invention, "element present in the virtual environment" should be understood as devices corresponding to existing physical elements which have been represented in / within the virtual environment.

[0053] On the other hand, "simultaneously" means that the state of the element and / or system represented and / or modeled in the virtual environment will be affected or altered as soon as or directly as a result of any alteration to the control means or controllers to which the user has access in the real world. By way of non-limiting example, it should be understood that if the user alters any of its components, the system will modify its state just as the same system would do in the real world.

[0054] Now, these devices can be represented in the virtual environment using a model developed in CAD.

[0055] This approach seeks to overcome the limitations inherent in conventional hands-on laboratories, such as the high cost of equipment, the need to share resources among students, and the risks associated with experimenting with real systems and components.

[0056] The present invention specifically addresses the restrictions of current practical laboratories by proposing a set of distinctive steps and features: i) Electromechanical and robotic systems are designed in a purely virtual manner using a CAD program.

[0057] (i) The CAD-designed systems are imported into a game engine, particularly Unity, to create the virtual environment. (iii) Through a graphical user interface, users view the purely virtual environment using their available processing devices (laptop, VR headset, mobile devices, among others). (iv) An external controller is used to control the behavior of the virtual system, allowing students to connect the virtual elements to real electrical and control components from their personal devices.

[0058] On the other hand, the present invention aims to provide and implement a unique and effective methodology to enhance remote learning in the field of control engineering using the previously mentioned system.

[0059] This methodology focuses on providing advanced simulation of a wide range of systems, encompassing diverse features and complexities. In doing so, the present invention not only addresses the lack of effective interaction with real electronic devices, but also overcomes the limitations of current technologies by offering a comprehensive and adaptable solution to the evolving demands of technical education. The present invention seeks to provide a totally advantageous and novel way for engineering students to access and participate in practical experiments; primarily, by immersing themselves in a virtual classroom, users have the freedom to explore and navigate within an immersive environment, providing a unique learning experience.

[0060] This innovative design is elevated by incorporating virtual elements, such as robots and electromechanical systems, enriching the simulation and providing a more tangible connection to theoretical concepts.

[0061] The application resulting from this advanced approach can be deployed on a variety of devices, from laptops to virtual reality headsets and tablets, allowing users to access the virtual laboratory flexibly and conveniently. However, the essence of the proposed invention lies in the embedded component, a technological advancement that overcomes the limitations of conventional virtual simulation and facilitates seamless connection with external electronic devices.

[0062] The embedded component, as described and claimed in this application, effectively addresses the complexities associated with sampling times, ensuring the timely delivery of simulation data, thereby ensuring that the simulation in which the user is immersed is substantially superior compared to the prior art, since the user will experience, when manipulating the control elements, how the virtual or virtually represented system is affected based on his actions.

[0063] This solves a common challenge when simulating virtual elements in environments like Unity or Simulink, where time synchronization can be an obstacle. Furthermore, the proposed technology eliminates the need for device-specific protocols, using embedded-generated electrical signals to communicate with a variety of devices, such as microcontrollers, PLCs, and operational amplifiers.

[0064] The integral methodology of the present invention ensures that the data sent and received are perfectly aligned with the required sampling times, essential for the success of control algorithms.

[0065] This approach not only overcomes time constraints but also enhances the connection between theoretical concepts and physical phenomena, providing students with an experience as effective as in-person, hands-on labs.

[0066] With the above, the present invention is focused as a holistic response to the persistent challenges in the teaching of control engineering. Through both its innovative laboratory system and its advanced methodology, the invention transcends current limitations and stands as a catalyst for transformation in the way practical engineering education is approached and as an advantageous solution to the technical problems identified throughout this, being that, the present invention - by effectively addressing the existing deficiencies in current technologies - offers an innovative platform that overcomes the limitations of the prior art, providing a truly accessible control laboratory, the ability to interact with real electronic devices and the simulation of diverse and complex systems.This technology is presented as a response to educational and professional needs in the field of control engineering.

[0067] Some aspects of the present invention will now be described in more detail using further reference to the accompanying drawings in which some embodiments and advantages of the present invention are shown.

[0068] Different aspects of the present invention relate to an Immersive Mixed Reality Laboratory; in particular, the present invention provides a virtual laboratory designed to provide students with an immersive experience. This environment combines virtual and real elements, allowing users to explore a virtual classroom that replicates a physical laboratory environment.

[0069] The immersive mixed reality laboratory, according to the present invention, merges real and virtual elements, providing a unique and enriching experience for engineering students. A clear description of each component that contributes to the uniqueness of this laboratory will be provided below: i) Detailed Virtual Environment: The virtual environment is meticulously created using a computer-aided design (CAD) program. This process ensures the accuracy and fidelity of each simulated element, faithfully replicating a physical laboratory. Users enter a digital space that authentically reproduces the components and devices present in a conventional laboratory environment.

[0070] i) Immersive Interactivity: Interaction within the virtual environment is achieved through an intuitive graphical user interface. Users can explore and manipulate virtual elements, creating an immersive experience that transcends the limitations of conventional learning methods. iii) Explorable Virtual Classroom: The mixed reality laboratory, according to the present invention, includes a virtual classroom that users can explore and navigate through in a manner similar to a physical environment. This virtual classroom serves as the canvas on which the virtual elements are integrated, offering a comprehensive educational experience. Users can navigate through this environment, interact with simulated equipment and manipulate them with electronic devices, as if they were physically present.iv) Replication of Physical Phenomena: The technology used in the laboratory in accordance with the present invention not only provides a high fidelity visual representation, but also accurately replicates physical phenomena.

[0071] In one embodiment, the developed models can be generated with the aid of a CAD system. Subsequently, in one embodiment, the models developed with the CAD system can be imported into a virtual engine and / or generator for simulations, architectural visualizations, and other interactive 2D and 3D applications; in a preferred embodiment, the virtual engine and / or generator can be a video game engine such as Unity or Unity 3d; however, other virtual engines or generators of the video game engine type can be used, such as any selected from the group comprising Unreal Engine, Godot Engine, CryEngine, Lumberyard, Godot Engine, Cocos2d, combinations thereof and / or similar, without departing from the teachings, essence and scope of the present invention.

[0072] In the context of the present invention, virtual engines and / or generators such as video game engines are systems used in the video game development industry, simulations, architectural visualizations, and other interactive applications in 2D and 3D.

[0073] These engines are known for being accessible to novice developers but powerful enough to meet the needs of experienced developers, since, among some of its virtues, it allows the development of games for a wide variety of platforms, including PC, consoles, mobile devices, virtual reality (VR), and augmented reality (AR), offers capabilities for both two-dimensional and three-dimensional development, allowing developers to create a wide range of visual experiences, allows programming in languages ​​such as C# and JavaScript, making development easier for those with different levels of programming experience, provides an integrated development environment that facilitates the creation, testing, and optimization of games.

[0074] Depending on the modality, this virtual motor / generator ensures a realistic simulation of electromechanical and control systems. This allows students to effectively combine theoretical concepts with practical experiences.

[0075] It is worth mentioning that the use of this virtual generation system or virtual motor as listed previously is not the essence of the invention per se, and, additionally, a person with experience in the matter will understand that, although this tool (virtual generator / motor) is used, within the present application and within the scope of protection thereof, it is NOT intended to include the software, program, code, or any other aspect of the virtual generator / motor; on the contrary, it must be clearly understood that, the virtual generator / motor is exclusively an auxiliary means that allows the present invention to be displayed in the physical world.

[0076] For the purposes of being even clearer on the above, it should be understood that, in the context of the present invention, the virtual generator / engine is an auxiliary element, just as the previously described CAD generation tool or system is. These elements depend their operation on a computing and / or processing system, so that, although the intervention of a computer or processing means is necessary to carry out the present invention, in no way should it be understood that the present invention is a computer program; on the contrary, it is an invention that uses a computing or processing means to be able to conduct itself and provide a result (technical effect) in the real world. v) Flexibility in Implementation: The versatility of the virtual laboratory according to the present invention is manifested in its ability to adapt to different devices, such as laptops, virtual reality viewers and tablets.

[0077] In one embodiment, the virtual laboratory, according to the present invention, comprises or interacts through an application designed to display the virtual laboratory, whereby a user can connect the control elements / controllers to his or her device (laptops, virtual reality viewers, tablets, among others) and begin to perform simulations within the space of the virtual laboratory of the present invention.

[0078] Taken together, the immersive mixed reality laboratory of the present invention redefines the way engineering education is approached, providing an educational environment that uniquely blends theory and practice, overcoming the limitations of traditional methods and anticipating the demands of modern education. b) Virtual Elements and Integrated Robots: Within the virtual classroom of the laboratory of the present invention, virtual elements, such as, but not limited to, robots and electromechanical systems, can be integrated to enrich the simulation; likewise, other elements can be incorporated, for example, decorative elements such as, for example, but not limited to, work tables, other static equipment, virtual students and / or the like, this with the purpose of providing an even more realistic environment.

[0079] In a preferred embodiment, these elements are designed with a CAD program and imported into a game engine, particularly Unity, to create a detailed and realistic virtual environment, as described throughout this application.

[0080] Within the immersive environment of the virtual laboratory, component b) of the present invention is highlighted by the integration of virtual elements, such as robots and electromechanical systems. This feature is essential to enrich the simulation and provide users with a more tangible educational experience. The key features of this section are detailed below: i) Precise Design with CAD Program: The virtual elements, which include robots and electromechanical systems, are meticulously designed using a computer-aided design (CAD) program. In one embodiment, the computer-aided design (CAD) programs can be any selected from the group comprising AutoCAD, SolidWorks, CATIA, Fusion 360, ANSYS, Revit, PTC Creo, combinations thereof and / or similar.

[0081] It is worth mentioning that the use of this computer-aided design (CAD) system or programs such as those listed previously is not the essence of the invention per se, and, additionally, a person with experience in the field will understand that, although this tool (CAD) is used, within the present application and within the scope of protection thereof, the software, program, code, or any other aspect of the CAD is NOT intended to be included; on the contrary, it must be clearly understood that the computer-aided design system or program is exclusively an auxiliary means that allows the present invention to be displayed in the physical world.

[0082] For the sake of clarity, it should be understood that, in the context of the present invention, the computer-aided design system or program is an auxiliary element. These elements depend on a computing and / or processing system for their operation. While the intervention of a computer or processing means is necessary to carry out the present invention, it should in no way be understood that the present invention is a computer program. Rather, it is an invention that employs a computing or processing means in order to operate and provide a result (technical effect) in the real world.

[0083] This approach ensures the precision and fidelity of each simulated component, accurately reproducing the geometry and, primarily, in conjunction with the use of a virtual motor / generator as previously described, favoring device behavior in a more realistic and efficient physical environment, which reacts accordingly to the manipulation performed by the user.

[0084] (i) Integration into the virtual engine / generator: Once designed in CAD, the virtual elements are imported and coherently integrated into any of the virtual engines / generators referred to in this application. In a particular or preferred embodiment, the engine / generator is a Unity-type video game engine.

[0085] As previously mentioned, the use of a virtual motor / generator not only facilitates detailed three-dimensional visualization but also enables more realistic interaction with these elements in the virtual environment. iii) Simulation Enrichment: The presence of virtual elements within the laboratory of the present invention, such as robots, adds an additional layer of complexity and realism to the simulation. Users can interact with these elements through electronic devices, allowing for the practical application of theoretical concepts and providing a deeper connection between theory and practice. iv) Replication of Real-World Behaviors: The simulation is not limited to visual representation; virtual elements accurately replicate the physical behaviors of their real-world counterparts.This simulation fidelity allows students to better understand the fundamental principles behind electromechanical and control systems, especially when the simulation directly depends on manipulation by the users themselves. As will be described in more depth later, users will be able to place manipulable elements connected to their device or computing / processing medium, and by modifying these manipulable elements electronically, the virtual laboratory and, more specifically, the simulated systems within it can be altered. v) Contribution to the Educational Experience: The inclusion of virtual elements not only serves an aesthetic purpose but also enhances the overall educational experience. It allows students to actively participate in the manipulation and observation of virtual devices, creating a deeper connection with engineering concepts.c) Versatile Multi-Device Application: The present invention includes a versatile application that can deploy the virtual lab on various devices, such as laptops, virtual reality headsets, and tablets. This allows users to view the lab across different devices, providing flexibility and convenience in the learning process.

[0086] The application that accompanies this invention serves as a fundamental element in maximizing user accessibility and flexibility in their interaction with the virtual laboratory. A clear description of each component that contributes to the uniqueness of this versatile application is provided below: i) Adaptive Platform: The application used in accordance with the present invention has been designed to adapt to a variety of devices, from conventional laptops to virtual reality headsets and the latest generation of tablets. This ensures that students can access the virtual laboratory without hardware restrictions, thus expanding the availability of the educational tool.

[0087] i) Intuitive Interface and Efficient Navigation: The application interface has been developed with a focus on usability, allowing users to fluidly navigate through the virtual environment. Intuitive controls have been implemented that are tailored to the specific platform, ensuring a consistent and comfortable experience for the user, whether on a conventional laptop or in an immersive virtual reality environment. ni) Optimization for Various Operating Platforms: The application has been developed to be compatible with different operating systems, ensuring its efficient execution in environments such as, but not limited to, Windows, macOS, Android, iOS, among others. This technical consideration allows a wide spectrum of users, regardless of their choice of device, to fully take advantage of the capabilities of the virtual laboratory in accordance with the present invention.iv) Continuous Updates and Improvements: The application is conceived as a dynamic tool, subject to periodic updates to incorporate improvements, new functionalities, and adaptations to emerging technologies. This approach ensures that the application evolves with changing educational and technological demands, providing an ever-contemporary educational experience. d) Innovative Embedded System Component (Hardware and Software): According to one embodiment of the present invention, there is a single embedded component, said component being the very essence of the invention.

[0088] This device acts as a bridge between the virtual environment and external electronic devices; in particular, it allows for the resolution of temporal challenges when simulating virtual elements, ensuring the availability of simulation data at specific times. As will become clearer upon a holistic reading of this application, the simulation times provided based on the present invention are advantageously reduced, to such an extent that the virtual system will undergo an alteration in its state immediately (or with a delay percentage that is much lower or close to zero), following the alterations executed by the user.

[0089] The embedded component, the central and innovative element of this invention, is detailed below in depth, highlighting its specific characteristics and essential role in the functionality of the virtual laboratory according to the present invention: i) Technological Bridge: The embedded component, according to one embodiment of the present invention, acts as a crucial technological bridge, efficiently connecting the virtual environment of the laboratory with the user's external electronic devices. This bridge ensures fluid and effective communication between the virtual simulation and the real world.

[0090] In this sense, "the user's external electronic devices" should be understood as the alterable or manipulable physical elements that the user has at his or her disposal. As previously mentioned, these elements are connected to the user's computing / processing medium. Once the user manipulates any element or state of these external electronic devices, these alterations can consequently modify the state of the virtual system.

[0091] i) Resolution of Temporal Challenges: One of the outstanding achievements of this component is its ability to solve temporal challenges inherent to the simulation of virtual elements, in more concise words, said embedded component ensures the timely availability of simulation data by simulating elements at specific intervals, thus overcoming the temporal limitations common in virtually generated environments such as Unity or Simulink (or any of those previously shown and listed in the present application). iii) Adaptive Programming: The embedded component, according to the present invention, has been programmed with an adaptive logic, considering the temporal constraints of different systems. This adaptability is essential to guarantee consistency in the simulation, regardless of the complexity of the system or variations in sampling times.iv) Generation of Electrical Signals: In order to achieve an effective connection with external electronic devices, the embedded component uses electrical signals, which advantageously emulate the communication that would be experienced with sensors of real systems, allowing the connection with devices such as microcontrollers, PLCs and operational amplifiers, with sampling times appropriate for each system, allowing a performance or response as if it were a real component. v) Uniform Communication Facilitator: By eliminating the need for specific protocols for each electronic device, the embedded component according to the present invention stands as a uniform communication facilitator, this characteristic being of utmost relevance since it simplifies the interaction between the virtual environment and a variety of devices, contributing to the versatility of the invention.vi) Time Synchronization: The embedded component's programming focuses on ensuring that the data sent and received are perfectly synchronized with the required sampling times. This aspect is essential for the control and stabilization of the virtual element through control algorithms, ensuring optimal performance in the simulation. vii) Consideration of Time Constraints: In its programming, the embedded component proactively addresses time constraints, ensuring that the electrical signals representing the virtual element's variables are updated at specific sampling periods. This consideration is vital for maintaining consistency in the simulation and interaction with external devices.

[0092] It is worth mentioning that, although “the programming” of the embedded component has been mentioned, it must be understood that programming and everything related to it is not the essence of the invention per se, and, additionally, a person with experience in the matter will understand that, although, the embedded component according to the present invention, given its nature, it is evident that this element comprises a certain programming (based on commands and / or codes), nevertheless, within the present application and within the scope of protection thereof, it is NOT intended to include the software, program, code, or any other aspect inherent to the programming of the embedded component; on the contrary, it must be clearly understood that, this element, in effect, can be denoted as a technical characteristic, since its incorporation and interaction with the rest of the elements that comprise the present invention cause a technical effect, which for this element,is to achieve an effective connection with external electronic devices, being that, as previously mentioned, based on the use of electrical signals, the embedded component of the present invention is advantageously capable of emulating the communication that would be experienced with sensors of real systems, allowing the connection with devices such as microcontrollers, PLCs and operational amplifiers, or any other control element, controllable or manipulable by the user, with a practically zero delay or very close to zero, allowing a performance or response as if it were a real component. e) Connection with External Electronic Devices: The embedded component allows the connection with a variety of external electronic devices, such as microcontrollers,PLCs and operational amplifiers, or any other control element currently used and / or commonly used in practice within the field of control electronics. This is achieved through electrical signals that emulate communication with sensors in real systems.

[0093] The connection between the embedded component and the external electronic devices is presented as a fundamental component of the present invention, providing detailed specificities that highlight its functionality and versatility: i) Wide Variety of Devices: The embedded component, previously described, belonging to the present invention and in accordance with one embodiment thereof, facilitates the connection with a wide variety of external electronic devices, including microcontrollers, PLCs and operational amplifiers or any other control element currently used and / or commonly used in practices within the field of control electronics. This ability to adapt to various devices significantly expands the scope and applicability of the invention in educational and research environments.

[0094] (i) Electrical Signal Interface: The connection with external electronic devices is achieved through the use of electrical signals generated by the embedded component. These signals, carefully designed to emulate communication with real-world sensor systems, ensure an efficient and effective interface between the virtual environment and the external devices. (iii) Elimination of Specific Protocols: The embedded component of the present invention eliminates the need for specific communication protocols for each type of device. This feature considerably simplifies the connection process, since no additional configuration is required for specific devices. Users can easily integrate their own devices without worrying about protocol compatibility. (iv) Data Transfer Security: The connection with external electronic devices is managed with a focus on data transfer security.The embedded component implements measures to guarantee data integrity during transmission, ensuring that information is transferred accurately and without loss of quality. v) Parameter Configuration: Users have the ability to configure specific parameters for connecting to external electronic devices. This feature allows for detailed customization based on system requirements, providing users with flexibility and control in implementing their own experiments and configurations. Therefore, it does not necessarily limit them to preset conditions as is the case with current technology (prior art). vi) Compatibility with Industrial Standards: The embedded component has been configured to ensure compatibility with common industrial standards.This allows the present invention to be compatible with a wide range of electronic devices used in industrial and educational environments, improving its utility and applicability. f) Time Synchronization and Efficient Data Transfer: The invention overcomes time limitations by considering and solving problems associated with the simulation of virtual elements. Time synchronization is achieved through embedding, ensuring efficient data transfer according to the required sampling times.

[0095] Time synchronization and efficient data transfer are fundamental pillars of the present invention, a detailed description of these key features is provided below: i) Sampling Time Management: The embedded component according to the present invention accurately addresses the management of sampling times crucial for the simulation of virtual elements. It establishes a system that guarantees the correct simulation of differential equations, displaying simulation data at specific intervals, either every 10 milliseconds or 0.5 milliseconds, or in a range from 20 to 0.01 milliseconds, adapting to the requirements of the system being worked with.

[0096] (i) Timely Data Availability: In order to ensure the timely availability of simulation data, the embedded component solves challenges inherent to simulation in virtual generator / engine programs, such as Unity or Simulink (or any of those listed in this application), where, in said virtual generators / engines, sampling times can be difficult to guarantee. The technology proposed in this application overcomes this limitation, advantageously providing a constant and accurate flow of simulation data. (iii) Time-Constraint-Adjusted Scheduling: The scheduling of the embedded component is performed considering specific time constraints. This results in a constant update of the electrical signals representing the variables of the virtual element, ensuring that each sampling period is met accurately, essential for the stability and control of the system.iv) Solving Problems Associated with Simulation: The embedded component efficiently solves technical problems associated with the simulation of virtual elements. When simulating within a computer, in virtual generation / engine programs, such as Unity or Simulink (or any of those listed in this application), sampling times can vary. The technology proposed in this application overcomes this challenge, providing a reliable and accurate solution. v) Sampling Time Compliance: The technology according to the present invention not only addresses the variability in sampling times, but also ensures that these times are rigorously adhered to. This guarantee is essential for the effective control and stabilization of the virtual element by control algorithms, ensuring that the values ​​of the variables are updated in synchrony with the system requirements.vi) Adaptability to Different Systems: The time-constraint-adjusted programming of the embedded component allows its adaptability to a variety of systems, from educational laboratory environments to advanced industrial applications. The proposed technology is positioned as a robust and versatile solution that adapts to the specific temporal demands of each implementation. vii) Improved Virtual Element Stability: By ensuring accurate updating of the virtual element variables, the proposed technology directly contributes to the stability and performance of the system. This translates into more accurate simulation and more effective interaction with external electronic devices, generally consolidating the virtues of the present invention.g) Comprehensive Learning Methodology: The invention does not limit itself to offering a virtual environment; its comprehensive methodology enhances the connection between theoretical concepts and physical phenomena. Students can effectively connect theory with practice, achieving an educational experience comparable to in-person practical labs.

[0097] The comprehensive learning methodology is the core that distinguishes and enhances the present invention, and is described in detail: i) Theory-Practice Connection: The present invention goes beyond simply offering a virtual environment; it focuses on bridging the gap between theory and practice. The comprehensive methodology provided by the laboratory of the present invention ensures that students not only visualize theoretical concepts, but also associate and experiment with them in a practical way, thus comparing them to the experience of face-to-face laboratories, consequently eliminating the need for sufficient physical equipment to cover the demand of users (students), since each student can be immersed in their particular virtual environment, collaborate actively and manipulate any state of the simulated system by means of the control devices in their direct environment.

[0098] i) Immersive Educational Experience: The integration of virtual and real elements within the laboratory provides an immersive educational experience. Students not only observe simulations, but also interact with virtual elements such as robots and electromechanical systems, generating a deeper and more practical understanding of the concepts studied. iii) Flexibility and Adaptability to Different Levels: The methodology adapts to different educational levels, from academic institutions to industrial environments. It is flexible in its implementation, allowing its use in various engineering disciplines and adjusting to the specific requirements of each educational context. v) Practical Application of Concepts: The direct connection with external electronic devices, facilitated by the embedded component, allows students to directly apply the learned concepts.The methodology focuses not only on simulation but also on how those concepts translate and apply to real-world environments. vi) Facilitating Interdisciplinarity: The invention is not limited to a specific discipline; its comprehensive methodology facilitates interdisciplinarity by adapting to a variety of engineering fields. This fosters collaboration and holistic learning, preparing students for complex challenges in their future careers. vii) Effective Performance Assessment: The methodology includes built-in assessment tools that allow for effective monitoring of student performance. This not only benefits educators by fine-tuning their pedagogical approach but also empowers students to understand and improve their areas of weakness.

[0099] Together, these elements form an invention that redefines engineering education by providing an immersive, flexible, and technically advanced virtual laboratory.

[0100] Having said that, and after a holistic reading of what is deposited in this application, a person with experience in the field to which the invention belongs will now be able to understand that the coherent integration of mathematical models of dynamic systems in a single simulation, operated in real time by an embedded system and synchronized with the animation of a virtual generator / motor (such as, for example, Unity or any other of those listed in this application), constitutes an extraordinary advance that surpasses conventional solutions.

[0101] In the context of this application, it is crucial to highlight how this unique combination of elements not only addresses obvious challenges but also unexpectedly produces a transformative effect.

[0102] The harmonious interconnection of these elements creates a revolutionary impact by ensuring a perfect correspondence between the system simulation and the real-time animation in the virtual environment. This unexpected achievement transcends conventional capabilities by providing not only an accurate visualization but also a coherent and authentic representation of the system's dynamics, something previously considered extraordinary in current technology.

[0103] The meticulous focus on sampling times, a fundamental requirement for an accurate description of system dynamics, adds another level of detail and accuracy to the simulation. The present invention distinguishes itself by ensuring that sampling times are consistently met, providing a more realistic and accurate simulation of dynamical systems, a vital aspect in applications that demand faithful representations of real-time phenomena.

[0104] The efficient communication of process variables between external controllers and the mixed reality lab is another unexpected and significant component of this invention. Beyond establishing a connection, the technology achieves this goal by strictly adhering to sampling times, which not only enhances the interactive experience but also expands the application possibilities in industrial and educational settings.

[0105] This synergistic interaction between previously imperceptible elements reveals an effect that redefines the capabilities of mixed reality laboratories. The invention not only overcomes previous barriers but also opens new perspectives in the simulation of dynamic systems and interconnection with external devices. This unexpected effect reinforces the uniqueness and innovation of the invention, positioning it as a transcendental contribution in the current technological field, as detailed in the previous sections of this specification. Furthermore, as previously described, this technology can be designed and consequently built and / or developed through a video game engine, such as (but not limited to) Unity, generating a virtual environment configured so that the user can explore and walk through it.

[0106] In this sense, by "the user can explore and walk through it" it should be understood that the virtual space called virtual laboratory according to the present invention, can comprise and / or simulate, in one of its variants and / or modalities, a laboratory (like a real laboratory space) with tables, benches, and laboratory equipment or also a room with seats, blackboard, windows, among others.

[0107] Within the virtual environment, it is possible to integrate virtual elements and / or systems intended to be controlled. These can include, but are not limited to, electromechanical systems, robots, manufacturing systems, automated guided vehicles, etc. In general, it is possible to include any element and / or system whose behavior is to be simulated once certain control parameters are altered.

[0108] These virtual elements can be designed from a CAD program into the virtual environment, allowing for their movements to be animated, making them dependent on variables that can be modified from external devices. This aims to enable the development of these elements and / or systems in the CAD program to allow for an "n" number of movements, reactions, state switching, among others, anticipating and / or projecting each and every one of the possible variations that the user can apply through the real control elements, thereby allowing for realistic control of the simulated system.

[0109] From the design and construction of the virtual environment and the integration of the virtual elements into the Unity program (or any other video game engine), the virtual laboratory, according to one modality, can be executed in a compatible application on different devices for viewing the virtual environment (either classroom or laboratory) that in some of its versions can be laptops, tablets, cell phones, virtual reality viewers, mixed reality viewers, among others.

[0110] Now, for a person with knowledge in the field to which the invention belongs, it must be understandable that, although the present invention can be executed through or in different computing and / or processing devices, the object of the present invention is not directed to protect a program (programming language, the use of CAD platforms, video game engines, simulation media or applications, among others), on the contrary, after a holistic reading of what is described in this application, an expert in the field must understand that the object to which the present invention is directed focuses essentially and mainly on a technical solution that can be assisted by a computing and / or processing system in order to, based on information supplied by a user, obtain a final result and / or effect.That is to say, the essence of the present invention covers and / or comprises a set or system called control elements, which, as has been mentioned throughout this, can be any controller and / or real element that the user can directly manipulate, and on the other hand, the directly causal effect of the modification of said controllers / real elements, that is to say, the alteration of a virtual element / system and, particularly, the simultaneous alteration thereby offering a totally realistic simulation that allows and / or favors the immersion of the same user, as if it were a real system (that is to say, as the state of a real system would be altered when manipulated by said real controllers).

[0111] In one embodiment, an embedded system allows communication between the virtual environment and external electronic devices.

[0112] This embedded represents the innovative part of the proposed technology.

[0113] In this sense, in the proposed invention, differential equations that determine the behavior of virtual elements (robots, electromechanical systems, etc.) are implemented and solved through numerical methods. The above is advantageously carried out by the previously mentioned embedding, which guarantees that the sampling times (or integration steps) are adequate for simulating the behavior of the virtual element.

[0114] The embedded system, according to the present invention, may include physical filters to process the electrical signals coming from external electronic devices, as well as digital analog signal converters to send information from the embedded system to external electronic devices, thereby, the information and / or data fed back by the control components once they are manipulated by the user are sent and processed by said embedded system, said configuration and / or arrangement between the control elements - embedded system - virtual and / or simulated system being an efficient way and / or an efficient result that allows, as previously mentioned, to achieve a realistic simulation, substantially in real time, that is, simultaneously based on the manipulation performed by the user to the external control components.

[0115] Furthermore, the embedded system, according to the present invention, is responsible for sending to the virtual environment, through serial communication, the values ​​of the variables of the system being simulated in the code so that the animation can be reflected within the virtual environment.

[0116] The variables that represent the virtual element's inputs and outputs can be read and manipulated from different electronic devices such as microcontrollers, PLCs, operational amplifiers, among others, where different control algorithms can be implemented to modify the behavior of the virtual elements.

[0117] With the above, the person skilled in the art will understand more clearly that, as previously mentioned, in effect, the technical result and / or effect is achieved through the combination and interaction of the different elements / components of the system of the claimed invention, and, in no way, is it intended in any sense, to seek the protection of a computer and / or software program.

[0118] Referring now to Figure 3, the mixed reality lab according to the present invention starts by opening the Unity application on the selected device (it can be a Laptop, Tablet, virtual reality headset, etc., as previously mentioned).

[0119] The Unity application connects serially to the embedded system, where the behavior of the virtual elements is simulated with differential equations.

[0120] Within the Unity application, the virtual element to be simulated is selected (it could be a robot, an electromechanical system, etc.), and the application sends data to the embedded system that corresponds to the system to be simulated.

[0121] The embedded system receives the data and decides which system to simulate using differential equations coded for the corresponding value.

[0122] The embedded reads data from the analog-to-digital converters (where the external electronic devices are connected), solves the differential equations using the Euler numerical method, and writes the updated output values ​​to the embedded digital-to-analog converter (also connected to the external electronic devices).

[0123] The embed also sends the values ​​of the variables through the serial port to the Unity application.

[0124] The embedded system is always doing this cyclically until it receives data corresponding to another virtual element from the application, or it receives data to end the simulation.

[0125] Within the Unity application, the data received from the embedded object is read, and the virtual element's animation is modified according to those values. This continues until the user decides to use another virtual element or terminate the simulation.

[0126] Finally, communication between the application and the embedded system is closed and the program is terminated.

[0127] Many modifications and other embodiments of the invention will come to mind to one skilled in the art to which the invention pertains, having the benefit of the teachings presented in the foregoing descriptions and associated drawings. Therefore, it should be understood that the invention should not be limited to the specific and exemplary embodiments described, but that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are used herein, they are used only in a generic and descriptive sense and not for limiting purposes. Furthermore, it should be understood that the materials from which the various components comprising the invention described herein can be manufactured, the geometries, dimensions, arrangements and other elements may vary without departing from the scope and spirit of the invention and therefore, the embodiments referred to should not be considered limiting.

Claims

MODIFIED CLAIMS received by the International Bureau on 30 October 2024 (30.10.2024) 1. A system for the interactive simulation of virtual elements in a virtual environment, or virtual laboratory, the system being configurable for carrying out design practices and validation of control algorithms, implemented in real hardware that controls virtual plants in real time, the system comprising: a) a virtual environment generator; b) an electronic device; c) an application that includes a set of virtual elements that graphically emulate the behavior of control systems; d) a set of virtual elements designed using a CAD program and exported to the virtual environment; e) control devices, said control devices being present in the real plane;characterized in that the electronic device comprises an embedded system, and said embedded system comprises at least one of: digital-analog signal converters and signal filters, and wherein the electronic device is configurable for: 1) the numerical resolution of the dynamic model of the control systems emulated in the application of section a), with a sampling rate that allows emulations to be made in real time, 2) the synchronization of the dynamic model with the emulation in the computer application of section a) at a sampling rate suitable for the correct visualization of the emulation, 3) the real-time synchronization of external electronic devices that implement control algorithms, at a sampling rate suitable for control purposes, by means of the communication of both analog and digital input and output signals without communication protocols;thus allowing the user to implement control algorithms in real hardware of different types, such as microcontrollers, PLCs, PCs, FPGAs, among others, and for these algorithms to act on the emulated systems in real time; the embedded system is adaptive logic, which allows and considers the temporal restrictions of different systems and establishes communication between the virtual environment and external electronic devices, processing electrical signals and allowing the reading and manipulation of variables of the simulated system; and the application comprises a signal communication module, configured to communicate with the electronic device, allowing the user to be able to emulate control systems from their own device without restrictions of place or time.

2. The system according to claim 1, wherein the application includes any set of virtual elements from the group comprising robots, inverted pendulum, seesaw, among others.

3. The system according to claim 1, wherein the serial communication between the application and the electronic device is carried out through serial cable or WiFi communication.

4. The system according to claim 1, wherein the application is configured to run on computing devices selected from the group comprising personal computers, tablets, laptops, and virtual reality and augmented reality viewers, combinations thereof, and / or similar.

5. The system according to claim 1, wherein the virtual environment generator is any selected from the group comprising video game engines such as Unity, or Unity 3d; Unreal Engine, Godot Engine, CryEngine, Lumberyard, Godot Engine, Cocos2d, combinations thereof and / or similar.

6. The system according to claim 1, wherein the set of virtual elements is designed using a CAD program, said CAD program being any selected from the group comprising AutoCAD, SolidWorks, CATIA, Fusion 360, ANSYS, Revit, PTC Creo, combinations thereof and / or similar.

7. The system according to claim 1, wherein the control devices are any device selected from the group comprising electromechanical systems, robots, manufacturing systems, automatically guided vehicles, or others, which can be directly manipulated by a user to alter control parameters, combinations thereof and / or the like.

8. The system according to claim 1, wherein the embedded system further comprises physical filters for processing electrical signals and digital to analog signal converters for feedback of information to external electronic devices.

9. A method for interactive simulation of control systems according to the system of claims 1 to 8, the method comprising the steps of: a) start the real-time graphical simulation of control systems in the application, allowing the synchronization of the graphical simulation with the electronic device with a sampling rate convenient for the fluid visualization of the movement of the virtual elements and that allows correct communication; b) carry out a real-time numerical simulation of the dynamic model of the control system, in the embedded system at a sampling rate sufficiently fast for real-time simulation, and different from the simulation rate of the system in the graphical application; c) carry out a synchronization of analog and digital input and output signals of the embedded system with the same sampling rate as the simulation of the dynamic model;wherein a sampling rate sufficiently fast for real-time simulation and different from the simulation rate of the system in the graphical application is achieved by a single control cycle at the control sampling rate and emulating a slower sampling rate for synchronization with the graphical application; and wherein synchronizing analog and digital input and output signals of the embedded system with the same sampling rate as the simulation of the dynamic model allows the user to implement control algorithms for different types of components by interacting using standard analog and digital signals without using communication protocols, and thus test closed-loop control algorithms in real time.

10. The method according to claim 9, wherein the simulation is performed cyclically until receiving data to simulate another virtual element or ending the simulation.

11. The method according to claim 9, wherein the animation of the virtual element is modified in real time according to the variables of the simulated system.

Images