Method for virtually simulating the location of a user inside a compartment of a piece of moving machinery

Abstract

The invention relates to a simulation method designed to provide an immersive experience to users in virtual environments in which heavy machinery is used, for example the mining, agriculture, port, maritime, electrical or construction industries. The method includes virtual reality anchor markers for positioning the user inside the compartment of a piece of heavy machinery or any type of structure, and comprises precisely simulating operation in underground and open-pit mining environments, agricultural fields and large constructions or movements of earth, without including sophisticated hardware other than a virtual reality headset.

Description

[0001] VIRTUAL SIMULATION METHOD FOR A USER'S LOCATION

[0002] INSIDE A MOBILE MACHINERY CAB

[0003] DESCRIPTIVE MEMORANDUM

[0004] PREVIOUS ART

[0005] Most of the methods and systems that exist in the world for virtual training or virtual simulation of the operation of heavy machinery are based on improvements to simulations through mechanical and / or electronic tools or hardware, and use hardware developments to generate the most immersive virtual simulation.

[0006] Simulation methods and systems present the problem that it is never possible to perfectly match or align electronic devices with virtual ones—joysticks, pedals, buttons, and the control panel—which results in a less than ideal training experience, as there is always a positioning mismatch. This causes the physical hand, the person's hand, to collide with the physical devices because the view from the headsets is not compatible with the physical world.

[0007] Furthermore, computational simulation allows for the virtual testing of different scenarios and the making of necessary adjustments before producing a physical prototype, thus reducing costs and development time. The physical prototyping stage is expensive and can significantly increase the cost of a team's project. Simulation allows you to explore hypothetical questions and scenarios without having to experiment with the system itself. It helps you identify bottlenecks in the flow of materials, information, and products. It helps you gain insights into which variables are most critical to the system's performance.

[0008] Regarding simulation methods, various solutions exist, such as the one described in document WO2017014733, which outlines a virtual reality training system for industrial work applications. Users wear virtual reality equipment, including a head-mounted device, and enter a virtual workplace filled with virtual industrial equipment, virtual hazards, and virtual tasks. Throughout the task performance, multiple sensors monitor the user's performance and identify knowledge gaps and stress levels. The system generates an evaluation for each user and then informs them of areas for improvement and alerts a manager to any potential weaknesses in the evaluated employees.

[0009] Document WO2024047671 describes computer-implemented systems and methods for creating and delivering interactive virtual learning within an organization for its employees through the gemification of standard operating procedures (SOPs). These systems comprise a user device, a remote / central server, and a processing and analysis device / module operationally coupled with the user device or server, or both. The server is configured with logic comprising a step-by-step algorithm that covers all crucial aspects / steps of an SOP document or any of its versions. These systems and methods help employees learn the rules for working in a specific area or on a particular machine within an organization in a much simpler and more engaging way than by simply reading the rules in the SOP documents.

[0010] EP4050163 describes a system that allows an operator to gain operational experience through simulation while observing a simulated operating scenario of a working machine, either by the operator themselves or another operator. A second operator can select a time series of images of the working environment from files stored in a database via a remote input interface. This second operator can then explore the selected time series of images on a remote output interface. The second operator can configure the remote output interface to display a time series of simulated images depicting a specific action sequence and an environmental change for a working machine in a virtual workspace, according to a simulated operating procedure, on a remote control.

[0011] US Patent 9818308 describes a simulator of an agricultural work machine that includes a computer unit coupled to a display unit and a control unit. The agricultural work machine to be simulated has adjustable working parts and control elements that implement the adjustment of the working parts, as well as a software module stored on the computer unit. The software module represents a process model of the crop processing operations to be implemented on the agricultural work machine and depicts the machine's behavior resulting from these operations. The process model and machine behavior are displayed on the display unit and can be edited from the control unit.

[0012] Document ES2913215 discloses a crane simulator comprising an input means for loading control commands, a graphical simulation module for calculating a virtual representation of the machine environment and machine components visible from the control station, such as boom and load hook, as well as a visualization device for displaying the calculated virtual representation; wherein a motion simulation module is provided for determining movements and / or deformations of the machine components based on the loaded control commands, based on which the graphical simulation module calculates the virtual representation.

[0013] US2003224333 describes a simulator comprising a cabin with a control mechanism and multiple windows. A screen is located near the cabin windows and includes a viewing area below the cabin. At least one window may be located at the bottom of the cabin. Multiple projectors are used to project images onto the screen. The cabin is suspended downward from a motion actuator adapted to move the cabin in response to adjustments made to the control mechanism. The simulator can be used for training operators of container gantry cranes, gantry cranes, dockside or harbor cranes, or tower cranes, for example.

[0014] Document WO 201 1067456 describes a forestry machinery simulator. The forestry machinery simulator aims to assist the user by adding visual cues to the simulated environment. For example, the simulator can add a grid indicating the simulated forestry machinery's working area, allowing the user to visualize the appropriate work area within the landscape. The method comprises:

[0015] • electronically determine a dimension of at least a portion of the working area of ​​the working machine, and

[0016] • Electronically indicate the working area of ​​the forestry machine so that the working area indication is at least partially visible on ground adjacent to the working machine.

[0017] • model the forestry work machine using a computer so that at least part of the forestry machine is a computer-modeled part, and

[0018] • To display the movement of the computer-modeled part of the forestry machine to a user of the device. Unlike prior art, the method of the invention can be adapted to existing applications, as it is software-based, utilizing the tools provided by virtual reality headsets that include a hardware and software hand tracking tool. The method of the invention uses these existing resources to position the user in a pre-established virtual area with an accuracy level of at least 85% (this is the percentage guaranteed by the manufacturers of virtual reality headsets).

[0019] The advantages of the invention lie in the fact that without the additional help of any mechanical or electronic device, it is possible to position a person within a virtual environment, while at the same time making the driving experience itself more immersive, because as the virtual machine moves within the scenario, the person already positioned inside the cabin will also move along with the machine, achieving a sense of immersion superior to other systems based on electronic devices.

[0020] The method adapts to all types of mining machinery and mining, agricultural, and construction environments, allowing users to experience operations firsthand under realistic and safe conditions. The user interface is simplified through two or three positions, with four actions on screen, facilitating the setup and positioning process.

[0021] BRIEF DESCRIPTION OF THE FIGURES

[0022] Figure 1: This figure illustrates the adjustment stage between the physical world and the virtual world, specifically the operator's hand position. Figure 2: This figure illustrates the adjustment of two or more points in the physical world to the virtual world within a simulated machine cab.

[0023] Figure 3: represents an illustration of the adjustment of the measurement and positioning of the control panel, with respect to the user and the position of the person sitting inside the cabin, with respect to the predefined area of ​​the machinery cabin.

[0024] Figure 4: represents an image and illustration of adjusting two anchor points, taking into account that the first anchor point is the viewer and the other is the virtual hand.

[0025] Figure 5: corresponds to an illustration of the on-screen action for configuring or setting the control panel, with the body physiognomy.

[0026] Figure 6: corresponds to an illustration for the configuration or setting of the seat, with respect to the operator's position.

[0027] Figure 7: corresponds to an illustration for the configuration or setting of joysticks, with respect to the position of the operator's hands.

[0028] Figure 8: corresponds to an illustration for the configuration or setting to centralize the position of the person, within the virtual booth area.

[0029] DESCRIPTION OF THE INVENTION

[0030] The virtual simulation model of a user's location within a mobile machinery cab involves configuring or setting up the user's body within predefined spaces for simulated machine cabs. This is achieved by introducing virtual anchor markers in three-dimensional spaces and virtual reality headsets to position the user within confined spaces in the cabs. The model includes configuring at least two and up to three points in the virtual space, where each configuration anchor point is set using a virtual reality headset. This establishes a perimeter that positions the person within a simulated machine cab. Strictly speaking, the anchor markers are used to calculate the user's position and orientation within the virtual environment.

[0031] The method understands that when a configuration is established outside the range or outside the required physical (real world) points, the method will establish that point as the configuration to be calculated (see Figure 1). This configuration is carried out with the person placing their hands on their toes, touching the corners of the structure in the simulation space and facing the simulated structure on the screen. Once this action is performed, an instructor will execute the first posture action (Figure 5). As an example, with reference to Figures 2 and 3, when three anchor points or markers are set, it considers that the person touches:

[0032] • Right fingertip.

[0033] • Left fingertip.

[0034] • Anchor point of the visor and head, in right side view.

[0035] Once the first anchor point is configured or set, keeping in mind that it is virtual and serves as a spatial guide for where the anchor points should be located, corresponding to a structure on the cockpit dashboard (see Figure 2), the seat configuration or setting action (Figure 6) is performed for a second position using the joysticks, which are defined as physical areas (see Figure 4) where these anchor points should be located. The person faces forward, grasps the joysticks, and performs the same steps as for the first position. Then, a third position involves looking at the right joystick, where the person activates the joystick configuration (Figure 7), and the person must:

[0036] • Anchor hand on right joystick.

[0037] • Set the visor and head anchor point by looking at the right joystick

[0038] The anchor points established through this method are nodes that appear after pressing assigned buttons on the simulation screen (see Figures 5 to 8). Without these physical elements, the virtual anchors cannot be correctly placed, as they are defined by the physical structures themselves, achieving a spatial relationship of 85% to 90%, which is determined by the capabilities of the virtual viewer's hardware.

[0039] The anchor points correspond to area positioning, such as the measurement and positioning of the control panel with respect to the user and the position of the person sitting inside the cabin with respect to the predefined area of ​​the cabin (see figures 3 and 4).

[0040] In the configuration or setting of two anchor points, taking into account that the first anchor point is the viewfinder and we add another which is the virtual hand, when executing the joystick configuration or setting action, a distance reference is established, achieving the fit of the virtual hand associated with the virtual joystick, with respect to the hands and physical joystick.

[0041] The method includes a centralized action (Figure 8), which consists of establishing, once the predefined action set by the viewers, degrees of rotation, displacement, and freedom in a virtual space. In this way, when offsets or special cases occur, such as when the user's view is outside the cab (see Figure 9), the calculation of the settings is prevented: 1. Setting and position with dashboard, 2. Setting and position with seat, 3. Setting and position with joysticks. When this action is executed, the method places the operator inside the cab, allowing the calculations associated with the other settings to be performed. The method includes a simplified user interface with the positions and actions executed displayed on screen (Figures 5 to 8) to facilitate the configuration and positioning process.Furthermore, the centralization stage allows the centralization of the individual, to bring him into the cabin area, thereby allowing the setting of his body physiognomy through the action of the actions mentioned above.

[0042] The primary objective of the invention is to enhance the learning and training processes for heavy equipment operators by providing a highly realistic simulation experience that can help reduce risks and improve workplace efficiency. Furthermore, the immersive experience achieved with this tool is expected to contribute to greater knowledge and skill retention by users.

[0043] The method allows achieving the primary objective, which is the anchoring and matching of real peripherals such as joysticks and buttons, along with the virtual ones that are within the virtual vision experience, with the additional advantages of increasing the immersive experience by reducing rejection, increasing the learning percentage and considerably reducing training time.

Claims

VIRTUAL SIMULATION METHOD FOR A USER'S LOCATION INSIDE A MOBILE MACHINERY CAB CLAIMS 1. A simulation method designed to provide an immersive experience to users within virtual environments where heavy machinery is used, such as in the mining, agriculture, construction, maritime and port, and electrical industries, CHARACTERIZED in that it comprises configuring or setting the body position within predefined spaces for simulated machine cabins by means of virtual anchor markers in three-dimensional spaces and virtual reality viewers to place the user within confined spaces in said cabins; where at least two and up to three anchor points are configured in the virtual space, where each configuration anchor point is set by means of a virtual reality viewer, establishing a perimeter that positions the person in a simulated cabin of a given machine.

2. Simulation method designed to provide an immersive experience to users according to claim 1, CHARACTERIZED in that it comprises the user placing their hands on their toes touching the corners of the simulation space structure and looking towards the simulated structure on the screen so as to position the user within the simulation range and thus define a first posture.

3. Simulation method designed to provide an immersive experience to users according to claim 2, CHARACTERIZED in that the action of touching the corners comprises the user touching the right fingertip, left fingertip and anchor point of the visor and head, in right side view.

4. Simulation method designed to provide an immersive experience to users according to claim 1, CHARACTERIZED in that it comprises configuring or setting the seat within the simulation space so as to generate a second posture, by means of joysticks that define anchor points.

5. Simulation method designed to provide an immersive experience to users according to claim 4, CHARACTERIZED in that said seat configuration comprises the user looking forward while holding the joysticks and performing the same steps as for the first posture.

6. Simulation method designed to provide an immersive experience to users according to claim 1, CHARACTERIZED in that it comprises generating a third posture where the user looks at the right joystick actuating joystick configuration.

7. Simulation method designed to provide an immersive experience to users according to claim 6, CHARACTERIZED in that in said third posture the user anchors his Place your hand on the right joystick and set the anchor point of the visor and head, looking at the right joystick.

8. Simulation method designed to provide an immersive experience to users according to claim 1, CHARACTERIZED in that it comprises a centralizing action, which consists of establishing degrees of rotation, displacement and freedom in a virtual space once the action already predefined by the viewers has been established.

9. Simulation method designed to provide an immersive experience to users according to claim 1, CHARACTERIZED in that it comprises a simplified user interface for positions within virtual space.

Images