Immersive safety simulator based on historic data tailored to demographic features

Abstract

A method for creating a personalized immersive safety simulator includes collecting demographic information relating to a first individual, identifying existing information relating to a work environment of the first individual, and combining the demographic information and the existing information into a semantic model for predicting risk to individuals in the work environment. Training scenarios displayable in an augmented reality or virtual reality device may be created using the combined information. The semantic model may be queried based on the individual and a real-world incident and near miss report related to the work environment or information in a safety protocol or safety standard. The semantic model may be enhanced by integrating information about high-risk zones, high-risk activities, and high-risk demographics into the semantic model, the integrated information being unique to the work environment associated with the semantic model. The semantic model may be updated based on observed tendencies or scene changes.

Description

IMMERSIVE SAFETY SIMULATOR BASED ON HISTORIC DATA TAILORED TO DEMOGRAPHIC FEATURESTECHNICAL FIELD

[0001] This application relates to augmented reality (AR) and virtual reality (VR) applications.BACKGROUND

[0002] Shop floors or other industrial facilities can be dangerous environments for human workers. Workers might not be aware of all possible risks at every location and may move around between many different places over the course of their worktime or shift so that their risks may change. In order to be aware of risks and avoid incidents, frequent (new) training may be conducted, and unfamiliar spaces may need to be learned quickly and efficiently. It is not only a challenge to teach and implement new safety protocols and regulations, but workers retention and carrying out appropriate action based on known situational awareness adds to these challenges.

[0003] Furthermore, not all safety hazards can be mitigated with standard procedures and equipment. Unique situations may arise that require customized solutions and specialized equipment. The nature of the work may change over time, introducing new risks that must be identified and addressed in a timely manner. Standard safety protocols might not always be sufficient in non-standard situations (e.g., corner cases). Different physical traits might impose different risks such as a bigger body height imposes more risks of hitting the head for example.SUMMARY

[0004] According to an embodiment described in this disclosure a method for creating an immersive safety simulator includes collecting demographic information relating to a first individual, identifying existing information relating to a work environment of the first individual, and combining the demographic information and the existing information into a semantic model for predicting risk to this individual in the work environment. A training scenario displayable in an augmented reality (AR) or virtual reality (VR) device may be created using the combined information. The semantic model may be queried based on the individual and real-world incident reports related to the work environment or information in a safety protocol or safety standard. The semantic model may be enhanced by integrating information about high-risk zones, high-risk activities, and high-risk demographics into the semantic model, the integrated information being unique to the work environment associated with the semantic model and relevant to the scene understanding. The semantic model may be updated based on observed tendencies of the individual.

[0005] According to other embodiments, a system for creating immersive training scenarios comprising a computer processor and a non-transitory computer memory in communication with the computer processor, the non-transitory computer memory storing instructions that when executed by the computer processor cause the computer processor to perform the steps of collecting demographic information relating to a first individual, identifying existing information relating to a work environment of the first individual, and combining the demographic information and the existing information into a semantic model for predicting risk to the individual in the work environment. A trainingscenario displayable in an augmented reality (AR) or virtual reality (VR) device may be created using the combined information. The semantic model may be queried based on the individual and real-world incident reports related to the work environment or information in a safety protocol or safety standard. The semantic model may be enhanced by integrating information about high-risk zones, high-risk activities, and high-risk demographics into the semantic model, the integrated information being unique to the work environment associated with the semantic model. The semantic model may be updated based on observed tendencies of the individual, demographic-related needs, and / or preferred user profile settings.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following figures:

[0007] FIG. 1 is an overview of a semantic model for producing immersive AR / VR training scenarios according to aspects of embodiments of this disclosure.

[0008] FIG. 2 is a detailed depiction of the semantic model according to aspects of embodiments of this disclosure.

[0009] FIG. 3 is a process flow diagram for creating and using an immersive AR / VR training scenarios according to aspects of embodiments of this disclosure.

[0010] FIG. 4 is a block diagram of a computer system that may implement methods and system according to aspects of embodiments of this disclosure.DETAILED DESCRIPTION

[0011] To address the above challenges, a comprehensive approach to safety in the workplace is necessary, one that includes ongoing training and education for workers, regular safety assessments, and the development of personalized safety plans for each employee. Technology has a critical role to play in mitigating safety risks, with the use of advanced sensors, monitoring systems, and Al-based safety protocols allowing the identification and response to risks in real time. By taking a holistic approach to safety in the workplace, employers can ensure the well-being of their employees while also promoting a culture of safety and responsibility.Safety training is conducted in many different modalities. In many places workers receive classroom training taught by a human instructor. Training material is provided via speech, textbooks, and demonstration. Workers must pass a proficiency test to move one. Moreover, on the job training is utilized in many facilities where a supervisor points out the actual risk and mitigation during the execution of a task.There are immersive AR / VR systems for training employees and ensuring safety protocols are demonstrated. These use 3D overlays that help workers understand risks in their environment and how to avoid them. Where these techniques fall short is being unable to consider all of the nuances for each individual worker. Systems show risks, where they are, what in proximity may exist that is currently unseen (some dangerous equipment around a corner) but depending on the individual, there may be additional or different risks where awareness is vital. Intelligent systems bring together a variety of datathat typically is combined in a cohesive manner within a semantic framework or layer. Tools such as AR devices use the semantic framework to inform or teach workers. While there are semantic structures for the nuances in worker differences, bringing all these together to inform immersive AR systems is lacking in industry today as a cohesive implementation. Some technologies for creating an immersive implementation include:

[0012] Human Frame Tracking (FMA): There are ontologies that track human body movements - and are actively used in industry tools - such as FMA. While these will hold data from, for instance, people of differing heights, they are not yet connected to other vital information such as a person’s disability, limits of motion, etc. Knowing that a person is in range of a danger proximal to their head is one level of 3D data comparison, however knowing that same individual has limited vision on the same side as a risk means perhaps warrants extra or even earlier warnings are needed for that individual.

[0013] Historic Information: Semantic frameworks often describe limited provenance information, but systems that consider ranges of historic data and human demographic information to tailor safety training require a full range of these details to be built out for storing and using previous workers' flags whenever a new user is established. This is not an area that is widely used if at all in combination with AR / VR and machine vision data.

[0014] Demographic Data: There are ontologies that represent human frames and associated demographic information. However, these are sometimes limiting and siloed to a specific domain such as varying languages a person may speak, physical or cognitive characteristics, movement patterns, and health background, and personal boundaries (for instance, who should be extra careful lifting if there is a history of back injury, etc.). Thesedetails are not combined for the purpose of AR-based safety training and are often utilized under different upper ontology modeling methods.

[0015] Safety Semantics: There are several existing projects and tools that combine semantics with safety regulations. These alone are substantial, however, a need exists to combine safety considerations with the other domain areas described here to tailor safety and learning experiences to individual workers.

[0016] Reporting: There are many ontological structures used for information structuring such as reporting, computerized details, event structuring, etc. However, these are also typically used in domain applications and not in combination with the above listed areas.

[0017] Embodiments presented in this application propose to build an immersive safety simulator based on existing data tailored to demographic features, through a comprehensive history-based personalized risk prediction approach based on a past accident and near miss report database.

[0018] FIG. 1 is a high-level illustration of a method for creating an immersive AR / VR training session according to embodiments of this disclosure. A first machine learning model 103 will first perform worker profiling from demographic 101 and related historical meta data. Incident reports 102 in a formalized ontology together with other available training data and safety protocols and standards (e.g., OSHA) may be input to the model 103 to support the scenario 104 build. Demographic data 101 may include information relating to an individual worker 105, which can be calculated within the ML model 103 to produce an individualized scenario 104.

[0019] FIG. 2 is an illustration showing additional inputs provided to the ML model 103 according to embodiments of this disclosure. As shown in FIG. 1 , demographics data and metadata 101 provide a worker profile to the ML model 103. Incident reports and other safety and training data 102 provide the conditions under which a scenario 104 may be built for a potentially hazardous area or condition. Additionally, historical data about high- risk zones, high-risk activities and high-risk demographics 201 need to be seamlessly integrated into the risk training modules to keep them updated and targeted towards bespoke conditions that are unique to a particular factory floor. This model 105 will be integrated with the current factory floor situations, working modes, worker type information 202 (elder vs. younger, tall vs. short, male vs. female, far-sighted vs. nearsighted, fully abled vs disabled) to predict hazards and corresponding risk levels to inform the worker in advance.

[0020] This solution is made user-intuitive by building an immersive risk awareness training simulation in which a worker is presented with different real-world scenarios based on historic data tailored to this individual's situation (individual risk scenarios). Immersive AR / VR devices present information based on a cohesive backend that is powered by semantics holding both demographic and historical user details and are adjusted based on observable tendencies of the individual given their movements in the environment (perhaps they are limping and need the recommendation to be adjusted for safety). The semantic backend (triple store, database, federated storage, or combination of these) combines all needed modeling and inference about (but not limited to) human body information, historic details of other workers and incidents, demographic data,certain health details, safety semantics, and others. These are presented to other system components though semantic endpoints.

[0021] FIG. 3 provides a process flow diagram for conducting regular safety assessments using AR / VR based on worker profiles and demographic information. This involves several steps:

[0022] Define worker profiles 301 : The first step is to define different worker profiles based on factors such as job function, experience, training, age, gender, and physical attributes. These profiles should be specific to the industry and job type, needed capabilities.

[0023] Develop AR / VR scenarios 302: Next, the safety team would develop AR / VR scenarios that simulate the hazards and risks that are specific to each worker profile. For example, an AR / VR scenario for a new worker in a manufacturing plant may focus on identifying potential tripping hazards, while an AR / VR scenario for an experienced worker may focus on identifying potential equipment failures. For a specific example, consider the worker is guided through different areas of the plant, such as assembly lines, storage areas, and walkways. The scenario is designed to help the new worker identify potential tripping hazards, such as loose cables, uneven flooring, obstructed walkways, and cluttered workspaces. For an experienced worker in the same manufacturing plant, the AR / VR scenario takes a different approach. The worker is placed in a simulated production line area with various machines and equipment. The scenario is designed to help the experienced worker identify potential equipment failures, such as unusual sounds, vibrations, overheating components, or irregular movements in the machinery.But these scenarios would be based on worker profiles and historical data extraction through semantics. The historical data include incident and near miss reports.

[0024] Incorporate worker-specific information 303: The AR / VR scenarios would incorporate worker-specific information such as the worker's physical attributes, training, and experience level. This can be done by using data from the worker profiles developed in step 301 .

[0025] Implement the AR / VR training 304: The VR training would be implemented regularly and integrated into the worker's overall training program. Workers should be encouraged to complete the AR / VR training and receive feedback from the safety team.

[0026] Evaluate and improve 305: Finally, the safety team would regularly evaluate the effectiveness of the AR / VR training by monitoring incident rates and soliciting feedback from workers. The VR scenarios should be updated and improved based on this feedback and changes in the work environment.

[0027] Overall, conducting regular safety assessments using AR / VR applications based on worker profiles and demographic information can help personalize safety training and improve worker awareness of hazards and risks in the workplace. It can also help to reduce incident rates and improve overall safety culture.

[0028] The core of embodiments herein is the buildup of a modular safety simulator based on existing data tailored to the needs of the trainee to make safety awareness as effective as possible. By using existing data and a semantic backend, generated scenarios can extend and update the scene based on the changing factory environment and risks, but more importantly based on the users' needs. All people learn differently andhave different risk factors based on their work experience, physical condition, prior accidents and such which may be taken into consideration for the novel safety tool. Also, over time adjustments to the training for the individual may be performed based on their personal development as well.

[0029] FIG. 4 illustrates an exemplary computing environment 400 within which embodiments of the invention may be implemented. Computers and computing environments, such as computer system 410 and computing environment 400, are known to those of skill in the art and thus are described briefly here.

[0030] As shown in FIG. 4, the computer system 410 may include a communication mechanism such as a system bus 421 or other communication mechanism for communicating information within the computer system 410. The computer system 410 further includes one or more processors 420 coupled with the system bus 421 for processing the information.

[0031] The processors 420 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as used herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and / or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and beconditioned using executable instructions to perform special purpose functions not performed by a general-purpose computer. A processor may be coupled (electrically and / or as comprising executable components) with any other processor enabling interaction and / or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

[0032] Continuing with reference to FIG. 4, the computer system 410 also includes a system memory 430 coupled to the system bus 421 for storing information and instructions to be executed by processors 420. The system memory 430 may include computer readable storage media in the form of volatile and / or nonvolatile memory, such as read only memory (ROM) 431 and / or random-access memory (RAM) 432. The RAM 432 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM 431 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 430 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 420. A basic input / output system 433 (BIOS) containing the basic routines that help to transfer information between elements within computer system 410, such as during start-up, may be stored in the ROM 431 . RAM 432 may contain data and / or program modules that are immediately accessible to and / or presently being operated on by the processors 420. System memory 430 may additionally include, for example, operating system 434, application programs 435, other program modules 436 and program data 437.

[0033] The computer system 410 also includes a disk controller 440 coupled to the system bus 421 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 441 and a removable media drive 442 (e.g., floppy disk drive, compact disc drive, tape drive, and / or solid-state drive). Storage devices may be added to the computer system 410 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).

[0034] The computer system 410 may also include a display controller 465 coupled to the system bus 421 to control a display or monitor 466, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computer system includes an input interface 460 and one or more input devices, such as a keyboard 462 and a pointing device 461 , for interacting with a computer user and providing information to the processors 420. The pointing device 461 , for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the processors 420 and for controlling cursor movement on the display 466. The display 466 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 461 . In some embodiments, an augmented reality device 467 that is wearable by a user, may provide input / output functionality allowing a user to interact with both a physical and virtual world. The augmented reality device 467 is in communication with the display controller 465 and the user input interface 460 allowing a user to interact with virtual items generated in the augmented reality device 467 by the display controller 465. The user may also providegestures that are detected by the augmented reality device 467 and transmitted to the user input interface 460 as input signals. Augmented reality device 467 may also receive input from a user, including alphanumeric data or through gestures. This information may be provided to computer system 410 as input from the user.

[0035] The computer system 410 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 420 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 430. Such instructions may be read into the system memory 430 from another computer readable medium, such as a magnetic hard disk 441 or a removable media drive 442. The magnetic hard disk 441 may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security. The processors 420 may also be employed in a multiprocessing arrangement to execute the one or more sequences of instructions contained in system memory 430. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0036] As stated above, the computer system 410 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processors 420 for execution. A computer readable medium may take many forms including, but not limited to, non- transitory, non-volatile media, volatile media, and transmission media. Non-limitingexamples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 441 or removable media drive 442. Non-limiting examples of volatile media include dynamic memory, such as system memory 430. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 421 . Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0037] The computing environment 400 may further include the computer system 410 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 480. Remote computing device 480 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 410. When used in a networking environment, computer system 410 may include modem 472 for establishing communications over a network 471 , such as the Internet. Modem 472 may be connected to system bus 421 via user network interface 470, or via another appropriate mechanism.

[0038] Network 471 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 410 and other computers (e.g., remote computing device 480). The network 471 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 471 .

[0039] An executable application, as used herein, comprises code or machine- readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine-readable instruction, subroutine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and / or parameters, performing operations on received input data and / or performing functions in response to received input parameters, and providing resulting output data and / or parameters.

[0040] A graphical user interface (GUI), as used herein, comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions. The GUI also includes an executable procedure or executable application. The executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user. The processor, under control of an executable procedure or executable application, manipulates the GUI display images inresponse to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.

[0041] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

[0042] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and / or combinations thereof.

Claims

CLAIMSWhat is claimed is:1 . A method for creating an immersive safety simulator comprising: collecting physiological and demographic information relating to a first individual; identifying existing information relating to risks in a work environment of the first individual; combining the demographic information and the existing information into a semantic model for predicting risk to the individual in the work environment; and proposing environmental parameters that should be in place or recommended for a given individual.

2. The method of Claim 1 , further comprising: constructing training scenarios displayable in an augmented reality (AR) or virtual reality (VR) device.

3. The method of Claim 2, further comprising: querying the semantic model based on the individual and a real-world scenario to construct non-standard training scenarios.

4. The method of Claim 2, wherein the existing information comprises incident or near miss reports related to the work environment.

5. The method of Claim 2, wherein the existing information includes a safety protocol or safety standard.

6. The method of Claim 2, further comprising:integrating information about high-risk zones, high-risk activities, and high-risk demographics into the semantic model, the integrated information being unique to the work environment associated with the semantic model.

7. The method of Claim 2, further comprising: updating the semantic model based on observed tendencies of the individual.

8. A system for creating an immersive training scenario comprising: a computer processor; and a non-transitory computer memory in communication with the computer processor, the non-transitory computer memory storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: collecting demographic information relating to a first individual; identifying existing information relating to a work environment of the first individual; and combining the demographic information and the existing information into a semantic model for predicting risk to the individual in the work environment.

9. The system of Claim 8, the non-transitory computer memory further storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: constructing training scenarios displayable in an augmented reality (AR) or virtual reality (VR) device.

10. The system of Claim 9, the non-transitory computer memory further storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: querying the semantic model based on the individual and a real-world scenario to construct the training scenarios.1 1 . The system of Claim 9, wherein the existing information comprises incident or near miss reports related to the work environment.

12. The system of Claim 9, wherein the existing information includes a safety protocol or safety standard.

13. The system of Claim 9, the non-transitory computer memory further storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: integrating information about high-risk zones, high-risk activities, and high-risk demographics into the semantic model, the integrated information being unique to the work environment associated with the semantic model.

14. The system of Claim 9, the non-transitory computer memory further storing instructions that when executed by the computer processor cause the computer processor to perform the steps of: updating the semantic model based on observed tendencies of the individual.

Images