Predictive Virtual Reconstruction of Physical Environments

Abstract

An embodiment of the present invention describes predictively reconstructing a physical event using augmented reality. The embodiment includes capturing data of the area of ​​the physical event using video analytics to analyze video feeds collected from the area of ​​the physical event before and after the physical event, the video feeds including at least one object, to identify relative positions of objects located in the area of ​​the physical event; generating a knowledge corpus including the video analytics and the collected video feeds related to the physical event and historical information; and displaying, by the computing device, the generated virtual reconstruction of the predicted physical event. Additionally, the embodiment describes identifying possible prior events based on the captured data and the knowledge corpus; generating a virtual reconstruction of the physical event using the possible prior events; and displaying, by the computing device, the generated virtual reconstruction of the predicted physical event; and overlaying the displayed virtual reconstruction of the predicted physical event on an image of the area of ​​the physical event.

Description

[Technical field]

[0001] The present invention relates generally to the field of augmented reality, and more particularly to predictive virtual reconstruction of a physical environment. [Background technology]

[0002] Augmented reality (AR) is an interactive experience of a real-world environment where real-world objects are augmented with computer-generated perceptual information across multiple perceptual modalities, including in some cases vision, sound, touch, body sensation, and smell. AR can be defined as a system that satisfies three basic characteristics: a combination of real and virtual worlds, real-time interaction, and accurate 3D description of virtual and real objects. The overlaid perceptual information can be added to or mask the natural environment. The AR experience can be one in which the physical world is seamlessly interwoven, so it can be understood as an immersive aspect of the real environment. In this way, augmented reality changes a person's ongoing perception of the real-world environment, replacing the user's real-world environment with a simulated one. Augmented reality is broadly related to two synonymous terms: mixed reality and computer-mediated reality.

[0003] Virtual reality is used to augment a natural environment or situation, providing a perceptually enhanced experience. With the aid of advanced AR technologies (e.g., adding computer vision, incorporating AR cameras in smartphone applications and object recognition), the information surrounding the user's real world becomes interactive. Information about the environment and its objects is overlaid on the real world. The immersive perceptual information is sometimes combined with supplemental information, such as scores overlaid on a live video feed of a sporting event. This combination is beneficial for both augmented reality and head-up display technologies (HUD). Summary of the Invention

[0004] Embodiments of the present invention disclose methods, computer program products and systems for predictively reconstructing a physical event using augmented reality, the computer implemented method including using video analytics to analyze video feeds collected from a region of a physical event before and after a physical event involving at least one object to identify a relative status of the object located within the region of the physical event, analyzing the video analytics and collected video feeds related to the physical event and historical information from a previous physical event to correlate how the relative positions of objects within the region of the physical event changed due to the physical event, a type of physical event associated with a subsequent event, and a duration of the physical event that gave rise to the subsequent event. generating a knowledge corpus including the video feed; capturing, with a computing device, data of an area of ​​the physical event; identifying possible prior events based on the captured data and the knowledge corpus; generating a virtual reconstruction of the physical event using the possible prior events, the virtual reconstruction of the physical event being a predicted physical event that is most likely to occur during the physical event; and displaying, by the computing device, the generated virtual reconstruction of the predicted physical event, the displayed virtual reconstruction of the predicted physical event being overlaid on an image of the area of ​​the physical event.

[0005] Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings in which: [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a functional block diagram illustrating a distributed data processing environment in accordance with an embodiment of the present invention. [Diagram 2]FIG. 2 illustrates operational steps of a virtual reconstruction tool in communication with computing devices in the distributed data processing environment of FIG. 1 for predictively reconstructing physical events using augmented reality according to an embodiment of the present invention. [Diagram 3] FIG. 3 illustrates a block diagram of components of a computing device executing a virtual reconstruction tool in the distributed data processing environment of FIG. 1 in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] During the testing of the physical environment, the user accesses the surrounding physical events and collects samples such as broken objects, relative positions of objects, different landmarks on the ground, damage to the surrounding environment, or any other information related to the physical event as known in the art, or combinations thereof. Thus, the user can analyze the physical event and the area surrounding the physical event (i.e., the physical event area) to determine the type of incident that may have occurred (e.g., a vehicle accident, a natural disaster, or any other event involving a physical event as known in the art). A physical event includes one or more physical objects interacting with one or more other physical objects (e.g., a vehicle accident, a natural disaster, or any other event involving a physical event as known in the art). Currently, in the art, approaches to determining events that precede a physical event (i.e., prior events) and events that occurred during a physical event include attempts by the user to estimate many different incidents that may have occurred within or around the physical event area. Often, it is extremely difficult for a user to accurately estimate or determine events that occurred before, during, and after a physical event, collectively referred to as preceding events. Furthermore, currently, it is often not possible in the art to predict and determine preceding events and events that are after a physical event (i.e., subsequent events), which also leads to delays and determination errors.

[0008] The present invention can solve the above-mentioned problems by providing an artificial intelligence (AI)-based augmented reality system that can analyze a surrounding area and predict what has occurred in the area of ​​a physical event using one or more standard learning models trained on historical (physical) events. For example, one or more machine learning models including Markov models, reinforcement learning (RL), recurrent neural networks, sequence mining models, or time series models, or combinations thereof, can be used to model the sequence of events and predict the pre-event. Additionally, the present invention can improve on the current technology by displaying the predicted pre-events, events that occurred during the physical event, and post-events as predicted by one or more machine learning models through augmented reality (AR) glasses or an AR application on a smartphone. The present invention can improve on the current technology by presenting a user with a list of possible scenarios of pre-events by generating sequences using a recurrent neural network (RNN) and events that occurred during the physical event, where the list of possible scenarios of the physical event are ranked from most likely to have occurred to least likely to have occurred.

[0009] Embodiments of the present invention may improve AR technology by using a data collection module that collects historical physical event information, including video feeds (i.e., captured videos) of different types of physical events, previously recorded or reported physical events, Internet of Things (IoT) feeds related to any physical events, and computer-generated simulation results of different types of physical events. Thus, embodiments of the present invention may improve AR technology by generating a knowledge corpus by collecting and storing historical information collected related to current and previous physical events using machine learning models. Since embodiments of the present invention generate a knowledge corpus by correlating various parameters of pre-events and post-events from previously collected data, embodiments of the present invention may be able to efficiently predict physical events and virtually reconstruct areas and physical events (e.g., describing how different objects were destroyed or damaged, showing how the relative positions of objects changed, a list of types of events as learned by one or more machine learning models that would have occurred to generate the physical events and post-events, and determining the degree of danger and severity of the physical events). Classification of the degree of danger and severity of a physical event can be performed using a bidirectional long short-term memory (LSTM) model.

[0010] When a user views a physical event around them through AR glasses, an embodiment of the present invention can access the area of ​​the physical event around them. An embodiment of the present invention can virtually display, via augmented reality, a simulation that describes how the physical event occurred, how the object's position moved, or how the object was damaged, including different types of possible prior events. An embodiment of the present invention can mimic, via AI-enabled augmented reality, the lighting, weather conditions, or environmental conditions of the area of ​​the physical event at the time the physical event occurred, or a combination thereof. For example, a user can observe whether the driver was temporarily blinded by a sunset or the sun's glare from the side mirror.

[0011] An embodiment of the present invention can predict the intensity type, direction of intensity, and duration of intensity of a physical event and therefore display the predicted event of the physical event to a user through AR glasses or a computing device with AR capabilities using one or more machine learning models (e.g., RNNs). An embodiment of the present invention can reconstruct the physical event and display the reconstruction of the physical event through an augmented reality component. An embodiment of the present invention can display animated virtual reality graphics to explain how the physical event occurred in the area of ​​the physical event to assist the user in reconstructing the prior event and understanding details about the physical event (e.g., earthquake, hurricane, traffic patterns, etc.). An embodiment of the present invention can overlay an AR simulated or learned and reconstructed physical event scenario on a current physical event at a similar location to identify similarities, differences, or both in the physical event, or post-event, or both. An embodiment of the present invention can predict the original positions of various objects before the physical event occurred and how the relative positions of the objects have changed due to the physical event. For example, visually displaying an object's trajectory via an AR simulation.

[0012] Note that in the described embodiments, the parties involved (i.e., users) have agreed to have their images captured, uploaded, and stored, and recorded and monitored. Additionally, the parties involved are aware that such recording and monitoring will occur. In various embodiments, for example, when downloading or operating an embodiment of the invention, the embodiment of the invention presents terms and conditions that allow interested parties to quickly agree or disagree to participate.

[0013] Implementations of embodiments of the present invention may take a variety of forms, and exemplary implementation details are described below with reference to the drawings (ie, FIGS. 1-3).

[0014] FIG. 1 is a functional block diagram illustrating a distributed data processing environment, generally designated 100, in accordance with one embodiment of the present invention. As used herein, the term "distributed" describes a computer system that includes multiple physically distinct devices that operate together as a single computer system. FIG. 1 is only intended to provide an illustration of one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made by one of ordinary skill in the art without departing from the scope of the present invention as recited in the claims. The distributed data processing environment 100 includes a computing device 110 and a server computer 120 interconnected over a network 130.

[0015] Network 130 may be, for example, a storage area network (SAN), a communications network, a local area network (LAN), a wide area network (WAN) such as the Internet, wireless technologies for exchanging data over short distances (such as short wavelength ultra-high frequency (UHF) radio waves, the 2.4-2.485 GHz Industrial, Scientific, and Medical (ISM) bands from fixed and mobile devices, and building personal area networks (PANs), or a combination of the three), and may include wired, wireless, or fiber optic connections. Network 130 may include one or more wired or wireless networks, or a combination thereof, capable of receiving data, voice, or video signals including voice, data, text, or video information, or a combination thereof. In general, network 130 may be any combination of connections and protocols supporting communication between computing device 110, server computer 120, and any other computing or storage devices (not shown in FIG. 1) in distributed data processing environment 100.

[0016] In some embodiments of the invention, computing device 110 may be, but is not limited to, a standalone device, a client, a server, a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a smart phone, a desktop computer, a smart television, a smart watch, a radio, a stereo system, a cloud-based service (e.g., a cognitive cloud-based service), an AR glass, a virtual reality headset, any HUD known in the art, or any programmable electronic computing device capable of communicating with various components and devices in distributed data processing environment 100 via network 130 or any combination therein. In general, computing device 110 may represent any programmable computing device or combination of programmable computing devices capable of executing machine-readable program instructions and communicating with users of other computing devices via network 130, or that execute machine-readable program instructions and communicate with server computer 120, or both.

[0017] In various embodiments, the camera component 108 executes on the computing device 110. In some embodiments, the camera component 108 may be located or executed elsewhere in the distributed data processing environment 100, or both. The camera component 108 may be one or more cameras as known in the art. In various embodiments, the camera component 108 may provide a video feed that a user may view at the computing device 110, or more specifically, at the virtual reconstruction tool 112. The video feed may be communicated in real-time as a live video feed, or may be recorded and communicated as a recorded video feed. The term "video feed" as used herein may refer to one or both types of video feed. In various embodiments, the virtual reconstruction tool 112 may enable the computing device 110 to store the captured video feed, or photographs, or both, on the shared storage 124, or on the local storage 104, or both. In various embodiments, the camera component 108 can record, communicate, and store video feeds, and can take, communicate, and store photographs.

[0018] In some embodiments of the present invention, computing device 110 may represent any programmable electronic computing device or combination of programmable electronic computing devices capable of executing machine-readable program instructions, manipulating the execution of machine-readable instructions, and communicating with server computer 120 and other computing devices (not shown) in distributed data processing environment 100 over a network, such as network 130. Computing device 110 may include a user interface instance 106 and local storage 104. In various embodiments, not shown in FIG. 1, computing device 110 may have multiple user interfaces. In other embodiments, not shown in FIG. 1, distributed data processing environment 100 may include one or a combination of multiple computing devices, multiple server computers, or multiple networks. Computing device 110 may include internal and external hardware components, as further illustrated and described in more detail with reference to FIG. 3.

[0019] The user interface 106 provides an interface to the predictive virtual reconstruction tool 112. Through the user interface 106, the computing device 110 allows users and / or clients to interact with the virtual reconstruction tool 112 and / or the server computer 120 in a variety of ways, such as sending program instructions, receiving program instructions, sending and / or receiving messages, updating data, sending data, inputting data, editing data, collecting data and / or receiving data, or combinations thereof. In one embodiment, the interface 106 can be a graphical user interface (GUI) or a web user interface (WUI) and can display at least text, documents, web browser windows, user options, application interfaces, and instructions for operation. The interface 106 can include information (such as graphics, text, and sound) that is presented to the user and can control the sequences that the user uses to control operations. In another embodiment, interface 106 may be mobile application software that provides an interface between a user of computing device 110 and server computer 120. Mobile application software, or "apps," may be designed to run on smart phones, tablet computers, and other computing devices. In an embodiment, interface 106 may enable a user of computing device 110 to at least send data, enter data, edit (annotate) data, correct data, or receive data, or any combination thereof.

[0020] The server computer 120 may be a standalone computing device, an administrative server, a web server, a mobile computing device, one or more client servers, or any other electronic device or computing system capable of receiving, delivering, and processing data. In other embodiments, the server computer 120 may represent a server computing system using multiple computers, such as, but not limited to, a server system in a cloud computing environment. In another embodiment, the server computer 120 may represent a computing system using cluster computers and components that function as a single pool of seamless resources when accessed in the distributed data processing environment 100 (e.g., database server computers, application server computers, etc.). The server computer 120 may include internal and external hardware components as shown and described in more detail with reference to FIG. 3.

[0021] Each of the shared storage 124 and the local storage 104 may be a data / knowledge repository or database that may be written to and / or read by one or a combination of the virtual reconstruction tool 112, the server computer 120, and the computing device 110. In the illustrated embodiment, the shared storage 124 resides on the server computer 120 and the local storage 104 resides on the computing device 110. In another embodiment, the shared storage 124 or the local storage 104, or both, may reside anywhere within the distributed data processing environment 100 such that they are accessible to and accessible by the computing device 110 and the server computer 120. The shared storage 124 or the local storage 104 may each be implemented using any type of storage device capable of storing data and configuration files, such as, but not limited to, a database server, hard disk, or flash memory, that may be accessed and used by the server computer 120.

[0022] In some embodiments of the invention, the shared storage 124, or the local storage 104, or both, may each be a hard drive, a memory card, computer output to a laser disk (cold storage), or any form of data storage known in the art, or a combination thereof. In some embodiments, the shared storage 124, or the local storage 104, or both, may each be one or more cloud storage systems, or databases linked to a cloud network, or a combination thereof. In various embodiments, the shared storage 124, or the local storage 104, or a combination thereof, may access, store, or contain, or a combination of, physical event data, or data, or both, shared across the distributed data processing environment 100.

[0023] In various embodiments, the virtual reconstruction tool 112 executes on the server computer 120. In other embodiments, the virtual reconstruction tool 112 may execute on the computing device 110. In some embodiments, the virtual reconstruction tool 112 may be located and / or executed anywhere in the distributed data processing environment 100. In various embodiments, the virtual reconstruction tool 112 may be connected to and in communication with the computing device 110 or the server computer 120 or both. In the illustrated embodiment, the virtual reconstruction tool 112 includes a data collection component 114, a machine learning component 116, and an augmented reality component 118. In the illustrated embodiment, the data collection component 114, the machine learning component 116, and the augmented reality component 118 each execute on the virtual reconstruction tool 112. In some embodiments of the present invention, the data collection component 114, the machine learning component 116, and the augmented reality component 118 may be located and / or executed anywhere within the distributed data processing environment 100, provided that the data collection component 114, the machine learning component 116, and the augmented reality component 118 are in communication with the virtual reconstruction tool 112, the computing device 110, or the server computer 120, or a combination thereof.

[0024] In various embodiments of the present invention, the virtual reconstruction tool 112 can collect historical information from various information sources, such as previously recorded video feeds of different types of physical events, IoT feeds related to any physical events, previous simulation results of computer graphics of different types of physical events, reports of physical events, live video feeds of the area of ​​the current physical event, or any other data from previous physical events known in the art, or combinations thereof, via the data collection component 114. In various embodiments of the present invention, the virtual reconstruction tool 112 can generate a knowledge corpus using the collected historical information (i.e., previously collected data) via the machine learning component 116, which consists of one or more machine learning models. In various embodiments of the present invention, the virtual reconstruction tool 112 can generate a knowledge corpus from the historical information collected via the machine learning component 116 by correlating various parameters of pre-events and post-events, so that the virtual reconstruction tool 112 can predict and determine pre-events and post-events of a current physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can reconstruct physical events and regions of physical events using correlation parameters identified from the knowledge corpus (e.g., indicating how much of an object was destroyed or damaged, indicating how the relative positions of objects may change, the type of physical event that generated the subsequent physical event, the physical event, and the degree of danger or severity of the physical event).

[0025] In some embodiments of the present invention, when a user uses the AR glasses and physically explores the surrounding physical event area, the virtual reconstruction tool 112 can simultaneously collect video and photos to access the surrounding physical event area through the camera component 108. In various embodiments of the present invention, the virtual reconstruction tool 112 can display different scenarios to the user, including the type of physical event, how the object's position was moved or damaged, and how the physical event could have occurred. In various embodiments of the present invention, the virtual reconstruction tool 112 can change the brightness of the generated reconstruction of the physical event, so that the user can "observe" whether the driver was temporarily blinded by the setting sun or the sun's glare from the side mirror.

[0026] In various embodiments of the present invention, the virtual reconstruction tool 112 can predict the type of physical event, the direction of the physical event, the intensity of the physical event, the duration of the physical event, and thus generate a replay of the predicted event, and display the generated replay through the computing device 110 via the augmented reality component 118. In various embodiments of the present invention, the virtual reconstruction tool 112 can generate and display animated graphics on the computing device 110 via the augmented reality component 118, where the animated graphics are augmented reality (virtual) reconstruction simulations that can explain and show how the physical event occurred and can help explain how the area of ​​the physical event occurred. In various embodiments of the present invention, the virtual reconstruction tool 112 can search and analyze previously reconstructed physical event scenarios at similar locations via the data collection component 114, and can overlay the previously reconstructed physical event scenarios on the current physical event and the area of ​​the physical event to look for similarities, differences, or both.

[0027] In various embodiments of the present invention, the virtual reconstruction tool 112 can predict the original positions of various items / objects before a physical event occurs and can display the relative positions of objects whose positions have been changed by the physical event. In various embodiments of the present invention, the virtual reconstruction tool 112 can generate and display, via the augmented reality component 118, animated movements or path trajectories of objects affected by a physical event. In one particular embodiment of the present invention, the virtual reconstruction tool 112 can gather the mechanism, sequence, cause and effect of physical events from written text reports. In one particular embodiment of the present invention, the virtual reconstruction tool 112 can gather the mechanism, sequence, cause and effect of physical events from written text reports. In this particular embodiment, the virtual reconstruction tool 112 can use the written text reports and previously collected data to construct an augmented reality representation of the proposed event sequence (i.e., a virtual reconstruction of the physical event). If the simulated positions, sequences, or outcomes of events and objects contradict the recorded final situation (for example, an item falls on top of another item, but they are in a different configuration than the real-world physical event), then the visualization simulation stops, highlights the contradiction, and requests the user to revise the report or add more information.

[0028] In various embodiments of the present invention, the virtual reconstruction tool 112 can enable and support collaborative explanation drafting and virtual reality simulation. The virtual reconstruction tool 112 allows different users to edit explanations in parallel in a virtual reconstruction simulation of physical events by branching from a master explanation, running the simulation on the branch, and then passing the received modifications back to the master, thus making the latest version of the explanation available to other collaborative users to view. In various embodiments of the present invention, the virtual reconstruction tool 112 can request and perform voting before passing to the master, which allows for crowdsourcing of explanations.

[0029] In various embodiments of the invention, the virtual reconstruction tool 112 allows for comparing the current reconstruction simulation of the physical event with various regulations and limitations obtained from a database: i) building regulations, ii) traffic regulations, iii) safety certifications, iv) technical standards, or v) product warranties. In various embodiments of the invention, the virtual reconstruction tool 112 allows for aspects (e.g., objects, sequences, cause-effect relationships) to be highlighted in the virtual reality or augmented reality simulation and for aspects to be annotated (e.g., with colors or symbols) to indicate when and where a deviation occurs from a restriction, regulation, certification, standard, warranty, or any other various regulations known in the art may occur in the domain of the physical event, or both. In another embodiment of the present invention, using objects, sequences of events, cause-and-effect relationships, and other data sources (e.g., local and remote sensing or crowdsourced data), the virtual reconstruction tool 112 can reconstruct and simulate predictions of how physical events may have occurred and display the predicted scenario of the physical events on the computing device 110 (i.e., virtual reality or AR system) as a replay of the physical events, showing possible risky patterns or “hot spot points” that may cause the events in the physical events, using annotations to mark the events, cross-referencing relevant regulations. The reconstructed replays can be shared from one computing device to another and converted to three-dimensional (3D) models as needed.

[0030] In some embodiments of the invention, the current physical event, the responses, and the associated collaboratively described / edited drafts can be managed and tracked using smart contracts and blockchain. In various embodiments of the invention, a user can wear paired AR-enabled glasses or a virtual reality headset and relive the pre-event, the physical event, and the post-event. For example, the user is driving a vehicle as if it were a vehicle in the physical event, and the AR-enabled glasses display a simulation of a vehicle cutting in front of him in a manner that the vehicle is about to crash. Alternatively, the virtual reconstruction tool 112 can display the virtual reconstruction simulation on the vehicle windshield in real time to the user like a HUD. In various embodiments of the invention, the virtual reconstruction tool 112, via the machine learning component 116, collects reports of historical physical events, simulations of historical physical events, video feeds related to the physical events, and IoT feeds to generate a knowledge corpus. In various embodiments of the present invention, the virtual reconstruction tool 112, via an augmented reality component 118, allows a user to view a virtual recreation of a predictive description of a current physical event through a computing device 110 (e.g., paired augmented reality glasses, or a smartphone).

[0031] In various embodiments of the invention, the virtual reconstruction tool 112 can collect previously recorded and stored video feeds of various physical events, including any dashboard camera, traffic camera, IoT feed, or any other camera or camera feed, or combination thereof, known in the art, via the data collection component 114. In various embodiments of the invention, the virtual reconstruction tool 112 can retrieve images or video or both from IoT feeds from fixed cameras, or IoT devices, or both, in the area of ​​the physical event, where the IoT feeds can include, but are not limited to, the type of force associated with the physical event, temperature, direction of impact (if necessary), time of day, and weather conditions. In various embodiments of the present invention, the virtual reconstruction tool 112 can collect or retrieve reports on physical events from users or from a knowledge repository via the data collection component 114 (e.g., local storage 104, or shared storage 124, or both) while simultaneously collecting video-based simulation results of different types of predicted physical events, where the predicted physical events are ranked from most likely to least likely. In various embodiments of the present invention, the virtual reconstruction tool 112 can cluster the collected data based on pre-events, physical events, and post-events. In various embodiments of the present invention, the virtual reconstruction tool 112 can compare pre-events and post-events to identify changes that have occurred to surrounding areas and objects as a result of events during the physical events.

[0032] In various embodiments of the invention, during the pre-event and post-event comparison, the virtual reconstruction tool 112 can perform a comparison between collected images, video frames, IoT feeds, or any other collected data related to a physical event known in the art, or a combination thereof. In various embodiments of the invention, the images, object boundaries, and object recognition allow the virtual reconstruction tool 112 to recognize what the relative positions and relative context of objects have changed as a result of the physical event.

[0033] In various embodiments of the present invention, the virtual reconstruction tool 112 can analyze the video portion related to the physical event through video analysis and identify the type of physical event, the type of impact (if required), the direction of the impact (if required), the duration of the impact (if required), and the amount / number of impacts of the physical event. In various embodiments of the present invention, the virtual reconstruction tool 112 can generate a knowledge corpus by collecting and storing historical information collected related to the current physical event and previous physical events through the machine learning component 116. In various embodiments of the present invention, the virtual reconstruction tool 112 can generate a knowledge corpus by using the collected data (i.e., previously collected data) to relate how the relative positions of objects can change due to physical events. The virtual reconstruction tool 112 can identify multiple types of physical events that can contribute to the current physical event and the duration of the physical event that resulted in generating such physical event. In various embodiments of the present invention, the knowledge corpus can assist in the virtual reconstruction of the physical event. In various embodiments of the invention, when a user runs the virtual reconstruction tool 112 through a computing device 110 (e.g., AR glasses), the virtual reconstruction tool 112 then connects to the knowledge corpus generated to reconstruct the physical event to retrieve data related to the physical event, where the virtual reconstruction tool 112 uses the knowledge corpus to retrieve data from similar physical events and similar post-events.

[0034] In various embodiments of the invention, the virtual reconstruction tool 112 can enable cameras and sensors (not shown in FIG. 1 ) to perform a capture of the physical event and the surroundings of the area of ​​the physical event at the computing device 110. Additionally, in various embodiments of the invention, the virtual reconstruction tool 112 can enable cameras and sensors at the computing device 110 to access, capture, or retrieve data from an IoT feed, or a combination thereof. In various embodiments of the invention, the virtual reconstruction tool 112 can recognize one or more objects in the area of ​​the physical event, where the virtual reconstruction tool 112 can recognize individual objects, the current location of the objects, and the condition of the objects (e.g., burned, damaged, intact, etc.). In various embodiments of the invention, the virtual reconstruction tool 112 can predict how the event may have occurred post-event based on the collected and analyzed data.

[0035] In various embodiments of the present invention, the virtual reconstruction tool 112 can identify possible explanations of the physical event based on the generated knowledge corpus. In various embodiments of the present invention, the virtual reconstruction tool 112 can send the identified prediction data to the computing device 110 (e.g., augmented reality glasses), where the virtual reconstruction tool 112 generates a virtually animated object related to the current physical event. In various embodiments of the present invention, the virtual reconstruction tool 112 can reconstruct the physical event based on the generated and predicted events that guided and generated the physical event, and can display the virtual reconstruction as an overlay on the area of ​​the physical event through the computing device 110.

[0036] 2 illustrates the operational steps of a virtual reconstruction tool 112, generally indicated at 200, in communication with a computing device 110 in the distributed data processing environment 100 of FIG. 1 for predictively reconstructing physical events using augmented reality according to an embodiment of the present invention. FIG. 2 provides an illustration of one implementation and is not intended to imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated embodiment may be made by one of ordinary skill in the art without departing from the scope of the present invention as recited in the claims.

[0037] In step 202, the virtual reconstruction tool 112 collects various images or video feeds. In various embodiments, the virtual reconstruction tool 112 can collect various images or video feeds from various sources (e.g., the camera component 108, IoT devices, or IoT sensors, or a combination thereof) via the data collection component 114. In various embodiments of the invention, the virtual reconstruction tool 112 can collect provided reports of current physical events and previously provided physical events via the data collection component 114.

[0038] In step 204, the virtual reconstruction tool 112 collects previously generated simulations of physical events. In various embodiments of the invention, the virtual reconstruction tool 112 can collect the previously generated simulations of physical events from a database (e.g., the local storage 104, or the shared storage 124, or both) via the data collection component 114.

[0039] In step 206, the virtual reconstruction tool 112 clusters the collected data. In various embodiments of the invention, the virtual reconstruction tool 112, via the data collection component 114, can cluster the collected data (e.g., collecting video feeds from various sources, submitted reports, or previously generated simulations of physical events) into three categories (e.g., pre-event, physical event, and post-event).

[0040] In step 208, the virtual reconstruction tool 112 analyzes the clustered data. In various embodiments of the present invention, the virtual reconstruction tool 112, via the machine learning component 116, can analyze the clustered data by comparing pre-events and post-events to identify changes caused by a physical event. In various embodiments of the present invention, the virtual reconstruction tool 112, via the machine learning component 116, can use object boundaries and object recognition in the images to identify how the relative positions and relative situations of objects have changed due to a physical event. For example, after an earthquake, the relative positions of objects in a store will change or become damaged (plates falling off shelves and damage to the floor).

[0041] At step 210, the virtual reconstruction tool 112 identifies the relative context of objects involved in the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 may use video analytics to analyze the collected video feed via the machine learning component 116 and identify portions of the collected video feed that are related to the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 may use video analytics to identify the type of physical event, the type of impact that occurred before, during, or after the physical event, the direction of the impact, the duration of the impact, and the number of impacts involved in the current physical event.

[0042] In step 212, the virtual reconstruction tool 112 generates a knowledge corpus. In various embodiments of the present invention, the virtual reconstruction tool 112, via the machine learning component 116, collects and stores historical information collected related to the current physical event and previous physical events. In various embodiments of the present invention, the virtual reconstruction tool 112 can generate the knowledge corpus using the collected historical information related to the current physical event and previous physical events to correlate how the physical event changed the relative positions of objects in the area of ​​the physical event, the type of physical event associated with the subsequent event, and the duration of the physical event that caused the subsequent event, using the machine learning component 116. In various embodiments of the present invention, the virtual reconstruction tool 112 can use the knowledge corpus to assist in the reconstruction of the physical event, using the clustered and identified relevant video portions to construct a simulated physical event scenario.

[0043] In step 214, the virtual reconstruction tool 112 captures data of the physical event area. In various embodiments of the present invention, the virtual reconstruction tool 112 can collect data from the area of ​​the physical event via the data collection component 114, and the user observes the physical event and the area of ​​the physical event with the computing device 110. The cameras and sensors mounted on the AR glasses can capture video and / or photos of the area of ​​the physical event and generate an IoT feed. The captured video and / or photos of the area of ​​the physical event and the generated IoT feed can be analyzed by the virtual reconstruction tool 112 for data points that aid in predicting the event that caused the physical event (i.e., the pre-event). In various embodiments of the present invention, the virtual reconstruction tool 112 can individually recognize each and every object, the current location of the object, the condition of the object (e.g., burned, damaged, intact, dented, scratched, or any other state and condition of the object known in the art). For example, a user wears AR glasses and walks around a vehicle accident, observing the scene for a written report and taking notes. In this embodiment, the virtual reconstruction tool 112 passively acquires video and captures photos of the vehicles involved in the vehicle accident, their damage, their accuracy, and their locations as the user gathers information for the written report.

[0044] In step 216, the virtual reconstruction tool 112 identifies possible explanations for the events that occurred during the physical event. In various embodiments of the present invention, the virtual reconstruction tool 112, the machine learning component 116 can identify possible prior events based on the collected data and the knowledge corpus. In various embodiments of the present invention, the virtual reconstruction tool 112 can identify possible reasons, or explanations, for how the physical event occurred.

[0045] In step 218, the virtual reconstruction tool 112 generates a virtual reconstruction of the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can generate a virtual reconstruction of the physical event via the augmented reality component 118, where the physical reconstruction of the physical event is an AR simulation. In various embodiments of the invention, the virtual reconstruction tool 112 generates a virtual reconstruction of the physical event based on the identified possible prior events.

[0046] In step 220, the virtual reconstruction tool 112 displays a virtual reconstruction of the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can display the generated virtual reconstruction of the physical event on the computing device 110 (e.g., a smartphone or AR glasses) via the augmented reality component 118. In various embodiments of the invention, the displayed virtual reconstruction of the physical event is superimposed on the actual physical event or an area of ​​the physical event, or both.

[0047] 3 illustrates a block diagram of components of a computing device 110 executing a virtual reconstruction tool 112 in the distributed data processing environment 100 of FIG. 1 according to an embodiment of the present invention. FIG. 3 provides an illustration of one implementation and is not intended to imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated embodiment may be made by one of ordinary skill in the art without departing from the scope of the present invention as recited in the claims.

[0048] 3 illustrates a computer system 300, where the computing device 110 represents an embodiment of the computer system 300 including the virtual reconstruction tool 112. The computer system includes a processor 301, a cache 303, a memory 302, a persistent storage 305, a communication unit 307, an input / output (I / O) interface 306, a display 309, an external device(s) 308, and a communication function 304. The communication function 304 provides communication between the cache 303, the memory 302, the persistent storage 305, the communication unit 307, and the input / output (I / O) interface(s) 306. The communication function 304 can be implemented in any architecture designed to pass data and / or control information between a processor (such as a microprocessor, communication, and network processor), system memory, peripheral devices, and any other hardware components in the system. For example, the communications function 304 may be implemented with one or more buses or crossbar switches.

[0049] Memory 302 and persistent storage 305 are computer-readable storage media. In this embodiment, memory 302 includes random access memory (RAM). In general, memory 302 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 303 is a subsequent memory that enhances the performance of processor 301 by holding recently accessed and near recently accessed data from memory 302.

[0050] Program instructions and data used to implement embodiments of the present invention may be stored in persistent storage 305 and memory 302 for execution by one or more respective processors 301 via cache 303. In an embodiment, persistent storage 305 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 305 may include a solid state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0051] The media used by persistent storage 305 can also be removable. For example, a removable hard drive can be used for persistent storage 305. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer readable storage medium that is also part of persistent storage 305.

[0052] Communications unit 307, in these embodiments, provides for communication with other data processing systems or devices. In these examples, communications unit 307 includes one or more network interface cards. Communications unit 307 provides for communication through the use of either or both physical or wireless communications links. Program instructions and data used to implement embodiments of the invention may be downloaded to persistent storage 305 through communications unit 307.

[0053] The I / O interface(s) 306 allow input and output of data to and from other devices that may be connected to the respective computer system. For example, the I / O interface 306 may provide a connection to an external device 308, such as a keyboard, a keypad, a touch screen, or some other suitable input device. The external device 308 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to implement embodiments of the present invention may be stored on such portable computer-readable storage media and may be loaded into the persistent storage 305 via the I / O interface(s) 306. The I / O interface(s) 306 also connect a display 309.

[0054] Display 309 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0055] The programs described herein are identified based on their application for implementing particular embodiments of the invention, however, it should be recognized that any particular program naming herein is used merely for convenience and thus is not intended to limit the invention to use only with any particular application identified or implied, or both, by such naming.

[0056] The present invention may be a system, method, or computer program product, or any combination thereof, including a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform features of the present disclosure.

[0057] A computer readable recording medium may be a tangible device capable of holding and storing a plurality of instructions for use by an instruction execution device. A computer readable medium may be, for example, but not limited to, an electrical recording device, a magnetic recording device, an optical recording device, an electro-magnetic recording device, a semiconductor recording device, or any suitable combination thereof. More specific examples of computer readable recording media include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device having protruding structures within a groove having instructions recorded therein, and any suitable combination thereof. As used herein, a computer-readable recording medium is not to be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves such as wave guides or other communications media (e.g., light pulses passing through fiber optic cable), or electrical signals communicated through wires.

[0058] The computer program instructions described herein can be downloaded from a computer readable recording medium into each computing / processing device, or can be downloaded to an external computer or external recording device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, and combinations thereof. The network can include copper cables, fiber optics, wireless networks, routers, firewalls, switches, gateway computers, and edge servers, and combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions into a computer readable recording medium in each computing / processing device for storage.

[0059] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or one or more conventional procedural programming languages, such as Smalltalk®, object oriented programming languages ​​such as C++, the “C” programming language, or similar programming languages. The computer readable program instructions may be executed entirely on the user computer, partially on the user computer as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user computer through any type of network, including a local area network (LAN), a wide area network (WAN), or the connection may be to an external computer (e.g., through an Internet service provider). In some embodiments, computer readable program instructions may be executed by electrical circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), and state information of the computer readable program instructions may be used to personalize the electrical circuitry to perform features of the present invention.

[0060] Aspects of the invention described herein have been described with reference to flowchart instructions and / or block diagrams of methods, apparatus (systems), and computer-readable storage media and computer program products according to embodiments of the invention. It will be understood that any combination of flowchart illustrations and / or block diagrams and / or blocks in flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions can be provided to a general purpose computer, a special purpose computer, or other processor or other programmable data processing device to produce a machine, which when executed by the processor of the computer or other programmable data processing device produces means for implementing the functions / operations specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof. These computer-readable program instructions that direct a computer, programmable data processing device, and other device, or combinations thereof, to function in a particular manner can also be stored on a computer-readable recording medium, and the computer-readable recording medium having instructions stored therein constitutes an article of manufacture including instructions that implement the functional / operational features specified in the block or blocks of the flowcharts and block diagrams, or combinations thereof.

[0062] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable apparatus, or other device to implement the functions / acts specified in the blocks of the flowcharts and block diagrams, or in a combination thereof, by causing a computer-implemented process to perform a series of operational steps on the computer, other programmable apparatus, or other device.

[0063] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementation operations of systems, methods, and computer programs according to various embodiments of the present invention. In this respect, the flowcharts or block diagrams may represent modules, segments, or portions of instructions, which include one or more executable instructions for implementing a particular logical function(s) (or functions). In some alternative implementations, the functions described in the blocks may be performed other than as illustrated. For example, two blocks shown in succession may be executed simultaneously, substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved. Also, the illustration of the blocks in the block diagrams and / or the illustration of the flowcharts or combinations thereof may be implemented by a system based on special purpose hardware that performs a particular function or operation or executes specific purpose hardware and computer instructions.

[0064] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limiting to the disclosed embodiments. Many modifications or variations will be obvious to those skilled in the art without departing from the scope and spirit of the present disclosure. The terms used in this specification are selected to best explain the principles of the present embodiments, practical applications, or technical improvements over the art found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for predictively reconstructing physical events using augmented reality, the computer-implemented method comprising: using video analytics to analyze video feeds collected from a region of a physical event before and after the physical event involving at least one object to identify a relative status of the object located within the region of the physical event; generating a knowledge corpus including the video analysis and the collected video feeds in conjunction with historical information from previous physical events to correlate the physical event and how the relative positions of objects within the area of ​​the physical event have changed due to the physical event, a type of physical event associated with a subsequent event, and a duration of the physical event that caused the subsequent event; capturing, with a computing device, data in a region of the physical event; identifying possible pre-events based on the captured data and the knowledge corpus; generating a virtual reconstruction of the physical event using the possible prior events, the virtual reconstruction of the physical event being a predicted physical event that is most likely to occur during the physical event; and displaying, by the computing device, the generated virtual reconstruction of the predicted physical events, wherein the displayed virtual reconstruction of the predicted physical events is overlaid on an image of an area of ​​the physical events. Computer-implemented method.

2. Displaying the generated virtual reconstruction of the physical events further comprises: mimicking the lighting, weather conditions and environmental conditions of the area of ​​the physical event when the physical event occurred; predicting original positions of various objects involved in the physical event before the physical event occurs; visually displaying how the relative positions of the objects have changed due to the physical events; and Collecting historical information from various sources of information such as video feeds of different types of physical events, IoT feeds related to any physical events, previously generated computer graphics simulation results of different types of physical events, reports of physical events, and live video feeds of the area of ​​the current physical event. The computer-implemented method of claim 1 , comprising:

3. moreover, comparing the generated virtual reconstruction of the physical event with various regulations and restrictions obtained from a database, the various regulations and restrictions including building regulations, traffic regulations, safety certifications, technical standards, and product warranties; highlighting objects, sequences, and cause-effect relationships within the generated virtual reconstruction of the physical event; and annotating the objects, the sequences, and cause-and-effect relationships with colors or symbols to indicate deviations from limits, rules, certifications, standards, and guarantees that have occurred in the physical events. The computer-implemented method of claim 1 , comprising:

4. moreover, presenting a list of possible pre-event scenarios to a user, the list of possible pre-event scenarios being presented to the user through the computing device; and Displaying to the user different scenarios of how the physical event occurred, including the type of the physical event and explaining how the position of the object was moved or damaged.

2. The computer-implemented method of claim 1, comprising:

5. A computer program product for causing a computer to execute the computer-implemented method according to any one of claims 1 to 4.

6. 1. A computer system for placing a virtual object in an augmented reality environment, the computer system comprising: one or more computer processors; one or more computer readable storage devices; program instructions stored in said one or more computer readable storage devices and for execution by at least one of said one or more computer processors, said stored program instructions comprising: program instructions for analyzing video feeds collected from a region of a physical event before and after the physical event involving at least one object, the program instructions using video analytics to identify a relative status of the object located within the region of the physical event; program instructions for generating a knowledge corpus including the video analysis and the collected video feeds in conjunction with historical information from previous physical events to correlate the physical event and how the relative positions of objects within the area of ​​the physical event have changed due to the physical event, a type of physical event associated with a subsequent event, and a duration of the physical event that caused the subsequent event; program instructions for capturing, by a computing device, data in a region of the physical event; program instructions for identifying possible pre-events based on the captured data and the knowledge corpus; program instructions for generating a virtual reconstruction of the physical event using the possible prior events, the virtual reconstruction of the physical event being a predicted physical event that is most likely to occur during the physical event; and program instructions for displaying, by the computing device, the generated virtual reconstruction of the predicted physical events, the program instructions overlaying the displayed virtual reconstruction of the predicted physical events onto an image of an area of ​​the physical events; Computer system.

7. Displaying the generated virtual reconstruction of the physical events further comprises: program instructions for mimicking lighting, weather conditions, and environmental conditions in an area of ​​the physical event when the physical event occurred; program instructions for predicting original positions of various objects involved in said physical event before said physical event occurs; program instructions for visually displaying how the relative positions of the objects have changed due to the physical events; and program instructions for collecting historical information from various sources of information, such as video feeds of different types of physical events, IoT feeds related to any physical events, previously generated computer graphics simulation results of different types of physical events, reports of the physical events, and live video feeds of the area of ​​the current physical event; 7. The computer system of claim 6, comprising:

8. moreover, program instructions for comparing the generated virtual reconstruction of the physical event with various regulations and restrictions obtained from a database, the various regulations and restrictions including building regulations, traffic regulations, safety certifications, technical standards, and product warranties; program instructions for highlighting objects, sequences, and cause-effect relationships within the generated virtual reconstruction of the physical events; and program instructions for annotating the objects, the sequences, and cause-and-effect relationships with colors or symbols to indicate deviations from constraints, rules, certifications, standards, and guarantees that have occurred in the physical events; 7. The computer system of claim 6, comprising:

9. moreover, presenting a list of possible pre-event scenarios to a user, wherein the list of possible pre-event scenarios is presented to the user through the computing device; and program instructions for displaying to the user different scenarios of how the physical event occurred, including the type of the physical event and describing how the position of the object was moved or damaged; 7. The computer system of claim 6, comprising:

Images