Predictive virtual reconstruction of a physical environment

Abstract

A method for predictively reconstructing a physical event using augmented reality, comprising: identifying a relative state of objects located in a physical event area by analyzing video feeds collected from the physical event area before and after a physical event involving at least one of the objects using video analytics, creating a knowledge corpus comprising the video analytics and collected video feeds associated with the physical event and historical information, and obtaining, by a computing device, data of the physical event area, identifying a possible precursor event based on the obtained data and the knowledge corpus, and generating a virtual reconstruction of the physical event using the possible precursor event, displaying, by the computing device, the generated virtual reconstruction of the predicted physical event, wherein the displayed virtual reconstruction of the predicted physical event overlays an image of the physical event area.

Description

Background Technology

[0001] This invention relates generally to the field of augmented reality, and more specifically to the predictive virtual reconstruction of physical environments.

[0002] Augmented reality (AR) is an interactive experience of a real-world environment in which objects residing in the real world are augmented by computer-generated perceptual information (sometimes across multiple perceptual modalities, including visual, auditory, tactile, haptic, and olfactory). AR can be defined as a system that achieves three fundamental characteristics: a combination of the real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects. Overlayed sensory information can be added to the natural environment or masked within it. AR experiences can be seamlessly interwoven with the physical world, making them perceived as immersive aspects of the real environment. In this way, augmented reality alters people's continuous perception of the real-world environment, whereas virtual reality completely replaces the user's real-world environment with a simulated one. Augmented reality involves two main synonymous terms: "mixed reality" and "computer-mediated reality."

[0003] Augmented reality (AR) is used to enhance natural environments or situations and provide a richer, more perceptual experience. With the help of advanced AR technologies (e.g., adding computer vision, incorporating AR cameras into smartphone applications, and object recognition), information about the real world around the user becomes interactive. Information about the environment and its objects is overlaid on the real world. Immersive perceptual information is sometimes combined with supplementary information such as scores in live video feeds of sporting events. This combines the benefits of both augmented reality and heads-up display (HUD) technologies. Summary of the Invention

[0004] This invention discloses a method, computer program product, and system for predictively reconstructing physical events using augmented reality. The computer-implemented method includes: identifying the relative state of objects located in a physical event region by analyzing video feeds collected from a physical event region before and after a physical event involving at least one of the objects using video analytics; creating a knowledge corpus including video analytics and collected video feeds associated with the physical event, as well as historical information from previous physical events, to correlate how the relative positions of objects in the physical event region change due to the physical event, the type of physical event associated with the subsequent event, and the duration of the physical event that produced the subsequent event; acquiring data of the physical event region by a computing device; identifying possible precursor events based on the acquired data and the knowledge corpus; generating a virtual reconstruction of the physical event using the possible precursor events, wherein the physical reconstruction of the physical event is a predicted physical event most likely to have occurred during the physical event; and displaying the generated virtual reconstruction of the predicted physical event by the computing device, wherein the displayed virtual reconstruction of the predicted physical event overlays an image of the physical event region.

[0005] Brief description of the attached figures

[0006] Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings, in which:

[0007] Figure 1 This is a functional block diagram illustrating a distributed data processing environment according to an embodiment of the present invention;

[0008] Figure 2 The following are the operational steps of a virtual reconstruction tool according to an embodiment of the present invention. Figure 1 Within a distributed data processing environment, it communicates with computing devices to predictively reconstruct physical events using augmented reality; and

[0009] Figure 3 The invention describes an embodiment of the invention in Figure 1 A block diagram of the components of a computing device that executes virtual reconfiguration tools within a distributed data processing environment. Detailed Implementation

[0010] During the inspection of the physical environment, the user assesses surrounding physical events, collecting samples such as broken objects, the relative positions of objects, different markings on the ground, damage to the surrounding area, and / or any other information known in the art related to the physical event. Thus, the user can analyze the physical event and the area surrounding it (i.e., the physical event area) to determine the type of event that may have occurred (e.g., a car accident, a natural disaster, or any situation known in the art involving a physical event). A physical event is an event involving one or more physical objects interacting with another physical object (e.g., a car accident, a natural disaster, or any situation known in the art involving a physical object). In the prior art, methods for determining what events occurred before a physical event (i.e., precursor events) and what events occurred during a physical event involve the user attempting to guess many different events that may have occurred in and around the physical event area. Often, it is difficult for the user to correctly guess or determine the events that led to the physical event, as well as the events that occurred during and after the physical event, collectively referred to as precursor events. Furthermore, in the prior art, in many cases, predicting and determining precursor events and events following the physical event (i.e., post-events) is impossible, which can lead to delays and decision-making errors.

[0011] Embodiments of the present invention can address the aforementioned problems by providing an artificial intelligence (AI)-based augmented reality system that can analyze the surrounding area and use one or more machine learning models trained on historical (physical) events to predict what happened in the physical event area. For example, one or more machine learning models, including Markov models, reinforcement learning (RL), recurrent neural networks, sequence mining models, and / or time series models, can be used to model event sequences and predict precursor events. Furthermore, embodiments of the present invention can improve upon the prior art by displaying predicted precursor events, events occurring during the physical event, and subsequent events as predicted by one or more machine learning models on a smartphone via augmented reality (AR) glasses or an AR application. Embodiments of the present invention can improve upon the prior art by presenting a list of potential scenarios for precursor events to a user using a recurrent neural network (RNN) and a sequence of events generated during the physical event, wherein the list of potential scenarios for the physical event is sorted from most likely to least likely to have occurred.

[0012] Embodiments of the present invention can improve AR technology by utilizing a data collection module to collect historical physical event information, which includes: video feeds (i.e., acquired videos) of different types of physical events (previously recorded or reported physical events); Internet of Things (IoT) feeds involving any physical event; and computer-generated simulation results of different types of physical events. Therefore, embodiments of the present invention can improve the domain of AR technology by utilizing machine learning models to create a knowledge corpus by collecting and storing historical information associated with current and previous physical events. Embodiments of the present invention create the knowledge corpus by correlating various parameters of precursor and post-event events from previously collected data, enabling embodiments of the present invention to effectively predict physical events and virtually reconstruct regions and physical events (e.g., demonstrating how different objects are destroyed or damaged, showing how the relative positions of objects may have changed, listing the types of events that may have occurred to create physical events and post-events as learned by one or more pattern machine learning models; and determining the criticality or severity of physical events). The classification of the criticality or severity of physical events can be accomplished using a bidirectional long short-term memory (LSTM) model.

[0013] When a user views a surrounding physical event area through AR glasses, embodiments of the present invention can assess the surrounding physical event area. Embodiments of the present invention can virtually display simulations demonstrating how physical events might occur via augmented reality, including different types of potential precursor events, how objects move, or how objects are damaged. Embodiments of the present invention can mimic lighting, weather conditions, and / or environmental conditions of the physical event area when a physical event occurs via AI-enabled augmented reality. For example, a user can observe whether a driver is temporarily blinded by the sunset or solar glare reflected in a side mirror.

[0014] Embodiments of the present invention can predict the type, direction, and duration of intensity of physical events, and accordingly use one or more machine learning models (e.g., RNNs) to display the predicted events to users via AR glasses or AR-enabled computing devices. Embodiments of the present invention can reconstruct physical events and display the reconstruction via augmented reality components. Embodiments of the present invention can display animated augmented reality graphics to explain how a physical event might create a physical event region to help users reconstruct precursor events and understand details about the physical event (e.g., earthquakes, hurricanes, traffic patterns, etc.). Embodiments of the present invention can overlay AR-simulated or learned reconstructed physical event scenarios at locations similar to the current physical event region to identify any similarities and / or differences between the physical event and / or the published event. Embodiments of the present invention can predict the original positions of different objects before a physical event occurs and display how the relative positions of objects change due to the physical event. For example, object trajectories can be virtually displayed via AR simulation.

[0015] It should be noted that in the described embodiments, the participants (i.e., users) have consented to their images being captured, uploaded, saved, recorded, and monitored. Furthermore, the participants are aware of the possibility of such recording and monitoring. In different embodiments, for example, when downloading or operating embodiments of the invention, these embodiments present terms and conditions that allow interested parties to opt in or out of participation.

[0016] The embodiments of the present invention can be implemented in various forms, and reference will be made subsequently to the accompanying drawings (i.e., Figures 1-3 Discuss the details of the exemplary implementation method.

[0017] Figure 1 This is a functional block diagram illustrating a distributed data processing environment, typically represented by 100, according to an embodiment of the present invention. The term "distributed" as used herein describes a computer system comprising multiple physically distinct devices that operate together as a single computer system. Figure 1 This illustration is provided only to illustrate one implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Those skilled in the art can make many modifications to the described environment without departing from the scope of the invention as set forth in the claims. The distributed data processing environment 100 includes computing devices 110 and server computers 120 interconnected via a network 130.

[0018] Network 130 may be, for example, a Storage Area Network (SAN), a telecommunications network, a Local Area Network (LAN), a Wide Area Network (WAN) (such as the Internet), a wireless technology for exchanging data over short distances (using short-wavelength ultra-high frequency (UHF) radio waves in the Industrial, Scientific, and Medical (ISM) band from 2.4 to 2.485 GHz from fixed and mobile devices, and establishing a Personal Area Network (PAN), or a combination of the three), and may include wired, wireless, or fiber optic connections. Network 130 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, and / or video signals, including multimedia signals containing voice, data, text, and / or video information. Typically, network 130 may be a connection between supporting computing device 110 and server computer 120 and any other computing and / or storage devices within the distributed data processing environment 100. Figure 1 Any combination of connections and protocols for communication between (not shown in the image).

[0019] In some embodiments of the present invention, computing device 110 may be, but is not limited to, a standalone device, client, server, laptop computer, tablet computer, netbook computer, personal computer (PC), smartphone, desktop computer, smart TV, smartwatch, radio, stereo system, cloud-based service (e.g., cognitive cloud-based service), AR glasses, virtual reality headset, any HUD known in the art, and / or any programmable electronic computing device capable of communicating with various components and devices within the distributed data processing environment 100 via network 130 or any combination thereof. Generally, computing device 110 may represent any programmable computing device or combination of programmable computing devices capable of executing machine-readable program instructions and communicating with users of other computing devices via network 130 and / or capable of executing machine-readable program instructions and communicating with server computer 120.

[0020] In various embodiments, camera component 108 is implemented on computing device 110. In some embodiments, camera component 108 may be located and / or implemented anywhere within distributed data processing environment 100. Camera component 108 may be one or more cameras known in the art. In various embodiments, camera component 108 may provide a video feed (video feed) of the content being viewed by a user to computing device 110 or more specifically to virtual reconstruction tool 112. The video feed may be transmitted in real time as a live video feed or recorded and transmitted as a recorded video feed. As used herein, the term "video feed" may refer to one or both types of video feed. In various embodiments, virtual reconstruction tool 112 may enable computing device 110 to store acquired video feeds and / or photographs to shared storage 124 and / or local storage 104. In various embodiments, camera component 108 may be capable of recording, transmitting, and storing video feeds and capturing, transmitting, and storing photographs.

[0021] In some embodiments of the invention, computing device 110 may represent any programmable electronic computing device or a combination of programmable electronic computing devices capable of executing machine-readable program instructions, manipulating executable machine-readable instructions, and communicating via a network (such as network 130) with server computer 120 and other computing devices (not shown) within distributed data processing environment 100. Computing device 110 may include instances of user interface (interface) 106 and local storage 104. Figure 1 In various embodiments not described herein, computing device 110 may have multiple user interfaces. Figure 1 In other embodiments not described herein, the distributed data processing environment 100 may include multiple computing devices, multiple server computers, and / or multiple networks. The computing devices 110 may include internal and external hardware components, such as those described above. Figure 3 Described and described in further detail.

[0022] User interface (interface) 106 provides an interface to predictive virtual reconstruction tool (virtual reconstruction tool) 112. Computing device 110, via user interface 106, enables users and / or clients to interact with virtual reconstruction tool 112 and / or server computer 120 in various ways, such as sending and receiving program instructions, sending and / or receiving messages, updating data, sending data, inputting data, editing data, collecting data, and / or receiving data. In one embodiment, interface 106 may be a graphical user interface (GUI) or a web user interface (WUI) and may display at least text, documents, a web browser window, user options, application interfaces, and operation instructions. Interface 106 may include information presented to the user (such as graphics, text, and sound) and control sequences used by the user to control operations. In another embodiment, interface 106 may be mobile application software that provides an interface between the user of computing device 110 and server computer 120. Mobile application software or "app" may be designed to run on smartphones, tablets, and other computing devices. In one embodiment, interface 106 enables a user of computing device 110 to at least send data, input data, edit data (annotate), collect data, and / or receive data.

[0023] Server computer 120 may be a standalone computing device, management server, web server, mobile computing device, one or more client servers, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, server computer 120 may represent a server computing system utilizing multiple computers, such as, but not limited to, server systems, such as in a cloud computing environment. In another embodiment, server computer 120 may represent a computing system utilizing cluster computers and components (e.g., database server computers, application server computers, etc.), which act as a single seamless resource pool when accessed within the distributed data processing environment 100. Server computer 120 may include internal and external hardware components, such as those relative to... Figure 3 As described and further detailed.

[0024] Each of shared storage 124 and local storage 104 can be a data / knowledge repository and / or database that can be written to and / or read by one or a combination of virtual reconfiguration tool 112, server computer 120, and computing device 110. In the depicted embodiment, shared storage 124 resides on server computer 120 and local storage 104 resides on computing device 110. In another embodiment, shared storage 124 and / or local storage 104 may reside elsewhere within the distributed data processing environment 100, provided that each is accessible to and can be accessed by computing device 110 and server computer 120. Shared storage 124 and / or local storage 104 may each be implemented using any type of storage device capable of storing data and configuration files accessible and utilized by server computer 120, such as, but not limited to, database servers, hard disk drives, or flash memory.

[0025] In some embodiments of the invention, shared storage 124 and / or local storage 104 may each be a hard disk drive, memory card, computer output to a laser disk (cold storage), and / or any form of data storage known in the art. In some embodiments, shared storage 124 and / or local storage 104 may each be one or more cloud storage systems and / or databases linked to a cloud network. In various embodiments, shared storage 124 and / or local storage 104 may access, store, and / or accommodate physical event data, and / or data shared throughout the distributed data processing environment 100.

[0026] In various embodiments, the virtual reconstruction tool 112 executes on server computer 120. In other embodiments, the virtual reconstruction tool 112 may execute on computing device 110. In some embodiments, the virtual reconstruction tool 112 may be located and / or executed anywhere within the distributed data processing environment 100. In various embodiments, the virtual reconstruction tool 112 may be connected to and communicate with computing device 110 and / or server computer 120. In the depicted embodiments, the virtual reconstruction tool 112 includes a data collection component 114, a machine learning component 116, and an augmented reality component 118. In the depicted embodiments, the data collection component 114, the machine learning component 116, and the augmented reality component 118 each execute on the virtual reconstruction tool 112. In some embodiments of the present invention, the data collection component 114, the machine learning component 116, and the augmented reality component 118 may be located and / or executed anywhere within the distributed data processing environment 100, as long as the data collection component 114, the machine learning component 116, and the augmented reality component 118 can communicate with the virtual reconstruction tool 112, the computing device 110, and / or the server computer 120.

[0027] In various embodiments of the invention, the virtual reconstruction tool 112 can collect historical information from various information sources via the data collection component 114, such as: video feeds of different types of previously recorded physical events, IoT feeds related to any physical event, computer graphics simulation results of different types of previously generated physical events, physical event reports, live video feeds of the current physical event area, and / or any other data from previous physical events known in the art. In various embodiments of the invention, the virtual reconstruction tool 112, via the machine learning component 116, consists of one or more machine learning models that can use the collected historical information (i.e., previously collected data) to create a knowledge corpus. In various embodiments of the invention, the virtual reconstruction tool 112, via the machine learning component 116, can create a knowledge corpus by correlating different parameters of precursor and follow-up events from the collected historical information; therefore, the virtual reconstruction tool 112 can predict and determine precursor and follow-up events of the current physical event. In various embodiments of the invention, the virtual reconstruction tool 112 may use associated parameters identified from a knowledge corpus to reconstruct physical events and physical event regions (e.g., showing how different objects are destroyed or damaged, showing that changes in the relative positions of objects can alter the physical events, identifying the type of physical event, the physical event itself, and the criticality or severity of the physical event after its creation).

[0028] In some embodiments of the invention, when a user uses AR glasses and physically observes the surrounding physical event area, the virtual reconstruction tool 112 can simultaneously collect video and photographs to assess the surrounding physical event area via camera component 108. In various embodiments of the invention, the virtual reconstruction tool 112 can display to the user different scenarios showing how physical events might occur, including the type of physical event and how the position of the demonstrated object has moved or been damaged. In various embodiments of the invention, the virtual reconstruction tool 112 can alter the lighting of the generated physical event reconstruction simulation, allowing the user to "observe" whether the driver is temporarily blinded by the sunset or solar glare reflected from the side mirrors.

[0029] In various embodiments of the invention, the virtual reconstruction tool 112 can predict the type, direction, intensity, and duration of a physical event, and accordingly generate a reproduction of the predicted event and display the generated reproduction via the augmented reality component 118 on the computing device 110. In different embodiments of the invention, the virtual reconstruction tool 112 can generate and display animated graphics on the computing device 110 via the augmented reality component 118, wherein the animated graphics are augmented reality (virtual) reconstruction simulations that can illustrate and depict how a physical event might occur and help explain how the physical event region is created. In various embodiments of the invention, the virtual reconstruction tool 112 can retrieve and analyze previously reconstructed physical event scenarios at similar locations via the data collection component 114, and overlay the previously reconstructed physical event scenarios onto the current physical event and the physical event region to find similarities and / or differences.

[0030] In various embodiments of the invention, the virtual reconstruction tool 112 can predict the original positions of different items / objects before a physical event occurs and can display the relative positions of objects changed due to the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can generate and display animated movement or path trajectories of objects affected by a physical event via the augmented reality component 118. In a particular embodiment of the invention, the virtual reconstruction tool 112 can collect the physical event mechanism, sequence, cause, and effect of a physical event from a written text report. In this particular embodiment, the virtual reconstruction tool 112 can analyze and interpret a documented written text report. In this particular embodiment, the virtual reconstruction tool 112 can use the collected written text report and previously collected data to construct an augmented reality representation of the provided event sequence (i.e., a virtual reconstruction of the physical event). When the simulated position, sequence, or outcome of events and objects is inconsistent with the recorded final state (e.g., an item falls on top of another item, even though they are configured differently in a real-world physical event), the visualization simulation stops, the inconsistency is highlighted, and the user is requested to change or add more information to the report.

[0031] In various embodiments of the invention, the virtual reconstruction tool 112 enables and supports collaborative interpretation drafting and virtual reality simulation. By starting from a main interpretation branch, running the simulation within that branch, and then submitting the received modifications to the master device, making it available to other collaborating users to view the latest version of the interpretation, the virtual reconstruction tool 112 allows different users to edit the interpretation in parallel with the virtual reconstruction simulation of physical events. In various embodiments of the invention, the virtual reconstruction tool 112 is capable of pulling requests and conducting approval voting before submission to the master device, enabling crowdsourced interpretation.

[0032] In various embodiments of the invention, the virtual reconstruction tool 112 can compare a current virtual reconstruction simulation of a physical event with different rules and constraints derived from a database (including: i) building codes, ii) traffic regulations, iii) safety certifications, iv) engineering specifications, and / or v) product warranties). In various embodiments of the invention, the virtual reconstruction tool 112 can highlight aspects of the virtual reality or augmented reality simulation (e.g., objects, sequences, causal relationships) and can annotate these aspects (e.g., by color or symbols) to indicate when and where deviations from constraints, specifications, certifications, specifications, warranties, and / or any other different rules known in the art have occurred in the physical event region. In another embodiment of the invention, using objects, event sequences, causal relationships, and other data sources (e.g., video feed data from local and remote sensing or crowdsourced data), the virtual reconstruction tool 112 can reconstruct and simulate predictions of how physical events will occur, and can display the predicted scenario of the physical event as a playback of the physical event on the computing device 110 (i.e., a virtual reality system or an AR system). The replay can use annotations to indicate "hot spots" to mark potential risk patterns or events that might trigger events in the physical event when cross-referencing relevant regulations. The reconstructed replay can be shared from one computing device to another and can be converted into a three-dimensional (3D) model as needed.

[0033] In some embodiments of the invention, smart contracts and blockchain may be used to manage and track current physical events, replays, and associated drafts of collaborative interpretations / editings. In various embodiments of the invention, a user may wear AR-enabled glasses or a virtual reality headset and re-store precursor events, physical events, and post-events. For example, a user drives a car as if it were the car in a physical event, and AR-enabled glasses display a simulation of the car cutting in front of him in a manner that could cause a car accident. Alternatively, the virtual reconstruction tool 112 may display the virtual reconstruction simulation as the user's HUD on the vehicle's windshield in real time. In different embodiments of the invention, the virtual reconstruction tool 112 may collect historical physical event reports, historical physical event simulations, physical event-related video feeds, and IoT feeds via machine learning component 114 to create a knowledge corpus. In various embodiments of the invention, the virtual reconstruction tool 112 may enable a user to view a virtual reconstruction of the interpretation of the predicted events in the current physical event via augmented reality component 118 through computing device 110 (e.g., a pair of augmented reality glasses or a smartphone).

[0034] In various embodiments of the invention, the virtual reconstruction tool 112 is capable of collecting video feeds of different physical events previously recorded and saved via the data collection component 114, including any dashboard camera, traffic camera, IoT feed, and / or any other camera or camera feed known in the art. In various embodiments of the invention, the virtual reconstruction tool 112 can retrieve images and / or videos from IoT feeds from fixed cameras and / or IoT devices in the physical event area, wherein the IoT feeds may include, but are not limited to, the type of force associated with the physical event, temperature, impact direction (if necessary), time of day, and weather conditions. In various embodiments of the invention, the virtual reconstruction tool 112 can collect or retrieve reports about physical events from users or from a knowledge repository (e.g., local storage 104 and / or shared storage 124) via the data collection component 114, while simultaneously collecting video-based simulation results of different types of predicted physical events, wherein the predicted physical events are ranked from most likely to least likely. In various embodiments of the invention, the virtual reconstruction tool 112 can aggregate the collected data based on precursor events, physical events, and post-events. In various embodiments of the invention, the virtual reconstruction tool 112 can compare precursor events and subsequent events to identify changes that have occurred in the surrounding area and objects due to events during a physical event.

[0035] In various embodiments of the invention, during comparisons between precursor and subsequent events, the virtual reconstruction tool 112 can perform comparisons between collected images, video frames, IoT feeds, and / or any other collected data known in the art related to physical events. In various embodiments of the invention, utilizing image object boundaries and object recognition, the virtual reconstruction tool 112 can identify how the relative positions and relative states of objects change due to physical events.

[0036] In various embodiments of the invention, the virtual reconstruction tool 112 can analyze video portions related to physical events via video analytics and identify the type of physical event, the type of impact (if necessary), the direction of the impact (if necessary), the duration of the impact (if necessary), and the amount / number of impacts in the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can create a knowledge corpus by collecting and storing historical information associated with the current physical event and previous physical events via the machine learning component 116. In various embodiments of the invention, the virtual reconstruction tool 112 can create a knowledge corpus by using collected data (i.e., previously collected data) to correlate how the relative positions of objects can change due to physical events. The virtual reconstruction tool 112 can identify various types of physical events that can produce the current subsequent event and the duration of the physical events that create such events. In various embodiments of the invention, the knowledge base can assist in the virtual reconstruction of physical events. In various embodiments of the invention, when a user executes the virtual reconstruction tool 112 via a computing device 110 (e.g., AR glasses), the virtual reconstruction tool 112 connects to a created knowledge corpus and retrieves data related to the physical event to reconstruct the physical event, wherein the virtual reconstruction tool 112 uses the knowledge corpus to retrieve data from similar physical events and similar subsequent events.

[0037] In various embodiments of the invention, the virtual reconstruction tool 112 enables the camera and sensors (which are executed on the computing device 110) to be reconstructed. Figure 1 (Not shown) Captures the physical event and the area surrounding the physical event. Additionally, in various embodiments of the invention, the virtual reconstruction tool 112 enables cameras and sensors running on the computing device 110 to access, acquire, and / or retrieve data from IoT feeds. In various embodiments of the invention, the virtual reconstruction tool 112 can identify one or more objects in the physical event area, wherein the virtual reconstruction tool 112 can individually identify the object, the object's current location, and the object's state (e.g., burned, cracked, intact, etc.). In various embodiments of the invention, the virtual reconstruction tool 112 can predict how the event will occur afterward based on the collected and analyzed data.

[0038] In various embodiments of the invention, the virtual reconstruction tool 112 can identify possible interpretations of a physical event based on a created knowledge base. In different embodiments of the invention, the virtual reconstruction tool 112 can send the identified prediction data to a computing device 110 (e.g., augmented reality glasses), wherein the virtual reconstruction tool 112 can create a virtual animated object associated with the current physical event. In different embodiments of the invention, the virtual reconstruction tool 112 can reconstruct a physical event based on generated predicted events that cause and occur during the physical event, and the virtual reconstruction can be displayed as an overlay on the physical event area by the computing device 110.

[0039] Figure 2 The invention illustrates an embodiment of the present invention. Figure 1 The virtual reconstruction tool 112, which communicates with computing device 110 within a distributed data processing environment 100, provides operational steps (generally labeled 200) for predictively reconstructing physical events using augmented reality. Figure 2 An illustration of an implementation is provided, but it does not imply any limitation regarding the environment in which different embodiments may be implemented. Those skilled in the art will make many modifications to the described environment without departing from the scope of the invention as set forth in the claims.

[0040] In step 202, the virtual reconstruction tool 112 collects various image or video feeds. In various embodiments, the virtual reconstruction tool 112 can collect various image and / or video feeds from various sources (e.g., camera component 108, IoT devices, and / or IoT sensors) via the data collection component 114. In various embodiments of the invention, the virtual reconstruction tool 112 can collect reports of current physical events and previously recorded physical events via the data collection component 114.

[0041] In step 204, the virtual reconstruction tool 112 collects previously created physical event simulations. In various embodiments of the invention, the virtual reconstruction tool 112 can collect previously created physical event simulations from a database (e.g., local storage 104 and / or shared storage 124) via a data collection component 114.

[0042] In step 206, the virtual reconstruction tool 112 clusters the collected data. In different embodiments of the invention, the virtual reconstruction tool 112 can cluster the collected data (e.g., video feeds, document reports, and / or previously created physical event simulations collected from different sources) into three categories (e.g., precursor events, physical events, and post-events) via the data collection component 114.

[0043] In step 208, the virtual reconstruction tool 112 analyzes the clustered data. In various embodiments of the invention, the virtual reconstruction tool 112, via the machine learning component 114, can analyze the clustered data by comparing precursor and post-event events to identify changes caused by physical events. In various embodiments of the invention, the virtual reconstruction tool 112, via the machine learning component 114, uses image object boundaries and object recognition to identify how the relative positions and states of objects have changed due to physical events. For example, after an earthquake, the relative positions of items in a store will have changed or they may have been damaged (plates falling from shelves and breaking on the floor).

[0044] In step 210, the virtual reconstruction tool 112 identifies the relative states of objects involved in the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can use video analytics via the machine learning component 114 to analyze collected video feeds and identify portions of the collected video feeds relevant to the physical event. In different embodiments of the invention, the virtual reconstruction tool 112 can use video analytics via the machine learning component 114 to identify the type of physical event, the type of impacts occurring before, after, or during the physical event, the direction of the impacts, the duration of the impacts, and the number of impacts involved in the current physical event.

[0045] In step 212, the virtual reconstruction tool 112 creates a knowledge corpus. In various embodiments of the invention, the virtual reconstruction tool 112, via the machine learning component 114, can create the knowledge corpus by collecting and storing historical information associated with the current physical event and previous physical events. In different embodiments of the invention, the virtual reconstruction tool 112, via the machine learning component 114, can create the knowledge corpus by using the collected historical information associated with the current physical event and previous physical events to correlate how the relative positions of objects in the physical event region change due to physical events, the type of physical event associated with subsequent events, and the duration of the physical event leading to the subsequent event. In various embodiments of the invention, the virtual reconstruction tool 112 can use the knowledge corpus to help reconstruct physical events by constructing simulated physical event scenes using clustered data and identified relevant video portions.

[0046] In step 214, the virtual reconstruction tool 112 acquires data at the physical event area. In various embodiments of the invention, the virtual reconstruction tool 112 can acquire and collect data from the physical event area via the data collection component 114 as the user observes the physical event and the physical event area using the computing device 110. Cameras and sensors mounted on the AR glasses can acquire video and / or photographs of the physical event area and create IoT feeds. The acquired video and / or photographs of the physical event area and the created IoT feeds can be analyzed by the virtual reconstruction tool 112 for data points that can help predict events that caused the physical event (i.e., precursor events). In various embodiments of the invention, the virtual reconstruction tool 112 can individually identify each object, the object's current location, and the state of objects involved in or associated with the physical event (e.g., burned, damaged, intact, dented, scratched, and / or any other condition or state of any object known in the art). For example, a user wearing AR glasses walks around a car accident, observes the scene, and takes notes for a written report. In this example, while the user is collecting information for a written report, the virtual reconstruction tool 112 passively collects video and obtains photos of the cars involved in the car accident, the damage to those cars, the angles of the cars, and the positions of the cars.

[0047] In step 216, the virtual reconstruction tool 112 identifies possible explanations for events that occurred during the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 and the machine learning component 116 can identify possible precursory events based on collected data and a knowledge base. In various embodiments of the invention, the virtual reconstruction tool 112 can identify possible causes or explanations regarding how the physical event occurred.

[0048] In step 218, the virtual reconstruction tool 112 generates a virtual reconstruction of the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can generate the virtual reconstruction of the physical event via the augmented reality component 118, wherein the physical reconstruction of the physical event is an AR simulation. In various embodiments of the invention, the virtual reconstruction tool 112 generates the virtual reconstruction of the physical event based on identified possible precursor events.

[0049] In step 220, the virtual reconstruction tool 112 displays a virtual reconstruction of the physical event. In various embodiments of the invention, the virtual reconstruction tool 112 can display the generated virtual reconstruction of the physical event on a computing device 110 (e.g., a smartphone or AR glasses) via an augmented reality component 118. In various embodiments of the invention, the displayed virtual reconstruction of the physical event may overlay the actual physical event and / or the physical event area.

[0050] Figure 3 The invention describes an embodiment of the invention in Figure 1 A block diagram of the components of a computing device 110 that executes a virtual reconfiguration tool 112 within a distributed data processing environment 100. Figure 3 An illustration of an implementation is provided, but it does not imply any limitation regarding the environment in which different embodiments may be implemented. Those skilled in the art will make many modifications to the described environment without departing from the scope of the invention as set forth in the claims.

[0051] Figure 3 A computer system 300 is depicted, wherein computing device 110 represents an example of a computer system 300 including a virtual reconfiguration tool 112. The computer system includes a processor 301, a cache 303, memory 302, persistent storage 305, a communication unit 307, an input / output (I / O) interface 306, a display 309, external devices 308, and a communication structure 304. The communication structure 304 provides communication between the cache 303, memory 302, persistent storage 305, communication unit 307, and input / output (I / O) interface 306. The communication structure 304 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communication structure 304 can be implemented using one or more buses or crossbars.

[0052] Memory 302 and persistent storage 305 are computer-readable storage media. In this embodiment, memory 302 includes random access memory (RAM). Typically, memory 302 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 303 is a fast memory that enhances the performance of processor 301 by storing recently accessed data from memory 302 and data near the recently accessed data.

[0053] Program instructions and data for implementing embodiments of the present invention may be stored in persistent storage 305 and memory 302 for execution by one or more of the respective processors 301 via cache 303. In an embodiment, persistent storage 305 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 305 may include a solid-state drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0054] The media used in persistent storage 305 can also be removable. For example, a removable hard drive can be used for persistent storage 305. Other examples include optical discs and disks, thumb drives and smart cards, which are inserted into the drive to be transferred to another computer-readable storage medium that is also part of persistent storage 305.

[0055] In these examples, communication unit 307 provides communication with other data processing systems or devices. In these examples, communication unit 307 includes one or more network interface cards. Communication unit 307 can provide communication via physical and / or wireless communication links. Program instructions and data for implementing embodiments of the invention can be downloaded to permanent storage 305 via communication unit 307.

[0056] I / O interface 306 enables data input and output with other devices that can be connected to each computer system. For example, I / O interface 306 can provide connectivity to external device 308 (such as a keyboard, keypad, touchscreen, and / or other suitable input devices). External device 308 may also include portable computer-readable storage media, such as thumb drives, portable optical discs or disks, and memory cards. Software and data used to implement embodiments of the invention can be stored on such portable computer-readable storage media and loaded onto persistent storage 305 via I / O interface 306. I / O interface 306 is also connected to display 309.

[0057] The display 309 provides a mechanism for displaying data to the user and may be, for example, a computer monitor.

[0058] The procedures described herein are identified based on their implementation in specific embodiments of the invention. However, it should be understood that any particular procedural terminology used herein is for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such terminology.

[0059] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0060] A computer-readable storage medium can be any tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards, or protrusions in recesses having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0062] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0063] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0064] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0065] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions that execute on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0066] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0067] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for predictively reconstructing physical events using augmented reality, the computer-implemented method comprising: The relative state of objects located in a physical event region is identified by analyzing video feeds collected from the physical event region before and after a physical event involving at least one of the objects; Create a knowledge corpus that includes video analysis and collected video feeds associated with the physical event, as well as historical information from previous physical events, to correlate how the relative positions of objects in the area of ​​the physical event change due to the physical event, the type of physical event associated with the subsequent event, and the duration of the physical event that produced the subsequent event. Data of the physical event region is acquired by a computing device; Based on the acquired data and the knowledge corpus, possible precursory events are identified; A virtual reconstruction of the physical event is generated using the possible precursor events, wherein the physical reconstruction of the physical event is the predicted physical event that is most likely to have occurred during the physical event. A virtual reconstruction of a predicted physical event generated by the computing device, wherein the displayed virtual reconstruction of the predicted physical event covers an image of the area of ​​the physical event; as well as The interpretation of editing collaboration in parallel with the resulting virtual reconstruction. The editing includes: From the main interpretation branch; and Multiple simulations are run on the branch derived from the main interpretation.

2. The computer-implemented method according to claim 1, wherein: Sunlight is the primary object; the position of the sun in the sky is the primary relative state with respect to the traffic intersection and nearby buildings. A possible precursor event is sunlight shining into the driver's eyes at an intersection; and The virtual reconstruction includes the view of the sun as seen by the driver's eyes.

3. The computer-implemented method according to claim 2, further comprising: Show the user the view of the sun as seen through the driver's eyes.

4. The computer-implemented method according to claim 3, wherein, The view of the sun is obtained through the rearview mirror of the car driven by the user.

5. The computer-implemented method according to claim 1, wherein, The virtual reconstruction of the generated physical events also includes: Simulate the lighting, weather conditions, and environmental conditions of the area where the physical event occurs; Predict the original positions of the various objects involved in the physical event prior to the occurrence of the physical event; The virtual display shows how the relative position of the object changes due to the physical event; and Historical information is collected from various information sources, including: video feeds of different types of previously recorded physical events, IoT feeds related to any physical event, computer graphics simulation results of different types of previously generated physical events, physical event reports, and live video feeds of the current physical event area.

6. The computer-implemented method according to claim 1, further comprising: The virtual reconstruction of the generated physical events is compared with different rules and constraints derived from the database, wherein the different rules and constraints derived from the database include: building codes, traffic regulations, safety certifications, engineering specifications, and product warranties; In the virtual reconstruction of the generated physical events, objects, sequences, and causal relationships are highlighted; and The objects, sequences, and causal relationships are annotated with colors or symbols to indicate deviations from constraints, specifications, certifications, descriptions, and warranties that have occurred in the physical event region.

7. The computer-implemented method according to claim 1, further comprising: Presenting a list of potential scenarios for a precursory event to the user, wherein the list of potential scenarios for a precursory event is presented to the user via the computing device; as well as The system displays different scenarios to the user, depicting how the physical event might occur, including the type of physical event and demonstrating how the object's position has moved or been damaged.

8. A computer program product for placing virtual objects in an augmented reality environment, the computer program product comprising: One or more computer-readable storage devices and program instructions stored on the one or more computer-readable storage devices, the stored program instructions including: Program instructions for identifying the relative state of an object located in a physical event region by analyzing video feeds collected from the physical event region before and after a physical event involving at least one of the objects; Program instructions for creating a knowledge corpus, the knowledge corpus including video analysis and collected video feeds associated with the physical event and historical information from previous physical events, to correlate how the relative positions of objects in the area of ​​the physical event change due to the physical event, the type of physical event associated with the subsequent event, and the duration of the physical event that produced the subsequent event. Program instructions for acquiring data of the physical event region via a computing device; Program instructions for identifying possible precursor events based on the acquired data and the knowledge corpus; Program instructions for generating a virtual reconstruction of the physical event using the possible precursor events, wherein the physical reconstruction of the physical event is a predicted physical event that is most likely to have occurred during the physical event; and Program instructions for displaying a virtual reconstruction of a predicted physical event generated by the computing device, wherein the virtual reconstruction of the predicted physical event displayed covers an image of the area of ​​the physical event; and Program instructions used for editing and collaborating in parallel with the resulting virtual reconstruction. The editing includes: From the main interpretation branch; and Multiple simulations are run on the branch derived from the main interpretation.

9. The computer program product according to claim 8, wherein: Sunlight is the primary object; the position of the sun in the sky is the primary relative state with respect to the traffic intersection and nearby buildings. A possible warning sign is sunlight shining into the driver's eyes at an intersection; as well as The virtual reconstruction includes the view of the sun as seen by the driver's eyes.

10. The computer program product according to claim 9, further comprising: The program instructions show the user the view of the sun as seen by the driver's eyes.

11. The computer program product according to claim 10, wherein, The view of the sun is obtained through the rearview mirror of the car driven by the user.

12. The computer program product according to claim 8, wherein, The virtual reconstruction of the generated physical events also includes: Program instructions for simulating lighting, weather conditions, and environmental conditions in the area of ​​the physical event when the physical event occurs; Program instructions for predicting the original positions of the various objects involved in the physical event prior to its occurrence; Program instructions for virtually displaying how the relative position of the object changes due to the physical event; and Program instructions for collecting historical information from various information sources, including: video feeds of different types of previously recorded physical events, IoT feeds related to any physical event, computer graphical simulation results of different types of previously generated physical events, physical event reports, and live video feeds of the current physical event area.

13. The computer program product according to claim 8, further comprising: Program instructions for comparing the virtual reconstruction generated by the physical event with different rules and constraints derived from the database, wherein the different rules and constraints derived from the database include: building codes, traffic regulations, safety certifications, engineering specifications, and product warranties; Program instructions used to highlight objects, sequences, and causal relationships in the virtual reconstruction of generated physical events; and Program instructions for using colors or symbols to annotate the objects, sequences, and causal relationships to indicate deviations from the constraints, specifications, certifications, descriptions, and warranties that have occurred in the physical event region.

14. The computer program product according to claim 8, further comprising: Program instructions for presenting a list of potential scenarios for a precursor event to a user, wherein the list of potential scenarios for a precursor event is presented to the user via the computing device; as well as Program instructions are used to show users different scenarios depicting how physical events might occur, including the type of physical event and demonstrating how the position of an object has moved or been damaged.

15. A computer system for placing virtual objects in an augmented reality environment, the computer system comprising: One or more computer processors; One or more computer-readable storage devices; Program instructions stored in the one or more computer-readable storage devices for execution by at least one of the one or more computer processors, the stored program instructions including: Program instructions for identifying the relative state of an object located in a physical event region by analyzing video feeds collected from the physical event region before and after a physical event involving at least one of the objects; Program instructions for creating a knowledge corpus, the knowledge corpus including video analysis and collected video feeds associated with the physical event and historical information from previous physical events, to correlate how the relative positions of objects in the area of ​​the physical event change due to the physical event, the type of physical event associated with the subsequent event, and the duration of the physical event that produced the subsequent event. Program instructions for acquiring data of the physical event region via a computing device; Program instructions for identifying possible precursor events based on the acquired data and the knowledge corpus; Program instructions for generating a virtual reconstruction of the physical event using possible precursor events, wherein the physical reconstruction of the physical event is a predicted physical event that is most likely to have occurred during the physical event; Program instructions for displaying a virtual reconstruction of a predicted physical event generated by the computing device, wherein the virtual reconstruction of the predicted physical event displayed overlays an image of the area of ​​the physical event; and Program instructions used for editing and collaborating in parallel with the resulting virtual reconstruction. The editing includes: From the main interpretation branch; and Multiple simulations are run on the branch derived from the main interpretation.

16. The computer system according to claim 15, wherein: Sunlight is the primary object; the position of the sun in the sky is the primary relative state with respect to the traffic intersection and nearby buildings. A possible warning sign is sunlight shining into the driver's eyes at an intersection; as well as The virtual reconstruction includes the view of the sun as seen by the driver's eyes.

17. The computer system of claim 16, further comprising: Program instructions for displaying a view of the sun to a user as seen by the driver's eyes, wherein the view of the sun is seen through the rearview mirror of the car driven by the user.

18. The computer system according to claim 15, wherein, The virtual reconstruction of the generated physical events also includes: Program instructions for simulating lighting, weather conditions, and environmental conditions in the area of ​​the physical event when the physical event occurs; Program instructions for predicting the original positions of the various objects involved in the physical event prior to its occurrence; Program instructions for virtually displaying how the relative position of the object changes due to the physical event; and Program instructions for collecting historical information from various information sources, including: video feeds of different types of previously recorded physical events, IoT feeds related to any physical event, computer graphical simulation results of different types of previously generated physical events, physical event reports, and live video feeds of the current physical event area.

19. The computer system of claim 15, further comprising: Program instructions for comparing a virtual reconstruction of the physical event with different rules and constraints derived from a database, wherein the different rules and constraints derived from the database include: building codes, traffic regulations, safety certifications, engineering specifications, and product warranties; Program instructions for highlighting objects, sequences, and causal relationships in a virtual reconstruction of the generated physical events; and Program instructions for using colors or symbols to annotate the objects, sequences, and causal relationships to indicate deviations from the constraints, specifications, certifications, descriptions, and warranties that have occurred in the region of physical events.

20. The computer system of claim 15, further comprising: Program instructions for presenting a list of potential scenarios for a precursor event to a user, wherein the list of potential scenarios for a precursor event is presented to the user via the computing device; as well as Program instructions are used to display to the user different scenarios depicting how physical events might occur, including the type of physical event and demonstrating how the position of an object has moved or been damaged.

Images