Earthquake detection and response via distributed visual input

Abstract

A visual input of a location is received from each of a plurality of cameras via a network. For each visual input from the plurality of cameras, a coupling correction between shakes of the cameras relative to the visual input is performed by subtracting velocity vectors of the plurality of cameras from velocity vectors of pixels defining the visual input to provide a processed input. A determination is made based on the processed input whether shakes identified in the processed input are above a predetermined threshold, thereby detecting one or more anomalies. Based on the one or more anomalies, at least one of a location, a magnitude, or a depth of an earthquake is inferred based on the shakes identified in the processed input of each of the plurality of cameras.

Description

Background Art Technical Field

[0002] The present disclosure relates generally to earthquake detection systems and methods, and more particularly, to systems and methods for measuring earthquake magnitude and intensity.

[0003] Description of Related Technology

[0004] Earthquakes (EQ) strike around the Earth, killing thousands and causing billions of dollars in damage. EQs are characterized by location (epicenter), depth, and their severity represented by magnitude (by the Richter scale) and intensity (e.g., by the modified Maxwell scale). Magnitude is the seismic energy released and is recorded by a dense network of cost-intensive seismographs. Intensity is the observed impact on the Earth's surface, ranging from perceived shaking to structural damage to buildings. Intensity is a truly interesting feature for disaster response because magnitude is independent of EQ effects. Intensity measurements are based on arbitrary grading and are assessed through questionnaires sent out long after the EQ. Real-time intensity measurements enable advance warning messages and determine evacuation and rescue missions. Experts believe that accurate prediction of earthquakes is virtually impossible with current methods and technologies. For magnitude, an earthquake warning system (EWS) provides timely detection to issue an alarm within seconds. Summary of the invention

[0005] According to various embodiments, a non-transitory computer-readable storage medium, a computer-implemented method, and a computer program product are provided that identify abnormal shaking on multiple cameras connected to a cloud interface during an earthquake. A visual input of a location is received from each of the multiple cameras over a network. For each visual input from the multiple cameras, a coupling correction between the shaking of the cameras relative to the visual input is performed by subtracting the velocity vectors of the multiple cameras from the velocity vectors of the pixels defining the visual input to provide a processed input. Based on the processed input, it is determined whether the shaking identified in the processed input is above a predetermined threshold, thereby detecting one or more anomalies. Based on the one or more anomalies, at least one of the location, magnitude, or depth of the earthquake is inferred based on the shaking identified in the processed input of each of the multiple cameras.

[0006] In one embodiment, the magnitude of the earthquake is determined based on the shaking identified in the processed input of each of the plurality of cameras.

[0007] In one embodiment, the inference of at least one of position, magnitude, or depth includes extracting and aggregating local approximations of at least one of a phase delay between a predetermined maximum pixel value applied to each pixel position in the visual input and a magnitude of the maximum value across the entirety of pixel positions in the visual input.

[0008] In one embodiment, shake is measured by one or more pixel-by-pixel changes in the processed input.

[0009] In one embodiment, for at least one visual input from a plurality of cameras, a spectral analysis of each pixel-by-pixel variation in at least one image of the visual input over time is performed. A spectral decomposition of the at least one image of the visual input is determined.

[0010] In one embodiment, an artificial intelligence (AI) model is trained to detect parameters of an earthquake based on visual input from multiple cameras. The trained AI model is applied to visual input from different groups of multiple cameras at separate locations.

[0011] In one embodiment, after the identified earthquake is completed, an indicator of at least one of the magnitude, location, or depth of the identified earthquake is stored in a predetermined memory.

[0012] In one embodiment, the calibration of the multiple cameras is refined via external seismic sensor data.

[0013] The technology described herein can be implemented in a variety of ways. Example implementations are provided below with reference to the following figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are illustrative embodiments. They do not show all embodiments. Additionally or alternatively, other embodiments may be used. Details that may be obvious or unnecessary may be omitted to save space or for more effective description. Some embodiments may be implemented with additional components or steps and / or without all components or steps shown. When the same number appears in different drawings, it refers to the same or similar components or steps.

[0015] Figure 1 The location of an earthquake that sends seismic waves into the ground and causes structural shaking is shown, consistent with an illustrative embodiment.

[0016] Figure 2 is a simplified process flow diagram of an earthquake detection and response system via distributed visual input, consistent with an illustrative embodiment.

[0017] Figure 3 A system for earthquake detection and response via distributed visual input consistent with an illustrative embodiment Figure 2 A detailed version of the process flow diagram.

[0018] Figure 4 is a block diagram of conditions prior to an earthquake and a graphical illustrative feature diagram of an earthquake detection and response system via distributed visual input consistent with an illustrative embodiment.

[0019] Figure 5 is a conceptual block diagram of monitoring a location and identifying abnormal shaking during an earthquake, consistent with an illustrative embodiment.

[0020] Figure 6 is a conceptual block diagram of correcting for pixel-by-pixel changes in an image during an earthquake, consistent with an illustrative embodiment.

[0021] Figure 7 is a conceptual block diagram of inferring the magnitude, location, and depth of an earthquake via seismic inversion, consistent with an illustrative embodiment.

[0022] Figure 8 is a conceptual block diagram of an assessment of the intensity of an earthquake during an earthquake, consistent with an illustrative embodiment.

[0023] Fig. 9 is a conceptual block diagram of providing an assessment of seismic intensity during an earthquake, consistent with an illustrative embodiment.

[0024] Fig.10 is a conceptual block diagram of a refinement of the calibration of multiple cameras via external seismic sensor data and additional assessment of seismic intensity, consistent with an illustrative embodiment.

[0025] Fig.11 is a functional block diagram illustration of a computer hardware platform that can be used to implement a specially configured computing device that can host an earthquake detection engine.

[0026] Fig.12 A cloud computing environment consistent with the illustrative embodiments is depicted.

[0027] Fig.13 Abstract model layers consistent with the illustrative embodiments are described. DETAILED DESCRIPTION

[0028] Overview

[0029] In the following detailed description, numerous specific details are set forth by way of example in order to provide a thorough understanding of the relevant teachings. However, it is apparent that the present teachings can be practiced without these details. In other instances, well-known methods, processes, components and / or circuits have been described at a relatively high level without details in order to avoid unnecessarily obscuring aspects of the present teachings.

[0030] Prior to an earthquake, multiple Internet Protocol (IP) cameras located in a particular area or location for surveillance or meteorological observation provide video streams and coordinates transmitted online via the Internet or to a private network. Clips from these cameras are often provided to the public free of charge. The teachings herein collect information from such cameras and provide meaningful insight into seismic activity at one or more locations.

[0031] In one aspect, during an earthquake, abnormal shaking of a captured scene is identified across multiple Internet Protocol (IP) cameras that provide distributed visual input of a location via a connection to a cloud or dedicated network interface, thereby defining at least a portion of a system for earthquake detection.

[0032] Various embodiments include steps of performing or instructions for performing or implementing: measuring the wobble of a location by pixel-by-pixel changes in at least one image appearing on at least one of a plurality of cameras. Correcting the coupling of pixel-by-pixel changes of natural objects or man-made objects or natural objects and man-made objects in the at least one image based on the pixel-by-pixel changes caused by the wobble of the at least one of the plurality of cameras. At least one of the magnitude, location, or depth of the earthquake may be inferred. An assessment of the intensity of the earthquake may then be provided.

[0033] In one embodiment, a spectral analysis of pixel-by-pixel changes in at least one image over time is performed to produce a spectral decomposition. The spectral analysis of pixel-by-pixel changes over time can be decoupled into a first group representing a scene in the at least one image and a second group representing at least one of the plurality of cameras according to phase delay, frequency shift, and amplitude. Note that in the present disclosure, an observer also refers to a camera.

[0034] In one embodiment, local approximations of phase delay, frequency shift, and amplitude are extracted and aggregated to infer at least one of the magnitude or intensity of an earthquake. The local approximations of phase delay, frequency shift, and amplitude are considered the local effects of an earthquake produced at the location of the camera, as calculated by a server located in a center among a group or multiple cameras. The location and depth of the earthquake can be inferred.

[0035] Inferring the characteristics of EQ(Y) from the inputs of various cameras (X) is a supervised regression problem with an unobservable function f when observing X and Y. However, since IP cameras are similar, the model on the central server is recalibrated. The central server has a model (f_estimate) that ideally relates visual input to EQ events, i.e., f_estimate(X) = Y_estimate.

[0036] After EQ has occurred (X and Y are available), Y derived from the seismic network almost perfectly infers the EQ location, etc. (summarized as Y), the model error (Y_estimate-Y) can be evaluated. The server (e.g., a seismic analysis engine running on the server) can recalibrate the model to f_estimate_new so that f_estimate_new(X) is more closely equal to Y.

[0037] After an earthquake, an indicator of at least one of the magnitude, location, or depth of the earthquake and an assessment of the earthquake intensity may be stored in an appropriate memory. In addition, the system calibration may be refined. For example, the central server may refine the learned EQ inference model via external seismic sensor data and an additional assessment of the earthquake intensity. In addition, more IP cameras may be deployed in locations where IP cameras sparsely occupy an area, such that the existing model lacks input from a pre-existing network of IP cameras.

[0038] After each EQ (and some non-EQ "events"), a record of the training data is appended, and the pre-existing AI is retrained based on the newly available data. This changes the AI's parameter values ​​("recalibrates" them). However, the model architecture, its inputs, and code remain the same. Furthermore, it can happen automatically and periodically.

[0039] As described above, before an earthquake, multiple IP cameras are located in a specific area or location for monitoring or meteorological observation. Video streams and coordinates are transmitted online via the Internet or to a dedicated network. Video streams from multiple IP cameras are utilized to facilitate various embodiments of the present disclosure.

[0040] According to the concept discussed here, seismology and intensity indicators are used, which already exist before an earthquake occurs. During an earthquake, the images provided by IP cameras can be decoupled and computer vision algorithms generated. The signals from the dedicated earthquake sensor network are processed by an artificial intelligence (AI) network that converts and improves the heuristics of the Maxwell scale for earthquake intensity measurement.

[0041] The AI ​​model can be trained or extended to new closed circuit televisions (CCTVs) in order to transfer detection capabilities from pre-existing CCTVs to new CCTVs. In one embodiment, a single camera going black may not provide enough information about earthquake detection because it may be malfunctioning, uninstalled, etc. However, multiple cameras going black (e.g., in a city) while other cameras around that location are spotting EQ events can indicate EQ occurring, which can result in power outages in the city (thus shutting down many cameras). In one embodiment, even a loss of signal (if it is tightly correlated in time and space, and other functioning cameras pick up the shake) indicates EQ.

[0042] The shake of the camera, like any object in its view, can slightly deviate from the overall frequency of earthquakes. The algorithm identifies the abnormal shake and the magnitude of the shake via a deep learning system. In one embodiment, triangulation between cameras is used to identify the amount of shake. The triangulation is implicitly learned by the AI ​​system. The AI ​​system is able to correlate inputs from different geographical locations in one system.

[0043] For example, consider mapping a sequence of IP camera images to an embedding vector via AI (e.g., via a deep neural network (DNN)). This embedding vector, which may include hundreds of floating point numbers, is a non-linear, low-dimensional, stored representation of what is going on in the video. These embedding vectors (plus latitude / longitude) are merged together as a joint (subsequent) input into the actual EQ detection.

[0044] However, since this AI system is learned "end-to-end", the embedding learning and actual EQ detection are jointly merged in one AI system. Therefore, in one embodiment, no explicit triangulation algorithm is used. The system learns on its own from the embedding vector and the latitude and longitude. When an EQ event at that approximate location is inferred, the model will automatically correlate the IP camera inputs from the same vicinity.

[0045] The earthquake network provides the ground truth of EQ characteristics (magnitude, etc.) represented by Y. As an example, assume that the video clip has a duration of 1 minute during an earthquake. The position is inferred at time t=30 seconds, and the vector value Y+ / -10 seconds connects the value, magnitude, and position together via spectral decomposition. A second video is superimposed to determine the X tilde (X~). Note that X is the endogenous model input, that is, the video input X is the original video input. X~ is the input derived by calculating the spectral decomposition along the video image. Therefore, this is just adding features to allow AI to identify EQ more accurately and faster. Therefore, as used in this article, the variable X is understood to be the entire input: the original plus the derived features, that is, X:=(X, X~).

[0046] Inferring the properties of EQ(Y) from the inputs (X) from the various cameras is a supervised regression problem with an unobservable function f when observing X and Y. However, since the IP cameras are identical, the model on the central server is recalibrated. The central server has a model (f_estimate) that ideally relates the visual input to the EQ event, i.e., f_estimate(X) = Y_estimate. After the EQ has occurred (X and Y are available), Y derived from the seismic network almost perfectly infers the EQ location, etc. (generalized as Y), and the model error (Y_estimate-Y) can be evaluated. The server can then recalibrate the model to f_estimate_new so that f_estimate_new(X) is more closely equal to Y. The shake in the image of one camera can simply be a noisy estimator of the intensity / location of the EQ.

[0047] While seismic sensors are well calibrated, they are expensive, sparsely located, and require additional knowledge (e.g., about the surrounding geological properties). In this regard, IP cameras are not only noisy estimators of seismic activity, but are also highly local, as knowledge outside the view frame is now provided. Furthermore, IP cameras are very common. The more cameras that are utilized, the more inputs are available to the AI, which automatically learns to infer EQ characteristics from distributed visual inputs.

[0048] As used herein, spectral synthesis refers to a fixed transformation of a simplified spectral excitation applied to a camera image to enhance the input signal. No explicit physical formulas are embedded. In one embodiment, the AI ​​model does not perform physical manipulations such as seismic inversion.

[0049] In one embodiment, the AI ​​model treats EQ detection as an end-to-end regression problem with (e.g., enhanced) video input (X) and EQ features (Y). Coupled correction / anomaly detection as defined herein refers to the inner workings of modern AI, components of which are not explicitly defined / modularized, where "X" is an endogenous model input, i.e., video input, and "Y" is an exogenous output (given as ground truth), i.e., characteristics of an earthquake (latitude, longitude, depth, magnitude, and intensity). In short, in one embodiment, Y is a 5-dimensional vector.

[0050] In one embodiment, Y is estimated by a network of seismic sensors, which may take several minutes. When EQ begins, Y is not available because the seismic sensors are still performing seismic inversion to obtain Y. However, the AI ​​seismic analysis engine can calculate a guess from the video input what Y (a characteristic of EQ) might be, and provide an alert if a predetermined (magnitude) threshold is believed to be exceeded.

[0051] In one embodiment, the seismic analysis engine applies deep learning via a convolutional neural network to filter out the shake, thereby enabling Y to calibrate itself. The camera has a specific shake pattern. Although a convolutional neural network is used by way of example, the teachings herein are not limited thereto.

[0052] Self-calibration refers only to the model retraining step as the ground truth Y of EQ is used after hours / days and compared with the Y_estimate from the EQ model, i.e., it refers to applying stochastic gradient descent (SGD) on the data with one new observation (i.e., the last observed X and Y of EQ).

[0053] Figure 1 A location 10 of a seismic EQ is shown which transmits seismic waves SW1 ... SWn towards a ground surface 15 where a first structure 21 and a second structure 22 are located.

[0054] The IP camera 20 is mounted on the second structure 22 at a height H above the ground 20. During an earthquake (EQ), the IP camera 20 vibrates vertically in the direction of the double arrow AA, while the first structure 21 vibrates vertically in the direction of the double arrow A'-A', thereby producing a distorted image captured by the IP camera 20.

[0055] As a very simplified illustration of the application of coupling correction and seismic analysis engine, the distorted image transmitted by IP camera 20 is coupled corrected via an algorithm applied by the seismic analysis engine, which will be discussed in more detail later. IP camera 20 has a field of view 25 including a first structure 21. During seismic EQ, first structure 21 has a velocity vector VS in the direction of arrow A'-A', while IP camera 20 has a velocity vector VC in the direction of arrow AA. Therefore, coupling correction is applied by subtracting the camera velocity vector VC from the structure velocity vector VS, where a reference plane such as the ground 15 at location 10 is established. Due to the difference in the forces applied by the seismic EQ, each pixel in the image captured by IP camera 20 will generally have its own unique velocity vector. Differences in angular velocity and horizontal and tilt oscillations will also affect the image and coupling correction requirements. This consideration can be addressed by deep learning and convolutional neural networks or alternative techniques with additional spectral decomposition applications. Spectral decomposition is a known technique in the field of seismic analysis, especially for oil field drilling. Therefore, as those skilled in the art will understand the process based on the present disclosure, the process is not graphically illustrated in more detail in the accompanying drawings.

[0056] Figure 2 is a simplified process flow diagram of an earthquake detection and response system 100 via distributed visual input according to an embodiment of the present disclosure.

[0057] Figure 3 20a ... 20n is a process flow diagram of an earthquake detection and response system via distributed visual input consistent with an illustrative embodiment. A plurality of IP cameras 20a ... 20n are pre-placed before an earthquake occurs and before an embodiment of the present invention is installed to monitor the location where the earthquake occurs. However, the plurality of IP cameras 20a ... 20n may be connected to an earthquake analysis engine / server 110 (see Figure 2 ) is located simultaneously or after the installation of the earthquake analysis engine / server dedicated to viewing the location for monitoring earthquake occurrences. In fact, this timing of the location of the earthquake analysis engine / server 110 relative to the interconnection with external software and hardware is applicable to various functions, such as earthquake warning systems, disaster response management and Figure 2 and Figure 3 The seismic network shown in .

[0058] A plurality of IP cameras 20a...20n communicate with the Internet, the cloud 102, and additionally or alternatively with a private network 104. The IP cameras 20a...20n transmit image data 40a...40n to the Internet cloud 102 or to the private network 104. The Internet cloud 102 or the private network 104 transmits the corresponding positions and video streams 40a'...40n' to the seismic analysis engine 110. In various embodiments, the seismic analysis engine 110 may be a software package running on a dedicated server and / or cloud. The seismic analysis engine 110 performs coupled correction of the positions and video streams 40a'...40n' to produce processed video streams 42a...42n of the corresponding positions, and converts the processed video streams 42a...42n of the corresponding positions into corresponding digital indicators or metrics 44a...44n for each corresponding camera and position video stream.

[0059] Coupling correction 44 refers to the ability of artificial intelligence (AI) algorithms (most notably, but not exclusively, involving convolutional neural networks) to filter out various types of image perturbations (e.g., shake, changed perspective, etc.) for image recognition tasks (e.g., object recognition, segmentation, etc.). In the supervised learning problem of inferring seismic properties from distributed visual inputs, for example, convolutions are adjusted during the model training process to effectively filter out such perturbations, learning a lower dimensional embedding vector that is approximately invariant to image perturbations such as camera shake. Since Internet Protocol (IP) cameras are (usually firmly) attached to the exterior or interior of buildings, the amplitude of their shake is fairly constant in time during a seismic event, while (unattached) objects in the image of various weights, robustness, and solidity shake to varying degrees, various AI algorithms find it easy to automatically decouple the shake of the camera itself from the effects of the earthquake on the objects in the image. However, it should be noted that this coupling correction is not exact, i.e., not every object in the image is fully trackable in its movement. However, this is not in any way a goal or function of the system, since it receives N camera inputs to infer only a few major seismic features. For this task, (approximate) coupling corrections are sufficient to reliably infer seismic events from distributed visual input.

[0060] As mentioned above Figure 1 As shown, a coupling correction 44 is applied to correct the coupling of the multiple cameras 20, 20a...20n and the scene being imaged by the multiple cameras to more accurately assess seismic energy. The coupling correction 44 is applied by subtracting the velocity vector VC of the camera from the velocity vector of the pixel defining the visual input. Applications such as convolutional neural networks and spectral decomposition are applied to perform the coupling correction. Spectral decomposition involves spectral analysis of the changes in the pixels of a given video stream over time. Decoupling of the spectrum of the pixels is performed in terms of phase delay, frequency shift and amplitude into two separate groups: the scene being imaged by the observer (i.e., the camera) and the camera itself.

[0061] Each reference to a spectrum here refers only to a set of proposed fixed transformations of an input video (image) to enhance the visual input of an AI subsequently ingested therewith, and therefore represents pixel-by-pixel changes as applied in this disclosure.

[0062] The processed video streams 42a...42n and metrics 44a...44n are then subjected to anomaly detection 46 by the seismic analysis engine 110. If one or more anomalies are detected, the seismic analysis engine 110 performs the functions of earthquake detection 120 and data storage 130, and the seismic analysis engine 110 internally enters an earthquake occurrence mode of operation. The earthquake detection system 120 then performs seismic inversion 122 on the magnitude, location and depth, and intensity measurements 124 via the seismic analysis engine 110 to assess the severity of the earthquake EQ and the intensity of the damage caused by the earthquake EQ.

[0063] It is important to recognize that the earthquake detection and response system 100 as described herein includes features that are actually subsystems whose functions are performed by the earthquake analysis engine 110. To this end, as an example, the earthquake detection system 120 is referred to as a system, but in reality, can be considered a subsystem with respect to the entire earthquake detection and response system 100. Other functions, such as seismic inversion 122 and intensity measurement 124 can also be considered subsystems of the earthquake detection and response system 100.

[0064] The seismic inversion 122 and intensity measurement 124 analysis results are forwarded to the earthquake early warning or response system 140, the response management system 150 and the existing seismic network 160. The video stream 126 and the earthquake indication value 128 are also forwarded from the data storage 130 to the earthquake early warning or response system 140, the response management system 150 and the existing seismic network 160.

[0065] After an earthquake, the existing seismic network 160 forwards the seismic analysis 48 to the seismic analysis engine 110 for a recalibration process 170, which in turn forwards the recalibrated analysis 50 to the seismic detection system 120 and anomaly detection 46 to correct for impending aftershocks and future earthquake occurrences.

[0066] Example Architecture

[0067] Figure 4-10 is a block diagram of a combined method of conditions before, during, and after an earthquake of an earthquake detection and response system 100 via distributed visual input consistent with an illustrative embodiment. More specifically, Figure 41 is a combined method block diagram of the conditions before an earthquake before actuating the earthquake detection and response system 100 via distributed visual input according to an embodiment of the present disclosure. Therein, before an earthquake EQ, a plurality of IP cameras 20a...20n are located in a particular area or location L for monitoring or meteorological observation and transmit image data 40a...40n to an Internet cloud 102 or to a private network 104. The video stream and the coordinates 42a...42n of the camera location are transmitted online via the Internet 102 or through a private network, usually without direct cost to the public. Various embodiments of the present disclosure are facilitated by video streams 42a...42n from the locations of a plurality of IP cameras 20a...20n. The locations of various pre-existing cameras are close to the possible span of the earthquake detection system 100 via distributed visual input. For practical purposes, when the location of an earthquake EQ appears within such an IP camera network (not far outside the network of IP cameras 20a...20n), the earthquake EQ can be inferred. The denser the area is populated with IP cameras, the more reliable the inferred characteristics of the earthquake event will be. Regardless, there is no required threshold for the number of cameras or camera density in an area to facilitate useful earthquake detection via distributed input. Thus, a pre-trained EQ detection system (from one area on Earth) could be used to infer earthquake events from a single IP camera on the other side of the world - although this is far from ideal, as earthquake properties will only be roughly estimated.

[0068] Furthermore, there is no fixed spatial resolution involved. An IP camera system can span a city, a state / province, an entire country or the globe. Figure 4 and Figure 5 This constitutes a subset of the IP cameras that have detected an earthquake event EQ by the earthquake detection system 120. The images shown show damaged areas in the same vicinity (although input from each IP is taken into account by the EQ detection system 120).

[0069] With Figure 4 In a similar way, Figure 5 The occurrence of an earthquake EQ is shown where the earthquake analysis engine 110 identifies abnormal shaking of multiple cameras 20a...20n providing distributed visual input of location L. Video streams 42a0...42n0 represent visual input at location L at time t0, while video streams 42a1...42n1 represent visual input at location L at time t1.

[0070] Figure 644a ... 44n, which is transmitted to the anomaly detection 46 as images 46a ... 46n.

[0071] Reference now Figure 7 , which is a conceptual block diagram of inferring the magnitude, location, and depth of an earthquake EQ via seismic inversion consistent with an illustrative embodiment. In one embodiment, with respect to anomaly detection 46, coupling correction 44 performed by the seismic analysis engine 110 may also include decoupling spectral analysis of pixel-by-pixel changes over time in terms of phase delay, frequency shift, and amplitude into a first group representing a scene in at least one image 40a...40n and a second group representing at least one of the plurality of cameras 20a...20n.

[0072] The decoupling of spectral analysis over time refers to transfer learning, i.e., after having a set of IP cameras 20a...20n with a learned AI and applying the learned AI to a new set of IP cameras (i.e., in a completely different area of ​​the world). The model from the first area (with a first "set" of cameras) can be used in the new area (which has a second "set" of IP cameras). However, the model is retrained (i.e., after an EQ event occurs).

[0073] The phase delay defined here (in seconds) is the time delay between the maximum pixel values ​​(applied to each pixel position in the predefined video). The amplitude defined here is the maximum value of the pixel positions across the entirety of the input video.

[0074] In one embodiment, with respect to anomaly detection 46, the seismic analysis engine 110 may also include extracting and aggregating local approximations of phase delays, frequency shifts, and amplitudes to infer at least the magnitude 122 and intensity 124 of the seismic EQ. Further with respect to anomaly detection 46, the seismic analysis engine 110 may also include inferring the location and depth 122 of the seismic EQ. Anomaly detection 46, just like decoupling correction 44, is implicitly learned by the model. In the same way that the distributed visual inputs X and X~ are used to learn the seismic characteristic Y in the case of an earthquake event, the model will infer the absence of an earthquake by always estimating Y (magnitude 0, depth 0, arbitrary latitude, longitude) through the visual inputs X and X~. Therefore, anomaly detection 46 refers only to the presence of abnormal values ​​of the corresponding features. In contrast to seismic sensors, there is no physical derivation of the features involved (velocity vectors, etc.) because the visual inputs from the IP cameras 20a...20n are learned end-to-end in conjunction with their locations to infer the presence / absence of an earthquake.

[0075] Fig. 9 1 is a conceptual block diagram of an assessment of seismic intensity following an earthquake (EQ) consistent with an illustrative embodiment. For example, following an earthquake EQ, the seismic analysis engine 110 stores an indicator 122 as a seismic indicator value 128 of at least one of the magnitude, location, or depth of the earthquake EQ and the seismic intensity assessment 124 and a key video stream 126 (see Figure 2 and Figure 3 ).

[0076] Fig.10 is a calibration of multiple cameras 20a...20n via external seismic sensor data and additional evaluation of seismic intensity consistent with the illustrative embodiment (see Figure 2 and 3 1 and 1 . A conceptual block diagram of a refinement of the recalibration process 170 and recalibration analysis 50). For example, after an earthquake, the earthquake analysis engine 110 maintains the AI ​​model architecture and visual input in a fixed state. However, the earthquake analysis engine 110 then causes the AI ​​model parameters to be re-evaluated through a retraining step, namely the recalibration process 170. There, the previous history of visual input and earthquake event characteristics is now added with new visual input and earthquake characteristic pairs. This AI model retraining step can be performed periodically and automatically triggered with a fixed periodicity (e.g., daily, weekly, monthly). The historical records include not only earthquake events, but also non-earthquake events (with a magnitude value of 0, an intensity value of 0, and arbitrary latitude / longitude), so that the earthquake detection system 120 can distinguish between earthquakes and non-earthquake events for various types of distributed visual inputs.

[0077] Example Computer Platforms

[0078] As described above, during an earthquake, functions related to identifying abnormal shaking on multiple Internet Protocol (IP) cameras can be performed using one or more computing devices connected via wireless or wired communications for data communication, wherein the multiple IP cameras are connected via an interface with a cloud or a private network to provide distributed visual input of the location, such as Figure 1-10 shown. Fig.11 It is a functional block diagram illustration of a computer hardware platform that can communicate with various networked components such as networked IP cameras, the cloud, etc. Specifically, Fig.11 A network or host computer platform 1100 is shown, which can be used to implement such Figure 2 A server of the earthquake analysis engine 110.

[0079] The computer platform 1100 may include a central processing unit (CPU) 1104 , a hard disk drive (HDD) 1106 , random access memory (RAM) and / or read-only memory (ROM) 1108 , a keyboard 1110 , a mouse 1112 , a display 1114 , and a communication interface 1116 , which are connected to the system bus 1102 .

[0080] In one embodiment, the HDD 1106 has the capability to include stored programs that can perform various processes such as the seismic analysis engine 1140 in the manner described herein. The seismic analysis engine 1140 can have various modules configured to perform different functions. For example, there can be a coupling correction module 1142 that is applied to subtract the velocity vector of the camera from the velocity vector of the pixels defining the visual input of the image displayed by the camera. The coupling correction can be performed via a convolutional neural network and spectral decomposition performed by the coupling correction module 1142.

[0081] There may be a video processing and indicator module 1144 that processes the video stream that has been coupling corrected and converts the coupling corrected video stream into indicators of the motion that occurred at the location and time of each individual image in the video stream.

[0082] There may be an anomaly detection module 1146 that analyzes the processed video stream and indicators to determine the occurrence of abnormal movement of one or more structures that have been imaged by the camera. The module 1146 then identifies such abnormal movement as an anomaly.

[0083] There may be a seismic detection module 1148 that processes one or more anomalies received from the anomaly detection module and performs seismic inversion and intensity assessment.

[0084] There may be an earthquake inversion and intensity assessment module 1150 that calculates the magnitude, location and depth of an earthquake and performs an intensity assessment. This information is then transmitted to an earthquake early warning system and used for disaster response management.

[0085] There may be a data storage module 1152 that stores anomalies identified by and received from the anomaly detection module 1146 .

[0086] There may be a critical video stream and earthquake indicator value module 1154, which receives higher priority video streams and earthquake indicator values ​​from the data storage module 1152, and then transmits the video streams and indicator values ​​to the earthquake early warning system and is used for disaster response management.

[0087] There may be an existing seismic network module 1156 that receives data from the seismic inversion and intensity assessment module 1150 and from the key video stream and seismic indicator module 1154, which may forward the data for recalibration after an earthquake.

[0088] There may be a recalibration module 1158 that recalibrates the data from the seismic inversion and intensity assessment module 1150 and from the key video stream and seismic indicator value module 1154, and which then forwards the results to the anomaly detection module 1146 and the earthquake detection module 1148 to improve the accuracy of upcoming aftershocks and future earthquakes.

[0089] Sample cloud platform

[0090] As described above, functions related to earthquake detection and response via distributed visual input may include cloud 102 or network 104 (see Figure 2 ). It should be understood that although the present disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, the embodiments of the present disclosure can be implemented in conjunction with any other type of computing environment now known or later developed.

[0091] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0092] Features are as follows:

[0093] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed without requiring manual interaction with the service provider.

[0094] Broad Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0095] Resource pooling: The computing resources of a provider are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence, as consumers generally do not control or know the exact location of the provided resources, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0096] Rapid elasticity: In some cases, the ability to scale out quickly and in quickly can be provisioned quickly and elastically. To the consumer, the capacity available for provisioning often appears to be unlimited and can be purchased in any quantity at any time.

[0097] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the utilized services.

[0098] The service model is as follows:

[0099] Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications are accessible from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0100] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications created or acquired by consumers using programming languages ​​and tools supported by the provider onto the cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but have control over deployed applications and possible configuration of the application hosting environment.

[0101] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0102] The deployment model is as follows:

[0103] Private Cloud: The cloud infrastructure is operated only for the organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0104] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0105] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0106] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community or public) that remain a unique entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0107] The cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure consisting of a network of interconnected nodes.

[0108] Reference now Fig.12 , depicts an illustrative cloud computing environment 1200. As shown, the cloud computing environment 1200 includes one or more cloud computing nodes 1210 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 1254A, a desktop computer 1254B, a laptop computer 1254C, and / or an automobile computer system 1254N. The nodes 1210 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 1250 to provide infrastructure, platform, and / or software as a service for which the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Fig.12 The types of computing devices 1254A-N shown in are intended to be illustrative only, and computing nodes 1210 and cloud computing environment 1250 may communicate with any type of computerized device over any type of network and / or network-addressable connection (eg, using a web browser).

[0109] Reference now Fig.13 , showing the cloud computing environment 1250 ( Fig.12 ) provides a set of functional abstraction layers. It should be understood in advance that Fig.13The components, layers, and functions shown in are intended to be illustrative only, and the embodiments of the present disclosure are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0110] The hardware and software layer 1360 includes hardware and software components. Examples of hardware components include: host 1361; server 1362 based on RISC (Reduced Instruction Set Computer) architecture; server 1363; blade server 1364; storage device 1365; and network and networking components 1366. In some embodiments, software components include network application server software 1367 and database software 1368.

[0111] Virtualization layer 1370 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 1371; virtual storage 1372; virtual networks 1373, including virtual private networks; virtual applications and operating systems 1374; and virtual clients 1375.

[0112] In one example, the management layer 1380 may provide the functionality described below. Resource provisioning 1381 provides dynamic procurement of computing resources and other resources used to perform tasks within a cloud computing environment. Metering and pricing 1382 provides cost tracking when resources are utilized in a cloud computing environment, as well as bills or invoices for consuming these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 1383 provides access to the cloud computing environment for consumers and system administrators. Service level management 1384 provides cloud computing resource allocation and management so that the required service levels are met. Service level agreement (SLA) planning and fulfillment 1385 provides pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0113] The workload layer 1390 provides examples of functions that the cloud computing environment can be used for. Examples of workloads and functions that can be provided from this layer include: mapping and navigation 1391; software development and lifecycle management 1392; virtual classroom education delivery 1393; data analysis processing 1394; transaction processing 1395; and seismic analysis engine 1396, as discussed herein. SUMMARY OF THE INVENTION

[0115] The description of various embodiments of the present teachings has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements over existing technologies on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0116] Although the above has described what is considered to be the best state and / or other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied to many applications, only some of which are described herein. The appended claims are intended to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.

[0117] The components, steps, features, objects, benefits, and advantages discussed herein are merely illustrative. None of them, or the discussion related to them, is intended to limit the scope of protection. Although various advantages have been discussed herein, it will be understood that not all embodiments necessarily include all advantages. Unless otherwise indicated, all measurements, values, ratings, positions, sizes, dimensions, and other specifications in this specification, including those set forth in the appended claims, are approximate and not exact. They are intended to have a reasonable range consistent with the functions to which they are related and the customary practices of the field to which they belong.

[0118] Many other embodiments are also contemplated. These include embodiments with fewer, additional and / or different components, steps, features, objects, benefits and advantages. These also include embodiments in which components and / or steps are arranged and / or sequenced in different ways.

[0119] Aspects of the present disclosure are described herein with reference to call flow diagrams and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each step of the flow diagram and / or block diagram and the combination of blocks in the call flow diagram and / or block diagram can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processor of a computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create components for implementing the functions / actions specified in one or more boxes of the call flow processing and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the call flow and / or block diagram.

[0121] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operating steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, whereby the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the call flow process and / or block diagram.

[0122] The flow chart and block diagram in the accompanying drawings show the architecture, function and operation of the possible implementation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each frame in the call flow process or block diagram can represent a module, segment or part of an instruction, and the instruction includes one or more executable instructions for realizing the specified logical function. In some alternative embodiments, the function noted in the frame may not occur in the order noted in the figure. For example, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved. It will also be noted that each frame of the block diagram and / or the call flow diagram and the combination of the frames in the block diagram and / or the call flow diagram can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

[0123] Although the foregoing has been described in conjunction with exemplary embodiments, it should be understood that the term "exemplary" is meant only as an example, rather than the best or optimal. Except as immediately stated above, nothing stated or illustrated is intended or should be construed to cause any component, step, feature, object, benefit, advantage, or equivalent to be dedicated to the public, whether or not stated in the claims.

[0124] It should be understood that, unless a specific meaning is elaborated in addition herein, the terms and expressions used herein have the common meaning consistent with these terms and expressions about their corresponding respective investigation and research fields. Relational terms such as the first and second etc. can be used only to distinguish one entity or action from another, without requiring or implying any actual such relationship or order between these entities or actions. The term "comprise", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only include those elements, but also may include other elements that are not clearly listed or are inherent to such process, method, article or device. In the absence of further constraints, the element with "one" or "an" in front does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0125] The abstract of the present disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. It should be understood that it is not used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, it can be seen that in order to make the present disclosure smooth, various features are grouped together in various embodiments. This method of disclosure should not be interpreted as reflecting the intention that the claimed embodiments have more features than the features explicitly recited in each claim. On the contrary, as reflected in the attached claims, the subject matter of the invention lies in less than all the features of a single disclosed embodiment. Therefore, the following claims are hereby incorporated into the specific implementation mode, wherein each claim is independently a subject matter claimed separately.

Claims

1. A computer-implemented method comprising: receiving, via a network, visual input of a position from each of a plurality of cameras; For each visual input from the multiple cameras: performing coupling correction between camera shake relative to the visual input by subtracting velocity vectors of the plurality of cameras from velocity vectors of pixels defining the visual input to provide a processed input; as well as determining, based on the processed input, whether shake identified in the processed input is above a predetermined threshold, thereby detecting one or more anomalies; as well as Based on the shaking identified in the processed input of each of the plurality of cameras, at least one of the location, magnitude, or depth of an earthquake is inferred from the one or more anomalies, wherein the inference of the at least one of the location, magnitude, or depth includes extracting and aggregating a local approximation of at least one of the phase delay between a predetermined maximum pixel value applied to each pixel position in the visual input and the amplitude of the maximum value across the pixel positions of the entirety of the visual input.

2. The computer-implemented method of claim 1, further comprising determining a magnitude of the earthquake based on the shaking identified in the processed input of each of the plurality of cameras.

3. The computer-implemented method of claim 1 , wherein: The shake is measured by one or more pixel-by-pixel changes in the processed input.

4. The computer-implemented method of claim 3, further comprising: For at least one visual input from the plurality of cameras: performing a spectral analysis of each pixel-by-pixel variation in at least one image of the visual input over time; as well as A spectral decomposition of the at least one image of the visual input is determined.

5. The computer-implemented method of claim 4, further comprising: training an artificial intelligence model based on the visual input from the plurality of cameras to detect parameters of an earthquake; as well as The trained artificial intelligence model is applied to visual input from multiple cameras in different groups at separate locations.

6. The computer-implemented method of claim 1 , further comprising: After the identified earthquake is completed, an indicator of at least one of the magnitude, location, or depth of the identified earthquake is stored.

7. The computer-implemented method of claim 1 , further comprising: Calibration of the plurality of cameras is refined via external seismic sensor data.

8. A computer program product comprising instructions executable by a processor to cause the processor to perform the method according to any one of claims 1 to 7.

9. A computing device comprising: processor; a storage device coupled to the processor; A seismic engine, wherein execution of the seismic engine on the processor causes the computing device to perform the method of any one of claims 1 to 7.

Images