Road incident detection and reconstruction

Abstract

A method and system are provided for detecting and reconstructing incidents using road information data collected by multiple source modalities. The method includes the step of processing road information data collected by a given source modality (SM) individually for each of the multiple source modalities to obtain one or more feature-level data modalities associated with the given SM, thereby yielding multiple obtained data modalities, each given data modality having information about one or more features extracted from the road information data collected by the associated SM during processing. If at least one of the multiple data modalities indicates a potential incident, the multiple data modalities are fused into one or more machine learning models (MLMs) to confirm the detection of the incident, and in response to this confirmation of incident detection, incident reconstruction and / or other incident-related processing are provided.

Description

[Technical Field]

[0001] The subject matter of this disclosure relates to traffic management technologies, and more specifically, to methods and systems for the automated detection and reconstruction of road incidents.

[0002] Cross-references to related applications This application claims the benefits of U.S. Provisional Application No. 63 / 491,217, filed on March 20, 2023, which is incorporated herein by reference in its entirety. [Background technology]

[0003] Automated detection and reconstruction of road incidents is a crucial element of modern traffic management systems. By rapidly detecting incidents, traffic management systems can reroute traffic and implement alternative strategies, thereby minimizing congestion and maintaining smoother traffic flow. Automated incident reconstruction helps investigators and authorities understand the sequence of events, determine fault, and improve the accuracy of post-incident analysis.

[0004] Similarly, automated detection and reconstruction can help optimize the allocation of resources for emergency services and law enforcement agencies. Furthermore, analyzing the collected incident data can identify patterns and potential causes, which can be used to implement preventative measures and improve infrastructure to reduce the likelihood of future incidents.

[0005] Thus, automated incident detection and restructuring contribute to improved safety, faster response, more efficient resource allocation, and enhanced post-incident analysis and prevention strategies.

[0006] As used herein, the terms “road incident” or “incident” refer to events that affect traffic, disrupt normal traffic flow, or pose a potential hazard. In this specification, the terms “road accident” or “accident” refer to harmful incidents, including collisions between vehicles or between vehicles and other objects. Accidents range from minor collisions to more serious events resulting in injury or death. Accidents can cause significant loss of life, property damage, and traffic disruption, and are of great concern to city authorities, traffic planners, insurance companies, and other stakeholders.

[0007] The problems of automated incident detection and remediation are recognized in conventional technologies, and various technologies have been developed to provide solutions. For example, the following:

[0008] U.S. Patent No. 9,773,281 discloses accident detection and recovery technology. One or more devices within an accident detection and recovery computing system may be configured to determine that a vehicle accident has occurred, collect and analyze accident characteristics and other relevant data, and provide customized accident recovery services. Mobile computing devices can detect accidents and receive accident-indicating data, either independently or in conjunction with vehicle-based systems and external devices. After determining that an accident has occurred, the mobile computing devices and / or vehicle-based systems may be configured to identify accident characteristics, retrieve vehicle and vehicle occupant data from one or more external servers, identify damage or potential damage resulting from the accident, and determine one or more accident recovery options or recommendations based on the damage caused by the accident. Various user interface screens can be generated and displayed via the user's mobile device and / or vehicle-based display device, thereby providing the user with accident information, damage, and recovery options or recommendations.

[0009] U.S. Patent No. 11,068,995 discloses a technique for reconstructing an accident scene using telematics data. The method comprises the steps of: receiving data from a vehicle in which a user is riding, indicating that the vehicle has been involved in an accident; sending a communication to the user's mobile device in response to the receipt of the data; displaying the communication via a mobile device application installed on the mobile device, wherein the communication prompts the user to answer one or more questions regarding the accident; determining the severity of the accident based on the received data, the determination of whether or not one or more injuries were likely to have occurred during the accident; receiving from the user the answers to one or more questions; prompting the user for emergency assistance recommendations based on the determination of the severity of the accident and the received answers to one or more questions; and performing one or more assessments based on the determination of the severity of the accident, the performance of one or more assessments including at least one of determining vehicle damage, determining necessary repairs to the vehicle, or determining user negligence in the accident.

[0010] U.S. Patent No. 11,620,862 discloses a technology for reconstructing information related to a vehicle accident. The system comprises an onboard computing system installed in a vehicle equipped with an accident reconstruction system. This system can generate an animated video depicting the accident using detection information from vehicle sensors and an animation engine. Furthermore, the system can take action based on information learned from the animated video and / or the underlying 3D model of the accident used to generate the video.

[0011] U.S. Patent No. 11,682,289 discloses a technology for integrated traffic incident detection and response. The method includes the step of an electronic device receiving motion data indicating the vehicle's operating characteristics from the electronic device's sensors, vehicle sensors, and / or images / videos captured by cameras. Based on an analysis of the motion data, the electronic device determines whether a potential incident has occurred to the vehicle and whether the potential incident has actually occurred. The electronic device also receives vehicle-related risk management data from a database and determines the severity level of the potential incident based on the motion data and risk management data. The electronic device then requests assistance by sending a notification indicating the potential incident to a third-party remote system (e.g., towing service, emergency service, or both) based on the likelihood that the potential incident has actually occurred and the severity level of the potential incident.

[0012] U.S. Patent Application Publication No. 2023 / 0074620 discloses an automated incident detection technology for vehicles. The computer implementation method comprises the steps of: receiving first data from vehicle sensors; a processing device processing the first data using a machine learning model to determine whether or not an incident outside the vehicle has occurred; if an incident outside the vehicle has been determined to have occurred, starting to record second data from the sensors; and if an incident outside the vehicle has been determined to have occurred, taking measures to control the vehicle.

[0013] U.S. Patent Application Publication 2023 / 0367006 discloses an incident detection technique using radar sensor data. The vehicle tracking system comprises at least one on-board sensor, such as a radar sensor, which can perceive the environment around the vehicle and acquire data about incidents observable by the sensor. The radar sensor can provide radar data that can be used to calculate the velocity vector, acceleration vector, azimuth angle, and elevation angle of other vehicles, and this data can be collected and stored for characterizing potential incidents and for accident investigation. A vehicle equipped with sensors can be configured to act as a third-party witness to potential incidents. Multiple sensors can be used in combination with each other, and one sensor can trigger another to initiate the acquisition of other data that cannot be acquired by one sensor.

[0014] The above references disclose background information applicable to the subject matter of this disclosure. Accordingly, all content of those documents is incorporated herein by reference as appropriate for additional or alternative details, features and / or technical background. [Overview of the Initiative]

[0015] In certain aspects of the subject matter of this disclosure, a computerized method is provided for detecting incidents using road information data collected by multiple source modalities. The method includes the step of processing road information data collected by a given SM individually for each of the multiple source modalities to obtain one or more feature-level data modalities associated with the given SM, thereby yielding multiple obtained data modalities, each given data modality having information about one or more features extracted from the road information data collected by the associated SM during processing. If at least one of the multiple data modalities indicates a potential incident, the multiple data modalities are fused into one or more machine learning models (MLMs) to confirm the detection of the incident, and in response to this confirmation of the detection of the incident, one or more incident-related processing is provided.

[0016] In a further embodiment, optionally in combination with other embodiments of the subject matter of this disclosure, a plurality of SMs may include a combination of at least one sensor configured to acquire road information data, a control unit configured to collect data from one or more road infrastructure elements, a V2X unit configured to receive vehicle motion data and / or safety data, and a cloud-based information module configured to collect road-related behavioral and aggregated data.

[0017] In a further aspect, optionally, in combination with other aspects of the subject matter of this disclosure, processing of road information data collected by a given SM is performed to obtain data modalities indicating a potential incident. Optionally, one or more incident detection models used during processing may depend on the SM that collected the respective road information data. Processing the road information data may include applying one or more anomaly detection models configured to detect a potential incident based on identifying anomaly data patterns that may be caused by an incident.

[0018] In a further embodiment, optionally in combination with other embodiments of the subject matter of the present disclosure, at least one data modality may provide information about a predefined set of features corresponding to the relevant SM and type of incident impact, the type of incident impact being selected from a group including direct impact, short-range indirect impact, medium-range indirect impact and long-range indirect impact.

[0019] In a further embodiment, optionally, in combination with other embodiments of the subject matter of this disclosure, the acquired data modalities may be fused into one or more MLMs using assigned fusion weights. Optionally, the fusion weights of a given data modality may depend on one or more techniques applied to obtain the respective associated SMs and / or given data modalities.

[0020] In a further embodiment, optionally in combination with other embodiments of the subject matter of the present disclosure, the method may further include the steps of: processing road information data collected by a given SM for each given SM of at least a subset of a plurality of SMs to obtain a first data modality and a second data modality associated therewith, thereby yielding a plurality of first data modalities and a plurality of second data modalities; fusing the plurality of first data modalities into a first MLM to detect potential incidents at a first confidence level; and further fusing the output of the first MLM and the plurality of second data modalities into a second MLM to detect potential incidents at a higher confidence level. Optionally, the first data modality may have information about a predefined set of features corresponding to the impact of one or more direct incidents, and the second data modality may have information about a predefined set of features corresponding to the impact of one or more short-range indirect incidents.

[0021] The method further includes the steps of processing road information data collected by a given SM for each SM among at least a portion of a plurality of SMs to obtain a third data modality and a fourth data modality associated therewith, thereby yielding a plurality of third data modalities and a plurality of fourth data modalities, wherein the third data modality has information relating to a pre-defined set of features corresponding to the impact of one or more medium-range indirect incidents, and the fourth data modality has information relating to a pre-defined set of features corresponding to the impact of one or more long-range indirect incidents; merging the output of a second MLM and the plurality of third data modalities into a third MLM to detect potential incidents with an even higher level of confidence; and merging the output of the third MLM and the plurality of fourth data modalities into a fourth MLM to confirm the detection of incidents.

[0022] According to a further aspect, optionally, in combination with other aspects of the subject matter of the present disclosure, incident-related processing may include incident reconstruction and / or initiation of respective alerts and / or reports.

[0023] According to a further aspect, optionally, in combination with other aspects of the subject matter of the present disclosure, incident reconstruction includes collecting incident information data corresponding to a time frame before and after the time when the incident occurred from a plurality of SMs, processing the collected incident information data to generate an incident reconstruction model, using the generated incident reconstruction model to generate a visual representation of the incident, expanding the incident representation, and rendering the reconstructed incident.

[0024] According to a further aspect, optionally, in combination with other aspects of the subject matter of the present disclosure, generating an incident reconstruction model includes detecting and aligning features extracted from incident information data, converting the collected incident information data into a common dimensional space and time frame, and reconstructing a 3D model.

[0025] According to other aspects, optionally, in combination with other aspects of the subject matter of the present disclosure, methods for reconstructing an incident and methods for reconstructing incident-related actions are provided.

[0026] According to another aspect, optionally, in combination with other aspects of the subject matter of the present disclosure, one or more computing devices comprising a processor and a memory are provided. The one or more computing devices are configured to perform operations for operating a system capable of detecting incidents using road information data collected by a plurality of source modalities in a cloud computing environment via computer-executable instructions. Those operations include, for each given source modality (SM) of the plurality of source modalities, processing the road information data collected by the given SM to obtain one or more feature-level data modalities related to the given SM, thereby resulting in a plurality of obtained data modalities, wherein each given data modality has information regarding one or more features extracted from the road information data collected by the associated SM during processing; when at least one of the plurality of data modalities indicates a potential incident, fusing the plurality of data modalities into one or more machine learning models (MLMs) to confirm the detection of the incident; and providing one or more incident-related processes in response to the confirmation of the detection of the incident.

[0027] According to another aspect of the subject matter of the present disclosure, a system capable of detecting incidents using road information data collected by a plurality of source modalities is provided, the system comprising a computer configured to perform the operations described above. Optionally, at least some of those operations may be provided in a cloud environment.

[0028] According to another aspect of the subject matter of the present disclosure, a non-transitory computer-readable medium is provided, which, when executed by a computing system comprising a memory storing a plurality of program components executable by the computing system, includes instructions for operating the computing system according to the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To understand the present invention and to see how it is actually carried out, embodiments will be described as non-limiting examples with reference to the accompanying drawings. [Figure 1] Figure 1 shows a schematic block diagram of an incident detection and recovery system (IDRS) according to a particular embodiment of the subject matter of this disclosure. [Figure 2] Figure 2 shows a schematic flowchart of the operation of IDRS according to a particular embodiment of the subject matter of this disclosure. [Figure 3] Figures 3a and 3b show schematic flowcharts of non-limiting examples of merging feature-level data modalities into one or more machine learning models according to a particular embodiment of the subject matter of this disclosure. [Figure 4] Figure 4 shows a schematic diagram of a non-limiting example of fusing feature-level data modalities into one or more machine learning models according to a particular embodiment of the subject matter of this disclosure. [Figure 5] Figure 5 shows a schematic diagram of a non-limiting example of merging feature-level data modalities into one or more machine learning models according to a particular embodiment of the subject matter of this disclosure. [Figure 6] Figure 6 shows a schematic flowchart of incident reconstruction according to a specific embodiment of the subject matter of this disclosure. [Figure 7] Figure 7 shows a schematic block diagram of an incident reconstruction module according to a particular embodiment of the subject matter of this disclosure. [Figure 8] Figure 8 shows a schematic flowchart for creating a 3D behavioral reconstruction model according to a specific embodiment of the subject matter of this disclosure. [Modes for carrying out the invention]

[0030] The following detailed description includes numerous specific details for a complete understanding of the present invention. However, those skilled in the art will understand that the subject matter of this disclosure can be implemented without these specific details. Furthermore, well-known methods, procedures, components, and circuits are not described in detail so as not to obscure the subject matter of this disclosure.

[0031] Unless otherwise specified, as will be apparent from the following description, throughout this specification, terms such as “processing,” “computation,” “representation,” “fusion,” “application,” “evaluation,” and “extraction” will be understood to refer to computer operations and / or processes that manipulate and / or transform data into other data, where such data is expressed as physical quantities, such as electronic quantities, and / or represents physical objects. The term “computer” should be interpreted broadly to encompass any kind of hardware-based electronic device with data processing capabilities, including, in non-limiting examples, the Incident Detection and Reconstruction System disclosed in this application and the Processing and Memory (PMC) circuits within it.

[0032] The operations described herein may be performed by a computer specially constructed for the desired purpose, or by a general-purpose computer specially configured for the desired purpose by a computer program stored on a non-temporary computer-readable storage medium.

[0033] With this in mind, Figure 1 shows a block diagram of an incident detection and reconstruction system (IDRS) according to a particular embodiment of the subject matter of this disclosure. The IDRS 100 is operably connected to a plurality of source modalities 111-115 (collectively referred to as source modalities 110), each source modality configured to collect road information data.

[0034] Unless otherwise specified, throughout this specification the term “road information data” refers to any data, metadata, and derived data that provide information about road users, roads (including roads, intersections, road structures, sidewalks, bicycle lanes, etc.) and their traffic. Road information data can be obtained from roads by various types of sensors at various bandwidths.

[0035] The term "road users" refers to any entity that uses a road, such as pedestrians, cyclists, motorcyclists, private cars, trucks, buses, and emergency vehicles.

[0036] Source modality 110 includes fixed sensors 111 (e.g., sensors attached to elements of road structures) and mobile sensors 112 (e.g., sensors attached to vehicles, mobile devices, or sensors connected to and / or integrated with road users, sensors attached to various types of unmanned aerial vehicles (UAVs), etc.). Sensors 111 and 112 can be of various types, such as cameras, LiDAR, long-range radar, and short-range radar.

[0037] At least some of the sensors 111 and 112 may include processing and memory circuits (PMC) configured to perform initial processing of acquired data to recognize road users, define at least some of the road user parameters such as their position, speed, acceleration, direction, past trajectory and predicted future trajectory, and track at least some of the road users. As a non-limiting example, sensors 111 and 112 may be configured to track road users whose parameters match predefined criteria (e.g., speeding, excessive acceleration, dangerous predicted trajectory, etc.).

[0038] The source modality 110 may further include one or more control units (CUs) 113, which are operably connected to road infrastructure (e.g., traffic control devices) and configured to collect data from there on the real-time status and duration of traffic signals, configuration information, timing and sequence of signals, and the presence or absence of associated signals or signs (e.g., dynamic message signs, dynamic lane indicators, etc.). In certain embodiments, at each intersection, a control unit 113 may be located in a traffic cabinet and configured to collect information data from a traffic control device at the intersection. The control unit 113 may include processing and memory circuits (PMCs) configured to provide initial processing of the collected data.

[0039] Furthermore, the source modality 110 may include a V2X (Vehicle-to-Everything) unit 114 that can receive data about the vehicle's movement and / or safety data from each vehicle and / or other appropriate entities using V2X messages. Typically, a set of V2X messages includes data about each vehicle's ID, its location, orientation, speed, acceleration, past trajectory, and predicted future trajectory. The V2X unit 114 may include processing and memory circuitry (PMC) configured to perform initial processing of the received data.

[0040] Therefore, the term “collected road information data” refers to road information data captured, collected, received or otherwise obtained by a given source modality, and / or derivatives of the obtained data obtained through preprocessing provided by a given source modality.

[0041] At least some of the source modalities 111, 112, 113, and 114 can be configured to locally record and store road information data collected within a predetermined time limit.

[0042] In certain embodiments, the source modality 110 may further include a cloud-based information module (CIM) 115 configured to collect and store road-related behavioral and aggregated data received from one or more external sources (e.g., Automatic Traffic Signal Performance Index (ATSPM) and other statistics, real-time and statistical data from navigation and mapping applications). Furthermore, the CIM 115 may collect and store data on external factors related to roads and their traffic (e.g., historical and current weather forecasts, historical and current traffic forecasts, data on public events that may have a potential impact on traffic at a particular time and place, data on road construction, geographic information system (GIS) data, etc.).

[0043] Furthermore, at least a portion of the data collected by source modalities 111, 112, 113, and 114 is transferred to and stored in CIM 115. Optionally, data from source modalities 111, 112, 113, and / or 114 is transferred to CIM 115 in response to only one or more pre-configured events (including timeouts for local storage of each data).

[0044] The IDRS100 includes a plurality of engines 121 to 125 (hereinafter collectively referred to as engine 120) operably connected to the processing and memory circuit (PMC) 130. Furthermore, the PMC 130 is connected to an input / output interface 140.

[0045] Engine 120 is configured to provide an operational connection between IDRS 100 and source modality 110. Engine 120 is configured to receive road information data collected by each source modality (in pull mode and / or push mode) and to process the received data to obtain data modality. Furthermore, engine 120 is configured to supply the obtained data modality to PMC 130.

[0046] Each engine is operationally connected to one of the multiple source modalities 120, and vice versa.

[0047] One or more sensors (or other sources) connected to the same engine are considered a single source modality in this specification.

[0048] As a non-limiting example, engine 120 may include one or more engines selected from at least one fixed sensor engine 121 corresponding to at least one fixed sensor source modality 111, at least one mobile sensor engine 122 corresponding to at least one mobile sensor source modality 112, at least one CU engine 123 corresponding to at least one CU source modality 113, at least one V2X engine 124 corresponding to at least one V2X source modality 114, and at least one CIM engine 125 corresponding to at least one CIM source modality 125.

[0049] Engine 120 is an executable software component that performs the functions described below. In certain embodiments, all engines can be configured to be executed by PMC 130. In other embodiments, at least a portion of Engine 120 can be executed at least partially by PMC (not shown) of each source modality. Engine 120 can be implemented in any suitable combination of software and firmware and / or hardware.

[0050] The PMC130 comprises a processor and memory (not shown separately within the PMC) and is operably connected to the engine 120 and the I / O interface 140. The PMC130 is configured to execute multiple program components according to computer-readable instructions implemented on an internal non-temporary computer-readable storage medium. Such executable program components are hereafter referred to as function modules included in the PMC. These function modules can be implemented in any suitable combination of software, firmware, and / or hardware.

[0051] The functional modules within the PMC130 may include an operationally connected incident detection module 131 and an incident reconstruction module 132. The incident detection module 131 is configured to accept and apply one or more trained machine learning models available to the IDRS100 when operating as detailed below. The incident reconstruction module 132 is configured to enable operations that will be described in more detail with reference to Figures 6 and 7.

[0052] The I / O interface 140 can be configured to interface with a web application that allows users to retrieve, search, filter, and / or download incident-related data. Alternatively or additionally, the I / O interface 140 can be configured to provide an API exposed to external companies or individuals to enable integration with their own applications. Furthermore, the I / O interface 140 can be configured to provide a dedicated API that facilitates connections between IDRS 100 and external systems.

[0053] Furthermore, the teachings of the subject matter of this disclosure are not limited to the Incident Detection and Reconstruction System (IDRS) described with reference to Figure 1. Equivalent and / or modified functions may be integrated or separated in other ways, implemented in any suitable combination of software, firmware, and / or hardware, and run on one or more suitable devices. Source modalities and / or engines may also be integrated or separated in other ways.

[0054] IDRS100 may be a standalone entity or may be fully or partially integrated with other entities. IDRS100 can be implemented, at least in part, in a distributed and / or cloud-based and / or virtualized computing environment. The functional module (and / or part thereof) shown in Figure 1 can be distributed across multiple local and / or remote computers (including computers located in a cloud environment) and linked via a communication network.

[0055] Referring to Figure 2, a method for operating an incident detection and restructuring system (IDRS) according to a particular embodiment of the subject matter of this disclosure is shown.

[0056] IDSR100 receives road information data collected by multiple source modalities (201). Each engine 121-125 processes the road information data collected by its respective source modality to obtain one or more feature-level data modalities (202).

[0057] Engine 120 is configured to use one or more incident detection techniques, as detailed below. As a non-limiting example, Engine 120 may use anomaly detection techniques to detect such potential incidents based on the identification of anomalous data patterns that could potentially be triggered by an incident.

[0058] A data modality obtained by a given engine is associated with its respective source modality and contains information about one or more features extracted by the given engine from road information data collected by its respective source modality.

[0059] A data modality obtained by a given engine may contain information about one or more individually extracted features (e.g., trajectory, position, velocity and / or acceleration, and their changes). Alternatively or additionally, the obtained data modality may contain information about a pre-defined set of features corresponding to the types of direct and indirect incident impacts and their respective source modalities.

[0060] When an incident occurs, it often results in direct and indirect impacts that can be detected as anomalies in data patterns.

[0061] Direct impacts include changes in vehicle trajectories, abnormal vehicle positions, and changes in speed and / or acceleration. Short-range indirect impacts include stranded vehicles, people disembarking and remaining stationary, crowds gathering, and changes in traffic. Medium-range indirect impacts may lead to changes in traffic statistics such as vehicle counts and delay times. Long-range indirect impacts include obstacle removal and recovery times, and the appearance of emergency vehicles.

[0062] In a particular embodiment, a given engine is, for example, - A statistical model that can analyze the statistical characteristics of the data and detect anomalies based on deviations from expected patterns, - A machine learning (ML) model that detects anomalies by training an algorithm using a dataset of normal traffic behavior (for example, a supervised learning algorithm can be trained to classify incidents based on sensor data, or an unsupervised learning algorithm can be used to identify clusters of abnormal behavior), - A rule-based model that defines a set of rules or thresholds for what constitutes normal behavior and flags anything that deviates from those rules as abnormal (for example, a rule-based method can flag sudden changes in a vehicle's speed or position as a potential incident). One or more incident detection models, including the following, can be implemented:

[0063] In certain embodiments, the incident detection technique applied may differ for each engine 120.

[0064] Sensor engines 121 or 122 can detect incidents using rule-based models that are input with information about the scene (e.g., second-level data such as whether a vehicle is stuck, whether a pedestrian is in a dangerous location, whether a traffic jam is occurring, etc.). Alternatively or additionally, sensor engines 121 or 122 can detect anomalies using ML / statistical models that are input with information such as traffic jam length, delay time, and vehicle and pedestrian trajectories (history of location, speed, and direction of travel). Similarly, long-shorter-term memory (LSTM) networks and / or transformer neural networks may help analyze temporal sequences of sensor (e.g., radar and / or camera) data to identify patterns preceding incidents such as erratic vehicle movements or sudden deceleration.

[0065] The CU engine 123 can detect incidents using a rule-based model that takes data available from the traffic control system (e.g., control system status, phase status, detector status, service status, etc.) and / or derived data therefrom (e.g., traffic decisions from the system that operates the traffic optimization mode). Alternatively or additionally, the CU engine 123 can detect anomalies using an ML / statistical model that takes information such as detector status and phase service (time service) as input.

[0066] The V2X engine 124 analyzes V2X messages collected from the vehicle (e.g., basic safety messages of the DSRC or C-V2X protocol) and can issue real-time alerts regarding sudden braking, airbag deployment, and emergency vehicle notifications.

[0067] The CIM engine 125 can detect anomalies by utilizing an ML / statistical model that takes ATSPMS-based statistical data (e.g., total and average vehicle delays, number of vehicles, number of pedestrians, cycle time, etc.) as input.

[0068] In certain embodiments, the engine 120 can utilize vehicle trajectory analysis to detect potential incidents. As a non-limiting example, applying a clustering algorithm (e.g., K-means, DBSCAN) to vehicle trajectories obtained from sensor data can help identify anomalous patterns that may indicate an incident, such as sudden stops or deviations from the normal path. As another non-limiting example, using a predictive model to estimate the future position of the vehicle may help detect deviations between the current trajectory and the predicted trajectory.

[0069] If at least one of the obtained data modalities suggests a potential incident, IDRS100 fuses the obtained feature-level data modalities into one or more machine learning models to confirm the detection of the incident (203). If the confidence level of the incident detection meets a predetermined criterion, the incident is considered confirmed. Figures 3 to 5 show non-limiting examples of fusion techniques according to specific embodiments of the subject matter of this disclosure.

[0070] In response to a confirmed incident detection, IDRS100 provides one or more incident-related actions (204). Non-exclusive examples of incident-related actions include reconstructing the incident and initiating alerts and / or reports to emergency services, law enforcement agencies, police, insurance companies, etc.

[0071] Referring to Figures 3a and 3b, schematic flowcharts of non-restrictive examples of merging the resulting feature-level data modalities into one or more MLMs are shown.

[0072] In the example shown in Figure 3a, when road information data is collected by multiple source modalities (301), IDRS100 processes the road information data collected by each source modality individually to obtain at least one feature-level data modality for each source modality (302). If at least one of the obtained data modalities indicates a potential incident, IDRS100 fuses the obtained data modalities into an MLM (303) and assigns their weights before fusion. The fusion weights of a given data modality may depend on each source modality and / or one or more techniques applied to obtain the given data modality. After fusion, IDRS100 applies the MLM to the fused data modality to detect incidents (304).

[0073] In the example shown in Figure 3b, when road information data is collected by multiple source modalities (301), IDRS100 processes the road information data collected by each source modality individually to obtain at least a first set for each of one or more data modalities and a second set for one or more data modalities (305). IDRS100 merges the first set of data modalities into a first MLM to detect potential incidents at a first confidence level (306), and further merges the output of the first MLM and the second set of data modalities into a second MLM to detect potential incidents at a higher confidence level (307).

[0074] Examples of the above-mentioned fusion technologies are shown in more detail in Figures 4 and 5.

[0075] As shown in Figure 4, road information data is collected by source modalities 401-1, 401-2, and 401-3. Engines 402-1, 402-2, and 402-3 process the data collected by the corresponding source modalities and, for each given source modality, obtain the first and second data modalities associated with them, thereby obtaining the first set of the first data modalities and the second set of the second data modalities. As shown in the figure, the first set includes the first data modalities 403-1, 403-2, and 403-3 associated with source modalities 401-1, 401-2, and 401-3, respectively, and the second set includes the second data modalities 404-1, 404-2, and 404-3 associated with the same source modalities, respectively.

[0076] Data modalities 403-1, 403-2, and 403-3 from the first set are merged into the first MLM405 to detect potential incidents at a first confidence level. Furthermore, the output of MLM405 and data modalities 404-1, 404-2, and 404-3 from the second set are merged into the second MLM406 to detect potential incidents at a higher confidence level (and / or make a final decision regarding incident detection).

[0077] As shown in the figure, the first data modalities 403-1, 403-2, and 403-3 have weights W 1 1. W 1 2, W 1 In 3, they are fused into MLM405 respectively, and at least one of their weights is different from the other weights. Similarly, the second data modalities 404-1, 404-2, and 404-3 have weights W 2 1. W 2 2, W 2 Each of the three MLMs is fused into MLM406, with at least one of its weights being different from the others. Optionally, the output of MLM405 can also be weighted before being fused into MLM406.

[0078] In certain embodiments, at least a portion of the data modalities from the same set can have information regarding the same feature. The fusion weights of such modalities can depend on the associated source modalities. As a non-limiting example, the fusion weight of the trajectory information data modality associated with the V2X source modality can be set higher than the fusion weight of the trajectory information data modality associated with the sensor source modality. Similarly, when the road visibility is good, the fusion weight of the trajectory information data modality associated with the camera-based sensor source modality can be set higher than the fusion weight of the trajectory information data modality associated with the radar-based sensor source modality, and the reverse setting can be applied when the road visibility is poor.

[0079] Alternatively or additionally, in certain embodiments, at least a portion of the data modalities from the same set can have information regarding different features. The fusion weights of such data modalities can depend on the respective features. As a non-limiting example, the fusion weight of the data modality including information regarding an abnormal vehicle position can be set higher than the fusion weight of the data modality including trajectory information.

[0080] In certain embodiments, at least a portion of the source modalities are associated with three or more sets of the data modalities obtained therefrom. In certain embodiments, the data modalities from the set N are fused with the output of the MLM N and the MLM N+1 to finally detect an incident at the required confidence level. In certain embodiments, the output of the MLM can be weighted before fusion. As a non-limiting example, the fusion weight of such an output can be increased for each subsequent MLM.

[0081] It should be noted that different source modalities can be associated with different sets of data modalities derived from them. Similarly, data modalities derived from one source modality can belong to multiple sets different from those derived from another source modality. Therefore, MLM I The source modality associated with the data modality that is fused is MLM. K The source modality associated with the fused data modality may differ, at least partially.

[0082] Furthermore, it should be noted that the teachings of the subject matter of this disclosure are not bound by the configuration of the MLM sequence described above. MLMs can be organized in chains, trees or other suitable configurations.

[0083] Figure 5 shows an unrestricted example of a chain of four MLMs. The first data modality (501-1 to 501-4) of the first set provides information about a pre-configured set of features corresponding to the impact of a direct incident; the second data modality (502-1 to 502-4) of the second set provides information about a pre-configured set of features corresponding to the impact of a short-range indirect incident; the third data modality (503-1 to 503-4) of the third set provides information about a pre-configured set of features corresponding to the impact of a medium-range indirect incident; and the fourth data modality (504-1 to 504-4) of the fourth set provides information about a pre-configured set of features corresponding to the impact of a long-range indirect incident.

[0084] The first data modality (501-1 to 501-4) is merged into the first MLM (505). Its output, which contains information on the detection of potential incidents, is merged with the second data modality (502-1 to 502-4) into the second MLM 506. The output of MLM 506 is based on a short-range field understanding and is merged with the third data modality (503-1 to 503-4) into the third MLM 507. The output of MLM 507 is based on a medium-range field understanding and is merged with the fourth data modality (504-1 to 504-4) into the fourth MLM 508. The output of MLM 508 is based on a long-range field understanding and can be used for final decisions regarding incident detection, and optionally for incident severity assessment.

[0085] Referring to Figure 6, a schematic flowchart of incident reconstruction according to a particular embodiment of the subject matter of this disclosure is shown. In response to incident detection, IDRS100 collects incident information data from one or more source modalities to obtain incident information data (601), and generates an incident reconstruction model using the collected incident information data and its derived data (602). Furthermore, IDRS100 generates a visual representation of the incident (606), enhances the incident representation (607), and enables rendering of the reconstructed incident (608).

[0086] Figure 7 shows the corresponding functional units of the incident reconstruction module 132. As shown in the figure, the incident reconstruction module 132 includes a data collection unit 701 operably connected to the incident reconstruction unit 702, which is further operably connected to a data enrichment unit 703. All units are operably connected to an incident reconstruction database 704 configured to store incident-related data and its derived data, as well as incident representations.

[0087] Data collection 601 includes collecting incident-related data from relevant source modalities and processing at least a portion of the collected data to derive inputs necessary for incident reconstruction.

[0088] In response to incident detection, the incident reconstruction module 132 requests engines 121-125 to collect road information data from source modalities 120. The acquired road information data corresponds to a specific time frame before and after the time when the incident is believed to have occurred.

[0089] Engines 121-125 receive the requested data and process at least a portion of it to derive the inputs necessary for incident reconstruction. The input and processing algorithms are specified for each engine and depend on the corresponding source modality. The road information data and its derived data thus collected constitute incident-related data and can be stored in database 704.

[0090] In certain embodiments, all road information data to be acquired can be received from the source modality from which the respective data was collected. In other embodiments, at least a portion of the road information data collected by source modalities 111-114 needs to be transferred to and retrieved from CIM 115 for storage when requested. In such cases, engines 121-124 can receive the necessary data by requesting it from engine 125.

[0091] As a non-limiting example, incident-related data may include video feeds, traffic signal status and duration data, radar data, V2X data, and third-party data, which are combined with their respective metadata. Furthermore, incident-related data may include the trajectories (location, speed, and acceleration) of all road users and other relevant data. Such data may be received from the source modality or derived by engines 121-124 upon request from the incident reconstruction module 132. The incident-related data corresponds to a first period before the incident and a second, pre-defined period after the traffic event. The length of these periods may vary depending on the type of data and the severity of the incident.

[0092] For example, if an accident is detected, the synchronized video feed will be sent. - A 10-minute high-resolution video including the 2 minutes before the accident, the moment of the accident, and the 6 minutes after the accident, - 100 minutes of low-resolution video at a rate of 1 FPS, including 30 minutes before the accident, 10 minutes at the time of the accident, and 60 minutes after the accident. It can include...

[0093] Traffic signal information provides the timestamped status of traffic lights and signs necessary to provide context for an incident. This context includes information on the status and duration of traffic lights, the timing and sequence of signals, and the presence or absence of relevant traffic lights and signs. This information may also include data on the state of the intersection at the time of a given incident, as well as data on traffic flow and patterns before and after the incident.

[0094] Radar data provides information about vehicle speed and location. V2X data transmitted from vehicles and infrastructure can provide insights into vehicle behavior and actions prior to an incident. Third-party data may include indirect data about intersection behavior (e.g., statistics on congestion that occurred, delay times, and other data about the intersection conditions leading up to the incident).

[0095] The step of processing the collected incident information data to generate an incident reconstruction model (602) includes feature detection and alignment (603), conversion to a unified dimension and time axis (604), and reconstruction of the 3D model (605).

[0096] As a non-limiting example, feature detection includes using computer vision and machine learning algorithms to detect key features in a video feed (e.g., relevant vehicles, pedestrians, and other notable objects). Radar and V2X data can then be used to further validate those features.

[0097] Furthermore, feature detection also includes identifying key timestamps that indicate important events in the incident sequence (e.g., sudden braking, impact of a collision). These timestamps help to further align data across multiple sources.

[0098] Data alignment involves aligning data from different source modalities (and / or different data modalities from the same source) based on detected features and timestamps. This includes creating a common reference frame, such as aligning radar detections and video frames, and V2X "frames" from all source modalities together, based on vehicle position, movement, and time.

[0099] By transforming aligned data into a unified dimension and time axis (604), a single source of truth can be created. All data modalities are appropriately normalized and scaled to fit the unified dimensional model.

[0100] Furthermore, a unified dimensional model is used to reconstruct a 3D model of the incident site (605). This model reflects the real-world location, movement, and interaction of all road users involved, giving it a sense of reality.

[0101] The generated 3D model enables a step-by-step visualization of the incident. This visualization can be used for analysis, investigation, and reproduction. Simulation and analysis tools can be used to analyze the dynamics of the incident and understand its causal factors.

[0102] Creating a reenactment video (606) involves analyzing incident data to identify the relevant causes of the incident. This identification can be achieved by comparing the incident parameters with a database of similar incidents (e.g., the IR database 704), thereby identifying commonalities and potential causes of the incidents. Relevant parameters include the type of incident, such as running a red light, sudden braking, or pedestrian involvement; background information, such as weather or a dangerous intersection; the types of parties involved, such as trucks and passenger cars, passenger cars together, or a chain of incidents; and severity.

[0103] The relevant causes of an incident can be identified by analyzing incident data in an IR database using machine learning models and comparing it to new incident data (input). This process is designed to identify commonalities and potential causes of incidents, providing a more comprehensive understanding of the situation. Machine learning models used for this purpose include clustering models such as k-means clustering and hierarchical clustering, or anomaly detection models such as autoencoders and one-class SVMs. By analyzing incident data and identifying patterns and relationships within it, the model generates outputs that highlight the relevant causes of the incident (scores for each parameter based on their importance and relevance to the incident - outputs). As mentioned above, causes include the type of incident, background information, and types of stakeholders. The outputs are used to create videos that visually highlight this information, helping viewers gain a deeper understanding of what happened and why.

[0104] The generated 3D model can be used to create an animated video that is visually appealing, easy to understand, and accurately represents the incident (606). This method involves using a machine learning model that maps incident-related data, such as intersection geometric information, trajectory data, traffic light status, time, and cause of the incident, to input in a graphics engine language. This can be achieved using deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs).

[0105] The model's output is a set of parameters used as input to the graphics engine, such as the size and color of relevant information that will be highlighted in the animation. The graphics engine takes these parameters and translates them into a visual representation.

[0106] Model training can utilize deep learning approaches, including neural networks that can learn from data. The model is trained using a dataset of incidents and their corresponding animations, aiming to learn patterns and relationships between the data and animations. The training process can be manual, with experts labeling and annotating relevant information within the dataset, or automated, using unsupervised learning techniques to train the model on a large dataset of incidents and their corresponding animations.

[0107] By using a model that maps all relevant data to input in a graphics engine language, process automation and the creation of animations for multiple incidents become possible. Furthermore, the model can be continuously improved by retraining it with new data.

[0108] Reconstructed incidents are added to the IR database along with all necessary fields, including location, time, and output from the reconstruction process. Users can view the reconstructed incidents and all related information in the web application.

[0109] In certain embodiments, faces and license plates in reconstructed videos can be blurred. This can be done using computer vision models that detect and blur faces and license plates while highlighting vehicle types and individuals, slowing down footage of the incident moment, and highlighting important features such as weather conditions and other relevant data. This can be achieved using object detection models such as YOLO, RCNN, and SSD, or image segmentation models such as U-Net, Mask R-CNN, and DeepLab. Once faces and license plates are detected, blur filters can be applied to their respective regions to conceal identifying information.

[0110] In certain embodiments, a visual representation of an incident can be generated as a schematic animated video. The video can provide a customized view of the incident and the surrounding area, and can remove noise and irrelevant information from intersections. Such videos also help to provide a situational understanding of the incident, a clear picture of what happened, and protect the privacy of those involved.

[0111] In certain embodiments, an animated reenactment video can be generated by further processing the RHR video. Alternatively, in other embodiments, an animated reenactment video can be automatically generated by combining all the collected incident-related data into a single, cohesive visual presentation.

[0112] After generating a visual representation of an incident, data augmentation (607) can be performed. Data augmentation goes beyond a basic reconstruction of the incident, providing a deeper level of analysis and a more comprehensive understanding of the circumstances leading to the incident. The augmented data provides rich information that can be used to classify and filter incidents based on various criteria, such as the type of incident, the types of vehicles involved, and even the time it took for traffic to return to normal. The augmented data is crucial for stakeholders such as insurance companies and traffic planners because it provides valuable insights into the underlying causes of incidents.

[0113] Data augmentation can be provided as a cloud-based service. The data augmentation service includes additional data such as weather conditions, intersection geometry, size and congestion, intersection behavior (including hazards), and historical incident information. This information helps identify whether an intersection is hazardous and prone to incidents. Included information includes incident types (such as crossing a red light or making a left turn), incident severity, arrival and departure times of emergency vehicles, time taken for traffic to return to normal, and types of vehicles involved. The data augmentation service also includes automated license plate recognition (ALPR) data, which helps identify vehicles involved in incidents and determine fault.

[0114] Each instance of the IR database can include various fields that provide users with valuable insights into incidents and their causes. - Metadata (time, location, weather data: this information helps users understand the environmental conditions and other factors that may have caused the incident), - Intersection data (size, congestion level, past incident information, risk score), - Collision data (this includes information on the severity of the incident, whether there were any injuries or fatalities, the types of vehicles involved in the incident, and other parties involved. Furthermore, the DB also includes information on the type of incident, such as running a red light or sudden braking.) and, - Indirect information (including details of emergency response personnel and the time it took them to arrive at the scene, the impact on traffic, and the time it took for traffic to return to normal), -Evidence (This field includes raw video files, metadata, traffic light status (TLS), reenactment videos, and animated videos.) It can include...

[0115] This allows users to filter and categorize incidents according to their needs. Users can add incidents to their favorites, mark them for further work, and share them with other users by writing both private and public comments. The application also allows users to associate incidents with specific cases or incidents, enabling a comprehensive understanding of the impact of incidents on the community.

[0116] Furthermore, IDRS100 can include APIs that can be made public to external companies and individuals. This allows for the integration of IR database 704 data with other systems and the automation of various processes.

[0117] In addition to the above, IDRS100 can be integrated with vehicle telematics systems, thereby facilitating data collection from vehicles involved in an incident and supporting comprehensive analysis and informed decision-making. This integration enables the identification of stakeholders for personalized solutions across various domains, stakeholder profiling based on driving data, and the development of customized service offerings.

[0118] Modern vehicles are increasingly equipped with telematics devices that monitor various metrics, including driver behavior and vehicle performance. This wealth of data is invaluable not only for personalizing services but also for enhancing operational efficiency, safety protocols, and customer engagement in areas such as car sales, vehicle rental services, fleet management, and smart city initiatives.

[0119] The process of identifying vehicles involved in an incident using telematics systems involves a crucial step called "pairing," which matches vehicles between telematics and IDRS. This is often achieved by correlating GPS data from both sources. Effective pairing techniques include: -Nearest Neighbor Algorithm (finds the closest match by calculating the distance between GPS coordinates), - Dynamic time warping algorithm (aligns time-series data from both systems to find the best match, especially effective when the data is not synchronized), - The Kalman filter algorithm (which removes noise from GPS data and estimates the vehicle's position using a predictive mathematical model), - Bayesian network algorithm (using a probabilistic model to estimate the probability of agreement between GPS coordinates) and It can include...

[0120] Additional parameters such as vehicle type and physical characteristics (e.g., color) can enhance the pairing process. If initial pairing is difficult, accuracy can be improved by incorporating additional data such as vehicle trajectory patterns and hybrid models that utilize machine learning.

[0121] Collaboration with stakeholders can be proactive, using telematics and incident data available through the VAR system to identify and respond to events, or passive, using data matching to investigate suspicious incidents retrospectively. Data matching and profiling techniques include the use of classification and clustering models, which are useful for personalized service delivery and operational improvements.

[0122] According to certain embodiments of the subject matter of this disclosure, machine learning models used for profiling can analyze accident parameters, driver behavior, and other relevant data to provide personalized pricing recommendations. For example, clustering models can group customers based on driving behavior, and decision trees can identify the most important parameters influencing pricing recommendations. Furthermore, IDRS100 can provide feature engineering that enables the uncovering of one or more new features to consider. In non-limiting examples, this can be done by decision trees, principal component analysis (PCA), etc.

[0123] In certain embodiments, 3D reconstructed models may be useful for behavioral analysis. By creating a 3D behavioral model reconstruction, behavioral information can be obtained, for example, the number of individuals involved in an accident, the cause of the accident, and actions immediately before, during, and after the event. Furthermore, such models may be useful for generating customized reports that include information on fault determination, the impact of actions on the accident outcome, and potential injuries.

[0124] Figure 8 shows a schematic flowchart for creating a 3D behavioral reconstruction model according to a particular embodiment of the subject matter of this disclosure.

[0125] IDRS100 uses data containing information about incidents collected from one or more source modalities to detect behavioral characteristics (801). Furthermore, IDRS100 analyzes each behavior and identifies its cause (802). Next, IDRS100 generates a unified behavioral model and provides a 3D reconstruction (803). The 3D reconstruction can be used for simulations and interactive estimations (804), as well as for reporting and analysis.

[0126] The detection of behavioral characteristics (801) is, - Human and object detection (using advanced computer vision algorithms to identify and track each individual and vehicle involved in the accident throughout the entire video footage), - Action recognition (implementing machine learning models trained to recognize specific actions, such as people getting on and off vehicles, and capturing important action moments) and It can include...

[0127] The extracted features provide the following information: - Signs of distraction (Identify behaviors that indicate distraction, such as the driver or pedestrian using a mobile phone, looking away from the road, or engaging in activities unrelated to driving.) - Aggressive driving patterns (Detects signs of aggressive driving before an accident, such as speeding, sudden braking, sudden lane changes without using turn signals, tailgating, and erratic driving.) - Seat belt use (Determining whether the driver and passengers were wearing seat belts at the time of the accident. This may affect injury compensation claims and liability assessments.) - Pedestrian behavior (analyzing pedestrian behaviors that could lead to accidents, such as crossing outside of designated crosswalks, ignoring traffic signals, and suddenly cutting in front of vehicles.) - Driver reaction time (Estimates the driver's reaction time to sudden obstacles, changes in traffic signals, or actions of other road users. This may indicate the driver's attention and adherence to safe driving practices.) - Vehicle condition and maintenance indicators (detecting visible signs of vehicle maintenance deficiencies that could lead to accidents, such as worn tires, malfunctioning lights, and damaged brakes.) - Weather and visibility conditions (evaluate the impact of weather conditions (rain, fog, snow) and visibility conditions (nighttime, glare) on the behavior of drivers and pedestrians.) - Obedience to traffic signals and signs (determining whether the vehicle and pedestrians were following traffic signals, stop signs, yield signs, and other traffic regulations at the time of the accident.) - Post-accident actions (analyzing individual actions immediately after the accident, such as rescue efforts, ensuring safety at the scene, information exchange, and actions that may suggest avoidance of responsibility.) - Road rage or confrontational behavior (Identify aggressive or confrontational behavior between individuals before, during, or after the incident. This may be relevant to understanding the context of the incident and its escalation.)

[0128] Behavioral analysis and causal identification (802) - Sequence analysis (analyzing the time-series sequence of detected behaviors to understand individual behavior before, during, and after the accident), - Causal modeling (using collected data to model potential causes of accidents. This includes analyzing vehicle telemetry data on sudden stops and accelerations and correlating it with video evidence indicating driver inattention or pedestrian behavior.) and It can include...

[0129] Data fusion and 3D reconstruction (803) can be provided in the manner detailed with reference to Figure 6. The unified behavioral model integrates the collected data into a unified model that represents both the physical and behavioral aspects of the accident scene. This includes creating a timeline of the event based on the detected sequence of behaviors and vehicle movements. 3D scene reconstruction reconstructs the accident scene using 3D modeling software, incorporating both the physical environment and the animated behaviors of individuals and vehicles. This model is generated to visually represent the timeline of the event and highlight key moments identified in the behavioral analysis.

[0130] Simulation and interactive estimation (804) involves developing interactive 3D models that allow various stakeholders to explore from different perspectives, zoom in on specific actions, and replay accident sequences from different angles. The models can be further annotated with important information such as timestamps of key events, vehicle speeds at the time of collision, and points of interest such as the initial or final contact locations of the vehicles and individuals.

[0131] It should be understood that the present invention is not limited in its application to the details described herein or shown in the drawings. Other embodiments are possible, and the invention can be implemented and carried out in a variety of ways. Therefore, it should be understood that the expressions and terms used herein are for illustrative purposes only and should not be considered limiting. Accordingly, those skilled in the art will understand that the underlying concepts of this disclosure can be readily used as a basis for designing other structures, methods, and systems to achieve various purposes of the subject matter of this disclosure.

[0132] Furthermore, it will be understood that the system according to the present invention can be implemented, at least in part, on a properly programmed computer. Similarly, the present invention also envisions a computer program readable by a computer for performing the method of the present invention. Moreover, the present invention also envisions a non-temporary computer-readable memory that tangibly embodies a program of instructions executable by a computer for performing the method of the present invention.

Claims

1. A computerized method for detecting incidents using road information data collected by multiple source modalities, A step of processing road information data collected by a given source modality (SM) from among the plurality of source modalities, individually for each given source modality, to obtain one or more feature-level data modalities associated with the given SM, thereby resulting in a plurality of obtained data modalities, wherein each given data modality has information about one or more features extracted from the road information data collected by the associated SM during processing. If at least one of the aforementioned data modalities indicates a potential incident, the steps include: merging the aforementioned data modalities into one or more machine learning models (MLMs) to confirm the detection of the incident; A method characterized by comprising the step of providing one or more incident-related processes in response to confirmation of the detection of the aforementioned incident.

2. In the method according to claim 1, The method is characterized in that the plurality of SMs include a combination of at least one sensor configured to acquire road information data, a control unit configured to collect data from one or more road infrastructure elements, a V2X unit configured to receive vehicle movement data and / or safety data, and a cloud-based information module configured to collect road-related behavioral data and aggregated data.

3. In the method according to claim 1 or 2, A method characterized in that the processing of road information data collected by the given SM is performed to obtain data modalities indicating potential incidents.

4. In the method according to claim 3, A method characterized in that one or more incident detection models used during processing depend on the SM that collected the respective road information data.

5. In the method according to any one of claims 1 to 4, A method for processing road information data, characterized by including the application of one or more anomaly detection models configured to detect potential incidents based on the identification of anomalous data patterns that may be caused by the incident.

6. In the method according to any one of claims 1 to 5, A method characterized in that at least one data modality has information about a predefined set of features corresponding to the relevant SM and type of incident impact, wherein the type of incident impact is selected from a group including direct impact, short-range indirect impact, medium-range indirect impact, and long-range indirect impact.

7. In the method according to any one of claims 1 to 6, A method characterized in that acquired data modalities are fused into one or more MLMs using assigned fusion weights.

8. In the method of claim 7, A method characterized in that the fusion weights of a given data modality depend on one or more techniques applied to obtain the respective associated SMs and / or a given data modality.

9. In the method according to any one of claims 1 to 8, For each given SM among at least a portion of the plurality of SMs, the road information data collected by the given SM is processed to obtain a first data modality and a second data modality associated therewith, thereby resulting in a plurality of first data modalities and a plurality of second data modalities. The steps include: integrating the aforementioned multiple first data modalities into a first MLM to detect potential incidents at a first confidence level; A method further comprising the step of further merging the output of the first MLM and the plurality of second data modalities into a second MLM to detect potential incidents at a higher level of confidence.

10. In the method according to claim 9, A method characterized in that the first data modality has information relating to a pre-configured set of features corresponding to the impact of one or more direct incidents, and the second data modality has information relating to a pre-configured set of features corresponding to the impact of one or more short-range indirect incidents.

11. In the method according to claim 10, A step of processing road information data collected by a given SM, at least a portion of the plurality of SMs, to obtain a third data modality and a fourth data modality associated therewith, thereby yielding a plurality of third data modalities and a plurality of fourth data modalities, wherein the third data modality has information relating to a preset set of features corresponding to the impact of one or more medium-range indirect incidents, and the fourth data modality has information relating to a preset set of features corresponding to the impact of one or more long-range indirect incidents. The steps include: merging the output of the second MLM and the plurality of third data modalities into a third MLM to detect potential incidents with an even higher level of confidence; A method further comprising the step of merging the output of the third MLM and the plurality of fourth data modalities into a fourth MLM to confirm the detection of an incident.

12. In the method according to any one of claims 1 to 11, A method characterized in that the incident-related processing includes the reconstruction of the incident.

13. In the method according to claim 12, The reconstruction of the aforementioned incident, The steps include: collecting incident information data from multiple service meters (SMs) corresponding to the time frame before and after the incident occurred; The steps include: processing the collected incident information data to generate an incident reconstruction model; Using the generated incident reconstruction model, the steps include generating a visual representation of the incident, Steps to expand incident descriptions, A method characterized by comprising the step of making the reconstructed incident renderable.

14. In the method according to claim 13, A method for generating the aforementioned incident reconstruction model, characterized in that it includes detecting and aligning features extracted from incident information data, converting the collected incident information data into a common dimensional space and time frame, and reconstructing a 3D model.

15. One or more computing devices comprising a processor and memory, The one or more computing devices are configured to perform operations via computer executable instructions to operate a system capable of detecting incidents using road information data collected by multiple source modalities in a cloud computing environment. The aforementioned operation, A step of processing road information data collected by a given source modality (SM) among the plurality of source modalities to obtain one or more feature-level data modalities associated with the given SM, thereby resulting in a plurality of obtained data modalities, wherein each given data modality has information about one or more features extracted from the road information data collected by the associated SM during processing. If at least one of the aforementioned data modalities indicates a potential incident, the steps include: merging the aforementioned data modalities into one or more machine learning models (MLMs) to confirm the detection of the incident; One or more computing devices, characterized by including the step of providing one or more incident-related processes in response to confirmation of the detection of the aforementioned incident.

16. In one or more computing devices according to claim 15, One or more computing devices further configured to perform the operations described in any one of claims 2 to 14.

17. A system capable of detecting incidents using road information data collected by multiple source modalities, A system comprising a computer configured to perform the operations described in any one of claims 1 to 14.

18. In the system described in claim 17, A system characterized in that at least a portion of the above operations is provided in a cloud environment.

19. A non-temporary computer-readable medium, A non-temporary computer-readable medium characterized by including instructions that cause a computing system to operate in accordance with any one of claims 1 to 14 when executed by a computing system having memory storing a plurality of program components that can be executed by a computing system.

Images