Intelligent driving abnormal event judgment method and device, electronic equipment and storage medium

Abstract

The invention relates to an intelligent driving abnormal event judgment method and device, electronic equipment and a storage medium, and belongs to the technical field of intelligent traffic. The target data comprises first video data and first driving system data associated with the suspected abnormal event; detecting the target data based on a time sequence anomaly detection strategy, and determining a time interval when an abnormal event occurs; extracting second video data and second driving system data associated with the abnormal event corresponding to the time interval; and judging the recognition accuracy of the abnormal event based on the second video data and the second driving system data. According to the intelligent driving abnormal event judgment method provided by the invention, the accuracy of abnormal event recognition can be accurately judged, and misrecognition is reduced while missed recognition is prevented.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to an intelligent driving abnormal event judgment method, device, electronic equipment and storage medium. Background Art

[0002] Currently, the detection of abnormal events that occur during the driving of smart driving cars is mainly completed by the vehicle side. The vehicle-side domain controller and other devices identify abnormal events based on changes in the internal data of the smart driving system, and then label them locally. The labeled data is stored in the local storage for subsequent tracing of abnormal problems.

[0003] The existing method performs real-time abnormal event judgment on the vehicle side, and reflects the abnormal problems on the vehicle side by establishing a rule-based data model. However, this method is limited to relatively simple rule logic, the signals involved are relatively simple, the events identified are relatively clear, and the working conditions when the events occur are relatively simple. When multiple events occur in parallel, the data changes in the same time period are diverse, and multi-dimensional analysis and identification problems are required. At this time, it is difficult to fully identify only through the internal data of the vehicle side. The limited computing and processing resources on the vehicle side cannot meet the simultaneous analysis of multi-dimensional data. At the same time, data quality, data integrity, data continuity, etc. all affect the judgment of abnormal triggered events. Afterwards, manual verification can only look at problems based on limited labeled data without other auxiliary information, and the accuracy and reliability of the triggered events are not high. Summary of the invention

[0004] In view of this, it is necessary to provide a method, device, electronic device and storage medium for judging abnormal events in intelligent driving to solve the problem of low accuracy and reliability of existing abnormal event identification.

[0005] In order to solve the above problems, the present invention provides a method for determining abnormal events in intelligent driving, comprising: Receiving target data sent by the vehicle end; the target data includes first video data and first driving system data associated with a suspected abnormal event; Detect the target data based on the time series anomaly detection strategy to determine the time interval when the abnormal event occurred; Extracting second video data and second driving system data associated with the abnormal event corresponding to the time interval; Based on the second video data and the second driving system data, the recognition accuracy of the abnormal event is determined.

[0006] In a possible implementation, the suspected abnormal event is determined when the vehicle side identifies a jump in a key signal; The key signal includes at least one of the following: Autopilot status, autopilot degradation status, autopilot lateral error, autopilot deceleration, autopilot braking, and lane line confidence.

[0007] In a possible implementation, the detecting the target data based on the time series anomaly detection strategy to determine the time interval in which the abnormal event occurs includes: Based on the time series anomaly detection strategy, an event recognition model is constructed; Based on the event recognition model, determining the time interval when the abnormal event occurs; The time series anomaly detection strategy includes at least one of the following: Signal mutation detection, signal anomaly detection, continuity judgment and joint judgment of multiple related signals.

[0008] In a possible implementation, extracting the second video data and the second driving system data associated with the abnormal event corresponding to the time interval includes: extracting the video data and driving system data within the time interval from a vehicle storage device; The video data and the driving system data within the time interval are time stamp synchronized to obtain the second video data and the second driving system data.

[0009] In a possible implementation manner, judging the recognition accuracy of the abnormal event based on the second video data and the second driving system data includes: Matching and fusing the second video data and the second driving system data to obtain fused data; Based on the fused data, the recognition accuracy of the abnormal event is determined.

[0010] In a possible implementation, judging the recognition accuracy of the abnormal event based on the fused data includes: Based on the fused data, displaying a change trend of the second driving system data in the second video data; Based on the change trend of the second driving system data, the recognition accuracy of the abnormal event is judged.

[0011] The present invention also provides an intelligent driving abnormal event judgment device, comprising: A receiving module, used to receive target data sent by the vehicle end; the target data includes first video data and first driving system data associated with a suspected abnormal event; A detection module, used to detect the target data based on a time series anomaly detection strategy to determine the time interval when the abnormal event occurs; An extraction module, used for extracting second video data and second driving system data associated with the abnormal event corresponding to the time interval; A judgment module is used to judge the recognition accuracy of the abnormal event based on the second video data and the second driving system data.

[0012] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the intelligent driving abnormal event judgment method described in any of the above implementation methods.

[0013] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for judging abnormal events in intelligent driving described in any of the above-mentioned implementation methods is implemented.

[0014] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the intelligent driving abnormal event judgment method described in any of the above-mentioned implementation methods.

[0015] The beneficial effects of the present invention are as follows: the intelligent driving abnormal event judgment method, device, electronic device and storage medium provided by the present invention can monitor whether a key signal mutates during the intelligent driving process, and can preliminarily identify suspected abnormal events through simple judgment logic, and send the first video data and the first driving system data associated with the suspected abnormal event to the cloud, so that the cloud can use the time series abnormality detection strategy to accurately identify the abnormal event and the time interval of the abnormal event, thereby extracting the second video data and the second driving system data associated with the abnormal event, and combining the second video data and the second driving system data to judge whether the abnormal event is accurate. Simplify the complex process of abnormal event recognition on the vehicle side, reduce the local computing pressure on the vehicle side, preliminarily screen out suspected abnormal problem points through simple logic design, report relevant information in combination with the wireless communication network, ensure that the strength of abnormal problem recognition is improved without wasting vehicle resources, and use cloud optimization to accurately locate abnormal events through the time series abnormality detection strategy, effectively integrate multi-dimensional information in combination with video and data, accurately judge the accuracy of abnormal event recognition, prevent missed recognition and reduce misrecognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 One of the method flow charts of an embodiment of the intelligent driving abnormal event judgment method provided by the present invention; Figure 2 A second method flow chart of an embodiment of the method for determining abnormal events in intelligent driving provided by the present invention; Figure 3 A schematic diagram of the structure of an embodiment of the intelligent driving abnormal event judgment device provided by the present invention; Figure 4 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] In the description of the embodiments of the present invention, unless otherwise specified, "multiple" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone.

[0020] The descriptions of "first", "second", etc. involved in the embodiments of the present invention are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0021] Figure 1 One of the method flow charts of an embodiment of the intelligent driving abnormal event judgment method provided by the present invention is as follows Figure 1 As shown, the intelligent driving abnormal event judgment method includes: S101, receiving target data sent by the vehicle end; the target data includes first video data and first driving system data associated with a suspected abnormal event; S102, detecting the target data based on a time series anomaly detection strategy to determine a time interval in which an abnormal event occurs; S103, extracting second video data and second driving system data associated with the abnormal event corresponding to the time interval; S104: Determine the recognition accuracy of the abnormal event based on the second video data and the second driving system data.

[0022] It should be noted that the execution subject of the intelligent driving abnormal event judgment method provided by the present invention can be the cloud, and the vehicle side can be the autonomous driving electronic control unit (ADECU). Video data can include video records of the vehicle's surroundings. Driving system data is the internal data of the intelligent driving system, for example, it can include various sensor records such as GPS location, vehicle speed, acceleration, environmental information, working condition information, etc.

[0023] Compared with the prior art, the method for judging abnormal events in intelligent driving provided by the embodiment of the present invention can monitor whether a key signal mutates during intelligent driving, and can preliminarily identify suspected abnormal events through simple judgment logic, and send the first video data and the first driving system data associated with the suspected abnormal event to the cloud, so that the cloud can use the time series anomaly detection strategy to accurately identify the abnormal event and the time interval of the abnormal event, thereby extracting the second video data and the second driving system data associated with the abnormal event, and combining the second video data and the second driving system data to judge whether the abnormal event is accurate. The complex process of abnormal event recognition on the vehicle side is simplified, the local computing pressure on the vehicle side is reduced, and the suspected abnormal problem points are preliminarily screened out through a simple logic design, and the relevant information is reported in combination with the wireless communication network, so as to improve the strength of abnormal problem recognition without wasting resources on the vehicle side, and utilize cloud optimization to accurately locate abnormal events through the time series anomaly detection strategy, effectively integrate multi-dimensional information in combination with video and data, accurately judge the accuracy of abnormal event recognition, and prevent missed recognition while reducing misrecognition.

[0024] In some embodiments of the present invention, the suspected abnormal event is determined when the vehicle side identifies a jump in a key signal; The key signal includes at least one of the following: Autopilot status, autopilot degradation status, autopilot lateral error, autopilot deceleration, autopilot braking, and lane line confidence.

[0025] When the vehicle side identifies a suspected abnormal event, it triggers the upload of relevant data. The vehicle-side ADECU can detect the internal data of the intelligent driving system in real time, and monitor key signals such as autonomous driving status, autonomous driving degradation status, autonomous driving lateral error, autonomous driving deceleration, autonomous driving braking, and lane line confidence.

[0026] By recording the time point T when the key signal jumps, the Internet of Vehicles communication terminal is triggered to upload the first driving system data and the first video data, i.e., the target data, for a period of time [TM, T+N] before and after the jump time point.

[0027] The signal jump of the autonomous driving state usually refers to the switching of the autonomous driving system from a normal state to an abnormal or degraded state. This switching may be caused by the system detecting an internal fault, insufficient environmental perception, or system overload.

[0028] In the degraded state of autonomous driving, some functions of the system may be turned off or transferred to manual control. This change in signal indicates that the system may have encountered a situation that it cannot handle completely autonomously and requires human intervention to ensure safety.

[0029] The lateral error of autonomous driving refers to the deviation between the vehicle's actual driving trajectory and the planned trajectory. When the lateral error increases suddenly, it may indicate that the vehicle is not following the planned path correctly, which may be caused by sensor errors, environmental interference or system failure.

[0030] The jump in the automatic driving deceleration signal indicates that the vehicle's deceleration rate is abnormal. This may be because the system predicts that there is an obstacle or an emergency situation ahead, such as emergency braking, and needs to decelerate quickly to avoid a collision.

[0031] The change of the automatic driving brake signal refers to the situation where the system automatically controls the brakes. If the brake signal is triggered suddenly, it may be because there is an emergency ahead that requires immediate stopping, or the system detects a potential collision risk and takes preventive measures.

[0032] Lane line confidence reflects the accuracy and reliability of the system's recognition of lane markings. When the lane line confidence decreases, it means that the system's recognition of the current lane information is not accurate enough, which may cause the vehicle to deviate from the lane or trigger other dangerous driving behaviors.

[0033] The intelligent driving abnormal event judgment method provided by the embodiment of the present invention determines suspected abnormal events by monitoring the jumps of various key signals. These signals cover multiple aspects such as automatic driving status, degradation status, lateral error, deceleration, braking signal and lane line confidence. By comprehensively analyzing these signals, the system can promptly identify and handle potential abnormal situations, thereby improving the safety and reliability of the automatic driving system.

[0034] In some embodiments of the present invention, the detecting the target data based on the time series anomaly detection strategy to determine the time interval in which the abnormal event occurs includes: Based on the time series anomaly detection strategy, an event recognition model is constructed; Based on the event recognition model, determining the time interval when the abnormal event occurs; The time series anomaly detection strategy includes at least one of the following: Signal mutation detection, signal anomaly detection, continuity judgment and joint judgment of multiple related signals.

[0035] The cloud performs accurate identification of abnormal events, accesses the data and videos uploaded by the Internet of Vehicles communication terminal, performs data storage and video file storage, and uses the terminal + timestamp naming rule for video file storage. An event recognition model based on complex logic is designed.

[0036] A time series anomaly detection strategy can be adopted to design an event recognition model based on theoretical and empirical rules according to the methods of signal mutation, signal anomaly, continuity judgment, and joint judgment of multiple related signals.

[0037] Accurately identify abnormal events in the intelligent driving system through offline batch processing. Batch process event data with a frequency of 100ms, input into the event recognition model, and accurately identify the time intervals [T1, T2] when various abnormal events occur.

[0038] For example, the time series anomaly detection strategy identifies outliers or abnormal patterns by analyzing time series data. Anomalies can be defined as data points or data patterns that are significantly different from normal behavior. In the field of intelligent driving, time series anomaly detection can help detect abnormal states of the system in a timely manner, prevent possible accidents, and improve the safety and reliability of the system.

[0039] Signal mutation refers to the phenomenon that the value of a variable suddenly changes significantly in time series data. The mutation point of the signal can be identified by calculating the gradient of the data or using differential operations, which can quickly locate the specific time point when the mutation occurs in the data.

[0040] Signal anomalies are individual points or short-term patterns in the data that are significantly different from the majority of the data. Such anomalies may be caused by system errors, external interference, or data entry errors. Detection methods include statistical-based methods (such as 3 σ Principle or box plot analysis) and distance-based methods such as DBSCAN are used to identify outliers in the data.

[0041] Continuity judgment focuses on the continuous change pattern of data in time series rather than the anomaly of a single data point. It helps to identify long-term trends or cyclical changes. The sliding window method can be combined with trend analysis or seasonal analysis to evaluate whether the change of data over a period of time is reasonable, so as to determine whether there is anomaly.

[0042] Multiple related signals are jointly judged. In intelligent driving systems, multiple sensors and data sources work simultaneously and generate multiple related signals. The relationship between these signals is crucial for accurately identifying abnormal events.

[0043] The joint judgment method can analyze the conditional dependency between different signals by constructing statistical models such as Bayesian networks and hidden Markov models to achieve joint judgment of multiple signals.

[0044] By combining the above methods, we can design an event recognition model based on theoretical and empirical rules, which can comprehensively evaluate the data from different angles and identify abnormal events.

[0045] The intelligent driving abnormal event judgment method provided by the embodiment of the present invention adopts a time series anomaly detection strategy, combines signal mutation, signal anomaly, continuity judgment, and a method of joint judgment of multiple related signals, and can design an event recognition model based on theoretical and empirical rules. The event recognition model can accurately identify abnormal events.

[0046] In some embodiments of the present invention, extracting the second video data and the second driving system data associated with the abnormal event corresponding to the time interval includes: extracting the video data and driving system data within the time interval from a vehicle storage device; The video data and the driving system data within the time interval are time stamp synchronized to obtain the second video data and the second driving system data.

[0047] According to the time interval when the abnormal event occurred, the video file data and the internal data of the intelligent driving system within the time range are extracted from the vehicle storage device. There are differences in the internal data information associated with various abnormal events. The relevant data is extracted and the timestamp is synchronized.

[0048] For example, in the event of autonomous driving takeover, the autonomous driving control system and the driving actions of the driver and passengers before and after the takeover are accurately analyzed, and information such as cruise speed, throttle, gear, brake, turn signal, steering wheel angle and obstacles ahead are extracted. Data from multiple modules such as perception, control, and positioning require time synchronization and time series alignment.

[0049] Exemplarily, after the time interval in which the abnormal event occurs is determined, the video data and driving system data associated with the time interval are first extracted from the original data stored in the cloud or other storage devices.

[0050] For example, all video data within the time interval can be extracted from the storage medium of the vehicle camera, that is, the video record of the vehicle's surrounding environment, which can include video clips from multiple cameras. Driving system data includes GPS records, vehicle sensor data (such as speed, acceleration, etc.), system logs, steering wheel angle, brake status, etc.

[0051] Video data is usually collected by multiple cameras, each of which may record video frames with different timestamps. In the extracted video data, timestamp synchronization is required to ensure that the video frames of different cameras can be aligned in time, so that the perspective and scene when the event occurred can be analyzed more accurately.

[0052] Driving system data may also come from different sensors or system records, and each system may record data with different timestamps. In the extracted driving system data, timestamp synchronization is also required to ensure that the records of each data source are consistent in time, so that the video data and driving system data can be effectively associated for comprehensive analysis.

[0053] After the timestamp synchronization processing, the obtained video data and driving system data are the second video data and the second driving system data, which can be directly used for subsequent abnormal event recognition, analysis and judgment.

[0054] The intelligent driving abnormal event judgment method provided by an embodiment of the present invention obtains second video data and second driving system data by synchronizing the timestamps of video data and driving system data within a time interval, thereby ensuring that data extracted from multiple data sources can be aligned and synchronized in time, thereby ensuring the consistency and accuracy of the data, so that the subsequent intelligent analysis system can accurately understand and evaluate the occurrence of abnormal events and related situations.

[0055] In some embodiments of the present invention, judging the recognition accuracy of the abnormal event based on the second video data and the second driving system data includes: Matching and fusing the second video data and the second driving system data to obtain fused data; Based on the fused data, the recognition accuracy of the abnormal event is determined.

[0056] In some embodiments of the present invention, judging the recognition accuracy of the abnormal event based on the fused data includes: Based on the fused data, displaying a change trend of the second driving system data in the second video data; Based on the change trend of the second driving system data, the recognition accuracy of the abnormal event is judged.

[0057] The cloud integrates and matches the abnormal event related signals and video information to determine the accuracy of abnormal event recognition. The extracted related data and video are integrated and matched, and a visual display of the data change process is added to the video. The accuracy of abnormal event recognition is manually checked based on the video information and data changes.

[0058] The cloud realizes the matching of abnormal event-related data and related video timestamps, uses ECharts and other visual chart libraries to animate the data jump status, embeds video images, displays data and video synchronously, and manually verifies the accuracy of event recognition.

[0059] For example, data matching fusion usually uses timestamps as a common reference to ensure that video data and driving system data are consistent in time. By aligning timestamps, it can be ensured that both reflect the situation at the same time point. Fusion methods can include time series alignment, that is, directly associating video frames at the same time point with system data. Fusion methods can also include Kalman filters or deep learning models, which can better handle noise and uncertainty.

[0060] Displaying the changing trend of the second driving system data in the second video data usually requires the use of data visualization technology. For example, a real-time curve graph or a numerical indicator of parameters such as speed and acceleration can be superimposed on the video playback interface.

[0061] Synchronous display helps users or analysts visually see how changes in system data are related to events in the video. For example, when a video shows a vehicle braking suddenly, the speed and deceleration curves in the system data will also show this sudden change.

[0062] Based on the fused data, especially the changing trend of the system data, the accuracy of the recognition of abnormal events can be judged more accurately. For example, if the system data shows that the automatic driving state changes from "normal driving" to "emergency braking" while the vehicle is decelerating, which is consistent with the situation in the video data, it can be confirmed that the recognition of abnormal events is accurate.

[0063] The method for judging abnormal events in intelligent driving provided by the embodiment of the present invention takes advantage of the cloud. The vehicle side can upload information related to the triggering event as completely as possible, perform accurate identification in the cloud, and extract related information. The related information can be dynamically configured to assist in troubleshooting and locating the root cause of the abnormal event. At the same time, the extracted accurate video information is used to grasp the working conditions of the vehicle, environment, etc. when the abnormal event occurs. After synchronous processing with the cloud timestamp, the data and video information are integrated to quickly judge the triggering event of intelligent driving, ensure that suspected problems are not missed, re-judge and eliminate non-abnormal problems through the fusion information, re-confirm abnormal problems, quickly and accurately match related information, assist in problem positioning, and improve problem solving efficiency.

[0064] The present invention is aimed at the intelligent driving high-speed trunk logistics scenario of commercial vehicles, and designs an intelligent driving trigger event judgment method based on cloud services. The vehicle side adopts a single signal design to simplify the logic to identify possible abnormal events, and uploads the data and video of a certain period of time before and after the abnormal event to the cloud. The cloud accurately identifies abnormal events through empirical models or big data model methods. At the same time, the cloud service is combined to obtain relevant video and environmental information data of the abnormal event segment to assist in judging the accuracy of the event.

[0065] The intelligent driving trigger event judgment method based on cloud services is mainly divided into the vehicle-side identification of suspected abnormal events and triggering the upload of relevant data, the cloud-side accurate identification of abnormal events, the cloud-side extraction of abnormal event video image information, the cloud-side extraction of abnormal event related signals, and the cloud-side fusion and matching of abnormal event related signals and video information to determine the accuracy of abnormal event recognition.

[0066] Figure 2 The second method flow chart of an embodiment of the method for determining abnormal events in intelligent driving provided by the present invention is as follows: Figure 2 As shown, the intelligent driving abnormal event judgment method includes: S201. The vehicle side identifies suspected abnormal events and triggers uploading of relevant data.

[0067] The vehicle-side ADECU detects the internal data of the intelligent driving system in real time, focusing on monitoring key signals such as autonomous driving status, autonomous driving degradation status, autonomous driving lateral error, autonomous driving deceleration, autonomous driving braking, and lane line confidence.

[0068] Record the time point T when the key signal jumps, and trigger the Internet of Vehicles communication terminal to upload system data and video for a period of time [TM, T+N] before and after the jump time point.

[0069] S202. Accurately identify abnormal events in the cloud.

[0070] The cloud accesses the data and videos uploaded by the Internet of Vehicles communication terminal for data storage and video file storage. The video file storage adopts the terminal + timestamp naming rule and designs an event recognition model based on complex logic.

[0071] A time series anomaly detection strategy can be adopted to design an event recognition model based on theoretical and empirical rules according to the methods of signal mutation, signal anomaly, continuity judgment, and joint judgment of multiple related signals.

[0072] Accurately identify abnormal events in the intelligent driving system through offline batch processing. Batch process event data with a frequency of 100ms, input into the event recognition model, and accurately identify the time intervals [T1, T2] when various abnormal events occur.

[0073] S203: Extract abnormal event related information in the cloud.

[0074] According to the time interval when the abnormal event occurred, the video files and internal data of the intelligent driving system within the time range are extracted. There are differences in the internal data information associated with various abnormal events. The relevant data is extracted and the timestamps are synchronized.

[0075] For example, in the event of an autonomous driving takeover, the autonomous driving control system and the driving actions of the driver and passengers before and after the takeover are accurately analyzed, and information such as cruise speed, throttle, gear, brake, turn signal, steering wheel angle and obstacles ahead are extracted. Data from multiple modules such as perception, control, and positioning require time synchronization and time series alignment.

[0076] S204: The cloud merges and matches abnormal event related signals and video information to determine the accuracy of abnormal event recognition.

[0077] The extracted relevant data and videos are fused and matched, a visualization of the data change process is added to the video, and the accuracy of abnormal event recognition is manually verified based on the video information and data changes.

[0078] The cloud realizes the matching of abnormal event-related data and related video timestamps, uses ECharts and other visual chart libraries to animate the data jump status, embeds video images, displays data and video synchronously, and manually verifies the accuracy of event recognition.

[0079] The method for judging abnormal events in intelligent driving provided by the embodiment of the present invention takes advantage of the cloud. The vehicle side can upload information related to the triggering event as completely as possible, perform accurate identification in the cloud, and extract related information. The related information can be dynamically configured to assist in troubleshooting and locating the root cause of the abnormal event. At the same time, the extracted accurate video information is used to grasp the working conditions of the vehicle, environment, etc. when the abnormal event occurs. After synchronous processing with the cloud timestamp, the data and video information are integrated to quickly judge the triggering event of intelligent driving, ensure that suspected problems are not missed, re-judge and eliminate non-abnormal problems through the fusion information, re-confirm abnormal problems, quickly and accurately match related information, assist in problem positioning, and improve problem solving efficiency.

[0080] The following describes in detail the intelligent driving abnormal event judgment method provided by the present invention in combination with specific application scenarios: S1. The vehicle side identifies suspected abnormal events.

[0081] Suspected abnormal events are identified based on key signals such as autonomous driving status, autonomous driving degradation status, autonomous driving lateral error, autonomous driving deceleration, autonomous driving braking, lane line confidence, etc. The identification results are notified to the Internet of Vehicles communication terminal through message information to report real-time data. The identification logic example is as follows: (a) Autonomous driving status.

[0082] Autonomous driving status definition: 0x0: not displayed; 0x1: The unavailable status indicator is always on, and the color is temporarily gray; 0x2: The standby status light is always on, and the color is temporarily set to white; 0x3: The standby status light flashes 0.5s on or 0.5s off; 0x4: The activation status light is always on, and the color is temporarily green; 0x5: The activation status light flashes, 0.5s on or 0.5s off; 0x6: The fault status indicator is always on, and the color is temporarily red; 0x7: The fault status light flashes, 0.5s on or 0.5s off; 0x8~0xF: Reserved.

[0083] When the autonomous driving status changes from 4 to 1, it can be judged that an abnormal autonomous driving takeover is suspected to have occurred. At this time, the Internet of Vehicles communication terminal is notified through message information to report data.

[0084] (b) Autonomous driving degraded state.

[0085] Definition of autonomous driving degradation status: Level_A_Event A downgrade; Level_B_Event B downgraded; Level_C_Event C downgrade; Level_D_Event D downgraded.

[0086] A value of 0 indicates no downgrade, and a value of 1 indicates downgrade.

[0087] When it is detected that the A\B\C\D degradation signal appears to be 1, it can be judged that an autonomous driving degradation event is suspected to have occurred. At this time, the Internet of Vehicles communication terminal is notified through message information to report data.

[0088] S2. Accurately identify abnormal events in the cloud.

[0089] The data reported by the vehicle is obtained for rule-based event recognition model analysis, and the abnormal event recognition results are output, including the abnormal event type, the start time of the event, the end time of the event, and other additional information of the event. The recognition logic example is as follows: The following precise identification is performed for suspected autonomous driving takeover events reported by the vehicle: S2.1. Determine whether the lever is actively taking over the lane change. The judgment logic is as follows: S2.1.1. Obtain data reported on suspected autonomous driving takeover events; S2.1.2, find the data frame T1 whose automatic driving state is 4 in the previous frame and changes to 1 in the next frame; S2.1.3. Extract the turn signal related data 5 seconds after time T1; S2.1.4. If there is a data jump in the turn signal data, indicating that the left turn signal or the right turn signal is turned on, the takeover is judged to be an active takeover of the lane change lever, and the time interval of the event is recorded as [T1, T1+5].

[0090] S2.2, determine whether the following vehicle has not stopped and taken over, the judgment logic is as follows: S2.2.1. Obtain data reported on suspected autonomous driving takeover events; S2.2.2. Find the time interval [T1, T2] when the vehicle is in the automatic driving state and the relative speed of the obstacle in front is 0; S2.2.3, determine whether the vehicle has deceleration within [T1, T2]. If so, jump to step S2.2.4. If not, continue to determine the next time interval that satisfies step S2.2.2; S2.2.4. Extract the automatic driving status information 10 seconds after time T2; S2.2.5. If the automatic driving status changes from 4 to 1, the takeover is judged as a takeover without braking from the following vehicle, and the time interval of the event is recorded as [T1, T2+10].

[0091] S3. Extract abnormal event related information in the cloud.

[0092] The cloud confirms the associated data that needs to be extracted by configuring the associated information of abnormal events. The configuration can be added, modified, and used flexibly. Different events can independently configure multiple different associated signals. The back-end service intercepts the corresponding associated data based on the configured signals and the time interval of the identified event. When the source of the associated data is inconsistent, the back-end service synchronizes the timestamp and performs structured processing on the associated data. At the same time, according to the time interval of the identified event, the video file is matched. The video file name is the timestamp, and the video contained in the time range of the event is extracted.

[0093] S4. Cloud-based fusion and matching of abnormal event-related signals and video information.

[0094] The cloud-based visualization interface displays the video and data associated with the event. The timestamp of the data matches the frame rate of the video. The visualization interface displays the video and data simultaneously and supports synchronous dynamic playback. Professionals re-check the accurately identified events to confirm the accuracy of event identification.

[0095] The intelligent driving abnormal event judgment method provided by the embodiment of the present invention simplifies the complex process of abnormal event recognition on the vehicle side, reduces the local computing pressure on the vehicle side, preliminarily screens out suspected abnormal problem points through simple logic design, and reports relevant information in combination with the wireless communication network, thereby improving the strength of abnormal problem recognition without wasting vehicle-side resources. By utilizing cloud optimization, a refined recognition logic is constructed to accurately locate abnormal events, and a visual interface is developed in combination with rich data-level video resources on the cloud side to fuse and match the video and data for display, effectively integrating multi-dimensional information, accurately judging the accuracy of abnormal event recognition, and preventing missed recognition while reducing misrecognition.

[0096] In order to better implement the intelligent driving abnormal event judgment method in the embodiment of the present invention, based on the intelligent driving abnormal event judgment method, the embodiment of the present invention also provides an intelligent driving abnormal event judgment device. Figure 3 A schematic diagram of the structure of an embodiment of the intelligent driving abnormal event judgment device provided by the present invention is shown as follows Figure 3 As shown, the intelligent driving abnormal event judgment device 300 includes: The receiving module 310 is used to receive target data sent by the vehicle end; the target data includes the first video data and the first driving system data associated with the suspected abnormal event; A detection module 320, configured to detect the target data based on a time series anomaly detection strategy to determine a time interval in which an abnormal event occurs; An extraction module 330, configured to extract second video data and second driving system data associated with the abnormal event corresponding to the time interval; The judgment module 340 is used to judge the recognition accuracy of the abnormal event based on the second video data and the second driving system data.

[0097] The intelligent driving abnormal event judgment device 300 provided in the above embodiment can implement the technical solution described in the above intelligent driving abnormal event judgment method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the intelligent driving abnormal event judgment method embodiment, which will not be repeated here.

[0098] like Figure 4 As shown, the present invention also provides an electronic device 400. The electronic device 400 includes a processor 401, a memory 402 and a display 403. Figure 4 Only some components of the electronic device 400 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0099] In some embodiments, the processor 401 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 402, such as the intelligent driving abnormal event judgment method of the present invention.

[0100] In some embodiments, processor 401 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, processor 401 may be local or remote. In some embodiments, processor 401 may be implemented in a cloud platform. In some embodiments, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0101] In some embodiments, the memory 402 may be an internal storage unit of the electronic device 400, such as a hard disk or memory of the electronic device 400. In other embodiments, the memory 402 may also be an external storage device of the electronic device 400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 400.

[0102] Furthermore, the memory 402 may include both an internal storage unit of the electronic device 400 and an external storage device. The memory 402 is used to store application software installed in the electronic device 400 and various data.

[0103] In some embodiments, the display 403 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an organic light-emitting diode (OLED) touch device, etc. The display 403 is used to display information on the electronic device 400 and to display a visual user interface. The components 401-403 of the electronic device 400 communicate with each other via a system bus.

[0104] In one embodiment, when the processor 401 executes the intelligent driving abnormal event judgment program in the memory 402, the following steps may be implemented: Receiving target data sent by the vehicle end; the target data includes first video data and first driving system data associated with a suspected abnormal event; Detect the target data based on the time series anomaly detection strategy to determine the time interval when the abnormal event occurred; Extracting second video data and second driving system data associated with the abnormal event corresponding to the time interval; Based on the second video data and the second driving system data, the recognition accuracy of the abnormal event is determined.

[0105] It should be understood that: when the processor 401 executes the intelligent driving abnormal event judgment program in the memory 402, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0106] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 400 mentioned, and the electronic device 400 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic device may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 400 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0107] Correspondingly, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions in the intelligent driving abnormal event judgment method provided by the above-mentioned method embodiments can be implemented.

[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps or functions in the intelligent driving abnormal event judgment method provided in the above-mentioned method embodiments.

[0109] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0110] The above is a detailed introduction to the intelligent driving abnormal event judgment method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for judging abnormal events in intelligent driving, characterized in that: include: Receive target data sent by the vehicle; The target data includes first video data and first driving system data associated with a suspected abnormal event; Detect the target data based on the time series anomaly detection strategy to determine the time interval when the abnormal event occurred; Extracting second video data and second driving system data associated with the abnormal event corresponding to the time interval; Based on the second video data and the second driving system data, the recognition accuracy of the abnormal event is determined.

2. The intelligent driving abnormal event judgment method according to claim 1 is characterized in that: The suspected abnormal event is determined when the vehicle side identifies a jump in the key signal; The key signal includes at least one of the following: Autopilot status, autopilot degradation status, autopilot lateral error, autopilot deceleration, autopilot braking, and lane line confidence.

3. The intelligent driving abnormal event judgment method according to claim 2 is characterized in that: The detecting the target data based on the time series anomaly detection strategy to determine the time interval of the abnormal event includes: Based on the time series anomaly detection strategy, an event recognition model is constructed; Based on the event recognition model, determining the time interval when the abnormal event occurs; The time series anomaly detection strategy includes at least one of the following: Signal mutation detection, signal anomaly detection, continuity judgment and joint judgment of multiple related signals.

4. The intelligent driving abnormal event judgment method according to claim 3 is characterized in that: The extracting the second video data and the second driving system data associated with the abnormal event corresponding to the time interval includes: extracting the video data and driving system data within the time interval from a vehicle storage device; The video data and the driving system data within the time interval are time stamp synchronized to obtain the second video data and the second driving system data.

5. The intelligent driving abnormal event judgment method according to claim 4 is characterized in that: The determining, based on the second video data and the second driving system data, the recognition accuracy of the abnormal event includes: Matching and fusing the second video data and the second driving system data to obtain fused data; Based on the fused data, the recognition accuracy of the abnormal event is determined.

6. The intelligent driving abnormal event judgment method according to claim 5 is characterized in that: The determining, based on the fused data, the recognition accuracy of the abnormal event includes: Based on the fused data, displaying a change trend of the second driving system data in the second video data; Based on the change trend of the second driving system data, the recognition accuracy of the abnormal event is judged.

7. An intelligent driving abnormal event judgment device, characterized in that: include: A receiving module, used for receiving target data sent by the vehicle end; The target data includes first video data and first driving system data associated with a suspected abnormal event; A detection module, used to detect the target data based on a time series anomaly detection strategy to determine the time interval when the abnormal event occurs; An extraction module, used for extracting second video data and second driving system data associated with the abnormal event corresponding to the time interval; A judgment module is used to judge the recognition accuracy of the abnormal event based on the second video data and the second driving system data.

8. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the intelligent driving abnormal event judgment method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent driving abnormal event judgment method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the intelligent driving abnormal event judgment method as described in any one of claims 1 to 6 is implemented.

Images