Log and video mutual positioning fault analysis method, system and product

Abstract

The embodiment of the invention provides a log and video mutual positioning fault analysis method, system and product, and the method comprises the steps: carrying out the recognition processing of various target video files and various target log files of a vehicle through an abnormal event recognition model, determining an abnormal video frame and a corresponding abnormal event type in the target video file, and an abnormal log line and a corresponding abnormal event type in the target log file; creating an abnormal event mapping relation corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type; and based on the abnormal event mapping relation, performing fault analysis on the vehicle to obtain a corresponding fault analysis result. The objective of the invention is to improve the fault fact restoration effect in fault analysis.

Description

Technical Field

[0001] The present application relates to the technical field of automobile fault analysis, and in particular to a fault analysis method, system and product for mutual positioning of logs and videos. Background Art

[0002] Log files are data that record the order and time relationship of events, and they play an important role in fault analysis. The specific fault analysis method is to restore the fault facts by analyzing the abnormal log lines in the log files. However, this kind of fault analysis based only on log files makes the restoration of fault facts poor. How to improve the restoration of fault facts in fault analysis is an urgent problem to be solved. Summary of the invention

[0003] In view of this, the present application provides a fault analysis method, system and product for mutual positioning of logs and videos, aiming to improve the effect of restoring fault facts in fault analysis.

[0004] The first aspect of the present application provides a fault analysis method for mutual positioning of logs and videos, the method comprising:

[0005] Identify and process various target video files and various target log files of the vehicle through the abnormal event recognition model, and determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files;

[0006] Creating an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type;

[0007] Based on the abnormal event mapping relationship, a fault analysis is performed on the vehicle to obtain a corresponding fault analysis result.

[0008] Optionally, various target video files and various target log files of the vehicle are identified and processed by an abnormal event identification model to determine abnormal video frames and corresponding abnormal event types in the target video files, and abnormal log lines and corresponding abnormal event types in the target log files, including:

[0009] Input various target video files and various target log files of the vehicle into the abnormal event recognition model, and extract the corresponding target video features and target log features;

[0010] By performing recognition processing on the target video features and the target log features, abnormal video frames and abnormal log lines therein are determined;

[0011] Performing model reasoning on the determined abnormal video frame and the abnormal log line to determine reasoning results corresponding to the abnormal video frame and the abnormal log line respectively;

[0012] According to all the inference results, the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line are determined.

[0013] Optionally, based on all the inference results, the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line are determined, including:

[0014] Determine the probability of the abnormal event type recorded in the target reasoning result;

[0015] When the probability is lower than a set threshold, obtaining other reasoning results under the occurrence period according to the occurrence period corresponding to the target reasoning result;

[0016] When the probability of the abnormal event type recorded in the other inference results is greater than or equal to a set threshold, the abnormal event type in the target inference result is updated with the abnormal event type recorded in the other inference results.

[0017] Optionally, the method further includes:

[0018] According to the abnormal event mapping relationship, a corresponding visualization interface is created to visualize the abnormal event mapping relationship;

[0019] Determining a target operation on a row of visualized data in the visualization interface;

[0020] When the target operation is to view details of a row of visual data, jump to the abnormal video frame and abnormal log row corresponding to the viewed visual data;

[0021] When the target operation is to reject the visualized data row, the corresponding rejection operation is recorded.

[0022] Optionally, the method further includes:

[0023] Determine the performance of the abnormal event recognition model based on the number of rejection operations on the visualized data rows;

[0024] According to the performance, the abnormal event recognition model is trained a second time to obtain a new abnormal event recognition model.

[0025] Optionally, when the abnormal event recognition model is deployed in the cloud, the method further includes:

[0026] Obtain various video files and log files of the vehicle;

[0027] Compressing the acquired video files and log files to obtain target video files and target log files;

[0028] Upload various target video files and various target log files to the cloud.

[0029] Optionally, when the video file is a display screen video file, obtaining the vehicle video file includes:

[0030] Monitor the video signal of the vehicle screen through the vehicle screen monitoring tool of the vehicle system;

[0031] Based on the video signal of the monitored vehicle computer screen, the video signal is persisted as a video file through a screen mirroring tool, and the persisted video file is obtained.

[0032] Optionally, compressing the acquired video file and log file to obtain a target video file and a target log file includes:

[0033] Determine the file type of the acquired data file;

[0034] When the file type is a video, compressing the video file by using a compression algorithm corresponding to the video file type to obtain a target video file;

[0035] In the case where the file type is a log, the log file is compressed using a compression algorithm corresponding to the log file type to obtain a target log file.

[0036] Optionally, the method further includes:

[0037] The online status of file acquisition, compression and upload processes is monitored in real time through the watchdog program;

[0038] When online, send a message to report the heartbeat to the cloud through the target protocol;

[0039] Restart the fetch, compress and upload process while offline.

[0040] A second aspect of the present application provides a fault analysis system for mutual positioning of logs and videos, the system comprising:

[0041] An abnormality determination module is used to identify and process various target video files and various target log files of the vehicle through an abnormal event recognition model, and determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files;

[0042] A mapping relationship creation module, used to create an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type;

[0043] The fault analysis module is used to perform fault analysis on the vehicle based on the abnormal event mapping relationship to obtain corresponding fault analysis results.

[0044] The third aspect of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps in the fault analysis method for mutual positioning of logs and videos as described in the first aspect of the present application are implemented.

[0045] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the fault analysis method for mutual positioning of logs and videos as described in the first aspect of the present application are implemented.

[0046] The fault analysis method for mutual positioning of logs and videos provided in this application has the following advantages:

[0047] The embodiment of the present application provides a fault analysis method for mutual positioning of logs and videos. First, various target video files and various target log files of the vehicle are identified and processed through an abnormal event recognition model to determine the abnormal video frames and corresponding abnormal event types in the target video files, as well as the abnormal log lines and corresponding abnormal event types in the target log files; according to the identification information of the vehicle, the abnormal video frames and the corresponding abnormal event types, and the abnormal log lines and the corresponding abnormal event types, an abnormal event mapping relationship corresponding to the identification information of the vehicle is created; based on the abnormal event mapping relationship, the vehicle is analyzed for faults to obtain corresponding fault analysis results. Therefore, when performing fault analysis, the present application not only considers various log files, but also considers various video files collected by the vehicle. By establishing a mapping relationship between video files and log files of abnormal events that occur in the same period and the same type, fault analysis can be performed based on the video files and log files with the mapping relationship at the same time, thereby effectively improving the restoration effect of the fault facts. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A flowchart of a fault analysis method for mutual positioning of logs and videos is shown as an embodiment of the present application;

[0050] Figure 2 A schematic diagram of mutual positioning of abnormal video frames and abnormal log lines in a fault analysis method for mutual positioning of logs and videos shown in one embodiment of the present application;

[0051] Figure 3 Another flow chart of a fault analysis method for mutual positioning of logs and videos shown in one embodiment of the present application;

[0052] Figure 4 A schematic diagram of a fault analysis system for mutual positioning of logs and videos is shown as an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] refer to Figure 1 , Figure 1 This is a flow chart of a fault analysis method for mutual positioning of logs and videos shown in one embodiment of the present application. Figure 1 As shown, the method includes:

[0055] Step S1: various target video files and various target log files of the vehicle are identified and processed by an abnormal event identification model to determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files.

[0056] In this embodiment, the vehicle will be equipped with a large number of monitoring sensors to monitor environmental data, and when each monitoring sensor monitors and obtains the corresponding environmental data, a corresponding log file will be generated to record whether there are abnormalities in the monitoring process and whether there are abnormalities in the environmental data obtained by monitoring. Therefore, there is a corresponding target log file for each monitoring sensor. At the same time, the vehicle will also collect and obtain a large amount of video data to generate video files, including video files generated by monitoring the video data obtained around the vehicle body.

[0057] Since the cockpit display screen is an extremely important component of current smart cars, it includes almost all the control components that directly interact with the user except the steering wheel, accelerator, and brake. If the operation interface it receives and responds to the user is incorrect, it may cause a major risk event. Therefore, in order to better restore the fault facts, this application also saves the video formed by the video signal generated by the vehicle display screen, so as to more completely restore the fault facts. Therefore, the collected vehicle video file also includes a video file generated by the interface display video signal based on the vehicle display screen. The video data obtained for monitoring the surrounding of the vehicle body is collected by each surround camera around the vehicle body. For each surround camera, there is a corresponding target video file. The video file generated by the interface display video signal of the vehicle display screen is also a target video file. By inputting various target video files and various target log files of the vehicle into the abnormal event recognition model for recognition processing, the abnormal video frames in various target video files and the abnormal event types corresponding to each abnormal video frame can be determined, and the abnormal log lines in various target log files and the abnormal event types corresponding to each abnormal log line can be determined. Since when there are abnormal video frames corresponding to abnormal events in the target video file, the possibility of only one abnormal video frame is low, and it is more likely to be abnormal video frames of a certain length. Therefore, the above description is the abnormal event type corresponding to each abnormal video frame. Similarly, when there are abnormal log lines corresponding to abnormal events in the target log file, the possibility of only one log line is low, and it is more likely to be multiple consecutive abnormal log lines. Therefore, the above description is the abnormal event type corresponding to each abnormal log line. Among them, the vehicle's log files, in addition to the log files corresponding to the sensors, also include other types of log files, including but not limited to collision warning log files, emergency light flashing log files, emergency braking log files, speed warning log files, vehicle system log files, memory monitoring log files, temperature overheating warning log files, etc.

[0058] In this embodiment, various sensors include but are not limited to laser radar, microwave radar, wheel speed sensor, positioning sensor, collision sensor, etc. Laser radar is used to obtain laser radar data. The acquisition process is that during the driving of the vehicle, the laser radar configured on the vehicle sends a laser pulse signal and is reflected back by surrounding objects. The radar system receives the reflected laser pulse signal and calculates the distance data based on the round-trip time of the laser pulse signal. At the same time, a model is established based on the data to form a three-dimensional model of the vehicle's surrounding environment. Microwave radar is used to obtain microwave radar data. The acquisition process is that during the driving of the vehicle, the microwave radar configured on the vehicle transmits a microwave signal and receives a reflected signal. According to the time difference and phase difference between the transmitted microwave signal and the received reflected signal, the distance, speed and direction of the objects around the vehicle are calculated. Wheel speed sensor is used to obtain vehicle driving speed data. The acquisition process is that during the driving of the vehicle, the transmitter in the wheel speed sensor configured on the vehicle sends a magnetic field or electromagnetic wave to the wheel. When the wheel rotates, the magnetic field or electromagnetic wave will be modulated by the gears or pores on the wheel to generate a change signal. The receiver in the wheel speed sensor detects the change signal and converts it into an electrical signal, thereby calculating the rotation speed of the wheel. The driving speed of the vehicle is calculated based on the rotation speed and the circumference of the wheel. The positioning sensor is used to obtain the positioning data and moving speed data of the vehicle. The acquisition process is that during the driving process of the vehicle, the positioning sensor (i.e., the global positioning system GPS receiver) configured on the vehicle receives the precise timestamp and position signal sent by the satellite, compares the time difference of the position signal received from at least four satellites, and uses triangulation to calculate the precise latitude, longitude and altitude of the vehicle on the earth to obtain the precise positioning data of the vehicle; based on the positioning data of the vehicle and the position change of the vehicle per unit time, the moving speed of the vehicle is calculated. The collision sensor is used to obtain the collision data of the vehicle. The acquisition process is that when the car collides, the sensitive elements inside the collision sensor configured on the vehicle are impacted and vibrated, and the physical changes such as impact and vibration are converted into electrical signal output to accurately capture the collision state of the vehicle. The surround view camera is used to obtain the panoramic monitoring image data of the vehicle. The acquisition process is that the surround view camera configured on the vehicle converts the light signal around the vehicle into a video signal, and the image processing unit processes and splices the video signals from each surround view camera to form seamless panoramic monitoring image data. The video file generated by the interface display video signal of the vehicle display screen will reflect the user's behavioral intention data in the display screen. The display screen sensor configured in the vehicle converts the user's touch position and pressure physical changes into electrical signals, senses the user's interaction with the vehicle, and determines the user's behavioral intention based on the interaction. The behavioral intention will be reflected in the video file generated based on the interface display video signal.Various sensors are constantly collecting signals, and the electrical signals output by the sensors are usually weak. The present application amplifies the signals through a signal amplifier, and the signal processing circuit filters, shapes and digitizes the amplified signals to ensure the stability and accuracy of the signals.

[0059] In this embodiment, the data information collected by the sensor will not only have corresponding log files generated and stored in the vehicle storage system, but also the collected data information will be sent to the vehicle control unit through wires or wirelessly. The specific process is that the signal processed by the signal amplifier and the signal processing circuit is transmitted to the cockpit electronic control unit (CECU Electronic Control Unit for Cockpit) through the internal communication network of the vehicle (such as CAN bus, LIN bus, MOST bus, etc.). The cockpit electronic control unit will store the collected data information in the vehicle's storage system for subsequent fault diagnosis and historical data analysis. At the same time, it will conduct comprehensive analysis and decision-making on various data information, process them according to preset algorithms and logic, and control the vehicle to perform corresponding control actions based on the processing results. The above complex logical processing and computing capabilities require the support of an intelligent vehicle system, which is composed of one or more chips and operating system programs. It completes the comprehensive analysis and calculation of various signal information through drivers and system programs, as well as upper-level applications.

[0060] Step S2: creating an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type.

[0061] In this embodiment, based on the identification information of the vehicle, the abnormal event types corresponding to the determined abnormal video frames and each segment of the abnormal video frames, and the abnormal event types corresponding to the determined abnormal log lines and each segment of the abnormal log lines, a corresponding abnormal event mapping relationship is created for abnormal video frames and abnormal log lines with the same abnormal event type and partial or complete overlap in their respective occurrence times, and the abnormal event mapping relationship records the identification information of the vehicle, the abnormal event type, the abnormal video frames, and the abnormal log lines. Among them, the abnormal event types include but are not limited to collision, scratch, breakdown, vehicle computer blur, stuck, black screen, vehicle computer restart, etc.

[0062] For example, based on the identification information of the vehicle, the determined abnormal video frames and the abnormal event types corresponding to each abnormal video frame, the determined abnormal log lines and the abnormal event types corresponding to each abnormal log line, it is determined that there is an abnormal log line corresponding to a collision sensor with a length of 10 to 20 lines, and the corresponding abnormal event type is a collision. The event corresponding to this abnormal log line is from 10:10:10 to 10:10:20 on a certain day. At the same time, it is determined that there is an abnormal video frame corresponding to a front camera with a length of 200 to 800 frames, and the corresponding abnormal event type is a collision. The occurrence time corresponding to this abnormal video frame is also on that certain day, and the specific time is 10 :10:10 to 10:10:40, since the abnormal event types corresponding to the abnormal log line and the abnormal video frame are the same, and there is a partial overlap in their respective occurrence times (the partially overlapping time is 10:10:10 to 10:10:20), an abnormal event mapping relationship between the abnormal log line and the abnormal video frame is established, and the abnormal event mapping relationship will record the identification information of the vehicle, the determined same collision abnormal event type and the abnormal log line and the abnormal video frame mapped to each other, as well as the longest occurrence time of each abnormal log line and each abnormal video frame mapped to each other. As shown in Table 1 below, Table 1 shows different abnormal event mapping relationships of multiple vehicles.

[0063] Table 1

[0064]

[0065]

[0066] Step S3: Based on the abnormal event mapping relationship, perform fault analysis on the vehicle to obtain corresponding fault analysis results.

[0067] In this embodiment, a vehicle fault analysis is performed based on the created abnormal event mapping relationship of the vehicle to obtain a corresponding fault analysis result. In the process of performing fault analysis on the vehicle based on the created abnormal event mapping relationship, not only various log files are considered, but also various video files collected by the vehicle are considered, thereby effectively improving the fault fact restoration effect of the fault analysis.

[0068] The embodiment of the present application provides a fault analysis method for mutual positioning of logs and videos. First, various target video files and various target log files of the vehicle are identified and processed through an abnormal event recognition model to determine the abnormal video frames and corresponding abnormal event types in the target video files, as well as the abnormal log lines and corresponding abnormal event types in the target log files; according to the identification information of the vehicle, the abnormal video frames and the corresponding abnormal event types, and the abnormal log lines and the corresponding abnormal event types, an abnormal event mapping relationship corresponding to the identification information of the vehicle is created; based on the abnormal event mapping relationship, the vehicle is analyzed for faults to obtain corresponding fault analysis results. Therefore, when performing fault analysis, the present application not only considers various log files, but also considers various video files collected by the vehicle. By establishing a mapping relationship between video files and log files of abnormal events that occur in the same period and the same type, fault analysis can be performed based on the video files and log files with the mapping relationship at the same time, thereby effectively improving the restoration effect of the fault facts.

[0069] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, step S1 may include:

[0070] Step S11: input various target video files and various target log files of the vehicle into the abnormal event recognition model to extract corresponding target video features and target log features.

[0071] In this embodiment, various target video files and various target log files of the vehicle are input into the abnormal event recognition model after being layered, cleaned and classified. The feature extraction layer of the model extracts the target video features corresponding to each target video file and the target log features corresponding to each target log file.

[0072] In this embodiment, since the gated recurrent unit (GRU) algorithm can effectively control the information flow, and has the advantages of fewer model parameters, high training efficiency, simple structure, etc., and especially in the case of limited resources, it has better performance than other deep learning algorithms, the abnormal event recognition model preferably adopts the gated recurrent unit (GRU) algorithm. It should be understood that the abnormal event recognition model can also be other algorithm models, which are not specifically limited here.

[0073] Step S12: determining abnormal video frames and abnormal log lines by identifying the target video features and the target log features.

[0074] In this embodiment, after obtaining each target video feature corresponding to each type of target video file, and each target log feature corresponding to each type of target log file, the abnormal event recognition model continues to identify and process the obtained target video features and target log features, determine each abnormal video frame in each type of target video file, and determine each abnormal log line in each type of target log file.

[0075] Step S13: performing model inference on the determined abnormal video frame and the abnormal log line to determine the inference results corresponding to the abnormal video frame and the abnormal log line respectively.

[0076] In this embodiment, after determining to obtain each abnormal video frame in each type of target video file and each abnormal log line in each type of target log file, the abnormal event recognition model continues to perform model reasoning on each abnormal video frame and each abnormal log line to determine the reasoning results corresponding to each abnormal video frame and each abnormal log line. Each abnormal video frame has a corresponding reasoning result, and each abnormal log line has a corresponding reasoning result, and each reasoning result records the abnormal event type corresponding to the corresponding abnormal video frame or abnormal log line.

[0077] Step S14: According to all the inference results, the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line are determined.

[0078] In this embodiment, based on all the obtained inference results, the abnormal event type of each abnormal video frame and the abnormal event type of each abnormal log line are determined.

[0079] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, step 14 may include steps S141 to S143:

[0080] Step S141: Determine the probability of the abnormal event type recorded in the target reasoning result.

[0081] In this embodiment, each inference result records the abnormal event type corresponding to a corresponding abnormal video frame or an abnormal log line, and also records the probability of belonging to the abnormal event type. In order to ensure the accuracy of the abnormal event type recorded in the inference result, when the probability of the abnormal event type recorded in the inference result is lower than a preset threshold, the abnormal event type recorded in the inference result is updated, wherein the threshold can be set according to the actual application scenario and is not specifically limited here. Specifically, all inference results are target inference results, and for each target inference result, the probability of the abnormal event type recorded in the target inference result is determined.

[0082] Step S142: when the probability is lower than a set threshold, other inference results in the occurrence time period are obtained according to the occurrence time period corresponding to the target inference result.

[0083] In this embodiment, when the probability of the abnormal event type recorded in the target reasoning result is lower than the set threshold, based on the occurrence time period of an abnormal video frame or the occurrence time period of an abnormal log line corresponding to the target reasoning result, other reasoning results belonging to the occurrence time period are obtained.

[0084] Step S143: When the probability of the abnormal event type recorded in the other reasoning results is greater than or equal to the set threshold, the abnormal event type in the target reasoning result is updated with the abnormal event type recorded in the other reasoning results.

[0085] In this embodiment, based on other reasoning results belonging to the occurrence period obtained in step S142, it is determined whether the probability of the abnormal event type recorded in the other reasoning results is greater than or equal to the set threshold. If it is greater than or equal to the set threshold, the abnormal event type recorded in the target reasoning result is replaced with the abnormal event type recorded in the other reasoning result to update the abnormal event type recorded in the target reasoning result. If it is less than the set threshold, the abnormal event type recorded in the target reasoning result is maintained.

[0086] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, the method further includes:

[0087] Step S4: according to the abnormal event mapping relationship, a corresponding visualization interface is created to visualize the abnormal event mapping relationship.

[0088] In this embodiment, based on the abnormal event mapping relationship obtained, a visualization interface corresponding to the abnormal event mapping relationship is created to visualize the abnormal event mapping relationship. The content of the visualization display is similar to that shown in Table 1. Each row displays the identification information of the vehicle, the occurrence event of the abnormal event, the event type of the abnormal event, the abnormal log line position, and the abnormal video frame position. The vehicle model can also be displayed. The content of the visualization display supports conditional screening based on the vehicle model, event type, etc.

[0089] In this embodiment, the visual display method is to use the client browser web page technology based on the Vue component to realize the online network page.

[0090] Step S5: Determine a target operation for the visualized data row in the visualization interface.

[0091] In this embodiment, the present application also sets corresponding details and rejection functions for each row of data recording the abnormal event mapping relationship. Determine whether the user has performed a corresponding target operation on a specific row of visualization data in the visualization interface, and the target operation includes executing the details function and executing the rejection function.

[0092] Step S6: When the target operation is to view details of a row of visualization data, jump to the abnormal video frame and abnormal log row corresponding to the viewed visualization data.

[0093] In this embodiment, when the target operation performed by the user is to execute the details function, it will jump to the abnormal event details analysis page of the row data, and the details analysis page is divided into two sections on the left and right. All relevant abnormal log lines are arranged from top to bottom on the left side. These abnormal log lines are loaded into the browser in the format of text files for display. The line number is displayed on the left side of each log window, and the drag bar is displayed on the right and bottom to ensure that the log content can be viewed more conveniently and in detail. On the right side of the details analysis page, the abnormal video frames are arranged from top to bottom. These abnormal video frames are loaded in the playback format supported by the browser. There are play and pause buttons and a drag progress bar at the bottom of each video window.

[0094] In this embodiment, the implementation effect of the details analysis page is to mark the corresponding range in each log window in red according to the line number of the abnormal log line, and mark the corresponding range progress bar in each video window in red according to the frame number of the abnormal video frame, and the red area is suspended to mark the abnormal event type. When the user clicks the event label, the log window automatically jumps to the abnormal log line, and the abnormal log line is also highlighted. The video progress bar automatically locates and plays the abnormal video frame segment while highlighting the corresponding video progress bar range. In this way, users can analyze and solve problems on the cloud platform webpage, realizing a fault analysis method in which log files and video files are located in each other's direction. Figure 2 As shown, Figure 2 A schematic diagram showing the mutual positioning of abnormal log lines and abnormal video frames in the same abnormal event type is shown. By clicking on the marked abnormal event type, the abnormal video frame and abnormal log line corresponding to the abnormal event type will be positioned.

[0095] Step S7: When the target operation is to reject the visualized data row, the corresponding rejection operation is recorded.

[0096] In this embodiment, when the target operation performed by the user is to perform a rejection operation, the rejection operation will be recorded, and the user's description of the rejection operation will be recorded at the same time. The rejection operation indicates that there is an error in the model inference result of the model for the row data corresponding to the rejection operation, and the description of the rejection operation is the user's description of the rejection operation. For example, some abnormal video frames in the row data are wrong, but they are not abnormal video frames; some abnormal log lines in the row data are wrong, but they are not abnormal log lines; the abnormal event type in the row data is wrong, and the specific abnormal event is described.

[0097] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, the method further includes:

[0098] Step S8: Determine the performance of the abnormal event recognition model based on the statistical number of rejection operations on the visualized data rows.

[0099] In this embodiment, the performance of the abnormal event recognition model is determined based on the number of rejection operations performed by users on the visual data rows and the descriptions of the rejection operations. In a data set that records a large number of visual data rows, if the number of rejection operations on the visual data rows exceeds the first preset ratio of the data set, it is determined that the performance of the abnormal event recognition model is poor, and the abnormal event recognition model needs to be retrained; or the descriptions of the rejection operations are counted to determine whether the number of descriptions of the same type exceeds the second preset ratio of the data set. If it exceeds, it is also determined that the performance of the abnormal event recognition model is poor, and the abnormal event recognition model needs to be retrained. For example, if the number of abnormal event type errors in the descriptions of all rejection operations exceeds 20% of the data set, it is determined that the performance of the abnormal event recognition model is poor, and the abnormal event recognition model needs to be retrained.

[0100] Step S9: According to the performance, the abnormal event recognition model is trained a second time to obtain a new abnormal event recognition model.

[0101] In this embodiment, when the performance of the abnormal event recognition model determined by step S8 is poor, the abnormal event recognition model is trained a second time to improve the performance of the abnormal event recognition model, thereby obtaining a new abnormal event recognition model with better performance.

[0102] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, when the abnormal event recognition model is deployed in the cloud, the method further includes:

[0103] Step S01: Acquire various video files and various log files of the vehicle.

[0104] In this embodiment, the abnormal event recognition model requires more computing resources for data processing. A preferred implementation is to deploy it in the cloud. It should be understood that the abnormal event recognition model can also be deployed on the vehicle side when the computing resources on the vehicle side are sufficient, and the results obtained from the data processing only need to be uploaded to the cloud later. Another optional implementation is to deploy the abnormal event recognition model in offline stores for fault detection. In the case where the abnormal event recognition model is deployed in the cloud, the present application will obtain relevant data files and upload them to the cloud for data processing. Specifically, various video files collected by the vehicle are obtained, including but not limited to video files corresponding to the environmental information around the vehicle body collected by the vehicle's surround view camera and video files corresponding to the interface video signal of the vehicle's vehicle display screen, as well as various log files of the vehicle. Among them, in the case where the abnormal event recognition model is deployed in the cloud, the data files received in the cloud will first be decompressed and layered, cleaned and classified, and then input into the abnormal event recognition model.

[0105] Step S02: compress the acquired video file and log file to obtain a target video file and a target log file.

[0106] In this embodiment, in order to improve processing efficiency, the present application compresses the various video files and various log files of the acquired vehicle before uploading them to the cloud to improve the transmission effect. By compressing the various video files and various log files acquired, the corresponding target video files and target log files are obtained.

[0107] Step S03: Upload various target video files and various target log files to the cloud.

[0108] In this embodiment, various target video files and various target log files obtained by compression processing are uploaded to the cloud through FTP (File Transfer Protocol), and the vehicle identification information is recorded in various video files and various log files, so that the cloud can know the specific vehicles to which the received target video files and target log files belong, and each vehicle has a unique and determined identification information.

[0109] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, when the video file is a display screen video file, obtaining the video file of the vehicle includes: monitoring the video signal of the vehicle screen through the vehicle screen monitoring tool of the vehicle system; based on the video signal of the monitored vehicle screen, persisting the video signal into a video file through the screen mirroring tool, and obtaining the persisted video file.

[0110] In this embodiment, when the video file is a display screen video file of the vehicle (i.e., a video file that records the display content of the display screen interface and the content of the interface jump during user operation), an optional implementation method for obtaining the video file of the vehicle is to monitor the video signal of the vehicle screen through the vehicle screen monitoring tool of the vehicle system, wherein the vehicle screen monitoring tool under the Android system is the ADB (Android Debug Bridge) tool, and under the iOS system, the vehicle screen detection tool can be the Xcode integrated development environment (IDE) and related command line tools (such as libimobiledevice and other open source projects). Based on the video signal of the monitored vehicle screen, the monitored video signal is persisted as a video file through the screen mirroring tool, and then the persisted video file is obtained.

[0111] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, step S02 may include steps S021 to S023:

[0112] Step S021: Determine the file type of the acquired data file.

[0113] In this embodiment, in order to reduce system resource overhead, the present application adopts different compression methods for different types of data files. Specifically, firstly, the file type of the acquired data file is determined, and the file type includes video files and log files.

[0114] Step S022: When the file type is a video, compress the video file using a compression algorithm corresponding to the video file type to obtain a target video file.

[0115] In this embodiment, when the file type of the determined data file is a video, the video file is compressed using a compression algorithm corresponding to the video file type to obtain a target video file. The compression algorithm corresponding to the video file type includes but is not limited to Adobe Premiere and Shotcut.

[0116] Step S023: When the file type is a log, compress the log file using a compression algorithm corresponding to the log file type to obtain a target log file.

[0117] In this embodiment, when the file type of the determined data file is a log, the log file is compressed using a compression algorithm corresponding to the log file type to obtain a target log file. The compression algorithm corresponding to the log file type includes but is not limited to the DBCC SHRINKFILE command in SQL Server.

[0118] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides a fault analysis method for mutual positioning of logs and videos. In the fault analysis method for mutual positioning of logs and videos, the method further includes: real-time monitoring of the online status of the file acquisition, compression and upload process through a watchdog program; when in the online state, sending a message to report the heartbeat to the cloud through the target protocol; when in the offline state, restarting the acquisition, compression and upload process.

[0119] In this embodiment, in order to ensure that the target video file and target log file of the vehicle can be uploaded to the cloud, the present application deploys a watchdog program in the vehicle system to monitor the online status of the corresponding process in real time. Specifically, the configured watchdog program monitors in real time the online status of the video file and log file acquisition process, the online status of the acquired video file and log file compression process, and the online status of the upload process of the compressed target video file and target log file; when all processes are online, a message is sent to report the heartbeat to the cloud through the target protocol, wherein the target protocol is preferably the TCP (Transmission Control Protocol) protocol. In the case of being offline, the acquisition, compression and upload processes are restarted to resume uploading the target video file and the target log file.

[0120] In this embodiment, if Figure 3 As shown, Figure 3Another example diagram of a fault analysis method for mutual positioning of logs and videos provided by the present application is shown. The fault analysis method for mutual positioning of logs and videos provided by the present application is first of all data collection. For the obtained log files, the vehicle's car intelligent system generates them, and for the obtained video files, the video stream program configured by the vehicle system obtains them. Based on the various target log files and various target video files obtained, the abnormal event recognition model performs feature extraction and abnormal log lines, as well as abnormal video frames. Based on the abnormal log lines and abnormal video frames obtained by recognition, the abnormal event type corresponding to the abnormal log line and the abnormal event type corresponding to the abnormal video frame are determined, and then the mapping relationship between the abnormal event and the abnormal log line and the abnormal video frame is created. Therefore, the fault analysis method for mutual positioning of logs and videos provided by the present application realizes a method for tracking and analyzing the vehicle fault based on the mutual positioning of log files and video files. Based on this method, users can grasp the environmental data when the fault occurs, solving the pain points of incomplete information data collection, slow offline pulling of vehicle data, and inability to completely and truly restore the fault scene. And based on the mutual positioning of log files and video files, users no longer need to compare and analyze log data line by line offline. By clicking on event labels, abnormal log lines, and abnormal video frames, logs and videos can be mutually positioned and analyzed, greatly improving the efficiency of troubleshooting. When users use this method to analyze faults, they sometimes need to hand them over to others for processing. If it is done offline, all relevant information needs to be compressed and packaged and sent to the other party for processing, which consumes time and resources. However, this method only requires sending the system link to the other party, and others can also quickly process and analyze faults on the web interface, improving work efficiency.

[0121] Based on the same inventive concept, an embodiment of the present application provides a fault analysis system for mutual positioning of logs and videos, such as Figure 4 As shown, the system 400 includes:

[0122] The abnormality determination module 401 is used to identify and process various target video files and various target log files of the vehicle through an abnormal event recognition model, and determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files;

[0123] A mapping relationship creation module 402, for creating an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type;

[0124] The fault analysis module 403 is used to perform fault analysis on the vehicle based on the abnormal event mapping relationship to obtain corresponding fault analysis results.

[0125] Optionally, the abnormality determination module 401 includes:

[0126] A feature extraction module is used to input various target video files and various target log files of the vehicle into the abnormal event recognition model to extract corresponding target video features and target log features;

[0127] An identification and processing module, used for identifying and processing the target video features and the target log features to determine abnormal video frames and abnormal log lines therein;

[0128] A model reasoning module, used for performing model reasoning on the determined abnormal video frame and the abnormal log line, and determining the reasoning results corresponding to the abnormal video frame and the abnormal log line respectively;

[0129] The abnormality determination submodule is used to determine the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line according to all the inference results.

[0130] Optional, abnormality determination submodule, including:

[0131] A probability determination module, used to determine the probability of the abnormal event type recorded in the target reasoning result;

[0132] An inference result acquisition module, used to acquire other inference results in the occurrence period according to the occurrence period corresponding to the target inference result when the probability is lower than a set threshold;

[0133] The data updating module is used to update the abnormal event type in the target reasoning result with the abnormal event type recorded in the other reasoning results when the probability of the abnormal event type recorded in the other reasoning results is greater than or equal to a set threshold.

[0134] Optionally, the system 400 further includes:

[0135] A visualization display module, used to create a corresponding visualization interface according to the abnormal event mapping relationship to visualize the abnormal event mapping relationship;

[0136] An operation determination module, used to determine a target operation on a row of visualized data in the visualization interface;

[0137] A jump module, used for jumping to the abnormal video frame and abnormal log line corresponding to the viewed visual data when the target operation is to view the details of the visual data line;

[0138] The data recording module is used to record the corresponding rejection operation when the target operation is to reject the visualized data row.

[0139] Optionally, the system 400 further includes:

[0140] A model performance determination module, used to determine the performance of the abnormal event recognition model according to the number of rejection operations performed on the visualized data rows;

[0141] The model training module is used to perform secondary training on the abnormal event recognition model according to the performance to obtain a new abnormal event recognition model.

[0142] Optionally, the system 400 further includes:

[0143] A data acquisition module, used to acquire various video files and various log files of the vehicle when the abnormal event recognition model is deployed in the cloud;

[0144] A compression module is used to compress the acquired video files and log files to obtain target video files and target log files;

[0145] The upload module is used to upload various target video files and various target log files to the cloud.

[0146] Optionally, the system 400 further includes:

[0147] A signal monitoring module, used for monitoring the video signal of the vehicle screen through the vehicle screen monitoring tool of the vehicle system when the video file is a display screen video file;

[0148] The data acquisition submodule is used to persist the video signal of the monitored vehicle computer screen into a video file through a screen mirroring tool, and obtain the persistent video file.

[0149] Optional, compression module, including:

[0150] A file type determination module, used to determine the file type of the acquired data file;

[0151] A first compression module, for compressing the video file by using a compression algorithm corresponding to the video file type to obtain a target video file when the file type is a video;

[0152] The second compression module is used to compress the log file by using a compression algorithm corresponding to the log file type to obtain a target log file when the file type is log.

[0153] Optionally, the system 400 further includes:

[0154] Status monitoring module, used to monitor the online status of file acquisition, compression and upload processes in real time through the watchdog program;

[0155] The heartbeat reporting module is used to send messages to report the heartbeat to the cloud through the target protocol when in the online state;

[0156] The restart module is used to restart the acquisition, compression and upload processes when in offline state.

[0157] Based on the same inventive concept, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored on the memory and running on the processor. When the computer program is executed by the processor, it implements the steps in a fault analysis method for mutual positioning of logs and videos as described in the first aspect of the present application.

[0158] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in a fault analysis method for mutual positioning of logs and videos as described in the first aspect of the present application are implemented.

[0159] As for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0161] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0163] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0167] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0168] The above is a detailed introduction to the fault analysis method, system and product for mutual positioning of logs and videos provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A fault analysis method for mutual positioning of logs and videos, characterized in that: The method comprises: Identify and process various target video files and various target log files of the vehicle through the abnormal event recognition model, and determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files; Creating an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type; Based on the abnormal event mapping relationship, a fault analysis is performed on the vehicle to obtain a corresponding fault analysis result.

2. A fault analysis method for mutual positioning of logs and videos according to claim 1, characterized in that: Various target video files and various target log files of the vehicle are identified and processed by the abnormal event identification model to determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files, including: Input various target video files and various target log files of the vehicle into the abnormal event recognition model, and extract the corresponding target video features and target log features; By performing recognition processing on the target video features and the target log features, abnormal video frames and abnormal log lines therein are determined; Performing model reasoning on the determined abnormal video frame and the abnormal log line to determine reasoning results corresponding to the abnormal video frame and the abnormal log line respectively; According to all the inference results, the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line are determined.

3. A fault analysis method for mutual positioning of logs and videos according to claim 2, characterized in that: Based on all the inference results, determine the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line, including: Determine the probability of the abnormal event type recorded in the target reasoning result; When the probability is lower than a set threshold, obtaining other reasoning results under the occurrence period according to the occurrence period corresponding to the target reasoning result; When the probability of the abnormal event type recorded in the other inference results is greater than or equal to a set threshold, the abnormal event type in the target inference result is updated with the abnormal event type recorded in the other inference results.

4. A fault analysis method for mutual positioning of logs and videos according to claim 1, characterized in that: The method further comprises: According to the abnormal event mapping relationship, a corresponding visualization interface is created to visualize the abnormal event mapping relationship; Determining a target operation on a row of visualized data in the visualization interface; When the target operation is to view details of a row of visual data, jump to the abnormal video frame and abnormal log row corresponding to the viewed visual data; When the target operation is to reject the visualized data row, the corresponding rejection operation is recorded.

5. A fault analysis method for mutual positioning of logs and videos according to claim 4, characterized in that: The method further comprises: Determine the performance of the abnormal event recognition model based on the number of rejection operations on the visualized data rows; According to the performance, the abnormal event recognition model is trained a second time to obtain a new abnormal event recognition model.

6. A fault analysis method for mutual positioning of logs and videos according to claim 1, characterized in that: In the case where the abnormal event recognition model is deployed in the cloud, the method further includes: Obtain various video files and log files of the vehicle; Compressing the acquired video files and log files to obtain target video files and target log files; Upload various target video files and various target log files to the cloud.

7. A fault analysis method for mutual positioning of logs and videos according to claim 6, characterized in that: When the video file is a display screen video file, the video file of the vehicle is obtained, including: Monitor the video signal of the vehicle screen through the vehicle screen monitoring tool of the vehicle system; Based on the video signal of the monitored vehicle computer screen, the video signal is persisted as a video file through a screen mirroring tool, and the persisted video file is obtained.

8. A fault analysis method for mutual positioning of logs and videos according to claim 6, characterized in that: Compress the acquired video file and log file to obtain the target video file and target log file, including: Determine the file type of the acquired data file; When the file type is a video, compressing the video file by using a compression algorithm corresponding to the video file type to obtain a target video file; In the case where the file type is a log, the log file is compressed using a compression algorithm corresponding to the log file type to obtain a target log file.

9. A fault analysis method for mutual positioning of logs and videos according to claim 6, characterized in that: The method further comprises: The online status of file acquisition, compression and upload processes is monitored in real time through the watchdog program; When online, send a message to report the heartbeat to the cloud through the target protocol; Restart the fetch, compress and upload process while offline.

10. A fault analysis system for mutual positioning of logs and videos, characterized in that: The system comprises: An abnormality determination module is used to identify and process various target video files and various target log files of the vehicle through an abnormal event recognition model, and determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files; A mapping relationship creation module, used to create an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type; The fault analysis module is used to perform fault analysis on the vehicle based on the abnormal event mapping relationship to obtain corresponding fault analysis results.

11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps in a fault analysis method for mutual positioning of logs and videos as described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the fault analysis method for mutual positioning of logs and videos as described in any one of claims 1 to 9 are implemented.

Images