A method, system and product for log and video cross-location fault analysis

Abstract

The embodiment of the application provides a log and video mutual positioning fault analysis method, system and product, the method comprises the following steps: identifying and processing various target video files and various target log files of a vehicle through an abnormal event identification model, determining abnormal video frames in the target video files and corresponding abnormal event types, and abnormal log lines in the target log files and corresponding abnormal event types; creating an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frames and the corresponding abnormal event types, and the abnormal log lines and the corresponding abnormal event types; based on the abnormal event mapping relationship, the vehicle is analyzed, and a corresponding fault analysis result is obtained. The purpose is to improve the fault fact restoration effect in fault analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobile fault analysis, in particular to a fault analysis method and system for mutual positioning of logs and videos, and a product. BACKGROUND

[0002] A log file, as a kind of data recording the sequence and time relationship of events, plays an important role in fault analysis. A specific fault analysis method is to restore fault facts by analyzing abnormal log lines in a log file. However, the restoration effect of fault facts is not good when fault analysis is only based on a log file. Therefore, how to improve the restoration of fault facts in fault analysis is an urgent problem to be solved. SUMMARY

[0003] Therefore, the present application provides a fault analysis method and system for mutual positioning of logs and videos, and a product. The purpose is to improve the restoration effect of fault facts in fault analysis.

[0004] The first aspect of the present application provides a fault analysis method for mutual positioning of logs and videos. The method comprises:

[0005] Identifying and processing various target video files and various target log files of a vehicle by using an abnormal event identification model to determine abnormal video frames in the target video files and corresponding abnormal event types, and abnormal log lines in the target log files and corresponding abnormal event types;

[0006] According to the identification information of the vehicle, the abnormal video frames and the corresponding abnormal event types, and the abnormal log lines and the corresponding abnormal event types, an abnormal event mapping relationship corresponding to the identification information of the vehicle is created;

[0007] Based on the abnormal event mapping relationship, fault analysis is performed on the vehicle to obtain a corresponding fault analysis result.

[0008] Optionally, identifying and processing various target video files and various target log files of a vehicle by using an abnormal event identification model to determine abnormal video frames in the target video files and corresponding abnormal event types, and abnormal log lines in the target log files and corresponding abnormal event types comprises:

[0009] Inputting various target video files and various target log files of a vehicle into an abnormal event identification model to extract corresponding target video features and target log features;

[0010] Identifying and processing the target video features and the target log features to determine abnormal video frames and abnormal log lines;

[0011] perform model inference on the determined abnormal video frame and the abnormal log line to determine a respective inference result of the abnormal video frame and the abnormal log line;

[0012] determine an abnormal event type of the abnormal video frame and an abnormal event type of the abnormal log line according to all the inference results.

[0013] Optionally, determining the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line according to all the inference results comprises:

[0014] determining a probability of an abnormal event type recorded in a target inference result;

[0015] in a case where the probability is lower than a set threshold, obtaining other inference results in a time period corresponding to the target inference result according to the time period;

[0016] in a case where a probability of an abnormal event type recorded in the other inference result is greater than or equal to the set threshold, updating the abnormal event type in the target inference result with the abnormal event type recorded in the other inference result.

[0017] Optionally, the method further comprises:

[0018] creating a corresponding visual interface to visually display the abnormal event mapping relationship according to the abnormal event mapping relationship;

[0019] determining a target operation on a visual data line in the visual interface;

[0020] in a case where the target operation is to view details of the visual data line, jumping to abnormal video frames and abnormal log lines corresponding to the visual data to be viewed;

[0021] in a case where the target operation is to reject the visual data line, recording a corresponding rejection operation.

[0022] Optionally, the method further comprises:

[0023] determining a performance of the abnormal event recognition model according to a number of rejection operations on the visual data line;

[0024] performing secondary training on the abnormal event recognition model according to the performance to obtain a new abnormal event recognition model.

[0025] Optionally, in a case where the abnormal event recognition model is deployed in the cloud, the method further comprises:

[0026] obtaining various video files and various log files of a vehicle;

[0027] The obtained video file and log file are compressed to obtain a target video file and a target log file.

[0028] The target video files and the target log files are uploaded to the cloud.

[0029] Optionally, in the case where the video file is a display screen video file, the video file of the vehicle is obtained, including:

[0030] The video signal of the car machine picture is monitored through a car machine picture monitoring tool of the car machine system.

[0031] Based on the monitored video signal of the car machine picture, the video signal is persisted as a video file through a screen mirroring tool, and the persisted video file is obtained.

[0032] Optionally, the obtained video file and log file are compressed to obtain a target video file and a target log file, including:

[0033] The file type of the obtained data file is determined.

[0034] In the case where the file type is a video, the video file is compressed through a compression algorithm corresponding to the video file type to obtain a target video file.

[0035] In the case where the file type is a log, the log file is compressed through a compression algorithm corresponding to the log file type to obtain a target log file.

[0036] Optionally, the method further includes:

[0037] The online state of the file obtaining, compressing and uploading process is monitored in real time through a watchdog program.

[0038] In the case where the online state is in an online state, a heartbeat is reported to the cloud through a target protocol.

[0039] In the case where the online state is in an offline state, the obtaining, compressing and uploading process is restarted.

[0040] The second aspect of the application provides a log and video mutual positioning fault analysis system, the system includes:

[0041] An abnormality determination module is configured to identify and process various target video files and various target log files of a vehicle through an abnormal event identification model, determine abnormal video frames in the target video files and corresponding abnormal event types, and determine abnormal log lines in the target log files and corresponding abnormal event types.

[0042] A mapping relationship creation module, configured to create an abnormal event mapping relationship corresponding to the vehicle identification information based on the vehicle identification information, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type;

[0043] The fault analysis module is used to perform fault analysis on the vehicle based on the abnormal event mapping relationship and obtain corresponding fault analysis results.

[0044] The third aspect of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored on the memory and running on the processor. When the computer program is executed by the processor, it implements the steps in the fault analysis method of mutual positioning of logs and videos as described in the first aspect of the present application.

[0045] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the fault analysis method for mutual positioning of logs and videos as described in the first aspect of the present application are implemented.

[0046] The fault analysis method for mutual positioning of logs and videos provided in this application has the following advantages:

[0047] The embodiment of the present application provides a fault analysis method for mutual positioning of logs and videos. First, various target video files and various target log files of the vehicle are identified and processed through an abnormal event recognition model to determine the abnormal video frames and corresponding abnormal event types in the target video files, as well as the abnormal log lines and corresponding abnormal event types in the target log files; based on the vehicle's identification information, abnormal video frames and corresponding abnormal event types, as well as abnormal log lines and corresponding abnormal event types, an abnormal event mapping relationship corresponding to the vehicle's identification information is created; based on the abnormal event mapping relationship, the vehicle is subjected to fault analysis to obtain corresponding fault analysis results. Therefore, when performing fault analysis, the present application not only considers various log files, but also considers various video files collected by the vehicle. By establishing a mapping relationship between video files and log files of abnormal events that occur in the same time period and of the same type, fault analysis can be performed based on the video files and log files with the mapping relationship at the same time, thereby effectively improving the restoration effect of the fault facts. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flow chart of a log and video interposition fault analysis method according to an embodiment of the present application;

[0050] Figure 2 A schematic diagram of interposition of abnormal video frames and abnormal log lines in a log and video interposition fault analysis method according to an embodiment of the present application;

[0051] Figure 3 Another flow chart of a log and video interposition fault analysis method according to an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a log and video interposition fault analysis system according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0054] Reference Figure 1 , Figure 1 A flow chart of a log and video interposition fault analysis method according to an embodiment of the present application. As shown in Figure 1 , the method comprises:

[0055] Step S1: identifying and processing various target video files and various target log files of the vehicle through an abnormal event identification model to determine abnormal video frames in the target video files and corresponding abnormal event types, and abnormal log lines in the target log files and corresponding abnormal event types.

[0056] In the embodiment, the vehicle is configured with a large number of monitoring sensors to monitor environmental data, and for each monitoring sensor, corresponding environmental data is monitored and obtained, and a corresponding log file is generated to record whether the monitoring process is abnormal and whether the obtained environmental data is abnormal. Therefore, for each monitoring sensor, there is a corresponding target log file. At the same time, the vehicle also collects a large amount of video data to generate video files, including video data obtained by monitoring the vehicle body to generate video files.

[0057] Since the cockpit display screen is an extremely important component of the current intelligent vehicle, it includes almost all the user interaction components except the steering wheel, accelerator and brake, and if it receives and responds to the incorrect user interface, it may cause a major risk event to occur, therefore, in order to better restore the fact of the failure, the video formed by the video signal generated by the vehicle display screen is also saved, so as to restore the fact of the failure more completely. Therefore, the collected vehicle video file also includes a video file generated based on the interface display video signal of the vehicle display screen, the video data obtained by monitoring the vehicle body is obtained by each surround camera around the vehicle body, and there is a target video file corresponding to each surround camera. The video file generated based on the interface display video signal of the vehicle display screen is also a target video file. By inputting various target video files and various target log files of the vehicle into the abnormal event recognition model for recognition processing, the abnormal video frames in the various target video files and the abnormal event types corresponding to each segment of the abnormal video frames can be determined, and the abnormal log lines in the various target log files and the abnormal event types corresponding to each segment of the abnormal log lines can be determined. Since there is an abnormal video frame corresponding to the abnormal event in the target video file, the possibility of only one frame of abnormal video frame is low, and the possibility of a long abnormal video frame is high, therefore, the above description is the abnormal event type corresponding to each segment of the abnormal video frame. Similarly, since there is an abnormal log line corresponding to the abnormal event in the target log file, the possibility of only one log line is low, and the possibility of continuous multiple abnormal log lines is high, therefore, the above description is the abnormal event type corresponding to each segment of the abnormal log line. Among them, the log files of the vehicle include various log files other than the log files corresponding to the sensors, including but not limited to collision warning log files, emergency light flashing log files, emergency braking log files, speed warning log files, car system log files, memory monitoring log files, temperature overheating warning log files, etc.

[0058] In this embodiment, various sensors include but are not limited to laser radar, microwave radar, wheel speed sensor, positioning sensor, collision sensor, etc. The laser radar is used to obtain laser radar data, and the obtaining process is that the laser radar configured by the vehicle sends a laser pulse signal during the vehicle driving process, and the laser pulse signal is reflected back by the surrounding objects. The radar system receives the reflected laser pulse signal, and calculates the distance data based on the round-trip time of the laser pulse signal, and establishes a model according to the data to form a three-dimensional model of the vehicle surrounding environment. The microwave radar is used to obtain microwave radar data, and the obtaining process is that the microwave radar configured by the vehicle transmits a microwave signal and receives a reflected signal during the vehicle driving process, and the distance, speed and direction of the objects around the vehicle are calculated according to the time difference and phase difference between the transmitted microwave signal and the received reflected signal. The wheel speed sensor is used to obtain the vehicle driving speed data, and the obtaining process is that the transmitter in the wheel speed sensor configured by the vehicle sends a magnetic field or electromagnetic wave to the wheel during the vehicle driving process, and when the wheel rotates, the magnetic field or electromagnetic wave is modulated by the gear or aperture on the wheel to generate a change signal. The receiver in the wheel speed sensor detects the change signal and converts it into an electrical signal, thereby calculating the rotational speed of the wheel, and calculating the driving speed of the vehicle based on the rotational speed and the circumference of the wheel. The positioning sensor is used to obtain the positioning data and moving speed data of the vehicle, and the obtaining process is that the positioning sensor (i.e. global positioning system GPS receiver) configured by the vehicle receives the precise timestamp and position signal sent by the satellite during the vehicle driving process, compares the time difference of the position signals received from at least four satellites, and calculates the precise longitude, latitude and height of the vehicle on the earth using the triangulation method to obtain the precise positioning data of the vehicle; based on the positioning data of the vehicle and the position change of the vehicle per unit time, the moving speed of the vehicle is calculated. The collision sensor is used to obtain the collision data of the vehicle, and the obtaining process is that when the vehicle collides, the sensitive elements inside the collision sensor configured by the vehicle are impacted and vibrated, and the physical changes such as impact and vibration are converted into electrical signals to accurately capture the collision state of the vehicle. The surround view camera is used to obtain the panoramic monitoring image data of the vehicle, and the obtaining process is that the surround view camera configured by the vehicle converts the light signals around the vehicle into video signals, and the image processing unit processes and splices the video signals from each surround view camera to form seamless panoramic monitoring image data. The interface display video signal generated by the video file obtained by the vehicle display screen will reflect the user's behavior intention data in the display screen, and the display screen sensor configured by the vehicle converts the touch position and pressure physical changes of the user into electrical signals to perceive the user's interaction action with the vehicle, and determines the user's behavior intention based on the interaction action. The behavior intention will be embodied in the video file generated based on the interface display video signal.Various sensors are constantly collecting signals, and the electrical signals output by the sensors are usually weak. The signal amplifier amplifies the signals, and the signal processing circuit filters, shapes, and digitizes the amplified signals to ensure the stability and accuracy of the signals.

[0059] In this embodiment, the data information collected by the sensor not only generates a corresponding log file and is stored in the vehicle storage system, but also sends the collected data information to the control unit of the vehicle through wires or wireless methods. The specific process is that the signals processed by the signal amplifier and the signal processing circuit are transmitted to the cockpit electronic control unit (CECU Electronic Control Unit for Cockpit) through the communication network (such as CAN bus, LIN bus, MOST bus, etc.) inside the vehicle. The cockpit electronic control unit stores the collected data information in the vehicle storage system for subsequent fault diagnosis and historical data analysis, and also performs comprehensive analysis and decision-making on various data information, processes according to the preset algorithm and logic, and controls the vehicle to perform corresponding control actions based on the processing results. The above complex logic processing and calculation capabilities require the support of an intelligent car machine system, which is composed of one or more chips and operating system programs. Through the driver program and system program, as well as the application program built on the upper layer, the intelligent car machine system jointly completes the comprehensive analysis and calculation of various signal information.

[0060] Step S2: According to the identification information of the vehicle, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type, an abnormal event mapping relationship corresponding to the identification information of the vehicle is created.

[0061] In this embodiment, based on the identification information of the vehicle, the determined abnormal video frame and the abnormal event type corresponding to each segment of the abnormal video frame, the determined abnormal log line and the abnormal event type corresponding to each segment of the abnormal log line, the abnormal video frame and the abnormal log line with the same abnormal event type and the occurrence time partially or completely overlapping are created. The corresponding abnormal event mapping relationship records the identification information of the vehicle, the abnormal event type, the abnormal video frame, and the abnormal log line. The abnormal event type includes but is not limited to collision, scratching, anchor throwing, car machine blurring, freezing, black screen, car machine restart, etc.

[0062] For example, based on the vehicle identification information, the determined abnormal video frames and the abnormal event types corresponding to each abnormal video frame, the determined abnormal log lines and the abnormal event types corresponding to each abnormal log line, it is determined that there is a section of 10 to 20 lines of abnormal log lines corresponding to the collision sensor, the corresponding abnormal event type is collision, and the event corresponding to this section of abnormal log lines is from 10:10:10 to 10:10:20 on a certain day. At the same time, it is determined that there is a section of 200 to 800 frames of abnormal video frames corresponding to the front camera, the corresponding abnormal event type is collision, and the occurrence time corresponding to this section of abnormal video frames is also on the certain day, specifically at 10 :10:10 to 10:10:40, because the abnormal event type corresponding to this abnormal log line and this abnormal video frame is the same, and there is partial overlap in their respective occurrence times (the partially overlapping time is 10:10:10 to 10:10:20), an abnormal event mapping relationship is established between this abnormal log line and this abnormal video frame. This abnormal event mapping relationship records the identification information of the vehicle, the determined identical collision abnormal event type, the mutually mapped abnormal log line and this abnormal video frame, and the longest occurrence time among the mutually mapped abnormal log lines and each abnormal video frame. As shown in Table 1 below, Table 1 shows different abnormal event mapping relationships for multiple vehicles.

[0063] Table 1

[0064]

[0065]

[0066] Step S3: Based on the abnormal event mapping relationship, perform fault analysis on the vehicle to obtain corresponding fault analysis results.

[0067] In this embodiment, a vehicle fault analysis is performed based on the created abnormal event mapping relationship of the vehicle to obtain corresponding fault analysis results. In the process of performing vehicle fault analysis based on the created abnormal event mapping relationship, not only various log files are considered, but also various video files collected by the vehicle are considered, thereby effectively improving the fault fact restoration effect of the fault analysis.

[0068] The log and video mutual positioning fault analysis method provided by the embodiments of the present application first identifies and processes various target video files and various target log files of a vehicle through an abnormal event identification model, determines abnormal video frames in the target video files and corresponding abnormal event types, and abnormal log lines in the target log files and corresponding abnormal event types; creates an abnormal event mapping relationship corresponding to the identification information of the vehicle according to the identification information of the vehicle, the abnormal video frames and the corresponding abnormal event types, and the abnormal log lines and the corresponding abnormal event types; and performs fault analysis on the vehicle based on the abnormal event mapping relationship to obtain a corresponding fault analysis result. Thus, when performing fault analysis, the embodiments of the present application not only consider various log files, but also consider various video files collected by the vehicle, and by establishing a mapping relationship between video files and log files of abnormal events occurring at the same time and of the same type, the video files and the log files having the mapping relationship can be used for fault analysis at the same time, thereby effectively improving the restoration effect of fault facts.

[0069] In combination with the above embodiments, in an implementation, the embodiments of the present application further provide a log and video mutual positioning fault analysis method. In the log and video mutual positioning fault analysis method, step S1 can include:

[0070] Step S11: input various target video files and various target log files of a vehicle into an abnormal event identification model to extract corresponding target video features and target log features.

[0071] In the present embodiment, after the various target video files and the various target log files of the vehicle are classified through hierarchical cleaning, they are input into the abnormal event identification model, and the feature extraction layer of the model extracts respective target video features corresponding to the various target video files and respective target log features corresponding to the various target log files.

[0072] In the present embodiment, the gated recurrent unit (GRU) algorithm can effectively control information flow, has fewer model parameters, high training efficiency, simple structure and other advantages, and has better performance than other deep learning algorithms in the case of limited resources, so the abnormal event identification model preferably adopts the gated recurrent unit (GRU) algorithm. It should be understood that the abnormal event identification model can also be other algorithm models, which are not limited here.

[0073] Step S12: determines abnormal video frames and abnormal log lines by identifying and processing the target video features and the target log features.

[0074] In the embodiment, after obtaining the target video features corresponding to each type of target video file and the target log features corresponding to each type of target log file, the obtained target video features and target log features are identified by the abnormal event identification model to determine the abnormal video frames in each type of target video file and the abnormal log lines in each type of target log file.

[0075] Step S13: Model inference is performed on the determined abnormal video frames and abnormal log lines to determine the inference results corresponding to the abnormal video frames and the abnormal log lines respectively.

[0076] In the embodiment, after determining the abnormal video frames in each type of target video file and the abnormal log lines in each type of target log file, model inference is performed on the obtained abnormal video frames and abnormal log lines by the abnormal event identification model to determine the inference results corresponding to each abnormal video frame and the inference results corresponding to each abnormal log line. Each abnormal video frame has a corresponding inference result, and each abnormal log line has a corresponding inference result. Each inference result records the type of abnormal event corresponding to the abnormal video frame or the abnormal log line.

[0077] Step S14: Determine the types of abnormal events of the abnormal video frames and the abnormal log lines according to all the inference results.

[0078] In the embodiment, the types of abnormal events of the abnormal video frames and the abnormal log lines are determined according to all the obtained inference results.

[0079] In combination with the above embodiments, in an implementation, the embodiments of the present application also provide a log and video interposition fault analysis method. In the log and video interposition fault analysis method, step 14 can include steps S141 to S143:

[0080] Step S141: Determine the probability of the type of abnormal event recorded in the target inference result.

[0081] In the embodiment, the abnormal event type corresponding to the abnormal video frame or the abnormal log line is recorded in each inference result, and the probability of the abnormal event type is recorded. In order to ensure the accuracy of the abnormal event type recorded in the inference result, the abnormal event type recorded in the inference result is updated when the probability of the abnormal event type recorded in the inference result is lower than a preset threshold value. The threshold value can be set according to the actual application scenario, and is not limited herein. Specifically, all inference results are target inference results, and the probability of the abnormal event type recorded in the target inference result is determined for each target inference result.

[0082] Step S142: When the probability is lower than the threshold value, other inference results in the occurrence period corresponding to the target inference result are obtained.

[0083] In the embodiment, when the probability of the abnormal event type recorded in the target inference result is lower than the threshold value, other inference results in the occurrence period corresponding to the abnormal video frame or the abnormal log line corresponding to the target inference result are obtained.

[0084] Step S143: When the probability of the abnormal event type recorded in the other inference result is greater than or equal to the threshold value, the abnormal event type recorded in the other inference result is used to update the abnormal event type in the target inference result.

[0085] In the embodiment, whether the probability of the abnormal event type recorded in the other inference result is greater than or equal to the threshold value is determined based on the other inference result in the occurrence period corresponding to the target inference result obtained in step S142. If the probability is greater than or equal to the threshold value, the abnormal event type recorded in the other inference result is used to replace the abnormal event type recorded in the target inference result, so as to update the abnormal event type recorded in the target inference result. If the probability is lower than the threshold value, the abnormal event type recorded in the target inference result is maintained.

[0086] In combination with the above embodiments, in an implementation, the embodiments of the present application further provide a log and video interposition fault analysis method. In the log and video interposition fault analysis method, the method further comprises:

[0087] Step S4: According to the abnormal event mapping relationship, a corresponding visual interface is created to visually display the abnormal event mapping relationship.

[0088] In this embodiment, based on the obtained abnormal event mapping relationship, a visual interface corresponding to the abnormal event mapping relationship is created to visually display the abnormal event mapping relationship. The content of the visual display is similar to that shown in Table 1. Each row displays the identification information of the vehicle, the occurrence event of the abnormal event, the event type of the abnormal event, the abnormal log line position, and the abnormal video frame position. The model of the vehicle can also be displayed. The content of the visual display supports conditional filtering according to the model of the vehicle, the event type, and the like.

[0089] In this embodiment, the visual display is achieved by using a client browser webpage technology based on a Vue component to realize an online network page.

[0090] Step S5: Determine the target operation on the visual data row in the visual interface.

[0091] In this embodiment, the application sets a corresponding detail and rejection function for each row of data recording the abnormal event mapping relationship. It is determined whether the user has performed a corresponding target operation on a specific visual data row in the visual interface. The target operation includes executing a detail function and executing a rejection function.

[0092] Step S6: In the case where the target operation is to view the details of the visual data row, jump to the abnormal video frame and abnormal log line corresponding to the visual data being viewed.

[0093] In this embodiment, in the case where the target operation performed by the user is to execute the detail function, the abnormal event detail analysis page of the row data will be jumped to. The detail analysis page is divided into left and right two panels. The left side sequentially arranges all related abnormal log lines from top to bottom. These abnormal log lines are loaded into the browser display in the format of a text file. The left side of each log window displays the line number, and the right side and the lower side display the drag bar, so as to ensure that the log content can be viewed more conveniently and in detail. The right side of the detail analysis page sequentially arranges each abnormal video frame from top to bottom. These abnormal video frames are loaded in a format supported by the browser. Each video window has a play, pause button, and a drag progress bar below.

[0094] In the embodiment, the implementation effect of the detail analysis page is that, according to the line number of the abnormal log line, the corresponding range in each log window is marked as red, according to the frame number of the abnormal video frame, the corresponding range of the progress bar in each video window is marked as red, and the red area is marked with the abnormal event type. When the user clicks the event label, the log window is automatically positioned to the abnormal log line, which is also highlighted, the video progress bar is automatically positioned to play the abnormal video frame segment and highlight the corresponding video progress bar range. In this way, the user can analyze and process problems on the cloud platform webpage, and a fault analysis method of mutual positioning of log files and video files is realized. As shown in Figure 2 Figure 2 An abnormal log line and an abnormal video frame are shown in the same abnormal event type, and by clicking the marked abnormal event type, the abnormal video frame and the abnormal log line corresponding to the abnormal event type are positioned.

[0095] Step S7: In the case where the target operation is a rejection of the visualization data line, record the corresponding rejection operation.

[0096] In the embodiment, in the case where the target operation performed by the user is a rejection operation, the rejection operation is recorded, and the user's explanation for the rejection operation is also recorded. The rejection operation indicates that the model inference result of the model for the row data corresponding to the rejection operation is incorrect, and the explanation for the rejection operation is the user's explanation for the rejection operation. For example, some abnormal video frames in the row data are incorrect, which are not abnormal video frames; some abnormal log lines in the row data are incorrect, which are not abnormal log lines; the abnormal event type in the row data is incorrect, and the specific abnormal event is explained.

[0097] In combination with the above embodiments, in an implementation, the embodiments of the present application also provide a fault analysis method of mutual positioning of logs and videos. In the fault analysis method of mutual positioning of logs and videos, the method further includes:

[0098] Step S8: According to the number of rejection operations on the visualization data line, determine the performance of the abnormal event recognition model.

[0099] ​In the embodiment, the performance of the abnormal event recognition model is determined according to the number of rejection operations performed by the user on the visual data row and the description of the rejection operation. In a data set in which a large number of visual data rows are recorded, if the number of rejection operations on the visual data row exceeds a first preset proportion of the data set, it is determined that the performance of the abnormal event recognition model is poor, and the abnormal event recognition model needs to be retrained at this time; or the description of the rejection operation is counted to determine whether the number of descriptions of the same type exceeds a second preset proportion of the data set, and if it exceeds, it is also determined that the performance of the abnormal event recognition model is poor, and the abnormal event recognition model needs to be retrained at this time. For example, if the number of descriptions of the abnormal event type error in all rejection operations exceeds 20% of the data set, it is determined that the performance of the abnormal event recognition model is poor, and the abnormal event recognition model needs to be retrained at this time.

[0100] Step S9: According to the performance, the abnormal event recognition model is retrained to obtain a new abnormal event recognition model.

[0101] In the embodiment, in the case where the performance of the abnormal event recognition model determined by step S8 is poor, the abnormal event recognition model is retrained to improve the performance of the abnormal event recognition model, so as to obtain a new abnormal event recognition model with better performance.

[0102] In combination with the above embodiments, in an implementation, the embodiments of the present application also provide a log and video mutual positioning fault analysis method. In the log and video mutual positioning fault analysis method, in the case where the abnormal event recognition model is deployed in the cloud, the method further comprises:

[0103] Step S01: Obtain various video files and various log files of the vehicle.

[0104] In the embodiment, the abnormal event identification model requires more computing resources for data processing. A preferred implementation is to deploy it in the cloud. It should be understood that the abnormal event identification model can also be deployed on the vehicle terminal when the computing resources are sufficient. Subsequently, only the results obtained by data processing need to be uploaded to the cloud. Another optional implementation is to deploy the abnormal event identification model in an offline store for fault detection. In the case of deploying the abnormal event identification model in the cloud, the application will obtain relevant data files and upload them to the cloud for data processing. Specifically, various types of video files collected by the vehicle are obtained, including but not limited to video files corresponding to environmental information around the vehicle body collected by the vehicle surround-view camera and video files corresponding to the interface video signal of the vehicle's vehicle terminal display screen, and various types of log files of the vehicle. In the case of deploying the abnormal event identification model in the cloud, the data files received by the cloud will be first decompressed and then classified by layer cleaning before being input into the abnormal event identification model.

[0105] Step S02: The obtained video files and log files are compressed to obtain target video files and target log files.

[0106] In the embodiment, in order to improve processing efficiency, the application compresses various types of video files and various types of log files of the vehicle obtained before uploading them to the cloud to improve transmission effect. By compressing various types of video files and various types of log files obtained, corresponding target video files and target log files are obtained.

[0107] Step S03: Upload various types of target video files and various types of target log files to the cloud.

[0108] In the embodiment, various types of target video files and various types of target log files obtained by compression are uploaded to the cloud through the FTP (File Transfer Protocol) protocol. At the same time, the identification information of the vehicle is recorded in various types of video files and various types of log files, so that the cloud can know the specific vehicle to which the received target video files and target log files belong. Each vehicle has a unique and unique identification information.

[0109] In combination with the above embodiments, in one implementation, the application embodiment further provides a log and video mutual positioning fault analysis method. In the case of a display screen video file, the video file of the vehicle is obtained, including: monitoring the video signal of the vehicle terminal screen through the vehicle terminal screen monitoring tool of the vehicle terminal system; based on the monitored video signal of the vehicle terminal screen, the video signal is persisted as a video file through a screen mirroring tool, and the persisted video file is obtained.

[0110] In the embodiment, in the case that the video file is a display screen video file of the vehicle (i.e., a video file recording the display content of the display screen interface and the interface jump content in the process of user operation), an optional implementation of obtaining the video file of the vehicle is to monitor the video signal of the vehicle screen through a screen monitoring tool of the vehicle system. In the Android system, the screen monitoring tool is an ADB (Android Debug Bridge) tool, and in the iOS system, the screen monitoring tool can be an Xcode integrated development environment (IDE) and related command line tools (such as the open source project libimobiledevice). Based on the monitored video signal of the vehicle screen, the screen mirroring tool is used to persist the monitored video signal as a video file, and then the persisted video file is obtained.

[0111] In combination with the above embodiments, in an implementation, the embodiments of the present application also provide a log and video mutual positioning fault analysis method. In the log and video mutual positioning fault analysis method, step S02 can include steps S021 to S023:

[0112] Step S021: determining the file type of the obtained data file.

[0113] In the embodiment, in order to reduce system resource overhead, the present application adopts different compression methods for different types of data files. Specifically, first, the file type of the obtained data file is determined, and the file type includes a video file and a log file.

[0114] Step S022: in the case that the file type is a video, the video file is compressed through a compression algorithm corresponding to the video file type to obtain a target video file.

[0115] In the embodiment, in the case that the file type of the determined data file is a video, the video file is compressed through a compression algorithm corresponding to the video file type to obtain a target video file. The compression algorithm corresponding to the video file type includes but is not limited to Adobe Premiere and Shotcut.

[0116] Step S023: in the case that the file type is a log, the log file is compressed through a compression algorithm corresponding to the log file type to obtain a target log file.

[0117] In the embodiment, in the case that the file type of the determined data file is a log file, the log file is compressed by a compression algorithm corresponding to the log file type to obtain a target log file. The compression algorithm corresponding to the log file type includes but is not limited to a DBCC SHRINKFILE command in SQL Server.

[0118] In combination with the above embodiments, in an implementation, the embodiments of the present application further provide a log and video mutual positioning fault analysis method. In the log and video mutual positioning fault analysis method, the method further includes: monitoring the online state of the file acquisition, compression and uploading process in real time by a watchdog program; in the case that the online state is in an online state, sending a heartbeat message to the cloud by a target protocol; in the case that the online state is in an offline state, restarting the acquisition, compression and uploading process.

[0119] In the embodiment, in order to ensure that the target video file and the target log file of the vehicle can be uploaded to the cloud, the present application deploys a watchdog program in the vehicle machine system to monitor the online state of the corresponding process in real time. Specifically, the online state of the video file and log file acquisition process, the online state of the acquired video file and log file compression process and the online state of the uploading process of the target video file and target log file obtained after compression are monitored in real time by the configured watchdog program; in the case that all processes are in an online state, a heartbeat message is sent to the cloud by a target protocol, and the target protocol is preferably a TCP (Transmission Control Protocol) protocol. In the case that the online state is in an offline state, the acquisition, compression and uploading process are restarted to restore the uploading of the target video file and the target log file.

[0120] In the embodiment, as shown in FIG. 6, the online state of the video file and log file acquisition process, the online state of the acquired video file and log file compression process and the online state of the uploading process of the target video file and target log file obtained after compression are monitored in real time by the configured watchdog program. Figure 3 Figure 3 ​Another example diagram of a log and video mutual positioning fault analysis method provided by the application is shown. The log and video mutual positioning fault analysis method provided by the application first is data collection. Target log files obtained are generated by a vehicle intelligent system of a vehicle, and target video files obtained are obtained by a video stream program configured by a vehicle infotainment system. Based on the obtained various target log files and various target video files, feature extraction and abnormal log line and abnormal video frame identification are performed by an abnormal event identification model. Based on the identified abnormal log line and abnormal video frame, an abnormal event type corresponding to the abnormal log line is determined, and an abnormal event type corresponding to the abnormal video frame is determined. Then, a mapping relationship between the abnormal event and the abnormal log line and the abnormal video frame is created. Thus, the log and video mutual positioning fault analysis method provided by the application realizes a method of mutual positioning based on log files and video files for tracking and analyzing and troubleshooting vehicle faults. Based on the method, a user can master environmental data when a fault occurs, and the pain points of incomplete information data collection, slow offline pulling of vehicle infotainment data, and inability to completely and truly restore a fault scene are solved. Based on the mutual positioning of the log files and the video files, the user no longer needs to analyze log data offline line by line. By clicking on an event label, an abnormal log line, and an abnormal video frame, the user can analyze the problem by mutual positioning of the log and the video, greatly improving the efficiency of fault troubleshooting. When a user analyzes a fault by the method, the user sometimes needs to transfer the fault to another person for processing. If offline, the user needs to compress and package all related materials and send them to the other person for processing, which consumes time and resources. However, the method only needs to send a system link to the other person, and the other person can quickly process and analyze the fault on a web interface, improving work efficiency.

[0121] Based on the same inventive concept, an embodiment of the application provides a log and video mutual positioning fault analysis system, as shown in Figure 4 The system 400 includes:

[0122] An abnormality determination module 401 is configured to determine abnormal video frames and corresponding abnormal event types in target video files of a vehicle and abnormal log lines and corresponding abnormal event types in target log files of the vehicle by identifying the target video files and the target log files of the vehicle through an abnormal event identification model.

[0123] A mapping relationship creation module 402 is configured to create an abnormal event mapping relationship corresponding to identification information of the vehicle according to the identification information of the vehicle, the abnormal video frames and the corresponding abnormal event types, and the abnormal log lines and the corresponding abnormal event types.

[0124] A fault analysis module 403 is configured to perform fault analysis on the vehicle based on the abnormal event mapping relationship to obtain a corresponding fault analysis result.

[0125] Optionally, the anomaly determination module 401 comprises:

[0126] a feature extraction module configured to input various types of target video files and various types of target log files of the vehicle into an anomaly event recognition model, and extract corresponding target video features and target log features;

[0127] a recognition processing module configured to determine abnormal video frames and abnormal log lines by performing recognition processing on the target video features and the target log features;

[0128] a model inference module configured to perform model inference on the determined abnormal video frames and abnormal log lines, and determine respective inference results of the abnormal video frames and the abnormal log lines;

[0129] an anomaly determination sub-module configured to determine anomaly event types of the abnormal video frames and anomaly event types of the abnormal log lines according to all the inference results.

[0130] Optionally, the anomaly determination sub-module comprises:

[0131] a probability determination module configured to determine a probability of an anomaly event type recorded in a target inference result;

[0132] an inference result acquisition module configured to, in a case where the probability is lower than a set threshold, acquire other inference results in a time period corresponding to the target inference result according to the time period;

[0133] a data update module configured to, in a case where a probability of an anomaly event type recorded in the other inference results is greater than or equal to a set threshold, update the anomaly event type in the target inference result with the anomaly event type recorded in the other inference results.

[0134] Optionally, the system 400 further comprises:

[0135] a visual display module configured to create a corresponding visual interface according to the anomaly event mapping relationship, and visually display the anomaly event mapping relationship through the visual interface;

[0136] an operation determination module configured to determine a target operation on a visual data line in the visual interface;

[0137] a jump module configured to, in a case where the target operation is to view details of a visual data line, jump to abnormal video frames and abnormal log lines corresponding to the visual data to be viewed;

[0138] a data recording module configured to, in a case where the target operation is to reject a visual data line, record a corresponding rejection operation.

[0139] Optionally, the system 400 further comprises:

[0140] a model performance determination module configured to determine the performance of the abnormal event identification model according to the number of rejected visual data rows in the statistics;

[0141] a model training module configured to retrain the abnormal event identification model according to the performance to obtain a new abnormal event identification model.

[0142] Optionally, the system 400 further comprises:

[0143] a data acquisition module configured to acquire various video files and various log files of the vehicle in the case that the abnormal event identification model is deployed in the cloud;

[0144] a compression module configured to compress the acquired video files and log files to obtain target video files and target log files;

[0145] an upload module configured to upload various target video files and various target log files to the cloud.

[0146] Optionally, the system 400 further comprises:

[0147] a signal monitoring module configured to monitor the video signal of the car machine picture through a car machine picture monitoring tool of a car machine system in the case that the video file is a display screen video file;

[0148] a data acquisition sub-module configured to persist the video signal as a video file through a screen mirroring tool based on the monitored video signal of the car machine picture, and acquire the persisted video file.

[0149] Optionally, the compression module comprises:

[0150] a file type determination module configured to determine the file type of the acquired data file;

[0151] a first compression module configured to compress the video file through a compression algorithm corresponding to the video file type to obtain a target video file in the case that the file type is a video;

[0152] a second compression module configured to compress the log file through a compression algorithm corresponding to the log file type to obtain a target log file in the case that the file type is a log.

[0153] Optionally, the system 400 further comprises:

[0154] a state monitoring module configured to monitor the online state of the file acquisition, compression and upload processes in real time through a watchdog program.

[0155] a heartbeat reporting module, configured to report a heartbeat to the cloud by sending a message through a target protocol when in an online state;

[0156] a restarting module, configured to restart the acquiring, compressing and uploading processes when in an offline state.

[0157] Based on the same inventive concept, one embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the steps in the method for log and video mutual positioning fault analysis according to the first aspect of the present application.

[0158] Based on the same inventive concept, one embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps in the method for log and video mutual positioning fault analysis according to the first aspect of the present application.

[0159] For the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0160] It should be noted that, for the method embodiment, in order to simply describe, the method embodiment is described as a series of action combinations, but those skilled in the art should know that the method embodiment is not limited to the action sequence described, because according to the method embodiment, some steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the method embodiment.

[0161] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0162] Those skilled in the art should know that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0163] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It is understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0164] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices, such that a series of operational steps are performed on the computer or other programmable terminal devices to create a computer implemented process so that the instructions executed on the computer or other programmable terminal devices provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0166] Although preferred embodiments of the present application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments once they have the benefit of the foregoing description. Accordingly, the appended claims are intended to cover all modifications and variations of the preferred embodiments that fall within the scope of the present application.

[0167] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0168] The log and video mutual positioning fault analysis method, system and product provided by the present application are described in detail above, and the principles and implementation modes of the present application are described by applying specific examples. The above example is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as a limitation on the present application.

Claims

1. A fault analysis method for mutual positioning of logs and videos, characterized in that: The method comprises: Identify and process various target video files and various target log files of the vehicle using the abnormal event recognition model to determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files; Creating an abnormal event mapping relationship corresponding to the vehicle identification information according to the vehicle identification information, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type; Based on the abnormal event mapping relationship, a fault analysis is performed on the vehicle to obtain a corresponding fault analysis result.

2. A fault analysis method for mutual positioning of logs and videos according to claim 1, characterized in that: The abnormal event recognition model is used to identify and process various target video files and various target log files of the vehicle, and abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files, including: Input various target video files and various target log files of the vehicle into the abnormal event recognition model to extract the corresponding target video features and target log features; By performing recognition processing on the target video features and the target log features, abnormal video frames and abnormal log lines are determined; Performing model reasoning on the determined abnormal video frame and the abnormal log line to determine reasoning results corresponding to the abnormal video frame and the abnormal log line respectively; Based on all the inference results, the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line are determined.

3. A fault analysis method for mutual positioning of logs and videos according to claim 2, characterized in that: Based on all the inference results, determine the abnormal event type of the abnormal video frame and the abnormal event type of the abnormal log line, including: Determine the probability of the abnormal event type recorded in the target inference result; When the probability is lower than the set threshold, obtaining other reasoning results under the occurrence period according to the occurrence period corresponding to the target reasoning result; When the probability of the abnormal event type recorded in the other inference results is greater than or equal to a set threshold, the abnormal event type in the target inference result is updated with the abnormal event type recorded in the other inference results.

4. The fault analysis method for mutual positioning of logs and videos according to claim 1, characterized in that: The method further comprises: According to the abnormal event mapping relationship, a corresponding visualization interface is created to visually display the abnormal event mapping relationship; Determining a target operation on a row of visualized data in the visualization interface; When the target operation is to view details of a row of visual data, jump to the abnormal video frame and abnormal log row corresponding to the viewed visual data; In the case where the target operation is to reject the visualized data row, the corresponding rejection operation is recorded.

5. The fault analysis method for mutual positioning of logs and videos according to claim 4 is characterized in that: The method further comprises: Determine the performance of the abnormal event recognition model based on the statistical number of rejection operations on the visualized data rows; According to the performance, the abnormal event recognition model is trained a second time to obtain a new abnormal event recognition model.

6. The fault analysis method for mutual positioning of logs and videos according to claim 1, characterized in that: In the case where the abnormal event recognition model is deployed in the cloud, the method further includes: Obtain various video files and log files of the vehicle; Compressing the acquired video file and log file to obtain a target video file and a target log file; Upload various target video files and various target log files to the cloud.

7. The fault analysis method for mutual positioning of logs and videos according to claim 6, characterized in that: If the video file is a display screen video file, obtain the vehicle video file, including: Monitor the video signal of the vehicle screen through the vehicle screen monitoring tool of the vehicle system; Based on the video signal of the monitored vehicle screen, the video signal is persisted into a video file through a screen mirroring tool, and the persisted video file is obtained.

8. The fault analysis method for mutual positioning of logs and videos according to claim 6, characterized in that: Compress the acquired video file and log file to obtain the target video file and target log file, including: Determine the file type of the acquired data file; In the case where the file type is a video, compressing the video file using a compression algorithm corresponding to the video file type to obtain a target video file; In the case that the file type is a log, the log file is compressed using a compression algorithm corresponding to the log file type to obtain a target log file.

9. The fault analysis method for mutual positioning of logs and videos according to claim 6, characterized in that: The method further comprises: The online status of file acquisition, compression and upload processes is monitored in real time through the watchdog program; When in online state, it sends a message to report the heartbeat to the cloud through the target protocol; Restart the fetching, compression, and uploading processes while offline.

10. A fault analysis system with mutual positioning of logs and videos, characterized in that: The system comprises: An abnormality determination module is used to identify and process various target video files and various target log files of the vehicle using an abnormal event recognition model, and determine abnormal video frames and corresponding abnormal event types in the target video files, as well as abnormal log lines and corresponding abnormal event types in the target log files; A mapping relationship creation module, configured to create an abnormal event mapping relationship corresponding to the vehicle identification information based on the vehicle identification information, the abnormal video frame and the corresponding abnormal event type, and the abnormal log line and the corresponding abnormal event type; The fault analysis module is used to perform fault analysis on the vehicle based on the abnormal event mapping relationship and obtain corresponding fault analysis results.

11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein when the computer program is executed by the processor, the steps of the fault analysis method for mutual positioning of logs and videos as described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a fault analysis method for mutual positioning of logs and videos according to any one of claims 1 to 9.

Images