Automobile remote detection and fault diagnosis data intelligent processing system

Abstract

The automobile remote detection and fault diagnosis data intelligent processing system of the present application provides automobile remote data processing solutions from three aspects: data parsing, loading and storage. The first is data parsing: defining the parsing rules of ASC files, DBC‑c files, and Report‑c files, and proposing parsing methods for different signal types of evolutionary calculations, establishing the parsing process of corresponding messages, and orderly and efficient vehicle data parsing; the second is data loading: constructing a data loading method suitable for different data types, with less data loading delay and good real-time performance; the third is data storage: building a distributed cluster based on the database Greenplum architecture, and building a system module-related database system; using indexing, distribution, partitioning and marking field methods to improve query speed and system security. The data storage and extraction speed is fast, query retrieval is easy, and remote data processing is efficient, which strongly supports the development of automobile detection and fault diagnosis systems from the perspective of data processing.

Description

Technical Field

[0001] The present application relates to a vehicle remote diagnosis data processing system, and more particularly to an automobile remote detection and fault diagnosis data intelligent processing system, belonging to the technical field of vehicle remote data processing. Background Art

[0002] With the rapid increase in the number of cars and their usage, the number of car failures will inevitably increase, making continuous diagnosis and maintenance of cars an indispensable link. The car fault diagnosis system uses the electronic control unit ECU on the car to monitor and detect the working condition of the car. Once a fault is detected, the system will trigger the automatic fault diagnosis program, track and store the fault code DTC in the car memory. If the fault code persists for a period of time, the signal circuit of the electronic component may have a problem, and the maintenance personnel can locate the fault location based on the fault code. For example, the oil and engine coolant sensors measure various physical parameters of the car, such as temperature and air pressure, and compare them with acceptable limits. If the actual value of the car exceeds the acceptable limit, then the fault is confirmed and the fault code is stored in the ECU.

[0003] Although fault codes can be obtained through the automobile fault diagnosis system, ordinary drivers without any professional background cannot identify the specific meaning of the fault codes and still need to seek help from professional maintenance personnel after a fault occurs. For maintenance manufacturers who lack experience and have poor hardware equipment, they will be helpless when encountering slightly complex automobile failures. Fault codes can be translated through a handheld fault detector during driving, but this requires hardware support, and fault information cannot be identified without a fault detector.

[0004] In order to meet the needs of intelligent monitoring, diagnosis and maintenance of automobiles, automobile remote diagnosis systems have gradually entered people's field of vision. The emergence of automobile remote diagnosis systems has enabled the timely discovery of hidden faults in automobiles, which has greatly guaranteed and improved driving safety for the frequent traffic accidents every year. For maintenance workshops and maintenance personnel, intelligent systems can help them analyze vehicle conditions and find faults faster and better, which not only improves maintenance efficiency but also plays an important role in improving products.

[0005] If maintenance personnel can be assisted in troubleshooting without complicated manual inspection and hardware support, it will not only greatly reduce the time of manual troubleshooting, but also avoid excessive energy consumption. The automobile remote diagnosis system transmits the detection information to the remote server for diagnosis through real-time monitoring of automobile equipment, thus achieving real prevention. One of the key bases for automatic detection is data. Statistics of historical records including fault data provide a basis for future big data analysis; various data and information of the equipment are collected and sorted to form corresponding digital information, providing technical support and experience for future fault detection; real-time analysis and processing of real-time data, and more clear and intuitive display in the form of charts, provide visual help for timely detection of faults and repairs. However, there are still many difficulties in data processing of existing automobile remote detection and fault diagnosis systems.

[0006] Since the automobile remote diagnosis system involves many technical fields, including wireless communication, positioning and navigation, and the research and development of each of these technologies is difficult, even in the more technologically advanced automobile companies, no breakthrough progress has been made in this field. In reality, automobile remote diagnosis technology has only achieved some simple functions, and more complex and substantial research is still ongoing. The more typical remote service systems currently available include: 1. General Motors Onstar Automobile Remote Diagnosis System in the United States, 2. Ford SYNC Automobile Remote Diagnosis System in the United States, 3. Toyota G-BOOK Automobile Remote Diagnosis System in Japan, and 4. BMW iDrive Automobile Remote Diagnosis System in Germany. However, these remote service systems do not have outstanding features and performance in automobile remote detection and fault diagnosis and processing, and cannot provide an effective way to obtain, store and manage vehicle data.

[0007] The data processing system is the key part of remote detection and fault diagnosis, and it is also the connecting part between the vehicle-mounted equipment and the data processing system. At present, there is an urgent need to develop a data processing module in the automobile remote diagnosis system, and further improve and optimize the data analysis, data loading and data storage therein, so as to provide a reliable and correct data source for the automobile remote diagnosis system.

[0008] In summary, there are several problems and defects in the existing automotive remote fault detection and diagnosis data processing. The problems and key technical difficulties to be solved in this application include:

[0009] (1) Since the automobile remote diagnosis system involves many technical fields, including wireless communication, positioning and navigation, and professional driving data processing, it is difficult to research and develop each of these technologies. At present, even in the more advanced automobile companies, no breakthrough progress has been made in this field. The existing automobile remote detection technology only realizes some simple functions, and requires complicated manual detection and hardware support to assist maintenance personnel in troubleshooting. The existing remote service system does not have outstanding features and performance in automobile remote detection and fault diagnosis processing, and cannot provide an effective way to obtain, store and manage vehicle data. At present, it is urgent to develop a data processing module in the automobile remote diagnosis system, and further improve and optimize the data analysis, data loading and data storage therein, so as to provide a reliable and correct data source for the automobile remote diagnosis system.

[0010] (2) The existing technology does not have a unified format standard for automobile remote detection and diagnosis data processing, and the data management and processing efficiency is low. The quality of data analysis is poor. There is a lack of on-board fault diagnosis-related protocols, CAN bus encoding formats and floating-point operation standards corresponding to the automobile remote diagnosis system, resulting in a disordered data management and processing state, frequent errors in data transmission and analysis, and a lack of signal analysis solutions based on the characteristics of different types of on-board data. It is impossible to optimize the cache to improve the analysis speed; the data processing system lacks a real-time monitoring solution, and the new files transmitted by the automatic monitoring on-board equipment cannot be divided into single-frame and multi-frame parsing diagnostic data. It is impossible to locate the navigation message parsing method, lack two solutions to improve data parsing efficiency, low vehicle data analysis efficiency, poor analysis quality, and unable to remotely diagnose the vehicle status in real time; it is impossible to store data in a distributed database, lack a data adaptive integration storage method, and cannot meet the needs of efficient loading and storage of on-board data. The storage solutions for different types of data are the same, with many I0 operations, slow storage speed, and low data query efficiency, which ultimately leads to low efficiency in vehicle remote diagnosis data processing, parsing, loading and storage, and poor quality of vehicle remote diagnosis data processing.

[0011] (3) The existing technology lacks solutions for automobile remote data processing in three aspects: data analysis, data loading and data storage. First, data analysis: there is a lack of corresponding file parsing rules, and there is a lack of parsing methods for different signal types in evolutionary calculations. Diagnostic data analysis cannot classify the messages in the initial ASC file according to different parsing rules: there is a lack of corresponding message parsing process, resulting in confusion and low efficiency in vehicle data analysis; second, data loading: there is a lack of data loading methods suitable for different data types, real-time data lacks external tables, and it is impossible to use intermediate temporary files to import data into the database in parallel at one time; the definition scope of non-real-time data is unclear, and there is also a lack of corresponding data loading solutions, resulting in a long data loading delay and poor real-time performance; third, data storage: there is a lack of a distributed cluster environment built based on the overall architecture of Greenplum, and there is a lack of an ER model based on the overall automobile remote diagnosis system, and it is impossible to build a system module-related database system; there is a lack of indexing, distribution, partitioning methods and field marking methods to improve query speed and system security, data storage and extraction speed is slow, query retrieval is difficult, and remote data processing efficiency is low, which seriously restricts the development of automobile detection and fault diagnosis systems. Summary of the invention

[0012] This application has greatly improved the data processing system part of the automobile remote monitoring and fault diagnosis system. First, it has constructed the relevant diagnostic protocols of the automobile remote diagnosis system and the relevant specifications of various signal analysis, designed the implementation plan for analyzing automobile data, and implemented the algorithm for analyzing various types of signals. Through this algorithm, standardized vehicle data can be obtained, which enhances the readability of the data and significantly improves the data analysis rate and quality. Second, based on the storage of automobile-related information including real-time vehicle data based on the distributed database Greenplum, different table structures are designed for different data to store, meeting the storage requirements of massive vehicle data. Third, different loading methods are designed for the problem of loading real-time data and non-real-time data into the warehouse. The data loading speed is fast and the delay is small. Fourth, it provides convenience for the data analysis system on the data processing system side. Fifth, it improves the real-time data transmission of vehicle-mounted equipment, designs a real-time monitoring plan, realizes automatic monitoring of new files and timely processing, and displays them more clearly and intuitively in the form of charts, providing visual help for timely detection of faults and repairs.

[0013] In order to achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0014] The intelligent processing system for automobile remote detection and fault diagnosis data analyzes and provides automobile remote data processing solutions from three aspects: data analysis, data loading and data storage. First, according to the automobile remote diagnosis system, the vehicle fault diagnosis related protocols, CAN bus encoding format and floating-point operation standards are constructed. Based on the characteristics of different types of vehicle data, different signal analysis schemes are designed, and the cache is optimized to improve the analysis speed; secondly, the data processing system establishes a real-time monitoring scheme, automatically monitors the new files transmitted by the vehicle-mounted equipment and divides them into single-frame and multi-frame parsed diagnostic data, constructs a positioning and navigation message parsing method, and proposes two solutions to improve data parsing efficiency; then, a distributed database is used to store data, and a data adaptive integration storage method is established to meet the efficient loading and storage of vehicle-mounted data. Different types of data use different solutions. Real-time data uses external tables for concurrent and rapid storage, and non-real-time data uses batch import to reduce I0 operations; finally, two improvement schemes are proposed to optimize the efficiency of data intelligent query, including an optimization method based on partition tables and an optimization method based on tag fields;

[0015] 1) Data parsing: Define the parsing rules for ASC files, DBC-c files, and Report-c files, and propose parsing methods for different signal types based on evolutionary computing. The test data parsing solution includes the parsing rules for vehicle-mounted equipment source data and data processing systems. Diagnostic data parsing divides the messages in the initial ASC file into three categories based on different parsing rules: diagnostic messages, positioning and navigation messages, and other vehicle messages, and establishes the parsing process for the corresponding messages.

[0016] 2) Data loading: Based on the characteristics of different data types, construct data loading methods suitable for different data types. For real-time data, design an external table method, use intermediate temporary files to import data into the database in parallel at one time, and read data concurrently through external tables; non-real-time data refers to data other than real-time data, including parsing rules used for parsing and data related to various functions in vehicle equipment;

[0017] 3) Data storage: Based on the overall architecture of the Greenplum database, a distributed cluster environment is built, an ER model is established according to the overall automobile remote diagnosis system, and a system module-related database system is constructed; indexing, distribution, partitioning methods and marking field methods are used to improve query speed and system security.

[0018] Preferably, a Report-c file is defined: the Report-c file is defined based on the parsing rules of the automobile electronic control unit status information and the fault code information. The Report-c file is created based on the CDD database and is jointly determined by the automobile type and the electronic control unit, that is, one electronic control unit of one vehicle type corresponds to one Report-c file. Each Report-c file has a unique identifier, which consists of two diagnostic identifiers: a diagnostic request identifier and a diagnostic response identifier.

[0019] The format of the electronic control unit status information is defined as follows:

[0020] 1) Byte: The starting byte number of the electronic control unit status information in the message. If it does not start from bit 0, the starting bit of the starting byte should also be marked;

[0021] 2) Byte description: status description of the electronic control unit;

[0022] 3) Default value: In the first frame, the second byte indicates the multi-frame byte length, and the third to fifth bytes indicate the electronic control unit status and operation represented by the information;

[0023] 4) Size: the number of bytes of different attributes of the ECU status;

[0024] 5) Escape type / escape name / escape identifier: indicates the format into which the value obtained by the byte should be converted;

[0025] The formats of fault code information are defined as follows:

[0026] Fault code: Fault code identifier, which is the same as the default value in the electronic control unit status information and is a diagnostic identifier;

[0027] Description: Description of the fault code;

[0028] Environmental data / snapshot data: They are different information of fault codes, which are separate files. The format is the same as the electronic control unit status information format. The two attribute values ​​in the fault code point to the links of the corresponding files respectively.

[0029] The Report-c file includes four types of sub-files: ECU status information, fault code information, environmental data and snapshot data. The environmental data and snapshot data are sub-files of the fault code information, which explain the content in more detail.

[0030] Preferably, the diagnostic data parsing method:

[0031] Step 1: After getting an ASC file, first extract the basic information of the file, including: file name, number of messages in the file, VIN of the car corresponding to the file, store this information, and then traverse each message in the file;

[0032] Step 2: Segment the row data according to the defined attributes, store the message data before parsing, and store the unparsed data to prepare for restoring the initial file by data condition query. Record the message ID of the message;

[0033] Step 3: If it is a diagnostic message, parse and store it using the Report-c file, otherwise execute step 2;

[0034] Step 4: If it is positioning navigation, parse and store it using the IEEE 754 standard, otherwise execute step 5;

[0035] Step 5: If it is other car signals, use DBC-c file to parse and store them.

[0036] Preferably, the diagnostic message parsing process:

[0037] (1) Single frame

[0038] The single frame information is all contained in one CAN frame. When a message is received, the frame is directly split and parsed. The specific process of single frame parsing includes:

[0039] Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0040] Step 2: The unit of data is byte. Check whether the number of data bytes is equal to the data length. If they are not equal, it means that the frame is wrong and the frame is discarded. Otherwise, go to step 3.

[0041] Step 3: Find the message ID in the DiagID container. If it is a diagnostic request ID, go to step 4. If it is a diagnostic response ID, go to step 5.

[0042] Step 4: Put the request message into the request queue until receiving its response message;

[0043] Step 5: First remove the corresponding request message in the request queue, and then parse the data part of the diagnostic response message according to the Report-c file;

[0044] During the transmission process, due to delay or other reasons, the request frame and the response frame do not appear in pairs, so a request queue is defined to store all request messages. When a request message receives a response message, the request is removed. The system uses diagnostic identifiers to match request and response messages.

[0045] (2) Multi-frame

[0046] The characteristic of multi-frame is that the information is contained in different CAN frames. When such messages are received, they cannot be processed directly, but are cached. When all CAN frames are received, they are split and parsed. The specific process of multi-frame parsing is as follows:

[0047] Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0048] Step 2: Determine whether it is the first frame according to the network protocol control information type. If it is the first frame, intercept the multi-frame length d and the diagnostic identifier, and put the data part into the pre-defined multi-frame data container List. If it is a continuous frame, execute step 3;

[0049] Step 3: According to the serial frame numbers, the data in the serial frames are stored in the List in order;

[0050] Step 4: If the number of bytes of data stored in List is greater than d, then execute step 5, otherwise continue to receive messages and execute step 1;

[0051] Step 5: At this point, all frames of the response message have been received, and the diagnostic response message is parsed according to the Report-c file;

[0052] When multiple frames are transmitted, the continuous frames received are out of order. An ordered container is set up. Whenever a continuous frame is received, it is inserted into the container according to the number of the continuous frame until the length of the received valid data is equal to the data length in the first frame.

[0053] Preferably, the positioning and navigation message parsing process: the message consists of a timestamp, a channel number, a message identifier, a request / response, a message type, a data length and other parts of the data, wherein whether a message belongs to a positioning and navigation message is determined according to the channel number and the message identifier. In the system, a message with a channel number of 9 and a message identifier ranging from 001 to 00A belongs to a positioning and navigation message;

[0054] The specific process of positioning and navigation data analysis is as follows:

[0055] Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0056] Step 2: Determine whether the message is a positioning navigation timestamp message. If so, calculate the time directly. If not, proceed to step 3.

[0057] Step 3: Calculate the longitude and latitude of positioning navigation according to the IEEE 764 standard;

[0058] Step 4: If a positioning and navigation signal has been completely collected, then store the positioning and navigation signal, otherwise continue to receive messages.

[0059] Preferably, other automobile message parsing processes: other message parsing processes except diagnostic messages and positioning navigation messages:

[0060] Process 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0061] Process 2: Find the corresponding DBC file file1 according to the car type obtained from the ASC file and the channel number of the message, and then determine the parsing rule in file1 according to the message identifier. If it cannot be found, it means that the message is wrong and discarded; if the parsing rule is found, then execute process 3;

[0062] Process 3: Parse the message according to the parsing rules and determine whether the message is marked. If marked, execute process 4; otherwise, execute process 5;

[0063] Process 4: Mark the corresponding fields of seconds, minutes, and hours according to the timestamp;

[0064] Process 5: Store the parsed signal data.

[0065] Preferably, the solution to improve data parsing efficiency is:

[0066] Solution 1: Use multithreading to improve the computing speed: Use multithreading programming to maximize the use of CPU resources, split a large ASC file into any small files, and process the small files at the same time. The time it takes to parse the smallest file with the longest parsing time is the final parsing time, and split the large file according to the message time.

[0067] Solution 2: Reduce the frequency of database access: During the data parsing process, the parsing rules used are stored in the database in advance. Each time a message is parsed, a connection is established with the database and the corresponding parsing rules are queried. An ASC file is the real-time data of a vehicle in a certain period of time. The parsing rules come from the DBC-c files corresponding to different channels. For a known vehicle model, these files are fixed and unchanged. During the parsing process, the corresponding parsing rules are searched from the cache, using a cache high-speed caching method.

[0068] Preferably, non-real-time data is stored in the database: non-real-time data is other data except real-time data, including newly added vehicles, parsed files and functional files;

[0069] 1) Adding a new vehicle: Before parsing an ASC file, first verify the vehicle to which the file belongs. If the vehicle already exists in the system, parse the data directly; if the vehicle does not exist, that is, the vehicle data has never been imported into the system before, then before parsing, add the vehicle to the system; adding a new vehicle not only adds a piece of data to the VEHICLE table, but also adds the corresponding MESSAGE and SIGNAL tables to store the vehicle;

[0070] 2) Parsing files: including DBC files and report files. All these files are loaded into the database before parsing. Among them, Report-c files have been fully loaded before the system is run. The vehicle remote diagnosis system provides users with an interface for uploading DBC files in the file management module. When loading parsing files, in order to improve loading efficiency, a custom batch processing solution is adopted;

[0071] 3) Function file: includes function external connection file and function internal signal file. The import method of this file is the same as that of the parsing file.

[0072] Preferably, real-time data storage: transmitted to the data processing system in the form of ASC files, the data processing system starts a background process to monitor the designated directory in real time, and once a new ASC file is found, it is immediately parsed, and the parsed message and signal data are timely imported into the database for use by the data analysis system;

[0073] All segments can import data from gpfdist concurrently and at high speed through external tables, including:

[0074] Process 1: Create gpfdist, create an external table on the Master server, and define the original data source of the external table. The vehicle remote diagnosis system transmits the vehicle data in real time, and creates gpfdist before and after the system runs;

[0075] Process 2: The data node connects to gpfdist to obtain data imported from the external table. If the number of gpfdist URLs is greater than 1, the data node starts the same concurrent call to gpfdist to obtain data.

[0076] Process 3: gpfdist first reads the file, splits the file into multiple blocks and distributes them to the data nodes. The data is redistributed by looking up the table. After each data loading is completed, the external table created in the process is automatically deleted.

[0077] Process 4: Reading data and redistribution are performed simultaneously;

[0078] To implement external data import: First, pass the parsed message and signal file address and the physical table to be inserted as parameters to the stored procedure. During the execution, an external table is created. Then, data is loaded into the physical table through the external table. Finally, when all data is inserted into the physical table, the external table is deleted.

[0079] Preferably, the optimization method based on the partition table: partitioning the real-time data table by range partitioning, selecting the signal time as the partition basis, including:

[0080] 1) Create a main table corresponding to the real-time data of the car. No data is stored in this table. At the same time, a sub-table is created according to the timestamp. The sub-table is created dynamically, and one minute of data is stored in a sub-table;

[0081] 2) Dynamically create indexes for the subtables. The solution adopted is to create the indexes three minutes after the partition is created, which improves query efficiency without affecting data loading speed due to redistribution.

[0082] Compared with the prior art, the innovations and advantages of this application are:

[0083] (1) This application has greatly improved the data processing system part of the automobile remote monitoring and fault diagnosis system, conducted an overall functional analysis of the automobile remote diagnosis system, and clarified the requirements of the data processing system part. First, it has constructed the relevant diagnostic protocols of the automobile remote diagnosis system and the relevant specifications of various signal analysis, designed an implementation plan for analyzing automobile data, and implemented algorithms for analyzing various types of signals. Through this algorithm, standardized vehicle data can be obtained, which enhances the readability of the data and significantly improves the data analysis rate and quality. Second, based on the storage of automobile related information including real-time vehicle data in the distributed database Greenplum, different table structures are designed for different data to store comprehensive data characteristics and query efficiency considerations, meeting the storage requirements of massive vehicle data. Third, in order to load real-time data and non-real-time data into the database, different loading methods are designed. The amount of real-time data transmitted by the vehicle-mounted equipment is very large and has high real-time requirements, so it is quickly imported based on the external table tool; the amount of non-real-time data is small, but in order to quickly load it into the database, a batch import algorithm is implemented, and the data loading speed is fast and the delay is small. Fourth, a number of stored procedures and dynamically increasing views are written on the data processing system side, providing convenient and fast query functions for the data analysis system. Fifth, improve the real-time data transmission of on-board equipment, design a real-time monitoring solution for the data processing system, and realize automatic monitoring of new files and timely processing. By analyzing and processing real-time data in real time and displaying it more clearly and intuitively in the form of charts, it provides visual help for timely detection of faults and repairs.

[0084] (2) The present application creatively proposes a data processing module in an automobile remote diagnosis system, and further improves and optimizes the data parsing, data loading and data storage therein, providing an effective way to acquire, store and manage vehicle data, and providing reliable and accurate data support for the automobile remote diagnosis system. The problem of inconsistent format standards for automobile remote detection and diagnosis data processing has been solved, data management and processing efficiency has been improved, and the quality of data analysis has been improved. The on-board fault diagnosis-related protocols, CAN bus encoding formats and floating-point operation standards corresponding to the automobile remote diagnosis system have been constructed, and data management and processing have entered an efficient state. Data transmission and analysis are fast and accurate. A signal analysis scheme based on the characteristics of different types of on-board data has been designed, and the cache has been optimized to improve the analysis speed. A real-time monitoring scheme has been proposed, which automatically monitors new files transmitted by on-board equipment and divides them into single-frame and multi-frame analysis diagnosis data, and an efficient positioning navigation message analysis method has been proposed. Two solutions to improve data analysis efficiency have been proposed, and the analysis efficiency of vehicle data is high and the analysis quality is good, so that the vehicle status can be diagnosed remotely in real time. A distributed database is used to store data, and a data adaptive integration storage method is designed to meet the needs of efficient loading and storage of on-board data. Different types of data storage schemes are different, with fewer I0 operations, fast storage speed, and high data query efficiency, which promotes the improvement of the efficiency of vehicle remote diagnosis data processing, loading and storage, and significantly improves the quality of vehicle remote diagnosis data processing.

[0085] (3) This application provides a high-quality solution for automobile remote data processing from three aspects: data parsing, data loading and data storage. First, data parsing: define the parsing rules of ASC files, DBC-c files and Report-c files, and propose the parsing methods of different signal types for evolutionary computing. The diagnostic data parsing is classified according to different parsing rules, and the parsing process of the corresponding message is established. The vehicle data parsing is orderly and efficient. Second, data loading: construct a data loading method suitable for different data types, design an external table method for real-time data, use intermediate temporary files to import data into the database in parallel at one time, read data concurrently through external tables, and have less data loading delay and good real-time performance. Third, data storage: based on the overall architecture of the database Greenplum, build a distributed cluster environment, establish an ER model according to the overall automobile remote diagnosis system, and build a system module related database system; use index, distribution, partitioning methods and marking field methods to improve query speed and system security. Data storage and extraction speed is fast, query retrieval is easy, remote data processing efficiency is high, and from the perspective of data processing, it has effectively promoted the development of automobile detection and fault diagnosis systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is the overall flow chart of real-time data analysis for automobile remote detection.

[0087] Figure 2 It is a single frame analysis flow chart for real-time data analysis of automobile remote detection.

[0088] Figure 3 This is a schematic diagram of other automobile message parsing processes.

[0089] Figure 4 It is a schematic diagram of experimental data comparing single-threaded and multi-threaded analysis.

[0090] Figure 5 This is a schematic diagram of the experimental results comparing cache and database access.

[0091] Figure 6 This is a schematic diagram of the optimized comparison query results based on the partition table.

[0092] Figure 7 This is a schematic diagram of the data results of adding an external table to the Analyze analysis.

[0093] Figure 8 This is a sample test image of the DBC-c file loading when uploading the page. DETAILED DESCRIPTION

[0094] The following, in conjunction with the accompanying drawings, further describes the technical solution of the automobile remote detection and fault diagnosis data intelligent processing system provided by the present application, so that technicians in this field can better understand the present application and implement it.

[0095] With the rapid growth of automobile production and sales, the maintenance industry has ushered in huge challenges. Traditional automobile maintenance must be carried out on the vehicle, which is limited by geographical, human and material resources. With the development of the Internet of Things and modern communication technology, the vehicle data can be continuously monitored to achieve remote fault diagnosis and real-time data query functions. Users can view the diagnosis and test results at any time through the browser, which has a very broad market prospect.

[0096] This application proposes relevant data processing technology in the automobile remote diagnosis system, which is an important component thereof.

[0097] The present application analyzes and solves the automobile remote data processing solution from three aspects: data parsing, data loading and data storage. Firstly, the vehicle fault diagnosis related protocols, CAN bus encoding format and floating point operation standard are constructed according to the automobile remote diagnosis system. Different signal parsing schemes are designed based on the characteristics of different types of vehicle data, and the cache is optimized to improve the parsing speed. Secondly, the data processing system establishes a real-time monitoring scheme, automatically monitors the new files transmitted by the vehicle-mounted equipment and divides them into single-frame and multi-frame parsed diagnostic data, constructs a positioning and navigation message parsing method, and proposes two solutions to improve the data parsing efficiency. Then, a distributed database is used to store the data, and a data adaptive integration and warehousing method is established to meet the efficient loading and warehousing of vehicle-mounted data. Different types of data use different schemes. Real-time data uses external tables for concurrent and rapid warehousing, and non-real-time data uses batch import to reduce I0 operations. Finally, in order to improve the system query efficiency, two improvement schemes are proposed to optimize the efficiency of data intelligent query, including an optimization method based on partition tables and an optimization method based on tag fields.

[0098] The vehicle data information collected from the automobile equipment belongs to the initial file, in which the content is unreadable. This type of file needs to be processed and converted into highly readable information. This process is data parsing. The initial file to be parsed is called ASC file. The content is real-time data information generated by the automobile equipment. It is not readable and generally difficult to understand directly. To facilitate user query, parallel parsing is performed during the data import process and converted into a readable form. Different signal types have different parsing rules. This application defines the parsing rules for ASC files, DBC-c files, and Report-c files, and proposes parsing methods for different signal types in evolutionary computing.

[0099] 1. Test data analysis solution

[0100] Data parsing is the key to the data processing system. How to efficiently and correctly process vehicle data is the focus of this section. CANoe is a bus development environment and a professional tool for network and ECU node development, ECU automated testing and verification. If third-party professional software such as CANoe is directly used, it will not only be costly, but more importantly, due to the transparency of the parsing process, the maintainability is poor and it is difficult to customize a processing module that meets the data characteristics of this application. This application is based on the DBC network database of CANoe software to establish a parsing rule file belonging to the data processing system.

[0101] 1. Vehicle equipment source data

[0102] The ASC file type is used to store the vehicle data saved on the server. The ASC file is designed to be divided into two parts: a header and real-time data. The header includes the starting time point for collecting the vehicle data in the file, accurate to seconds; the real-time data part is in lines, each line is a message, and the message is the real-time data of the vehicle generated by the vehicle system.

[0103] (II) Data processing system analysis rules

[0104] The parsing rule file suitable for this system is defined based on the DBC database, which is called the DBC-c file. In order to more conveniently process diagnostic messages, a new diagnostic data file is defined based on the CDD database, which is called the Report-c file.

[0105] (1) Define DBC-c file

[0106] The DBC-c file parses general messages in the ASC file except for diagnostic messages and positioning navigation messages. The DBC-c file is determined by the vehicle type and channel number, including:

[0107] 1) Car type: the car type to which the DBC-c file belongs;

[0108] 2) Channel number: the channel to which the DBC-c file belongs;

[0109] 3) Message ID: corresponds to the message ID in the ASC file and consists of hexadecimal numbers;

[0110] 4) Message name: the message name recognized by the device;

[0111] 5) Signal name: the signal name recognized by the device. The message name and signal name are recognized by the device. The signal name and the attribute it refers to are configured in the signal configuration function;

[0112] 6) Message size: the number of bytes in the message, corresponding to the data length in the ASC file;

[0113] 7) Diagnostic request / response: Whether the message is a diagnostic request or a diagnostic response. If both attribute values ​​are NO, the message is not a diagnostic message.

[0114] 8) Signal bit range: The position of the signal in the message. After converting the message bytes into binary bits, the data calculated within this range is the initial calculated value of the signal.

[0115] 9) Signal size: the bit length of a signal;

[0116] 10) Signal attributes: all states included in a signal;.

[0117] 11) Displacement / multiplier factor: Based on these two attributes and the initial calculated value obtained based on the data attributes of the real-time message in the ASC file, the real signal value can be obtained.

[0118] (2) Define Report-c file

[0119] The Report-c file is defined based on the parsing rules of the vehicle's electronic control unit status information and fault code information. The Report-c file is created based on the CDD database and is determined by the vehicle type and the electronic control unit. That is, one electronic control unit of a vehicle type corresponds to one Report-c file. Each Report-c file has a unique identifier, which consists of two diagnostic identifiers: a diagnostic request identifier and a diagnostic response identifier.

[0120] The format of the electronic control unit status information is defined as follows:

[0121] 1) Byte: The starting byte number of the electronic control unit status information in the message. If it does not start from bit 0, the starting bit of the starting byte should also be marked;

[0122] 2) Byte description: status description of the electronic control unit;

[0123] 3) Default value: In the first frame, the second byte indicates the multi-frame byte length, and the third to fifth bytes indicate the electronic control unit status and operation represented by the information;

[0124] 4) Size: the number of bytes of different attributes of the ECU status;

[0125] 5) Escape type / escape name / escape identifier: indicates the format into which the value obtained by the byte should be converted;

[0126] The formats of fault code information are defined as follows:

[0127] Fault code: Fault code identifier, which is the same as the default value in the electronic control unit status information and is a diagnostic identifier;

[0128] Description: Description of the fault code;

[0129] Environmental data / snapshot data: They are different information of fault codes, which are separate files. The format is the same as the electronic control unit status information format. The two attribute values ​​in the fault code point to the links of the corresponding files respectively.

[0130] The Report-c file includes four types of sub-files: ECU status information, fault code information, environmental data and snapshot data. The environmental data and snapshot data are sub-files of the fault code information, which explain the content in more detail.

[0131] 2. Diagnostic Data Analysis Method

[0132] The DBC-c file stores the parsing rules for general messages except diagnostic messages and positioning and navigation messages. The Report-c file stores the parsing rules for diagnostic messages. Positioning and navigation messages are parsed according to the double-precision standard defined in IEEE 754. According to different parsing rules, the messages in the initial ASC file are divided into three categories: diagnostic messages, positioning and navigation messages, and other vehicle messages. The overall parsing flow chart is as follows: Figure 1 shown.

[0133] Step 1: After getting an ASC file, first extract the basic information of the file, including: file name, number of messages in the file, VIN of the car corresponding to the file, store this information, and then traverse each message in the file;

[0134] Step 2: Segment the row data according to the defined attributes, store the message data before parsing, and store the unparsed data to prepare for restoring the initial file by data condition query. Record the message ID of the message;

[0135] Step 3: If it is a diagnostic message, parse and store it using the Report-c file, otherwise execute step 2;

[0136] Step 4: If it is positioning navigation, parse and store it using the IEEE 754 standard, otherwise execute step 5;

[0137] Step 5: If it is other car signals, use DBC-c file to parse and store them.

[0138] (I) Diagnostic message parsing process

[0139] The diagnostic signal is parsed using a Report-c file, which contains a diagnostic request identifier and a diagnostic response identifier. These two identifiers in all Report-c files are centrally managed and placed in the container DiagID. If the message identifier of a message to be parsed belongs to DiagID, then the message is a diagnostic message.

[0140] The diagnostic message data format is identified as a 29-bit extended address, the high byte is fixed at 0x18DA, and the remaining two low bytes represent the destination address and source address respectively. The data information is divided into single frame and multi-frame. When parsing, the diagnostic message is divided into single frame and multi-frame accordingly.

[0141] (1) Single frame

[0142] The single frame information is all contained in one CAN frame. When a message is received, the frame is directly split and parsed. The single frame parsing flow chart is as follows: Figure 2 As shown in the figure, the specific process of single frame analysis includes:

[0143] Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0144] Step 2: The unit of data is byte. Check whether the number of data bytes is equal to the data length. If they are not equal, it means that the frame is wrong and the frame is discarded. Otherwise, go to step 3.

[0145] Step 3: Find the message ID in the DiagID container. If it is a diagnostic request ID, go to step 4. If it is a diagnostic response ID, go to step 5.

[0146] Step 4: Put the request message into the request queue until receiving its response message;

[0147] Step 5: First remove the corresponding request message in the request queue, and then parse the data part of the diagnostic response message according to the Report-c file;

[0148] During the transmission process, due to delays or other reasons, request frames and response frames do not appear in pairs, so a request queue is defined to store all request messages. When a request message receives a response message, the request is removed. The system uses diagnostic identifiers to match request and response messages.

[0149] (2) Multi-frame

[0150] The characteristic of multi-frame is that the information is contained in different CAN frames. When such messages are received, they cannot be processed directly, but are cached. When all CAN frames are received, they are split and parsed. The specific process of multi-frame parsing is as follows:

[0151] Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0152] Step 2: Determine whether it is the first frame according to the network protocol control information type. If it is the first frame, intercept the multi-frame length d and the diagnostic identifier, and put the data part into the pre-defined multi-frame data container List. If it is a continuous frame, execute step 3;

[0153] Step 3: According to the serial frame numbers, the data in the serial frames are stored in the List in order;

[0154] Step 4: If the number of bytes of data stored in List is greater than d, then execute step 5, otherwise continue to receive messages and execute step 1;

[0155] Step 5: At this point, all frames of the response message have been received, and the diagnostic response message is parsed according to the Report-c file;

[0156] When multiple frames are transmitted, the continuous frames received are out of order. An ordered container is set up. Whenever a continuous frame is received, it is inserted into the container according to the number of the continuous frame until the length of the received valid data is equal to the data length in the first frame.

[0157] (II) Positioning and navigation message analysis process

[0158] A message consists of a timestamp, channel number, message identifier, request / response, message type, data length and data part. Whether a message is a positioning and navigation message is determined based on the channel number and message identifier. In the system, messages with a channel number of 9 and a message identifier ranging from 001 to 00A are positioning and navigation messages.

[0159] The specific process of positioning and navigation data analysis is as follows:

[0160] Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0161] Step 2: Determine whether the message is a positioning navigation timestamp message. If so, calculate the time directly. If not, proceed to step 3.

[0162] Step 3: Calculate the longitude and latitude of positioning navigation according to the IEEE 764 standard;

[0163] Step 4: If a positioning and navigation signal has been completely collected, then store the positioning and navigation signal, otherwise continue to receive messages.

[0164] There are two problems in parsing positioning and navigation data:

[0165] (1) When the car is just started, the data is inaccurate. This is because when the car is just started, it needs to initialize the internal data and search for navigation satellites, which leads to abnormal positioning and navigation data when the car is just started.

[0166] (2) If data is accidentally lost during the transmission process, resulting in a large difference in the time distance between the two positioning navigation points, the displayed effect is an unreasonable straight line.

[0167] In order to solve these two problems, limiting filtering is used, in which the maximum deviation value d is set to 0.1, and the maximum acceptable distance difference between adjacent time points is 11 kilometers. All positioning navigation points are traversed to find the first group of adjacent positioning navigation points whose longitude and latitude deviation values ​​are not greater than d, and the positioning navigation points with later time points in the group are set as legal points.

[0168] Taking firstValidPoint as the starting point, the longitude and latitude of two adjacent positioning navigation points are verified respectively. The time of gpsA point is in front and the time of gpsB point is in the back. gpsA has been verified to be a normal point. If the longitude deviation value or latitude deviation value of gpsA and gpsB is greater than or equal to d, gpsB is regarded as an abnormal point, otherwise gpsB is set as a legal point.

[0169] (III) Other automobile news analysis processes

[0170] In addition to the two special signals mentioned above, the vehicle system also has signals such as vehicle speed and door status. The analysis of such signals depends entirely on the DBC-c file.

[0171] The message parsing flow chart other than diagnostic messages and positioning navigation messages is as follows Figure 3 As shown in the figure, the analysis process of other signals except special signals is as follows:

[0172] Process 1: Split the message to obtain timestamp, channel number, message identifier, data length and data attributes;

[0173] Process 2: Find the corresponding DBC file file1 according to the car type obtained from the ASC file and the channel number of the message, and then determine the parsing rule in file1 according to the message identifier. If it cannot be found, it means that the message is wrong and discarded; if the parsing rule is found, then execute process 3;

[0174] Process 3: Parse the message according to the parsing rules and determine whether the message is marked. If marked, execute process 4; otherwise, execute process 5;

[0175] Process 4: Mark the corresponding fields (seconds, minutes, hours) according to the timestamp;

[0176] Process 5: Store the parsed signal data.

[0177] (IV) Solutions to improve data analysis efficiency

[0178] One of the important issues that needs to be solved in data analysis is the speed problem. Based on this problem, this application proposes two solutions.

[0179] Solution 1: Use multi-threading to improve the computing speed: In order to maximize the use of CPU resources, this application proposes to use multi-threaded programming to split a large ASC file into any small files, and process the small files at the same time. The time it takes to parse the smallest file that takes the longest time is the final parsing time, and the large file is divided according to the message time.

[0180] Solution 2: Reduce the frequency of database access: During the data parsing process, the parsing rules used have been stored in the database in advance. Under normal circumstances, each message parsed needs to establish a connection with the database and then query the corresponding parsing rules. An ASC file is the real-time data of a vehicle in a certain period of time. The parsing rules come from the DBC-c files corresponding to different channels. For a known vehicle model, these files are fixed. Therefore, if all the parsing rules of a vehicle model are put into the cache before parsing, and then the corresponding parsing rules are searched from the cache during the parsing process, frequent access to the database can be avoided, and the cache high-speed caching method can be adopted.

[0181] (V) Data analysis test

[0182] The ASC file records the real-time data of the car over a period of time, and parses the message line in the file into a readable signal and signal value. The two solutions proposed in this application to improve the parsing efficiency are experimentally verified respectively.

[0183] (1) Single-threaded and multi-threaded analysis comparison experiment

[0184] Ten files were randomly selected from the vehicle historical data for testing. Each file contained 500,000 messages, and the maximum number of signals included in one message was 39.

[0185] right Figure 4 The ten files in the experiment are divided into N small files according to the time range of the messages in the file (the last file may be smaller than the previous N-1 files). The N small files correspond to N threads. These N small files are parsed at the same time, and the split file time and parsing data time are recorded for comparative analysis. The number of splits N for each ASC file is taken into different values, and ten experiments are conducted respectively, and the average value is finally taken.

[0186] Experiments were conducted on ten random files. The total parsing time of single-threaded parsing was significantly higher than that of multi-threaded parsing. As the number of threads increased, the total parsing time tended to decrease. When the number of threads was greater than 4, the total parsing time tended to increase slightly.

[0187] From the results of the ten experiments, we can see that the efficiency of multi-threaded parsing is significantly better than that of single-threaded parsing. Although multi-threaded parsing requires additional time to split files, the IO operation time for reading and writing files is relatively fixed and has little impact on the total time.

[0188] (2) Comparative experiment between cache and database access

[0189] Five files were randomly selected from the existing vehicle data, and 8 seconds of data were extracted from each file for testing. The number of messages was about 25,000 to 26,000.

[0190] This experiment compares the efficiency of accessing the database and reading after a one-time cache for parsing rules. Two types of experiments are conducted for each file. The first type uses the method of accessing the database. Each time a message is parsed, the database is queried once and the required parsing rules are returned. The second type uses the cache mechanism to do preparatory work before parsing, load the parsing rules into the cache at one time, and traverse to find the required parsing rules during the parsing process. Five files are tested, and each file is tested ten times according to the above two methods. The total parsing time is recorded and the average value is taken as the final result.

[0191] The experimental results are as follows Figure 5 As shown in the figure, all five sets of results show that the time spent accessing the database for each message parsed is much longer than accessing it once and then putting it into the cache. Although the parsing rules are read into the cache once and then a loop query is performed each time, the time spent is significantly reduced compared to accessing a database for each message. The reason why only 8 seconds of data are selected for the test is that the longer the frequent database access time is, the more time spent on accessing the database will increase exponentially. It takes 3000 seconds to parse 8 seconds of data, which is very slow.

[0192] 3. Adaptive integration of data into the database

[0193] Data types can be divided into two categories: real-time data and non-real-time data. Real-time data refers to data generated by vehicle-mounted equipment in real time, which is transmitted in the form of ASC files. The tables involved are those in the real-time data class. Non-real-time data refers to data other than real-time data, including the parsing rules used for parsing and data related to various functions in vehicle equipment. The tables involved are those in the vehicle information class, parsing file class, and functional component class.

[0194] (I) Non-real-time data storage

[0195] Non-real-time data refers to data other than real-time data, including newly added vehicles, parsed files, and functional files.

[0196] (1) Adding a new vehicle: Before parsing an ASC file, first verify the vehicle to which the file belongs. If the vehicle already exists in the system, then parse the data directly. If the vehicle does not exist, that is, the vehicle data has never been imported into the system before, then before parsing, add the vehicle to the system. Adding a new vehicle not only adds a piece of data to the VEHICLE table, but also adds the corresponding MESSAGE and SIGNAL tables to store the vehicle.

[0197] (2) Parsing files: including DBC files and report files. All of these files are loaded into the database before parsing. Among them, Report-c files have been fully loaded before the system is run. The vehicle remote diagnosis system provides users with an interface for uploading DBC files in the file management module. When loading parsing files, in order to improve loading efficiency, a customized batch processing solution is adopted.

[0198] (3) Function file: includes function external connection file and function internal signal file. The import method of this file is the same as that of the parsing file.

[0199] (II) Real-time data storage

[0200] Real-time data refers to data generated by vehicles in real time. These data are transmitted to the data processing system in the form of ASC files. The data processing system needs to start a background process to monitor the specified directory in real time. Once a new ASC file is found, it will be parsed immediately. The parsed message and signal data will be imported into the database in time for use by the data analysis system.

[0201] All segments can import data from gpfdist concurrently and at high speed through external tables, including:

[0202] Process 1: Create gpfdist, create an external table on the Master server, and define the original data source of the external table. The vehicle remote diagnosis system transmits the vehicle data in real time, and creates gpfdist before and after the system runs;

[0203] Process 2: The data node connects to gpfdist to obtain data imported from the external table. If the number of gpfdist URLs is greater than 1, the data node starts the same concurrent call to gpfdist to obtain data.

[0204] Process 3: gpfdist first reads the file, splits the file into multiple blocks and distributes them to the data nodes. The data is redistributed by looking up the table. After each data loading is completed, the external table created in the process is automatically deleted.

[0205] Process 4: Data reading and redistribution are performed simultaneously.

[0206] To implement external data import: First, pass the parsed message and signal file address and the physical table to be inserted as parameters to the stored procedure. During the execution, an external table is created. Then, data is loaded into the physical table through the external table. Finally, when all data is inserted into the physical table, the external table is deleted.

[0207] 4. Optimizing the efficiency of intelligent data query

[0208] (I) Optimization method based on partition table

[0209] To optimize the query, this application uses range partitioning to partition the real-time data table, selecting signal time as the partition basis, including:

[0210] 1) Create a main table corresponding to the real-time data of the car. No data is stored in this table. At the same time, a sub-table is created according to the timestamp. The sub-table is created dynamically, and one minute of data is stored in a sub-table;

[0211] 2) Dynamically create indexes for the subtables. The solution adopted is to create the indexes three minutes after the partitions are created. This can improve query efficiency without affecting data loading speed due to redistribution.

[0212] To verify the optimization degree of partition table query performance, two tables, partitioned and unpartitioned, were tested. The test data in the tables were exactly the same, with a data volume of about 7.07 million. The same query SQL "select * from test where id>3000and id<5000order by timestamp;" was executed ten times. The query results are as follows: Figure 6 As shown, the results show that the query efficiency is improved by 50% after partitioning.

[0213] (II) Optimization method based on tag fields

[0214] Since the car generates data in real time, a new message data is generated every 0.0003 seconds on average, and 500,000 messages are generated every 150 seconds. Each message corresponds to multiple signals, and the data volume is very large. If you operate directly on these data, the efficiency is very low. And considering that the time interval between each two messages is very close, the data difference is almost negligible. In this case, a mark bit is used in the real-time data table, and a total of two are set: mark by second and mark by minute.

[0215] The benefit of this is that the query efficiency is greatly improved within the allowable error range. The specific method is: for each signal, mark the first data within one second ( / minute), there is no interference between different signals, and the marking status of each signal must be recorded. Every time a new second ( / minute) comes, the corresponding status is cleared.

[0216] This method uses a similar data compression method to improve query efficiency by reducing the number of query targets, but this approach will add additional query conditions. To verify the optimization degree of the tag field, the same data table is queried. The first query is all data, the second query is based on the second tag, and the third query is based on the minute tag. Each query is performed ten times, with an average time consumption ratio of 381:58:1. Although the query conditions are increased, the query efficiency is significantly improved due to the significant reduction in the number of queries.

[0217] 5. Database operation test

[0218] (1) Real-time data loading test

[0219] The temporary file BMWcar_sig after ASC file parsing is used as the test object. The file size is 500M, the time length is 0 to 149.214s, and there are 7,076,093 pieces of data. Through Analyze the process of importing real-time data using external tables, the data is successfully imported. The results are as follows: Figure 7 shown.

[0220] It takes about 9 seconds to import the parsed data from the external table. Although it is much shorter than the total time of the data in the file, the imported data will constantly change the tree structure due to the built-in index in the table. As the data increases, the import efficiency will become worse and worse. Considering the import results, the real-time data import solution is feasible, but considering the long-term operation of the system, the database still needs to be further optimized.

[0221] (2) DBC-c file loading test

[0222] DBC-c files are provided by users, and the upload interface is provided directly in the browser for users to use. First, the user selects "DBCFile" in the file type; then, clicks the "ChooseFiles" button and selects the DBC-c file to be uploaded in the pop-up dialog box; the above process is as follows Figure 8 shown.

[0223] After selecting the file to be uploaded, fill in the file configuration information; finally, click the "Submit" button to import the data. According to the import results, the DBC-c file custom batch import algorithm is feasible.

[0224] (3) Report-c file loading test

[0225] As a fixed file, the Report-c file will not be modified during system operation. It needs to be imported into the database before actual operation. There are 24 Report-c files in total, and the time consumption of importing the database at one time is reduced. According to the query results, the rules in the Report-c file are successfully imported, indicating that this import algorithm is feasible and efficient.

Claims

1. The intelligent processing system for automobile remote detection and fault diagnosis data is characterized by: From the three aspects of data analysis, data loading and data storage, this paper analyzes and provides automotive remote data processing solutions. First, according to the automotive remote diagnosis system, the on-board fault diagnosis related protocols, CAN bus encoding format and floating-point operation standards are constructed. Based on the characteristics of different types of on-board data, different signal analysis schemes are designed, and the cache is optimized to improve the analysis speed; secondly, the data processing system establishes a real-time monitoring scheme, automatically monitors the new files transmitted by the on-board equipment and divides them into single-frame and multi-frame parsed diagnostic data, constructs a positioning and navigation message parsing method, and proposes two solutions to improve data parsing efficiency; then, a distributed database is used to store data, and a data adaptive integration and warehousing method is established to meet the efficient loading and warehousing of on-board data. Different types of data use different schemes. Real-time data uses external tables for concurrent and rapid warehousing, and non-real-time data uses batch import to reduce I0 operations; finally, two improvement schemes are proposed to optimize the efficiency of data intelligent query, including an optimization method based on partition tables and an optimization method based on tag fields; 1) Data parsing: Define the parsing rules for ASC files, DBC-c files, and Report-c files, and propose parsing methods for different signal types based on evolutionary computing. The test data parsing solution includes the parsing rules for vehicle-mounted equipment source data and data processing systems. Diagnostic data parsing divides the messages in the initial ASC file into three categories based on different parsing rules: diagnostic messages, positioning and navigation messages, and other vehicle messages, and establishes the parsing process for the corresponding messages. 2) Data loading: Based on the characteristics of different data types, construct data loading methods suitable for different data types. For real-time data, design an external table method, use intermediate temporary files to import data into the database in parallel at one time, and read data concurrently through external tables; non-real-time data refers to data other than real-time data, including parsing rules used for parsing and data related to various functions in vehicle equipment; 3) Data storage: Based on the overall architecture of the Greenplum database, a distributed cluster environment is built, an ER model is established based on the overall automotive remote diagnosis system, and a database system related to the system modules is constructed; indexing, distribution, partitioning methods, and field marking methods are used to improve query speed and system security; Diagnostic data analysis method: Step 1: After getting an ASC file, first extract the basic information of the file, including: file name, number of messages in the file, VIN of the car corresponding to the file, store this information, and then traverse each message in the file; Step 2: Segment the row data according to the defined attributes, store the message data before parsing, store the unparsed data to prepare for restoring the initial file for data condition query, and record the message ID of the message; Step 3: If it is a diagnostic message, parse and store it using the Report-c file, otherwise execute step 2; Step 4: If it is positioning navigation, parse and store it using the IEEE 754 standard, otherwise execute step 5; Step 5: If it is other car signals, use DBC-c file to parse and store them.

2. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Define Report-c file: Define Report-c file based on the parsing rules of automobile electronic control unit status information and fault code information. Report-c file is created based on CDD database and is determined by automobile type and electronic control unit. That is, one electronic control unit of one vehicle type corresponds to one Report-c file. Each Report-c file has a unique identifier, which consists of two diagnostic identifiers: diagnostic request identifier and diagnostic response identifier. The format of the electronic control unit status information is defined as follows: 1) Byte: The starting byte number of the electronic control unit status information in the message. If it does not start from bit 0, the starting bit of the starting byte should also be marked; 2) Byte description: status description of the electronic control unit; 3) Default value: In the first frame, the second byte indicates the multi-frame byte length, and the third to fifth bytes indicate the electronic control unit status and operation represented by the information; 4) Size: the number of bytes of different attributes of the ECU status; 5) Escape type / escape name / escape identifier: indicates the format into which the value obtained by the byte should be converted; The formats of fault code information are defined as follows: Fault code: Fault code identifier, which is the same as the default value in the electronic control unit status information and is a diagnostic identifier; Description: Description of the fault code; Environmental data / snapshot data: They are different information of fault codes, which are separate files. The format is the same as the electronic control unit status information format. The two attribute values ​​in the fault code point to the links of the corresponding files respectively. The Report-c file includes four types of sub-files: ECU status information, fault code information, environmental data and snapshot data. The environmental data and snapshot data are sub-files of the fault code information, which explain the content in more detail.

3. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Diagnostic message parsing process: (1) Single frame The single frame information is all contained in one CAN frame. When a message is received, the frame is directly split and parsed. The specific process of single frame parsing includes: Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes; Step 2: The unit of data is byte. Check whether the number of data bytes is equal to the data length. If they are not equal, it means that the frame is wrong and the frame is discarded. Otherwise, go to step 3. Step 3: Find the message ID in the DiagID container. If it is a diagnostic request ID, go to step 4. If it is a diagnostic response ID, go to step 5. Step 4: Put the request message into the request queue until receiving its response message; Step 5: First remove the corresponding request message in the request queue, and then parse the data part of the diagnostic response message according to the Report-c file; During the transmission process, due to delay or other reasons, the request frame and the response frame do not appear in pairs, so a request queue is defined to store all request messages. When a request message receives a response message, the request is removed. The system uses diagnostic identifiers to match request and response messages. (2) Multi-frame The characteristic of multi-frame is that the information is contained in different CAN frames. When such messages are received, they cannot be processed directly, but are cached. When all CAN frames are received, they are split and parsed. The specific process of multi-frame parsing is as follows: Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes; Step 2: Determine whether it is the first frame according to the network protocol control information type. If it is the first frame, intercept the multi-frame length d and the diagnostic identifier, and put the data part into the pre-defined multi-frame data container List. If it is a continuous frame, execute step 3; Step 3: According to the serial frame numbers, the data in the serial frames are stored in the List in order; Step 4: If the number of bytes of data stored in List is greater than d, then execute step 5, otherwise continue to receive messages and execute step 1; Step 5: At this point, all frames of the response message have been received, and the diagnostic response message is parsed according to the Report-c file; When multiple frames are transmitted, the continuous frames received are out of order. An ordered container is set up. Whenever a continuous frame is received, it is inserted into the container according to the number of the continuous frame until the length of the received valid data is equal to the data length in the first frame.

4. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Positioning and navigation message parsing process: The message consists of a timestamp, channel number, message identifier, request / response, message type, data length and other parts of the data. The channel number and message identifier are used to determine whether a message is a positioning and navigation message. In the system, messages with a channel number of 9 and a message identifier ranging from 001 to 00A are positioning and navigation messages. The specific process of positioning and navigation data analysis is as follows: Step 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes; Step 2: Determine whether the message is a positioning navigation timestamp message. If so, calculate the time directly. If not, proceed to step 3. Step 3: Calculate the longitude and latitude of positioning navigation according to the IEEE 764 standard; Step 4: If a positioning and navigation signal has been completely collected, then store the positioning and navigation signal, otherwise continue to receive messages.

5. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Other car message parsing process: Other message parsing process except diagnostic messages and positioning navigation messages: Process 1: Segment the message to obtain timestamp, channel number, message identifier, data length and data attributes; Process 2: Find the corresponding DBC file file1 according to the car type obtained from the ASC file and the channel number of the message, and then determine the parsing rule in file1 according to the message identifier. If it cannot be found, it means that the message is wrong and discarded; if the parsing rule is found, then execute process 3; Process 3: Parse the message according to the parsing rules and determine whether the message is marked. If marked, execute process 4; otherwise, execute process 5; Process 4: Mark the corresponding fields of seconds, minutes, and hours according to the timestamp; Process 5: Store the parsed signal data.

6. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Solutions to improve data analysis efficiency: Solution 1: Use multithreading to improve the computing speed: Use multithreading programming to maximize the use of CPU resources, split a large ASC file into any small files, and process the small files at the same time. The time it takes to parse the smallest file with the longest parsing time is the final parsing time, and split the large file according to the message time. Solution 2: Reduce the frequency of database access: During the data parsing process, the parsing rules used are stored in the database in advance. Each time a message is parsed, a connection is established with the database and the corresponding parsing rules are queried. An ASC file is the real-time data of a vehicle in a certain period of time. The parsing rules come from the DBC-c files corresponding to different channels. For a known vehicle model, these files are fixed and unchanged. During the parsing process, the corresponding parsing rules are searched from the cache, using a cache high-speed caching method.

7. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Non-real-time data storage: Non-real-time data refers to data other than real-time data, including newly added vehicles, parsing files, and function files; 1) Adding a new vehicle: Before parsing an ASC file, first verify the vehicle to which the file belongs. If the vehicle already exists in the system, parse the data directly; if the vehicle does not exist, that is, the vehicle data has never been imported into the system before, then before parsing, add the vehicle to the system; adding a new vehicle not only adds a piece of data to the VEHICLE table, but also adds the corresponding MESSAGE and SIGNAL tables to store the vehicle; 2) Parsing files: including DBC files and report files. All these files are loaded into the database before parsing. Among them, Report-c files have been fully loaded before the system is run. The vehicle remote diagnosis system provides users with an interface for uploading DBC files in the file management module. When loading parsing files, in order to improve loading efficiency, a custom batch processing solution is adopted; 3) Function file: includes function external connection file and function internal signal file. The import method of this file is the same as that of the parsing file.

8. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Real-time data storage: The data is transmitted to the data processing system in the form of ASC files. The data processing system starts a background process to monitor the specified directory in real time. Once a new ASC file is found, it will be parsed immediately. The parsed message and signal data will be imported into the database in time for use by the data analysis system. All segments can import data from gpfdist concurrently and at high speed through external tables, including: Process 1: Create gpfdist, create an external table on the Master server, and define the original data source of the external table. The vehicle remote diagnosis system transmits the vehicle data in real time, and creates gpfdist before and after the system runs; Process 2: The data node connects to gpfdist to obtain data imported from the external table. If the number of gpfdist URLs is greater than 1, the data node starts the same concurrent call to gpfdist to obtain data. Process 3: gpfdist first reads the file, splits the file into multiple blocks and distributes them to the data nodes. The data is redistributed by looking up the table. After each data loading is completed, the external table created in the process is automatically deleted. Process 4: Reading data and redistribution are performed simultaneously; To implement external data import: First, pass the parsed message and signal file address and the physical table to be inserted as parameters to the stored procedure. During the execution, an external table is created. Then, data is loaded into the physical table through the external table. Finally, when all data is inserted into the physical table, the external table is deleted.

9. The automobile remote detection and fault diagnosis data intelligent processing system according to claim 1 is characterized in that: Optimization method based on partition table: Use range partitioning to partition the real-time data table, and select signal time as the partition basis, including: 1) Create a main table corresponding to the real-time data of the car. No data is stored in this table. At the same time, a sub-table is created according to the timestamp. The sub-table is created dynamically, and one minute of data is stored in a sub-table; 2) Dynamically create indexes for the subtables. The solution adopted is to create the indexes three minutes after the partition is created, which improves query efficiency without affecting data loading speed due to redistribution.

Images