Data consistency checking method, system, equipment and medium

Abstract

The invention relates to a data consistency checking method, system and device and a medium. The method comprises the following steps: obtaining a consistency check demand, and obtaining a first number of vehicle end information records in a preset sampling time period based on the demand; performing down-sampling processing on the first number of vehicle end information records based on a preset time interval to obtain a second number of vehicle end information records; wherein the first number is greater than the second number; calling a cloud record corresponding to the vehicle end information record according to the sampling time period; and determining a corresponding inspection mechanism based on the demand, and performing consistency inspection on the cloud record and each vehicle end information record in the second number of vehicle end information records one by one to obtain an inspection result. According to the invention, the efficiency and accuracy of data consistency check are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a data consistency verification method, system, device and medium. Background Art

[0002] One of the applications of the Internet of Vehicles is to connect the vehicle to the cloud platform through the wireless communication network or cellular network, use sensors to sense the vehicle status information, realize information transmission between the vehicle and the cloud platform, and share vehicle data in real time. The more common function is the application of big data in the Internet of Vehicles: integrating the information uploaded from the vehicle to the cloud to realize the control of the intelligent vehicle status.

[0003] At present, some applications and research methods of IoV big data focus on mass data storage, while others focus on data feature extraction. The stability and reliability acceptance of these IoV big data application research is a prerequisite for ensuring the stability, security and reliability of cloud-based intelligent control. Therefore, there are higher standards for the consistency acceptance of IoV big data upload functions.

[0004] For the Internet of Vehicles big data involving massive amounts of data, the existing test process for testers is: 1. Record the vehicle-side CAN message (Controller Area Network) or Flexray message (FlexRay is an automotive bus technology with fault tolerance); 2. The cloud platform exports the data uploaded by the vehicle; 3. Perform manual testing and comparison to check whether the vehicle-side message data and cloud data are consistent at a fixed time point. The above-mentioned manual testing method has the disadvantages of a small number of data comparisons and a long data comparison time. Therefore, it is necessary to provide a data consistency verification method, system, equipment and medium. Summary of the invention

[0005] In view of the shortcomings of the prior art described above, an object of the present invention is to provide a data consistency verification method, system, device and medium, which improves the problem of low efficiency and accuracy when performing consistency verification between vehicle-side data and cloud data in the prior art.

[0006] To achieve the above-mentioned purpose and other related purposes, the present invention provides a data consistency verification method, including: obtaining a consistency verification requirement, and obtaining a first number of vehicle-side information records within a preset sampling period based on the requirement; based on a preset time interval, downsampling the first number of vehicle-side information records to obtain a second number of vehicle-side information records; wherein the first number is greater than the second number; based on the sampling period, retrieving the cloud record corresponding to the vehicle-side information record; determining the corresponding verification mechanism based on the requirement, and performing consistency verification on the cloud record and each vehicle-side information record in the second number of vehicle-side information records one by one to obtain a verification result.

[0007] In one embodiment of the present invention, the requirement for obtaining a consistency check and obtaining a first number of vehicle-side information records within a preset sampling period based on the requirement include: obtaining a requirement for a consistency check and obtaining a third number of vehicle-side information records within the sampling period based on the requirement; checking the data integrity of each vehicle-side information record, eliminating vehicle-side information records with missing data, and obtaining a first number of vehicle-side information records; wherein the first number is less than or equal to the third number.

[0008] In one embodiment of the present invention, the retrieving of cloud records corresponding to the vehicle-side information records based on the sampling period includes: determining a cloud extraction period based on the sampling period and a preset buffer zone; and retrieving cloud records corresponding to the vehicle-side information records within the cloud extraction period.

[0009] In one embodiment of the present invention, the corresponding inspection mechanism is determined based on the demand, and the cloud record is checked for consistency with each of the second number of vehicle-side information records one by one to obtain the inspection result, including: determining the corresponding inspection mechanism based on the category of the demand: wherein each category of demand is pre-corresponding to a inspection mechanism; if the inspection mechanism is an inspection mechanism based on the success rate, the inspection success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record is calculated, and the inspection result is generated based on the success rate; if the inspection mechanism is an inspection mechanism based on the attribute, the inspection success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record is calculated, and the inspection result is obtained based on the inspection success rate and the abnormal situation of the corresponding attribute of the cloud record.

[0010] In one embodiment of the present invention, the calculation of the verification success rate of each category of attributes in the second number of vehicle-side information records and the corresponding attributes in the cloud records, and the generation of verification results based on the success rate, includes: extracting all categories of attributes from the second number of vehicle-side information records; wherein each vehicle-side information record includes multiple different categories of attributes; for all attributes of the same category: performing consistency checks on each attribute of the category with the attributes of the corresponding category in the cloud records, and recording the number of attributes that have successfully been verified; calculating the verification success rate of the attributes of the category based on the number of attributes that have successfully been verified and the second number; comparing the verification success rate with a preset first success rate threshold, and when it is greater than the success rate threshold, marking the attributes of the category as verified attributes; generating detection results based on all successfully verified attributes in the second number of vehicle-side information records.

[0011] In one embodiment of the present invention, the verification success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute in the cloud record is calculated, and the verification result is obtained based on the verification success rate and the abnormal situation of the corresponding attribute in the cloud record, including: extracting all categories of attributes from the second number of vehicle-side information records; wherein each vehicle-side information record includes multiple attributes of different categories; for all attributes of the same category: identifying whether the attribute of the category in the cloud record is an abnormal value; wherein the abnormal value includes a null value and a value without a character; if so, marking the attribute of the category as an attribute that failed the verification; otherwise, performing a consistency check on each attribute of the category with the attribute of the corresponding category in the cloud record, and calculating the verification success rate of the attribute of the category; and comparing the verification success rate with a preset second success rate threshold, and when it is greater than the second success rate threshold, marking the attribute of the category as an attribute that succeeded in the verification; generating a detection result based on all the attributes that succeeded in the verification in the second number of vehicle-side information records.

[0012] In one embodiment of the present invention, the corresponding verification mechanism is determined based on the demand, and the cloud record is checked for consistency with each of the second number of vehicle-side information records one by one. After the verification result is obtained, it also includes: visually presenting the information in the verification result.

[0013] In one embodiment of the present invention, a data consistency verification system is also provided, which includes: a vehicle-side record acquisition module, used to obtain the requirement for consistency verification, and obtain a first number of vehicle-side information records within a preset sampling period based on the requirement; a data cleaning module, used to downsample the first number of vehicle-side information records based on a preset time interval to obtain a second number of vehicle-side information records; wherein the first number is greater than the second number; a cloud record acquisition module, used to retrieve the cloud record corresponding to the vehicle-side information record based on the sampling period; and a verification module, used to determine the corresponding verification mechanism based on the requirement, and perform consistency verification on the cloud record and each vehicle-side information record in the second number of vehicle-side information records one by one to obtain a verification result.

[0014] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements any of the data consistency verification methods described above.

[0015] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the above-mentioned data consistency verification methods.

[0016] As described above, a method, system, device and medium for verifying the consistency of data of the present invention have the following beneficial effects: when verifying data, a large number of vehicle-side information records within a preset sampling period are obtained according to the requirements of consistency verification, and the amount of data is reduced by downsampling so that the number of vehicle-side information records and cloud records are at the same level. Cloud records within the same period are obtained in the cloud to ensure the synchronization of cloud records with vehicle-side record data. The corresponding verification mechanism is determined according to the requirements, and the cloud records and the vehicle-side information records after downsampling are verified for consistency one by one according to the verification mechanism. The most suitable verification method can be selected according to different requirements, thereby improving the data processing rate and the accuracy of the verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of a data consistency verification method provided by an embodiment of the present invention;

[0018] Figure 2 Displayed is a visual interface of the data consistency verification method described in the present invention;

[0019] Figure 3 Shown is a visual schematic diagram of the test results of the present invention;

[0020] Figure 4 Shown is a structural block diagram of a data consistency verification system provided by an embodiment of the present invention;

[0021] Figure 5 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0024] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0025] The inventors found that the current method for verifying the function of big data in the Internet of Vehicles still uses manual data walkthrough for testing. However, this method has the following shortcomings in testing efficiency: testers are required to go through each signal one by one and change each signal. With the increase in the amount of big data signals uploaded on the vehicle side (such as 2000 groups), this manual comparison method has significantly increased the workload and led to an increase in labor costs. In order to meet the current requirements of agile development, testers only compare the consistency of the signal once within a limited time, and cannot perform long-term and multiple stress tests. As a result, occasional problems take a long time to be exposed, which is not conducive to the control of software quality. Therefore, seeking a more efficient data comparison method is a technical problem that needs to be overcome at present.

[0026] The present invention provides a method for verifying the consistency of data. When verifying the data, a large number of vehicle-side information records within a preset sampling period are obtained according to the needs of the consistency check, and the amount of data is reduced by downsampling so that the number of vehicle-side information records and cloud records are at the same level. Cloud records within the same period are obtained in the cloud to ensure the synchronization of cloud records with vehicle-side record data. A corresponding verification mechanism is determined according to the needs, and consistency checks are performed one by one on the cloud records and the vehicle-side information records after downsampling according to the verification mechanism. The most suitable verification method can be selected according to different needs, thereby improving the data processing rate and the accuracy of the verification.

[0027] See also Figure 1 ,The data consistency verification method includes the following steps:

[0028] S1. Obtaining a consistency check requirement, and based on the requirement, obtaining a first number of vehicle-side information records within a preset sampling period.

[0029] When it is necessary to perform consistency check on the data, the sampling period and sampling frequency are determined according to the current consistency check requirements, and the vehicle-side information records within the sampling period are extracted from the vehicle-side according to the sampling frequency to obtain the first number of vehicle-side information records. Each vehicle-side information record has a sampling timestamp. It should be noted that the first number is the total amount of original vehicle-side information records extracted within the sampling period. Due to the high sampling frequency, the first number is very large. It can be understood that the vehicle-side information record refers to the message data transmitted through the internal network of the vehicle, wherein the type of internal network includes but is not limited to CAN bus or FlexRay bus, and each vehicle-side information record records the current operating status of the vehicle and related sensor data and other information.

[0030] In an embodiment of the present invention, the step of obtaining a consistency check requirement and obtaining a first number of vehicle-side information records within a preset sampling period based on the requirement includes:

[0031] Obtaining a consistency check requirement, and obtaining a third number of vehicle-side information records within the sampling period based on the requirement;

[0032] Check the data integrity of each vehicle-side information record, eliminate the vehicle-side information records with missing data, and obtain a first number of vehicle-side information records; wherein the first number is less than or equal to the third number.

[0033] According to the current consistency test requirements, the identification of the vehicle-side information record that needs to be tested is determined, and within the preset sampling period, all vehicle-side information records with the identification are collected from the vehicle-side to obtain the third number of vehicle-side information records. Among them, the type of identification includes but is not limited to the message ID or the type of data to be analyzed, etc., as long as the corresponding vehicle-side information record can be obtained according to the identification, it is not limited here. Since the third number of vehicle-side information records collected are original records, there may be data missing. In order to ensure the accuracy of subsequent data processing, it is necessary to perform integrity checks on these data to ensure that each vehicle-side information record contains necessary data fields. It should be noted that due to the presence of abnormal values ​​in the vehicle-side signal during the test process in the vehicle development stage, the abnormal values ​​are not processed. In the above manner, after deleting the records with missing data, all remaining vehicle-side information records are the first number of vehicle-side information records. Due to the existence of deleted records, the first number will be less than or equal to the third number.

[0034] S2. Based on a preset time interval, downsample the first number of vehicle-side information records to obtain a second number of vehicle-side information records; wherein the first number is greater than the second number.

[0035] Since the cycle of the vehicle-side signals (such as CAN signals or FlexRay signals) currently processed is in the millisecond level, while the vehicle-side signals reported by the cloud data are in the second level, the two are of different magnitudes. In order to make a consistency comparison, it is necessary to ensure that the two are at the same magnitude. Therefore, it is necessary to downsample the first number of vehicle-side information records to reduce the amount of data. Specifically, a time interval (i.e., sampling frequency) may be set based on specific needs, such as a sampling frequency of once every 100 milliseconds. Since the first number of vehicle-side information records are arranged in sequence according to the collection time, each vehicle-side information record may be traversed according to the time interval, starting from the first vehicle-side information record collected, as the current record, and the difference between the timestamp of the traversed vehicle-side information record and the timestamp of the current record is calculated, and the difference is compared with the preset time interval: if the difference is greater than or equal to the time interval, the traversed record is retained as the downsampling result, and the timestamp of the current record is updated to the timestamp of the record. Otherwise, the record is skipped, and the next vehicle-side information record is analyzed until all the first number of vehicle-side information records are traversed to obtain the second number of vehicle-side information records. By downsampling, the number of records is greatly reduced, so that the second number is kept at the same order of magnitude as the number of corresponding vehicle-side information records on the cloud. For example, python cantools may be used to load the CAN DBC file of the original signal of the vehicle-side for parsing, extract the original collected data at intervals of 100ms, and save it in an SQL database with the file name Bigdata1<Timer1,A,B,C,D…> , where A, B, C, and D are the attribute information in the record, and Timer1 is the sampling period.

[0036] S3. According to the sampling period, retrieve the cloud record corresponding to the vehicle-side information record.

[0037] Specifically, in one embodiment of the present invention, retrieving the cloud record corresponding to the vehicle-side information record according to the sampling period includes:

[0038] Determining a cloud extraction period according to the sampling period and a preset buffer zone;

[0039] During the cloud extraction period, the cloud record corresponding to the vehicle-side information record is retrieved.

[0040] Taking into account the existence of time deviations, network delays and other factors during data transmission, the cloud data and the vehicle-side data may not be completely synchronized in time. To improve this problem, the present invention presets a buffer zone, and obtains an extended sampling period through the buffer zone and the preset sampling period as the cloud extraction period. Exemplarily, if the sampling period is from 10:00 to 10:01, and the buffer zone is ±0.1 seconds, then the cloud extraction period is from 9:59:59 to 10:01:01. During this cloud extraction period, the cloud records corresponding to the vehicle-side information records are extracted based on the timestamp and corresponding identifier of the cloud records. Exemplarily, log in to the cloud data platform, export the cloud records uploaded during the cloud extraction period corresponding to the collected data, and upload them to the automated comparison tool for storage. Among them, the data attributes in Bigdata2 include<ID,TimerI,A,B,C,D> , where the upper and lower limits of TimerI are the difference between the sampling period and the cache interval timecount, and the sum of the sampling period and the buffer interval timecount, that is, TimerI = [Timer1-timecount, Timer1+timecount]. It should be noted that the specific data between the buffers can be adaptively set, and can be set in a dynamically variable form so that the user can make adaptive adjustments based on actual needs. For example, the range between the buffers is [0.01, 0.99].

[0041] S4. Determine a corresponding verification mechanism based on the demand, and perform consistency verification on the cloud record and each vehicle-side information record in the second number of vehicle-side information records one by one to obtain a verification result.

[0042] Determine the corresponding inspection mechanism according to the current consistency inspection requirements, where the inspection mechanism includes a success rate-based inspection mechanism and an attribute-based inspection mechanism. For each vehicle-side information record: compare it with the cloud record, and obtain the consistency inspection result of the record by comparing the data of the relevant attributes in the two records. After all the second number of vehicle-side information records are compared, summarize all consistency inspection results to obtain the final inspection result.

[0043] Specifically, in one embodiment of the present invention, the corresponding verification mechanism is determined based on the demand, and the cloud record is verified for consistency with each of the second number of vehicle-side information records one by one to obtain a verification result, including:

[0044] Determine a corresponding inspection mechanism based on the category of the requirement: wherein each category of the requirement corresponds to a inspection mechanism in advance;

[0045] If the verification mechanism is a verification mechanism based on success rate, calculating the verification success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record, and generating a verification result based on the success rate;

[0046] If the inspection mechanism is an attribute-based inspection mechanism, the inspection success rate of each category attribute in the second number of vehicle-side information records and the corresponding attributes of the cloud records is calculated, and the inspection result is obtained based on the inspection success rate and the abnormal situation of the corresponding attributes of the cloud records.

[0047] According to the category of the current consistency test requirement, the corresponding test mechanism is determined. Exemplarily, if some data requires an accurate match of the attribute data, an attribute-based test mechanism can be selected, and if some data requires an overall consistency test, an attribute-based test mechanism can be selected. It can be understood that the category of each requirement is determined by comparing the correct answer rates of the two test mechanisms after repeated tests in advance. The corresponding test mechanism of the category is determined. After determining the test mechanism, the corresponding test process is selected according to the mechanism. For example, when the test mechanism is a test mechanism based on the success rate, for each vehicle-side information record: by comparing the success rate of each attribute data and the corresponding attribute data in the cloud record, the overall test result of the second number of vehicle-side information records is obtained. When the test mechanism is an attribute-based test mechanism, for each vehicle-side information record: not only will the success rate of the attribute data and the corresponding attribute data in the cloud record be compared, but also whether there is a preset abnormality in each attribute data in the cloud record, thereby obtaining the overall test result of the second number of vehicle-side information records.

[0048] Specifically, in one embodiment of the present invention, the calculating of the verification success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record, and generating the verification result based on the success rate, includes:

[0049] Extracting all types of attributes from the second number of vehicle-side information records; wherein each vehicle-side information record includes multiple different types of attributes;

[0050] For all attributes of the same category:

[0051] Perform consistency check on each attribute of the category with the attributes of the corresponding category in the cloud record, and record the number of attributes that have been successfully checked;

[0052] Calculating a successful inspection rate of the attributes of the category according to the number of attributes that have been successfully inspected and the second number;

[0053] Compare the inspection success rate with a preset first success rate threshold, and when the success rate is greater than the success rate threshold, mark the attribute of the category as an attribute of successful inspection;

[0054] Generate a test result based on all successfully tested attributes in the second number of vehicle-side information records.

[0055] When the verification mechanism is a verification mechanism based on success rate, for the second number of vehicle-side information records, first, the data needs to be read one by one to obtain multiple attributes of different categories (such as vehicle speed, engine speed, etc.) contained in each record, and these attributes are stored according to the corresponding categories to ensure that the attributes of the same category form an attribute sequence according to the timestamp. When comparing, the attribute in the cloud record is compared with each attribute in the attribute sequence respectively, and by comparing whether the cloud attribute value and the vehicle-side attribute value are consistent, it is determined whether the two are successfully verified: if they are consistent, it means that the cloud attribute value and the current vehicle-side attribute value match, and the verification is successful, otherwise, it means that the two do not match and the verification is unsuccessful. After all attributes in the attribute sequence of the same category are processed, the ratio of the number of attributes that have been successfully verified to the second number is calculated to obtain the verification success rate of the attributes of this category. When the verification success rate is greater than the preset first success rate threshold of the category (such as 70%), it means that the attribute verification of this category is successful, otherwise, the attribute verification of this category fails. After the attributes of all categories are verified, all the inspection results are summarized to obtain the final inspection result. It should be noted that the final inspection result may include the success rate of each category attribute, whether the inspection is successful, and possible problems, etc. It is understandable that the first success rate threshold can be set for attributes of different categories, or all categories of attributes can be set to the same success rate threshold, which is not limited here.

[0056] In one embodiment of the present invention, the calculating of the verification success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record, and obtaining the verification result according to the verification success rate and the abnormality of the corresponding attribute of the cloud record, includes:

[0057] Extracting all types of attributes from the second number of vehicle-side information records; wherein each vehicle-side information record includes multiple different types of attributes;

[0058] For all attributes of the same category:

[0059] Identify whether the attribute of the category in the cloud record is an abnormal value; wherein the abnormal value includes a null value and a value without a character;

[0060] If yes, then mark the attribute of this category as an attribute that fails the test;

[0061] Otherwise, each attribute of the category is checked for consistency with the attributes of the corresponding category in the cloud record, and the success rate of the check of the attributes of the category is calculated; and

[0062] Compare the inspection success rate with a preset second success rate threshold, and when the success rate is greater than the second success rate threshold, mark the attribute of the category as an attribute of successful inspection;

[0063] Generate a test result based on all successfully tested attributes in the second number of vehicle-side information records.

[0064] When the inspection mechanism is an attribute-based inspection mechanism, for the second number of vehicle-side information records, first, the data needs to be read one by one to obtain multiple attributes of different categories (such as vehicle speed, engine speed, etc.) contained in each record, and these attributes are stored according to the corresponding categories to ensure that the attributes of the same category form an attribute sequence according to the timestamp. During the inspection, first check whether the attribute of the category in the cloud record is an abnormal value. If there is an abnormal value such as a null value or a character value, the attribute of the category is marked as a test failure, and the subsequent inspection process is not performed. If there is no abnormal value, for each attribute in the attribute sequence of the category: compare the attribute value with the corresponding attribute value in the cloud to determine whether the two are consistent, so as to determine whether the two are successfully inspected. After all attributes in the attribute sequence are analyzed, the ratio of the number of successful inspections in the attribute sequence to the second number is calculated as the inspection success rate of the attribute of the category. If the success rate is greater than the second success rate (such as 90%) threshold preset for the category, the attribute of the category is marked as a successful inspection. After the inspection of all categories of attributes is completed, all inspection results are summarized to obtain the final inspection result. It should be noted that the final inspection result may include the success rate of each category attribute, whether the inspection is successful, and possible problems, etc. It is understandable that the second success rate threshold can be set for attributes of different categories, or all categories of attributes can be set to the same success rate threshold, which is not limited here.

[0065] Furthermore, in order to further improve the detection efficiency, the present invention can also distribute the corresponding inspection tasks to multiple nodes or servers for parallel processing through a distributed architecture according to the detection requirements. Specifically, before performing the consistency check, the operation status of each node will be evaluated based on the information sent back by each node, and the second number of vehicle-side information records can be distributed to at least two nodes based on the priority, requirements and other information of the task. Each node can perform a consistency check of a part of the attributes in the vehicle-side information record, or different nodes can perform a consistency check of different vehicle-side information records. After all records are verified, the results of each node are summarized to obtain the final inspection result.

[0066] In an embodiment of the present invention, the corresponding verification mechanism is determined based on the demand, and the cloud record is verified for consistency with each of the second number of vehicle-side information records one by one, and after the verification result is obtained, it also includes: visually presenting the information in the verification result. The generated verification result is presented on the user interface in the form of a visual chart, so as to facilitate viewing and analysis.

[0067] Furthermore, in order to improve operational efficiency and simplify user operations, the method of the present invention is developed by combining Python with HeidiSql. HeidiSQL is an open source graphical interface for simplifying mini server and database management. HeidiSQL provides an easy-to-use interface with syntax highlighting for switching SQL queries and tags between database browsing. Desktop development based on python and SQL can provide a visual interface for testers. It has the characteristics of no installation and easy operation. The detailed interface of the tool is shown as follows Figure 2 As shown in the Internet of Vehicles Big Data Comparison Tool. Figure 3 As shown in the figure, this tool mainly includes the following functions: 1) Upload the original vehicle-side CAN message / Flexary message. The tester needs to use the CAN data acquisition device to export the original signal of the vehicle-side and save it as a blf format file. Click Figure 2 Select the blf file and upload it; 2) Load the DBC file on the vehicle side as follows Figure 2 As shown, select the FR database and upload the vehicle-side DBC file; 3) The cloud data file can be exported on the cloud platform and uploaded to the big data comparison tool, such as Figure 2 . As shown; 4) Signal requirement matrix table (ie Figure 2 Upload the signal table as Figure 2 As shown, select the signal table to upload; 5) The user can press the "Delete" key to delete the uploaded content box; 6) The consistency check method, select Figure 2 7) Select the consistency test method. When selecting the consistency test method 1 described above, check "Success rate priority". When selecting the consistency test method 2, do not check it. Figure 2 8) The consistency test progress bar is as shown in Figure 2 9) Adjustment of the time parameter of “cloud-side and vehicle-side time difference”. In the process of developing the automatic comparison tool, the original data collection time and the cloud collection time are inconsistent. Therefore, the present invention can manually adjust the time parameter for this situation, such as Figure 2The “time difference between the cloud and the vehicle” is used to calibrate the time to ensure that the time uploaded to the cloud and the actual vehicle-side signal acquisition time remain accurate. The adjustment range of the calibration parameters can be manually entered; 10) In order to increase the sample comparison amount of each attribute such as A, the present invention provides an adjustable parameter input box function of “verification time deviation value” such as Figure 2 As shown, the adjustable time parameter, i.e., the parameter timecount mentioned in the step, can be automatically input or adjusted in steps of 0.01, ranging from [-0.99, 0.99]; 11) For the time length of the consistency check, the present invention provides an editable time input box (start time and end time), refer to Figure 2 As shown, the tester can select the data to be compared according to the time when the test case was executed. 12) In order to quickly query the location of the test results, this interface provides users with personalized prompts to guide the tester to query the test results. The guide language is as follows: Figure 2 As shown in the figure: "The test report has been generated", the test report result query is as follows Figure 3 shown.

[0068] Specifically, Figure 3 The relevant information is as follows: PortName is the identifier of the vehicle-side information record, Collect period is the sampling period, and Data Element is the attribute in the record. Figure 3 It can be seen that all attributes (such as WhlMotSysIdc, IsgModSts, HvBattLmnIndcn, etc.) failed the consistency check because the success rate was lower than the preset threshold.

[0069] In summary, compared with manual testing, the present invention releases the current test resources: CAN data acquisition tool. The manual comparison method requires the use of CAN data acquisition tools and computers to perform real-time / offline playback of the CAN signal of the vehicle end, and then query the signal reported on the cloud at the current time point for comparison one by one. The tool implemented by the present invention only requires one computer to run, and no other equipment is required during the data consistency comparison process. And the current automated comparison tool is a lightweight offline tool developed based on Python language and hedisql database, which occupies less memory and runs fast. Python language provides a wealth of data processing tools such as Python cantools (i.e. CAN bus toos, a CAN bus tool), which can load DBC (Database Container) files (CAN data definition files) and perform offline parsing. Heidisql does not need to be connected to the Internet, can store data locally, is more convenient, faster, and supports SQL language and other characteristics. Compared with manual testing, the test time is shortened. In terms of manual comparison test efficiency, it takes about 20 days to compare the consistency of 2000 signals. Using automated comparison tools saves 15 days of work time and improves test efficiency.

[0070] See also Figure 4 , the data consistency verification system 100 includes: a vehicle-side record acquisition module 110, a data cleaning module 120, a cloud record acquisition module 130 and a verification module 140. The above-mentioned vehicle-side record acquisition module 110 is used to obtain the demand for consistency verification, and obtain a first number of vehicle-side information records within a preset sampling period based on the demand. The data cleaning module 120 is used to downsample the first number of vehicle-side information records based on a preset time interval to obtain a second number of vehicle-side information records; wherein the first number is greater than the second number. The cloud record acquisition module 130 is used to retrieve the cloud record corresponding to the vehicle-side information record according to the sampling period. The verification module 140 is used to determine the corresponding verification mechanism based on the demand, and perform consistency verification on the cloud record and each vehicle-side information record in the second number of vehicle-side information records one by one to obtain a verification result.

[0071] For the specific definition of the data consistency verification system, please refer to the definition of the data consistency verification method above, which will not be repeated here. Each module in the above data consistency verification system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0072] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0073] See also Figure 5 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a data consistency check program.

[0074] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as codes for data consistency verification, etc., but can also be used to temporarily store data that has been output or is to be output.

[0075] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules (such as data consistency check programs, etc.) stored in the memory 12, and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0076] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above data consistency verification method.

[0077] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a vehicle-side record acquisition module 110, a data cleaning module 120, a cloud record acquisition module 130, and a verification module 140.

[0078] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the data consistency verification method described in each embodiment of the present application.

[0079] In summary, the present invention discloses a method, system, device and medium for consistency verification of data, which improves the problems of low efficiency and long problem discovery cycle of manual consistency comparison of big data. The method of the present invention has the functions of cleaning and storing the original data, and automatically generating a result report after consistency verification. During the verification, the vehicle-side information records are sampled and extracted at preset time intervals to achieve data cleaning. The original vehicle-side information records are stored in a data table in the form of key-value pairs and data dictionaries. The downloaded cloud upload data is compared with each original information record on the vehicle side for consistency, and the comparison results are automatically output. The whole process does not require manual intervention, is easy to operate, significantly improves the efficiency of data consistency verification, and can quickly output consistency comparison results. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0080] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A data consistency verification method, characterized in that: The method comprises: Obtaining a consistency check requirement, and obtaining a first number of vehicle-side information records within a preset sampling period based on the requirement; Based on a preset time interval, down-sampling the first number of vehicle-side information records to obtain a second number of vehicle-side information records; wherein the first number is greater than the second number; According to the sampling period, retrieve the cloud record corresponding to the vehicle-side information record; A corresponding verification mechanism is determined based on the demand, and consistency checks are performed one by one between the cloud record and each vehicle-side information record in the second number of vehicle-side information records to obtain a verification result.

2. The data consistency verification method according to claim 1, characterized in that: The obtaining of the consistency check requirement, and obtaining a first number of vehicle-side information records within a preset sampling period based on the requirement, includes: Obtaining a consistency check requirement, and obtaining a third number of vehicle-side information records within the sampling period based on the requirement; Check the data integrity of each vehicle-side information record, eliminate the vehicle-side information records with missing data, and obtain a first number of vehicle-side information records; wherein the first number is less than or equal to the third number.

3. The data consistency verification method according to claim 1, characterized in that: The retrieving the cloud record corresponding to the vehicle-side information record according to the sampling period includes: Determining a cloud extraction period according to the sampling period and a preset buffer zone; During the cloud extraction period, the cloud record corresponding to the vehicle-side information record is retrieved.

4. The data consistency verification method according to claim 1, characterized in that: The determining a corresponding verification mechanism based on the demand, performing consistency verification on the cloud record and each vehicle-side information record in the second number of vehicle-side information records one by one, and obtaining a verification result, includes: Determine a corresponding inspection mechanism based on the category of the requirement: wherein each category of the requirement corresponds to a inspection mechanism in advance; If the verification mechanism is a verification mechanism based on success rate, calculating the verification success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record, and generating a verification result based on the success rate; If the inspection mechanism is an attribute-based inspection mechanism, the inspection success rate of each category attribute in the second number of vehicle-side information records and the corresponding attributes of the cloud records is calculated, and the inspection result is obtained based on the inspection success rate and the abnormal situation of the corresponding attributes of the cloud records.

5. The data consistency verification method according to claim 4, characterized in that: The calculating the verification success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record, and generating the verification result based on the success rate, includes: Extracting all types of attributes from the second number of vehicle-side information records; wherein each vehicle-side information record includes multiple different types of attributes; For all attributes of the same category: Perform consistency check on each attribute of the category with the attributes of the corresponding category in the cloud record, and record the number of attributes that have been successfully checked; Calculating a successful inspection rate of the attributes of the category according to the number of attributes that have been successfully inspected and the second number; Compare the inspection success rate with a preset first success rate threshold, and when the success rate is greater than the success rate threshold, mark the attribute of the category as an attribute of successful inspection; Generate a test result based on all successfully tested attributes in the second number of vehicle-side information records.

6. The data consistency verification method according to claim 4, characterized in that: The calculating of the verification success rate of each category attribute in the second number of vehicle-side information records and the corresponding attribute of the cloud record, and obtaining the verification result according to the verification success rate and the abnormality of the corresponding attribute of the cloud record, includes: Extracting all types of attributes from the second number of vehicle-side information records; wherein each vehicle-side information record includes multiple different types of attributes; For all attributes of the same category: Identify whether the attribute of the category in the cloud record is an abnormal value; wherein the abnormal value includes a null value and a value without a character; If yes, then mark the attribute of this category as an attribute that fails the test; Otherwise, each attribute of the category is checked for consistency with the attributes of the corresponding category in the cloud record, and the success rate of the check of the attributes of the category is calculated; and Compare the inspection success rate with a preset second success rate threshold, and when the success rate is greater than the second success rate threshold, mark the attribute of the category as an attribute of successful inspection; Generate a test result based on all successfully tested attributes in the second number of vehicle-side information records.

7. The data consistency verification method according to claim 1, characterized in that: The corresponding inspection mechanism is determined based on the demand, and the cloud record is checked for consistency with each of the second number of vehicle-side information records one by one. After the inspection result is obtained, it also includes: visually presenting the information in the inspection result.

8. A data consistency verification system, characterized in that: The system comprises: A vehicle-side record acquisition module, used to obtain a consistency check requirement, and based on the requirement, obtain a first number of vehicle-side information records within a preset sampling period; A data cleaning module, configured to perform downsampling processing on the first number of vehicle-side information records based on a preset time interval to obtain a second number of vehicle-side information records; wherein the first number is greater than the second number; A cloud record acquisition module, used to retrieve the cloud record corresponding to the vehicle-side information record according to the sampling period; The inspection module is used to determine the corresponding inspection mechanism based on the demand, and to perform consistency inspection on the cloud record and each vehicle-side information record in the second number of vehicle-side information records one by one to obtain the inspection result.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the data consistency verification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the data consistency verification method described in any one of claims 1 to 7.

Images