An automobile extended warranty multi-source heterogeneous data cleaning and evaluation system

Abstract

The application discloses a kind of automobile extended warranty multi-source heterogeneous data cleaning evaluation systems, it is related to data cleaning evaluation technical field, to solve the problem of low data cleaning efficiency, by integrating the data to be cleaned, calling each data entry time and adding its record location to additional information by information guidance, processing and setting abnormal time period to entry time, according to abnormal time period selection mark data, detect the entry and storage condition of each marked data, mark different abnormal state by comprehensively entering condition and storage condition, count the abnormal state quantity of each marked data and calculate the cleaning decision value of the data set to be cleaned, set decision rule and generate decision result after cleaning decision value is input into decision rule, different cleaning methods are selected based on decision result to clean the data set to be cleaned, so as to improve the cleaning efficiency while reducing the cleaning cost, guarantee the accuracy and integrity of data quality.

Description

Technical Field

[0001] This invention relates to the field of data cleaning and evaluation technology, and more specifically, to a multi-source heterogeneous data cleaning and evaluation system for automotive extended warranty. Background Technology

[0002] With the rapid development of automotive extended warranty services and after-sales management systems, a large amount of business data from different sources and types is collected and stored, including customer information, vehicle information, maintenance records, etc. This data usually exhibits multi-source heterogeneity, with differences in data format, entry time, and storage method among different data sources. Existing data cleaning methods are mostly targeted at a single data source or use fixed rules to remove outliers and correct formats.

[0003] The existing technology has the following shortcomings:

[0004] Currently, existing technologies only perform cleaning processing on a single data source or rely on fixed rules to remove abnormal data. They lack the ability to uniformly integrate multi-source heterogeneous data and dynamically identify anomalies. They cannot accurately identify various types of abnormal data and generate intelligent cleaning decisions, resulting in reduced data cleaning efficiency and increased data quality risks. This affects the data quality of extended warranty business and subsequent analysis and applications. Therefore, a multi-source heterogeneous data cleaning and evaluation system for automotive extended warranty is proposed.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multi-source heterogeneous data cleaning and evaluation system for automotive extended warranty, which solves the problems mentioned in the background art by using multi-source heterogeneous data integration, data entry time correlation analysis, and intelligent cleaning decision-making mechanism.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multi-source heterogeneous data cleaning and evaluation system for automotive extended warranty, comprising a data integration module, a recording and judgment module, a data detection module, and a cleaning and evaluation module, wherein the modules are electrically connected to each other.

[0008] The data integration module merges the data to be cleaned into a dataset to be cleaned, calls the entry time of different data in the dataset to be cleaned, sets information guidance to add the record position of the entry time to the additional information of each data in the dataset to be cleaned, and passes the dataset to be cleaned into the record judgment module.

[0009] The recording and judgment module identifies each data in the dataset to be cleaned and calls the information in the additional information of each data to obtain the corresponding entry time. Based on the entry time, it sets abnormal recording time segments, selects and marks each data in the dataset to be cleaned according to the abnormal time segments, and passes the marked data to the data detection module.

[0010] The data detection module detects the input and storage status of each labeled data, identifies different abnormal states based on the input and storage status, counts the number of abnormal states of each labeled data, calculates the cleaning decision value of the dataset to be cleaned, and sends the cleaning decision value of the dataset to be cleaned to the cleaning evaluation module.

[0011] The cleaning assessment module generates decision results by setting decision rules and feeding the received cleaning decision values ​​into the decision rules. Based on the decision results, different cleaning methods are selected to clean the dataset to be cleaned.

[0012] In a preferred embodiment, before integrating the data to be cleaned, the data integration module identifies the car extended warranty service folder selected by the user and obtains the data contained in the folder as the data to be cleaned.

[0013] The setting information guide is used to temporarily store the record position of each data entry time in the dataset to be cleaned. The information guide set for the record position of each data entry time is added to the additional information of the corresponding data.

[0014] In a preferred embodiment, after the recording and judgment module obtains the storage location information of the entry time of each data in the dataset to be cleaned, it jumps to the storage location and calls the entry time corresponding to each data.

[0015] The K-nearest neighbor algorithm is used to set time segments for abnormal records based on the entry time.

[0016] In a preferred embodiment, the K-nearest neighbor algorithm is used to set the time segments for abnormal records as follows:

[0017] Merging data: The entry time corresponding to each data point in the dataset to be cleaned is used as the time dataset;

[0018] Set the K value: Use the median of the data in the time dataset as the K value;

[0019] Dataset sorting: Sort the entry times in chronological order and calculate the time difference between adjacent entry times;

[0020] Calculate the sample distance: Set the time adjustment ratio, and use the product of the time adjustment ratio and the maximum time difference between adjacent input times as the sample distance of the time dataset;

[0021] K-value screening: Using the K-value as the center point, compare the distance between the center point and its two adjacent points with the sample distance. If the distance is less than the sample distance, then use the left and right adjacent points as two new center points and perform a new round of comparison with the left or right adjacent points.

[0022] Set a stopping condition: stop traversing when the distance between the center point and its left or right neighboring point exceeds the sample distance;

[0023] Set abnormal recording time segments: Select the maximum and minimum values ​​of the center point as the upper and lower interval segments of the recording time, and set the recording time segments outside the upper and lower interval segments as abnormal recording time segments.

[0024] In a preferred embodiment, the recording judgment module compares the entry time of each data in the dataset to be cleaned with the abnormal record time segment. When the entry time falls within the abnormal record time segment, the corresponding data is marked and the marked data is passed to the data detection module.

[0025] In a preferred embodiment, the input status refers to the input time, and the storage status refers to the storage time;

[0026] Call the storage record log of each tag data, extract each tag data and put it into the storage time field as the storage time;

[0027] The difference between the storage time and the entry time is calculated to obtain the entry-storage time difference.

[0028] The difference between the input and storage time is compared and analyzed with the preset time anomaly threshold.

[0029] If the time difference between the input and storage exceeds the time anomaly threshold, the current marked data will be recorded as a time anomaly state.

[0030] If the time difference between the recorded and stored data is lower than the time anomaly threshold, the currently marked data will not be recorded as a time anomaly.

[0031] In a preferred embodiment, a detection time is set, and historical information of the marked data is retrieved within the detection time.

[0032] Historical information refers to the name of the historical record;

[0033] The historical record name is generated by calling the primary key value of the marked data and the table name field in the database record log;

[0034] Compare the names of each historical record within the detection period. If there are identical historical record names, it is determined that there is duplicate recording behavior, and the current marked data is marked as a duplicate abnormal state.

[0035] If no identical historical record name exists, it is determined that there is no duplicate record behavior, and the current marked data is not recorded as a duplicate exception state.

[0036] In a preferred embodiment, when performing input and storage operations on the input time and storage time of each marker data, the format of each marker data is detected. If the format of the marker data in the input operation is inconsistent with the format of the marker data in the storage operation, the current marker data is marked as an abnormal type.

[0037] If the format of the marked data in the input operation is the same as that in the storage operation, then the current marked data will not be recorded as a type exception.

[0038] The number of abnormal states in each labeled data is counted and accumulated to obtain the cleaning decision value of the dataset to be cleaned.

[0039] In a preferred embodiment, decision rules are set to generate different decision results for the dataset to be cleaned based on a high-level cleaning threshold and a low-level cleaning threshold.

[0040] The cleaning decision value of the dataset to be cleaned is compared with the high-level cleaning threshold and the low-level cleaning threshold. If the cleaning decision value of the dataset to be cleaned exceeds the high-level cleaning threshold, the high-level decision result is applied to the dataset to be cleaned.

[0041] If the cleaning decision value of the dataset to be cleaned is lower than the high-level cleaning threshold but higher than the low-level cleaning threshold, then the dataset to be cleaned will be subject to a medium-level decision.

[0042] If the cleaning decision value of the dataset to be cleaned is lower than the low-level cleaning threshold, then the low-level decision result is applied to the dataset to be cleaned.

[0043] In a preferred embodiment, a high-level cleaning monitoring method, a medium-level cleaning monitoring method, and a low-level cleaning monitoring method are selected based on the high-level decision results, the medium-level decision results, and the low-level decision results to clean the dataset to be cleaned.

[0044] The technical effects and advantages of this invention are as follows:

[0045] This invention integrates the data to be cleaned and calls up the entry times of different data. Information guidance is set to add the record position of the entry time to the additional information of each data point. The entry times of each data point within the dataset to be cleaned are processed, and abnormal recording time segments are set. Based on these abnormal time segments, each data point in the dataset to be cleaned is selected and marked. The entry and storage status of each marked data are detected. Different abnormal states are marked based on a combination of entry and storage status. The number of abnormal states for each marked data point is counted, and the cleaning decision value of the dataset to be cleaned is calculated. Decision rules are set, and the cleaning decision value is fed into the decision rules to generate a decision result. Based on the decision result, different cleaning methods are selected to clean the dataset, thereby improving cleaning efficiency while reducing cleaning costs and ensuring the accuracy and integrity of data quality. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the implementation of a multi-source heterogeneous data cleaning and evaluation system for automotive extended warranty according to the present invention.

[0047] Figure 2 This is a module framework diagram of a multi-source heterogeneous data cleaning and evaluation system for automotive extended warranty according to the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example: A system for cleaning and evaluating multi-source heterogeneous data from automotive extended warranty applications, such as... Figure 1 and Figure 2 As shown, it includes a data integration module, a record judgment module, a data detection module, and a cleaning evaluation module, with electrical signal connections between the modules;

[0050] The functions of each module are as follows:

[0051] The data integration module merges the data to be cleaned into a dataset to be cleaned, calls the entry time of different data in the dataset to be cleaned, sets information guidance to add the record position of the entry time to the additional information of each data in the dataset to be cleaned, and passes the dataset to be cleaned into the record judgment module.

[0052] The recording and judgment module identifies each data in the dataset to be cleaned and calls the information in the additional information of each data to obtain the corresponding entry time. Based on the entry time, it sets abnormal recording time segments, selects and marks each data in the dataset to be cleaned according to the abnormal time segments, and passes the marked data to the data detection module.

[0053] The data detection module detects the input and storage status of each labeled data, identifies different abnormal states based on the input and storage status, counts the number of abnormal states of each labeled data, calculates the cleaning decision value of the dataset to be cleaned, and sends the cleaning decision value of the dataset to be cleaned to the cleaning evaluation module.

[0054] The cleaning assessment module generates decision results by setting decision rules and feeding the received cleaning decision values ​​into the decision rules. Based on the decision results, different cleaning methods are selected to clean the dataset to be cleaned.

[0055] It should be noted that the information guide is an information space for storing information or data location. In this example, it is used to store the storage location information of the entry time of the data to be cleaned, so as to facilitate the subsequent retrieval of the entry time of each data in the dataset to be cleaned.

[0056] Before integrating the data to be cleaned, the data integration module identifies the car extended warranty service folder selected by the user, obtains the data contained in the folder and uses it as the data to be cleaned, and merges the data to be cleaned into a dataset to be cleaned as the evaluation target.

[0057] The data integration module obtains the record position of each data entry time in the dataset to be cleaned by accessing the service information dataset, sets information guidelines to temporarily store the record position of each data entry time in the dataset to be cleaned, adds the information guidelines set for the record position of each data entry time to the additional information of the corresponding data, and passes the above-processed dataset to be cleaned into the record judgment module.

[0058] It should be noted that a service information dataset is a collection of data that gathers and stores information services, including information on extended car warranty services and their locations. In this example, the location of the record used to retrieve the entry time of the extended car warranty service information is shown.

[0059] The recording and judgment module identifies the additional information of each data in the dataset to be cleaned and calls the storage location information of the corresponding information guide to the entry time. After obtaining the storage location information of the entry time of each data in the dataset to be cleaned, it jumps to the storage location and calls the entry time corresponding to each data.

[0060] The record judgment module uses the K-nearest neighbor algorithm to set abnormal record time segments based on the input time. The specific steps are as follows:

[0061] Merging data: The entry time corresponding to each data point in the dataset to be cleaned is used as the time dataset;

[0062] Set the K value: Use the median of the data in the time dataset as the K value;

[0063] Dataset sorting: Sort the entry times in chronological order and calculate the time difference between adjacent entry times;

[0064] Calculate the sample distance: Set the time adjustment ratio, and use the product of the time adjustment ratio and the maximum time difference between adjacent input times as the sample distance of the time dataset;

[0065] K-value screening: Using the K-value as the center point, compare the distance between the center point and its two adjacent points with the sample distance. If the distance is less than the sample distance, then use the left and right adjacent points as two new center points and perform a new round of comparison with the left or right adjacent points.

[0066] Set a stopping condition: stop traversing when the distance between the center point and its left or right neighboring point exceeds the sample distance;

[0067] Set abnormal recording time segments: Select the maximum and minimum values ​​of the center point as the upper and lower interval segments of the recording time, and set the recording time segments outside the upper and lower interval segments as abnormal recording time segments.

[0068] The record judgment module compares the entry time of each data in the dataset to be cleaned with the abnormal record time segment. When the entry time falls within the abnormal record time segment, the corresponding data is marked and the marked data is sent to the data detection module.

[0069] It should be noted that the K-nearest neighbors algorithm is a supervised learning algorithm used for classification and regression tasks. In this example, it is used to select and label the data to be cleaned. The time adjustment ratio mentioned above can be set according to the screening accuracy. For example, if the time adjustment ratio is set to 80%, the sample distance will be 80% of the maximum time difference between adjacent entry times. This will not be analyzed in detail here.

[0070] The recording and judgment module detects the input and storage status of each marked data.

[0071] Specifically, "entry status" refers to the entry time, and "storage status" refers to the storage time.

[0072] Among them, storage time refers to the storage timestamp information of the storage record log generated when each marked data enters the system storage unit. Its acquisition logic is to call the storage record log of each marked data and extract the storage time field of each marked data as the storage time.

[0073] It should be noted that the storage record log refers to the data write operation log that is automatically generated and recorded when each marked data is written to the system storage unit.

[0074] The difference between the storage time and the entry time is calculated to obtain the entry-storage time difference.

[0075] The difference between the input and storage time is compared and analyzed with the preset time anomaly threshold.

[0076] If the time difference between the input and storage exceeds the time anomaly threshold, the current marked data will be recorded as a time anomaly state.

[0077] If the time difference between the input and storage is lower than the time anomaly threshold, the currently marked data will not be recorded as a time anomaly.

[0078] It should be noted that the time anomaly threshold is a reference threshold set by researchers in this field based on the normal delay characteristics of the data acquisition link and the system storage write latency characteristics. It is used to measure the reasonable range of difference between the input time and the storage time. The specific threshold setting can be determined by combining factors such as network transmission latency, system processing latency, and database write latency under different application scenarios, which will not be elaborated here.

[0079] Set the detection time and retrieve the historical information of the marked data within the detection time;

[0080] Among them, historical information refers to the name of the historical record;

[0081] The historical record name is a name identifier field that can uniquely identify the same data object in the historical record information. It is generated by combining the data unique identifier field and the data feature field and is used to distinguish different data records. Its acquisition logic is to generate the historical record name by calling the primary key value of the marked data and the table name field in the database record log.

[0082] Among them, the database log refers to the operation log automatically generated and saved by the database management system during the process of marking data writing, updating or deleting operations, including but not limited to: the primary key value of the data, the table name field, the operation type (write, update or delete), the operation timestamp, the operation result status (success, failure or retry information), etc.

[0083] It should be noted that the primary key value of the marked data refers to the key field in the database table used to uniquely identify a single data record. It is set to a unique value in the database design to distinguish different data objects. The table name field refers to the name identifier field of the database table where the marked data is located. The table name field is used to identify the storage location and logical belonging of the data, and is used to further distinguish data records from different sources or of different types.

[0084] It should be noted that primary key values ​​in database design are typically unique within a single table to distinguish different data records within that table. However, those skilled in the art will understand that duplicate representations may still exist in practical applications. For example, when multiple different database tables generate primary key values, even if the primary keys within each table are not duplicated, the same primary key values ​​may appear between different tables. In the event of business system failure or logical anomalies, duplicate primary key values ​​may be generated.

[0085] Compare the names of each historical record within the detection period. If there are identical historical record names, it is determined that there is duplicate recording behavior, and the current marked data is marked as a duplicate abnormal state.

[0086] If no identical historical record name exists, it is determined that there is no duplicate record behavior, and the current marked data is not recorded as a duplicate exception state;

[0087] When performing input and storage operations on the input and storage times of each tagged data, the format of each tagged data is checked. If the format of the tagged data in the input operation is inconsistent with the format of the tagged data in the storage operation, the current tagged data is marked as an abnormal type.

[0088] If the format of the marked data in the input operation is the same as that in the storage operation, then the current marked data will not be recorded as a type exception.

[0089] It should be noted that the abnormal states are cumulative, that is, the same labeled data may have both time abnormality and duplication abnormality at the same time. For example, when the time difference between the entry and storage of a certain labeled data exceeds the time abnormality threshold, and there is also duplicate recording behavior within the set detection time, then the labeled data is marked as both time abnormality state and duplication abnormality state.

[0090] The number of abnormal states in each labeled data is counted and accumulated to obtain the cleaning decision value of the dataset to be cleaned;

[0091] Send the cleaning decision values ​​of the dataset to be cleaned to the cleaning evaluation module;

[0092] It should be noted that the larger the cleaning decision value of the dataset to be cleaned, the more abnormal states there are in the dataset, the higher the overall data quality risk, and the more necessary it is to adopt higher-level cleaning measures to ensure the targeting and effectiveness of data cleaning.

[0093] In the cleaning assessment module, decision rules are set to generate different decision results for the dataset to be cleaned based on high-level cleaning thresholds and low-level cleaning thresholds.

[0094] The cleaning decision value of the dataset to be cleaned is compared with the high-level cleaning threshold and the low-level cleaning threshold. If the cleaning decision value of the dataset to be cleaned exceeds the high-level cleaning threshold, the high-level decision result is applied to the dataset to be cleaned.

[0095] If the cleaning decision value of the dataset to be cleaned is lower than the high-level cleaning threshold but higher than the low-level cleaning threshold, then the dataset to be cleaned will be subject to a medium-level decision.

[0096] If the cleaning decision value of the dataset to be cleaned is lower than the low-level cleaning threshold, then the low-level decision result is applied to the dataset to be cleaned.

[0097] It should be noted that the high-level cleaning threshold and the low-level cleaning threshold were set by the researchers based on the abnormal distribution characteristics of the historical dataset and the processing resources and cleaning effect requirements of different cleaning levels, and will not be elaborated here.

[0098] Based on the results of high-level, mid-level, and low-level decision-making, high-level cleaning and monitoring methods, mid-level cleaning and monitoring methods, and low-level cleaning and monitoring methods are selected to clean the dataset to be cleaned.

[0099] Among them, advanced cleaning and monitoring methods refer to the processing methods used in the data cleaning process to perform full consistency verification, cross-source comparison, logical rule verification, and abnormal data repair on the dataset to be cleaned. Specifically, this includes, for example, detecting the time consistency, field integrity, and cross-table dependencies of each marked data, and calling anomaly repair algorithms to correct data with missing, conflicting, or logically inconsistent data.

[0100] Intermediate cleaning and monitoring methods refer to the processing methods used in the data cleaning process, such as partial field detection, duplicate data removal, and basic integrity verification of the dataset to be cleaned. Specifically, these methods include, for example, comparing the field values ​​of various tagged data from the same source to see if there are duplicates or conflicts, deleting redundant data, and performing integrity checks on key fields.

[0101] Low-level cleaning monitoring methods refer to the processing methods of standardizing the format and filling default values ​​in the dataset to be cleaned during the data cleaning process. Specifically, this includes, for example, uniformly formatting data fields (such as time format, numerical units, etc.) and filling default values ​​in some default fields to ensure that the dataset can be correctly identified and used in subsequent processing stages.

[0102] It should be noted that cleaning refers to the process of performing corresponding data quality improvement operations on the dataset to be cleaned according to the preset cleaning monitoring methods based on the decision results generated by the cleaning assessment module. Its goal is to minimize data redundancy, improve data accuracy and consistency, and ensure that the cleaned data can meet the needs of subsequent analysis or business calls through means such as abnormal data removal, defective data repair, format standardization and logical consistency verification.

[0103] Finally, it should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0104] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] In this document, the singular forms “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that terms such as “comprising / including” or “having” specify the presence of the stated features, integrals, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, integrals, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0106] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0107] The above description of the disclosed embodiments will enable those skilled in the art to make or use various modifications to these embodiments. It will be readily apparent to those skilled in the art that the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A system for cleaning and evaluating multi-source heterogeneous data from automotive extended warranty applications, characterized in that: It includes a data integration module, a record judgment module, a data detection module, and a cleaning evaluation module, with electrical signal connections between the modules; The data integration module merges the data to be cleaned into a dataset to be cleaned, calls the entry time of different data in the dataset to be cleaned, sets information guidance to add the record position of the entry time to the additional information of each data in the dataset to be cleaned, and passes the dataset to be cleaned into the record judgment module. The recording and judgment module identifies each data in the dataset to be cleaned and calls the information in the additional information of each data to obtain the corresponding entry time. Based on the entry time, it sets abnormal recording time segments, selects and marks each data in the dataset to be cleaned according to the abnormal time segments, and passes the marked data to the data detection module. The K-nearest neighbor algorithm is used to set the time segments for abnormal records as follows: Merging data: The entry time corresponding to each data point in the dataset to be cleaned is used as the time dataset; Set the K value: Use the median of the data in the time dataset as the K value; Dataset sorting: Sort the entry times in chronological order and calculate the time difference between adjacent entry times; Calculate the sample distance: Set the time adjustment ratio, and use the product of the time adjustment ratio and the maximum time difference between adjacent input times as the sample distance of the time dataset; K-value screening: Using the K-value as the center point, compare the distance between the center point and its two adjacent points with the sample distance. If the distance is less than the sample distance, then use the left and right adjacent points as two new center points and perform a new round of comparison with the left or right adjacent points. Set a stopping condition: stop traversing when the distance between the center point and its left or right neighboring point exceeds the sample distance; Set abnormal recording time segments: Select the maximum and minimum values ​​of the center point as the upper and lower interval segments of the recording time, and use the recording time segments outside the upper and lower interval segments as abnormal recording time segments. The data detection module detects the input and storage status of each labeled data, identifies different abnormal states based on the input and storage status, counts the number of abnormal states of each labeled data, calculates the cleaning decision value of the dataset to be cleaned, and sends the cleaning decision value of the dataset to be cleaned to the cleaning evaluation module. The cleaning assessment module generates decision results by setting decision rules and feeding the received cleaning decision values ​​into the decision rules. Based on the decision results, different cleaning methods are selected to clean the dataset to be cleaned.

2. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 1, characterized in that: Before integrating the data to be cleaned, the data integration module identifies the user-selected car extended warranty service folder and obtains the data contained in the folder as the data to be cleaned. The setting information guide is used to temporarily store the record position of each data entry time in the dataset to be cleaned. The information guide set for the record position of each data entry time is added to the additional information of the corresponding data.

3. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 2, characterized in that: After the record judgment module obtains the storage location information of the entry time of each data in the dataset to be cleaned, it jumps to the storage location and calls the entry time corresponding to each data. The K-nearest neighbor algorithm is used to set time segments for abnormal records based on the entry time.

4. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 1, characterized in that: The record judgment module compares the entry time of each data in the dataset to be cleaned with the abnormal record time segment. When the entry time falls within the abnormal record time segment, the corresponding data is marked and the marked data is sent to the data detection module.

5. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 1, characterized in that: Entry status refers to the entry time, and storage status refers to the storage time; Call the storage record log of each tag data, extract each tag data and put it into the storage time field as the storage time; The difference between the storage time and the entry time is calculated to obtain the entry-storage time difference. The difference between the input and storage time is compared and analyzed with the preset time anomaly threshold. If the time difference between the input and storage exceeds the time anomaly threshold, the current marked data will be recorded as a time anomaly state. If the time difference between the recorded and stored data is lower than the time anomaly threshold, the currently marked data will not be recorded as a time anomaly.

6. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 1, characterized in that: Set the detection time and retrieve the historical information of the marked data within the detection time; Historical information refers to the name of the historical record; The historical record name is generated by calling the primary key value of the marked data and the table name field in the database record log; Compare the names of each historical record within the detection period. If there are identical historical record names, it is determined that there is duplicate recording behavior, and the current marked data is marked as a duplicate abnormal state. If no identical historical record name exists, it is determined that there is no duplicate record behavior, and the current marked data is not recorded as a duplicate exception state.

7. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 1, characterized in that: When performing input and storage operations on the input and storage times of each tagged data, the format of each tagged data is checked. If the format of the tagged data in the input operation is inconsistent with the format of the tagged data in the storage operation, the current tagged data is marked as an abnormal type. If the format of the marked data in the input operation is the same as that in the storage operation, then the current marked data will not be recorded as a type exception. The number of outliers in each labeled data set is counted and accumulated to obtain the cleaning decision value for the dataset to be cleaned.

8. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 7, characterized in that: Set decision rules to generate different decision results for the dataset to be cleaned based on high-level and low-level cleaning thresholds; The cleaning decision value of the dataset to be cleaned is compared with the high-level cleaning threshold and the low-level cleaning threshold. If the cleaning decision value of the dataset to be cleaned exceeds the high-level cleaning threshold, the high-level decision result is applied to the dataset to be cleaned. If the cleaning decision value of the dataset to be cleaned is lower than the high-level cleaning threshold but higher than the low-level cleaning threshold, then the dataset to be cleaned will be subject to a medium-level decision. If the cleaning decision value of the dataset to be cleaned is lower than the low-level cleaning threshold, then the low-level decision result is applied to the dataset to be cleaned.

9. The automotive extended warranty multi-source heterogeneous data cleaning and evaluation system according to claim 8, characterized in that: Based on the results of high-level, mid-level, and low-level decision-making, high-level cleaning and monitoring methods, mid-level cleaning and monitoring methods, and low-level cleaning and monitoring methods are selected to clean the dataset to be cleaned.

Images