Data processing method and device, equipment, program product and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment, a program product and a storage medium, and the method comprises the steps: receiving the processing data, sent by a data processing unit, of each processing node in a data processing process, the processing data comprises to-be-processed data of each Internet of Vehicles data source sent by the message queue subunit, effective data sent by the data distribution subunit and application data sent by the service subunit; performing data statistics on the processed data to obtain upstream statistical data corresponding to the to-be-processed data, dimension statistical data corresponding to the valid data and landing statistical data corresponding to the application data; performing consistency detection on the upstream statistical data, the dimension statistical data and the landing statistical data to obtain a consistency detection result; if the consistency detection result is abnormal, abnormal prompt information is generated according to the consistency detection result, and the abnormal prompt information is sent to the user. According to the method, the data consistency in the stream processing process can be ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data, and in particular to a data processing method, apparatus, device, program product and storage medium. Background Art

[0002] In the Internet of Vehicles scenario, with the surge in the number of vehicles and the explosive growth in the amount of data, how to efficiently process and analyze these massive amounts of data has become an urgent problem to be solved. Traditional data processing methods often cannot meet the real-time and high concurrency requirements in complex analysis scenarios of massive data. Therefore, the data warehouse design pattern based on the snowflake model is introduced to meet this challenge. The snowflake model can quickly adapt to data changes and generate highly timely data reports by splitting complex master data into multiple sub-tables and using a distributed computing framework for real-time stream processing.

[0003] However, although the snowflake model performs well in improving the efficiency of complex data analysis, in the snowflake model, it is difficult to confirm the consistency of the data volume between the data production end and the final analysis report landing application end. In the Internet of Vehicles scenario, upstream data usually comes from various sensors. After real-time screening, classification, aggregation and other processing, these data will be output to the downstream application database. Since the business meaning of the upstream data is inconsistent with the business meaning of the downstream application database, it is impossible to intuitively confirm the data consistency from the size of the upstream and downstream data. For example, in the scenario of counting vehicle charging fragments, the upstream data is the charging status of each frame of the message, and after streaming data processing, multiple messages will be merged into a vehicle charging fragment data after analysis and output to the downstream application database. In this case, due to the different number of message frames contained in each charging fragment, the consistency of the data cannot be ensured. Summary of the invention

[0004] The embodiments of the present invention provide a data processing method, apparatus, device, program product and storage medium, which can perform full-link data consistency detection on data in the stream processing of the snowflake model data structure, and promptly remind the user when the data is inconsistent, thereby ensuring the consistency of the data in the stream processing process.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0006] Receive processing data of each processing node in the data processing process sent by the data processing unit, wherein the processing data includes the to-be-processed data of each vehicle network data source sent by the message queue subunit, the valid data sent by the data distribution subunit, and the application data sent by the business subunit;

[0007] Performing data statistics on the processed data to obtain upstream statistical data corresponding to the data to be processed, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data;

[0008] Performing consistency check on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain a consistency check result;

[0009] If the consistency detection result is abnormal, abnormal prompt information is generated according to the consistency detection result, and the abnormal prompt information is sent to the user.

[0010] In a second aspect, an embodiment of the present invention provides a data processing device, the device comprising:

[0011] A data acquisition module, used to receive the processing data of each processing node in the data processing process sent by the data processing unit, wherein the processing data includes the to-be-processed data of each IoV data source sent by the message queue subunit, the valid data sent by the data distribution subunit, and the application data sent by the business subunit;

[0012] A data statistics module is used to perform data statistics on the processed data to obtain upstream statistical data corresponding to the data to be processed, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data;

[0013] A data detection module is used to perform consistency detection on the upstream statistical data, the dimension statistical data and the landing statistical data to obtain a consistency detection result;

[0014] The result display module is used to generate abnormal prompt information according to the consistency detection result if the consistency detection result is abnormal, and send the abnormal prompt information to the user.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a data processing method as described in any one of the embodiments of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data processing method as described in any one of the embodiments of the present invention.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements a data processing method as described in any one of the embodiments of the present invention.

[0018] In an embodiment of the present invention, the processing data of each processing node in the data processing process sent by the data processing unit is received, and the processing data includes the to-be-processed data of each Internet of Vehicles data source sent by the message queue subunit, the valid data sent by the data diversion subunit, and the application data sent by the business subunit; data statistics are performed on the processed data to obtain upstream statistical data corresponding to the to-be-processed data, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data; consistency detection is performed on the upstream statistical data, dimensional statistical data, and landing statistical data to obtain a consistency detection result; if the consistency detection result is abnormal, abnormal prompt information is generated according to the consistency detection result, and the abnormal prompt information is sent to the user. The method of the embodiment of the present invention can obtain the data of each processing node in the data processing process, and perform data statistics and full-link data consistency detection on it. When the consistency detection result is abnormal, the user can be prompted in time, so that the user can understand the problems in the data processing process in time, thereby ensuring the consistency of the data in the stream processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A first flow chart of a data processing method provided by an embodiment of the present invention;

[0021] Figure 2 A second flow chart of a data processing method provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the structure of a data processing unit and a consistency detection unit provided in an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of the structure of a data processing device provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0026] Figure 1 This is the first flow chart of a data processing method provided by an embodiment of the present invention. The method of the embodiment of the present invention can perform full-link data consistency detection on data in the stream processing of the snowflake model data structure, and promptly remind the user when the data is inconsistent, thereby ensuring the consistency of the data in the stream processing process. The method can be executed by a data processing device provided by an embodiment of the present invention, and the device can be implemented in software and / or hardware. The following embodiments will be described by taking the device integrated in an electronic device as an example. The electronic device can be a server or a computer device, etc., refer to Figure 1 , the method may specifically include the following steps:

[0027] Step 101: Receive processing data of each processing node in the data processing process sent by a data processing unit.

[0028] Among them, the data processing unit includes a message queue subunit, a data diversion subunit and a business subunit. The message queue subunit includes each message queue, and the business subunit includes each business system and software module. The processed data includes the pending data of each Internet of Vehicles data source sent by the message queue subunit, the valid data sent by the data diversion subunit, and the application data sent by the business subunit. The Internet of Vehicles data source includes the vehicle, other equipment or software module that generates the Internet of Vehicles data. The Internet of Vehicles data source in this solution includes the vehicle terminal's vehicle machine module, log system, other equipment terminals (such as in-vehicle sensors and navigation systems, etc.), internal systems of automobile companies and third-party systems, etc. The pending data is the Internet of Vehicles data generated by each Internet of Vehicles data source stored in each message queue. The data diversion subunit is used to process and analyze the pending data in each message queue according to specific business logic and actual needs, etc. The valid data is the data obtained after the data diversion subunit processes the pending data. The business subunit can process the valid data and apply the processed valid data to each business system and software module.

[0029] Specifically, the processing method of the Internet of Vehicles data adopts the data stream processing method, which continuously processes the new data events received. Since this method is a continuous processing, generally only the initial data and the final data can be obtained in this method. In this scheme, in order to ensure data consistency and understand the data processing situation in the data stream processing process, the processing data of each processing node in the data processing process can be obtained in real time through the consistency detection unit in the server. The processing nodes include data acquisition nodes, data processing nodes and data application nodes. When the server obtains the data to be processed through the message queue subunit, each message queue can synchronize the data to be processed to the consistency detection unit. After the data diversion subunit processes the data to be processed and obtains valid data, the valid data can be synchronized to the consistency detection unit. After the business subunit obtains the business data based on the valid data, the business data can be synchronized to the consistency detection unit. The consistency detection unit can receive the processing data of each processing node in the data processing process.

[0030] Step 102: Perform data statistics on the processed data to obtain upstream statistical data corresponding to the data to be processed, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data.

[0031] Specifically, after obtaining the data to be processed, valid data and application data, the consistency detection unit can perform format consistency conversion on the data to be processed, valid data and application data according to their data formats, so that the processed data of each processing node maintains format consistency. Data preprocessing is performed on the processed data after the conversion format, and duplicate data is removed, missing data is filled, and erroneous data is corrected. After obtaining the preprocessed data to be processed, data sorting and statistics are performed on the data to be processed to obtain upstream statistical data. Data sorting and statistics are performed on the valid data to obtain dimensional statistical data. Data sorting and statistics are performed on the application data to obtain landing statistical data.

[0032] For example, in the process of counting vehicle charging segments, the data to be processed is the charging status of each frame of the message, and after the streaming data processing of the data splitting subunit, multiple frames of messages will be merged and analyzed into a vehicle charging segment data output to the business subunit. The consistency detection unit can convert the format of the data to be processed, the valid data and the application data, and convert them into data in a unified format for consistency detection.

[0033] Step 103: Perform consistency check on the upstream statistical data, dimension statistical data and landing statistical data to obtain consistency check results.

[0034] Among them, the consistency test result is normal or abnormal, and normal means that there is no abnormal state such as missing data or incorrect data processing during the data processing. After obtaining the upstream statistical data, dimension statistical data and landing statistical data, the three types of data are subjected to consistency test in each dimension to obtain the consistency test result. In this scheme, optionally, the upstream statistical data, dimension statistical data and landing statistical data are subjected to consistency test to obtain the consistency test result, including the following steps A1-A2:

[0035] Step A1: According to the data identifier of each piece of processed data, perform a data volume consistency check on the dimensional statistical data and the landing statistical data to obtain a data volume consistency check result; according to the timestamp of each piece of data, perform a time consistency check on the upstream statistical data, the dimensional statistical data and the landing statistical data to obtain a time consistency check result; according to the data content of each piece of data, perform a content consistency check on the upstream statistical data, the dimensional statistical data and the landing statistical data to obtain a content consistency check result.

[0036] Among them, the data identifier is an identifier used to uniquely identify each data record, such as a unique key (such as a primary key) for each data or other fields that can ensure the uniqueness of the data. After obtaining all the processed data, the upstream statistical data, dimension statistical data and landing statistical data are respectively counted according to the data identifier of each data, and their data volumes are compared to obtain the data volume consistency test result. Exemplarily, there are 80 upstream statistical data, 45 dimension statistical data, and 35 landing statistical data. It can be directly determined that the data has been lost (the data volume of the landing statistical data is less than the data volume of the dimension statistical data), and the data volume consistency test result is abnormal. Exemplarily, there are 80 upstream statistical data, 45 dimension statistical data, and 45 landing statistical data. According to the data identifier, the consistency detection unit detects that the data identifiers of 45 data in the dimension statistical data are all present in the 80 data of the upstream statistical data (indicating that the dimension statistical data is indeed obtained based on the upstream statistical data). The data identifiers of 40 data in the landing statistical data exist in both the upstream statistical data and the dimension statistical data, while the data identifiers of 5 data only exist in the upstream statistical data. It means that the dimension statistical data may have been lost, and the business system may have processed 5 invalid data. Further, it is determined that the data volume consistency test result is abnormal.

[0037] Specifically, during the data processing process, each processing node will timestamp the data in real time when processing the data to record the time when the data was processed. After obtaining the processed data, the consistency detection unit determines whether the timestamps of each piece of data in the upstream statistical data, dimensional statistical data, and landing statistical data are within a processing cycle, for example, whether they are all data from the same day or the same hour. If the timestamp of each piece of data is within a processing cycle, the time consistency detection result is determined to be normal. If there is at least one piece of data that has a timestamp that is not within a processing cycle with other data, the time consistency detection result is determined to be abnormal.

[0038] After obtaining the data volume consistency test results and the time consistency test results, the data content of each piece of processed data is checked for consistency. Specifically, the consistency test unit can extract and process the data with the same data identifier in the upstream statistical data, dimension statistical data, and landing statistical data according to the data identifier, and analyze whether the data content with the same data identifier is consistent, thereby obtaining the content consistency test result. For example, if a car is charged within a certain period of time, the change in power during this period should match the charging record.

[0039] Step A2: Obtain consistency test results based on the data volume consistency test results, the time consistency test results and the content consistency test results.

[0040] After obtaining the data volume consistency test results, the time consistency test results, and the content consistency test results, if at least one of the three consistency test results is abnormal, then the consistency test result of the processed data is determined to be abnormal. If the data volume consistency test results, the time consistency test results, and the content consistency test results are all normal, then the consistency test result of the processed data is determined to be normal.

[0041] By performing data volume consistency test, time consistency test results and content consistency test on the processed data, the consistency test results of the processed data can be accurately obtained, providing a reliable foundation for subsequent data analysis and application.

[0042] Step 104: If the consistency detection result is abnormal, generate abnormal prompt information according to the consistency detection result, and send the abnormal prompt information to the user.

[0043] Among them, the abnormal prompt information is used to prompt the user to deal with the abnormal data in time. If the consistency test result is normal, there is no need to send a prompt message to the user. If the consistency test result is abnormal, an abnormal prompt message is generated according to the abnormal consistency test result. For example, if the data volume consistency test result is abnormal, a prompt message is generated according to the data volume of the processed data and displayed to the user.

[0044] In this solution, the consistency detection unit can also provide query services for users, providing a basis for users to conduct full-link inspection, analysis and verification of the data processing process. In this solution, optionally, it also includes: receiving data traceability conditions sent by users, finding target data corresponding to the traceability conditions in the processed data, and visualizing the target data to obtain visualized target data; and displaying the visualized target data to users.

[0045] Among them, the traceability condition includes one or more of the processing node, the traceability dimension and the filtering condition. The processing node includes the data acquisition node, the data processing node and the data application node. The traceability dimension is used to trace the specific dimension of the data, such as the vehicle identification, the charging station name or the time range. The filtering condition is a further refined query condition, such as a specific time period or a specific data type. Specifically, when the user needs to query the data through the consistency detection unit, the data traceability condition can be input according to the query requirements. After receiving the data traceability condition, the server parses the data traceability condition and determines the specific fields and values ​​to be searched. The data of each processing node is searched to find the target data that matches the condition. For example, if the user specifies the vehicle charging record within a certain time period, the server can search for the record within the time period in the data to be processed. The retrieved data is filtered according to the filtering condition to ensure that only the records that meet the conditions are retained, that is, the target data. After obtaining the target data, the data visualization tool is used to visualize the target according to the data characteristics of the target data and the user needs to obtain the visualized target data. For example, the target data is converted into a suitable bar chart, line chart, or pie chart. Further, the visualized target data is displayed to the user. By providing data traceability condition query services, users can track each link of data processing, improving data transparency and user experience.

[0046] The technical solution of this embodiment receives the processing data of each processing node in the data processing process sent by the data processing unit, and the processing data includes the pending data of each Internet of Vehicles data source sent by the message queue subunit, the valid data sent by the data diversion subunit, and the application data sent by the business subunit; performs data statistics on the processed data to obtain the upstream statistical data corresponding to the pending data, the dimensional statistical data corresponding to the valid data, and the landing statistical data corresponding to the application data; performs consistency detection on the upstream statistical data, the dimensional statistical data, and the landing statistical data to obtain the consistency detection result; if the consistency detection result is abnormal, generates abnormal prompt information according to the consistency detection result, and sends the abnormal prompt information to the user. The technical solution of this embodiment can obtain the data of each processing node in the data processing process, and perform data statistics and full-link data consistency detection on it. When the consistency detection result is abnormal, it can promptly prompt the user, so that the user can promptly understand the problems in the data processing process, thereby ensuring the consistency of the data in the stream processing process.

[0047] Figure 2 The second flow chart of a data processing method provided by an embodiment of the present invention is a refinement of the above embodiment. The specific method can be as follows Figure 2 As shown, the method may include the following steps:

[0048] Step 201: Obtain the data to be processed from each Internet of Vehicles data source through the message queue subunit.

[0049] Among them, the message queue subunit includes multiple message queues, and the message queues are used to store data from various Internet of Vehicles data sources. Different types of data to be processed correspond to different message queues. Internet of Vehicles data sources include the vehicle terminal's vehicle computer module, log system, other device terminals (such as in-vehicle sensors and navigation systems, etc.), car company internal systems and third-party systems, etc. The server can obtain vehicle network data in real time through each message queue. For example, the vehicle's status data includes basic information such as speed, location and fuel consumption. Vehicle status data usually requires high throughput and low latency processing. Therefore, Kafka message queues that can efficiently process large amounts of real-time data streams can be used to store vehicle status data.

[0050] Step 202: pre-divide the data to be processed according to the preset dimension tables to obtain valid data and invalid data.

[0051] Among them, each dimension table can be a data table obtained under the data structure of the snowflake model. The snowflake model can further normalize the original data table into multiple dimension tables to reduce redundancy and data storage space. Valid data is data that can meet the filtering conditions of each dimension and the server in real time. In this solution, optionally, data pre-diversion is performed on the data to be processed according to each pre-set dimension table to obtain valid data and invalid data, including the following steps B1-B2:

[0052] Step B1: Perform dimensional analysis on the data to be processed, and input the data to be processed into each dimensional table according to the dimensional analysis result to obtain data of each dimensional table;

[0053] Specifically, after obtaining the data to be processed, the data to be processed is analyzed to identify different dimensions and measurement values ​​in the data. For example, in the Internet of Vehicles data, dimensions such as vehicle status, charging events, and fault diagnosis may be included. After obtaining the data to be processed, at least one dimension to which each piece of data to be processed belongs is determined, and the data to be processed is mapped to the corresponding dimension table according to its attributes and content to obtain data in each dimension table.

[0054] Step B2: Obtain the filtering conditions at the current moment, and filter out valid data from each dimensional data table according to the filtering conditions; determine the data other than the valid data in each dimensional data table as invalid data.

[0055] Among them, the filtering conditions can be pre-set according to business logic and data analysis requirements, or they can be sent to the server in real time according to the needs of the user. The filtering conditions are used to filter the data in each dimensional data table to obtain valid data that can meet the filtering conditions. For example, if the user needs the vehicle speed data within a certain time period, the filtering conditions can be determined to include the time range and the vehicle speed. After determining the filtering conditions, the server determines the dimensional data table including the vehicle speed in each dimensional data table according to the filtering conditions, and then filters out the vehicle speed data within the time period from these dimensional data tables and determines it as valid data. Data other than valid data in each dimensional data table is determined as invalid data.

[0056] By classifying and storing the data to be processed through pre-set dimension tables, the data retrieval and processing speed can be accelerated. When the data needs to be queried or analyzed, it can be operated directly in the relevant dimension table without traversing the entire data set. Through dimension analysis and setting of filtering conditions, valid data that meets business needs can be effectively screened out, while invalid or irrelevant data can be identified and excluded, which improves the accuracy and reliability of the data and provides a data basis for subsequent data analysis and decision-making.

[0057] After determining the invalid data, it can be further determined whether to retain the invalid data based on the amount and data characteristics of the invalid data. In this solution, optionally, after obtaining the invalid data, it also includes: performing data retention analysis on the invalid data based on a pre-set invalid retention index to obtain a data retention analysis result; when the retention analysis result is to retain, the invalid data is stored in a pre-established invalid database.

[0058] Among them, the retention analysis result is to retain or not retain. The invalid retention index is used to determine whether the server retains invalid data. The invalidity includes indicators including: one or more of the amount of data, the type of data, and the frequency of data generation. After obtaining the invalid data, the invalid data is analyzed. According to the analysis results of the invalid data, it is determined whether to retain the invalid data in combination with the invalid retention index. For example, if the invalid retention index includes that the amount of data is greater than 80, and the amount of invalid data is 100, it is determined to retain the invalid data. Furthermore, if it is determined to retain the invalid data, the invalid data can be stored in a pre-established invalid database for subsequent analysis and use.

[0059] By performing dimensional analysis on the data to be processed and inputting the data into each dimensional table according to the dimensional analysis results, the data of each dimensional table can be obtained. This process helps to reduce data redundancy, improve storage efficiency, and provide a high-quality data foundation for subsequent data analysis and visualization.

[0060] Step 203: Process the valid data according to the business requirements of the business sub-unit to obtain application data of the business sub-unit.

[0061] Among them, the business subunit includes various business systems and software modules, and the business subunit can apply valid data to various business systems and software modules. Business requirements can be pre-set according to business logic, or sent to the server in real time by users according to requirements. According to business requirements, valid data is processed, including data cleaning (removing outliers and missing values) and data conversion (converting data formats into a form suitable for analysis), etc., to obtain application data of the business subunit, so that the application data can be applied to various business systems and software modules.

[0062] Figure 3 The structure diagram of the data processing unit and the consistency detection unit provided by the embodiment of the present invention is as follows. Figure 3As shown, the data source of the Internet of Vehicles includes a vehicle computer module, a log system, and a third-party application. The data processing unit includes a message queue subunit, a data diversion subunit, and a business subunit. The message queue subunit includes a Kafka message queue, a RabbitMQ message queue, and a RocketMQ message queue. The message queue subunit obtains the data to be processed of the Internet of Vehicles data source. The data diversion subunit diverts the data sent by the message queue subunit, performs data analysis on the valid data, and sends the analyzed valid data to the business subunit. The business subunit applies the valid data to various business scenarios. The data diversion subunit performs exception processing on the invalid data that needs to be retained and stores it in an invalid database. During the data processing process, the consistency detection unit can receive the data sent by the message queue subunit, the data diversion subunit, and the business subunit in real time, and perform data statistics on it to obtain upstream statistical data, dimension statistical data, and landing statistical data. The consistency detection unit includes a consistency detection module and a query module. The consistency detection unit can perform consistency detection on the upstream statistical data, dimension statistical data, and landing statistical data to obtain a consistency detection result. When the consistency detection result is abnormal, a prompt message is sent to the user. The user can trace the data through the query module.

[0063] Step 204, receiving the processing data of each processing node in the data processing process sent by the data processing unit, the processing data includes the to-be-processed data of each Internet of Vehicles data source sent by the message queue subunit, the valid data sent by the data diversion subunit, and the application data sent by the business subunit.

[0064] Step 205: Perform data statistics on the processed data to obtain upstream statistical data corresponding to the data to be processed, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data.

[0065] Step 206: Perform consistency check on the upstream statistical data, dimension statistical data, and landing statistical data to obtain consistency check results.

[0066] Step 207: If the consistency detection result is abnormal, generate abnormal prompt information according to the consistency detection result, and send the abnormal prompt information to the user.

[0067] In the technical solution of this embodiment, the data to be processed of each vehicle network data source is obtained through the message queue subunit; the data to be processed is pre-divided according to each pre-set dimension table to obtain valid data and invalid data; the valid data is processed according to the business needs of the business subunit to obtain the application data of the business subunit. The processing data of each processing node in the data processing process sent by the data processing unit is received, and the processing data includes the data to be processed of each vehicle network data source sent by the message queue subunit, the valid data sent by the data diversion subunit, and the application data sent by the business subunit. Data statistics are performed on the processed data to obtain the upstream statistical data corresponding to the data to be processed, the dimension statistical data corresponding to the valid data, and the landing statistical data corresponding to the application data. The upstream statistical data, the dimension statistical data, and the landing statistical data are checked for consistency to obtain the consistency test result. If the consistency test result is abnormal, an abnormal prompt information is generated according to the consistency test result, and the abnormal prompt information is sent to the user. In the technical solution of this embodiment, the data to be processed is obtained through the message queue subunit, which can ensure the real-time collection and transmission of data. By pre-dividing the data and separating the valid data from the invalid data, it can be ensured that the subsequent data processing is only performed on the valuable data, thereby improving the accuracy and efficiency of data processing. In addition, in this embodiment, the data of each processing node in the data processing process can be obtained, and data statistics and full-link data consistency detection can be performed on it. When the consistency detection result is abnormal, the user can be prompted in time so that the user can understand the problems in the data processing process in time, thereby ensuring the consistency of the data in the stream processing process.

[0068] Figure 4 The structure diagram of a data processing device provided in an embodiment of the present invention is suitable for executing the data processing method provided in an embodiment of the present invention. Figure 4 As shown, the device may specifically include:

[0069] The data acquisition module 401 is used to receive the processing data of each processing node in the data processing process sent by the data processing unit, wherein the processing data includes the to-be-processed data of each IoV data source sent by the message queue subunit, the valid data sent by the data distribution subunit, and the application data sent by the business subunit;

[0070] A data statistics module 402 is used to perform data statistics on the processed data to obtain upstream statistical data corresponding to the data to be processed, dimension statistical data corresponding to the valid data, and landing statistical data corresponding to the application data;

[0071] The data detection module 403 is used to perform consistency detection on the upstream statistical data, the dimension statistical data and the landing statistical data to obtain a consistency detection result;

[0072] The result display module 404 is used to generate abnormal prompt information according to the consistency detection result if the consistency detection result is abnormal, and send the abnormal prompt information to the user.

[0073] Optionally, the data detection module 403 is specifically used to: perform data volume consistency detection on the dimension statistical data and the landing statistical data according to the data identifier of each piece of data of the processed data, and obtain a data volume consistency detection result;

[0074] According to the timestamp of each piece of data, a time consistency test is performed on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain a time consistency test result;

[0075] According to the data content of each piece of data, a content consistency test is performed on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain a content consistency test result;

[0076] The consistency detection result is obtained based on the data volume consistency detection result, the time consistency detection result and the content consistency detection result.

[0077] Optionally, the data detection module 403 is further used to: receive the data tracing condition sent by the user, wherein the tracing condition includes one or more of a processing node, a tracing dimension and a screening condition; find the target data corresponding to the tracing condition in the processed data, and perform visualization processing on the target data to obtain visualized target data;

[0078] The visualized target data is presented to the user.

[0079] Optionally, the data detection module 403 is further used to: obtain the to-be-processed data of each Internet of Vehicles data source through the message queue subunit;

[0080] Pre-dividing the data to be processed according to each pre-set dimension table to obtain valid data and invalid data;

[0081] The valid data is processed according to the business requirements of the business sub-unit to obtain application data of the business sub-unit.

[0082] Optionally, the data detection module 403 is further used to: perform dimensional analysis on the data to be processed, and input the data to be processed into each dimensional table according to the dimensional analysis result to obtain data of each dimensional table;

[0083] Obtaining the filtering conditions at the current moment, and filtering out the valid data from the data tables of each dimension according to the filtering conditions;

[0084] The data other than the valid data in the dimensional data tables are determined as the invalid data.

[0085] Optionally, the data detection module 403 is further used to: perform data retention analysis on the invalid data based on a preset invalid retention index to obtain a data retention analysis result; wherein the invalidity index includes: one or more of the data volume, data type and data generation frequency; the retention analysis result is retention or non-retention;

[0086] When the retention analysis result is retention, the invalid data is stored in a pre-established invalid database.

[0087] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, reference can be made to the description in any method embodiment of the present invention.

[0088] An embodiment of the present invention also provides a computer program product.

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer program products, which may include one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, referring to Figure 5 , Figure 5 The electronic device 12 shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Figure 5As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0091] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0092] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0093] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0094] A program / utility 40 having a set (at least one) of program modules 46 may be stored, for example, in the memory 28, such program modules 46 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 46 generally perform the functions and / or methods of the embodiments described herein.

[0095] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0096] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a data processing method provided by an embodiment of the present invention: receiving processing data of each processing node in the data processing process sent by the data processing unit, the processing data including the to-be-processed data of each Internet of Vehicles data source sent by the message queue subunit, the valid data sent by the data diversion subunit, and the application data sent by the business subunit; performing data statistics on the processed data to obtain upstream statistical data corresponding to the to-be-processed data, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data; performing consistency check on the upstream statistical data, the dimensional statistical data, and the landing statistical data to obtain a consistency check result; if the consistency check result is abnormal, generating abnormal prompt information according to the consistency check result, and sending the abnormal prompt information to the user.

[0097] The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, a data processing method as provided in all the embodiments of the present invention is implemented: receiving the processing data of each processing node in the data processing process sent by a data processing unit, the processing data includes the to-be-processed data of each vehicle network data source sent by a message queue subunit, the valid data sent by a data diversion subunit, and the application data sent by a business subunit; performing data statistics on the processed data to obtain the upstream statistical data corresponding to the to-be-processed data, the dimension statistical data corresponding to the valid data, and the landing statistical data corresponding to the application data; performing consistency detection on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain a consistency detection result; if the consistency detection result is abnormal, generating abnormal prompt information according to the consistency detection result, and sending the abnormal prompt information to the user. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction-executing electronic device, apparatus, or device.

[0098] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction-executing electronic device, apparatus, or device.

[0099] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0100] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Receive processing data of each processing node in the data processing process sent by the data processing unit, wherein the processing data includes the to-be-processed data of each vehicle network data source sent by the message queue subunit, the valid data sent by the data distribution subunit, and the application data sent by the business subunit; Performing data statistics on the processed data to obtain upstream statistical data corresponding to the data to be processed, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data; Performing consistency check on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain a consistency check result; If the consistency detection result is abnormal, abnormal prompt information is generated according to the consistency detection result, and the abnormal prompt information is sent to the user.

2. The method according to claim 1, characterized in that Perform consistency check on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain consistency check results, including: According to the data identifier of each piece of data of the processed data, a data volume consistency test is performed on the dimension statistical data and the landing statistical data to obtain a data volume consistency test result; According to the timestamp of each piece of data, a time consistency test is performed on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain a time consistency test result; According to the data content of each piece of data, a content consistency test is performed on the upstream statistical data, the dimension statistical data, and the landing statistical data to obtain a content consistency test result; The consistency detection result is obtained based on the data volume consistency detection result, the time consistency detection result and the content consistency detection result.

3. The method according to claim 1, characterized in that: The method further comprises: receiving the data tracing condition sent by the user, wherein the tracing condition includes one or more of a processing node, a tracing dimension, and a screening condition; searching for target data corresponding to the tracing condition in the processed data, and performing visualization processing on the target data to obtain visualized target data; The visualized target data is presented to the user.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire the to-be-processed data of each IoV data source through the message queue subunit; Pre-dividing the data to be processed according to each pre-set dimension table to obtain valid data and invalid data; The valid data is processed according to the business requirements of the business sub-unit to obtain application data of the business sub-unit.

5. The method according to claim 4, characterized in that Pre-dividing the data to be processed according to each preset dimension table to obtain valid data and invalid data includes: Performing dimensional analysis on the data to be processed, and inputting the data to be processed into each dimensional table according to the dimensional analysis result to obtain data of each dimensional table; Obtaining the filtering conditions at the current moment, and filtering out the valid data from the data tables of each dimension according to the filtering conditions; The data other than the valid data in the dimensional data tables are determined as the invalid data.

6. The method according to claim 4, characterized in that The method further comprises: Performing data retention analysis on the invalid data based on a preset invalid retention index to obtain a data retention analysis result; wherein the invalidity index includes: one or more of data volume, data type and data generation frequency; the retention analysis result is retention or non-retention; When the retention analysis result is retention, the invalid data is stored in a pre-established invalid database.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements a data processing method according to any one of claims 1 to 6.

8. A data processing device, characterized in that: include: A data acquisition module, used to receive the processing data of each processing node in the data processing process sent by the data processing unit, wherein the processing data includes the to-be-processed data of each IoV data source sent by the message queue subunit, the valid data sent by the data distribution subunit, and the application data sent by the business subunit; A data statistics module is used to perform data statistics on the processed data to obtain upstream statistical data corresponding to the data to be processed, dimensional statistical data corresponding to the valid data, and landing statistical data corresponding to the application data; A data detection module is used to perform consistency detection on the upstream statistical data, the dimension statistical data and the landing statistical data to obtain a consistency detection result; The result display module is used to generate abnormal prompt information according to the consistency detection result if the consistency detection result is abnormal, and send the abnormal prompt information to the user.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data processing method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images