Real-time data statistics method and device, computer equipment, readable storage medium and program product
By reading batch vehicle status data from the data source message queue in the automotive industry, VIN aggregation and in-depth aggregation driven by business scenario demands, the problem of IO pressure in traditional methods is solved, and real-time and effective massive data processing is achieved. Improves system performance and efficiency.
Patent Information
- Application Number
- CN202411038992.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-15
AI Technical Summary
In the automotive industry, traditional data statistics methods have over-consuming system resources due to frequent IO calls, which has become a bottleneck in the performance of cloud storage systems. How to process massive streaming data in real time and effectively has become an urgent problem to be solved.
By reading batch vehicle status data from the data source message queue, performing first aggregation based on the vehicle identification code VIN, determining the aggregation parameters for the second aggregation based on the business scenario requirements, generating the final data set and storing it to the cloud server, using Batch processing with local aggregation strategy to reduce system IO pressure.
It reduces the amount of data transmission between the network, improves the data flow speed and processing efficiency, reduces the frequency of direct write requests to cloud servers, realizes real-time and effective data processing, and enhances the reliability and response speed of the system.
Smart Images

Figure CN118966255A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of big data processing, and in particular to a real-time statistical data method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] The automotive industry is undergoing an unprecedented transformation, with the trend towards intelligence and networking becoming increasingly prominent. This change has not only reshaped the industry landscape, but has also given rise to consumers' strong desire for advanced autonomous driving technology. To keep up with this trend, automakers have stepped up their efforts to build a huge data collection network by equipping advanced hardware such as environmental perception sensors to capture every detail of vehicle operation. Upload key data to the cloud periodically, and use its powerful computing power to conduct in-depth analysis, format conversion, and secure storage. Real-time data statistics in this process are particularly critical, as they can quickly capture data fluctuations and provide decision makers with accurate and timely information support, thereby accelerating the decision-making process and improving overall efficiency.
[0003] When counting massive streaming data, traditional data statistics methods use a one-by-one processing method. Each data processing is accompanied by an independent IO operation. As the amount of data grows, frequent IO calls will lead to excessive consumption of system resources and even become a bottleneck for system performance. After receiving massive vehicle data in the cloud, how to count this data in real time and effectively and alleviate the IO pressure of the cloud storage system has become a key issue that needs to be solved urgently. Summary of the invention
[0004] Based on this, it is necessary to provide a real-time statistical data method, device, computer equipment, computer-readable storage medium and computer program product that can process massive data in real time and effectively in response to the above technical problems.
[0005] In a first aspect, the present application provides a real-time statistical data method, comprising:
[0006] Read batches of vehicle status data from the data source message queue to generate a status data set;
[0007] According to the vehicle identification code VIN contained in each vehicle status data in the status data set, the status data set is first aggregated to generate an intermediate data set; the intermediate data set includes a first aggregate set corresponding to each of the multiple VINs;
[0008] Determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes second aggregation sets corresponding to respective ranges of aggregation parameter values;
[0009] The final data sets corresponding to each state data set are stored in the cloud server.
[0010] In one embodiment, the first aggregation of the status data set according to the vehicle identification code VIN included in each vehicle status data in the status data set includes:
[0011] The vehicle status data with the same VIN in the status data set is taken as a first aggregated set.
[0012] In one of the embodiments, performing a second aggregation on the current first aggregation set according to the aggregation parameter includes:
[0013] Extracting an aggregation parameter value from each piece of vehicle status data included in the current first aggregation set;
[0014] Based on the aggregation parameter value and the preset aggregation rule, a second aggregation is performed on the vehicle status data included in the current first aggregation set.
[0015] In one embodiment, before first aggregating the status data set according to the vehicle identification code VIN included in each vehicle status data in the status data set, the method further includes:
[0016] Each vehicle status data in the status data set is judged, and if the vehicle status data can be parsed successfully, the vehicle status data is retained; if the parsing fails, the vehicle status data is deleted.
[0017] In one embodiment, storing the final data sets corresponding to the respective state data sets in the cloud server includes:
[0018] For the final data set corresponding to each status data set, obtain the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set, splice the VIN and the aggregation parameter value range to obtain the splicing result, use the splicing result as the key and the second aggregation set as the value, and store them in the cloud server.
[0019] In one of the embodiments, before performing a second aggregation on the current first aggregation set according to the aggregation parameter for the first aggregation set corresponding to each VIN in the intermediate data set, the method further includes:
[0020] Sending the intermediate data set to a transit message queue;
[0021] A batch of intermediate data sets are read from the transit message queue.
[0022] In a second aspect, the present application also provides a real-time statistical data device, comprising:
[0023] A data acquisition module is used to read batch vehicle status data from a data source message queue and generate a status data set;
[0024] A first aggregation module, configured to perform a first aggregation on the status data set according to a vehicle identification code VIN included in each vehicle status data in the status data set to generate an intermediate data set; the intermediate data set includes a first aggregation set corresponding to each of the multiple VINs;
[0025] A second aggregation module is used to determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes a plurality of second aggregation sets corresponding to respective aggregation parameter value ranges;
[0026] The data storage module is used to store the final data sets corresponding to each state data set in the cloud server.
[0027] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the above-mentioned real-time statistical data method when executing the computer program.
[0028] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned real-time statistical data method are implemented.
[0029] In a fifth aspect, the present application also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned real-time statistical data method are implemented.
[0030] The above-mentioned real-time data statistics method, device, computer equipment, computer-readable storage medium and computer program product, first, read batch vehicle status data from the data source message queue to generate a status data set; according to the vehicle identification code VIN contained in each vehicle status data in the status data set, the batch status data set is first aggregated, and the first aggregation here reduces the amount of data transmission between networks, reduces bandwidth usage, speeds up data flow, and improves data processing efficiency; thereby generating an intermediate data set, the intermediate data set includes a first aggregation set corresponding to each of the multiple VINs; then determine the aggregation parameters in combination with the business scenario requirements, for the first aggregation set corresponding to each VIN in the intermediate data set, and then perform a second aggregation on the current first aggregation set according to the aggregation parameters, and the second aggregation here can make the data deeply aggregated, ensuring the accuracy of data analysis; thereby obtaining the final data set of the current first aggregation set; the final data set includes a second aggregation set corresponding to each of the multiple aggregation parameter value ranges; finally, the final data set corresponding to each of the status data sets is stored in the cloud server. Through batch processing and local aggregation strategies, the frequency of direct write requests to the cloud server is greatly reduced, the system IO pressure is reduced, real-time and effective data processing is achieved, and the reliability and response speed of the system are enhanced. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0032] Figure 1 A diagram of an application environment of a real-time statistical data collection method in one embodiment;
[0033] FIG2 is a schematic diagram of a flow chart of a method for collecting real-time statistical data in one embodiment;
[0034] Figure 3 is a schematic flow diagram of a first polymerization method in one embodiment;
[0035] Figure 4 is a schematic flow diagram of a second polymerization method in one embodiment;
[0036] Figure 5 A structural diagram of a real-time statistical data model in one embodiment;
[0037] Figure 6 is a structural diagram of a real-time statistical data storage device in one embodiment;
[0038] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0040] The real-time statistical data method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the vehicle 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 reads batches of vehicle status data from the data source message queue to generate a status data set. The server 104 performs a first aggregation on the status data set according to the vehicle identification code VIN contained in each vehicle status data in the status data set to generate an intermediate data set; the intermediate data set includes a first aggregation set corresponding to each of the multiple VINs. The server 104 determines the aggregation parameters in combination with the business scenario requirements, and performs a second aggregation on the current first aggregation set according to the aggregation parameters for the first aggregation set corresponding to each VIN in the intermediate data set to obtain the final data set of the current first aggregation set; the final data set includes a second aggregation set corresponding to each of the multiple aggregation parameter value ranges. The final data sets corresponding to each of the status data sets are stored in the server 104. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0041] In an exemplary embodiment, as shown in FIG2 , a real-time statistical data method is provided, which is applied to Figure 1 102-106 in the example are used to illustrate, including the following steps 102 to 108. Among them:
[0042] Step 202, read batch vehicle status data from the data source message queue to generate a status data set.
[0043] The data source message queue may be one of the following message queues: RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, Pulsar, etc.
[0044] The data source message queue stores a large amount of vehicle status data. Each piece of vehicle status data is sent by the vehicle to the data source message queue. The vehicle status data is collected by sensors on the vehicle. For example, the vehicle status data includes data related to autonomous driving, such as time, fault information, etc.
[0045] Step 204: Perform a first aggregation on the status data set according to the vehicle identification code VIN contained in each vehicle status data in the status data set to generate an intermediate data set; the intermediate data set includes a first aggregation set corresponding to each of the multiple VINs.
[0046] Among them, the vehicle identification code VIN consists of 17 characters and is the unique identification code of the vehicle.
[0047] Exemplarily, the vehicle identification code VIN of each vehicle status data in the status data set is counted, and then the vehicle status data with the same vehicle identification code VIN are combined together to generate multiple first aggregation sets, which are intermediate data sets, namely, hash tables of VIN and VIN statistical data.
[0048] Step 206, determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a second aggregation set of the current first aggregation set; the final data set contains multiple second aggregation sets corresponding to each of the aggregation parameter value ranges.
[0049] Among them, the aggregation parameter is at least one data in the vehicle status data, which is determined by business needs. For example, when fault information needs to be processed, the relevant fault information can be processed specifically according to the fault information and the vehicle identification code VIN.
[0050] Exemplarily, for each first aggregation set in the intermediate data set, the first aggregation set includes multiple vehicle status data obtained by aggregation, and the aggregation parameter values in each vehicle status data are counted. The vehicle status data with the same aggregation parameter values in the first aggregation set are combined together to obtain multiple second aggregation sets corresponding to each first aggregation set.
[0051] Exemplarily, for each first aggregation set in the intermediate data set, the first aggregation set includes multiple vehicle status data obtained by aggregation, all fault information in each vehicle status data is counted, and the vehicle status data with the same fault information in the first aggregation set are combined together to obtain multiple second aggregation sets of the current first aggregation set.
[0052] Step 208: Store the final data sets corresponding to each state data set in the cloud server.
[0053] Among them, the cloud server can be the following databases: Mysql, Oracle, Sql Server, DB2, Access, PostgreSQL, SqLite, OceanBase, GaussDB, TDSQL, DM, Redis, Memcached, MongoDB, CouchDB, Couchbase, HBase, Neo4j, db4o, BaseX, etc.
[0054] Exemplarily, the final data sets corresponding to each state data set are stored in the Redis database in a PipeLine manner.
[0055] Exemplarily, for the final data set corresponding to each status data set, the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set are obtained, the VIN and the aggregation parameter value range are spliced to obtain the splicing result, and the splicing result is used as the key and the second aggregation set as the value and stored in the cloud server.
[0056] In the above embodiment, first, batch vehicle status data is read from the data source message queue to generate a status data set; according to the vehicle identification code VIN contained in each vehicle status data in the status data set, the status data set is first aggregated, and the first aggregation here reduces the amount of data transmission between networks, reduces bandwidth usage, speeds up data flow, and improves data processing efficiency; thereby generating an intermediate data set, the intermediate data set includes a first aggregation set corresponding to each of the multiple VINs; then, the aggregation parameters are determined in combination with the business scenario requirements, and the first aggregation set corresponding to each VIN in the intermediate data set is then second aggregated according to the aggregation parameters. Here, the second aggregation can enable the data to be deeply aggregated and ensure the accuracy of data analysis; thereby obtaining the final data set of the current first aggregation set; the final data set includes a second aggregation set corresponding to each of the multiple aggregation parameter value ranges; finally, the final data set corresponding to each of the status data sets is stored in the cloud server. The batch processing and local aggregation strategies greatly reduce the frequency of direct write requests to the cloud server, reduce the system IO pressure, achieve real-time and effective data processing, and enhance the reliability and response speed of the system.
[0057] In an exemplary embodiment, Figure 3 As shown, the first aggregation of the status data set is performed according to the vehicle identification code VIN contained in each vehicle status data in the status data set, including: taking the vehicle status data with the same VIN in the status data set as a first aggregation set.
[0058] Exemplarily, the vehicle identification code VIN of each vehicle status data in the statistical status data set is counted, all vehicle status data are classified according to the vehicle identification code VIN, and then the data belonging to the same vehicle identification code VIN are combined together to generate multiple first aggregation sets, and the multiple first aggregation sets are intermediate data sets, namely, hash tables of VIN and VIN statistical data.
[0059] Among them, the hash table is also called a hash table, which is extended from an array and is a data structure that directly accesses the storage location in the memory based on the key.
[0060] In this embodiment, the first aggregation reduces the amount of data transmitted between networks, reduces bandwidth usage, speeds up data flow, and improves data processing efficiency.
[0061] In an exemplary embodiment, Figure 4 The second aggregation of the current first aggregation set according to the aggregation parameter includes: extracting the aggregation parameter value in each vehicle status data contained in the current first aggregation set; based on the aggregation parameter value and the preset aggregation rule, performing the second aggregation on the vehicle status data contained in the current first aggregation set.
[0062] The aggregation rule is a rule for implementing a set rule for classifying the data in the first aggregation set according to the value of the aggregation parameter.
[0063] Exemplarily, all types of fault information of the current first aggregation set are counted, such as engine fault, transmission system fault, brake system fault, etc. For different fault information, the data in the current first aggregation set are classified according to the fault information, and the data belonging to the same type of fault are combined together to obtain multiple second aggregation sets of the current first aggregation set, that is, a hash table with fault information and vin concatenated as the key and vin statistical information as the value.
[0064] Exemplarily, all fault information of the current first aggregation set is counted. The data in the first aggregation set whose fault information is engine fault are combined together to obtain a second aggregation set of the current first aggregation set. Similarly, the first data set is grouped according to fault information such as transmission system fault and brake system fault, and multiple second aggregation sets of the current first aggregation set can be obtained, that is, a hash table with fault information and vin concatenated as keys and vin statistical information as values.
[0065] In this embodiment, the data can be deeply aggregated through the second aggregation, thereby ensuring the accuracy of data analysis.
[0066] In an exemplary embodiment, before performing the first aggregation on the status data set based on the vehicle identification code VIN contained in each vehicle status data in the status data set, the method further includes: judging each vehicle status data in the status data set, and retaining the vehicle status data if the vehicle status data can be parsed successfully; and deleting the vehicle status data if the parsing fails.
[0067] Exemplarily, each vehicle status data in the status data set is read in batches, and each status data is judged. If the vehicle status data can be parsed successfully, the vehicle status data is retained; if the parsing fails, the vehicle status data is deleted.
[0068] In this embodiment, by judging each piece of vehicle status data, the data that cannot be parsed can be effectively removed, thereby alleviating the pressure of data processing.
[0069] In an exemplary embodiment, the storing of the final data sets corresponding to each status data set in the cloud server includes: for the final data set corresponding to each status data set, obtaining the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set, splicing the VIN and the aggregation parameter value range to obtain a splicing result, using the splicing result as a key and the second aggregation set as a value, and storing them in the cloud server.
[0070] Exemplarily, for the final data set corresponding to each status data set, the VIN and fault information corresponding to each second aggregation set in the final data set are obtained, the VIN and the fault information are spliced to obtain a splicing result, and the splicing result is used as a key and the second aggregation set is used as a value to be stored in the cloud server.
[0071] In this embodiment, by aggregating the VIN and the aggregation parameters, data can be extracted in a targeted manner according to business needs, and the required data can be processed efficiently.
[0072] In an exemplary embodiment, before performing a second aggregation on the current first aggregation set according to the aggregation parameters for the first aggregation set corresponding to each VIN in the intermediate data set, it also includes: sending the intermediate data set to a transit message queue; and reading batches of intermediate data sets from the transit message queue.
[0073] Exemplarily, the intermediate data set is sent to a transit message queue on a server, and the intermediate data is read in batches from the transit message queue.
[0074] In this embodiment, by saving the intermediate data set to the transit message queue, the intermediate data set can be read from the transit message queue at any time when the intermediate data set is needed.
[0075] In an exemplary embodiment, Figure 5 As shown, a model diagram of real-time statistical data is provided. First, batch vehicle status data is read from the data source message queue to generate a status data set; then, according to the vehicle identification code VIN contained in each vehicle status data in the status data set, the status data set is first aggregated to generate an intermediate data set, which is stored in the transit message queue and the first message queue; then, the aggregation parameters are determined in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set in the transit message queue, the current first aggregation set is second aggregated according to the aggregation parameters to obtain the final data set of the current first aggregation set; similarly, the above operations are performed in other containers, that is, the above operations are performed in multiple containers such as container 2 and container 3, and the final data sets corresponding to multiple status data sets are obtained, and finally the final data sets corresponding to each status data set are stored in the cloud server Redis.
[0076] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0077] In an exemplary embodiment, Figure 6 As shown, a real-time statistical data device is provided, comprising: a data acquisition module, a first aggregation module, a second aggregation module and a data storage module, wherein:
[0078] A data acquisition module is used to read batch vehicle status data from a data source message queue and generate a status data set;
[0079] A first aggregation module, configured to perform a first aggregation on the status data set according to a vehicle identification code VIN included in each vehicle status data in the status data set to generate an intermediate data set; the intermediate data set includes a first aggregation set corresponding to each of the multiple VINs;
[0080] A second aggregation module is used to determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes a plurality of second aggregation sets corresponding to respective aggregation parameter value ranges;
[0081] The data storage module is used to store the final data sets corresponding to each state data set in the cloud server.
[0082] In one embodiment, the first aggregation module is further configured to:
[0083] The vehicle status data with the same VIN in the status data set is taken as a first aggregated set.
[0084] In one embodiment, the second aggregation module is further configured to:
[0085] Extracting an aggregation parameter value from each piece of vehicle status data included in the current first aggregation set;
[0086] Based on the aggregation parameter value and the preset aggregation rule, a second aggregation is performed on the vehicle status data included in the current first aggregation set.
[0087] In one embodiment, the data acquisition module is further used to:
[0088] Each vehicle status data in the status data set is judged, and if the vehicle status data can be parsed successfully, the vehicle status data is retained; if the parsing fails, the vehicle status data is deleted.
[0089] In one embodiment, the second aggregation module is further configured to:
[0090] For the final data set corresponding to each status data set, obtain the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set, splice the VIN and the aggregation parameter value range to obtain the splicing result, use the splicing result as the key and the second aggregation set as the value, and store them in the cloud server.
[0091] In one embodiment, the first aggregation module is further configured to:
[0092] Sending the intermediate data set to a transit message queue;
[0093] A batch of intermediate data sets are read from the transit message queue.
[0094] Each module in the above-mentioned insulating medium loss detection device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.
[0095] Each module in the above-mentioned real-time statistical data device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0096] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for the final data set. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a real-time statistical data method is implemented.
[0097] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0098] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0099] Read batches of vehicle status data from the data source message queue to generate a status data set;
[0100] According to the vehicle identification code VIN contained in each vehicle status data in the status data set, the status data set is first aggregated to generate an intermediate data set; the intermediate data set includes a first aggregate set corresponding to each of the multiple VINs;
[0101] Determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes second aggregation sets corresponding to respective ranges of aggregation parameter values;
[0102] The final data sets corresponding to each state data set are stored in the cloud server.
[0103] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0104] The vehicle status data with the same VIN in the status data set is taken as a first aggregated set.
[0105] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0106] Extracting an aggregation parameter value from each piece of vehicle status data included in the current first aggregation set;
[0107] Based on the aggregation parameter value and the preset aggregation rule, a second aggregation is performed on the vehicle status data included in the current first aggregation set.
[0108] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0109] Each vehicle status data in the status data set is judged, and if the vehicle status data can be parsed successfully, the vehicle status data is retained; if the parsing fails, the vehicle status data is deleted.
[0110] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0111] For the final data set corresponding to each status data set, obtain the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set, splice the VIN and the aggregation parameter value range to obtain the splicing result, use the splicing result as the key and the second aggregation set as the value, and store them in the cloud server.
[0112] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0113] Sending the intermediate data set to a transit message queue;
[0114] A batch of intermediate data sets are read from the transit message queue.
[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0116] Read batches of vehicle status data from the data source message queue to generate a status data set;
[0117] According to the vehicle identification code VIN contained in each vehicle status data in the status data set, the status data set is first aggregated to generate an intermediate data set; the intermediate data set includes a first aggregate set corresponding to each of the multiple VINs;
[0118] Determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes second aggregation sets corresponding to respective ranges of aggregation parameter values;
[0119] The final data sets corresponding to each state data set are stored in the cloud server.
[0120] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0121] The vehicle status data with the same VIN in the status data set is taken as a first aggregated set.
[0122] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0123] Extracting an aggregation parameter value from each piece of vehicle status data included in the current first aggregation set;
[0124] Based on the aggregation parameter value and the preset aggregation rule, a second aggregation is performed on the vehicle status data included in the current first aggregation set.
[0125] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0126] Each vehicle status data in the status data set is judged, and if the vehicle status data can be parsed successfully, the vehicle status data is retained; if the parsing fails, the vehicle status data is deleted.
[0127] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0128] For the final data set corresponding to each status data set, obtain the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set, splice the VIN and the aggregation parameter value range to obtain the splicing result, use the splicing result as the key and the second aggregation set as the value, and store them in the cloud server.
[0129] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0130] Sending the intermediate data set to a transit message queue;
[0131] A batch of intermediate data sets are read from the transit message queue.
[0132] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0133] Read batches of vehicle status data from the data source message queue to generate a status data set;
[0134] According to the vehicle identification code VIN contained in each vehicle status data in the status data set, the status data set is first aggregated to generate an intermediate data set; the intermediate data set includes a first aggregate set corresponding to each of the multiple VINs;
[0135] Determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes second aggregation sets corresponding to respective ranges of aggregation parameter values;
[0136] The final data sets corresponding to each state data set are stored in the cloud server.
[0137] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0138] The vehicle status data with the same VIN in the status data set is taken as a first aggregated set.
[0139] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0140] Extracting an aggregation parameter value from each piece of vehicle status data included in the current first aggregation set;
[0141] Based on the aggregation parameter value and the preset aggregation rule, a second aggregation is performed on the vehicle status data included in the current first aggregation set.
[0142] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0143] Each vehicle status data in the status data set is judged, and if the vehicle status data can be parsed successfully, the vehicle status data is retained; if the parsing fails, the vehicle status data is deleted.
[0144] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0145] For the final data set corresponding to each status data set, obtain the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set, splice the VIN and the aggregation parameter value range to obtain the splicing result, use the splicing result as the key and the second aggregation set as the value, and store them in the cloud server.
[0146] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0147] Sending the intermediate data set to a transit message queue;
[0148] A batch of intermediate data sets are read from the transit message queue.
[0149] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0151] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for real-time statistical data, characterized in that: The method comprises: Read batches of vehicle status data from the data source message queue to generate a status data set; According to the vehicle identification code VIN contained in each vehicle status data in the status data set, the status data set is first aggregated to generate an intermediate data set; the intermediate data set includes a first aggregate set corresponding to each of the multiple VINs; Determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes second aggregation sets corresponding to respective ranges of aggregation parameter values; The final data sets corresponding to each state data set are stored in the cloud server.
2. The method according to claim 1, characterized in that The first aggregation of the status data set according to the vehicle identification code VIN contained in each vehicle status data in the status data set includes: The vehicle status data with the same VIN in the status data set is taken as a first aggregated set.
3. The method according to claim 1, characterized in that The performing a second aggregation on the current first aggregation set according to the aggregation parameter includes: Extracting an aggregation parameter value from each piece of vehicle status data included in the current first aggregation set; Based on the aggregation parameter value and the preset aggregation rule, a second aggregation is performed on the vehicle status data included in the current first aggregation set.
4. The method according to claim 1, characterized in that: Before performing a first aggregation on the status data set according to the vehicle identification code VIN included in each piece of vehicle status data in the status data set, the method further includes: Each vehicle status data in the status data set is judged, and if the vehicle status data can be parsed successfully, the vehicle status data is retained; if the parsing fails, the vehicle status data is deleted.
5. The method according to claim 1, characterized in that The storing of the final data sets corresponding to the respective status data sets in the cloud server includes: For the final data set corresponding to each status data set, obtain the VIN and aggregation parameter value range corresponding to each second aggregation set in the final data set, splice the VIN and the aggregation parameter value range to obtain the splicing result, use the splicing result as the key and the second aggregation set as the value, and store them in the cloud server.
6. The method according to claim 1, characterized in that Before performing a second aggregation on the current first aggregation set according to the aggregation parameter for the first aggregation set corresponding to each VIN in the intermediate data set, the method further includes: Sending the intermediate data set to a transit message queue; A batch of intermediate data sets are read from the transit message queue.
7. A real-time statistical data device, characterized in that: The device comprises: The data acquisition module is used to read batch vehicle status data from the data source message queue and generate a status data set; A first aggregation module, configured to perform a first aggregation on the status data set according to a vehicle identification code VIN included in each vehicle status data in the status data set to generate an intermediate data set; the intermediate data set includes a first aggregation set corresponding to each of the multiple VINs; A second aggregation module is used to determine the aggregation parameters in combination with the business scenario requirements, and for the first aggregation set corresponding to each VIN in the intermediate data set, perform a second aggregation on the current first aggregation set according to the aggregation parameters to obtain a final data set of the current first aggregation set; the final data set includes a plurality of second aggregation sets corresponding to respective aggregation parameter value ranges; The data storage module is used to store the final data sets corresponding to each state data set in the cloud server.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
