Data processing methods, apparatus, electronic devices, media and computer programs

Abstract

The application relates to the technical field of big data, in particular to a data processing method and device, electronic equipment, medium and computer program, wherein the method comprises the following steps: acquiring vehicle-end signal data; obtaining a numerical variable by performing real-time or offline calculation on the vehicle-end signal data; performing grouped calculation on the numerical variable according to a data range until a target chi-square value is obtained after a preset condition is met; determining the data characteristics of the vehicle-end signal data based on the target chi-square value; and realizing intelligent binning of the vehicle-end signal data according to the data characteristics. Thus, the problems of poor flexibility and poor expansibility in the related art, which are caused by the mode of storing data in a database and obtaining data by manually adding statements, are solved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, electronic device, medium and computer program. Background Technology

[0002] The process of vehicles uploading data signals to the cloud involves multiple stages, including cleaning, storage, computation, and application. The main technologies employed are traditional OLTP (On-Line Transaction Processing) databases based on SQL (Structured Query Language) for integrated processing and data mining at each stage. The general technical requirements are MySQL, Hive, or other databases for data storage, and the use of standard SQL for analysis and computation.

[0003] However, when the data volume is large, MySQL database storage is at risk of read and write blocking, and when the upstream adds fields, DDL (Data Definition Language) statements need to be added manually to retrieve the data, resulting in poor data flexibility. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, medium, and computer program to solve the problems of poor flexibility and poor scalability in related technologies that store data in a database and retrieve data by manually adding statements.

[0005] The first aspect of this application provides a data processing method, including the following steps: acquiring vehicle-side signal data; performing real-time or offline calculations on the vehicle-side signal data to obtain numerical variables; grouping and calculating the numerical variables according to the data range until a preset condition is met to obtain a target chi-square value; determining the data characteristics of the vehicle-side signal data based on the target chi-square value; and implementing intelligent binning of the vehicle-side signal data according to the data characteristics.

[0006] Optionally, the numerical variables are grouped and calculated according to the data range until a target chi-square value is obtained after a preset condition is met. This includes: calculating the chi-square value after merging adjacent groups; merging the pair of groups with the smallest chi-square value until the number of bins is equal to a preset number, and / or the smallest chi-square value is less than a preset threshold, and then stopping the calculation to obtain the target chi-square value.

[0007] Optionally, intelligent binning of vehicle-side signal data can be implemented based on data characteristics, including generating a universally unique identifier based on a front-end page request; and querying the data stream warehouse based on the universally unique identifier and the data characteristics of the vehicle-side signal data to achieve intelligent binning.

[0008] Optionally, intelligent binning is achieved by querying the data stream warehouse based on the universally unique identifier and the data characteristics of the vehicle-side signal data. This includes: obtaining the data indicators requested by the front-end page and the corresponding placeholders for the front-end page request; configuring a universally unique identifier based on the data indicators requested by the front-end page; and querying the data stream warehouse based on the configured universally unique identifier, the corresponding placeholders for the front-end page request, and the data characteristics of the vehicle-side signal data to achieve intelligent binning.

[0009] Optionally, before obtaining numerical variables after real-time or offline calculation of the vehicle-side signal data, the process also includes: consuming the vehicle-side data signals using the Flink framework; and storing the consumed vehicle-side data signals in a data stream warehouse.

[0010] A second aspect of this application provides a data processing apparatus, comprising: an acquisition module for acquiring vehicle-end signal data; a calculation module for performing real-time or offline calculations on the vehicle-end signal data to obtain numerical variables, grouping the numerical variables according to the data range for calculation until a target chi-square value is obtained after satisfying a preset condition, and determining the data characteristics of the vehicle-end signal data based on the target chi-square value; and a processing module for implementing intelligent binning of the vehicle-end signal data according to the data characteristics.

[0011] Optionally, the calculation module is further used to calculate the chi-square value after merging adjacent groups; merge the pair of groups with the smallest chi-square value until the number of bins is equal to the preset number, and / or the smallest chi-square value is less than the preset threshold, and then stop the calculation to obtain the target chi-square value.

[0012] Optionally, the processing module is further used to generate a universally unique identifier based on the front-end page request; the query module is used to query the data stream warehouse based on the universally unique identifier and the data characteristics of the vehicle-side signal data to achieve intelligent bin sorting.

[0013] Optionally, the processing module is further configured to obtain the data metrics of the front-end page request and the corresponding padding placeholders; configure a universally unique identifier based on the data metrics of the front-end page request; and query the data stream warehouse based on the configured universally unique identifier, the padding placeholders corresponding to the front-end page request, and the data characteristics of the vehicle-side signal data.

[0014] Optionally, the data processing device further includes: a storage module, used to consume vehicle-side data signals using the Flink framework before obtaining numerical variables after real-time or offline calculation of vehicle-side signal data; and to store the consumed vehicle-side data signals to a data stream warehouse.

[0015] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data processing method as described in the above embodiments.

[0016] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the data processing method as described in the above embodiments.

[0017] A fifth aspect of this application provides a computer program product, including: a computer program or instructions, which, when executed, implement the data processing method as described in the above embodiments.

[0018] Therefore, this application has at least the following beneficial effects:

[0019] This application's embodiment can dynamically group vehicle-side signal data according to the data range. As the data volume increases and the data query interval increases, intelligent bins are dynamically divided based on data characteristics. This requires no manual intervention, adapts to changes in data characteristics, avoids the risk of read / write congestion, ensures the timeliness of data read / write, and eliminates the need for complex data interfaces, resulting in high data structure flexibility. Therefore, it solves the problems of poor flexibility and scalability in related technologies that rely solely on database storage and manual data retrieval via added statements.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0022] Figure 1 This is a flowchart of a data processing method provided according to an embodiment of this application;

[0023] Figure 2 This is a data processing technology architecture diagram provided according to an embodiment of this application;

[0024] Figure 3 This is a data processing application architecture diagram provided according to one embodiment of this application;

[0025] Figure 4 This is a timing diagram of data processing according to an embodiment of this application;

[0026] Figure 5 This is an example diagram of a data processing apparatus provided according to an embodiment of this application;

[0027] Figure 6 This is a structural schematic diagram of a vehicle provided according to an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0029] The following description, with reference to the accompanying drawings, outlines a data processing method, apparatus, electronic device, storage medium, and computer program according to embodiments of this application. Addressing the problems mentioned in the background section, this application provides a data processing method. In this method, vehicle-end signal data is dynamically grouped according to data range. As the data volume increases, the data query interval also increases. Intelligent bins are dynamically divided based on data characteristics, eliminating the need for manual intervention. This method adapts to changes in data characteristics, avoiding the risk of read / write congestion, ensuring the timeliness of data read / write, and eliminating the need for complex data interfaces, resulting in high data structure flexibility. This solves the problems of poor flexibility and scalability inherent in related technologies that rely on database storage and manual data retrieval via added statements.

[0030] Specifically, Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this application.

[0031] like Figure 1 As shown, the data processing method includes the following steps:

[0032] In step S101, vehicle-side signal data is acquired.

[0033] The vehicle-side signal data can include TBOX high-frequency signals, TBOX low-frequency signals, event signals, etc. In actual execution, such as... Figure 2 As shown, in this embodiment of the application, vehicle-side signal data can be obtained through Kafka message integration. TBOX high-frequency signals and BOX low-frequency signals can be obtained through topic1, and event signals can be obtained through topic2.

[0034] In step S102, the vehicle-end signal data is calculated in real time or offline to obtain numerical variables. The numerical variables are grouped and calculated according to the data range until the preset conditions are met to obtain the target chi-square value. The data characteristics of the vehicle-end signal data are determined based on the target chi-square value.

[0035] It is understandable that, before obtaining numerical variables after real-time or offline calculation of vehicle-side signal data, embodiments of this application may also utilize the Flink framework to consume vehicle-side data signals and store the consumed vehicle-side data signals in a data stream warehouse.

[0036] Specifically, such as Figure 2As shown in the embodiment of this application, data consumed from Kafka via the Flink framework can be stored in the real-time data stream warehouse Hudi. It should be noted that, as shown in Figure 2, the distributed Kafka, Flink, Spark, Hudi, Clickhouse, Yarn HA cluster, and cloud object storage or distributed HDFS storage components are deployed independently and interconnected via a multi-component network.

[0037] Furthermore, in this embodiment, real-time stream processing analysis of the data can be performed using Flink, and offline scheduling computation can be performed using Spark, thereby forming layers such as the raw data layer, detailed data layer, summary data layer, dimensional data layer, and wide table data layer, as detailed below. Figure 3 As shown.

[0038] After performing real-time or offline calculations on the vehicle-side signal data to obtain numerical variables, the embodiments of this application can dynamically group the numerical variables, and further realize intelligent grouping.

[0039] In one embodiment of this application, numerical variables are grouped and calculated according to the data range until a target chi-square value is obtained after a preset condition is met. This includes: calculating the chi-square value after merging adjacent groups; merging the pair of groups with the smallest chi-square value until the number of bins is equal to a preset number, and / or the smallest chi-square value is less than a preset threshold, and then stopping the calculation to obtain the target chi-square value.

[0040] In this embodiment of the application, the data range can be determined according to the amount of data. For example, if the initial data range is 1-100, the intelligent bin can be divided into 1-50 and 51-100. As the data increases, the data range is 1-10000, and the intelligent bin can be divided into 1-500, 501-2000, 2001-6000, 6001-10000, etc., without manual intervention. At the same time, the number of bins can be set according to the conditions, such as a fixed 5-8 bins, but this setting is also omitted and no specific limitation is made.

[0041] Specifically, the embodiments of this application can determine data characteristics for intelligent binning based on the following steps:

[0042] 1) Sort the numerical variable A and divide it into several groups with a large number of intervals, denoted as A_1, A_2, ..., A_n.

[0043] 2) Calculate the chi-square value of the combined A_1 and A_2, the chi-square value of the combined A_2 and A_3, and so on, until the chi-square value of the combined A_(n-1) and A_n.

[0044] 3) Find the smallest chi-square value among all the merged values ​​from the previous step, let's say it's A_(i-1) and A_i, and merge them to form a new A_(i-1).

[0045] 4) Repeat steps 2 and 3 until the number of bins equals the preset number and / or the minimum chi-square value is less than the preset threshold, then stop the calculation. The preset number and preset threshold can be set according to actual conditions and are not specifically limited.

[0046] In step S103, intelligent sorting of vehicle-side signal data is implemented based on data characteristics.

[0047] It is understood that the embodiments of this application use intelligent binning. As the amount of data continues to increase and the data query interval continues to increase, intelligent binning can be dynamically divided according to the data characteristics. For example, the data may be normally distributed in the range of 0-100000, and may exhibit a skewed distribution in the range of 2000-10000. No manual intervention is required, and the binning can change with the changes in data characteristics.

[0048] In one embodiment of this application, intelligent binning of vehicle-side signal data is implemented based on data characteristics, including: generating a universally unique identifier based on a front-end page request; and querying a data stream warehouse based on the universally unique identifier and the data characteristics of the vehicle-side signal data to achieve intelligent binning.

[0049] Furthermore, intelligent binning is achieved by querying the data stream warehouse based on the universally unique identifier and the data characteristics of the vehicle-side signal data. This includes: obtaining the data indicators requested by the front-end page and the corresponding placeholders for the front-end page request; configuring a universally unique identifier based on the data indicators requested by the front-end page; and querying the data stream warehouse based on the configured universally unique identifier, the corresponding placeholders for the front-end page request, and the data characteristics of the vehicle-side signal data.

[0050] In this embodiment of the application, the report application can be configured using dynamic SQL for the detailed data layer. The specific timing process is as follows: Figure 4 As shown: 1) User / Frontend front-end page request — 2) Receive request and convert to UUID (Universally Unique Identifier) ​​— 3) Query Database SQL based on UUID through Dashboard back-end service — 4) Fill SQL placeholders according to User / Frontend parameters and query Database — 5) Return query results.

[0051] As can be seen from the sequence diagram, dynamic SQL implements the reporting logic, eliminating the need for complex interfaces. A single, universal interface can be used to query database data by configuring different universally unique identifiers for the SQL query. The SQL query conditions are passed to the interface through SQL placeholders (named parameters can be defined by the user); another parameter, UUID, specifies the unique indicator logic. Therefore, this embodiment of the application, through dynamic SQL configuration on the front end, can configure the data indicators to be displayed on the front end page without changing the underlying data (i.e., indicator result1 can be queried through SQL1, indicator result2 can be queried through SQL2, and when the front end page is upgraded and the indicators need to be iterated, only the corresponding SQL logic needs to be changed).

[0052] The data processing method proposed in this application can dynamically group vehicle-side signal data according to the data range. As the data volume increases and the data query interval increases, the method dynamically divides the data into intelligent bins based on data characteristics. This eliminates the need for manual intervention, adapts to changes in data characteristics, avoids the risk of read / write congestion, ensures the timeliness of data read / write, and does not require complex data interfaces, resulting in high data structure flexibility. Therefore, it solves the problems of poor flexibility and scalability associated with storing data in a database and manually adding statements to retrieve data in related technologies.

[0053] Next, the data processing apparatus proposed according to embodiments of this application is described with reference to the accompanying drawings.

[0054] Figure 5 This is a block diagram of a data processing apparatus according to an embodiment of this application.

[0055] like Figure 5 As shown, the data processing device 10 includes: an acquisition module 100, a calculation module 200, and a processing module 300.

[0056] The acquisition module 100 is used to acquire vehicle-side signal data; the calculation module 200 is used to perform real-time or offline calculations on the vehicle-side signal data to obtain numerical variables, group the numerical variables according to the data range, and obtain the target chi-square value after meeting the preset conditions, and determine the data characteristics of the vehicle-side signal data based on the target chi-square value; the processing module 300 is used to realize intelligent binning of vehicle-side signal data according to the data characteristics.

[0057] In one embodiment of this application, the calculation module 200 is further used to calculate the chi-square value after merging adjacent groups; merge the pair of groups with the smallest chi-square value until the number of bins is equal to the preset number, and / or the smallest chi-square value is less than the preset threshold, and then stop the calculation to obtain the target chi-square value.

[0058] In one embodiment of this application, the processing module 300 is further configured to generate a universally unique identifier based on a front-end page request; the query module is configured to query the data stream warehouse based on the universally unique identifier and the data characteristics of the vehicle-end signal data to achieve intelligent bin sorting.

[0059] In one embodiment of this application, the processing module 300 is further configured to obtain the data indicators of the front-end page request and the fill placeholders corresponding to the front-end page request; configure a universally unique identifier based on the data indicators of the front-end page request; and query the data stream warehouse to achieve intelligent binning based on the configured universally unique identifier, the fill placeholders corresponding to the front-end page request, and the data characteristics of the vehicle-end signal data.

[0060] In one embodiment of this application, the data processing device 10 further includes: a storage module, used to consume vehicle-side data signals using the Flink framework before obtaining numerical variables after real-time or offline calculation of vehicle-side signal data; and to store the consumed vehicle-side data signals to a data stream warehouse.

[0061] It should be noted that the foregoing explanation of the data processing method embodiments also applies to the data processing apparatus of this embodiment, and will not be repeated here.

[0062] The data processing apparatus proposed in this application can dynamically group vehicle-end signal data according to the data range. As the data volume increases and the data query interval increases, it dynamically divides the data into intelligent bins based on data characteristics. This eliminates the need for manual intervention, adapts to changes in data characteristics, avoids the risk of read / write congestion, ensures the timeliness of data read / write, and does not require complex data interfaces, resulting in high data structure flexibility. Therefore, it solves the problems of poor flexibility and scalability associated with storing data in a database and manually adding statements to retrieve data in related technologies.

[0063] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0064] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0065] When the processor 602 executes the program, it implements the data processing method provided in the above embodiments.

[0066] Furthermore, electronic devices also include:

[0067] Communication interface 603 is used for communication between memory 601 and processor 602.

[0068] The memory 601 is used to store computer programs that can run on the processor 602.

[0069] The memory 601 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0070] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0071] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0072] The processor 602 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0073] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data processing method described above.

[0074] This application also provides a computer program product, including: a computer program or instructions, which, when executed, implement the data processing method described above.

[0075] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0076] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0077] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0078] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0079] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0080] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: acquiring vehicle terminal signal data; performing real-time or offline calculation on the vehicle terminal signal data to obtain numerical variables, performing grouping calculation on the numerical variables according to a data range, and obtaining a target chi-square value after a preset condition is met, and determining data characteristics of the vehicle terminal signal data based on the target chi-square value; realizing intelligent binning of the vehicle terminal signal data according to the data characteristics; before performing real-time or offline calculation on the vehicle terminal signal data to obtain numerical variables, further comprising: consuming the vehicle terminal signal data by using a flink framework; and storing the consumed vehicle terminal signal data to a data stream warehouse; the realizing intelligent binning of the vehicle terminal signal data according to the data characteristics comprises: generating a universally unique identifier based on a front-end page request; and querying the data stream warehouse to realize intelligent binning according to the universally unique identifier and the data characteristics of the vehicle terminal signal data; the querying the data stream warehouse to realize intelligent binning according to the universally unique identifier and the data characteristics of the vehicle terminal signal data comprises: acquiring data indexes of the front-end page request and filling placeholders corresponding to the front-end page request; configuring a universally unique identifier based on the data indexes of the front-end page request; and querying the data stream warehouse to realize intelligent binning according to the configured universally unique identifier, the filling placeholders corresponding to the front-end page request and the data characteristics of the vehicle terminal signal data.

2. The data processing method according to claim 1, characterized in that, the grouping calculation on the numerical variables according to a data range, and obtaining a target chi-square value after a preset condition is met, comprises: calculating chi-square values after adjacent groups are combined; combining the groups with the smallest chi-square values until the number of bins is equal to a preset number, and / or stopping calculation to obtain the target chi-square value when the smallest chi-square value is less than a preset threshold.

3. A data processing apparatus, characterized by, comprise: an acquisition module, configured to acquire vehicle terminal signal data; a calculation module, configured to perform real-time or offline calculation on the vehicle terminal signal data to obtain numerical variables, perform grouping calculation on the numerical variables according to a data range, obtain a target chi-square value after a preset condition is met, and determine data characteristics of the vehicle terminal signal data based on the target chi-square value; a processing module, configured to realize intelligent binning of the vehicle terminal signal data according to the data characteristics; a storage module, configured to, before performing real-time or offline calculation on the vehicle terminal signal data to obtain numerical variables, consume the vehicle terminal signal data by using a flink framework, and store the consumed vehicle terminal signal data to a data stream warehouse; the processing module is further configured to: generate a universally unique identifier based on a front-end page request; and query the data stream warehouse to realize intelligent binning according to the universally unique identifier and the data characteristics of the vehicle terminal signal data; the processing module is further configured to: acquire data indexes of the front-end page request and filling placeholders corresponding to the front-end page request; configure a universally unique identifier based on the data indexes of the front-end page request; and query the data stream warehouse to realize intelligent binning according to the configured universally unique identifier, the filling placeholders corresponding to the front-end page request and the data characteristics of the vehicle terminal signal data.

4. The data processing apparatus according to claim 3, characterized by the calculation module is further configured to: calculating a chi-square value of adjacent bin combinations; combining the pair of bins with the smallest chi-square value until the number of bins equals a preset number and / or the smallest chi-square value is smaller than a preset threshold, and stopping the calculation to obtain the target chi-square value.

5. An electronic device, comprising: comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the data processing method of claim 1 or 2.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor for implementing the data processing method of claim 1 or 2.

7. A computer program product, comprising: Computer program or instructions, characterized in that, when the computer program or instructions are executed, the data processing method of claim 1 or 2 is implemented.

Images