Data processing method and device, communication device, storage medium and vehicle

Abstract

This invention relates to a data processing method, apparatus, communication equipment, storage medium, and vehicle. The method involves obtaining industrial signal data corresponding to table fields in a preset business detail table; filtering the industrial signal data to obtain target industrial signal data; aggregating the target industrial signal data based on its corresponding second-level time interval to obtain aggregated industrial signal data; and determining the aggregated industrial signal data corresponding to the table fields in the preset business detail table based on the aggregated industrial signal data. This application, by filtering industrial signal data and retaining target industrial signal data that meets requirements and quality, and by performing second-level time aggregation on the target industrial signal data, can eliminate redundant data from multiple data points per second. This allows for the retention of valid data while eliminating redundant and irrelevant invalid data, improving overall data granularity and meeting data business needs with less data.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a data processing method, apparatus, communication equipment, storage medium, and vehicle. Background Technology

[0002] With the development of industrial big data, the demand for intelligent services and development in various industries is increasing, especially in the field of industrial equipment. For example, with the gradual improvement of intelligent vehicle networking technology, users' needs for vehicles are no longer limited to vehicle driving performance, but are expanding to areas such as vehicle comfort, functional adaptability, intelligent cockpit, intelligent human-vehicle interaction, and intelligent driving. In order to achieve the above functions, the vehicle needs to collect a large amount of vehicle-related data and transmit this data back to the car manufacturer for processing and backup.

[0003] However, in the existing technology, in the process of realizing industrial signal data transmission and processing, such as vehicle CAN signals and other industrial intelligent big data acquisition signals, there are many invalid data entries and the data requires a large space, which leads to excessive pressure on the big data cluster, affecting the timely access of subsequent data analysis and processing or consuming a large amount of computing resources. Summary of the Invention

[0004] One objective of this invention is to provide a data processing method to solve the problems of large number of invalid data entries and large space requirements caused by the large storage volume of industrial intelligent big data acquisition signal uploads in the prior art; a second objective is to provide a data processing device; a third objective is to provide a communication device; a fourth objective is to provide a storage medium; and a fifth objective is to provide a vehicle.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A first aspect of this application provides a data processing method, the data processing method comprising:

[0007] The industrial signal data corresponding to the table fields in the preset business details table is obtained, wherein the industrial signal data is obtained from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud.

[0008] The industrial signal data is filtered to obtain the target industrial signal data;

[0009] The target industrial signal data is aggregated based on the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data.

[0010] The aggregated industrial signal data corresponding to the table fields in the preset business details table is determined based on the aggregated industrial signal data.

[0011] Furthermore, the step of aggregating the target industrial signal data based on the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data includes:

[0012] Based on the second-level time corresponding to the target industrial signal data and the different signal codes corresponding to the target industrial signal data, all the industrial signal data are aggregated to obtain aggregated industrial signal data.

[0013] Furthermore, the signal encoding is based on the functional category corresponding to the target industrial signal data.

[0014] Furthermore, the step of aggregating all the industrial signal data according to the second-level time corresponding to the target industrial signal data and the different signal codes corresponding to the target industrial signal data to obtain aggregated industrial signal data includes:

[0015] Based on the second-level time corresponding to the target industrial signal data, all the target industrial signal data are aggregated to obtain a main table, which includes a unique device identifier and a second-level time field.

[0016] Based on the different signal codes corresponding to the target industrial signal data and the second-level time, all the target industrial signal data are aggregated to obtain a sub-table corresponding to the target industrial signal data with different signal codes. The sub-table includes the unique identifier of the equipment, the second-level time field, and the aggregated industrial signal data corresponding to the table field.

[0017] The main table and at least one of the sub-tables are aligned to obtain an event detail table, wherein the event detail table includes aggregated industrial signal data.

[0018] Furthermore, the step of aggregating all the target industrial signal data according to the different signal codes corresponding to the target industrial signal data and the second-level time to obtain the sub-table corresponding to the target industrial signal data with different signal codes includes:

[0019] All the target industrial signal data are split according to the signal encoding to obtain multiple target industrial signal data corresponding to different signal codes;

[0020] If it is detected that the second target industrial signal data corresponding to any signal code within the same second-level time field is a non-null value and the first target industrial signal data is a null value, or if the second target industrial signal data corresponding to the same signal code within the same second-level time field is a null value and the first target industrial signal data is a non-null value, the non-null values ​​corresponding to the first target industrial signal data and the non-null values ​​corresponding to the second target industrial signal data are aggregated according to the second-level time to obtain a sub-table corresponding to the target industrial signal data corresponding to the signal code. The sub-table includes the signal code, the vehicle unique identifier, the second-level time field, and the data field.

[0021] Furthermore, the step of splitting all the target industrial signal data according to the signal encoding to obtain multiple target industrial signal data corresponding to different signal codes includes:

[0022] The target industrial signal data is split according to the signal code to obtain multiple sets of target industrial signal data, wherein each set of target industrial signal data corresponds to one of the signal codes;

[0023] The first or last target industrial signal data in each group of target industrial signal data, sorted by millisecond time, is selected as the target industrial signal data corresponding to different signal codes.

[0024] Furthermore, the step of aligning the main table and at least one of the sub-tables to obtain the event details table includes:

[0025] The event details table is obtained by aligning the device unique identifier and the second-level time field in the main table and the sub-table.

[0026] Furthermore, the step of obtaining the industrial signal data corresponding to the table fields in the preset business details table includes:

[0027] Receive business function instructions sent by users;

[0028] A preset business detail table is generated according to the business function instruction, wherein the preset business detail table includes table fields, the table fields include a second-level time field and a data field, and one second-level time field corresponds to multiple data fields;

[0029] The industrial signal data corresponding to the data field is obtained from the preset acquisition signal matrix according to the data field, wherein one data field corresponds to at least one industrial signal data.

[0030] Furthermore, the process of filtering the industrial signal data to obtain the target industrial signal data includes:

[0031] The industrial signal data is filtered according to preset filtering rules to obtain target industrial signal data.

[0032] Furthermore, the preset filtering rules include that the value corresponding to the industrial signal data is a non-empty value, and that the frequency corresponding to the industrial signal data is greater than the target threshold.

[0033] Furthermore, the second-level time is the time extracted in second-level units from the millisecond-level time string corresponding to the industrial signal data.

[0034] In another aspect of this application, a data processing apparatus is also provided, the data processing apparatus comprising:

[0035] The acquisition module is used to acquire industrial signal data corresponding to the table fields in the preset business details table, wherein the industrial signal data is acquired from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud.

[0036] The filtering module is used to filter the industrial signal data to obtain target industrial signal data;

[0037] The aggregation module is used to aggregate the target industrial signal data according to the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data.

[0038] The determination module is used to determine the aggregated industrial signal data corresponding to the table fields in the preset business details table based on the aggregated industrial signal data.

[0039] Furthermore, the aggregation module includes:

[0040] The first aggregation submodule is used to aggregate all the industrial signal data according to the second-level time corresponding to the target industrial signal data and the different signal codes corresponding to the target industrial signal data to obtain aggregated industrial signal data, wherein the signal code is the signal channel corresponding to the target industrial signal data.

[0041] The first aggregation submodule includes:

[0042] The first aggregation unit is used to aggregate all the target industrial signal data according to the second-level time corresponding to the target industrial signal data to obtain a main table, the main table including a unique device identifier and a second-level time field.

[0043] The second aggregation unit is used to aggregate all the target industrial signal data according to the different signal codes corresponding to the target industrial signal data and the second-level time, to obtain a sub-table corresponding to the target industrial signal data with different signal codes, wherein the sub-table includes the unique identifier of the equipment, the second-level time field and the aggregated industrial signal data corresponding to the table field;

[0044] An alignment unit is used to align the main table and at least one of the sub-tables to obtain an event detail table, wherein the event detail table includes aggregated industrial signal data.

[0045] Furthermore, the second aggregation submodule includes:

[0046] The sub-unit is used to divide all the target industrial signal data according to the signal encoding to obtain multiple target industrial signal data corresponding to different signal encodings;

[0047] An aggregation subunit is used to aggregate the non-null values ​​corresponding to the first target industrial signal data and the non-null values ​​corresponding to the second target industrial signal data based on the second-level time field when it is detected that the second target industrial signal data corresponding to any signal code within the same second-level time field is non-null and the first target industrial signal data is null, or when the second target industrial signal data corresponding to the same signal code within the same second-level time field is null and the first target industrial signal data is non-null, to obtain a sub-table corresponding to the target industrial signal data corresponding to the signal code. The sub-table includes a signal code, a unique device identifier, a second-level time field, and a data field.

[0048] Furthermore, the alignment unit includes:

[0049] The alignment sub-unit is used to perform alignment processing based on the device unique identifier and the second-level time field in the main table and the sub-table to obtain the event details table.

[0050] Furthermore, the acquisition module includes:

[0051] The receiving submodule is used to receive business function instructions sent by the user;

[0052] A generation submodule is used to generate a preset business detail table according to the business function instruction. The preset business detail table includes table fields, which include a second-level time field and a data field. One second-level time field corresponds to multiple data fields.

[0053] The acquisition submodule is used to acquire industrial signal data corresponding to the data field in a preset acquisition signal matrix according to the data field, wherein one data field corresponds to at least one industrial signal data.

[0054] Furthermore, the filtering module includes:

[0055] The filtering submodule is used to filter the industrial signal data according to preset filtering rules to obtain target industrial signal data.

[0056] Furthermore, the filtering submodule is specifically used for the preset filtering rules, which include the industrial signal data having a non-empty value and the industrial signal data having a frequency greater than a target threshold.

[0057] Furthermore, the aggregation module is specifically used to extract the second-level time from the millisecond-level time string corresponding to the industrial signal data.

[0058] In another aspect of this application, a communication device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0059] Memory, used to store computer programs;

[0060] A processor, when executing a program stored in memory, implements any of the data processing methods described above.

[0061] In another aspect of this application, a computer-readable storage medium is also provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform any of the data processing methods described above.

[0062] In another aspect of this application, a vehicle is also provided, the vehicle including any of the data processing devices described above.

[0063] The beneficial effects of this invention are:

[0064] The data processing method provided in this application involves obtaining industrial signal data corresponding to table fields in a preset business detail table. The industrial signal data is obtained from the full volume of industrial signal data uploaded from preset industrial equipment to a preset cloud. The industrial signal data is then filtered to obtain target industrial signal data. This target industrial signal data is then aggregated based on its corresponding second-level time interval to obtain aggregated industrial signal data. Finally, the aggregated industrial signal data corresponding to the table fields in the preset business detail table is determined based on the aggregated industrial signal data. In other words, by filtering the obtained industrial signal data, this application can optimize the original full volume of data in the first step, retaining target industrial signal data that meets the requirements and quality. Further aggregation based on second-level time intervals on the target industrial signal data yields aggregated industrial signal data. This process can remove redundant data from multiple data entries per second, achieving the retention of valid data while eliminating redundant and irrelevant invalid data. This improves the overall data granularity and retention, meeting data business needs with less data and more data. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0066] Figure 1 The flowchart illustrating the steps of the data processing method provided in the embodiments of this application is shown. Figure 1 ;

[0067] Figure 2 The flowchart illustrating the steps of the data processing method provided in the embodiments of this application is shown. Figure 2 ;

[0068] Figure 3 The flowchart illustrating the steps of the data processing method provided in the embodiments of this application is shown. Figure 3 ;

[0069] Figure 4 This invention provides a block diagram of a data processing method apparatus according to an embodiment of the present application.

[0070] Figure 5 This paper shows a structural block diagram of a communication device provided in an embodiment of the present application;

[0071] Figure 6 This illustration shows a schematic diagram of acquiring industrial signal data in data processing according to an embodiment of this application;

[0072] Figure 7 This illustration shows a schematic diagram of filtering industrial signal data in data processing according to an embodiment of this application;

[0073] Figure 8 It shows Figure 2 The data processing method provided in the embodiments of this application has the following steps. Figure 2 The flowchart for step 203. Detailed Implementation

[0074] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0075] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of this application to enable readers to better understand this application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in this application can be implemented. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0076] It should be noted that the embodiments of this application can be applied to big data processing centers or big data cluster centers. Big data cluster centers can use the data processing methods in the embodiments of this application for subsequent development. For example, the processed aggregated industrial signal data can be applied to data analysis, artificial intelligence research, and initial model optimization.

[0077] To facilitate understanding of the technical solutions of this application by those skilled in the art, the embodiments of this application will subsequently use the CAN signal, a commonly used signal in vehicle transmission in industrial signals, as an example for illustration. Through the data processing of existing vehicle-transmitted CAN signals, it is found that the same functional information may be uploaded by different CAN signals. The frequency of the data uploaded by different CAN signals is different, but most of them are in milliseconds. The CAN signal data upload contains a large amount of invalid information such as null values. In daily data analysis needs, there are rarely millisecond-level precision requirements for information such as speed, mileage, energy consumption, and energy reserves. Furthermore, the accumulation volume of CAN signal data is indeed too large, putting significant pressure on the performance of traditional big data clusters: large storage space, slow data retrieval, and a high proportion of computing program resources. In the industrial intelligent big data scenario, the functional scope of the data returned by the CAN signal is limited for each data analysis, and the coverage of the business content of interest in a single analysis is narrow. If the data returned by the CAN signal is retrieved from the entire CAN signal data table for each retrieval, it will cause even more pressure on the big data cluster.

[0078] Therefore, the embodiments of this application can be based on big data warehouse technology, and through the analysis of data application scenarios, the exploration of data quality, and the summarization of data presentation forms, and through the aggregation, improvement, and alignment of data granularity, the data processing language can be used to perform operations such as splitting, cleaning, storing, and reusing the massive industrial big data signals that are frequently uploaded.

[0079] Reference Figure 1 The flowchart illustrates the steps of the data processing method provided in the embodiments of this application. Figure 1 The method may include:

[0080] Step 101: Obtain the industrial signal data corresponding to the table fields in the preset business details table. The industrial signal data is obtained from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud.

[0081] Further, step 101 may include: receiving a service function instruction sent by a user; generating a preset service detail table according to the service function instruction, wherein the preset service detail table includes table fields, the table fields include a second-level time field and a data field, one second-level time field corresponds to multiple data fields; obtaining industrial signal data corresponding to the data field from a preset acquisition signal matrix according to the data field, wherein one data field corresponds to at least one industrial signal data.

[0082] It should be noted that in this embodiment of the application, the device transmits various full-volume signal data to a preset cloud (cloud server). The big data cluster center can pull these full-volume industrial signal data from the preset cloud. Therefore, the industrial signal data is obtained from the full-volume industrial signal data uploaded by the preset industrial equipment to the preset cloud.

[0083] After acquiring the initial industrial signal data, the actual business needs can be met. For example, the CAN signal data in vehicles can be processed to facilitate subsequent intelligent research. Therefore, the corresponding business domains should be defined, and a business detail table should be pre-determined. This business detail table includes the table fields required by the actual needs. In other words, the business detail table should be pre-modeled based on the actual needs or business scenarios.

[0084] Specifically, as shown in Table 1, which is an example of a preset business details table, it can be clearly seen that the table fields include a unique vehicle identifier, a time field, and a data field. The data field can be a type of vehicle-related data, such as speed, real-time fuel consumption, total mileage, vehicle operating mode, etc. In addition, different data fields can be determined according to different business domains. The example in the table is not unique.

[0085] Table 1 - Exemplary Preset Service Details Table

[0086]

[0087] Step 102: Filter the industrial signal data to obtain the target industrial signal data.

[0088] Further, step 102 may include: filtering the industrial signal data according to a preset filtering rule to obtain target industrial signal data.

[0089] The preset filtering rules include that the value corresponding to the industrial signal data is non-empty, and that the frequency corresponding to the industrial signal data is greater than the target threshold.

[0090] It should be noted that, as Figure 7 As shown, after determining and acquiring the required industrial signal data, since the number of industrial signals uploaded per second is extremely large and the uploaded industrial signal data contains a large amount of invalid information such as null values, the original industrial signal data is first filtered to obtain the filtered target industrial signal data.

[0091] Specifically, filtering can be achieved by setting filtering rules in advance. For example, industrial signal data must not be null, and the values ​​corresponding to different industrial signal data must meet the preset range corresponding to different industrial signal data. Furthermore, the frequency of the current industrial signal data uploaded to the big data center or vehicle enterprise center must be greater than the target threshold. By setting filtering rules to filter the original industrial signal data, it can be ensured that some redundant information is removed.

[0092] Step 103: Aggregate the target industrial signal data according to the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data.

[0093] Furthermore, the second-level time is the time extracted in second-level units from the millisecond-level time string corresponding to the industrial signal data.

[0094] It should be noted that, in the embodiments of this application, after filtering the original industrial signal data, some invalid information containing null values ​​can be excluded. Furthermore, since the same functional information may be uploaded by different industrial signals, and each time the industrial signal data is retrieved, it will cause pressure on the large data clusters.

[0095] Therefore, in this application, multiple industrial signal data per second are aggregated into only one data point per second through second-level aggregation. For details, please refer to the following discussion.

[0096] Step 104: Determine the aggregated industrial signal data corresponding to the table fields in the preset business details table based on the aggregated industrial signal data.

[0097] It should be noted that, in this embodiment of the application, after the target industrial signal data is aggregated in step 103, aggregated industrial signal data can be obtained. Therefore, the aggregated industrial signal data can be filled into a preset business detail table to obtain the final second-level aggregated detail table that meets the needs of the business scenario. The aggregated vehicle-related data in second-level units can be clearly and intuitively displayed in the table.

[0098] Specifically, as shown in Table 2, which is an example of a pre-defined service detail table generated by aggregation, it can be clearly seen that 45 seconds and 46 seconds correspond to multiple CAN data.

[0099] Table 2 - Example of Preset Business Details Generated by Aggregation

[0100]

[0101] By aggregating and transforming full data storage into detailed business table storage that meets the needs of business scenarios, the impact of redundant data on data processing is reduced. This is suitable for reducing the pressure of data on hardware, improving data application efficiency, and enabling industrial data to generate higher value.

[0102] The data processing method provided in this application involves obtaining industrial signal data corresponding to table fields in a preset business detail table. The industrial signal data is obtained from the full volume of industrial signal data uploaded from preset industrial equipment to a preset cloud. The industrial signal data is then filtered to obtain target industrial signal data. This target industrial signal data is then aggregated based on its corresponding second-level time interval to obtain aggregated industrial signal data. Finally, the aggregated industrial signal data corresponding to the table fields in the preset business detail table is determined based on the aggregated industrial signal data. In other words, by filtering the obtained industrial signal data, this application can optimize the original full volume of data in the first step, retaining target industrial signal data that meets the requirements and quality. Further aggregation based on second-level time intervals on the target industrial signal data yields aggregated industrial signal data. This process can remove redundant data from multiple data entries per second, achieving the retention of valid data while eliminating redundant and irrelevant invalid data. This improves the overall data granularity and retention, meeting data business needs with less data and more data.

[0103] In addition, the signal data processed by the data processing method in this application embodiment can be applied to subsequent business scenarios. For example, for data analysis, low-quality data can be excluded in advance, and only one data point can be generated per second on a second-by-second basis, ensuring that the corresponding data point per second contains all the data required for the business scenario.

[0104] Therefore, the data processing method in this application embodiment can generate only one data per second on a second-level basis, and ensure that the corresponding data per second contains all the data required for the business scenario, thereby significantly reducing storage costs and saving computing resources. By cleaning and filtering the signal data, the data quality can be improved, and a large amount of transmission costs can be saved when calling the signal data in the future.

[0105] Reference Figure 2 The flowchart illustrates the steps of the data processing method provided in the embodiments of this application. Figure 2 The method may include:

[0106] Step 201: Obtain the industrial signal data corresponding to the table fields in the preset business details table. The industrial signal data is obtained from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud.

[0107] Step 202: The industrial signal data is filtered to obtain the target industrial signal data.

[0108] It should be noted that steps 201-202 above are based on the previous discussion and will not be repeated here.

[0109] Step 203: Aggregate all the industrial signal data according to the second-level time corresponding to the target industrial signal data and the different signal codes corresponding to the target industrial signal data to obtain aggregated industrial signal data.

[0110] The signal encoding is based on the functional category corresponding to the target industrial signal data.

[0111] It should be noted that, as Figure 6 As shown, in this embodiment of the application, the corresponding target industrial signal data can be determined by CAN-ID (the signal code corresponding to the CAN signal).

[0112] Furthermore, such as Figure 8 As shown, step 203 may include:

[0113] Step 2031: Aggregate all the target industrial signal data according to the second-level time corresponding to the target industrial signal data to obtain a main table. The main table includes a unique device identifier and a second-level time field.

[0114] It should be noted that, in the data aggregation part of this application embodiment, after determining which functional data corresponding to the acquisition signals to be extracted, i.e. the target industrial signal data, data aggregation can be achieved by writing data warehouse SQL code. Among them, the device unique identifier is the unique identifier information corresponding to the industrial equipment transmitting industrial signals. For example, taking vehicle CAN signal transmission as an example, the second dimension of the relevant vehicle with signal upload can be extracted and aggregated by taking the unique identifier of a single vehicle as a unit to obtain the main table.

[0115] It should be noted that for second-level time aggregation, industrial signals are transmitted in milliseconds as the smallest unit, transmitting a large amount of signal data. Therefore, second-level time aggregation can remove a large amount of null information within milliseconds.

[0116] Specifically, Table 3 is an example main table. As shown in Table 3 below, it can be clearly seen that the table fields in the main table include a vehicle as a unique identifier and a second-level time field.

[0117] Table 3 - Example Main Table

[0118]

[0119]

[0120] Step 2032: Aggregate all the target industrial signal data according to the different signal codes corresponding to the target industrial signal data and the second-level time to obtain a sub-table corresponding to the target industrial signal data with different signal codes. The sub-table includes the unique identifier of the equipment, the second-level time field, and the aggregated industrial signal data corresponding to the table field.

[0121] It should be noted that the signal code is a different ID of the industrial signal setting number, or the corresponding signal channel, used to distinguish the signal. In the embodiments of this application, taking the CAN signal as an example, the signal code corresponding to the CAN signal is the CAN-ID. Therefore, after determining the main table, all target CAN data are aggregated according to the signal code (i.e. the ID of the CAN signal) and the second-level time to obtain the non-null value data corresponding to all second-level times for each signal code.

[0122] Further, step 2032 includes the following steps: splitting all the target industrial signal data according to the signal code to obtain multiple target industrial signal data corresponding to different signal codes; when it is detected that the second target industrial signal data corresponding to any signal code in the same second-level time field is a non-null value and the first target industrial signal data is a null value, or when the second target industrial signal data corresponding to the same signal code in the same second-level time field is a null value and the first target industrial signal data is a non-null value, the non-null values ​​corresponding to the first target industrial signal data and the non-null values ​​corresponding to the second target industrial signal data are aggregated according to the second-level time to obtain a sub-table corresponding to the target industrial signal data corresponding to the signal code, wherein the sub-table includes the signal code, the unique device identifier, the second-level time field, and the data field.

[0123] It should be noted that all target industrial signal data can be divided according to the signal code. For example, for the signal code CAN-ID, there are 381, 287 and 315, etc. Then, according to the different signal codes, it can be divided into 381, as shown in Table 4. Table 4 is an exemplary sub-table before aggregation processing. Taking the signal code 381 as an example, it can be clearly seen that in step 204, the target industrial signal data can first be divided into different groups of data according to different signal codes. The signal codes corresponding to each group of data are the same. Then, under the premise that the signal codes are the same, time aggregation is performed in units of seconds.

[0124] Table 4 - Example Sub-table Before Aggregation Processing

[0125]

[0126]

[0127] As shown in Table 5, which is an exemplary sub-table after aggregation processing, it can be clearly seen that, under the premise of the same signal encoding, after time aggregation in seconds, the result is CAN data of the vehicle without null values ​​within a second.

[0128] Table 5 - Example Sub-table after Aggregation Processing

[0129]

[0130] Similarly, based on different signal codes, the target industrial signal data can be divided into different groups, and each group of data is processed using the same method described above.

[0131] Further, the step of splitting all the target industrial signal data according to the signal code to obtain multiple target industrial signal data corresponding to different signal codes includes: splitting all the target industrial signal data according to the signal code to obtain multiple sets of target industrial signal data, wherein each set of target industrial signal data corresponds to one signal code; selecting the first or last target industrial signal data in each set of target industrial signal data based on millisecond-level time sorting as the target industrial signal data corresponding to different signal codes.

[0132] It should be noted that when extracting second-level data, each function signal selects the data that the user is concerned about based on the actual business or scenario requirements, such as the first or last data within a second.

[0133] Therefore, in this embodiment of the application, for the secondary table, all target industrial signal data are first split according to the signal encoding and then aggregated. The advantage of splitting all signal data is that it can increase data processing efficiency, that is, multi-threaded synchronous operation, and at the same time, in big data scenarios, it can ensure the normal operation of the data processing program under limited resources, and reduce the impact of a large amount of data processing on the system.

[0134] Step 2033: Align the main table and at least one of the sub-tables to obtain an event detail table, wherein the event detail table includes aggregated industrial signal data.

[0135] It should be noted that, in the embodiments of this application, after determining the main table and at least one sub-table, each sub-table corresponds to a unique signal code and a unique device identifier.

[0136] Therefore, a secondary table can be created by using the unique identifier of the device (e.g., the unique identifier of the vehicle) and the time (seconds) as the granularity. This table can then be linked with the aggregated results of various signal codes (e.g., CAN signals) corresponding to different signal codes. The target industrial signal data of the secondary table corresponding to each industrial signal will coexist in parallel within the same industrial device (e.g., vehicle) and the second-level time. Specifically, if the device uploads signal data within a given second, it will appear in that result data. This results in a result table with various acquired signal data attached, using industrial devices and second-level time units as the granularity. After this result table is generated, it does not need to exclude null values ​​and can be directly used as a transaction detail table for the relevant subject area. The relevant subject area refers to different business domains, such as intelligent driving and human-vehicle interaction for vehicles.

[0137] Furthermore, since only devices that upload data and the time in seconds will appear in the event details table, meaning that every data row has at least one data information uploaded by a collection signal, a significant amount of computational resources used to exclude null values ​​can be saved.

[0138] Step 204: Determine the aggregated industrial signal data corresponding to the table fields in the preset business details table based on the aggregated industrial signal data.

[0139] It should be noted that step 204 above is based on the previous discussion and will not be repeated here.

[0140] The data processing method provided in this application involves obtaining industrial signal data corresponding to table fields in a preset business detail table. The industrial signal data is obtained from the full volume of industrial signal data uploaded from preset industrial equipment to a preset cloud. The industrial signal data is then filtered to obtain target industrial signal data. This target industrial signal data is then aggregated based on its corresponding second-level time interval to obtain aggregated industrial signal data. Finally, the aggregated industrial signal data corresponding to the table fields in the preset business detail table is determined based on the aggregated industrial signal data. In other words, by filtering the obtained industrial signal data, this application can optimize the original full volume of data in the first step, retaining target industrial signal data that meets the requirements and quality. Further aggregation based on second-level time intervals on the target industrial signal data yields aggregated industrial signal data. This process can remove redundant data from multiple data entries per second, achieving the retention of valid data while eliminating redundant and irrelevant invalid data. This improves the overall data granularity and retention, meeting data business needs with less data and more data.

[0141] In addition, the signal data processed by the data processing method in this application embodiment can be applied to subsequent business scenarios. For example, for data analysis, low-quality data can be excluded in advance, and only one data point can be generated per second on a second-by-second basis, ensuring that the corresponding data point per second contains all the data required for the business scenario.

[0142] Therefore, the data processing method in this application embodiment can generate only one data per second on a second-level basis, and ensure that the corresponding data per second contains all the data required for the business scenario, thereby significantly reducing storage costs and saving computing resources. By cleaning and filtering the signal data, the data quality can be improved, and a large amount of transmission costs can be saved when calling the signal data in the future.

[0143] Reference Figure 3 The flowchart illustrates the steps of the data processing method provided in the embodiments of this application. Figure 3 The method may include:

[0144] Step 301: Obtain the industrial signal data corresponding to the table fields in the preset business details table, wherein the industrial signal data is obtained from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud.

[0145] Step 302: The industrial signal data is filtered to obtain the target industrial signal data.

[0146] Step 303: Aggregate all the target industrial signal data according to the second-level time corresponding to the target industrial signal data to obtain a main table. The main table includes a unique device identifier and a second-level time field.

[0147] Step 304: Aggregate all the target industrial signal data according to the different signal codes corresponding to the target industrial signal data and the second-level time to obtain a sub-table corresponding to the target industrial signal data with different signal codes. The sub-table includes the unique identifier of the equipment, the second-level time field, and the aggregated industrial signal data corresponding to the table field.

[0148] It should be noted that steps 301-304 above are based on the previous discussion and will not be repeated here.

[0149] Step 305: Align the device unique identifier and second-level time field in the main table and the sub-table to obtain the event detail table, wherein the event detail table includes aggregated industrial signal data.

[0150] It should be noted that, in this embodiment of the application, after obtaining the main table with the device unique identifier and time (second) as the granularity in step 303, the secondary table of the aggregation results of each acquired signal (CAN data) is associated with the device unique identifier and the second-level time field as the association condition in the main table. The data of each signal secondary table will coexist in the same device and second-level time. If the device has signal data uploaded within that second, it will appear in the result data, thereby obtaining an event detail table with the device and second-level time unit as the granularity and various acquired signal data.

[0151] Once the event details table is generated, it can be directly used as a details table for the relevant subject area without needing to exclude null values. This is because only vehicles that have uploaded data and their corresponding times in seconds will appear in the data table. In other words, every data row has at least one data entry for the acquired signal, thus saving a significant amount of computational resources used for excluding null values.

[0152] Step 306: Determine the aggregated industrial signal data corresponding to the table fields in the preset business details table based on the aggregated industrial signal data.

[0153] It should be noted that step 306 above is based on the previous discussion and will not be repeated here.

[0154] The data processing method provided in this application involves obtaining industrial signal data corresponding to table fields in a preset business detail table. The industrial signal data is obtained from the full volume of industrial signal data uploaded from preset industrial equipment to a preset cloud. The industrial signal data is then filtered to obtain target industrial signal data. This target industrial signal data is then aggregated based on its corresponding second-level time interval to obtain aggregated industrial signal data. Finally, the aggregated industrial signal data corresponding to the table fields in the preset business detail table is determined based on the aggregated industrial signal data. In other words, by filtering the obtained industrial signal data, this application can optimize the original full volume of data in the first step, retaining target industrial signal data that meets the requirements and quality. Further aggregation based on second-level time intervals on the target industrial signal data yields aggregated industrial signal data. This process can remove redundant data from multiple data entries per second, achieving the retention of valid data while eliminating redundant and irrelevant invalid data. This improves the overall data granularity and retention, meeting data business needs with less data and more data.

[0155] Reference Figure 4 , Figure 4 An embodiment of this application provides a data processing method apparatus, the apparatus comprising:

[0156] The acquisition module 401 is used to acquire industrial signal data corresponding to the table fields in the preset business details table, wherein the industrial signal data is acquired from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud.

[0157] The filtering module 402 is used to filter the industrial signal data to obtain target industrial signal data;

[0158] Aggregation module 403 is used to aggregate the target industrial signal data according to the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data;

[0159] The determination module 404 is used to determine the aggregated industrial signal data corresponding to the table fields in the preset business details table based on the aggregated industrial signal data.

[0160] The data processing method provided in this application involves obtaining industrial signal data corresponding to table fields in a preset business detail table. The industrial signal data is obtained from the full volume of industrial signal data uploaded from preset industrial equipment to a preset cloud. The industrial signal data is then filtered to obtain target industrial signal data. This target industrial signal data is then aggregated based on its corresponding second-level time interval to obtain aggregated industrial signal data. Finally, the aggregated industrial signal data corresponding to the table fields in the preset business detail table is determined based on the aggregated industrial signal data. In other words, by filtering the obtained industrial signal data, this application can optimize the original full volume of data in the first step, retaining target industrial signal data that meets the requirements and quality. Further aggregation based on second-level time intervals on the target industrial signal data yields aggregated industrial signal data. This process can remove redundant data from multiple data entries per second, achieving the retention of valid data while eliminating redundant and irrelevant invalid data. This improves the overall data granularity and retention, meeting data business needs with less data and more data.

[0161] This application also provides a communication device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0162] Memory 503 is used to store computer programs;

[0163] When processor 501 executes the program stored in memory 503, it can perform the following steps:

[0164] The industrial signal data corresponding to the table fields in the preset business details table is obtained, wherein the industrial signal data is obtained from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud.

[0165] The industrial signal data is filtered to obtain the target industrial signal data;

[0166] The target industrial signal data is aggregated based on the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data.

[0167] The aggregated industrial signal data corresponding to the table fields in the preset business details table is determined based on the aggregated industrial signal data.

[0168] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0169] The communication interface is used for communication between the aforementioned terminal and other devices.

[0170] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0171] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0172] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the data processing methods described in the above embodiments.

[0173] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data processing methods described in the above embodiments.

[0174] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or third-party database to another website, computer, server, or third-party database via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or third-party database that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0175] It should be noted that, in this document, relational terms such as "first" and "first" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0176] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0177] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A data processing method, characterized by, The data processing method includes: The industrial signal data corresponding to the table fields in the preset business details table is obtained, wherein the industrial signal data is obtained from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud. The industrial signal data is filtered to obtain the target industrial signal data; The target industrial signal data is aggregated based on the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data. Based on the aggregated industrial signal data, determine the aggregated industrial signal data corresponding to the table fields in the preset business details table; The process of aggregating the target industrial signal data based on the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data includes: Based on the second-level time corresponding to the target industrial signal data and the different signal codes corresponding to the target industrial signal data, all the industrial signal data are aggregated to obtain aggregated industrial signal data, including: Based on the second-level time corresponding to the target industrial signal data, all the target industrial signal data are aggregated to obtain a main table, which includes a unique device identifier and a second-level time field. Based on the different signal codes corresponding to the target industrial signal data and the second-level time, all the target industrial signal data are aggregated to obtain a sub-table corresponding to the target industrial signal data with different signal codes. Specifically, the target industrial signal data is divided into different groups based on the different signal codes, with each group corresponding to the same signal code. Under the premise that the signal codes are the same, time aggregation is performed in second-level units. The sub-table includes the unique device identifier, the second-level time field, and the aggregated industrial signal data corresponding to the table fields. The main table and at least one of the sub-tables are aligned to obtain an event detail table, wherein the event detail table includes aggregated industrial signal data.

2. The data processing method according to claim 1, characterized in that, The signal encoding is based on the functional category corresponding to the target industrial signal data.

3. The data processing method of claim 1, wherein, The step of aggregating all the target industrial signal data according to the different signal codes corresponding to the target industrial signal data and the second-level time to obtain a sub-table corresponding to the target industrial signal data with different signal codes includes: All the target industrial signal data are split according to the signal encoding to obtain multiple target industrial signal data corresponding to different signal codes; If it is detected that the second target industrial signal data corresponding to any signal code within the same second-level time field is a non-null value and the first target industrial signal data is a null value, or if the second target industrial signal data corresponding to the same signal code within the same second-level time field is a null value and the first target industrial signal data is a non-null value, the non-null values ​​corresponding to the first target industrial signal data and the non-null values ​​corresponding to the second target industrial signal data are aggregated according to the second-level time to obtain a sub-table corresponding to the target industrial signal data corresponding to the signal code. The sub-table includes the signal code, the unique device identifier, the second-level time field, and the data field.

4. The data processing method according to claim 3, characterized in that, The step of splitting all the target industrial signal data according to the signal encoding to obtain multiple target industrial signal data corresponding to different signal codes includes: The target industrial signal data is split according to the signal code to obtain multiple sets of target industrial signal data, wherein each set of target industrial signal data corresponds to one of the signal codes; The first or last target industrial signal data in each group of target industrial signal data, sorted by millisecond time, is selected as the target industrial signal data corresponding to different signal codes.

5. The data processing method according to claim 3, characterized in that, The step of aligning the main table and at least one of the sub-tables to obtain the event details table includes: The event details table is obtained by aligning the device unique identifier and the second-level time field in the main table and the sub-table.

6. The data processing method of claim 1, wherein, The step of obtaining the industrial signal data corresponding to the table field based on the table field in the preset business details table includes: Receive business function instructions sent by users; A preset business detail table is generated according to the business function instruction, wherein the preset business detail table includes table fields, the table fields include a second-level time field and a data field, and one second-level time field corresponds to multiple data fields; The industrial signal data corresponding to the data field is obtained from the preset acquisition signal matrix according to the data field, wherein one data field corresponds to at least one industrial signal data.

7. The data processing method according to claim 1, characterized in that, The process of filtering the industrial signal data to obtain the target industrial signal data includes: The industrial signal data is filtered according to preset filtering rules to obtain target industrial signal data.

8. The data processing method according to claim 7, characterized in that, The preset filtering rules include that the value corresponding to the industrial signal data is non-empty, and that the frequency corresponding to the industrial signal data is greater than the target threshold.

9. The data processing method according to claim 1, characterized in that, The second-level time is the time extracted in seconds from the millisecond-level time string corresponding to the industrial signal data.

10. A data processing apparatus, characterized in that, The data processing device includes: The acquisition module is used to acquire industrial signal data corresponding to the table fields in the preset business details table, wherein the industrial signal data is acquired from the full amount of industrial signal data uploaded from the preset industrial equipment to the preset cloud. The filtering module is used to filter the industrial signal data to obtain target industrial signal data; The aggregation module is used to aggregate the target industrial signal data according to the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data. The determination module is used to determine the aggregated industrial signal data corresponding to the table fields in the preset business details table based on the aggregated industrial signal data. The device further includes: The process of aggregating the target industrial signal data based on the second-level time corresponding to the target industrial signal data to obtain aggregated industrial signal data includes: Based on the second-level time corresponding to the target industrial signal data and the different signal codes corresponding to the target industrial signal data, all the industrial signal data are aggregated to obtain aggregated industrial signal data, including: Based on the second-level time corresponding to the target industrial signal data, all the target industrial signal data are aggregated to obtain a main table, which includes a unique device identifier and a second-level time field. Based on the different signal codes corresponding to the target industrial signal data and the second-level time, all the target industrial signal data are aggregated to obtain a sub-table corresponding to the target industrial signal data with different signal codes. Specifically, the target industrial signal data is divided into different groups based on the different signal codes, with each group corresponding to the same signal code. Under the premise that the signal codes are the same, time aggregation is performed in second-level units. The sub-table includes the unique device identifier, the second-level time field, and the aggregated industrial signal data corresponding to the table fields. The main table and at least one of the sub-tables are aligned to obtain an event detail table, wherein the event detail table includes aggregated industrial signal data.

11. A communication device, characterized in that, include: A transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; The processor is configured to read a program from the memory to implement the data processing method as described in any one of claims 1-9.

12. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the data processing method as described in any one of claims 1-9.

13. A vehicle, characterized in that, The vehicle is equipped with the data processing device as described in claim 10.

Images