Data processing method, device and equipment

Abstract

The invention discloses a data processing method and equipment, and the method comprises the steps: determining a target field used for describing the data attribute of a vehicle terminal, and determining a field identifier of the target field; wherein the target field can comprise a plurality of fields, and each field uniquely corresponds to one field identifier. And splicing the field identifier of the target field, and determining a storage field identifier. Based on the hierarchical relationship of the vehicle terminal data, an information bit identifier of the vehicle terminal data is determined. That is, the information bit identification may be used to represent the data hierarchy at which the vehicle terminal data is at. And determining metadata corresponding to the vehicle terminal data based on the storage field identifier and the information bit identifier. Through the method provided by the invention, the metadata describing the vehicle terminal data can be determined according to a unified processing method, and due to the same processing method, the data processing efficiency can be improved, and the data can be conveniently traced.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a method, apparatus, and device for processing data. Background Art

[0002] During the production and application process of vehicles, a large amount of data is generated due to business requirements, scenario testing, etc., and the data will flow through multiple processing steps to various business departments. In order to better analyze the performance and function usage of vehicles, etc., it is necessary to analyze the backhaul data.

[0003] However, currently, there is a lack of unified data processing rules and storage methods. After the data goes through various levels of processing and transfer, the following problems are likely to occur: If information is missing during the data transfer process, it may be difficult to trace back to the root node or intermediate nodes of the data due to different data processing methods at each level. If the data is defined differently, it may cause the same data to be split multiple times, or the same data to be stored repeatedly, resulting in duplicate data processing and low data processing efficiency. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, and device for processing data, so as to improve the data processing efficiency and facilitate data traceability.

[0005] In a first aspect, this application provides a method for processing data, the method comprising:

[0006] Determine a target field for describing the data attributes of a vehicle terminal and a field identifier of the target field based on a field extraction rule;

[0007] Based on a splicing rule, splice the field identifiers to determine a storage field identifier;

[0008] Determine an information bit identifier of the vehicle terminal data based on an information bit generation rule and the hierarchical relationship of the vehicle terminal data;

[0009] Based on the storage field identifier and the information bit identifier, determine the metadata corresponding to the vehicle terminal data.

[0010] In a possible implementation manner, before determining the metadata of the vehicle terminal data based on the storage field identifier and the information bit identifier, the method further comprises:

[0011] Perform encryption processing on the vehicle terminal data based on an information digest generation rule to determine an encryption identifier of the vehicle terminal data;

[0012] The determining the metadata corresponding to the vehicle terminal data based on the storage field identifier and the information bit identifier comprises:

[0013] Determine the metadata based on the storage field identifier, the information bit identifier, and the encryption identifier.

[0014] In a possible implementation, the method further includes:

[0015] In response to an operation to update the vehicle terminal data, obtain the storage field identifier and the information bit identifier in the metadata;

[0016] Based on the update data in the vehicle terminal data, determine the field to be updated in the target field corresponding to the storage field identifier, and update the field to be updated based on the update data;

[0017] Update the information bit identifier based on the update data.

[0018] In a possible implementation, the determining the storage field identifier by splicing the field identifiers based on the splicing rule includes:

[0019] Splice the field identifiers of the target field based on the splicing rule to determine a spliced field identifier;

[0020] Perform encoding processing on the spliced field identifier based on the encoding rule to determine the storage field identifier.

[0021] In a possible implementation, the determining the information bit identifier of the vehicle terminal data based on the hierarchical relationship of the vehicle terminal data includes:

[0022] Determine the hierarchical level to which the vehicle terminal data belongs and the splitting order in the hierarchical level based on the hierarchical relationship;

[0023] Determine a hierarchical identifier based on the hierarchical level, and determine an order identifier based on the splitting order;

[0024] Determine the information bit identifier based on the hierarchical identifier and the order identifier.

[0025] In a possible implementation, the method further includes:

[0026] Map the target field and the field identifier to a dictionary based on the data dictionary specification, where the field identifier represents the keyword of the dictionary and the target field represents the value of the dictionary.

[0027] In a second aspect, the present application provides a method for processing data, the method including:

[0028] Obtain the field identifier of the data to be processed;

[0029] Among the storage field identifiers included in the metadata database, determine a target field identifier that matches the field identifier. The metadata database includes multiple metadata corresponding to multiple vehicle terminal data. The metadata includes the storage field identifier and the information bit identifier of the vehicle terminal data, and the metadata is determined based on any implementation manner of the above first aspect;

[0030] Obtain the target information bit identifier corresponding to the target field identifier;

[0031] Determine the data lineage based on the target information bit identifier.

[0032] In a possible implementation manner, the determining, among the storage field identifiers included in the metadata database, a target field identifier that matches the field identifier includes:

[0033] Determine the first identifier corresponding to the first field based on the field identifier;

[0034] Determine the second identifier corresponding to the second field based on the storage field identifier;

[0035] Among the multiple second identifiers included in the metadata database, determine the target field identifier that matches the first identifier.

[0036] In a possible implementation manner, when the field identifier and the storage field identifier are encoded identifiers, the determining, among the storage field identifiers included in the metadata database, a target field identifier that matches the field identifier includes: performing a decoding process on the field identifier to obtain the decoded field identifier; performing a decoding process on the storage field identifier to obtain the decoded storage field identifier; among the decoded storage field identifiers, determine the target field identifier that matches the decoded field identifier.

[0037] In a possible implementation manner, the method further includes: obtaining the encrypted identifier corresponding to the target field identifier, where the encrypted identifier is obtained by encrypting the vehicle terminal data based on the information digest generation rule; generating a summary table based on the target field identifier, the target information bit identifier, and the encrypted identifier.

[0038] In a possible implementation manner, the method further includes: when there are identical encrypted identifiers in the encrypted identifier, deleting the duplicate encrypted identifiers and deleting the target field identifier and the target information bit identifier corresponding to the duplicate encrypted identifiers to update the summary table.

[0039] In a possible implementation, determining the data lineage based on the target information bit identifier includes: generating a data lineage map based on the updated summary table for data traceability based on the data lineage map.

[0040] In a third aspect, the present application provides a data processing device, the device includes:

[0041] A first determination unit, configured to determine a target field for describing the data attributes of the vehicle terminal and a field identifier of the target field based on a field extraction rule;

[0042] A second determination unit, configured to splice the field identifiers based on a splicing rule to determine a storage field identifier;

[0043] A third determination unit, configured to determine an information bit identifier of the vehicle terminal data based on an information bit generation rule and a hierarchical relationship of the vehicle terminal data;

[0044] A fourth determination unit, configured to determine metadata corresponding to the vehicle terminal data based on the storage field identifier and the information bit identifier.

[0045] In a possible implementation, the device further includes: an encryption unit;

[0046] Before determining the metadata of the vehicle terminal data based on the storage field identifier and the information bit identifier, the encryption unit is configured to perform encryption processing on the vehicle terminal data based on an information digest generation rule to determine an encryption identifier of the vehicle terminal data;

[0047] The fourth determination unit is specifically configured to determine the metadata based on the storage field identifier, the information bit identifier, and the encryption identifier.

[0048] In a possible implementation, the device further includes: an update unit, configured to, in response to an operation of updating the vehicle terminal data, obtain the storage field identifier and the information bit identifier in the metadata; determine a field to be updated in the target field corresponding to the storage field identifier based on the update data in the vehicle terminal data, and update the field to be updated based on the update data; update the information bit identifier based on the update data.

[0049] In a possible implementation, the second determination unit is specifically configured to splice the field identifiers of the target field based on the splicing rule to determine a spliced field identifier; perform encoding processing on the spliced field identifier based on an encoding rule to determine the storage field identifier.

[0050] In a possible implementation manner, the third determination unit is specifically configured to determine the belonging level of the vehicle terminal data and the splitting order in the belonging level based on the hierarchical relationship; determine a level identifier based on the belonging level, and determine an order identifier based on the splitting order; and determine the information bit identifier based on the level identifier and the order identifier.

[0051] In a possible implementation manner, the apparatus further includes: a mapping unit, configured to map the target field and the field identifier into a dictionary based on a data dictionary specification, where the field identifier represents a keyword of the dictionary, and the target field represents a value of the dictionary.

[0052] Fourthly, the present application provides a data processing apparatus, where the apparatus includes:

[0053] A first acquisition unit, configured to acquire a field identifier of data to be processed;

[0054] A fifth determination unit, configured to determine a target field identifier that matches the field identifier in storage field identifiers included in a metadata database, where the metadata database includes multiple metadata corresponding to multiple vehicle terminal data, the metadata includes a storage field identifier and an information bit identifier of the vehicle terminal data, and the metadata is determined based on any one of the implementation manners of the first aspect above;

[0055] A second acquisition unit, configured to acquire a target information bit identifier corresponding to the target field identifier;

[0056] A sixth determination unit, configured to determine a data lineage relationship based on the target information bit identifier.

[0057] In a possible implementation manner, the fifth determination unit is specifically configured to determine a first identifier corresponding to a first field based on the field identifier; determine a second identifier corresponding to a second field based on the storage field identifier; and determine a target field identifier that matches the first identifier among multiple second identifiers included in the metadata database.

[0058] In a possible implementation manner, when the field identifier and the storage field identifier are encoded identifiers, the fifth determination unit is specifically configured to perform decoding processing on the field identifier to obtain a decoded field identifier; perform decoding processing on the storage field identifier to obtain a decoded storage field identifier; and determine the target field identifier that matches the decoded field identifier among the decoded storage field identifiers.

[0059] In a possible implementation, the device further includes: a third acquisition unit, configured to acquire an encryption identifier corresponding to the target field identifier, where the encryption identifier is obtained by encrypting vehicle terminal data based on an information digest generation rule; a generation unit, configured to generate a summary table based on the target field identifier, the target information bit identifier, and the encryption identifier.

[0060] In a possible implementation, the device further includes: an update unit, configured to, when there are identical encryption identifiers in the encryption identifiers, delete the duplicate encryption identifiers and delete the target field identifier and the target information bit identifier corresponding to the duplicate encryption identifiers, so as to update the summary table.

[0061] In a possible implementation, the sixth determination unit is specifically configured to generate a data lineage map based on the updated summary table, so as to perform data traceability based on the data lineage map.

[0062] In a fifth aspect, the present application provides an electronic device, where the device includes: a memory and a processor;

[0063] The memory is used to store relevant program codes;

[0064] The processor is used to call the program codes to execute the data processing method according to any one of the implementations in the first aspect or the second aspect above.

[0065] In a sixth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the data processing method according to any one of the implementations in the first aspect or the second aspect above.

[0066] Thus, the present application has the following beneficial effects:

[0067] In the above implementation manners of the present application, the target fields for describing the attributes of vehicle terminal data can be determined based on the field extraction rules, and the field identifiers of the target fields can be determined. Among them, the target fields can include multiple fields, and each field uniquely corresponds to a field identifier. The field identifiers of the target fields are concatenated based on the concatenation rules to determine the stored field identifier. Based on the information bit generation rules and the hierarchical relationship of the vehicle terminal data, the information bit identifier of the vehicle terminal data is determined. That is, the information bit identifier can be used to represent the data hierarchy in which the vehicle terminal data is located. Based on the stored field identifier and the information bit identifier, the metadata corresponding to the vehicle terminal data is determined. Through the method provided by the present application, the metadata for describing the vehicle terminal data can be determined according to a unified processing method, solving problems such as unclear data sources and duplicate data storage during the data flow process. Since the processing methods are the same, the data processing efficiency can be improved, and it is also convenient to trace the data. Description of the Drawings

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments provided in the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings.

[0069] Figure 1 Schematic diagram of an application scenario provided for an embodiment of the present application;

[0070] Figure 2 Flowchart of a data processing method provided for an embodiment of the present application;

[0071] Figure 3 Schematic diagram of an information bit generation rule provided for an embodiment of the present application;

[0072] Figure 4 Flowchart of determining metadata provided for an embodiment of the present application;

[0073] Figure 5 Flowchart of updating metadata provided for an embodiment of the present application;

[0074] Figure 6 Flowchart of another data processing method provided for an embodiment of the present application;

[0075] Figure 7 Schematic diagram of a data lineage provided for an embodiment of the present application;

[0076] Figure 8 Schematic diagram of a processing device provided for an embodiment of the present application;

[0077] Figure 9Schematic diagram of a data processing device provided by an embodiment of the present application;

[0078] Figure 10 Schematic diagram of another data processing device provided by an embodiment of the present application;

[0079] Figure 11 Schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0080] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. The described embodiments are only exemplary embodiments of the present application, not all implementation manners. Those skilled in the art can obtain other embodiments without creative efforts in combination with the embodiments of the present application, and these embodiments are also within the protection scope of the present application.

[0081] In the production and application process of vehicles, a large amount of data will be generated due to business requirements, scenario testing, etc., and the data will flow to each business department through multiple processing steps. In order to better analyze the performance and function usage of vehicles, etc., it is necessary to analyze the backhaul data.

[0082] However, currently, the data lacks unified processing rules and storage methods. After the data goes through various levels of processing and circulation, the following problems are likely to occur: If information is missing during the data transmission process, due to different data processing methods at each level, it may be difficult to trace back to the root node or path nodes of the data. If the data definition methods are different, it may cause the same data to split multiple times, or the same data to be stored repeatedly, resulting in repeated data processing and low data processing efficiency.

[0083] Based on this, the embodiments of the present application provide a data processing method to improve the data processing efficiency and facilitate data tracing. Specifically, when implemented, the target fields for describing the data attributes of the vehicle terminal can be determined based on the field extraction rules, and the field identifiers of the target fields can be determined. Among them, the target fields can include multiple fields, and each field uniquely corresponds to a field identifier. The field identifiers of the target fields are concatenated based on the concatenation rules to determine the storage field identifier. Based on the information bit generation rules and the hierarchical relationship of the vehicle terminal data, the information bit identifier of the vehicle terminal data is determined. That is, the information bit identifier can be used to represent the data level at which the vehicle terminal data is located. Based on the storage field identifier and the information bit identifier, the metadata corresponding to the vehicle terminal data is determined. Through the method provided by the embodiments of the present application, the metadata describing the vehicle terminal data can be determined according to a unified processing method. Since the processing methods are the same, the data processing efficiency can be improved, and it is also convenient to trace the data.

[0084] SeeFigure 1 , Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application.

[0085] In this application scenario, vehicle terminal data can be collected according to business requirements. For example, the vehicle terminal data can be data collected by the vehicle for road information. Then, the vehicle terminal data is processed to obtain multiple data sets. For example, it can be divided according to business types, etc. The data sets are screened according to different functional scenarios to obtain a bid submission data set. The bid submission data set is audited and labeled based on the annotation specification to obtain an annotated data set. The annotated data set that passes the audit and annotation is sent to the business department for use.

[0086] To facilitate understanding of the method provided by the embodiments of the present application, the following will be specifically introduced in conjunction with the accompanying drawings in the embodiments.

[0087] See Figure 2 , Figure 2 It is a flowchart of a data processing method provided by an embodiment of the present application.

[0088] This method can be executed by a processing device, and this method may include the following steps:

[0089] S201: Determine the target field for describing the attributes of the vehicle terminal data and the field identifier of the target field based on the field extraction rule.

[0090] Among them, the attributes of the vehicle terminal data may include business type, data type, vehicle information, collection information, update time, etc. The business type may include road collection, vehicle testing, etc.; the data type represents the identification due to data format differences, such as video data, compressed package data, etc.; the vehicle information may include information describing the vehicle ID, such as vehicle coding, etc.; the collection information may include the spatio-temporal information where the data occurs, such as the occurrence time, occurrence location, etc.; the update time may represent the time when the content of the data is updated during the transfer process.

[0091] In a possible implementation manner, since the original fields recording the attributes of the vehicle terminal data may include redundant data, the required target fields can be extracted from the original fields. That is, the original fields describing the attributes of the vehicle terminal data can be obtained first, and then according to the unified field extraction rule, the required target fields are extracted from the original fields, and the field identifiers corresponding to the target fields are determined. The target fields may include multiple fields, for example, business type, data type, vehicle information, collection information, update time, etc. Each field can uniquely correspond to a field identifier. Among them, the embodiments of the present application do not limit the setting method of the field identifier. For example, it can be set in ways such as letters, numbers, etc., as long as each field uniquely corresponds to a field identifier.

[0092] Among them, the field extraction rule can be pre-determined, which stipulates the content, format, etc. of the fields to be extracted. In a possible implementation, before determining the target field, the integrity of the original fields can also be verified according to the field extraction rule, mainly including verifying whether the types of the original fields are complete and whether each type of original field is a non-null value, that is, whether the original fields lack specific content. For example, for the attribute description of vehicle terminal data, the types of the original fields can include service type, data type, vehicle information, etc. When the original fields are verified to be complete fields, then the target field is determined from the original fields using the field extraction rule.

[0093] In a possible implementation, the target field and the field identifier corresponding to the target field can be mapped to a dictionary based on the data dictionary specification, where the field identifier represents the keyword key of the dictionary, and the target field represents the value value of the dictionary. This facilitates subsequent retrieval of fields according to the keyword. Among them, the data dictionary specification can be a pre-determined data mapping rule.

[0094] When the target field includes multiple fields, each field and the field identifier corresponding to the field can be stored as a dictionary. For example, when the field content representing the service type is "data collection" and the field identifier is "00R", the field identifier "00R" can be used as the keyword of the dictionary and "data collection" can be used as the value of the dictionary for storage. When the field content representing the service type is "road test" and the field identifier is "00J", the field identifier "00J" can be used as the keyword of the dictionary and "road test" can be used as the value of the dictionary for storage.

[0095] S202: Concatenate the field identifiers based on the concatenation rule to determine the stored field identifier.

[0096] When the target field includes multiple fields and each field uniquely corresponds to a field identifier, the field identifiers of the multiple fields can be concatenated to determine the stored field identifier. Optionally, the field identifier concatenation rule can be pre-determined, and according to the field identifier concatenation rule, the multiple field identifiers are concatenated to determine the stored field identifier. For example, when the target field includes service type, data type, vehicle information, collection information, and update time, the corresponding concatenation rule can be service type - data type - vehicle information - collection information - update time.

[0097] In a possible implementation, the field identifiers of the target fields can be concatenated based on the concatenation rule to determine the concatenated field identifier. Then, the concatenated field identifier can be encoded based on the encoding rule to determine the stored field identifier. By encoding the concatenated field identifier, the storage space occupied by the field identifier can be reduced, and the data security can be improved. Moreover, encoding the concatenated field identifier according to the unified encoding rule can improve the consistency of data storage. For example, the concatenated field identifier can be encoded using the Base64 encoding rule to obtain the stored field identifier. Among them, Base64 is a common encoding method for transmitting 8-bit byte codes, and binary data can be represented based on 64 printable characters.

[0098] S203: Determine the information bit identifier of the vehicle terminal data based on the information bit generation rule and the hierarchical relationship of the vehicle terminal data.

[0099] In the application scenario, the vehicle terminal data may be generated by other parent data, or may generate different child data, etc. Therefore, the information bit identifier corresponding to the vehicle terminal data can be determined according to the hierarchical relationship of the vehicle terminal data. This hierarchical relationship can represent the "parent-child relationship" between data. For example, when the first data can split to generate the second data and the third data, then the first data is the parent data of the second data and the second data, the first data is at the upper level of the second data and the third data, the second data and the third data belong to the same level, and are at the lower level of the first data.

[0100] In specific implementation, the hierarchical level to which the vehicle terminal data belongs and the splitting order in the hierarchical level can be determined based on the hierarchical relationship of the vehicle terminal data. Among them, the hierarchical level can represent the number of layers through which the vehicle terminal data flows, and the splitting order represents the order in which different data are generated in the same level. Then, the hierarchical identifier is determined based on the hierarchical level to which the vehicle terminal data belongs, and the order identifier is determined based on the splitting order in the hierarchical level. Based on the hierarchical identifier and the order identifier, the information bit identifier is determined. That is, the information bit identifier is determined by combining the hierarchical identifier and the order identifier.

[0101] In a possible implementation, the information bit identifier of the vehicle terminal data can be determined according to the pre-determined information bit generation rule. For example, the number of bits of the information bit identifier can be determined first, and then a part of the bits is used to represent the hierarchical identifier, and the remaining bits are used to represent the order identifier.

[0102] For specific reference, see Figure 3As shown in the figure, this application example provides a schematic diagram of an information bit generation rule. In this application scenario, it can be determined that the information bit identifier is 9 bits. The first 3 bits of the information bit identifier are used to represent the hierarchical identifier, that is, the number of layers of vehicle terminal data flow. The last 6 bits of the information bit identifier are used to represent the sequence identifier, that is, the splitting sequence of vehicle terminal data at the corresponding layer. For example, the information bit identifier is "003000009". Among them, "003" indicates that the vehicle terminal data has experienced 3 layers of data flow, and the corresponding layer is the 3rd layer. "000009" indicates that the vehicle terminal data is the 9th data generated in the 3rd layer.

[0103] Among them, when the vehicle terminal data represents the parent node data or the child node data, the corresponding information bit generation rules are similar. For example, when the information bit identifier of the parent node data is "003000009", "003" indicates that the parent node data has experienced 3 layers of data flow, and the corresponding layer is the 3rd layer. "000009" indicates that the parent node data is the 9th data generated in the 3rd layer. When the information bit identifier of the child node data is "003000009", "003" indicates that the child node data has experienced 3 layers of data flow, and the corresponding layer is the 3rd layer. "000009" indicates that the child node data is the 9th data generated in the 3rd layer.

[0104] S204: Based on the storage field identifier and the information bit identifier, determine the metadata corresponding to the vehicle terminal data.

[0105] After determining the storage field identifier and the information bit identifier of the vehicle terminal data, the storage field identifier and the information bit identifier can be determined as the metadata of the vehicle terminal data.

[0106] In the actual application scenario, the metadata corresponding to different vehicle terminal data is different. The metadata of multiple vehicle terminal data can be stored as a metadata database. Subsequently, if it is necessary to trace the source of the data to be processed, the data associated with the data to be processed can be determined according to the metadata stored in the metadata database, so as to achieve problem tracing.

[0107] In a possible implementation, during the data flow of vehicle terminal data, some data content may be updated. The fields corresponding to the updated data are the same as the original data fields, but the specific content has changed. To determine the uniqueness of vehicle terminal data and avoid duplicate processing of the same data, the vehicle terminal data can be encrypted based on the information digest generation rule to determine the encryption identifier corresponding to the vehicle terminal data. Optionally, the Message-Digest Algorithm MD5 can be used to encrypt the vehicle terminal data to determine the uniquely corresponding encryption identifier. Then, the storage field identifier, information bit identifier, and encryption identifier can be determined as the metadata corresponding to the vehicle terminal data.

[0108] In a possible implementation, the process of determining the metadata described above can be executed by the data production end. Among them, the data production end can be a functional module of the processing device to determine the metadata of the vehicle terminal data.

[0109] For details, please refer to Figure 4 shown in the flowchart of determining metadata provided by the embodiment of the present application.

[0110] The process of the data production end determining the metadata may include the following steps:

[0111] A1: Obtain the original fields describing the attributes of the vehicle terminal data;

[0112] A2: Perform integrity verification on the original fields based on the field extraction rule;

[0113] A3: When the verification passes, determine the target fields from the original fields according to the field extraction rule, and determine the field identifiers of the target fields;

[0114] A4: Map the target fields and field identifiers to a dictionary for storage based on the data dictionary specification;

[0115] A5: Concatenate the field identifiers based on the concatenation rule to determine the concatenated field identifier;

[0116] A6: Encode the concatenated field identifier based on the encoding rule to determine the storage field identifier;

[0117] A7: Determine the information bit identifier based on the information bit generation rule and the hierarchical relationship of the vehicle terminal data;

[0118] A8: Encrypt the vehicle terminal data based on the information digest generation rule to determine the encryption identifier;

[0119] A9: Store the storage field identifier, information bit identifier, and encryption identifier as metadata in the metadata database.

[0120] Optionally, the field extraction rule, data dictionary specification, splicing rule, encoding rule, information bit generation rule, and information digest generation rule can be used as unified rules and stored in the common policy end. The common policy end can represent a functional module in the processing device and is used to store rules. Subsequently, the corresponding rules can be obtained from the common policy end for data processing.

[0121] In a possible implementation manner, in response to an operation of updating vehicle terminal data, the metadata of the vehicle terminal data can be obtained, that is, the storage field identifier and the information bit identifier are obtained. Based on the updated data in the vehicle terminal data, the fields to be updated in the target field corresponding to the storage field identifier are determined, and the fields to be updated are updated based on the updated data. That is, the updated data can be partial fields in the target field, so the fields to be updated in the target field that need to be updated can be determined based on the updated data, and then the content of the fields to be updated is updated. When the vehicle terminal data is updated, the hierarchical relationship of the vehicle terminal data also changes, and the information bit identifier can be updated based on the updated data.

[0122] For example, when the content of the current data node changes, the original data of the data node is used as the parent node, and the updated data of the data node is used as the corresponding child node. Then, according to the hierarchical relationship between the parent node and the child node, the information bit identifier is updated. That is, the information bit identifier of the child node corresponds to the information bit identifier of the changed parent node, the hierarchical information of the child node is incremented by one bit, and the sequence identifier of the child node changes simultaneously.

[0123] In a possible implementation manner, the above process of updating metadata can be executed by the data processing end, where the data processing end can be a functional module of the processing device.

[0124] Specifically, refer to Figure 5 as shown, which is a flowchart of updating metadata provided by an embodiment of the present application.

[0125] B1: Obtain the updated data in the vehicle terminal data;

[0126] Among them, the updated data can include newly added data, or data obtained through steps such as data cleaning, removing abnormal data, and manual annotation.

[0127] B2: Decode the storage field identifier corresponding to the vehicle terminal data to obtain the target field;

[0128] According to the above embodiment, it can be known that the storage field identifier can be obtained by encoding the splicing field identifier with the encoding rule. Therefore, after decoding the storage field identifier, the field identifier and the target field can be determined.

[0129] B3: Determine the fields to be updated in the target field based on the updated data, and update the fields to be updated.

[0130] For example, the fields to be updated can be determined according to the specific content of the updated data. For example, when the data type of the vehicle terminal data changes, the fields to be updated can include the data type and the update time.

[0131] B4: Update the information bit identifier based on the updated data.

[0132] B5: Encrypt the updated vehicle terminal data to obtain an encryption identifier.

[0133] Through the method provided in the embodiments of the present application, the metadata describing the vehicle terminal data can be determined according to a unified processing method. Since the processing methods are the same, the data processing efficiency can be improved, and it is also convenient to trace the data.

[0134] Based on the above method embodiments, the embodiments of the present application further provide a data processing method. Refer to Figure 6 As shown, it is a flowchart of another data processing method provided by the embodiments of the present application.

[0135] S601: Obtain the field identifier of the data to be processed.

[0136] Among them, the data to be processed can be determined artificially according to business requirements and can be data for problem tracing.

[0137] S602: Determine the target field identifier that matches the field identifier among the stored field identifiers included in the metadata database.

[0138] According to the above embodiments, it can be known that the metadata corresponding to the vehicle terminal data can be stored in the metadata database. That is, the metadata database can include multiple metadata corresponding to multiple vehicle terminal data, and each metadata includes the stored field identifier and the information bit identifier of the vehicle terminal data. Therefore, the field identifier of the data to be processed can be matched with multiple stored field identifiers in the metadata database to determine the target field identifier.

[0139] When the vehicle terminal data is updated, in the target fields corresponding to the storage field identifiers, only the specific contents of some fields may be updated, while the specific contents of other fields remain unchanged. Therefore, all the data associated with the data to be processed can be determined based on the fields that have not changed. That is, some fields describing the attributes of the vehicle terminal data have an inheritance relationship. In one possible implementation, the first identifier corresponding to the first field can be determined based on the field identifier. The second identifier corresponding to the second field is determined based on the storage field identifier in the metadata database. Among the multiple second identifiers included in the metadata database, the target field identifier that matches the first identifier is determined. Among them, there can be multiple target field identifiers.

[0140] For example, when the target fields describing the attributes of the vehicle terminal data include business type, data type, vehicle information, collection information, and update time, the first field can include business type, data type, vehicle information, and collection information. That is, the update time usually changes and is not used as the first field.

[0141] S603: Obtain the target information bit identifier corresponding to the target field identifier.

[0142] After determining the target field identifier, the target information bit identifier corresponding to the target field identifier can be determined according to the metadata.

[0143] In one possible implementation, the metadata can also include an encryption identifier. Therefore, after determining the target field identifier, the target information bit identifier and the target encryption identifier corresponding to the target field identifier can be obtained.

[0144] After determining the target information bit identifier and the target encryption identifier, optionally, it can be determined whether there are the same information bit identifiers in the target information bit identifier, and whether there are the same encryption identifiers in the target encryption identifier. If so, it indicates that there is the same data, and the duplicate data can be deleted.

[0145] S604: Determine the data lineage based on the target information bit identifier.

[0146] According to the above embodiments, the information bit identifier can represent the hierarchical relationship of the vehicle terminal data. The target information bit identifier determined according to the target field identifier includes multiple data with an association relationship. Therefore, the data lineage can be determined according to the target information bit identifier. As Figure 7 shown, it is a schematic diagram of a data lineage provided by an embodiment of the present application. Subsequently, problem tracing can be performed based on the data lineage.

[0147] In a possible implementation, the above method for processing data can be executed by a data consumer, where the data consumer can also be a functional module of a processing device and execute the data processing method.

[0148] According to the above embodiments, the field identifier can be an encoded identifier. That is, the stored field identifier in the metadata database can also be an encoded identifier. Based on this, the target field identifier can be determined in the following manner. First, decode the field identifier to obtain the decoded field identifier. Decode the stored field identifier to obtain the decoded stored field identifier. Then, in the decoded stored field identifier, determine the target field identifier that matches the decoded field identifier.

[0149] To determine the uniqueness of vehicle terminal data and avoid duplicate processing of the same data, the above embodiments can also encrypt the vehicle terminal data based on an information digest generation rule to determine the encryption identifier corresponding to the vehicle terminal data. Based on this, the encryption identifier corresponding to the target field identifier can also be obtained, and then a summary table can be generated based on the target field identifier, the target information bit identifier, and the encryption identifier, so as to determine the data lineage according to the summary table.

[0150] In a possible implementation, when there are identical encryption identifiers in the encryption identifier, that is, there are duplicate encryption identifiers, the duplicate encryption identifiers can be deleted, and the target field identifier and the target information bit identifier corresponding to the duplicate encryption identifier can be deleted to update the summary table.

[0151] In a possible implementation, a data lineage map can be generated based on the updated summary table for data tracing. For example, a tree-like relationship structure linked list can be formed according to the updated summary table, and then a data lineage map can be generated according to the tree-like relationship structure linked list. Subsequently, data tracing can be performed according to the data lineage map.

[0152] See Figure 8 As shown, it is a schematic diagram of a processing device provided by an embodiment of the present application. The processing device can include a common policy end, a data production end, a data processing end, and a data consumer. Among them, the working principles of each functional module can be referred to the above method embodiments and will not be elaborated here.

[0153] Based on the above method embodiments, an embodiment of the present application provides a data processing device. See Figure 9 As shown, it is a schematic diagram of a data processing device provided by an embodiment of the present application.

[0154] The device 900 includes:

[0155] The first determination unit 901 is configured to determine a target field for describing the attributes of vehicle terminal data and a field identifier of the target field based on a field extraction rule;

[0156] The second determination unit 902 is configured to splice the field identifiers based on a splicing rule to determine a storage field identifier;

[0157] The third determination unit 903 is configured to determine an information bit identifier of the vehicle terminal data based on an information bit generation rule and a hierarchical relationship of the vehicle terminal data;

[0158] The fourth determination unit 904 is configured to determine metadata corresponding to the vehicle terminal data based on the storage field identifier and the information bit identifier.

[0159] In a possible implementation manner, the apparatus further includes: an encryption unit; before determining the metadata of the vehicle terminal data based on the storage field identifier and the information bit identifier, the encryption unit is configured to perform an encryption process on the vehicle terminal data based on an information digest generation rule to determine an encryption identifier of the vehicle terminal data;

[0160] The fourth determination unit 904 is specifically configured to determine the metadata based on the storage field identifier, the information bit identifier, and the encryption identifier.

[0161] In a possible implementation manner, the apparatus further includes: an update unit, configured to, in response to an operation of updating the vehicle terminal data, obtain a storage field identifier and an information bit identifier in the metadata; determine a field to be updated in the target field corresponding to the storage field identifier based on the update data in the vehicle terminal data, and update the field to be updated based on the update data; update the information bit identifier based on the update data.

[0162] In a possible implementation manner, the second determination unit 902 is specifically configured to splice the field identifiers of the target field based on the splicing rule to determine a spliced field identifier; perform an encoding process on the spliced field identifier based on an encoding rule to determine the storage field identifier.

[0163] In a possible implementation manner, the third determination unit 903 is specifically configured to determine a hierarchical level to which the vehicle terminal data belongs and a splitting order in the hierarchical level based on the hierarchical relationship; determine a hierarchical identifier based on the hierarchical level, and determine an order identifier based on the splitting order; determine the information bit identifier based on the hierarchical identifier and the order identifier.

[0164] In a possible implementation, the apparatus further includes: a mapping unit, configured to map the target field and the field identifier to a dictionary based on a data dictionary specification, where the field identifier represents a keyword of the dictionary, and the target field represents a value of the dictionary.

[0165] In addition, an embodiment of the present application further provides a data processing apparatus. Refer to Figure 10 As shown, it is a schematic diagram of another data processing apparatus provided by an embodiment of the present application.

[0166] The apparatus 1000 includes:

[0167] A first acquisition unit 1001, configured to acquire a field identifier of data to be processed;

[0168] A fifth determination unit 1002, configured to determine a target field identifier that matches the field identifier in the stored field identifiers included in the metadata database, where the metadata database includes a plurality of metadata corresponding to a plurality of vehicle terminal data, the metadata includes the stored field identifier and the information bit identifier of the vehicle terminal data, and the metadata is determined based on any one of the implementation manners in the above first aspect;

[0169] A second acquisition unit 1003, configured to acquire a target information bit identifier corresponding to the target field identifier;

[0170] A sixth determination unit 1004, configured to determine a data lineage based on the target information bit identifier.

[0171] In a possible implementation, the fifth determination unit 1002 is specifically configured to determine a first identifier corresponding to a first field based on the field identifier; determine a second identifier corresponding to a second field based on the stored field identifier; and determine a target field identifier that matches the first identifier among the plurality of second identifiers included in the metadata database.

[0172] In a possible implementation, when the field identifier and the stored field identifier are encoded identifiers, the fifth determination unit 1002 is specifically configured to perform a decoding process on the field identifier to obtain a decoded field identifier; perform a decoding process on the stored field identifier to obtain a decoded stored field identifier; and determine the target field identifier that matches the decoded field identifier in the decoded stored field identifiers.

[0173] In a possible implementation, the apparatus further includes: a third obtaining unit, configured to obtain an encryption identifier corresponding to the target field identifier, where the encryption identifier is obtained by encrypting vehicle terminal data based on an information digest generation rule; and a generating unit, configured to generate a summary table based on the target field identifier, the target information bit identifier, and the encryption identifier.

[0174] In a possible implementation, the apparatus further includes: an updating unit, configured to, when there are identical encryption identifiers in the encryption identifiers, delete the duplicate encryption identifiers and delete the target field identifier and the target information bit identifier corresponding to the duplicate encryption identifiers, so as to update the summary table.

[0175] In a possible implementation, the sixth determining unit 1004 is specifically configured to generate a data lineage map based on the updated summary table, so as to perform data traceability based on the data lineage map.

[0176] Based on the above method embodiments and apparatus embodiments, an embodiment of the present application further provides an electronic device. This will be introduced below with reference to the accompanying drawings.

[0177] See Figure 11 , Figure 11 which is a schematic diagram of an electronic device provided by an embodiment of the present application.

[0178] The electronic device 1100 includes: a memory 1101 and a processor 1102;

[0179] The memory 1101 is used to store relevant program codes;

[0180] The processor 1102 is used to call the program codes and execute the data processing method described in the above method embodiments.

[0181] In addition, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the data processing method described in the above method embodiments.

[0182] It should be noted that the technical features in the upper-layer means provided in the embodiments of the present application are clear to those skilled in the art, and the problems to be solved by the upper-layer means are also clear. How to obtain the means for the corresponding features can be selected by those skilled in the art according to specific implementation requirements. The means provided in the present application should not be regarded as a limitation to the solution or the only implementation means.

[0183] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments. The device embodiments described above are merely illustrative. The units or modules described as separate components may or may not be physically separated. The components shown as units or modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network units. Some or all of the units or modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of methods, devices, and equipment according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0185] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist at the same time. Here, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (one)" or its similar expression below refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0186] It should also be noted that in this application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0187] The steps of the methods or algorithms described in connection with the embodiments disclosed in this application can be implemented directly in hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well known in the art.

[0188] The foregoing description of the disclosed embodiments enables those skilled in the art to practice or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this application can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed in this application.

Claims

1. A method for processing data, characterized in that, The method includes: Determining a target field for describing the attributes of vehicle terminal data and the field identifier of the target field based on field extraction rules; Concatenating the field identifiers based on concatenation rules to determine a stored field identifier; Determining the information bit identifier of the vehicle terminal data based on information bit generation rules and the hierarchical relationship of the vehicle terminal data; Determining the metadata corresponding to the vehicle terminal data based on the stored field identifier and the information bit identifier.

2. The method according to claim 1, characterized in that, Before determining the metadata corresponding to the vehicle terminal data based on the stored field identifier and the information bit identifier, the method further includes: Performing encryption processing on the vehicle terminal data based on information digest generation rules to determine the encryption identifier of the vehicle terminal data; The determining the metadata corresponding to the vehicle terminal data based on the stored field identifier and the information bit identifier includes: Determining the metadata based on the stored field identifier, the information bit identifier, and the encryption identifier.

3. The method according to claim 1, wherein The method further includes: In response to an operation to update the vehicle terminal data, obtaining the stored field identifier and the information bit identifier in the metadata; Determining the field to be updated in the target field corresponding to the stored field identifier based on the update data in the vehicle terminal data, and updating the field to be updated based on the update data; Updating the information bit identifier based on the update data.

4. The method according to claim 1, characterized in that, The concatenating the field identifiers based on concatenation rules to determine a stored field identifier includes: Concatenating the field identifiers of the target fields based on the concatenation rules to determine a concatenated field identifier; Performing encoding processing on the concatenated field identifier based on encoding rules to determine the stored field identifier.

5. The method according to claim 1, characterized in that, The determining the information bit identifier of the vehicle terminal data based on the hierarchical relationship of the vehicle terminal data includes: Determining the belonging level of the vehicle terminal data and the splitting order in the belonging level based on the hierarchical relationship; Determining a level identifier based on the belonging level and determining an order identifier based on the splitting order; Determining the information bit identifier based on the level identifier and the order identifier.

6. The method according to claim 1, characterized in that The method further includes: Mapping the target field and the field identifier to a dictionary based on data dictionary specifications, where the field identifier represents the keyword of the dictionary and the target field represents the value of the dictionary.

7. A data processing method, characterized in that, The method includes: Obtaining the field identifier of the data to be processed; Determining a target field identifier that matches the field identifier among the stored field identifiers included in the metadata database, where the metadata database includes multiple metadata corresponding to multiple vehicle terminal data, the metadata includes the stored field identifier and the information bit identifier of the vehicle terminal data, and the metadata is determined based on the method according to any one of claims 1 to 6; Obtaining the target information bit identifier corresponding to the target field identifier; Determining the data lineage based on the target information bit identifier.

8. The method according to claim 7, wherein The determining a target field identifier that matches the field identifier among the stored field identifiers included in the metadata database includes: Determine a first identifier corresponding to the first field based on the field identifier; Determine a second identifier corresponding to the second field based on the stored field identifier; Among the multiple second identifiers included in the metadata database, determine a target field identifier that matches the first identifier.

9. The method according to claim 7, wherein When the field identifier and the stored field identifier are encoded identifiers, the determining, in the stored field identifiers included in the metadata database, a target field identifier that matches the field identifier includes: Perform a decoding process on the field identifier to obtain the decoded field identifier; Perform a decoding process on the stored field identifier to obtain the decoded stored field identifier; Among the decoded stored field identifiers, determine the target field identifier that matches the decoded field identifier.

10. The method according to claim 7, wherein The method further includes: Obtain an encrypted identifier corresponding to the target field identifier, where the encrypted identifier is obtained by encrypting vehicle terminal data based on an information digest generation rule; Generate a summary table based on the target field identifier, the target information bit identifier, and the encrypted identifier.

11. The method according to claim 10, wherein The method further includes: When there are identical encrypted identifiers in the encrypted identifiers, delete the duplicate encrypted identifiers and delete the target field identifier and the target information bit identifier corresponding to the duplicate encrypted identifiers to update the summary table.

12. The method according to claim 11, wherein The determining the data lineage based on the target information bit identifier includes: Generate a data lineage map based on the updated summary table for data tracing based on the data lineage map.

13. A data processing device, characterized in that, The apparatus includes: A first determination unit, configured to determine a target field for describing the attributes of vehicle terminal data and the field identifier of the target field based on a field extraction rule; A second determination unit, configured to splice the field identifiers based on a splicing rule to determine a stored field identifier; A third determination unit, configured to determine an information bit identifier of the vehicle terminal data based on an information bit generation rule and the hierarchical relationship of the vehicle terminal data; A fourth determination unit, configured to determine metadata corresponding to the vehicle terminal data based on the stored field identifier and the information bit identifier.

14. A data processing device, characterized in that, The apparatus includes: A first acquisition unit, configured to acquire a field identifier of data to be processed; A fifth determination unit, configured to determine, in the stored field identifiers included in a metadata database, a target field identifier that matches the field identifier, where the metadata database includes multiple metadata corresponding to multiple vehicle terminal data, the metadata includes the stored field identifier and the information bit identifier of the vehicle terminal data, and the metadata is determined based on the method according to any one of claims 1 to 7; A second acquisition unit, configured to acquire a target information bit identifier corresponding to the target field identifier; A sixth determination unit, configured to determine a data lineage based on the target information bit identifier.

15. An electronic device, characterized in that, The device includes: a memory and a processor; The memory is used to store relevant program codes; The processor is used to call the program codes to execute the method according to any one of claims 1 to 6, or any one of claims 7 to 12.

Images