Battery data processing method, device and platform and storage medium

Abstract

The embodiment of the invention discloses a battery data processing method and device, a platform and a storage medium, and the method comprises the steps: obtaining first battery data and a format declaration, the first battery data comprises data items and attribute information, and the format declaration comprises a target format corresponding to the data items; compiling and generating a data processing function module based on a data item in the first battery data and a target format corresponding to the data item in the format declaration; and the format of the attribute information of the data item is converted into the target format based on the data processing function module, so that a user does not need to preset different code templates for different battery data, and the data processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a battery data processing method, device, platform and storage medium. Background Art

[0002] Batteries have become the preferred energy source for new energy vehicles and energy storage power stations due to their high energy density, long service life and zero pollution.

[0003] In order to obtain information such as battery life or battery health status, it is necessary to obtain battery data for calculation. However, the format of attribute information of the same data item may not be unified between battery data obtained from different sources and cannot be directly calculated. Therefore, it is necessary to unify the format of attribute information of the same data item in battery data obtained from different sources before calculating the battery life or battery health status.

[0004] At present, in order to unify the format of attribute information of the same data item in battery data from different sources, the user inputs the target format of the attribute information of each data item, generates a data processing function module according to the target format and a pre-set code template, and uses the data processing function module to unify the format of the attribute information of the same data item.

[0005] However, the above method requires the user to pre-set different code templates to adapt to different battery data, resulting in low data processing efficiency. Summary of the invention

[0006] In view of the above problems, the present application provides a battery data processing method, device, platform and storage medium, which can improve data processing efficiency without the need for users to pre-set different code templates for different battery data.

[0007] In a first aspect, the present application provides a battery data processing method, comprising: acquiring first battery data and a format declaration, wherein the first battery data comprises a data item and attribute information, and the format declaration comprises a target format corresponding to the data item;

[0008] Compile and generate a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration;

[0009] The format of the attribute information of the data item is converted into the target format based on the data processing function module.

[0010] The battery data processing method provided in the technical solution of the embodiment of the present application obtains the first battery data and the format declaration, compiles and generates a data processing function module based on the data item of the first battery data and the target format corresponding to the data item in the format declaration, and converts the format of the attribute information of the data item into the target format based on the data processing function module. Compared with generating a data processing function module according to the target format in the format declaration and the pre-set code template, for different battery data, there is no need for the user to pre-set different code templates. In the process of generating the data processing function module, the data processing function module can be directly generated by compiling according to the data item of the first battery data and the target format corresponding to the data item in the format declaration, without having to search for the code template corresponding to the format declaration of the battery data in many pre-set code templates, which simplifies the processing flow and improves data processing efficiency; at the same time, since there is no need to pre-set different code templates, storage space can also be saved.

[0011] In some embodiments, before compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration, the method further includes:

[0012] The target formats in the format declaration are sorted according to the order of the data items.

[0013] In this embodiment, the arrangement order of multiple target formats in the format declaration is adjusted so that battery data with different data item arrangement orders can generate corresponding data processing function modules with the same format declaration, thereby processing the attribute information of different battery data. Since in the embodiment of the present application, when processing different battery data, it is only necessary to obtain the target format once, and there is no need to obtain different target formats multiple times, the workflow is simplified, thereby improving data processing efficiency.

[0014] In some embodiments, in the process of compiling and generating a data processing function module based on data items in the first battery data and target formats corresponding to the data items in the format declaration, the target formats in the format declaration are sorted according to the order of the data items.

[0015] In this embodiment, the order of arrangement of the target format in the format declaration is changed by adjusting the order of arrangement of the subtrees in the abstract syntax tree. When adjusting the order of the subtrees, only the position of the root node in the subtree needs to be changed, and the positions of the subnodes under the root node also change accordingly. The operation is simple and the execution efficiency is high.

[0016] In some embodiments, in the process of compiling and generating a data processing function module based on data items in the first battery data and a target format corresponding to the data items in the format declaration, an abstract syntax tree generated based on the format declaration is optimized.

[0017] In the process of compiling and generating the data processing function module in this embodiment, the abstract syntax tree generated based on the format declaration is optimized, and several performance optimizations are made at the machine instruction level, so that this solution has obvious advantages in big data application scenarios.

[0018] In some embodiments, optimizing an abstract syntax tree generated based on the format declaration includes:

[0019] When a data item corresponding to a subtree in the abstract syntax tree does not appear in a data item in the first battery data, the subtree is pruned.

[0020] In this embodiment, pruning the subtree of the abstract syntax tree can avoid generating redundant data processing functions, reduce the amount of code corresponding to the final data processing function module, speed up the generation of the data processing function module and save memory.

[0021] In some embodiments, the optimizing the abstract syntax tree generated based on the format declaration includes:

[0022] Optimize logical expressions of nodes of each subtree in the abstract syntax tree and / or arrange child nodes of the same type in the nodes of each subtree in the abstract syntax tree under the same parent node.

[0023] In the embodiment of the present application, by optimizing the logical expressions of the child nodes of each subtree of the abstract syntax tree, the amount of calculation during code execution can be minimized. By arranging the child nodes of the same type in the child nodes of each subtree of the abstract syntax tree under the same parent node, the cyclomatic complexity of the code finally generated can be reduced, the code branches can be reduced, the branch prediction performance during processor execution can be improved, the number of pipeline switching times can be reduced, and the instruction cache hit rate can be improved.

[0024] In some embodiments, the obtaining the first battery data includes:

[0025] Get initial battery data;

[0026] Determining a data parsing component for the initial battery data according to a data input format of the initial battery data and a correspondence between a preset data input format and a data parsing component;

[0027] The data parsing component is used to parse the data item and the attribute information corresponding to the data item from the initial battery data to obtain the first battery data.

[0028] In this embodiment, by identifying the data input format of the initial battery data, the initial battery data in different data input formats can be directly parsed into multiple data items and attribute information corresponding to the multiple data items using the corresponding data parsing components, that is, parsed into first battery data that is independent of the data input format, without having to first convert the initial battery data into a specific data input format and then parse the data items and attribute information, which can further improve data processing efficiency.

[0029] In some embodiments, after obtaining the format declaration, the method further includes:

[0030] When the format declaration is updated, a temporary form is generated in the data warehouse according to the updated format declaration;

[0031] The data storage address of the historical form of the format declaration is determined as the data storage address of the temporary form.

[0032] In this embodiment, when the format declaration is updated, a new temporary form is generated instead of updating the original historical form. In this way, when the format declaration is updated, the historical form can still be used, and the data processing function module is still generated by relying on the historical form without interruption, thereby realizing online update of the format declaration and improving data processing efficiency.

[0033] In some embodiments, after obtaining the format declaration, the method further includes:

[0034] Verifying the legality of the format declaration according to pre-set legality rules;

[0035] When the verification is passed, the input format declaration is stored in the cache database.

[0036] In this embodiment, after obtaining the format declaration, the legality of the format declaration needs to be verified first to avoid generating an erroneous data processing function module due to an illegal format declaration, thereby improving the accuracy of data processing.

[0037] In some embodiments, before compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration, the method further includes:

[0038] According to the data items in the first battery data and the preset correspondence between the historical data items and the historical data processing function modules, searching for the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data;

[0039] If the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data is not found, the step of compiling and generating the data processing function module based on the data item in the first battery data and the target format corresponding to the data item in the format declaration is performed again.

[0040] In some embodiments, after searching for a historical data processing function module corresponding to a historical data item identical to a data item in the first battery data according to the data item in the first battery data and the preset correspondence between the historical data item and the historical data processing function module, the method further includes:

[0041] When a history data processing function module corresponding to the history data item identical to the data item in the first battery data is found, the format of the attribute information corresponding to the data item is converted into the target format based on the history data processing function module.

[0042] In this embodiment, for data items that have already generated data processing function modules, when subsequently processing the attribute information of the same data items, there is no need to repeat the process of generating the data processing function modules. The historical data processing function modules can be directly used for processing. This has the advantage of generating once and running multiple times, avoiding the operation of repeatedly generating data processing function modules and speeding up data processing.

[0043] In some embodiments, the data processing function module is implemented in the form of Java bytecode. In the embodiments of the present application, the data processing function module finally generated is in the form of Java bytecode, which can be directly executed by the Java virtual machine of the data processing node. Compared with generating a high-level programming language, it does not need to be compiled again, and can be directly integrated with common big data processing software without stopping for loading, thereby speeding up data processing.

[0044] In a second aspect, an embodiment of the present invention further provides a battery data processing device, the battery data processing device comprising:

[0045] an acquisition module, configured to acquire first battery data and a format declaration, wherein the first battery data includes data items and attribute information, and the format declaration includes a target format corresponding to the data items;

[0046] A processing module is used to compile and generate a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration; and convert the format of the attribute information of the data item into the target format based on the data processing function module.

[0047] The battery data processing device provided in this embodiment obtains the first battery data and the format declaration, compiles and generates a data processing function module based on the data items of the first battery data and the target format corresponding to the data items in the format declaration, and converts the format of the attribute information of the data items into the target format based on the data processing function module. Compared with generating a data processing function module according to the target format in the format declaration and the pre-set code template, for different battery data, there is no need for the user to pre-set different code templates. In the process of generating the data processing function module, the data processing function module can be directly generated by compiling according to the data items of the first battery data and the target format corresponding to the data items in the format declaration, without having to search for the code template corresponding to the format declaration of the battery data in many pre-set code templates, which simplifies the processing flow and improves data processing efficiency; at the same time, since there is no need to pre-set different code templates, storage space can also be saved.

[0048] In a third aspect, an embodiment of the present invention further proposes a battery data processing platform, which is used to execute the method described above.

[0049] In a fourth aspect, an embodiment of the present invention further provides a storage medium, on which a battery data processing instruction is stored. When the battery data processing instruction is executed by a processor, the battery data processing method as described above is implemented.

[0050] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0052] Figure 1 A structural block diagram of an embodiment of a battery data processing platform proposed in an embodiment of the present invention;

[0053] Figure 2 A structural block diagram of a processing node in a battery data processing platform proposed in an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a flow chart of an embodiment of a battery data processing method proposed in an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a flow chart for generating a data processing function module is provided for an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of a process for proposing a cache format declaration according to an embodiment of the present invention;

[0057] Figure 6 A flowchart of an implementation method for generating a data processing function module according to battery data and format declaration is proposed for an embodiment of the present invention;

[0058] Figure 7 This is a structural block diagram of a battery data processing device proposed in an embodiment of the present invention.

[0059] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" in the specification and claims of the present invention and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0062] In the description of the embodiments of the present invention, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present invention, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0063] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0064] In the description of the embodiments of the present invention, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0065] In the description of the embodiments of the present invention, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0066] In the description of the embodiments of the present invention, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the embodiments of the present invention.

[0067] In the description of the embodiments of the present invention, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0068] As the country vigorously promotes new energy technologies, batteries have become the preferred energy source for new energy vehicles and energy storage power stations due to their high energy density, long service life and zero pollution.

[0069] At present, in order to obtain the battery life or monitor the health status of the battery, it is necessary to obtain the battery data for calculation. Battery data generally includes data items and attribute information of the data items. For example, a data item is "voltage" and the attribute information of the data item is 50V. However, the format of the attribute information of battery data from different sources is not the same. For example, the unit of the attribute information of the data item "voltage" in the battery data reported by the vehicle OEM is mV, while the unit of the attribute information of the data item "voltage" in the battery data reported by the energy storage power station is V. Therefore, in order to calculate the battery life or monitor the health status of the battery, etc., it is first necessary to unify the attribute information of the same type of data items in different battery data into the same target format. For example, the target format can be a national standard or a customized format standard of the enterprise.

[0070] In order to unify the format of attribute information of the same type of data items in different battery data, the user needs to manually input the target format of the attribute information and generate a data processing function module according to the target format and a pre-set code template. However, for different battery data, the user needs to pre-set different code templates. Before generating the data processing function module, it is necessary to search for a code template corresponding to the format declaration of the battery data among many pre-set code templates, resulting in low data processing efficiency.

[0071] Based on the above considerations, a battery data processing method is proposed in an embodiment of the present application, which obtains the first battery data and the format declaration, compiles and generates a data processing function module based on the data items of the first battery data and the target format corresponding to the data items in the format declaration, and converts the format of the attribute information of the data items into the target format based on the data processing function module. Compared with generating a data processing function module according to the target format in the format declaration and the pre-set code template, for different battery data, there is no need for the user to pre-set different code templates. In the process of generating the data processing function module, the data processing function module can be directly generated by compiling according to the data items of the first battery data and the target format corresponding to the data items in the format declaration, without having to search for the code template corresponding to the format declaration of the battery data in many pre-set code templates, which simplifies the processing flow and improves data processing efficiency; at the same time, since there is no need to pre-set different code templates, storage space can also be saved.

[0072] After in-depth research, the embodiments of the present application propose a battery data processing method, device, platform and storage medium, which are suitable for various scenarios of battery data processing during the actual use of the battery, for example, the processing of battery data reported by the vehicle main manufacturer, the processing of battery data reported by the battery swap station, the processing of battery data reported by the energy storage station, etc.

[0073] In order to better understand the embodiments of the present application, the battery data processing method, device, platform and storage medium provided according to the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0074] The execution subject of the battery data processing method provided in the embodiment of the present application may be a battery data processing platform, which can be used to process battery data from different sources, such as battery data reported by vehicle OEMs, battery data reported by battery swap stations, and battery data reported by energy storage stations. The format of attribute information of similar data items in battery data from different sources is unified, and the processed battery data can be used to calculate the lifespan, health status, etc. of battery of vehicle OEMs, battery of battery swap stations, and battery of energy storage stations.

[0075] like Figure 1 As shown, the battery data processing platform can be a big data platform, which includes a master node and multiple processing nodes connected to the master node, such as Figure 1 As shown, the big data platform includes processing node 1, processing node 2, and processing node N, where N is an integer greater than 2. Each processing node includes a memory and a processor, and the memory of each processing node stores the battery data processing method in the embodiment of the present application.

[0076] The master node is used to generate multiple tasks and distribute the multiple tasks to multiple processing nodes. Each processing node is used to receive battery data according to the received tasks and process the battery data using the battery data processing method in the embodiment of the present application.

[0077] In one example, a Java virtual machine is installed on each processing node of the big data platform. In this way, the battery data processing method in the embodiment of the present application can be developed using the Java programming language, packaged as a Java archive file, and distributed to each processing node of the big data platform. The Java virtual machine of each processing node can load the Java archive file.

[0078] For different big data tools, the Java archive file can be used in different ways on the processing node. For big data tools developed using Structured Query Language (SQL), such as Spark SQL, Flink SQL, Hive, etc., the data processing function module generated after running the above Java archive file can be directly loaded as a user-defined function (UDF) for use. For big data tools developed using Java and Scala (Scala is a multi-paradigm programming language, a programming language similar to Java), such as Spark and Flink, the data processing function module generated after running the above Java archive file can be directly called.

[0079] For the sake of convenience, here we use Figure 1 The battery data processing platform shown serves as a specific application environment of the battery data processing method provided in this embodiment. The battery data processing platform is only a possible example, and does not limit the specific application environment of the battery data processing method provided in the embodiment of the present application. The battery data processing method provided in the embodiment of the present application can also be applied to other battery data processing platforms.

[0080] The processing nodes in the battery data processing platform can include a front-end interaction layer and a back-end management layer, such as Figure 2 shown.

[0081] The front-end interaction layer can be used as an interface for data collection in the battery data processing method, and can obtain battery data from different sources through different interfaces and standards. For example, the front-end interaction layer can obtain real-time uploaded battery data from vehicle OEMs, battery swap stations, energy storage stations, etc. through IoT protocols such as message queue telemetry transport (MQTT). For example, the front-end interaction layer can also obtain battery data imported in batches by vehicle OEMs, battery swap stations, energy storage stations, etc. through a USB interface. Battery data may include: measurement data, time data, etc., wherein the measurement data includes current data, voltage data, temperature data, etc., and the time data includes the start charging time, the end charging time, etc. The front-end interaction layer can also interact with the user to obtain various format declarations input by the user. The format declaration includes the target format corresponding to the data item. It is worth noting that for offline batch imported battery data, the front-end interaction layer input processing node is used; for real-time uploaded battery data, after the front-end interaction layer is configured for the first time, it can be directly obtained by the back-end management layer.

[0082] The back-end management layer receives the first battery data and format declaration obtained by the front-end interaction layer, compiles and generates a data processing function module based on the data item of the first battery data and the target format corresponding to the data item in the format declaration, and converts the format of the attribute information of the data item into the target format based on the data processing function module. Compared with generating a data processing function module according to the target format in the format declaration and a pre-set code template, for different battery data, there is no need for the user to pre-set different code templates. In the process of generating a data processing function module, the data processing function module can be directly generated by compiling according to the data item of the first battery data and the target format corresponding to the data item in the format declaration, without having to search for the code template corresponding to the format declaration of the battery data in many pre-set code templates, which simplifies the processing flow and improves data processing efficiency; at the same time, since there is no need to pre-set different code templates, storage space can also be saved.

[0083] In some embodiments of the present invention, reference Figure 3 , an embodiment of the present invention proposes a battery data processing method, which may include:

[0084] Step S10: Acquire first battery data and format declaration.

[0085] Specifically, this step includes two steps of "obtaining first battery data" and "obtaining a format declaration". These two steps may be performed simultaneously or successively, which is not limited in this embodiment.

[0086] First, “obtaining first battery data” will be described in detail.

[0087] The first battery data includes data items and attribute information. The first battery data may include multiple data items and attribute information corresponding to the multiple data items, and one data item corresponds to one attribute information. For example, the data item may be "voltage", and the attribute information corresponding to the data item may be "220V"; for another example, the data item may be "current", and the attribute information corresponding to the data item may be "50A".

[0088] It is worth noting that the attribute information of the data item can be empty. The multiple data items of battery data often include multiple types of data items, and the names of different types of data items are different. For example, "voltage" and "current" belong to different types of data items. In an example, the multiple data items of battery data may include: 2 "voltages", 1 "current", 2 "times" and other data items.

[0089] Next, the "Get Format Declaration" is explained in detail.

[0090] The format declaration is used to specify the target format of the attribute information of the data item, and the data item corresponds to the target format.

[0091] It is feasible to pre-set multiple format declarations and store them. When processing the first battery data, the multiple format declarations stored are displayed on the display interface, and the staff can select the format declaration required to process the first battery data. It is worth noting that the format declaration of the first battery data can be filled in by the user on the web page, and then recognized and stored by the machine by converting the web page content into json format, or the user directly enters the format declaration in json format and stores it.

[0092] Due to the huge amount of battery data, a format declaration often includes multiple target formats, and one target format corresponds to one data item. In the embodiment of the present application, the target format of the attribute information of a data item may include the following:

[0093] 1. Field name, used to match data items in the battery data to determine which data item the target format corresponds to.

[0094] 2. Data type, used to specify the data type of the attribute information corresponding to the data item of the battery data. Data types include but are not limited to integer type int, single-precision floating point type float, double-precision floating point type double, text type text, time type date, Boolean type bool, enumeration type, structure type struct, array type array. Among them, the time type date can additionally specify the date format; the structure type struct and array type array are composite data structures, and the internal fields can be recursively declared.

[0095] 3. Verification rules are used to verify whether the attribute information corresponding to the data item of the battery data is legal. Verification rules include but are not limited to: whether it is necessary (true means that the attribute information corresponding to the data item must be verified, false means it is not necessary), whether it exceeds the maximum value, whether it exceeds the minimum value, step length, enumeration range, array length, etc. Among them, whether it must apply to all data types; whether it exceeds the maximum value, whether it exceeds the minimum value, and step length apply to all numeric types; enumeration range applies to enumeration types; array length applies to array types.

[0096] 4. Processing rules, used to process the attribute information corresponding to the data items of the battery data, including but not limited to: performing algebraic operations on attribute information of numerical type, performing operations such as concatenation, interception, and regular matching on attribute information of string value, decrypting encrypted attribute information, and decompressing compressed attribute information.

[0097] 5. Custom extensions allow user-defined declarations, including the full class name of the user-defined format declaration and the function name of the user-defined format declaration. For example, the full class name of the custom extension in the format declaration of a data item is: com.catl.cari.pdbd.algo.PreProcessing, and the function name of the custom extension is "preProcess". Before using the user-defined format declaration, the user-defined program needs to be submitted to the memory.

[0098] It is worth noting that the target format of each data item includes the field name and data type, and also includes any one or any combination of validation rules, processing rules, and custom extensions.

[0099] Taking the input first battery data including multiple data items "VIN", "VOLT", and "date" as an example, the format declaration of the first battery data when displayed in json format may be as follows:

[0100]

[0101]

[0102] The above format declaration includes three target formats, which correspond to the three data items "VIN", "VOLT", and "date" one by one. The target format corresponding to the data item VIN is:

[0103]

[0104] The target format of the data item "VIN" includes: the field name identifier "VIN", the data type dateType is text type text, and the verification rule includes "required" as "false" to indicate that verification is not required. The target format also includes the access mode accessMode as "r", that is, the read mode. The target format also includes the unified name name "VIN code". Among them, the access mode accessMode and the unified name name are optional items.

[0105] The target format corresponding to the data item "VOLT" is:

[0106]

[0107] The target format of the data item "VOLT" includes: the field name identifier "VOLT", the data type dateType is the double-precision floating point type double, the verification rule includes "required" as "true" to indicate that verification is required, and the rule description "specs" indicates that the legal minimum value of the attribute information "min" is 0, and the maximum value "max" is 100; the processing rule normalize includes "value / 1000", that is, the value of the data item is divided by 1000; the custom extension plugin is: com.catl.cari.cpbd.algo.PreProcesser:"preprocess", that is, when the attribute information of the data item is output, the preProcess method in the com.catl.cari.pdbd.algo.PreProcessing class needs to be called additionally for processing. The target format also includes the access mode accessMode as "r", that is, the read mode. The target format also includes the unified name name "voltage". Among them, the access mode accessMode and the unified name name are optional items.

[0108] The target format corresponding to the data item "data" is:

[0109]

[0110] The target format of the data item "data" includes: field name identifier "data", data type dateType is time type date, verification rules include: "Required" required is "true" to indicate verification is required, rule description "specs" data item "data" format is "yyyy-MM-dd HH:mm:ss" or "yyyy / MM / dd HH:mm:ss". The target format also includes access mode accessMode as "r", i.e. read mode. The target format also includes a unified name name "timestamp". Among them, access mode accessMode and unified name name are optional items.

[0111] It should be noted that the field name identifier can be set to one or more. For example, the field name in the target format of the data item "VIN" can include not only "VIN" but also "VIN code", "vehicle identification code", etc.; for another example, the field name in the target format of the data item "VOLT" can include not only "VOLT" but also "voltage", "volt", "voltage", etc. In other words, multiple different field names can be declared for data items with the same meaning, avoiding the situation where data items with the same meaning use different names and cannot be matched to the target format, and repeated format declaration is required.

[0112] Step S20: compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration.

[0113] In this embodiment, taking the case where there are multiple data items in the first battery data as an example, the process of compiling and generating the data processing function module is described in detail:

[0114] (1) Deserialize the format declaration to obtain an abstract syntax tree, where the abstract syntax tree includes multiple subtrees, and the multiple subtrees correspond to the multiple target formats in the format declaration one by one and in the same order.

[0115] Specifically, a data processing function module is automatically generated by compiling according to a plurality of data items and format declarations, wherein the data processing function module can be implemented in the form of machine code. The compilation process can be referred to Figure 5 , which generally includes: deserializing the format declaration to generate an abstract syntax tree (Abstract Syntax Tree, referred to as AST), and then generating a data processing function module according to the nodes on each subtree in the abstract syntax tree and the mapping relationship between the pre-set nodes and instructions.

[0116] When deserializing the format declaration to generate an abstract syntax tree, the target format of each data item will form a subtree. The field name of the root node of the subtree is the same as the name of the data item, and below the root node are the subnodes corresponding to the specific content of the target format of the data item. Each subtree has different structures according to different data types. For example, the subtree of the data item of numeric type has greater than and less than nodes, while other types of subtrees do not. For example, the subtree of the data item of enumeration type has multiple nodes, which record the name and value of the enumeration, etc.

[0117] (2) Generate a data processing function module based on the nodes of the abstract syntax tree and the mapping relationship between nodes and instructions.

[0118] Specifically, the mapping relationship between various nodes and instructions is configured in the backend management layer. According to the various nodes of the abstract syntax tree and the mapping relationship between nodes and instructions, the data processing function module can be automatically generated.

[0119] In this embodiment, according to the mapping relationship between nodes and instructions, different instructions are sequentially bound to generate data processing function modules, including: binding initialization empty array instructions, specifically including: the backend management layer will use the ANEWARRAY instruction to create an output empty array based on the number of subtrees in the intermediate tree as an operand, and the empty array is used to store attribute information after data processing. Binding static field initialization instructions, specifically including: the backend management layer will generate several static fields based on the static nodes of the intermediate tree, for example: first generate <init>Hide the function body, then allocate memory space for static variables through the NEW instruction in turn, and then call the constructors of different static variable types for initialization through the INVOKESPECIAL instruction.

[0120] The backend management layer processes each subtree of the abstract syntax tree in turn, records an integer index, and generates code for reading attribute information from the attribute information through the AALOAD instruction, where the operand for reading the attribute information is the same as the value of the integer index.

[0121] Next, the backend management layer will sequentially perform data type conversion, verification rules, processing rule instruction binding, static call user-defined method instruction binding, and write empty array instruction binding. Among them, the data type conversion is completed by statically calling the corresponding Java encapsulation type parsing function through the INVOKE_STATIC instruction. For example, the conversion of the int type is completed by generating an instruction to call the Integer.parseInt function. The verification rule is completed by generating a number of logical judgment instructions, such as LCMP, DCMPG, etc. When the verification fails, an exception message is thrown through the ATROW instruction. For the operation part of the processing rule, the corresponding code is generated through the expression engine technology, and the INVOKE_VIRTUAL instruction is used for calling; other parts of the processing rule, such as encryption and decryption, can be implemented by calling pre-set modules, such as by generating static call logic. Next, the backend management layer will generate static call logic for the user-defined extension code through the INVOKE_STATIC instruction. Finally, the backend management layer generates an instruction to write the processing result back to the generated empty array through the AASTORE instruction, where the operand of the processing result written into the empty array is the same as the value of the recorded integer index.

[0122] Determine whether all subtrees of the abstract syntax tree have been processed. When all subtrees of the abstract syntax tree have been processed in a loop, the backend management layer will generate a function signature for the data processing function module. The data processing function module can be called through this function signature and the generated data processing function module will be returned.

[0123] At present, the method of automatically generating data processing function modules is based on template generation. The generated data processing function modules are implemented in high-level programming languages. They need to be compiled twice to be machine-readable codes before they can be put online, and the publishing efficiency is low. In this embodiment, the data processing function modules are automatically generated by compilation, and they are implemented in machine code, that is, the underlying assembly code (such as Java). After the data processing function modules are automatically generated, they can be directly loaded and used without compilation, realizing hot loading of the data processing function modules.

[0124] Step S30: Convert the format of the attribute information of the data item into a target format based on the data processing function module.

[0125] Specifically, the generated data processing function module can convert the format of the attribute information of multiple data items into the target format. For example: take multiple data items as ["VIN", "VOLT", "date"], the format declaration is as shown in the above example, and the attribute information before conversion is assumed to be [LUAU2AUB3GE383467, 50000mV, 1658976485]. After using the data processing function module to convert the format of the attribute information of multiple data items into the target format, the converted attribute information is [LUAU2AUB3GE383467, 50V, 2022-07-28 02:48:05]. Among them, the data processing function module unifies the attribute information "LUAU2AUB3GE383467" corresponding to the data item "VIN" into text format. The attribute information "50000mV" corresponding to the data item "VOLT" is unified into the double-precision floating point type double, and the attribute information is divided by 1000, so that the unit mV of the data item "VOLT" is unified into V, and it is verified whether the data item "VOLT" after the unit is unified is between 0 and 100. The attribute information "1658976485" corresponding to the data item "date" is converted from the timestamp to the time format to obtain "2022-07-28 02:48:05".

[0126] The battery data processing method provided in the technical solution of the embodiment of the present application obtains the first battery data and the format declaration, compiles and generates a data processing function module based on the data item of the first battery data and the target format corresponding to the data item in the format declaration, and converts the format of the attribute information of the data item into the target format based on the data processing function module. Compared with generating a data processing function module according to the target format in the format declaration and the pre-set code template, for different battery data, there is no need for the user to pre-set different code templates. In the process of generating the data processing function module, the data processing function module can be directly generated by compiling according to the data item of the first battery data and the target format corresponding to the data item in the format declaration, without having to search for the code template corresponding to the format declaration of the battery data in many pre-set code templates, which simplifies the processing flow and improves data processing efficiency; at the same time, since there is no need to pre-set different code templates, storage space can also be saved.

[0127] In some embodiments, the first battery data and the second battery data share a format declaration, the second battery data and the first battery data include the same data items, and an arrangement order of the data items in the second battery data is different from an arrangement order of the multiple data items in the first battery data.

[0128] Assuming that the order of multiple target formats in the format declaration is "VIN", "VOLT", "date", and the order of multiple data items in the first battery data is ["VIN", "VOLT", "date"], the order of data items in the second battery data that shares the format declaration with the first battery data can be ["date", "VOLT", "VIN"], ["date", "VIN", "VOLT"], ["VOLT", "date", "VIN"], ["VOLT", "VIN", "date"], or ["VIN", "date", "VOLT"].

[0129] In the above example, since the order of data items in the second battery data is different from the order of multiple target formats in the format declaration, when generating a data processing function module based on the second battery data and the format declaration, it is necessary to adjust the arrangement order of the multiple target formats in the format declaration. After the adjustment, the arrangement order of the multiple target formats is the same as the arrangement order of the multiple data items in the second battery data. In this way, the generated data processing function module can process the attribute information in sequence according to the order of the data items in the second battery data.

[0130] Similarly, if the order of multiple data items in the first battery data is different from the order of multiple target formats in the format declaration, for example, the order of multiple target formats is "VIN", "VOLT", "date", and the order of multiple data items in the first battery data is ["VIN", "date", "VOLT"], it is also necessary to adjust the arrangement order of the multiple target formats in the format declaration so that the arrangement order of the multiple target formats after adjustment is the same as the arrangement order of the multiple data items in the first battery data. In this way, the generated data processing function module can process the attribute information in sequence according to the order of the data items in the first battery data.

[0131] In the embodiment of the present application, the second battery data and the first battery data include the same data items, the arrangement order of the data items in the second battery data is different from the arrangement order of the data items in the first battery data, and the first battery data and the second battery data share a format declaration, that is, the same format declaration can be applied to battery data with the same data items but different arrangement orders of the data items, avoiding the user from manually entering the target format corresponding to different battery data, and there is no need to obtain multiple different target formats multiple times. Afterwards, the arrangement order of multiple target formats in the format declaration is adjusted, and the arrangement order of the multiple target formats after adjustment is the same as the arrangement order of the multiple data items, so that the battery data with different arrangement orders of the data items can generate corresponding data processing function modules respectively with the same format declaration, thereby processing the attribute information of different battery data. Since in the embodiment of the present application, when processing different battery data, it is only necessary to obtain the target format once, and there is no need to obtain different target formats multiple times, the workflow is simplified, thereby improving data processing efficiency.

[0132] In one example, before compiling and generating the data processing function module based on the data items in the first battery data and the target formats corresponding to the data items in the format declaration, the method further includes: sorting the target formats in the format declaration according to the order of the data items.

[0133] Specifically, before compiling and generating a data processing function module based on the data items in the first battery data and the target formats corresponding to the data items in the format declaration, the arrangement order of multiple target formats in the format declaration is adjusted to be consistent with the arrangement order of multiple data items by matching the names of the data items in the first battery data with the field names of the target format in the format declaration, so as to ensure that the generated data processing function module can process the attribute information in sequence according to the order of the data items in the first battery data.

[0134] In another example, in the process of compiling and generating the data processing function module based on the data items in the first battery data and the target formats corresponding to the data items in the format declaration, the target formats in the format declaration are sorted according to the order of the data items.

[0135] Specifically, the multiple subtrees in the abstract syntax tree are sorted according to the arrangement order of the multiple data items in the first battery data, and the arrangement order of the multiple subtrees in the sorted abstract syntax tree is the same as the arrangement order of the multiple data items.

[0136] The subtrees of the abstract syntax tree correspond to the target format one by one, and the name of the root node of the subtree is the same as the field name of the target format. Therefore, the root nodes of each subtree of the abstract syntax tree are traversed, and the arrangement order of the root nodes of each subtree is adjusted according to the arrangement order of multiple data items in the first battery data, so that the arrangement order of the root nodes of each subtree in the adjusted abstract syntax tree is consistent with the arrangement order of multiple data items in the first battery data. In this embodiment, the arrangement order of the target format in the format declaration is changed by adjusting the arrangement order of the subtrees in the abstract syntax tree. Since when adjusting the order of the subtrees, only the position of the root node in the subtree needs to be changed, and the position of the subnodes under the root node also changes accordingly, the operation is simple and the execution efficiency is high.

[0137] In some embodiments, in the process of compiling and generating a data processing function module based on data items in the first battery data and a target format corresponding to the data items in the format declaration, an abstract syntax tree generated based on the format declaration is optimized.

[0138] In big data application scenarios, the number of calls to the data processing function module is an astronomical number. Any minor performance problem will result in significant loss of time and money. In this embodiment, during the process of compiling and generating the data processing function module, the abstract syntax tree generated based on the format declaration is optimized, and several performance optimizations are made at the machine instruction level, which makes this solution have obvious advantages in big data application scenarios.

[0139] The optimization of the abstract syntax tree is described in detail below.

[0140] In one example, optimizing the abstract syntax tree generated based on the format declaration includes: pruning the subtree in the abstract syntax tree when a data item corresponding to the subtree in the abstract syntax tree does not appear in the data item in the first battery data.

[0141] Specifically, refer to Figure 4 , the target format of a data item appears in the format declaration, but the data item does not appear in the first battery data. At this time, the data item corresponding to the subtree in the abstract syntax tree generated based on the format declaration does not appear in the first battery data. At this time, the subtree corresponding to the data item that does not appear in the first battery data will be deleted from the abstract syntax tree to avoid generating unnecessary data processing functions. By pruning the abstract syntax tree, it is possible to avoid generating unnecessary data processing functions, reduce the final amount of code generation, speed up code generation, and save memory.

[0142] It is worth noting that the embodiment of the present application does not limit the order of pruning and sorting. The abstract syntax tree may be pruned first and then the pruned abstract syntax tree may be sorted; or the abstract syntax tree may be sorted first and then the sorted abstract syntax tree may be pruned.

[0143] It should be noted that, for data items that appear in the first battery data but are not declared in the format declaration, a new empty subtree will be created.

[0144] In another example, optimizing the abstract syntax tree generated based on the format declaration includes: optimizing logical expressions of nodes of each subtree in the abstract syntax tree and / or arranging child nodes of the same type in the nodes of each subtree in the abstract syntax tree under the same parent node.

[0145] Specifically, refer to Figure 4 In this embodiment, optimizing the child nodes of each subtree of the abstract syntax tree includes at least the following three methods.

[0146] (1) Optimize the logical expressions of the child nodes of each subtree of the abstract syntax tree. For example, for a subtree that is configured with both a maximum value and a minimum value, as well as a legal value and an illegal value, the comparison of each value can be merged into the same logical expression and simplified. In one example, the comparison logic in a subtree, such as comparing with the maximum value and then comparing with the minimum value, can be merged into the following logical expression: if the comparison is greater than the maximum value, return without performing the subsequent comparison with the minimum value; if the comparison is less than the maximum value, perform the comparison with the minimum value. Another example: for a data item that is configured with both a normalization operation and a value check, the value check is logically performed after the evaluation of the normalization operation. However, if the inverse function of the normalization operation can be obtained, the value check can be advanced. In one example, a subtree has a subnode normalization operation "divide by 1000" and a subnode value check "whether it is greater than 100mV and less than 10V". The node corresponding to the value check can be placed before the node corresponding to the normalization operation, so that if the value check fails, the normalization operation does not need to be performed. In this embodiment, the amount of calculation during code execution can be reduced as much as possible by optimizing the logical expression.

[0147] (2) Arrange the subnodes of the same type in the subnodes of each subtree of the abstract syntax tree under the same parent node. For example, the same type of exceptions can be merged. In one example, the same type of data conversion nodes are arranged under the same parent node to jointly handle the same type of data conversion exceptions. By merging subnodes, the cyclomatic complexity of the final generated code can be reduced, code branches can be reduced, the branch prediction performance of the processor during execution can be improved, the number of pipeline switches can be reduced, and the instruction cache hit rate can be improved.

[0148] (3) Optimizing logical expressions of child nodes of each subtree of the abstract syntax tree, and arranging child nodes of the same type among the child nodes of each subtree of the abstract syntax tree under the same parent node.

[0149] In the embodiment of the present application, by optimizing the logical expressions of the child nodes of each subtree of the abstract syntax tree, the amount of calculation during code execution can be minimized. By arranging the child nodes of the same type in the child nodes of each subtree of the abstract syntax tree under the same parent node, the cyclomatic complexity of the code finally generated can be reduced, the code branches can be reduced, the branch prediction performance during processor execution can be improved, the number of pipeline switching times can be reduced, and the instruction cache hit rate can be improved.

[0150] In another example, refer to Figure 4 After the abstract syntax tree is generated, other optimizations can also be done. For example, statically parse the configuration parameters of the child nodes, such as converting the configuration values ​​of the legal values ​​and illegal values ​​of the time type field into timestamps in advance, and using timestamps for comparison when the code is running, instead of converting the time type fields into timestamps in sequence each time the code is run; for example, if there are loop nodes in the abstract syntax tree, expand the loop nodes in advance, and execute one instruction at a time when the code is running, instead of calculating the number of loops each time the code is run to implement the loop. The execution time of the code generated by the optimized abstract syntax tree is less than the execution time of the code generated by the abstract syntax tree before optimization.

[0151] In some embodiments, obtaining first battery data includes: obtaining initial battery data; determining a data parsing component for the initial battery data based on a data input format of the initial battery data and a correspondence between a pre-set data input format and a data parsing component; and using the data parsing component to parse data items and attribute information of corresponding data items from the initial battery data to obtain first battery data.

[0152] Specifically, the initial battery data can be divided into the following two categories according to different input methods.

[0153] One type is real-time input streaming data, which can be uploaded in real time through IoT protocols such as Message Queuing Telemetry Transport (MQTT). This type of battery data can be input in data input formats such as JavaScript Object Notation (JSON) and eXtensible Markup Language (XML). When the initial battery data is input in these data input formats, each line of initial battery data includes both data items and attribute information after the data items. For example, a line of initial battery data input in JSON format can be {"VIN": LUAU2AUB3GE383467, "VOLT": 50V, "date": 1658976485}.

[0154] The other type is one-time batch input, which can be imported in batches through the file system. This type of initial battery data can be input in data input formats such as comma-separated values ​​(CSV), tab-separated values ​​(TSV), and Excel. When the initial battery data is input in these data input formats, the first row includes all data items, and the columns store the attribute information of the corresponding data items. The initial battery data input in the form of an Excel table can be shown in Table 1 below:

[0155] Table 1

[0156]

[0157] In this embodiment, after obtaining the input initial battery data, it is necessary to first identify the data input format of the initial battery data. Realistically, the data input format of the initial battery data can be identified by identifying the file extension or by using a regular expression to match the content. For example, if the file extension is identified as ".xlsx" or ".xls", it indicates that the initial battery data is input in Excel format; if the file extension is ".csv", it indicates that the initial battery data is input in CSV format. For another example, if the method of using a regular expression to match the content identifies that the initial battery data begins with "{", it indicates that the initial battery data is streamed in json format; if the method of using a regular expression to match the content identifies that the initial battery data begins with "<", it indicates that the initial battery data is streamed in XML format.

[0158] After identifying the data input format of the initial battery data, it is necessary to determine the data parsing component according to the data input format of the battery data. The data parsing component is used to identify the data items and attribute information in the initial battery data. For example, the battery data input in CSV format can be parsed using the data parsing component openCSV, and the battery data input in JSON format can be parsed using the data parsing component jackson. The data items and attribute information in the battery data after being parsed by the data parsing component can be recognized by the machine. In this embodiment, the corresponding relationship between the data input format and the data parsing component is pre-set to find the corresponding data parsing component.

[0159] It is worth noting that for the initial battery data input in JSON, XML and other formats, each line of initial battery data includes both data items and attribute information located after the data items. When parsing using the data parsing component, the input initial battery data can be read line by line. For each line of data, read sequentially starting from the first field. The previous field is the data item, and the following one or more fields are the attribute information of the data item.

[0160] For the initial battery data input in CSV, TSV, Excel and other formats, the first row includes all the data items, and the columns store the attribute information of the corresponding data items. When parsing using the data parsing component, the input battery data can be read according to the form. For each form, the first row of data is read first, and the first row of data is all data items. After that, the data is read by column, and each column of data is the attribute information corresponding to the data item in that column.

[0161] In the embodiment of the present application, by identifying the data input format of the initial battery data, the initial battery data in different data input formats can be directly parsed into multiple data items and attribute information corresponding to the multiple data items using the corresponding data parsing components, that is, parsed into first battery data that is independent of the data input format, without having to first convert the initial battery data into a specific data input format and then parse the data items and attribute information, which can further improve data processing efficiency.

[0162] It should be noted that, when each of the multiple data items corresponds to one attribute information, the attribute information corresponding to the multiple data items can be represented by an array; when each of the multiple data items corresponds to multiple attribute information, the attribute information corresponding to the multiple data items can be represented by multiple arrays. For example, a row of initial battery data input in JSON format is such as: {"ID": 1, 2; "State of Charge": 55, 33}. After parsing, multiple data items ["ID", "State of Charge"] can be obtained, and the attribute information corresponding to the multiple data items is [1, 55] and [2, 33]. In this embodiment, the array formed by the multiple data items is called the data header, and the array formed by the attribute information is called the data body.

[0163] In some embodiments, after obtaining the format declaration, the method further includes: when updating the format declaration, generating a temporary form in the data warehouse according to the updated format declaration; and determining the data storage address of the historical form of the format declaration as the data storage address of the temporary form.

[0164] In this embodiment, when the format declaration is updated, a new temporary form is generated instead of updating the original historical form. In this way, when the format declaration is updated, the historical form can still be used, and a data processing function module can be generated based on the historical form without interruption, thereby realizing online update of the format declaration and improving data processing efficiency.

[0165] Specifically, before generating the data processing function module, the front-end interaction layer first receives the format declaration input by the user. The format declaration of the battery data can be filled in by the user on the web page, and then recognized by the machine by converting the web page content into json format; or, the user directly enters the format declaration in json format. The format declaration can be stored in the cache database in json format. The cache database is, for example, a distributed storage device (such as Redis) of a big data platform, which can be accessed by any processing node in the big data platform. At the same time, the back-end management layer will generate a form in the data warehouse (Operational Data Store, referred to as ODS) according to the format declaration. Metadata is stored on the form, and the corresponding format declaration in the cache database can be accessed through the metadata of the temporary form.

[0166] The backend management layer will automatically generate / update the tables in the Operational Data Store (ODS) based on the format declaration. Figure 5 As shown in the figure, the backend management layer will generate different target table names according to the identifier id of the format declaration. For example, if the id of the format declaration is abcd, a feasible generated target table name is `ods`.`ods_measurement_abcd`. The backend management layer will read each target format in the format declaration in turn and convert the data in the target format into metadata. Specifically, the column names of the table are first generated according to the field names in the target format, where the illegal characters in the original field names are replaced with underscores. Then, the attribute information corresponding to the field name is read, and the column attributes of the table are generated according to the attribute information. For example, the required attribute in the target format generates the nullable attribute in the table, and the field type decimal(18, 8) in the table is generated according to the field type double in the target format.

[0167] Determine whether all field names have been processed. After all field names in the target format have been processed, a temporary form is generated in the ODS data warehouse based on the generated target table name and the parsed column metadata. The table name of the temporary form is the target table name plus a random suffix, such as `ods`.`ods_measurement_abcd_12345`.

[0168] In the case where the data warehouse does not include the historical form of the format declaration, that is, in the case of adding a new ODS form, the data storage address of the temporary form points to a new blank data storage address. In the case where the data warehouse includes the historical form of the format declaration, that is, in the case of updating the ODS form, the data storage address of the temporary form points to the data storage address of the historical form.

[0169] Afterwards, the data at the data storage address is read, that is, the data stored in the cache database is read to reconstruct the partition information of the temporary form, and the corresponding data in the database can be accessed through the metadata of the temporary form.

[0170] When the partition information is rebuilt, if the data warehouse does not include the historical form of the format declaration, that is, if the ODS form is newly added, the temporary form is renamed and the suffix of the temporary form name is removed. If the data warehouse includes the historical form of the format declaration, that is, if the ODS form is updated, the metadata in the temporary form is exchanged with the metadata in the historical form, and then the exchanged temporary form is deleted.

[0171] In this embodiment, when the format declaration is updated, a new temporary form is generated instead of updating the original historical form. In this way, when the format declaration is updated, the historical form can still be used, and the data processing function module is still generated by relying on the historical form without interruption, thereby realizing online update of the format declaration and improving data processing efficiency.

[0172] In some embodiments, after obtaining the format declaration, the method further includes: verifying the legality of the format declaration according to a preset legality rule; if the verification passes, storing the input format declaration in a cache database.

[0173] Specifically, legality rules can be pre-set, and the legality of the obtained format declaration can be verified using these pre-set legality rules. Pre-set legality rules may include but are not limited to the following: 1. Pre-compile the operation of the processing rule to check the correctness of the expression; 2. Check the legality of various constant parameters input by the user, such as the correctness of the string regular expression; 3. Check the legality of the configuration of the verification rule, such as the maximum and minimum values ​​of string type data cannot be checked. If the check fails, an error is reported and the user is reminded to modify it. If the check passes, the input format declaration is stored.

[0174] In the embodiment of the present application, after obtaining the format declaration, the legality of the format declaration needs to be verified first to avoid generating an erroneous data processing function module due to an erroneous format declaration, thereby improving the accuracy of the battery data processing method.

[0175] In some embodiments, before compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration, it also includes: searching for a historical data processing function module corresponding to a historical data item identical to the data item in the first battery data according to the data items in the first battery data and a pre-set correspondence between the historical data items and the historical data processing function modules; if no historical data processing function module corresponding to a historical data item identical to the data item in the first battery data is found, then executing the step of compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration.

[0176] Specifically, the cache of this embodiment stores the historical data processing function module that has been generated, and the historical data processing function module corresponds to the data item in the first battery data. Therefore, it is possible to find out whether the historical data processing function module exists based on the multiple data items in the first battery data and the pre-set correspondence between the historical data items and the historical data processing function module.

[0177] When the historical data processing function module is found, it indicates that the format declaration corresponding to the multiple data items in the first battery data has generated a data processing function module. At this time, the historical data processing function module in the cache can be directly loaded to process the multiple data items. In the embodiment of the present application, for multiple data items that have generated a data processing function module, when the same data item is subsequently processed, there is no need to repeat the process of generating the data processing function module, which has the advantage of generating once and running multiple times, avoiding the operation of repeatedly generating the data processing function module and speeding up the data processing speed.

[0178] In one example, the cache also stores hash values ​​generated by multiple data items and format declarations in the first battery data. In this way, when searching for a historical data processing function module corresponding to the format declaration in the cache, a hash value can be first generated based on the multiple data items and format declarations in the first battery data, and the cache can be searched for the same hash value. If the same hash value exists, it can be determined that the historical data processing function module exists in the cache. In this embodiment, the historical data processing function module can be searched based on the hash value.

[0179] It is understandable that if the content in the cache is regarded as a dictionary, the hash value can be understood as the page number of the dictionary, and the content corresponding to the page number is the historical data processing function module, and multiple data items and format declarations in the first battery data that generate the hash value.

[0180] Since there may be multiple different data items and format declarations that generate the same hash value, in order to improve the cache hit rate, in some examples, after finding the same hash value in the cache, the multiple data items can be compared with the multiple data items in the cache, and the format declaration can be compared with the format declaration in the cache. If they are the same, it is determined that there is a historical data processing function module in the cache. If the multiple data items in the cache are different from the multiple data items to be processed, or the format declaration is different from the format declaration in the cache, it is determined that there is no historical data processing function module in the cache.

[0181] In some embodiments, after searching for the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data based on the correspondence between the pre-set historical data item and the historical data processing function module, it also includes: when the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data is found, converting the format of the attribute information corresponding to the data item into a target format based on the historical data processing function module.

[0182] Specifically, when the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data is not found, it indicates that the format declaration corresponding to the data item in the first battery data has not generated a data processing function module. At this time, it is necessary to generate a new data processing function module based on the data item in the first battery data and the format declaration corresponding to the data item, and use the new data processing function module to process the attribute information of the corresponding data item.

[0183] In this embodiment, for data items that have already generated data processing function modules, when subsequently processing the attribute information of the same data items, there is no need to repeat the process of generating the data processing function modules. The historical data processing function modules can be directly used for processing. This has the advantage of generating once and running multiple times, avoiding the operation of repeatedly generating data processing function modules and speeding up data processing.

[0184] In some embodiments, the data processing function module is implemented in the form of Java bytecode. In the embodiments of the present application, the data processing function module finally generated is in the form of Java bytecode, which can be directly executed by the Java virtual machine of the data processing node. Compared with generating a high-level programming language, there is no need to compile again, and no need to stop the machine for loading, which speeds up the data processing speed.

[0185] The data processing method in the embodiment of the present application is described below by taking a group of battery data processing process as an example. Figure 6 shown.

[0186] Assume that the input battery data is {"VOLT": 200000mV; "data": 1404958872}. First identify the data input format of the battery data, such as using regular expression content matching to identify the data input format of the battery data. It can be identified that the string at the beginning of the battery data is "{", indicating that the battery data is streamed in json format. After that, use the data parsing tool you want to listen to, parse the initial battery data to obtain multiple data items and attribute information corresponding to the multiple data items. For example, the data parsing component jackson can be used for parsing, and the input battery data can be parsed to include two data items "VOLT" and "data", and two attribute information 200000mV and 1404958872.

[0187] In this embodiment, the format declaration input by the user is also obtained, and the legality of the format declaration is checked. After the check passes, the format declaration can be temporarily stored in the cache, and an ODS form is generated in the data warehouse according to the format declaration.

[0188] In this example, it is assumed that the format declaration is as follows:

[0189]

[0190]

[0191] The format declaration includes two target formats. The first target format corresponds to the data item "data":

[0192]

[0193]

[0194] The second target format corresponds to the data item "VOLT":

[0195]

[0196] If the historical data processing function module corresponding to the multiple data items ["VOLT", "data"] is found, the historical data processing code in the cache can be directly loaded to process the attribute information corresponding to the multiple data items. If the historical data processing code is not found, it indicates that the format declaration corresponding to the multiple data items has not generated data processing code. At this time, it is necessary to automatically generate new data processing code through compilation according to the multiple data items and format declarations, and use the new data processing code to process the attribute information corresponding to the multiple data items, wherein the generated data processing code is the underlying machine code, such as JAVA bytecode.

[0197] Specifically, first, deserialize according to the format declaration to obtain an abstract syntax tree, and the target format of each field will form a subtree, and each subtree has a different structure according to different data types. After the abstract syntax tree is generated, the abstract syntax tree is optimized to obtain an intermediate tree. The optimization method includes but is not limited to: pruning the subtree in the abstract syntax tree when the data item corresponding to the subtree does not appear in the multiple data items; sorting the subtrees corresponding to the data items in the abstract syntax tree according to the order of the data items in the multiple data items; optimizing the logical expressions of the child nodes of each subtree of the abstract syntax tree; and arranging the child nodes of the same type in the child nodes of each subtree of the abstract syntax tree under the same parent node. Afterwards, generate data processing code according to the mapping relationship between each node and the instruction of the intermediate tree. Finally, use the data processing code to process the attribute information [200000mV, 1404958872] corresponding to the multiple data items to obtain the new array [200V, 2014-07-10 10:21:12] after processing.

[0198] It should be noted that if the current batch battery data has not been processed, or is stream processing, the step of parsing the data input format will be returned to the loop. After buffering the return value of the attribute information that has been processed, the system management layer will store the processed attribute information, and an ODS form can be generated in the data warehouse to store the processed attribute information. After the offline data import is completed, or after a certain period of time has passed since the online data import, the system management layer will automatically rebuild the ODS table metadata.

[0199] In this embodiment, the data processing function module is compiled and generated, and the data processing function module is output in the form of underlying assembly code (such as JAVA bytecode). On the one hand, it can be directly loaded and used without compilation, thus realizing hot loading of the code; on the other hand, several performance optimizations can be made at the machine instruction level, which has obvious advantages in big data application scenarios.

[0200] To achieve the above purpose, the present invention also provides a battery data processing device, such as Figure 7 As shown, the battery data processing device includes:

[0201] The acquisition module 10 is used to acquire first battery data and a format declaration, wherein the first battery data includes data items and attribute information, and the format declaration includes a target format corresponding to the data item.

[0202] The processing module 20 is used to compile and generate a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration; and convert the format of the attribute information of the data items into the target format based on the data processing function module.

[0203] The battery data processing device provided in this embodiment obtains the first battery data and the format declaration, compiles and generates a data processing function module based on the data items of the first battery data and the target format corresponding to the data items in the format declaration, and converts the format of the attribute information of the data items into the target format based on the data processing function module. Compared with generating a data processing function module according to the target format in the format declaration and the pre-set code template, for different battery data, there is no need for the user to pre-set different code templates. In the process of generating the data processing function module, the data processing function module can be directly generated by compiling according to the data items of the first battery data and the target format corresponding to the data items in the format declaration, without having to search for the code template corresponding to the format declaration of the battery data in many pre-set code templates, which simplifies the processing flow and improves data processing efficiency; at the same time, since there is no need to pre-set different code templates, storage space can also be saved.

[0204] In some embodiments, the processing module 20 is used to sort the target formats in the format declaration according to the order of the data items before compiling and generating a data processing function module based on the data items in the first battery data and the target formats corresponding to the data items in the format declaration.

[0205] In this embodiment, the arrangement order of multiple target formats in the format declaration is adjusted so that battery data with different data item arrangement orders can generate corresponding data processing function modules with the same format declaration, thereby processing the attribute information of different battery data. Since in the embodiment of the present application, when processing different battery data, it is only necessary to obtain the target format once, and there is no need to obtain different target formats multiple times, the workflow is simplified, thereby improving data processing efficiency.

[0206] In some embodiments, the processing module 20 is used to sort the target formats in the format declaration according to the order of the data items in the first battery data and the target formats corresponding to the data items in the format declaration during the process of compiling and generating a data processing function module.

[0207] In this embodiment, the order of arrangement of the target format in the format declaration is changed by adjusting the order of arrangement of the subtrees in the abstract syntax tree. When adjusting the order of the subtrees, only the position of the root node in the subtree needs to be changed, and the positions of the subnodes under the root node also change accordingly. The operation is simple and the execution efficiency is high.

[0208] In some embodiments, the processing module 20 is used to optimize the abstract syntax tree generated based on the format declaration during the process of compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration.

[0209] In the process of compiling and generating the data processing function module in this embodiment, the abstract syntax tree generated based on the format declaration is optimized, and several performance optimizations are made at the machine instruction level, so that this solution has obvious advantages in big data application scenarios.

[0210] In some embodiments, the processing module 20 is configured to prune the subtree in the abstract syntax tree when a data item corresponding to the subtree in the abstract syntax tree does not appear in the data items in the first battery data.

[0211] In this embodiment, pruning the subtree of the abstract syntax tree can avoid generating redundant data processing functions, reduce the amount of code corresponding to the final data processing function module, speed up the generation of the data processing function module and save memory.

[0212] In some embodiments, the processing module 20 is used to optimize the logical expressions of the nodes of each subtree in the abstract syntax tree and / or arrange the child nodes of the same type in the nodes of each subtree in the abstract syntax tree under the same parent node.

[0213] In the embodiment of the present application, by optimizing the logical expressions of the child nodes of each subtree of the abstract syntax tree, the amount of calculation during code execution can be minimized. By arranging the child nodes of the same type in the child nodes of each subtree of the abstract syntax tree under the same parent node, the cyclomatic complexity of the code finally generated can be reduced, the code branches can be reduced, the branch prediction performance during processor execution can be improved, the number of pipeline switching times can be reduced, and the instruction cache hit rate can be improved.

[0214] In some embodiments, the acquisition module 10 is used to acquire initial battery data;

[0215] Determining a data parsing component for the initial battery data according to a data input format of the initial battery data and a correspondence between a preset data input format and a data parsing component;

[0216] The data parsing component is used to parse the data item and the attribute information corresponding to the data item from the initial battery data to obtain the first battery data.

[0217] In this embodiment, by identifying the data input format of the initial battery data, the initial battery data in different data input formats can be directly parsed into multiple data items and attribute information corresponding to the multiple data items using the corresponding data parsing components, that is, parsed into first battery data that is independent of the data input format, without having to first convert the initial battery data into a specific data input format and then parse the data items and attribute information, which can further improve data processing efficiency.

[0218] In some embodiments, the processing module 20 is used to generate a temporary form in the data warehouse according to the updated format declaration when the format declaration is updated after obtaining the format declaration;

[0219] The data storage address of the historical form of the format declaration is determined as the data storage address of the temporary form.

[0220] In this embodiment, when the format declaration is updated, a new temporary form is generated instead of updating the original historical form. In this way, when the format declaration is updated, the historical form can still be used, and the data processing function module is still generated by relying on the historical form without interruption, thereby realizing online update of the format declaration and improving data processing efficiency.

[0221] In some embodiments, the processing module 20 is used to, after obtaining the format declaration,

[0222] Verifying the legality of the format declaration according to pre-set legality rules;

[0223] When the verification is passed, the input format declaration is stored in the cache database.

[0224] In this embodiment, after obtaining the format declaration, the legality of the format declaration needs to be verified first to avoid generating an erroneous data processing function module due to an illegal format declaration, thereby improving the accuracy of data processing.

[0225] In some embodiments, the processing module 20, before compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration, is further configured to:

[0226] According to the data items in the first battery data and the preset correspondence between the historical data items and the historical data processing function modules, searching for the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data;

[0227] If the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data is not found, the step of compiling and generating the data processing function module based on the data item in the first battery data and the target format corresponding to the data item in the format declaration is performed again.

[0228] In some embodiments, the processing module 20, after searching for the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data according to the data item in the first battery data and the preset correspondence between the historical data item and the historical data processing function module, is further configured to:

[0229] When a history data processing function module corresponding to the history data item identical to the data item in the first battery data is found, the format of the attribute information corresponding to the data item is converted into the target format based on the history data processing function module.

[0230] In this embodiment, for data items that have already generated data processing function modules, when subsequently processing the attribute information of the same data items, there is no need to repeat the process of generating the data processing function modules. The historical data processing function modules can be directly used for processing. This has the advantage of generating once and running multiple times, avoiding the operation of repeatedly generating data processing function modules and speeding up data processing.

[0231] In some embodiments, the data processing function module is implemented in Java bytecode form.

[0232] The data processing function module finally generated in the embodiment of the present application is in the form of Java bytecode. The Java bytecode can be directly executed by the Java virtual machine of the data processing node. Compared with generating high-level programming languages, it does not need to be compiled again and can be directly integrated with common big data processing software without stopping for loading, thereby speeding up data processing.

[0233] To achieve the above objective, an embodiment of the present invention further provides a storage medium, on which a program is stored, and when the program is executed by a processor, the battery data processing method as described above is implemented.

[0234] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.< / init>

Claims

1. A battery data processing method, It is characterized in that include: Acquire first battery data and a format declaration, wherein the first battery data includes data items and attribute information, and the format declaration includes a target format corresponding to the data item; Compile and generate a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration; The format of the attribute information of the data item is converted into the target format based on the data processing function module.

2. The battery data processing method according to claim 1, It is characterized in that Before compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration, the method further includes: The target formats in the format declaration are sorted according to the order of the data items.

3. The battery data processing method according to claim 1, It is characterized in that In the process of compiling and generating a data processing function module based on the data items in the first battery data and the target formats corresponding to the data items in the format declaration, the target formats in the format declaration are sorted according to the order of the data items.

4. The battery data processing method according to any one of claims 1 to 3, It is characterized in that In the process of compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration, an abstract syntax tree generated based on the format declaration is optimized.

5. The battery data processing method according to claim 4, It is characterized in that The abstract syntax tree generated based on the format declaration is optimized, including: When a data item corresponding to a subtree in the abstract syntax tree does not appear in a data item in the first battery data, the subtree is pruned.

6. The battery data processing method according to claim 4, It is characterized in that The optimizing the abstract syntax tree generated based on the format declaration includes: Optimize logical expressions of nodes of each subtree in the abstract syntax tree and / or arrange child nodes of the same type in the nodes of each subtree in the abstract syntax tree under the same parent node.

7. The battery data processing method according to claim 1, It is characterized in that The obtaining of the first battery data includes: Get initial battery data; Determining a data parsing component for the initial battery data according to a data input format of the initial battery data and a correspondence between a preset data input format and a data parsing component; The data parsing component is used to parse the data item and the attribute information corresponding to the data item from the initial battery data to obtain the first battery data.

8. The battery data processing method according to claim 1, It is characterized in that After obtaining the format declaration, it also includes: When the format declaration is updated, a temporary form is generated in the data warehouse according to the updated format declaration; The data storage address of the historical form of the format declaration is determined as the data storage address of the temporary form.

9. The battery data processing method according to claim 1, It is characterized in that After obtaining the format declaration, it also includes: Verifying the legality of the format declaration according to pre-set legality rules; When the verification is passed, the input format declaration is stored in the cache database.

10. The battery data processing method according to any one of claims 1 to 9, It is characterized in that Before compiling and generating a data processing function module based on the data items in the first battery data and the target format corresponding to the data items in the format declaration, the method further includes: According to the data items in the first battery data and the preset correspondence between the historical data items and the historical data processing function modules, searching for the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data; If the historical data processing function module corresponding to the historical data item identical to the data item in the first battery data is not found, the step of compiling and generating the data processing function module based on the data item in the first battery data and the target format corresponding to the data item in the format declaration is performed again.

11. The battery data processing method according to claim 10, It is characterized in that After searching for a historical data processing function module corresponding to a historical data item identical to a data item in the first battery data according to the data item in the first battery data and the preset correspondence between the historical data item and the historical data processing function module, the method further includes: When a history data processing function module corresponding to the history data item identical to the data item in the first battery data is found, the format of the attribute information corresponding to the data item is converted into the target format based on the history data processing function module.

12. The battery data processing method according to any one of claims 1 to 11, It is characterized in that The data processing function module is implemented in the form of Java bytecode.

13. A battery data processing device, It is characterized in that include: an acquisition module, configured to acquire first battery data and a format declaration, wherein the first battery data includes data items and attribute information, and the format declaration includes a target format corresponding to the data items; A processing module, used for compiling and generating a data processing function module based on a data item in the first battery data and a target format corresponding to the data item in the format declaration; The format of the attribute information of the data item is converted into the target format based on the data processing function module.

14. A battery data processing platform, It is characterized in that The battery data processing method according to any one of claims 1 to 12 is executed on the battery data processing platform.

15. A storage medium, It is characterized in that The storage medium stores a battery data processing instruction, and when the battery data processing instruction is executed, the battery data processing method as described in any one of claims 1 to 12 is implemented.

Images