Data acquisition method and device, equipment and storage medium

Abstract

The invention discloses a data acquisition method and device, equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the steps of obtaining to-be-collected battery data; calling a corresponding analytical model according to the battery data; the battery data is analyzed through an analysis rule in the analysis model to obtain a standardized data structure, and the battery data of different sources and different formats can be analyzed through the analysis model after data collection is completed, so that the standardized battery data structure is obtained, and the battery data of the uniform data format is collected; and the data acquisition efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data acquisition method, device, equipment and storage medium. Background Art

[0002] Before analyzing the battery, a large amount of battery data needs to be collected to provide a data source for the analysis. Currently, when collecting battery data, the difference in data leads to low data collection efficiency. Summary of the invention

[0003] The embodiments of the present invention provide a data collection method, device, equipment and storage medium, aiming to solve the technical problem of low data collection efficiency.

[0004] In a first aspect, an embodiment of the present application provides a data collection method, the data collection method comprising:

[0005] Obtain battery data to be collected;

[0006] Retrieving a corresponding analytical model according to the battery data;

[0007] The battery data is parsed using the parsing rules in the parsing model to obtain a standardized data structure, thereby completing data collection.

[0008] In this embodiment, the battery data to be collected is obtained, and the corresponding parsing model is called according to the battery data, so that the battery data is parsed according to the parsing rules in the parsing model to obtain a standardized data structure, and the data collection is completed. The parsing model can be used to parse battery data from different sources and in different formats to obtain a standardized battery data structure, collect battery data with a unified data format, and improve the efficiency of data collection.

[0009] In some embodiments, the retrieving a corresponding analytical model according to the battery data includes:

[0010] Analyzing the battery data to determine the battery type of the battery data;

[0011] The corresponding analytical model is retrieved according to the battery type.

[0012] In the technical solution of the embodiment of the present application, the battery type of the battery data can be determined by analyzing the required battery, and the analytical model that can be used to process this battery type can be quickly queried based on the battery type, which can be flexibly applied to various types of battery data.

[0013] In some embodiments, the retrieving a corresponding analytical model according to the battery type includes:

[0014] determining a data format of the battery data according to the battery type;

[0015] Querying a parsing model corresponding to the data format according to the data format;

[0016] Retrieve the parsing model corresponding to the data format.

[0017] In the technical solution of the embodiment of the present application, the data formats of battery data corresponding to different battery types are different. Therefore, the parsing model that can process this data format can be queried through the data format. When there is a parsing model that can process the battery data in the data format, the parsing model corresponding to the data format is called up. The corresponding parsing model can be quickly determined according to the data format, which facilitates the rapid matching of different types of battery data with the corresponding parsing models.

[0018] In some embodiments, the method further comprises:

[0019] If no parsing model corresponding to the data format is found, updating the parsing model to obtain an updated parsing model;

[0020] Obtaining update parsing rules of the update parsing model;

[0021] The update parsing model and the update parsing rules are stored in a preset cache library.

[0022] When the parsing model corresponding to the data format is not queried, this embodiment updates the parsing model to obtain an updated parsing model, configures the updated parsing rules in the updated parsing model, and stores the updated parsing model and the updated parsing rules in a preset cache library, so that the updated parsing model and the updated parsing rules can be directly retrieved according to the preset cache library, thereby improving the efficiency of parsing.

[0023] In some embodiments, after retrieving the corresponding analytical model according to the battery data, the method further includes:

[0024] Acquire the update parsing rule from the preset cache library;

[0025] Initialize and load the update parsing rules.

[0026] In the technical solution of the embodiment of the present application, before parsing, the updated parsing rules are initialized and loaded to ensure that the changed rules are loaded and executed, thereby ensuring that the updated parsing rules take effect on time.

[0027] In some embodiments, before calling the corresponding analytical model according to the battery data, the method includes:

[0028] Collect data of different types of battery samples;

[0029] Parsing the battery sample data to determine the battery data format of the battery sample data;

[0030] The corresponding parsing model is configured through the battery data format.

[0031] This embodiment obtains battery sample data under different operating conditions of the battery by collecting massive data in advance, and determines the data format corresponding to the battery sample data by parsing the battery sample data, and configures the corresponding data format with a parsing model that can parse this format, so as to flexibly handle the needs of data parsing in various scenarios. Through the unified design and definition of the parsing model, the accessed battery data can be uniformly processed through the parsing model during data collection, thereby improving the processing efficiency.

[0032] In some embodiments, before configuring the corresponding parsing model through the battery data format, the method further includes:

[0033] Determining the attributes of the battery data by using the battery data format;

[0034] Configure the corresponding parsing rules through the properties;

[0035] A parsing model is constructed according to the parsing rules.

[0036] This embodiment determines the attributes of the battery data through the battery data format; configures corresponding parsing rules through the attributes; and builds a parsing model through the parsing rules. Since the parsing model includes different parsing rules, different parsing rules can be used to parse battery data with different attributes. Therefore, corresponding parsing rules can be configured for the battery data according to the attributes of the battery data, thereby improving the efficiency of battery data parsing.

[0037] In some embodiments, parsing the battery data using the parsing rules in the parsing model to obtain a standardized data structure and complete data collection includes:

[0038] determining an attribute of the battery data according to the battery data;

[0039] querying a corresponding parsing rule from the parsing model based on the attribute;

[0040] The battery data is parsed according to the parsing rules to obtain a standardized data structure, thereby completing data collection.

[0041] This embodiment determines the attributes of the accessed battery data, and then queries the corresponding parsing rules from the parsing model through the relationship between the attributes and the parsing rules, quickly locates the corresponding parsing rules through the attributes, thereby improving the parsing efficiency, parses the battery data through the parsing rules, and standardizes the battery data to obtain a standardized data structure, completes data collection, and improves the efficiency of the subsequent data processing process of battery access.

[0042] In some embodiments, after retrieving the corresponding analytical model according to the battery data, the method further includes:

[0043] Obtaining parsing rules in the parsing model;

[0044] Initialize and load the parsing rules.

[0045] This embodiment can ensure that all parsing rules take effect in a timely manner and improve the efficiency of parsing by initializing and loading the parsing rules in the parsing model.

[0046] In some embodiments, the method further comprises:

[0047] Verifying the battery data according to the attributes of the battery data to obtain a verification result;

[0048] When the verification result is that the verification is passed, the step of parsing the battery data according to the parsing rules in the parsing model is executed to obtain a standardized data structure, thereby completing the step of data collection.

[0049] In the technical solution of the embodiment of the present application, before parsing the battery data, in order to avoid incorrect battery data input leading to parsing errors or abnormalities, the battery data can be verified, thereby improving the accuracy of the accessed battery data and improving the fault tolerance of the analysis.

[0050] In some embodiments, the method further comprises:

[0051] When the verification result is that the verification fails, generating a reminder message;

[0052] The battery data format query reminder is performed through the reminder information.

[0053] In the technical solution of the embodiment of the present application, when the verification result is verification failure, it means that the connected battery data is incompatible or there is an error, and a reminder message can be generated to remind the staff to query and correct the battery data format.

[0054] In some embodiments, verifying the battery data according to the attributes of the battery data to obtain a verification result includes:

[0055] Retrieving corresponding verification rules according to the attributes of the battery data;

[0056] The battery data is verified using the verification rule to obtain a verification result.

[0057] In the technical solution of the embodiment of the present application, the verification rules corresponding to the attributes can be called through the attributes of the battery data. The verification rules can verify the attributes of the battery data to determine whether there are incompatibilities or errors in the battery data. The battery data can be verified in advance through the verification rules to speed up the subsequent data analysis time.

[0058] In some embodiments, before calling the corresponding verification rule according to the attribute of the battery data, the method further includes:

[0059] Collect attribute sample data of battery data;

[0060] Determining a verification process of the attribute sample data by parsing the attribute sample data;

[0061] Verification rules are constructed based on the verification process.

[0062] This embodiment collects attribute sample data of battery data in advance, parses the attribute sample data, determines the verification process of the attribute sample data, and then constructs verification rules through the verification process, and statistically summarizes different verification rules. When verifying the battery data, by calling the corresponding verification rules, the needs of data verification in various scenarios can be flexibly handled.

[0063] In some embodiments, parsing the battery data using the parsing rules in the parsing model to obtain a standardized data structure and complete data collection includes:

[0064] Parsing the battery data using the parsing rules in the parsing model to obtain keyword data of the battery data;

[0065] The keyword data of the battery data is used as a standardized data structure to complete data collection.

[0066] When parsing battery data using parsing rules, this embodiment extracts keywords from the battery data to obtain keyword data of the battery data, and uses the extracted keyword data as a standardized data structure. By standardizing the battery data, the effect of battery data collection is improved.

[0067] In some embodiments, the obtaining of the battery data to be collected includes:

[0068] Obtaining a collection agreement for the data to be collected;

[0069] Determining a collection interface based on the collection protocol;

[0070] The battery data to be collected is acquired through the collection interface.

[0071] This embodiment obtains the collection protocol of the data to be collected, so that the collection interface during collection is determined according to the collection protocol, and the battery data to be collected of the corresponding collection protocol can be accessed through the collection interface, thereby improving the efficiency of data access.

[0072] In a second aspect, an embodiment of the present invention further provides a data acquisition system, the data acquisition system comprising: a data access gateway module, a parsing module, a user management module, a cache module and a parsing module, the data access gateway module is connected to the user management module, the cache module and the parsing module respectively, and the parsing module stores a parsing model;

[0073] The user management module is used to obtain user information and send the user information to the data access gateway module;

[0074] The data access gateway module is used to load the corresponding acquisition protocol according to the user information, acquire the corresponding battery data to be acquired through the acquisition protocol, retrieve the corresponding analysis model from the analysis module based on the battery data to be acquired, and analyze the battery data to be acquired according to the analysis model to obtain a standardized data structure;

[0075] The cache module is used to store standardized data structures.

[0076] In a third aspect, an embodiment of the present invention further proposes a data acquisition device, comprising: a memory, a processor, and a data acquisition program stored in the memory and executable on the processor, wherein the data acquisition program is configured to implement the data acquisition method described above.

[0077] In a fourth aspect, an embodiment of the present invention further provides a storage medium, on which a data acquisition program is stored, and when the data acquisition program is executed by a processor, the data acquisition method as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0079] Figure 1 A schematic diagram of the structure of a data acquisition device in a hardware operating environment according to an embodiment of the present application;

[0080] Figure 2 A schematic diagram of the structure of a big data platform proposed in an embodiment of the present invention;

[0081] Figure 3 A schematic diagram of a flow chart of an embodiment of a data collection method proposed in an embodiment of the present invention;

[0082] Figure 4 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0083] Figure 5 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0084] Figure 6 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0085] Figure 7 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0086] Figure 8 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0087] Fig. 9 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0088] Fig.10 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0089] Fig.11 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0090] Fig.12 This is another flow chart of an embodiment of the data collection method provided by the embodiment of the present invention;

[0091] Fig.13 A schematic diagram of the structure of a data acquisition system proposed in an embodiment of the present invention.

[0092] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0093] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0094] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" in the specification and claims of the present invention and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0095] In the description of the embodiments of the present invention, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present invention, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0096] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0097] In the description of the embodiments of the present invention, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0098] In the description of the embodiments of the present invention, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0099] In the description of the embodiments of the present invention, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the embodiments of the present invention.

[0100] In the description of the embodiments of the present invention, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0101] Currently, when testing power batteries, massive amounts of battery data in different scenarios can be collected, and the collected data can be transmitted to the subsequent processing layer to achieve testing of the power batteries.

[0102] However, there are certain differences in battery data in different scenarios. For example, the data formats provided by various data providers are different. Therefore, it is necessary to process the data in a unified manner to ensure the consistency of the collected data. If you want to process the collected data in various formats, the usual processing method is to use hard code to parse. For example, for the access scenario of business A, develop a collection program for A, write logic directly in the code to parse and judge, parse each field in turn, and generate our unified format data; add a new business B access scenario, and develop a new B collection service. Similarly, do customized hard-coded parsing. For each new business scenario, you need to repeatedly develop different collection services, which has low collection efficiency, and the collection service is only for the corresponding business data and cannot be reused, and the overall R&D cost is high. When the data format changes, the collection program needs to be remodeled and released, and the design is not flexible enough.

[0103] In order to solve the above-mentioned defects, the inventive concept of the present application is as follows:

[0104] By dynamically accessing different types of battery data and analyzing the battery data, the corresponding format of the battery data is determined. Therefore, a parsing model that can parse the current battery data format can be configured from a pre-established parsing database according to the format of the battery data. The battery data can be parsed by configuring corresponding parsing rules through the parsing model. Corresponding parsing rules can be configured for different types of battery data, thereby building a standardized battery data structure, improving the efficiency of battery collection, and also improving the efficiency of subsequent battery processing.

[0105] Based on the above considerations, in order to solve the problem of low battery data collection efficiency, after in-depth research, a data collection method, device, equipment and storage medium are proposed, which are suitable for various scenarios of data collection during the actual use of the battery, for example, in the specific application scenario of safety monitoring of batteries of equipment such as electric vehicles, ships or aircraft during driving or charging.

[0106] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the structure of a battery internal structure display device in the hardware operating environment involved in an embodiment of the present application.

[0107] like Figure 1 As shown, the battery internal structure display device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0108] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the device for displaying the internal structure of the battery, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0109] like Figure 1As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a battery internal structure display program.

[0110] exist Figure 1 In the battery internal structure display device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the battery internal structure display device of the present invention can be set in the battery internal structure display device, and the battery internal structure display device calls the battery internal structure display program stored in the memory 1005 through the processor 1001, and executes the battery internal structure display method provided in the embodiment of the present invention.

[0111] The data collection method provided in the embodiment of the present application may be executed by a data collection device, which may be used to collect a large amount of battery data. The data collection device may be a data collector, which may be embedded in the data collection layer of a big data cloud platform, and the big data cloud platform may be a battery early warning platform. Figure 2 The battery warning platform shown serves as a specific application environment of the data collection method provided in this embodiment. The battery warning platform is only a possible example, and does not limit the specific application environment of the data collection method provided in the embodiment of the present application. The data collection method provided in the embodiment of the present application can also be applied to other big data platforms.

[0112] like Figure 2 As shown, Figure 2 The schematic diagram of the structure of the big data cloud platform shows that the battery early warning platform also includes a processing layer. The data collection layer is connected to the processing layer. The data collected by the data collection layer is transmitted to the processing layer. The processing layer processes the received collected data to obtain processed data. The data collection layer includes a data access gateway module. A data collector is set in the data access gateway module. The massive battery data is collected through the data collector. The data collection layer is also configured with a variety of analytical models. The multiple analytical models can be located in the collector or outside the collector. When the multiple analytical models are located in the collector, the collector obtains the battery data to be collected. The multiple analytical models directly analyze the collected data, thereby transmitting the analyzed data to the processing layer. When the analytical model is located outside the collector, the battery data is connected to the data access gateway module. The analytical model first analyzes the data to obtain a standardized data structure, and then collects it to the processing layer through the collector. Thus, the accessed battery data is analyzed through multiple analytical models to obtain a standardized data structure, thereby improving the efficiency of the subsequent processing layer in processing data.

[0113] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0114] In some embodiments of the present invention, reference Figure 3 , an embodiment of the present invention proposes a data collection method, which may include:

[0115] Step S10: Obtain the battery data to be collected.

[0116] It should be understood that when the acquisition layer is needed for data acquisition or the user uploads data to the acquisition layer, the battery data to be collected can be obtained first. The battery data can be different types of battery data. The battery data can be different battery operating conditions and different battery configuration data and battery measurement data. The battery operating conditions may include battery charging and discharging conditions or other conditions. The corresponding battery operating conditions can be obtained and the battery data representing the battery operating conditions can be accessed. The battery data may include battery configuration data, battery voltage and current data, etc., and may also include other data, which is not limited in this embodiment. The data source of the battery data to be collected may include upstream and downstream related parties, from battery raw materials, battery equipment manufacturers, electric vehicle manufacturers, energy storage operators, charging pile equipment parties, operators, electric vehicle operators, users, battery recycling companies, etc., by accessing the configuration data, measurement data, event data and other related data representing the battery operating conditions. The battery configuration data includes the vehicle series, model, manufacturer, vehicle battery type and vehicle battery model, etc., and may also include other configuration data. The measurement data may include current data, voltage data, etc. of the battery under various conditions. Time data includes the start and end of charging, etc. The collection layer of the cloud platform can collect data from various upstream and downstream links of the battery. The data comes from various third parties related to the battery. Compared with a single terminal obtaining a single data, the data collected by the big data cloud platform is more comprehensive and covers a wider range.

[0117] Step S20: Retrieve the corresponding analytical model according to the battery data.

[0118] It should be noted that the battery data to be collected can be analyzed to determine the type or other features of the battery data, and then the analytical model corresponding to the features of the battery data can be retrieved through the features of the battery data. The analytical model is a model for standardizing the battery data. Standardization processing may include keyword extraction, useless data removal, and data normalization of the battery data, thereby forming a standardized data structure, which is convenient for the subsequent processing layer to directly process the standardized data structure and improve the processing efficiency.

[0119] The analytical model can be stored in the analytical database or in other locations. This embodiment does not limit this. A variety of different analytical models are configured in the analytical database. Different analytical models can parse battery data with different characteristics. The analytical model can be constructed by analyzing different types of battery sample data collected in advance. By summarizing multiple analytical models, a analytical database is constructed.

[0120] Step S30: parse the battery data using the parsing rules in the parsing model to obtain a standardized data structure, thereby completing data collection.

[0121] It should be understood that parsing the battery data may include extracting keywords from the battery data, graying the image data, eliminating useless data, normalizing the data, etc., and may also include other parsing processes, which are not limited in this embodiment.

[0122] The battery data can be parsed by configuring in advance, without the need to develop code every time, regardless of whether the battery technology format is adding fields, deleting fields, or changing fields. The collection side service code side does not need to be modified, and can flexibly adapt to various data change scenarios. It has good scalability, greatly reduces the frequent modification and release of the code side, further reduces the risk of anomalies or errors, and further improves the stability of the collection service.

[0123] In this embodiment, the battery data to be collected is obtained, and the corresponding parsing model is called according to the battery data, so that the battery data is parsed according to the parsing rules in the parsing model to obtain a standardized data structure, and the data collection is completed. The parsing model can be used to parse battery data from different sources and in different formats to obtain a standardized battery data structure, collect battery data with a unified data format, and improve the efficiency of data collection.

[0124] In some embodiments, due to different types of battery data, the corresponding analytical model can be retrieved to determine the different types of battery data. Figure 4 , step S20 specifically includes:

[0125] Step S201: Analyze the battery data to determine the battery type of the battery data.

[0126] It should be noted that the battery data to be processed can be analyzed to determine how many types of batteries there are in the battery data to be processed, and the battery type of the battery data can be obtained. For example, the battery types include oxide batteries, lithium-ion batteries, sodium-ion batteries, etc.

[0127] As an example, the battery type of the battery data may also be directly acquired, for example, the accessed battery data may carry various battery types.

[0128] Step S202: Retrieve the corresponding analytical model according to the battery type.

[0129] It is understandable that since different types of battery data correspond to different parsing models, after obtaining the battery type, the parsing model that can parse the current type of battery data can be retrieved through the battery type of the connected battery data.

[0130] In the technical solution of the embodiment of the present application, the battery type of the battery data can be determined by analyzing the required battery, and the analytical model that can be used to process this battery type can be quickly queried based on the battery type, which can be flexibly applied to various types of battery data.

[0131] In some embodiments, due to the different battery types of battery data, the parsing model called is also different. Figure 5 As shown, the step of S202 specifically includes:

[0132] Step S2021: Determine the data format of the battery data according to the battery type.

[0133] It should be noted that since the battery data to be accessed is provided by different battery storage parties, different provision methods may result in different formats of statistical battery data, so the data format of the battery data to be accessed can be determined according to the battery type, for example, the data format is json format, xml format, etc.

[0134] Step S2022: query the parsing model corresponding to the data format according to the data format.

[0135] In a specific implementation, the parsing model corresponding to the data format can be queried through the data format, and the corresponding relationship between the parsing model and the data format can be established in advance, so as to directly query whether there is a parsing model corresponding to the data format through the data format. If the parsing model is stored in a parsing database, it can be queried in the parsing database whether there is a corresponding parsing model.

[0136] For example, if the data format is json, you can configure parsing model 1 in advance to parse battery data in json format. If the data format is xml, you can configure parsing model 2 in advance to parse battery data in xml format.

[0137] Step S2023: Retrieve the parsing model corresponding to the data format.

[0138] It should be noted that when a corresponding parsing model is found according to a data format, the parsing model corresponding to the data format can be directly retrieved, thereby loading the parsing rules in the parsing model to parse the battery data.

[0139] Optionally, after the battery data is connected, in order to facilitate the subsequent rapid analysis of the battery data, the analysis database can be started, and all analysis models and all analysis rules in all analysis models can be initialized to ensure that all analysis rules can take effect in time. When the analysis model or analysis rule changes, the analysis model or analysis rule can be stored in the secondary cache first, and the corresponding analysis model and analysis rule can be directly retrieved from the cache. After the analysis is completed, the updated analysis model or analysis rule can be updated to the analysis database. The dynamic configuration and expansion of the analysis database is realized.

[0140] In the technical solution of the embodiment of the present application, the data formats of battery data corresponding to different battery types are different. Therefore, the parsing model that can process this data format can be queried through the data format. When there is a parsing model that can process the battery data in the data format, the parsing model corresponding to the data format is called up. The corresponding parsing model can be quickly determined according to the data format, which facilitates the rapid matching of different types of battery data with the corresponding parsing models.

[0141] In some embodiments, there may be a situation where the parsing model is not compatible with the data format and the model needs to be updated. Therefore, after step S2022, the following steps are also included:

[0142] When no parsing model corresponding to the data format is found, the parsing model is updated to obtain an updated parsing model;

[0143] Obtaining update parsing rules of the update parsing model;

[0144] The updated parsing model and the updated parsing rules are stored in a preset cache library.

[0145] It should be noted that if the parsing model corresponding to the data format is not found according to the data format, the parsing model of the current data format can be reconfigured and updated to obtain the configured updated parsing model, without updating the updated parsing rules corresponding to the parsing model configuration.

[0146] To ensure that the change of the parsing model takes effect in time, a preset cache library can be set. The preset cache library is a secondary Redis cache. Redis is an open source, in-memory data structure storage system. The reconfigured updated parsing model is stored in this cache, and the corresponding updated parsing rules are configured for the updated parsing model. The updated parsing rules are synchronized to the cache. When parsing the battery data corresponding to this data format, the updated parsing model can be directly obtained from the preset cache library, and parsed through the updated parsing rules in the updated parsing model, thereby improving the efficiency and comprehensiveness of the parsing.

[0147] When the parsing model corresponding to the data format is not queried, this embodiment updates the parsing model to obtain an updated parsing model, configures the updated parsing rules in the updated parsing model, and stores the updated parsing model and the updated parsing rules in a preset cache library, so that the updated parsing model and the updated parsing rules can be directly retrieved according to the preset cache library, thereby improving the efficiency of parsing.

[0148] As an example, after the update parsing model and the update parsing rules are configured, the update parsing rules can be obtained from the preset cache library before loading the rules, and the method further includes:

[0149] Get the updated parsing rules from the preset cache library;

[0150] The updated parsing rules will be initially loaded.

[0151] It should be noted that after configuring the update resolution rules, you can obtain the update resolution faults from the Redis cache before loading the rules, so as to ensure that the changed rules are loaded and executed, ensuring that the changed rules take effect in a timely manner.

[0152] In the technical solution of the embodiment of the present application, before parsing, the updated parsing rules are initialized and loaded to ensure that the changed rules are loaded and executed, thereby ensuring that the updated parsing rules take effect on time.

[0153] In some embodiments, the parsing model is established in advance, and when parsing the connected battery data, the corresponding parsing model can be directly called to improve the efficiency of data parsing. Before step S20, the step of constructing the parsing model is also included, such as Figure 6 As shown, the steps of constructing the analytical model may include:

[0154] Step S11: Collecting different types of battery sample data.

[0155] It should be noted that the battery sample data is a large amount of battery data collected in advance, including battery data in different scenarios.

[0156] Step S12: parse the battery sample data to determine the battery data format of the battery sample data.

[0157] It should be understood that different parsing rules can be used to parse the battery sample data to obtain the battery data format of the battery sample data. For example, data A in the battery sample data can be parsed using the json format parsing rules to obtain the json format of the battery data, and data B in the battery sample data can be parsed using the xml format parsing rules to obtain the xml format of the battery data.

[0158] Step S13: configuring a corresponding analytical model through the battery data format.

[0159] In a specific implementation, the parsing rules for parsing the battery data format can be obtained through the battery data format, so as to establish a corresponding relationship between the parsing rules and the battery data format, and thus configure the parsing model through multiple parsing rules.

[0160] This embodiment acquires battery sample data under different operating conditions of the battery by collecting massive data in advance, and determines the data format corresponding to the battery sample data by parsing the battery sample data, and configures the corresponding data format with a parsing model that can parse this format, thereby constructing a parsing model to flexibly handle the needs of data parsing in various scenarios. Through the unified design and definition of the parsing model, the accessed battery data can be uniformly processed through the parsing model during data collection, thereby improving processing efficiency.

[0161] In some embodiments, the attributes of battery data in the same battery data format may also be different, so different parsing rules can be constructed in advance according to the attributes of the corresponding battery data to meet the parsing of battery data with different attributes and improve the flexibility of parsing.

[0162] In some embodiments, before configuring the corresponding parsing model through the battery data format, the method further includes:

[0163] Determine the attributes of the battery data through the battery data format;

[0164] Configure the corresponding parsing rules through attributes;

[0165] Construct parsing models through parsing rules.

[0166] It should be noted that the battery data in the same format may be analyzed based on the format of the battery data to determine the attributes of the battery data, such as the voltage attribute, current attribute, capacity attribute, etc. of the battery, and may also include other attributes.

[0167] Parsing rules of different attributes can be configured through the attributes of battery data, so as to build a parsing model through multiple parsing rules.

[0168] This embodiment determines the attributes of the battery data through the battery data format; configures the corresponding parsing rules through the attributes; and builds a parsing model through the parsing rules. Since the parsing model includes different parsing rules, different parsing rules can be used to parse battery data with different attributes. Therefore, corresponding parsing rules can be configured for the battery data according to the attributes of the battery data, thereby improving the efficiency of battery data parsing.

[0169] In some embodiments, since the result of parsing the battery data by the parsing rules in the parsing model is a standardized battery data structure, the uniformity of the collected data can be improved.

[0170] like Figure 7 As shown, step S30 specifically includes:

[0171] Step S301: Determine the attributes of the battery data according to the battery data.

[0172] It should be noted that the attributes of the accessed battery data can be determined by analyzing the battery data.

[0173] Step S302: querying the corresponding parsing rules from the parsing model based on the attributes.

[0174] It can be understood that there is a corresponding relationship between the attributes of the battery data and the parsing rules, so the corresponding parsing rules can be queried from the parsing model based on the attributes.

[0175] Step S303: parse the battery data using parsing rules to obtain a standardized data structure, thereby completing data collection.

[0176] It should be noted that the corresponding parsing rules can be configured for the current battery data through the attributes of the battery data, so that the battery data can be parsed through the parsing rules. The parsing may include normalizing the battery data, extracting keywords, etc., so as to obtain a standardized data structure that meets the needs of subsequent data processing and complete the data collection at the collection layer.

[0177] This embodiment determines the attributes of the connected battery data, and then queries the corresponding parsing rules from the parsing model through the relationship between the attributes and the parsing rules, quickly locates the corresponding parsing rules through the attributes, thereby improving the parsing efficiency, parses the battery data through the parsing rules, and standardizes the battery data to obtain a standardized data structure, completes data collection, and improves the efficiency of the subsequent data processing process of battery access.

[0178] In some embodiments, in order to improve the parsing efficiency when parsing data, after the corresponding parsing model is retrieved, the parsing rules in the parsing model may be loaded to improve the parsing efficiency. Then, after step S20, the following steps may be further included:

[0179] Get the parsing rules in the parsing model;

[0180] Initialize the parsing rules.

[0181] It should be noted that when the parsing rules are started for the first time, they can be started through the container, by obtaining all the parsing rules in the parsing model and initializing all the parsing rules, thereby ensuring that all the parsing rules take effect in a timely manner.

[0182] This embodiment can ensure that all parsing rules take effect in a timely manner and improve the efficiency of parsing by initializing and loading the parsing rules in the parsing model.

[0183] In some embodiments, since the parsing rules may be incompatible or erroneous with the battery data type during configuration, resulting in parsing errors or anomalies, affecting the normal collection of battery data, it is necessary to verify the collected battery data before parsing the data to ensure the accuracy of the accessed battery data.

[0184] like Figure 8 As shown, the steps of verifying the battery data specifically include:

[0185] Step S21: verify the battery data according to the attributes of the battery data to obtain a verification result.

[0186] It should be noted that the attributes of the battery data may be verified to determine whether there are errors in the battery data.

[0187] Step S22: When the verification result is that the verification is passed, the battery data is parsed according to the parsing rules in the parsing model to obtain a standardized data structure, thereby completing the step of data collection.

[0188] It should be understood that when the battery data is verified to be free of errors, it is proved that the accessed battery data can be parsed normally, and then the step of parsing the battery data using the parsing rules in the parsing model can be executed.

[0189] For example, if the attribute of the accessed battery data is user information, such as the user's mobile phone number, it can be verified using the verification rules for the corresponding mobile phone number. For example, if the verification result is that the mobile phone number is an 11-digit number, the verification passes. If the verification result is a 12-digit number or 11-digit characters, etc., the verification passes.

[0190] It should be noted that the step of verifying the battery data may be performed simultaneously with the step of retrieving the corresponding analytical model based on the battery data, or may be performed before or after the step of retrieving the corresponding analytical model based on the battery data, and this embodiment does not impose any limitation on this.

[0191] In the technical solution of the embodiment of the present application, before parsing the battery data, in order to avoid incorrect battery data input leading to parsing errors or abnormalities, the battery data can be verified, thereby improving the accuracy of the accessed battery data and improving the fault tolerance of the analysis.

[0192] Optionally, when the battery data verification fails, in order to avoid affecting the progress of subsequent data analysis, it is necessary to promptly remind the battery data with verification errors.

[0193] In some embodiments, Fig. 9 As shown, after obtaining the verification result, the method further includes:

[0194] Step S23: When the verification result is verification failure, a reminder message is generated.

[0195] It should be noted that, for example, when verifying the user's mobile phone number in the battery data, if the verified string is "abcd1112345", the verification result is verification failure, and a reminder message can be generated. The reminder message carries the reason for the verification failure, such as "the number input is incorrect."

[0196] Step S24: Perform a battery data format query reminder through reminder information.

[0197] In a specific implementation, the reminder information can be transmitted to the front-end page to remind the user who uploaded the battery data to query and modify the battery data format, thereby improving the accuracy of data analysis.

[0198] In the technical solution of the embodiment of the present application, when the verification result is verification failure, it means that the connected battery data is incompatible or there is an error, and a reminder message can be generated to remind the staff to query and correct the battery data format.

[0199] In some embodiments, when verifying the battery data, in order to improve the verification efficiency, the battery data may be verified using verification rules to improve the processing efficiency.

[0200] like Fig.10 As shown, in step S21, the battery data is verified according to the attributes of the battery data, and the verification result obtained specifically includes:

[0201] Step S211: Retrieve corresponding verification rules according to the attributes of the battery data.

[0202] It should be noted that the corresponding verification rules can be retrieved according to the attributes of the battery data. The verification rules are the standards for verifying the battery data. Rules for verifying battery data with different attributes can be collected in advance to construct multiple verification rules. There is a correlation between the verification rules and the attributes of the battery data, and the corresponding verification rules can be retrieved through the attributes of the battery data and the correlation.

[0203] Step S212: Verify the battery data using verification rules to obtain verification results.

[0204] It should be understood that the battery data may be verified by verification rules to determine whether there are errors or incompatibilities in the battery data.

[0205] In the technical solution of the embodiment of the present application, the verification rules corresponding to the attributes can be called through the attributes of the battery data. The verification rules can verify the attributes of the battery data to determine whether there are incompatibilities or errors in the battery data. The battery data can be verified in advance through the verification rules to speed up the subsequent data analysis time.

[0206] In some embodiments, verification rules may be established in advance, so that when verifying battery data, the verification rules may be directly called to improve the efficiency of verification. Before calling the corresponding verification rules, the following steps may also be included:

[0207] Collect attribute sample data of battery data;

[0208] By parsing the attribute sample data, the verification process of the attribute sample data is determined;

[0209] Build validation rules based on the validation process.

[0210] It should be noted that the attribute sample data of the battery data may be collected in advance, and the attribute sample data may be parsed to determine the verification process of the attribute sample data.

[0211] By analyzing the verification process, different verification rules are configured for different verification processes. For example, the verification process is to verify user information or to verify the data length of battery data.

[0212] It should be understood that after configuring different verification rules, the verification rules can be summarized and stored, for example, the verification rules are stored in a verification rule library, which is also set in the data access gateway module in the acquisition layer, and can also be stored elsewhere. When verifying the battery data, the corresponding verification rules in the verification rule library can be called, so that the battery data can be verified through the verification rules.

[0213] This embodiment collects attribute sample data of battery data in advance, parses the attribute sample data, determines the verification process of the attribute sample data, and then constructs verification rules through the verification process, and statistically summarizes different verification rules. When verifying the battery data, the needs of data verification in various scenarios can be flexibly handled by calling the corresponding verification rules.

[0214] In some embodiments, since the result of parsing the battery data by the parsing rules in the parsing model is a standardized battery data structure, the uniformity of the collected data can be improved.

[0215] like Fig.11 As shown, step S30 specifically includes:

[0216] Step S301 ′: parse the battery data using the parsing rules in the parsing model to obtain keyword data of the battery data.

[0217] It should be noted that the battery data can be parsed for keyword extraction through the parsing rules in the parsing model to obtain the keyword data of the battery data. Since the accessed battery data includes a large amount of invalid data, the keyword data useful for subsequent data processing can be directly extracted from the battery data to obtain the keyword data of the battery data.

[0218] If the accessed battery data includes image data, the image data may be grayed out using the image parsing rules in the parsing rules.

[0219] Step S302 ′: Use the keyword data of the battery data as a standardized data structure to complete data collection.

[0220] In a specific implementation, the keyword data of the processed battery data can be used as a standardized data structure to complete data collection, and the data corresponding to the standardized data structure is transmitted to the processing layer for data processing, thereby improving the efficiency of battery data access to the processing layer.

[0221] When parsing battery data using parsing rules, this embodiment extracts keywords from the battery data to obtain keyword data of the battery data, and uses the extracted keyword data as a standardized data structure. By standardizing the battery data, the effect of battery data collection is improved.

[0222] In some embodiments, since different battery parties may upload battery data in different ways, for example, the collection program has accessed the data of business A and has been released online. Once the data format of A changes, for example, the original fields are a, b, and c, and a new field d is added, the collection needs to be synchronously modified to continue, and d needs to be customized and parsed before it is released online. Or, field a needs to be renamed m, or field b is invalidated and deleted, and the collection side is not informed in time, the collection service will have parsing exceptions or errors, thereby affecting the business. The battery data to be collected can be analyzed to further improve the consistency of the collection.

[0223] like Fig.12 As shown, the steps of obtaining the battery data to be collected specifically include:

[0224] Step S101: Acquire the collection protocol of the data to be collected.

[0225] It should be noted that when preparing for online data collection, the collection protocol of the data to be collected can be obtained. For example, if there are three different battery data A, B, and C to be collected, the collection protocol all uses Kafka.

[0226] The data layer can also access offline data, such as data collected by users in advance, and upload the data to the collection layer through the collection interface.

[0227] Step S102: Determine the acquisition interface based on the acquisition protocol.

[0228] In a specific implementation, different collection interfaces can be configured at the collection layer. For example, collection interfaces corresponding to different collection protocols can be configured in advance, including Kafka collection interface, Http collection interface, etc.

[0229] Step S103: Acquire the battery data to be collected through the collection interface.

[0230] After the collection interface is determined, the battery data to be collected with the same collection protocol can be uniformly accessed through the collection interface. For example, the collection protocols of three different battery data to be collected, A, B, and C, all use Kafka, and the collection interface corresponding to Kafka is collection interface 1. Then, the battery data to be collected with the Kafka protocol can be uniformly collected through collection interface 1.

[0231] When performing data collection, this embodiment can obtain the collection protocol of the data to be collected, so as to determine the collection interface during collection according to the collection protocol, and can access the battery data to be collected of the corresponding collection protocol through the collection interface, thereby improving the efficiency of data access.

[0232] To achieve the above-mentioned purpose, an embodiment of the present invention further proposes a data acquisition system, which includes a data access gateway module, a parsing module, a user management module, a cache module and a parsing module, wherein the data access gateway module is connected to the user management module, the cache module and the parsing module respectively, and a parsing model is stored in the parsing module; the user management module is used to obtain user information and send the user information to the data access gateway module; the data access gateway module is used to load the corresponding acquisition protocol according to the user information, and acquire the corresponding battery data to be collected through the acquisition protocol, and retrieve the corresponding parsing model from the parsing module based on the battery data to be collected, and parse the battery data to be collected according to the parsing model to obtain a standardized data structure; the cache module is used to store the standardized data structure.

[0233] like Fig.13 As shown, Fig.13It is a structural diagram of the data acquisition system. The data acquisition system also includes an acquisition interface. The acquisition interface accesses the corresponding battery data to be collected. Specifically, the corresponding acquisition protocol can be loaded and configured according to the user information of the user management module. For example, if data A, B, and C are all kafka acquisition protocols, then data A, B, and C are uniformly accessed, and the data A, B, and C to be collected are accessed to the data access gateway module. The data access gateway module mainly completes the adaptation of acquisition protocols such as http and kafka, and also performs flow control and authentication on the accessed battery data and verification of device registration. After completing these steps, the accessed battery data will also be analyzed to determine the type of battery data. The corresponding parsing model is configured from the parsing module according to the battery type of the battery data, and the corresponding parsing rules are configured, so that the battery data is parsed and processed through the parsing rules, a standardized data structure is generated, and the standardized data structure is stored in the cache module. When there is no parsing rule that can parse the battery data in the parsing model, the parsing rule can be reconfigured to obtain an updated parsing rule, and the updated parsing rule is stored in the redis cache. The corresponding parsing rule is called through the redis cache, so that the parsing engine in the parsing rule is used to parse the data. The parsing rule engine may include a groovy engine, a pb engine, a qlexpress engine or other engines.

[0234] To achieve the above objective, an embodiment of the present invention further provides a storage medium, on which a data acquisition program is stored, and when the data acquisition program is executed by a processor, the data acquisition method as described above is implemented.

[0235] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0236] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data collection method, characterized in that: The data collection method comprises: Obtain battery data to be collected; Retrieving a corresponding analytical model according to the battery data; The battery data is parsed using the parsing rules in the parsing model to obtain a standardized data structure, thereby completing data collection.

2. The data collection method according to claim 1, characterized in that: The retrieving a corresponding analytical model according to the battery data includes: Analyzing the battery data to determine the battery type of the battery data; The corresponding analytical model is retrieved according to the battery type.

3. The data collection method according to claim 2, characterized in that: The retrieving a corresponding analytical model according to the battery type includes: determining a data format of the battery data according to the battery type; Querying a parsing model corresponding to the data format according to the data format; Retrieve the parsing model corresponding to the data format.

4. The data collection method according to claim 3, characterized in that: The method further comprises: If no parsing model corresponding to the data format is found, updating the parsing model to obtain an updated parsing model; Obtaining update parsing rules of the update parsing model; The update parsing model and the update parsing rules are stored in a preset cache library.

5. The data collection method according to claim 4, characterized in that: After the corresponding analytical model is retrieved according to the battery data, the method further includes: Acquire the update parsing rule from the preset cache library; Initialize and load the update parsing rules.

6. The data collection method according to any one of claims 1 to 4, characterized in that: Before calling the corresponding analytical model according to the battery data, the method includes: Collect data of different types of battery samples; Parsing the battery sample data to determine the battery data format of the battery sample data; The corresponding parsing model is configured through the battery data format.

7. The data collection method according to claim 5, characterized in that: Before configuring the corresponding parsing model through the battery data format, the method further includes: Determining the attributes of the battery data by using the battery data format; Configure the corresponding parsing rules through the properties; A parsing model is constructed according to the parsing rules.

8. The data collection method according to any one of claims 1 to 7, characterized in that: The battery data is parsed by the parsing rules in the parsing model to obtain a standardized data structure and complete data collection, including: determining an attribute of the battery data according to the battery data; querying a corresponding parsing rule from the parsing model based on the attribute; The battery data is parsed according to the parsing rules to obtain a standardized data structure, thereby completing data collection.

9. The data collection method according to claim 8, characterized in that: The method further comprises: Verifying the battery data according to the attributes of the battery data to obtain a verification result; When the verification result is that the verification is passed, the step of parsing the battery data according to the parsing rules in the parsing model is executed to obtain a standardized data structure, thereby completing the step of data collection.

10. The data collection method according to claim 9, characterized in that: The method further comprises: When the verification result is that the verification fails, generating a reminder message; The battery data format query reminder is performed through the reminder information.

11. The data collection method according to claim 9 or 10, characterized in that: The verifying the battery data according to the attributes of the battery data to obtain a verification result includes: Retrieving corresponding verification rules according to the attributes of the battery data; The battery data is verified using the verification rule to obtain a verification result.

12. The data collection method according to claim 11, characterized in that: Before calling the corresponding verification rule according to the attribute of the battery data, the method further includes: Collect attribute sample data of battery data; Determining a verification process of the attribute sample data by parsing the attribute sample data; Verification rules are constructed based on the verification process.

13. The data collection method according to any one of claims 1 to 12, characterized in that: The battery data is parsed by the parsing rules in the parsing model to obtain a standardized data structure and complete data collection, including: Parsing the battery data using the parsing rules in the parsing model to obtain keyword data of the battery data; The keyword data of the battery data is used as a standardized data structure to complete data collection.

14. The data collection method according to any one of claims 1 to 13, characterized in that: After the corresponding analytical model is retrieved according to the battery data, the method further includes: Obtaining parsing rules in the parsing model; Initialize and load the parsing rules.

15. The data collection method according to any one of claims 1 to 14, characterized in that: The obtaining of the battery data to be collected includes: Obtaining a collection agreement for the data to be collected; Determining a collection interface based on the collection protocol; The battery data to be collected is acquired through the collection interface.

16. A data acquisition system, characterized in that: The data acquisition system comprises: a data access gateway module, a parsing module, a user management module, a cache module and a parsing module, wherein the data access gateway module is connected to the user management module, the cache module and the parsing module respectively, and the parsing module stores a parsing model; The user management module is used to obtain user information and send the user information to the data access gateway module; The data access gateway module is used to load the corresponding acquisition protocol according to the user information, acquire the corresponding battery data to be acquired through the acquisition protocol, retrieve the corresponding analysis model from the analysis module based on the battery data to be acquired, and analyze the battery data to be acquired according to the analysis model to obtain a standardized data structure; The cache module is used to store standardized data structures.

17. A data acquisition device, characterized in that: The data acquisition device comprises: a memory, a processor, and a data acquisition program stored in the memory and executable on the processor, wherein the data acquisition program is configured to implement the data acquisition method according to any one of claims 1 to 15.

18. A storage medium, characterized in that: The storage medium stores a data acquisition program, and when the data acquisition program is executed by the processor, the data acquisition method according to any one of claims 1 to 15 is implemented.

Images