Data processing method and device, equipment and storage medium

Abstract

The invention relates to the field of data processing, and provides a data processing method and device, equipment and a storage medium. The method is implemented based on a data processing system, the data processing system comprises an access layer, a configuration layer and a data layer, and the method comprises the following steps: accessing to-be-processed data through the access layer; obtaining a message analysis script set by the configuration layer through the access layer, and converting the to-be-processed data into standard data; and acquiring the equipment model set by the configuration layer through the data layer, and mapping the standard data to a data table corresponding to the equipment model. According to the data processing method and the data processing system, the message analysis is carried out on the data through the message analysis script set by the configuration layer, the data processing requirements of different standards can be met, the analyzed data can be mapped through the corresponding data table only by correspondingly updating the equipment model on the configuration layer, and the downstream data convenient to use can be effectively obtained.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to data processing methods, devices, equipment and storage media. Background Art

[0002] With the development of new energy technology, battery technology has been widely used. In order to better optimize battery performance or monitor the battery status, a large amount of battery data will be collected during the production or use of the battery.

[0003] Due to the diversity of data standards, including national standards, enterprise standards, and custom standards, the content of the collected battery data is diverse. The data content of different standards varies greatly and cannot effectively generate data that can be used downstream. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an aura processing method, device, equipment and storage medium to solve the problem in the prior art that due to the diversity of data standards, the content of the collected battery data is diverse, the data content of different standards is quite different, and it is impossible to effectively generate downstream usable data.

[0005] A first aspect of an embodiment of the present application provides a data processing method, which is implemented based on a data processing system, wherein the data processing system includes an access layer, a configuration layer and a data layer, and the method includes: accessing the data to be processed through the access layer; obtaining the message parsing script set by the configuration layer through the access layer, and converting the data to be processed into standard data; obtaining the device model set by the configuration layer through the data layer, and mapping the standard data to the data table corresponding to the device model.

[0006] For the data to be processed accessed by the access layer, the message parsing script set by the configuration layer can be used to parse the message, and the standard data corresponding to the data to be processed of different standards can be obtained, so as to adapt to the processing requirements of data of different standards. The data layer maps the standard data to the data table corresponding to the data model according to the device model set by the configuration layer, so that the standard data converted from the data of different standards can be mapped to data that is convenient for downstream use. For data of different standards, you only need to update the device model accordingly in the configuration layer, and you can map the parsed data through the corresponding data table, which is conducive to improving the convenience of using data downstream.

[0007] In combination with the first aspect, in a first possible implementation of the first aspect, before mapping the standard data to the data table corresponding to the device model, the method includes: obtaining a change message of the device model through the data layer; and updating the data table corresponding to the device model through the data layer according to the change relationship between the change message of the device model and the existing device model, wherein the data table is used to map the standard data.

[0008] When mapping standard data to the data layer, it is necessary to set the data table corresponding to the data to be processed so that the data to be processed can be accurately and effectively mapped to the data layer. After the device model is published in the configuration layer, the device model can be converted into data defined in JSON (JavaScript Object Notation in English, JS Object Notation in Chinese), including data in the object model format, and the converted data is sent to the data layer. The data layer can update the data table in the data according to the change message of the device model, so that the updated data table matches the changed device model, thereby facilitating the mapping of the data obtained by the access layer to the data layer.

[0009] In combination with the first possible implementation method of the first aspect, in a second possible implementation method of the first aspect, the data table corresponding to the device model is updated through the data layer according to the change relationship between the change message of the device model and the existing device model, including: determining the change relationship between the change message of the device model and the existing device model through the data layer; when the change message of the device model is a change message of an existing device model, adding a new field to the existing model table according to the change message of the device model through the data layer, and the existing model table is a data table mapped to the existing model; when the change message of the device model is a change message of an unknown device model, the data layer creates a data table according to the change message of the device model, or determines the newly added field according to the change message of the device model, and updates the data table according to the newly added field.

[0010] When the data to be processed is obtained through data layer mapping, a change message of the device model can be sent to the data layer according to the device model set by the setting layer. The data layer determines the data table corresponding to the changed device model based on the change message of the device model. If the change message of the device model is a change message of an existing device model, a new field can be added to the existing model table mapped to the existing model to obtain an updated data table. If the change message of the device model is a change message of an unknown device model, a new data table can be created, or the fields in the change message of the device model can be determined, the fields in the message can be compared with the existing data table, the newly added fields in the existing data table can be determined, and the data table can be updated according to the newly added fields, thereby obtaining the data table corresponding to the device model.

[0011] In combination with the second possible implementation method of the first aspect, in a third possible implementation method of the first aspect, after determining the change relationship between the change message of the device model and the existing device model through the data layer, the method also includes: when the fields included in the change message of the device model are inherited from the basic model, adding new fields to the basic model table according to the change message of the device model through the data layer.

[0012] In order to obtain the data table corresponding to the device model more quickly, after comparing the change message of the device model with the existing device model, you can first determine whether the fields included in the change message of the device model are inherited from the basic model. If it is inherited from the basic model, you can directly add new fields to the basic model table to quickly obtain the data table corresponding to the change message of the device model.

[0013] In combination with the second possible implementation method of the first aspect, in a fourth possible implementation method of the first aspect, after obtaining the change message of the device model through the data layer, the method also includes: determining the type of the change message of the device model according to the data identifier in the change message of the device model.

[0014] Since the device model includes at least one of a device attribute model, a device event model, and a device measurement data model, the change messages of different types of device models correspond to different data tables. In order to accurately map the accessed data to the data layer, data tables can be set separately according to different types of device models. For example, the data table can include at least one of a device attribute table, a device event table, and a device measurement data table.

[0015] In combination with any one of the first aspect to the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the message parsing script set by the configuration layer is obtained through the access layer, and the data to be processed is converted into standard data, including: identifying the message header and message body in the data to be processed through the access layer; performing message header conversion and message body conversion on the identified data according to the message parsing script set by the configuration layer through the access layer; and obtaining the standard data according to the converted message through the access layer.

[0016] When converting the data to be processed into standard data, the data to be processed is first identified and converted to obtain the message header and message body corresponding to the data to be processed, and the message header in the data to be processed is converted according to the message header conversion script included in the message parsing script set in the configuration layer, and the message body in the data to be processed is converted according to the message body parsing script in the message parsing script set in the configuration layer, and the standard data is obtained according to the converted message header and message body.

[0017] In combination with the fifth possible implementation method of the first aspect, in a sixth possible implementation method of the first aspect, the standard data is obtained through the access layer according to the converted message, including: the access layer obtains the constraint rules set by the configuration layer; the access layer verifies the data to be processed according to the constraint rules, and determines the standard data based on the verified data.

[0018] In order to improve data processing efficiency, before mapping the data, you can also obtain the constraint rules set by the configuration layer and verify the data to be processed obtained by the access layer. For battery data scenarios, the verification content may include: verification of integer, single-precision floating-point, and double-precision floating-point data, mainly to verify whether the actual data type is consistent and whether the maximum and minimum values ​​meet the requirements. Verification of enumeration and Boolean data may include verifying whether the actual data is included in the defined range. For time data, it may include verifying whether the time format conforms to the predefined format (timestamp, custom time format). For structured data, it may include verifying whether the data is a data structure such as a JSON structure.

[0019] In combination with the fifth possible implementation manner of the first aspect, in the seventh possible implementation manner of the first aspect, after identifying the message header and message body in the data to be processed through the access layer, and before the access layer performs message header conversion and message body conversion on the identified data according to the message parsing script set by the configuration layer, the method also includes: obtaining routing rules of different versions of message parsing scripts set by the configuration layer through the access layer; and routing the data to be processed to the corresponding version of the message parsing script for parsing through the access layer according to the routing rules.

[0020] In order to effectively process data while reducing the loss caused by errors in data processing by the new version of the message parsing script, you can set the routes corresponding to different versions of the message parsing script, and send different data to be processed to different message parsing scripts for parsing. For example, the first version of the message parsing script can be routed to the first range of data to be processed, and the second version of the message parsing script can be routed to the second range of data to be processed. After determining the stability and reliability of the new version of the message parsing script, increase the amount of data routed to the new version of the message parsing script for processing.

[0021] In combination with the seventh possible implementation method of the first aspect, in the eighth possible implementation method of the first aspect, the routing rules include HASH value routing rules and / or whitelist routing rules; the access layer routes the data to be processed to the corresponding version of the message parsing script for parsing according to the routing rules, including: determining the message parsing script corresponding to the version of the data to be processed according to the numerical range of the HASH value of a predetermined field of the data to be processed by the access layer; or, determining the message parsing version of the principal and information corresponding to the data to be processed according to the whitelist corresponding to the version of the message parsing script by the access layer.

[0022] When routing data to different versions of message parsing scripts, the whitelist and / or blacklist corresponding to different versions of message parsing scripts can be determined by setting whitelists and / or blacklists, and the data to be processed in the whitelist can be routed to the corresponding version of the message parsing script. Alternatively, specific fields in the data to be processed, such as device identification, can be hashed to obtain data to be processed within a predetermined range, and then routed to the corresponding version of the message parsing script for parsing.

[0023] In combination with the first aspect to the eighth possible implementation method of the first aspect, in the ninth possible implementation method of the first aspect, accessing the data to be processed through the access layer includes: accessing the data to be processed through the access layer through at least one of http (Hyper Text Transfer Protocol in English, Hypertext Transfer Protocol in Chinese) protocol adaptation, csv import, mq (Message Queue in English, Message Sequence in Chinese) real-time message access, oss (Object Storage Service in English, Object Storage Service in Chinese) docking, MinIO (an object storage service based on the Apache License v2.0 open source protocol) docking and ftp (File Transfer Protocol in English, File Transfer Protocol in Chinese) docking.

[0024] By setting up multiple data access methods at the access layer, it can effectively adapt to the docking of data with different access methods, including completing the access of HTTP protocol data through the HTTP access gateway, completing the access of MQ messages through the MQ message middleware, and adapting the import of CSV data through offline import. By setting the corresponding adaptation module or by data extraction, it can effectively dock the data to be processed with multiple access methods.

[0025] The second aspect of an embodiment of the present application provides a data processing device, which is implemented based on a data processing system. The data processing system includes an access layer, a configuration layer and a data layer. The device includes: an access unit, which is used to access the data to be processed through the access layer; a parsing unit, which is used to obtain the message parsing script set by the configuration layer through the access layer, and convert the data to be processed into standard data; a mapping unit, which is used to obtain the device model set by the configuration layer through the data layer, and map the standard data to the data table corresponding to the device model.

[0026] A third aspect of an embodiment of the present application provides a data processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0027] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0028] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 It is a schematic diagram of an application scenario of a data processing method provided in an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;

[0032] Figure 3 This is a schematic diagram of an implementation flow of a message conversion method provided in an embodiment of the present application;

[0033] Figure 4 It is a schematic diagram of an implementation flow of a method for generating standard data provided in an embodiment of the present application;

[0034] Figure 5 It is a schematic diagram of an implementation flow of a method for updating a device model provided in an embodiment of the present application;

[0035] Figure 6 It is a schematic diagram of an implementation flow of a routing method for determining a message parsing version provided in an embodiment of the present application;

[0036] Figure 7 It is a schematic diagram of an implementation flow of a data table updating method provided in an embodiment of the present application;

[0037] Figure 8 It is a structural diagram of a platform system provided in an embodiment of the present application;

[0038] Fig. 9 is a schematic diagram of a data processing device provided in an embodiment of the present application;

[0039] Fig.10 It is a schematic diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0041] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0042] Figure 1 A schematic diagram of an application scenario of a data processing method provided in an embodiment of the present application. Figure 1 As shown, the data for processing includes battery data, and the application scenario for processing battery data includes a data terminal and a platform. Among them, the data terminal is used to collect data to be processed, and the data terminal may include new energy vehicles, charging piles, energy storage power stations or other battery-powered devices. The platform can connect to data of various different access methods through a variety of data access methods set by the access layer. The data accessed by the access layer is parsed through the message parsing script set by the configuration layer, and the collected data is processed into standard data through the message parsing script. Through the device model set by the configuration layer, the data table in the data layer is updated through the change message of the device model, so as to facilitate the effective mapping of the data accessed by the access layer to the data layer after parsing, which is convenient for subsequent use.

[0043] Figure 2 A schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application is implemented based on a data processing system, wherein the data processing system includes an access layer, a configuration layer, and a data layer. The method is described in detail as follows:

[0044] In S101, data to be processed is accessed through an access layer.

[0045] Specifically, the data to be processed in the embodiments of the present application may include data from a variety of different docking scenarios. For example, the data to be processed may include data in http protocol format accessed based on http protocol, mq real-time messages accessed based on mq message middleware, csv data imported based on offline data, and data imported based on object storage service OSS, distributed file storage service MinIO, and file transfer protocol ftp.

[0046] By adapting different data access modules at the access layer, including at least one of the http protocol adaptation module, mq message middleware, csv import module, and OSS / MinIO / ftp docking module, http protocol data, mq message, csv offline data, and access data based on OSS / MinIO / ftp can be accessed accordingly.

[0047] In a possible implementation, a new adaptation protocol or component may be added through the configuration layer so that the access layer can adapt to the data connection of the newly added access method.

[0048] In S102, the message parsing script set by the configuration layer is obtained through the access layer, and the data to be processed is converted into standard data.

[0049] In the embodiment of the present application, the configuration layer can set a corresponding message parsing script according to the field information included in the data to be processed accessed by the access layer. The data to be processed is parsed by the message parsing script to obtain standard data that can be mapped to the data layer.

[0050] The message parsing script can be configured through the message parsing script configuration module in the configuration layer. The message parsing script configuration module can include functional modules such as a script content module, a script verification module, a script debugging module, a script compilation module, a script version module, and a grayscale configuration module.

[0051] Among them, the script content module may include a message header conversion method and a message body conversion method. The script content module can convert the message header and message content in the data to be processed according to the message header conversion method and the message conversion method. For example, the accessed data to be processed can be converted into a standard format of a message header and a message body, and the message header therein can be converted by the message header conversion method, and the message body therein can be converted by the message body conversion method.

[0052] The message header conversion can be understood as the mapping of the key in the data to be processed to the attribute key in the attribute field of the device model set by the configuration layer. The message body conversion can be understood as the conversion of the value in the data to be processed to the value in the attribute field of the device model set by the configuration layer.

[0053] Message body conversion includes but is not limited to data magnification, data reduction, constant value increase, constant value decrease, etc. For example, if the units of the values ​​in the data to be processed are different from those in the set device model, such as voltage units and current units, the data needs to be magnified or reduced. If the reference points of the data are different, the data needs to be processed by adding constant values ​​or reducing constant values. The message body conversion method in the message parsing script can be set through the configuration layer according to the format of the accessed data to be processed and the standard data, and the requirements of the measurement standard.

[0054] The script verification module can be used for basic syntax verification. If the syntax verification in the script content fails, it cannot be submitted successfully. The verification rules and logic of the script syntax structure can be set through the configuration layer to automatically verify the script content.

[0055] The script compilation module can be used to compile the edited script into a binary executable file to improve the efficiency of execution.

[0056] The script version module can be used to manage new and old versions. When users need to upgrade the scripts in the production environment, they edit the published scripts. The version management system will generate a new script version based on the content of the latest published script. The newly generated script version is in draft state, and the version number is increased by 1 based on the original one. When the user edits the logic of the script based on the newly generated script, completes the debugging and compilation of the script, the latest version can be published.

[0057] The grayscale configuration module is used to complete the filtering and allocation of traffic between different versions. The version of the message parsing script for routing can be determined according to the HASH modulo value of the predetermined field in the data to be processed. By configuring the predetermined field HASH routing rules, the user can route the HASH value of the predetermined field to the specified version of the message parsing script, such as routing to the latest version of the message parsing script, thereby completing the comparison test of the new and old scripts, and finally achieving smooth script launch.

[0058] In a possible implementation, for the same HASH result, such as belonging to a predetermined value range, it can be routed to one or more versions of the message parsing script. When two or more message parsing scripts parse the data to be processed at the same time, the accuracy and reliability of the new version of the message parsing script can be determined based on the comparison of the parsing results.

[0059] In a possible implementation, a blacklist or whitelist of data end identifiers corresponding to different versions of message parsing scripts may also be set. When the data end identifier of the data to be processed belongs to the whitelist of the message parsing script of that version, the data to be processed is allowed to be routed to the message parsing script. If the data end identifier of the data to be processed belongs to the blacklist of the message parsing script of that version, the data to be processed is prohibited from being routed to the message parsing script.

[0060] The access layer obtains the message parsing script set by the configuration layer and converts the data to be processed into standard data. Figure 3 As shown, including:

[0061] In S301, a message header and a message body in the data to be processed are identified through the access layer.

[0062] The message header and message body in the data to be processed can be determined according to the format of the data to be processed. The message header is the key included in the data to be processed, and the message body is the value included in the data to be processed. The message header and message body in the collected data to be processed can be determined according to the data format defined by the data collection system of different data terminals.

[0063] In S302, the access layer performs message header conversion and message body conversion on the identified data according to the message parsing script set by the configuration layer.

[0064] After determining the message header and message body in the data to be processed, the message header conversion method and message body conversion method in the message parsing script obtained from the configuration layer can be used to convert the data to be processed. That is, through the message header conversion method, the message header after the data to be processed is converted into the message header in the device model. For fields that cannot be found in the message parsing script, the message header in the data to be processed is passed down as is.

[0065] When converting the message body, the message body of the data to be processed can be converted into the message body of the device model by magnifying, reducing, decreasing, or increasing the constant value according to the message body conversion method in the message parsing.

[0066] In S303, the standard data is obtained according to the converted message through the access layer.

[0067] The standard data includes data that can be used to map to the device model set by the configuration layer.

[0068] In order to improve the reliability and efficiency of data mapping, when generating standard data, you can also Figure 4 As shown, including:

[0069] In S401, the constraint rules set by the configuration layer are obtained through the access layer.

[0070] The attribute type in the attribute field of the device model includes at least one of integer, single-precision floating point, double-precision floating point, enumeration, Boolean, string, time, and structure. The constraint rules set by the configuration layer may include constraint rules of the attribute field.

[0071] There are different constraint rules for different field types. For battery data scenarios, the main constraint rules include the following:

[0072] For integer, single-precision floating-point, and double-precision floating-point data, constraint rules include verifying whether the actual data types are consistent and whether the maximum and minimum values ​​meet the requirements.

[0073] For enumeration and Boolean data, constraint rules include verifying whether the actual data is contained in the defined range.

[0074] For time-type data, constraint rules include checking whether the time format conforms to a predefined format (timestamp, custom time format).

[0075] For structured data, the constraint rules include checking whether the data is of JSON structure.

[0076] In S402, the data to be processed is verified according to the constraint rules through the access layer, and the standard data is determined according to the verified qualified data.

[0077] The validity of the data to be processed can be checked according to the constraint rules of the attribute fields defined by the device model configured in the configuration layer. Data that fails the check can be set as dirty data labels. The subsequent mapping process can be facilitated according to the set data labels. For example, it is possible to prohibit the mapping of the dirty data standard to be processed data to the data layer to improve data processing efficiency.

[0078] When the configuration layer sets or defines a device model, it sends a change message notification to the access layer. When the access layer receives the change message notification, it can Figure 5 The method shown updates the locally cached device model.

[0079] In S501 , a device model change message is received.

[0080] The configuration layer can receive configuration information from the user. The configuration information can include configuration information for setting the field attributes of the model. After the user completes the configuration of the device model, it can be uploaded to the platform and a device model change message can be sent to the access layer or the data layer. At this time, the access layer or the data layer can receive the device model change message sent by the configuration layer. For example, the access layer can receive the configuration information about the message parsing script in the change message in the device model, and the data layer can receive the field information in the device model and update the data table according to the field information.

[0081] In S502, the HASH value of the updated data is calculated.

[0082] The HASH value of the overall data of the change message of the device model can be calculated. Due to the one-to-one correspondence between HASH and data content, when the change message changes value, the corresponding HASH value will also change.

[0083] In S503, it is determined whether the current update data is cached locally according to the HASH value.

[0084] The HASH value of the locally cached data may be compared with the HASH value of the device model that currently receives the change message to determine whether the two are consistent.

[0085] In S504, if the updated data of the device model is not cached locally, the updated data of the device model is cached locally.

[0086] If the HASH value of the device model corresponding to the received change message is different from the HASH value of the locally cached device model, due to the one-to-one correspondence between the HASH value and the device model, it means that the device model corresponding to the change message sent by the configuration layer is not cached locally, then the data of the device model corresponding to the change message can be cached to the access layer.

[0087] In S505, if it has been cached locally, the update is terminated.

[0088] If the HASH value of the device model corresponding to the received change message is the same as the HASH value of the locally cached device model, it means that the device model corresponding to the change message sent by the configuration layer has been cached locally.

[0089] In addition, the new version of the device model may have defects or loopholes when processing data. In order to reduce the data errors caused by the new version of the device model when processing data, it can be analyzed through the grayscale configuration method, such as Figure 6 As shown, the method of parsing through grayscale configuration includes:

[0090] In S601, the routing rules of different versions of the message parsing scripts set by the configuration layer are obtained through the access layer.

[0091] Among them, the routing rule refers to the path for transmitting the data to be processed to different versions of the message parsing script. The routing rule can be set in the configuration interface of the message parsing script through the configuration layer. For example, the HASH value of the predetermined field of the data to be processed (including data terminal identification, etc.) can be calculated, and the version of the message parsing script corresponding to the data to be processed can be determined according to the data range to which the HASH value belongs. Alternatively, the version of the message parsing script corresponding to the data to be processed can be determined according to the whitelist or blacklist corresponding to different versions of the message parsing script. Among them, the same data to be processed can be routed to one or more versions of the message parsing script. The same message parsing script can parse one or more data to be processed.

[0092] In S602, the access layer routes the data to be processed to the corresponding version of the message parsing script for parsing according to the routing rule.

[0093] According to the predetermined field HASH value of the data to be processed, or according to the whitelist or blacklist corresponding to the message parsing script, the data to be processed is routed to the corresponding message parsing script, and the data to be processed is parsed, including message header parsing conversion and message body parsing conversion.

[0094] In S103, the device model set in the configuration layer is obtained through the data layer, and the standard data is mapped to the data table corresponding to the device model.

[0095] The data layer updates the data table according to the change message, which can be done as follows Figure 7 As shown, including:

[0096] In S701, a change message of the device model is obtained through the data layer.

[0097] The configuration layer includes a device model definition interface, which can be used to define the device model. After the configuration layer completes the definition of the device model, a change message can be sent to the data layer. The data layer can update the data table in the data layer according to the change message.

[0098] In order to update the table in the data layer more accurately, when the change message of the device model is obtained, the type of the change message of the device model can also be determined according to the data identifier included in the change message of the device model.

[0099] For example, the types of the change message of the device model may include device attribute type, device event type, and device measurement data type.

[0100] Among them, the identifier field of the device event type can be used to distinguish different event types. For the device measurement data type, it can include a basic model of battery data, and the basic model of battery data can include basic fields of national standards. For example, the basic model of battery data can include at least one of device code, total voltage, cell voltage array, maximum voltage, minimum voltage, maximum voltage cell number, minimum voltage cell number, current, sensor temperature array, maximum temperature, minimum temperature, maximum temperature sensor number, minimum temperature sensor number, insulation resistance, SOC (full name of State of Charge in English, full name of state of charge in Chinese), cumulative mileage, vehicle status, and charging status.

[0101] The basic model of the battery data may be expanded to obtain a device model, including at least one of a device measurement data model, a device attribute model, and a device event model.

[0102] In S702, according to the change relationship between the change message of the device model and the existing device model, the data table corresponding to the device model is updated through the data layer, and the data table is used to map the standard data.

[0103] After the data layer receives the change message of the device model, the data layer may update the data table corresponding to the device model according to the change relationship between the change message of the device model and the existing device model.

[0104] When the device model change message is a change message of an existing device model, the data layer adds a new field in the existing model table according to the device model change message, and the existing model table is a data table mapped by the existing model.

[0105] When the change message of the device model is a change message of an unknown device model, the data layer creates a data table according to the change message of the device model, or determines a new field according to the change message of the device model and updates the data table according to the new field.

[0106] For example, the user expands on the basis of basic model A to obtain device model A1. The change message of the device model determines that the change message corresponds to the change of device model A1. Then, based on the existing data table B1 corresponding to the existing model A1, the fields corresponding to the change message can be added on the basis of the existing data table B1. If the change message is a change message of an unknown model, a data table can be created based on the fields of the change message, or the new fields can be determined based on the change message and added to the existing data table. Among them, the existing data table can include the basic model table corresponding to the basic model.

[0107] When the device model of the change message is of the device attribute model type, fields can be added directly according to the change message.

[0108] When the type of the device model of the change message is the setting event model, it can be determined whether the change message corresponds to the change of an existing event model. If it is a change of an existing event model, a calculation can be performed based on the change message to determine the newly added field to the existing event table relative to the existing event model, and the field is added based on the existing event table corresponding to the existing event model. If it is not a change message of an existing event model, a new data table can be created based on the change message.

[0109] When the type of the device model of the change message is the device measurement data model, it can be determined whether the fields included in the change message of the current device measurement data model are inherited from the basic model. If inherited from the basic model, the fields can be directly added to the basic model table corresponding to the basic model to obtain the data table corresponding to the change message.

[0110] If it is not inherited from the basic model, it can be determined whether it is a change to the existing device measurement data model. If it is a change to the existing device test data model, a field corresponding to the change message can be added based on the existing data table corresponding to the existing device test data model.

[0111] If the existing equipment test data model is obtained by updating the basic model, the fields required to be added in the change message can be added on the basis of the basic model table to obtain a data table corresponding to the change message.

[0112] If the device test data model change message is an unknown device test data model, the fields that need to be added can be determined by comparing the change message with the existing device test data table or the basic model table.

[0113] After updating the fields in the data table in the data layer through the change message of the device model, the message body of the data to be processed can be accurately and effectively mapped to the data table through the mapping relationship to obtain accurate mapping data. In addition, data with different access methods can be uniformly stored through the data layer, which is conducive to the subsequent use of data.

[0114] Figure 8 A schematic diagram of a data processing system for a platform provided in an embodiment of the present application. Figure 8 As shown, the system includes an access layer, a configuration layer and a data layer.

[0115] Among them, the configuration layer includes a device type management module, a device model configuration module and a message parsing script configuration module.

[0116] The device type management module may include information management such as basic information, description, and label of the device. The basic information of the device may include at least one of the device name, device creation time, and device modification time.

[0117] The device model configuration module includes at least one of a device attribute configuration module, a device measurement data configuration module, and a device event configuration module. The device model configuration module can use one or more basic attribute fields. The device event configuration includes an identifier field for device events to distinguish different event types. For the device measurement data configuration, a basic model of battery data is included. The basic model of battery data can include various fields defined by the national standard. For example, it can include at least one of the fields such as device code, total voltage, cell voltage array, maximum voltage, minimum voltage, maximum voltage cell number, minimum voltage cell number, current, sensor temperature array, maximum temperature, minimum temperature, maximum temperature sensor number, minimum temperature sensor number, insulation resistance, SOC, cumulative mileage, vehicle status, charging status, etc.

[0118] When configuring a device model, you can expand any required device measurement data model based on the basic model.

[0119] In the device attribute configuration module, the attribute type may include at least one of common integer type, single-precision floating point type, double-precision floating point type, enumeration type, Boolean type, string type, time type, and structure.

[0120] You can set constraint rules in the device attribute fields. There will be different constraint rules for different field types. For example, for battery data scenarios, the constraint rules for data of the following field types can be included: integer, single-precision floating point, and double-precision floating point data can verify whether the actual data type is consistent and whether the maximum and minimum values ​​meet the requirements; enumeration and Boolean data can verify whether the actual data is included in the defined range; time data can verify whether the time format conforms to the predefined format (timestamp, custom time format); the structure mainly verifies whether it is JSON structured data.

[0121] The above-mentioned device attributes, device measurement data and device events can be configured to be uniformly managed based on the device model version. When the user needs to upgrade the device model definition of the production environment, the user edits the published device model, and the system copies the content of the latest published device model definition to generate new device model definition version data. The newly generated device model definition version data is in draft status, and the version number is increased by 1 based on the original one. When the user completes the editing of the latest device model definition, the latest version can be published.

[0122] The message parsing script configuration module mainly includes the script content module, script verification module, script debugging module, script compilation module, script version module and grayscale configuration module. The script content module includes the message header conversion method and the message body conversion method. The function of the script content can be used to complete the data conversion of the message header and the message content.

[0123] The script verification module mainly performs basic syntax verification. If the script syntax verification fails, it cannot be submitted successfully.

[0124] The script compilation module is mainly used to compile the edited script into a binary executable file to improve the efficiency of execution.

[0125] The script version module is mainly used for the management of new and old versions. When the user needs to upgrade the script in the production environment, the user edits the published script, and the system generates a new script version based on the script content of the latest published version. The newly generated script version is in draft status, and the version number is increased by 1 based on the original one. When the user edits the logic of the script based on the newly generated script and completes the debugging and compilation of the script, the latest version can be published.

[0126] The grayscale configuration module mainly completes the ability to route traffic between different versions according to the field HASH modulus. By configuring specific field HASH routing rules, users can route the values ​​of specific fields to the new version, thereby completing the comparison test of new and old scripts, and finally achieving smooth launch of the script.

[0127] The data layer is used to establish relevant models for the device attributes, device measurement data, and device events defined in the device model, including operations such as creating data tables and adding new fields. The created data tables include device attribute model tables, device event model tables, and device measurement data model tables.

[0128] After the user completes the configuration of the device attributes, events, and attribute fields of the measurement data of the device model in the configuration layer and publishes them, the configuration layer will send a device model change message to notify the data layer. After the data layer receives the device model change message from the configuration layer, it creates a table or adds a new field.

[0129] The access layer includes data access module, parsing module and constraint module. The data access module completes the data input docking by adapting the http protocol, mq real-time message, offline csv import, OSS / MinIO / ftp, data extraction and other docking methods. The main process is as follows:

[0130] After receiving the battery data to be processed, the http protocol adapter module, the mq message middleware in the mq real-time message module, the csv import module, and the oss / MinIO / ftp docking module first convert the data to be processed into a standardized format of message header + message body.

[0131] The parsing module of the access layer obtains the configuration information of the device model published in the configuration layer from the cache, obtains the grayscale configuration information of the message parsing, and obtains the corresponding message parsing script configuration in the grayscale configuration information. According to the routing rules of the grayscale configuration, the different versions of the message parsing script corresponding to the data to be processed are determined.

[0132] The message header conversion script in the message parsing script converts the message header, thereby completing the mapping of the key in the message header of the data to be processed to the attribute key in the attribute field in the device model definition. For the fields that cannot be found in the message header conversion script, they can be passed down as is.

[0133] The message body conversion script in the message script can convert the content in the message body, including data preprocessing such as magnification, reduction, adding or subtracting some constant values.

[0134] The constraint module can perform logical checks on common fields according to the constraint rules in the attribute fields in the device model definition. Data that fails the check can be considered dirty data and labeled as dirty data for easy subsequent processing.

[0135] The verified data is formed into standard data of the device model, and the standard data can be written into the data table corresponding to the data layer.

[0136] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0137] Fig. 9 A schematic diagram of a data processing device provided in an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0138] The access unit 901 is used to access the data to be processed through the access layer.

[0139] The parsing unit 902 is used to obtain the message parsing script set by the configuration layer through the access layer, and convert the data to be processed into standard data.

[0140] The mapping unit 903 is used to obtain the device model set by the configuration layer through the data layer, and map the standard data to the data table corresponding to the device model.

[0141] Fig. 9 The data processing device shown, and Figure 2 The data processing method shown corresponds to.

[0142] Fig.10 Schematic diagram of a data processing device provided by an embodiment of the present application. Fig.10 As shown, the data processing device 10 of this embodiment includes: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100, such as a data processing program. When the processor 100 executes the computer program 102, the steps in the above-mentioned data processing method embodiments are implemented. Alternatively, when the processor 100 executes the computer program 102, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0143] Exemplarily, the computer program 102 may be divided into one or more modules / units, which are stored in the memory 101 and executed by the processor 100 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 102 in the data processing device 10.

[0144] The data processing device may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will appreciate that Fig.10 It is only an example of the data processing device 10 and does not constitute a limitation of the data processing device 10. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the data processing device may also include input and output devices, network access devices, buses, etc.

[0145] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0146] The memory 101 may be an internal storage unit of the data processing device 10, such as a hard disk or memory of the data processing device 10. The memory 101 may also be an external storage device of the data processing device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the data processing device 10. Further, the memory 101 may also include both an internal storage unit of the data processing device 10 and an external storage device. The memory 101 is used to store the computer program and other programs and data required by the data processing device. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0147] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0148] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0150] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0154] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: Based on the data processing system, the data processing system includes an access layer, a configuration layer and a data layer, and the method includes: Accessing the data to be processed through the access layer; Acquire the message parsing script set by the configuration layer through the access layer, and convert the data to be processed into standard data; The device model set by the configuration layer is obtained through the data layer, and the standard data is mapped to the data table corresponding to the device model.

2. The method according to claim 1, characterized in that Before mapping the standard data to the data table corresponding to the device model, the method includes: Obtaining a change message of a device model through the data layer; According to the change relationship between the change message of the device model and the existing device model, the data table corresponding to the device model is updated through the data layer, and the data table is used to map the standard data.

3. The method according to claim 2, characterized in that According to the change relationship between the change message of the device model and the existing device model, updating the data table corresponding to the device model through the data layer includes: Determine, through the data layer, the change relationship between the change message of the device model and the change of the existing device model; When the change message of the device model is a change message of an existing device model, a new field is added to the existing model table according to the change message of the device model through the data layer, and the existing model table is a data table mapped by the existing model; When the device model change message is a change message of an unknown device model, a data table is created according to the device model change message through the data layer, or a new field is determined according to the device model change message, and the data table is updated according to the new field.

4. The method according to claim 3, characterized in that After determining the change relationship between the change message of the device model and the existing device model through the data layer, the method further includes: When the fields included in the change message of the device model are inherited from the basic model, a new field is added to the basic model table according to the change message of the device model through the data layer.

5. The method according to claim 3, characterized in that: After acquiring the change message of the device model through the data layer, the method further includes: The type of the device model change message is determined according to the data identifier in the device model change message.

6. The method according to any one of claims 1 to 5, characterized in that: Obtaining the message parsing script set by the configuration layer through the access layer, and converting the data to be processed into standard data, including: Identify, through the access layer, a message header and a message body in the data to be processed; The access layer performs message header conversion and message body conversion on the identified data according to the message parsing script set by the configuration layer; The standard data is obtained through the access layer according to the converted message.

7. The method according to claim 6, characterized in that Obtaining the standard data according to the converted message through the access layer includes: Acquire the constraint rules set by the configuration layer through the access layer; The access layer verifies the data to be processed according to the constraint rules, and determines the standard data according to the verified data.

8. The method according to claim 6, characterized in that After the access layer identifies the message header and the message body in the data to be processed, and before the access layer performs message header conversion and message body conversion on the identified data according to the message parsing script set by the configuration layer, the method further includes: Acquire routing rules of different versions of message parsing scripts set by the configuration layer through the access layer; The access layer routes the data to be processed to the corresponding version of the message parsing script for parsing according to the routing rule.

9. The method according to claim 8, characterized in that The routing rules include HASH value routing rules and / or whitelist routing rules; Routing the data to be processed to the message parsing script of the corresponding version for parsing according to the routing rule through the access layer includes: Determine, by the access layer, the message parsing script of the corresponding version of the data to be processed according to the numerical range of the HASH value of the predetermined field of the data to be processed; Alternatively, the access layer determines the message parsing version of the principal and information corresponding to the data to be processed according to the white list corresponding to the version of the message parsing script.

10. The method according to any one of claims 1 to 9, characterized in that: Accessing the data to be processed through the access layer includes: The data to be processed is accessed through the access layer through at least one access method selected from the group consisting of http protocol adaptation, CSV import, mq real-time message access, oss docking, MinIO docking and ftp docking.

11. A data processing device, characterized in that: Based on the data processing system, the data processing system includes an access layer, a configuration layer and a data layer, and the device includes: An access unit, used for accessing the data to be processed through the access layer; A parsing unit, configured to obtain a message parsing script set by the configuration layer through the access layer, and convert the data to be processed into standard data; A mapping unit is used to obtain the device model set by the configuration layer through the data layer, and map the standard data to a data table corresponding to the device model.

12. A data processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images