Data analysis method, system and device and electronic equipment

Abstract

The invention provides a data analysis method, system and device and electronic equipment, and relates to the technical field of Internet of Things, the method is applied to data acquisition middleware in a data analysis system, the data analysis system comprises the data acquisition middleware and at least one service module, and the method comprises the following steps: receiving an original message uploaded by an equipment terminal, the original message is equipment point table information, the point table information is obtained through configuration based on a protocol model corresponding to the equipment terminal, and the protocol model corresponding to the equipment terminal is used for defining a data structure of a protocol; pulling a protocol model corresponding to the equipment terminal, and performing protocol analysis on an original message according to the original message and the equipment identifier uploaded by the equipment terminal to obtain an original attribute value; packaging the original attribute value into a JSON message of the equipment terminal, and issuing the JSON message to a target message queue through a kafka message queue; wherein the message theme of the target message queue is obtained based on the identifier of the target object model, and the universality of the data analysis method is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a data analysis method, system, device and electronic equipment. Background Art

[0002] As the application scenarios of the Internet of Things continue to expand, the types of connected devices are increasing. Different devices use different protocols, data types, and functions. The complexity of the Internet of Things platform has increased dramatically, bringing a huge burden to the platform construction. Data analysis for devices is also particularly important.

[0003] The conventional practice is to carry out customized development for data analysis of each device type. When there are many types of devices, there are problems such as large development workload, long development cycle, and difficulty in subsequent maintenance. To solve this problem, some products on the market use configurable data analysis templates to parse protocol data, but the data analysis templates only support data of specific types or formats, and the versatility of this data analysis method is poor. Summary of the invention

[0004] The present invention provides a data analysis method, system, device and electronic device to solve the defect of poor versatility of the data analysis method in the prior art and improve the versatility of the data analysis method.

[0005] In a first aspect, the present invention provides a data analysis method, which is applied to a data acquisition middleware in a data analysis system, wherein the data analysis system includes the data acquisition middleware and at least one business module; the method includes the following steps: Receive an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0006] According to a data parsing method provided by the present invention, when the protocol model corresponding to the device terminal is the address splicing protocol model, the original message uploaded by the device terminal includes at least one message message and a message address corresponding to at least one message message; Pulling the protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal to obtain the original attribute value of the device terminal includes: Matching the message addresses corresponding to the message packets; According to the message address corresponding to each of the message messages and the field order configured by the address splicing protocol model, the data of each field in each of the message addresses is read in sequence; The original attribute value of the device terminal is determined according to the data of each field in each message address.

[0007] According to a data parsing method provided by the present invention, the data of each field is sequentially read according to the message address corresponding to each message message and the field order configured by the address splicing protocol model, including: In the case where the original message uploaded by the device terminal is a message-nested original message, according to the first-level message address, the second-level message address and the field order configured by the address splicing protocol model included in each message message, the data of each field corresponding to the first-level message address and the data of each field corresponding to the second-level message address are read in sequence; The original attribute value of the device terminal is determined according to the data of each field corresponding to the primary message address and the data of each field corresponding to the secondary message address.

[0008] According to a data parsing method provided by the present invention, when the protocol model corresponding to the device terminal is the key-value pair protocol model, the original message uploaded by the device terminal includes a message message consisting of at least one group of key-value pairs; Pulling the protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal to obtain the original attribute value of the device terminal includes: Determining original attribute values ​​of each attribute field in the device terminal according to the key in each of the key-value pairs and the identifier predefined in the key-value pair protocol model; When it is determined that the value in the key-value pair is of an object or array type, obtaining a sub-protocol model corresponding to the value in the key-value pair; the sub-protocol model is used to define the value in the key-value pair; The original attribute value of the device terminal is determined according to the original attribute value of each attribute field in the device terminal and the sub-protocol model corresponding to the value in the key-value pair.

[0009] According to a data parsing method provided by the present invention, when the protocol model corresponding to the device terminal is the delimiter protocol model, the original message uploaded by the device terminal is a message message divided and arranged by the configured delimiter, and the delimiter includes a row delimiter and a field delimiter; the row delimiter is used to divide the parent model, and the field delimiter is used to divide the child model; Pulling the protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal to obtain the original attribute value of the device terminal includes: The original message is segmented according to the segmentation character to obtain a character array; The original attribute value of the device terminal is obtained by matching the attribute order with the character array subscript.

[0010] According to a data parsing method provided by the present invention, encapsulating the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal includes: Performing verification conversion on the original attribute value of the device terminal to obtain a conversion value of the device terminal; The conversion value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal.

[0011] In a second aspect, the present invention further provides a data analysis system, the data analysis system comprising an operation and maintenance management platform, a data collection middleware and at least one business module; The operation and maintenance management platform is used to maintain basic data; the basic data maintenance includes equipment management, object model management and protocol model management; the equipment management is used to maintain the object model bound to the terminal device; the object model management is used to maintain the model number, model name, corresponding protocol model, business and capability label of the object model; the protocol model management is used to define the data structure of the protocol; The data acquisition middleware is used to receive the original message uploaded by the device terminal; the original message is the device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal includes at least one of the following: address splicing protocol model, key-value pair protocol model, and separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; wherein the message subject of the target message queue is obtained based on the identifier of the target object model; Each of the business modules is used to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0012] According to the data analysis system provided by the present invention, the data analysis system further includes a visualization module; Each of the business modules is further used to search for a first object model that meets a specific tag condition; obtain device data of the first object model from a first message queue where a message topic corresponding to the identifier of the first object model is located according to the identifier of the first object model; the target object model is one or more of the first object models; The visualization module visualizes the device data according to the specific tag condition, the first object model, and the device data of the first object model.

[0013] In a third aspect, the present invention further provides a data analysis device, which is applied to data acquisition middleware in a data analysis system, wherein the data analysis system includes the data acquisition middleware and at least one business module; the device includes the following modules: A receiving module, used to receive an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; A data parsing module, used to pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0014] In a fourth aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the data analysis methods described above when executing the computer program.

[0015] In a fifth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the data parsing methods described above.

[0016] In a sixth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the data parsing methods described above.

[0017] The data analysis method, system, device and electronic device provided by the present invention are applied to data acquisition middleware in a data analysis system. The data analysis system includes data acquisition middleware and at least one business module. First, an original message uploaded by a device terminal is received, wherein the original message is device point table information, and the device point table information is obtained based on the configuration of a protocol model corresponding to the device terminal. The protocol model corresponding to the device terminal is used to define the data structure of the protocol. The protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; then, the protocol model corresponding to the device terminal is pulled, and the device terminal receives the data structure of the protocol according to the configuration of the protocol model corresponding to the device terminal. The uploaded original message and the device identifier of the device terminal are used to perform protocol parsing on the original message to obtain the original attribute value of the device terminal; then, the original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, and the JSON message of the device terminal is published to the target message queue through the Kafka message queue; wherein, the message subject of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0018] The protocol model in the present invention is used to define the data structure of the protocol. The protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol. The model original message (device point table information) is obtained based on the configuration of the protocol model corresponding to the device terminal, and a custom general data structure is realized. Then, by pulling the protocol model corresponding to the device terminal, the original message uploaded by the device terminal is parsed to obtain the original attribute value of the device terminal. The original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, and published to the target message queue through the kafka message queue. The present invention realizes the conversion of device point table information into a protocol model through a custom general data structure, and the versatility of the data parsing method is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of the data analysis method provided by the present invention.

[0021] Figure 2 It is one of the structural schematic diagrams of the data analysis system provided by the present invention.

[0022] Figure 3 A protocol model for fire alarm monitoring provided by the present invention.

[0023] Figure 4a This is one of the protocol models of the access control and attendance system provided by the present invention.

[0024] Figure 4b This is the second protocol model of the access control and attendance system provided by the present invention.

[0025] Figure 5 This is the attribute definition page of the address splicing protocol model provided by the present invention.

[0026] Figure 6a It is one of the attribute definition pages of the address key-value pair protocol model provided by the present invention.

[0027] Figure 6b This is the second attribute definition page of the address key-value pair protocol model provided by the present invention.

[0028] Figure 7 This is the second structural diagram of the data analysis system provided by the present invention.

[0029] Figure 8 It is a schematic diagram of the effect of the equipment management provided by the present invention.

[0030] Fig. 9 It is a schematic diagram of the effect of the object model management provided by the present invention.

[0031] Fig.10 It is a schematic diagram of the effect of the protocol model management provided by the present invention.

[0032] Fig.11 It is a structural schematic diagram of the data analysis device provided by the present invention.

[0033] Fig.12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Combine the following Figure 1-Figure 12The data analysis method, system, device and electronic device of the present invention are described.

[0036] Figure 1 is a flow chart of a data analysis method provided by the present invention, the method is applied to a data acquisition middleware in a data analysis system, the data analysis system includes a data acquisition middleware and at least one business module; Figure 1 As shown, the method includes the following: Step 101, receiving an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of a protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define a data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; Specifically, it should be noted that the execution subject of this embodiment is the data acquisition middleware in the data analysis system, which is used to realize the conversion of the equipment point table information into a universal data structure, thereby improving the universality of the data analysis method.

[0037] in, Figure 2 is one of the structural diagrams of the data analysis system provided by the present invention, such as Figure 2 As shown, the data analysis system includes data collection middleware and at least one business module. The data analysis system also includes an operation and maintenance management platform, a device terminal, multiple business modules, and a visualization page. The operation and maintenance management platform is responsible for the maintenance of basic data, including functions such as device management, physical model management, and protocol model management. The device management maintains the terminal devices accessing the platform. Each terminal device is bound to a physical model. The physical model management maintains the physical model used by the platform. The physical model structure consists of three parts: model number, model name and other basic information, protocol model, and business and capability labels. The protocol model management maintains the protocol model defined by the platform. The protocol model defines the data structure of the protocol.

[0038] Correspondingly, in this embodiment, the process of data collection middleware implementing data parsing is as follows: First, the data acquisition middleware receives the original message uploaded by the device terminal. The data acquisition middleware consists of a communication management model, a data parsing module, and a data publishing module. The original message is the device point table information, which is obtained based on the configuration of the protocol model corresponding to the device terminal. For example, the communication management module can be used to maintain communication with the device, such as receiving the device point table information uploaded by the device terminal.

[0039] The present invention customizes a universal protocol model structure, and the protocol model corresponding to the device terminal is used to define the data structure of the protocol. The protocol model corresponding to the device terminal in the present invention includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model.

[0040] For example, Table 1 is a point table structure of a fire alarm monitoring system provided by the present invention. Figure 3 A protocol model for fire alarm monitoring provided by the present invention.

[0041] Table 1:

[0042] As shown in Table 1, the original message is in binary format, and each message is 14 bytes long. Bytes 0-4 represent the temperature of the monitoring point, bytes 4-8 represent the humidity of the monitoring point, bytes 8-12 represent the smoke concentration, bytes 13 represent the smoke concentration of the monitoring point, and bytes 14 represent the spray status of the detection point.

[0043] The uploaded temperature value is 50 + temperature in degrees Celsius * 10. For example, if the temperature is 23.5 degrees, 50 + 23.5×10=285 will be uploaded. The temperature detection range is -50℃-1000℃, so the uploaded original value range is 0-10500.

[0044] The value uploaded by relative humidity is relative humidity*10. For example, if the relative humidity is 55.0%, 550 will be uploaded. The relative humidity detection range is 0%-100%, so the original value uploaded range is 0-1000.

[0045] The uploaded value of smoke concentration will not be modified. For example, if the smoke concentration is 700ppm, the uploaded value will be 700. The smoke detection range is 0-10000ppm, so the original value range uploaded is 0-10000.

[0046] When the alarm is triggered, the alarm status value is 1, otherwise it is 0.

[0047] When the sprinkler is on, the sprinkler status is 1, otherwise it is 0.

[0048] like Figure 3 As shown, the protocol model type is splicing by address, and the protocol model is as follows. Since the device protocol does not design a message ID, the primary address and the secondary address are empty.

[0049] For example, Table 2 is a point table structure of a certain access control and attendance system provided by the present invention. Figure 4a One of the protocol models of the access control and attendance system provided by the present invention, Figure 4b This is the second protocol model of the access control and attendance system provided by the present invention.

[0050] Table 2:

[0051] The access control system uses a comma-separated values ​​(CSV) structure to regularly upload personnel entry and exit information to the platform in batches. For example, "1261, Zhang San, 1, 09:01:02CRLF1275, Li Si, 2, 09:01:12CRLF1311, Wang Wu, 1, 09:01:42" includes the entry and exit information of three people at 9:01. CRLF is a carriage return and line feed character.

[0052] Therefore, it is necessary to use two protocol models, the parent and child. The protocol types are both delimiter protocol models. The delimiter type of the parent protocol model is CRLF, and the delimiter of the child protocol model is comma. The child protocol model - the personnel attendance information sub-protocol model - is referenced in the parent protocol model.

[0053] Step 102: Pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal; Specifically, after receiving the original message uploaded by the device terminal, the data parsing module can be used to perform data parsing, for example, to pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal. The protocol model corresponding to the device terminal is predetermined, because the terminal device is bound to the object model, and the object model is bound to its corresponding protocol model, so that the protocol model corresponding to the device terminal can be determined (that is, the data structure definition).

[0054] After pulling the corresponding protocol model, the original message can be parsed by referring to the definition of the data structure in the protocol model corresponding to the device terminal to obtain the original attribute value of the device terminal, thereby realizing universal data parsing with strong versatility.

[0055] Step 103: Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0056] Specifically, after parsing the original attribute value of the device terminal, the parsed original attribute value of the data can be further encapsulated to facilitate subsequent data publishing. The data publishing process can use the data module of the data acquisition middleware. The data publishing module publishes data through the kafka message queue, thereby completing data publishing and facilitating subscription by various business modules.

[0057] The topic of the message queue is obtained based on the identifier of the target object model, for example, the topic is the object model id. The device data includes the key device identifier and the JSON message encapsulated by the value.

[0058] The method provided in this embodiment is applied to data acquisition middleware in a data analysis system. The data analysis system includes data acquisition middleware and at least one business module. First, an original message uploaded by a device terminal is received, wherein the original message is device point table information, and the device point table information is obtained based on the configuration of a protocol model corresponding to the device terminal. The protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; then, the protocol model corresponding to the device terminal is pulled, and the original message is parsed according to the original message uploaded by the device terminal and the device identifier of the device terminal to obtain the original attribute value of the device terminal; then, the original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, and the JSON message of the device terminal is published to the target message queue through a kafka message queue; wherein the message subject of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0059] The protocol model in the present invention is used to define the data structure of the protocol. The protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol. The model original message (device point table information) is obtained based on the configuration of the protocol model corresponding to the device terminal, and a custom general data structure is realized. Then, by pulling the protocol model corresponding to the device terminal, the original message uploaded by the device terminal is parsed to obtain the original attribute value of the device terminal. The original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, and published to the target message queue through the kafka message queue. The present invention realizes the conversion of device point table information into a protocol model through a custom general data structure, and the versatility of the data parsing method is improved.

[0060] According to a data parsing method provided by the present invention, when the protocol model corresponding to the device terminal is an address splicing protocol model, the original message uploaded by the device terminal includes at least one message message and a message address corresponding to at least one message message; Pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal, including: Match the message address corresponding to each message message; According to the message address corresponding to each message message and the field order configured by the address splicing protocol model, read the data of each field in each message address in sequence; Determine the original attribute value of the device terminal based on the data in each field of each message address.

[0061] Specifically, in some embodiments, the protocol model corresponding to the device terminal is an address splicing protocol model. When the protocol model corresponding to the device terminal is the address splicing protocol model, the original message uploaded by the device terminal includes at least one message message and at least one message address corresponding to the message message.

[0062] For example, Figure 5 This is the attribute definition page of the address splicing protocol model provided by the present invention, such as Figure 5 As shown, the address concatenation protocol model uses a binary message format of the form "message A address + message A message + message B address + message B message + ...".

[0063] Specifically, first, match the message address corresponding to each message message; then, read the data of each field in each message address in turn according to the message address corresponding to each message message and the field order configured by the address splicing protocol model; further, the original attribute value of the device terminal can be determined based on the data of each field in each message address.

[0064] For example, in the process of processing the original message, the original value of the attribute can be obtained by matching the message address and reading the data of the corresponding byte (bit) length in sequence according to the configured field order.

[0065] The method provided in this embodiment, when the protocol model corresponding to the device terminal is an address splicing protocol model, the original message uploaded by the device terminal includes at least one message message and at least one message address corresponding to the message message, and by matching the message address corresponding to each message message, the data of each field in each message address is read in sequence according to the message address corresponding to each message message and the field order configured by the address splicing protocol model; further, according to the data of each field in each message address, the original attribute value of the device terminal is determined to realize data analysis. The data analysis method of the present invention has strong versatility.

[0066] According to a data parsing method provided by the present invention, data of each field is read in sequence according to the message address corresponding to each message message and the field order configured by the address splicing protocol model, including: In the case where the original message uploaded by the device terminal is a message-nested original message, the data of each field corresponding to the first-level message address and the data of each field corresponding to the second-level message address are read in sequence according to the first-level message address, the second-level message address and the field order configured by the address splicing protocol model included in each message message; The original attribute value of the device terminal is determined according to the data of each field corresponding to the primary message address and the data of each field corresponding to the secondary message address.

[0067] Specifically, in some embodiments, the process of sequentially reading the data of each field according to the message address corresponding to each message message and the field order configured by the address splicing protocol model can be implemented by the following steps, including: First, in the case where the original message uploaded by the device terminal is a message-nested original message, the data of each field corresponding to the first-level message address and the data of each field corresponding to the second-level message address are read in sequence according to the first-level message address, the second-level message address and the field sequence configured by the address splicing protocol model included in each message message. Further, the original attribute value of the device terminal is determined according to the data of each field corresponding to the first-level message address and the data of each field corresponding to the second-level message address.

[0068] For example, the message body of message A contains two sub-messages A1 and A2. For example, for a point table containing nested messages, such as "message A address + (message A1 address + message A1 message + message A2 address + message A2 message)", the present invention adopts a secondary address method to implement parsing of the point table containing nested messages. In the above-mentioned message, the message A address and the message B address are primary addresses, and the message A1 address and the message A2 address are secondary addresses.

[0069] In the method provided in this embodiment, the original message uploaded to the device terminal is an original message with nested messages. The present invention adopts a secondary address method to implement parsing of the point table containing nested messages, thereby improving the accuracy of data parsing.

[0070] According to a data parsing method provided by the present invention, when the protocol model corresponding to the device terminal is a key-value pair protocol model, the original message uploaded by the device terminal includes a message message consisting of at least one set of key-value pairs; Pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal, including: Determine the original attribute value of each attribute field in the device terminal according to the key in each key-value pair and the identifier predefined in the key-value pair protocol model; When it is determined that the value in the key-value pair is an object or array type, the sub-protocol model corresponding to the value in the key-value pair is obtained; the sub-protocol model is used to define the value in the key-value pair; The original attribute value of the device terminal is determined according to the original attribute value of each attribute field in the device terminal and the sub-protocol model corresponding to the value in the key-value pair.

[0071] Specifically, in some embodiments, the protocol model corresponding to the device terminal is a key-value pair protocol model. In this case, the original message uploaded by the device terminal includes a message message composed of at least one group of key-value pairs.

[0072] For example, Figure 6a It is one of the attribute definition pages of the address key-value pair protocol model provided by the present invention, such as Figure 6a As shown, the key-value pair protocol model is used to define a message message consisting of a series of key-value pairs, such as JSON and Protobuf formats.

[0073] Correspondingly, the process of pulling the protocol model corresponding to the device terminal in step 102 and performing protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal can be implemented by the following steps, including: First, the original attribute value of each attribute field in the device terminal is determined according to the key in each key-value pair and the identifier predefined in the key-value pair protocol model. Specifically, the original attribute value of the attribute is obtained by matching the key of the original message with the identifier defined in the protocol.

[0074] Furthermore, when it is determined that the value in the key-value pair is an object or array type, the sub-protocol model corresponding to the value in the key-value pair is obtained, wherein the sub-protocol model is used to define the value in the key-value pair. For example, for a key whose value is a JSON object or array, the present invention completes the definition of the JSON object or array by selecting the data type as an object or array and referencing the sub-protocol model. Figure 6b This is the second attribute definition page of the address key-value pair protocol model provided by the present invention.

[0075] According to the original attribute values ​​of each attribute field in the device terminal and the sub-protocol model corresponding to the value in the key-value pair, the original attribute value of the device terminal is determined to realize data parsing.

[0076] The method provided in this embodiment, when the protocol model corresponding to the device terminal is a key-value pair protocol model, the original message uploaded by the device terminal includes a message message composed of at least one group of key-value pairs, and the data parsing process determines the original attribute value of each attribute field in the device terminal according to the key in each key-value pair and the identifier predefined in the key-value pair protocol model; then, when it is determined that the value in the key-value pair is an object or array type, the sub-protocol model corresponding to the value in the key-value pair is obtained; the sub-protocol model is used to define the value in the key-value pair; and then, according to the original attribute value of each attribute field in the device terminal and the sub-protocol model corresponding to the value in the key-value pair, the original attribute value of the device terminal is determined. The present invention customizes the universal protocol model structure, converts the content of the protocol point table into a protocol model, and changes the field definition in the protocol without changing the code, thereby improving the efficiency of data parsing.

[0077] According to a data parsing method provided by the present invention, when the protocol model corresponding to the device terminal is a delimiter protocol model, the original message uploaded by the device terminal is a message message divided and arranged by the configured delimiter, and the delimiter includes a row delimiter and a field delimiter; the row delimiter is used to divide the parent model, and the field delimiter is used to divide the child model; Pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal, including: Split the original message according to the separator to obtain a character array; According to the order of the attributes, the original attribute value of the device terminal is obtained by matching the character array subscript.

[0078] Specifically, in some embodiments, the protocol model corresponding to the device terminal is a delimiter protocol model, such as Figure 6aAs shown, the delimiter protocol type model is used to define a message in which each field is separated and arranged by a specific delimiter. When the protocol model corresponding to the device terminal is the delimiter protocol model, the original message uploaded by the device terminal is a message message separated and arranged by the configured delimiter.

[0079] Correspondingly, the data analysis process in step 102 can be implemented through the following steps, including: First, the original message is segmented according to the separator to obtain a character array. In the present invention, there are two types of separators, namely, a row separator and a field separator, wherein the row separator is used to segment the parent model, and the field separator is used to segment the child model. Furthermore, the original attribute value of the device terminal can be obtained by matching the attribute sorting with the character array subscript.

[0080] Exemplarily, the data parsing module in the middle of data collection divides the original message according to the configured delimiters (row delimiters and field delimiters) to obtain a character array, and then matches the array subscript according to the order of the attributes to obtain the original value of the attribute. It supports nested delimiters. For table data in CSV format, it contains two types of delimiters, row delimiters (carriage return and line feed characters) and field delimiters (commas). The present invention adopts two parent-child protocol models. The delimiter of the parent model uses a row delimiter, the data type selects an array, and references the child protocol model. The delimiter of the child model uses a field delimiter. In the child protocol model, the identifiers of the attributes of each column of CSV, the definition of data types, units, etc. are completed.

[0081] The method provided in this embodiment converts the content of the protocol point table into a protocol model by customizing the general protocol model structure, and changes the field definition in the protocol without changing the code. The data parsing efficiency is improved and the data parsing method has strong versatility.

[0082] According to a data parsing method provided by the present invention, the original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, including: Verify and convert the original attribute value of the device terminal to obtain the conversion value of the device terminal; The conversion value, attribute name, field identifier, and data unit of the device terminal are encapsulated into the JSON message of the device terminal.

[0083] Specifically, in some embodiments, the process of encapsulating the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal in step 103 can be implemented by the following steps, including: First, the original attribute value of the device terminal is verified and converted to obtain the conversion value of the device terminal. The purpose of the verification conversion is to make the data more standardized and in line with the original value range. For example, the original attribute value of the device terminal is verified after the original value range using the data parsing module of the data acquisition middleware, and the data is converted according to the precision, offset value, and decimal places to obtain the conversion value of the device terminal.

[0084] Furthermore, the conversion value, attribute name, field identifier, and data unit of the device terminal may be encapsulated into a JSON message of the device terminal to implement data encapsulation, thereby facilitating subsequent publishing to the corresponding target message queue.

[0085] The method provided in this embodiment can further verify and convert the original attribute value of the device terminal after parsing to obtain the original attribute of the device terminal, obtain the converted value of the device terminal, and then encapsulate the converted value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal. The present invention realizes the encapsulation of the converted value, which is convenient for subsequent publishing to the corresponding target message queue so that the business module can understand the data.

[0086] Figure 7 This is the second structural diagram of the data analysis system provided by the present invention. Figure 7 As shown, the data analysis system includes an operation and maintenance management platform 710, a data collection middleware 720 and at least one business module 730; The operation and maintenance management platform 710 is used to maintain basic data; the basic data maintenance includes device management, object model management and protocol model management; the device management is used to maintain the object model bound to the terminal device; the object model management is used to maintain the model number, model name, corresponding protocol model, business and capability label of the object model; the protocol model management is used to define the data structure of the protocol; Specifically, the present invention provides a data analysis system, which includes an operation and maintenance management platform 710 , a data acquisition middleware 720 and at least one business module 730 .

[0087] The operation and maintenance management platform 710 is used to maintain basic data; the maintenance of basic data includes equipment management, object model management and protocol model management.

[0088] Among them, 1) device management is used to maintain the object model bound to the terminal device. For example, Figure 8 It is a schematic diagram of the effect of the device management provided by the present invention, such as Figure 8 As shown, the device management interface maintains terminal devices accessing the platform, and each terminal device is bound to a thing model.

[0089] 2) Object model management is used to maintain the model number, model name, corresponding protocol model, service and capability label of the object model. For example, Fig. 9 It is a schematic diagram of the effect of the object model management provided by the present invention, such as Fig. 9 As shown in the figure, the object model management interface maintains the object model used by the platform. The object model structure consists of three parts: model number, model name and other basic information, protocol model, and business and capability labels.

[0090] 3) The protocol model manages the data structures used to define the protocol. For example, Fig.10 It is a schematic diagram of the effect of the protocol model management provided by the present invention, such as Fig.10 As shown, the protocol model management interface maintains the protocol model defined by the platform, and the protocol model defines the data structure of the protocol.

[0091] The data acquisition middleware 720 is used to receive the original message uploaded by the device terminal; the original message is the device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal includes at least one of the following: address splicing protocol model, key-value pair protocol model, and separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; wherein the message subject of the target message queue is obtained based on the identifier of the target object model; Specifically, the data acquisition middleware 720 is used to realize data reception, data analysis and data publishing. The data acquisition middleware is composed of a communication management model, a data analysis module and a data publishing module. Specifically, first, the communication management module is used to receive the original message uploaded by the device terminal. The original message is the device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal includes at least one of the following: address splicing protocol model, key-value pair protocol model, and separator protocol model. For example, Netty is used to implement tcp communication, and EMQX+Paho is used to implement mqtt communication. When the communication management module receives the data uploaded by the device, it presents the original message and the device identifier to the data analysis module.

[0092] Furthermore, the data parsing module pulls the protocol model corresponding to the device terminal, performs protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtains the original attribute value of the device terminal.

[0093] Furthermore, the data publishing module encapsulates the original attribute value, attribute name, field identifier, and data unit of the device terminal into the JSON message of the device terminal, and publishes the JSON message of the device terminal to the target message queue through the Kafka message queue; wherein, the message topic of the target message queue is obtained based on the identifier of the target object model.

[0094] Each of the business modules 730 is used to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0095] Specifically, each of the business modules 730 is used to subscribe to the device data in the target message queue according to the identifier of the target object model, and the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0096] The data analysis system in this embodiment includes an operation and maintenance management platform, a data acquisition middleware and at least one business module. The operation and maintenance management platform maintains basic data, including device management, object model management and protocol model management; device management maintains the object model bound to the terminal device; object model management is used to maintain the model number, model name, corresponding protocol model, business and capability tags of the object model; protocol model management is used to define the data structure of the protocol; the data acquisition middleware is used to receive the original message uploaded by the device terminal, wherein the original message is the device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal, and the protocol model corresponding to the device terminal includes at least one of the following: address splicing protocol model, key-value pair Protocol model, delimiter protocol model; pull the protocol model corresponding to the device terminal, parse the original message according to the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal; encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into the JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the kafka message queue; wherein, the message subject of the target message queue is obtained based on the identifier of the target object model; each business module subscribes to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0097] The protocol model in the present invention is used to define the data structure of the protocol. The protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol. The model original message (device point table information) is obtained based on the configuration of the protocol model corresponding to the device terminal, and a custom general data structure is realized. Then, by pulling the protocol model corresponding to the device terminal, the original message uploaded by the device terminal is parsed to obtain the original attribute value of the device terminal. The original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, and published to the target message queue through the kafka message queue. The present invention realizes the conversion of device point table information into a protocol model through a custom general data structure, and the versatility of the data parsing method is improved.

[0098] According to the data analysis system 700 provided by the present invention, the data analysis system 700 further includes a visualization module; Each of the business modules 730 is further configured to search for a first object model that meets a specific tag condition; obtain device data of the first object model from a first message queue where a message topic corresponding to the identifier of the first object model is located according to the identifier of the first object model; the target object model is one or more of the first object models; The visualization module visualizes the device data according to the specific tag condition, the first object model, and the device data of the first object model.

[0099] Specifically, in some embodiments, the data analysis system 700 further includes a visualization module. The visualization module is used to provide different visualization effects and different operations for the device according to the physical model of the device and its label.

[0100] Each of the business modules 730 searches for a first object model that meets a specific tag condition; then, based on the identifier of the first object model, obtains the device data of the first object model from the first message queue where the message topic corresponding to the identifier of the first object model is located, wherein the target object model is one or more of the first object models.

[0101] Furthermore, the visualization module visualizes the device data according to the specific tag condition, the first object model, and the device data of the first object model.

[0102] In the data analysis system provided by the present invention, each business module searches for a set of object models that meet specific label conditions, obtains data of these object models, and performs corresponding business processing. The visualization interface provides different visualization effects for the device according to the device's object model and its label, and provides different operations, thereby providing a better user experience.

[0103] The data analysis device provided by the present invention is described below. The data analysis device described below and the data analysis method described above can be referenced to each other.

[0104] Fig.11 1 is a schematic diagram of the structure of the data analysis device provided by the present invention, which is applied to the data acquisition middleware in the data analysis system. The data analysis system includes the data acquisition middleware and at least one business module. The data analysis device 1100 includes the following modules: The receiving module 1110 is used to receive the original message uploaded by the device terminal; the original message is the device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; The data parsing module 1120 is used to pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0105] The device provided in this embodiment is applied to data acquisition middleware in a data analysis system. The data analysis system includes data acquisition middleware and at least one business module. First, the receiving module 1110 receives the original message uploaded by the device terminal, wherein the original message is the device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal. The protocol model corresponding to the device terminal is used to define the data structure of the protocol. The protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; then, the data analysis module 1120 pulls the protocol model corresponding to the device terminal, and according to the device terminal, The uploaded original message and the device identifier of the device terminal are used to perform protocol parsing on the original message to obtain the original attribute value of the device terminal; then, the original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, and the JSON message of the device terminal is published to the target message queue through the Kafka message queue; wherein, the message subject of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0106] The protocol model in the present invention is used to define the data structure of the protocol. The protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol. The model original message (device point table information) is obtained based on the configuration of the protocol model corresponding to the device terminal, and a custom general data structure is realized. Then, by pulling the protocol model corresponding to the device terminal, the original message uploaded by the device terminal is parsed to obtain the original attribute value of the device terminal. The original attribute value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal, and published to the target message queue through the kafka message queue. The present invention realizes the conversion of device point table information into a protocol model through a custom general data structure, and the versatility of the data parsing method is improved.

[0107] According to the data parsing device 1100 provided by the present invention, when the protocol model corresponding to the device terminal is the address splicing protocol model, the original message uploaded by the device terminal includes at least one message message and a message address corresponding to at least one message message; The data analysis module 1120 is specifically used for: Matching the message addresses corresponding to the message packets; According to the message address corresponding to each of the message messages and the field order configured by the address splicing protocol model, the data of each field in each of the message addresses is read in sequence; The original attribute value of the device terminal is determined according to the data of each field in each message address.

[0108] According to the data analysis device 1100 provided by the present invention, the data analysis module 1120 is further used for: In the case where the original message uploaded by the device terminal is a message-nested original message, according to the first-level message address, the second-level message address and the field order configured by the address splicing protocol model included in each message message, the data of each field corresponding to the first-level message address and the data of each field corresponding to the second-level message address are read in sequence; The original attribute value of the device terminal is determined according to the data of each field corresponding to the primary message address and the data of each field corresponding to the secondary message address.

[0109] According to the data parsing device 1100 provided by the present invention, when the protocol model corresponding to the device terminal is the key-value pair protocol model, the original message uploaded by the device terminal includes a message message composed of at least one set of key-value pairs; The data analysis module 1120 is specifically used for: Determining original attribute values ​​of each attribute field in the device terminal according to the key in each of the key-value pairs and the identifier predefined in the key-value pair protocol model; When it is determined that the value in the key-value pair is of an object or array type, obtaining a sub-protocol model corresponding to the value in the key-value pair; the sub-protocol model is used to define the value in the key-value pair; The original attribute value of the device terminal is determined according to the original attribute value of each attribute field in the device terminal and the sub-protocol model corresponding to the value in the key-value pair.

[0110] According to the data parsing device 1100 provided by the present invention, when the protocol model corresponding to the device terminal is the delimiter protocol model, the original message uploaded by the device terminal is a message message divided and arranged by the configured delimiter, and the delimiter includes a row delimiter and a field delimiter; the row delimiter is used to divide the parent model, and the field delimiter is used to divide the child model; The data analysis module 1120 is specifically used for: The original message is segmented according to the segmentation character to obtain a character array; The original attribute value of the device terminal is obtained by matching the attribute order with the character array subscript.

[0111] According to a data analysis method provided by the present invention, the data analysis module is further used for: Performing verification conversion on the original attribute value of the device terminal to obtain a conversion value of the device terminal; The conversion value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal.

[0112] Fig.12 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.12 As shown, the electronic device may include: a processor 1210, a communications interface 1220, a memory 1230 and a communication bus 1240, wherein the processor 1210, the communications interface 1220 and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call the logic instructions in the memory 1230 to execute the data analysis method, which is applied to the data acquisition middleware in the data analysis system, and the data analysis system includes the data acquisition middleware and at least one business module; the method includes: Receive an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0113] In addition, the logic instructions in the above-mentioned memory 1230 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the data analysis method provided by the above methods, the method is applied to a data acquisition middleware in a data analysis system, the data analysis system includes the data acquisition middleware and at least one business module; the method includes: Receive an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0115] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data analysis method provided by the above methods is implemented, and the method is applied to a data acquisition middleware in a data analysis system, wherein the data analysis system includes the data acquisition middleware and at least one business module; the method includes: Receive an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data analysis method, characterized in that: A data acquisition middleware applied to a data analysis system, wherein the data analysis system comprises the data acquisition middleware and at least one business module; the method comprises: Receive an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

2. The data analysis method according to claim 1, characterized in that: In the case where the protocol model corresponding to the device terminal is the address splicing protocol model, the original message uploaded by the device terminal includes at least one message message and a message address corresponding to at least one message message; Pulling the protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal to obtain the original attribute value of the device terminal includes: Matching the message addresses corresponding to the message packets; According to the message address corresponding to each of the message messages and the field order configured by the address splicing protocol model, the data of each field in each of the message addresses is read in sequence; The original attribute value of the device terminal is determined according to the data of each field in each message address.

3. The data analysis method according to claim 1, characterized in that: The step of sequentially reading data of each field according to the message address corresponding to each message message and the field order configured by the address splicing protocol model includes: In the case where the original message uploaded by the device terminal is a message-nested original message, according to the first-level message address, the second-level message address and the field order configured by the address splicing protocol model included in each message message, the data of each field corresponding to the first-level message address and the data of each field corresponding to the second-level message address are read in sequence; The original attribute value of the device terminal is determined according to the data of each field corresponding to the primary message address and the data of each field corresponding to the secondary message address.

4. The data analysis method according to claim 1, characterized in that: In the case where the protocol model corresponding to the device terminal is the key-value pair protocol model, the original message uploaded by the device terminal includes a message message consisting of at least one set of key-value pairs; Pulling the protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal to obtain the original attribute value of the device terminal includes: Determining original attribute values ​​of each attribute field in the device terminal according to the key in each of the key-value pairs and the identifier predefined in the key-value pair protocol model; When it is determined that the value in the key-value pair is of an object or array type, obtaining a sub-protocol model corresponding to the value in the key-value pair; the sub-protocol model is used to define the value in the key-value pair; The original attribute value of the device terminal is determined according to the original attribute value of each attribute field in the device terminal and the sub-protocol model corresponding to the value in the key-value pair.

5. The data analysis method according to claim 1, characterized in that: In the case where the protocol model corresponding to the device terminal is the delimiter protocol model, the original message uploaded by the device terminal is a message message divided and arranged by the configured delimiter, and the delimiter includes a row delimiter and a field delimiter; the row delimiter is used to divide the parent model, and the field delimiter is used to divide the child model; Pulling the protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal to obtain the original attribute value of the device terminal includes: The original message is segmented according to the segmentation character to obtain a character array; The original attribute value of the device terminal is obtained by matching the attribute order with the character array subscript.

6. The data analysis method according to any one of claims 1 to 5, characterized in that: The encapsulating the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal includes: Performing verification conversion on the original attribute value of the device terminal to obtain a conversion value of the device terminal; The conversion value, attribute name, field identifier, and data unit of the device terminal are encapsulated into a JSON message of the device terminal.

7. A data analysis system, characterized in that: The data analysis system includes an operation and maintenance management platform, a data collection middleware and at least one business module; The operation and maintenance management platform is used to maintain basic data; the basic data maintenance includes equipment management, object model management and protocol model management; the equipment management is used to maintain the object model bound to the terminal device; the object model management is used to maintain the model number, model name, corresponding protocol model, business and capability label of the object model; the protocol model management is used to define the data structure of the protocol; The data collection middleware is used to receive the original message uploaded by the device terminal; The original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; Pulling a protocol model corresponding to the device terminal, performing protocol parsing on the original message uploaded by the device terminal and a device identifier of the device terminal to obtain an original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; wherein the message subject of the target message queue is obtained based on the identifier of the target object model; Each of the business modules is used to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

8. The data analysis system according to claim 7, characterized in that: The data analysis system also includes a visualization module; Each of the business modules is further used to search for a first object model that meets a specific tag condition; obtain device data of the first object model from a first message queue where a message topic corresponding to the identifier of the first object model is located according to the identifier of the first object model; the target object model is one or more of the first object models; The visualization module visualizes the device data according to the specific tag condition, the first object model, and the device data of the first object model.

9. A data analysis device, characterized in that: A data acquisition middleware applied to a data analysis system, wherein the data analysis system comprises the data acquisition middleware and at least one business module; the device comprises: A receiving module, used to receive an original message uploaded by a device terminal; the original message is device point table information, and the device point table information is obtained based on the configuration of the protocol model corresponding to the device terminal; the protocol model corresponding to the device terminal is used to define the data structure of the protocol, and the protocol model corresponding to the device terminal includes at least one of the following: an address splicing protocol model, a key-value pair protocol model, and a separator protocol model; A data parsing module, used to pull the protocol model corresponding to the device terminal, perform protocol parsing on the original message uploaded by the device terminal and the device identifier of the device terminal, and obtain the original attribute value of the device terminal; Encapsulate the original attribute value, attribute name, field identifier, and data unit of the device terminal into a JSON message of the device terminal, and publish the JSON message of the device terminal to the target message queue through the Kafka message queue; Among them, the message topic of the target message queue is obtained based on the identifier of the target object model; the identifier of the target object model is used for each business module to subscribe to the device data in the target message queue according to the identifier of the target object model; the device data includes the device identifier of the device terminal and the JSON message of the device terminal.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data parsing method according to any one of claims 1 to 6 is implemented.

Images