Multi-version message data storage and analysis system and method

Abstract

The present invention provides a multi-version message data storage and parsing system and method, and relates to the field of data processing technology. The present invention caches the original data message queue, and intercepts the minimum storage unit from the cache queue according to the sliding window mechanism for persistent storage. The purpose is to enable data acquisition and sending devices with different sampling frequencies to save the original data in the minimum storage unit determined by the persistence cycle, so that it has a standardized data structure and length, and can adapt to different acquisition cycles, and then package and compress the existing minimum storage units in the minimum compression unit, thereby reducing the pressure on the network and the receiving end; when processing with the minimum storage unit and the minimum compression unit, two types of file names are given according to the naming specification, and a standardized multi-version parsing configuration file is placed in the original data query module to parse each version of the original data, avoiding version confusion and realizing the simultaneous storage and parsing of different versions of original data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-version message data storage and parsing system and method. Background Art

[0002] With the rapid development of IoT technology, installing IoT devices to collect, transmit, store and analyze various types of data has become an important means to improve industry efficiency and intelligence. IoT devices, as the perception layer of IoT, can capture various physical quantity information such as temperature, humidity, pressure, etc. in real time through built-in sensors and other components. This information is the basis for subsequent data analysis and decision-making, and has immeasurable value for smart manufacturing, smart agriculture, smart cities and other fields.

[0003] When faced with the processing and management of the IoT big data collected above, the existing technology usually adopts the technical solution of big data storage. Figure 2 The big data technology architecture solution of the S0 module shown in the figure has powerful data processing capabilities and high scalability, and can cope with the complex analysis needs of massive data. However, this solution is often accompanied by high cost investment, which is not only reflected in the configuration of hardware resources, but also includes the training and maintenance costs of professionals; its technical complexity is also relatively high, and the operation and maintenance personnel need to have a deep technical background to ensure the stable operation and continuous optimization of the system.

[0004] Therefore, a multi-version message data storage and analysis system and method are needed to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multi-version message data storage and analysis system, including several data collection and sending devices, an original data file receiving module, an original data file storage module, an original data query module and an original data query service module; wherein,

[0006] Data collection and transmission device: used to collect or receive the original data to be collected from the outside, convert the collected original data into message data according to the message protocol used, and establish a message queue for caching the original data, take out the message queue according to the persistence period and store and package and compress it; after completing the packaging and compression, upload the compressed file to the original data file receiving module;

[0007] The raw data file receiving module is used to receive the compressed files of the raw data; after receiving the compressed files uploaded by different data acquisition and sending devices through the file receiving interface, the compressed files are uploaded to the raw data file storage module;

[0008] The original data file storage module is used to provide storage space and archive management for the compressed files of the original data; create a compressed folder in the storage space according to the storage specification, and archive the compressed files into the corresponding compressed folder; respond to the query request of the original data query module for the storage space, and return the corresponding compressed files in the compressed folder;

[0009] The original data query module is used to generate a corresponding query request according to the service request parameters of the original data query service module, receive the compressed file returned by the original data file storage module, and automatically parse the file through the parsing program; wherein different versions of parsing configuration files are set for different message protocols;

[0010] Original data query business module: Generates corresponding business request parameters according to its own business needs and sends them to the original data query module, and receives the business request data parsed by the original data query module.

[0011] As a further solution, the data collection and sending device includes a data integration collection module, a raw data message queue, a local file system and a file transfer module; wherein the data integration collection module is used to collect or receive external raw data to be collected, and convert the collected raw data into message data according to the message protocol; the raw data message queue is used to cache the collected or received raw message data, intercept the raw data message queue according to a sliding window of a specific size, and write the message data corresponding to the minimum storage unit to the local file system; the file transfer module packages and compresses multiple minimum storage units that have been written to the local file system with the minimum compression unit, and assigns a compressed file name according to the file name specification, and uploads the compressed file through the file receiving interface provided by the raw data file receiving module.

[0012] A method for storing and parsing multi-version message data is applied to a multi-version message data storage and parsing system as described in any one of the above items, and performs multi-version message data storage and parsing through the following steps:

[0013] S11: Each data collection and transmission device collects or receives the raw data to be collected from the outside, converts the collected or received raw data into message data according to the message protocol used, and establishes a message queue for caching the raw data, dynamically takes out the cached raw data according to a sliding window of a specific size, processes it into the minimum storage unit, and then packages and compresses it according to the minimum compression unit; after completing the packaging and compression, the compressed file is uploaded to the raw data file receiving module;

[0014] S12: The raw data file receiving module receives the compressed file of the raw data; after receiving the compressed file uploaded by different data acquisition and sending devices through the file receiving interface, the compressed file is uploaded to the raw data file storage module;

[0015] S13: The raw data file storage module provides storage space and archive management for the compressed files of the raw data; creates a compressed folder in the storage space according to the storage specification, and archives the compressed files into the corresponding compressed folder; responds to the query request of the raw data query module for the storage space, and returns the corresponding compressed files in the compressed folder;

[0016] S14: The data query module starts to generate a corresponding query request according to the service request parameters of the original data query service module, receives the compressed file returned by the original data file storage module, and automatically parses the file through the parsing program; wherein different versions of parsing configuration files are set for different message protocols;

[0017] S15: The original data query business module generates corresponding business request parameters according to its own business requirements and sends them to the original data query module, and receives the business request data parsed by the original data query module.

[0018] As a further solution, the time length of the minimum storage unit is t=f*n, and the queue length of the sliding window is w=2*n-1; wherein n represents the number of collected data points contained in the minimum storage unit, and f is the data collection period of the data collection and sending device.

[0019] As a further solution, when the data collection period f can be divided into the persistence period, the number of collected data points n contained in the minimum storage unit is n=x / f; where x is the persistence period; when the data collection period f cannot be divided into the persistence period, the number of collected data points n contained in the minimum storage unit is the minimum integer that makes the time length t not less than the persistence period.

[0020] As a further solution, when the data collection and sending device is performing a message protocol upgrade and update, if the time required to perform the upgrade and update operation is less than the persistence period, the missing time is made up by increasing the pause time of the restart interval in the restart script file.

[0021] As a further solution, the raw data file receiving module and the raw data file storage module are provided with multiple groups. The collection and sending device selects the raw data file receiving module and the raw data file storage module of the corresponding group for file upload according to the established grouping strategy. At the same time, the raw data file receiving module selectively receives the uploaded data of each data collection and sending device according to the grouping strategy.

[0022] As a further solution, the original data query module divides the directory tree hierarchy according to the strategy of first time and then device number when creating a compressed folder.

[0023] As a further solution, when executing S14, the raw data query module performs the following steps:

[0024] S140: parsing the service request parameters, calculating the compressed folder list to be read and recording it as list L1, recording all the fields to be read as a field set F;

[0025] S141: traverse each compressed folder recorded in the list L1, obtain the names of all compressed files in each compressed folder, and add the file names that meet the time range specified by the request parameter to the list L2;

[0026] S142: decompress and read the message data names in all corresponding compressed files in the list L2, and automatically load the corresponding version of the parsing configuration file into the parsing program according to the message data name;

[0027] S143: For each line of message data, the parsing program parses only the fields included in the F field set according to the adapted parsing configuration file;

[0028] S144: Assemble all the data groups obtained after parsing according to the device number, and return them in the manner specified by the service request parameters.

[0029] As a further solution, when executing S141, if the data collection period f cannot be divided into an entire persistence period, at most one compressed file is additionally acquired as needed before the specified query time.

[0030] Compared with the related art, the multi-version message data storage and analysis system and method provided by the present invention has the following beneficial effects:

[0031] The present invention caches the original data message queue, and intercepts the minimum storage unit from the cache queue according to a sliding window of a specific size for persistent temporary storage. Its purpose is to enable data acquisition and sending devices with different sampling frequencies to use the same number of collected data as storage units, and then package and compress them according to the minimum compression unit, thereby reducing the processing pressure of network transmission and receiving ends, and having a standardized data structure and being adaptable to different acquisition cycles; the minimum storage unit message data that has been stored in the local file system is packaged and compressed with the minimum compression unit, and the minimum storage unit and the minimum compression unit file name are assigned according to the file name specification, and the message data of each version is automatically parsed by placing multiple versions of parsing configuration files in the original data query module, thereby avoiding the problem of version confusion and realizing the simultaneous storage and parsing of message data of different versions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of a multi-version message data storage and analysis system provided by the present invention;

[0033] Figure 2 A schematic diagram of a big data storage technology solution used in the prior art;

[0034] Figure 3 An example diagram of a message protocol design provided by the present invention;

[0035] Figure 4 A schematic diagram of sliding windows with different queue lengths provided by the present invention;

[0036] Figure 5 A schematic diagram of a 100 millisecond sliding window provided by the present invention;

[0037] Figure 6 An example diagram of the data collection and storage specification design provided by the present invention;

[0038] Figure 7 A comparison chart of examples of archived content provided by the present invention;

[0039] Figure 8 A specific embodiment diagram of the JSON configuration file provided by the present invention;

[0040] Fig. 9 A diagram of a structured raw data query interface request parameter table provided by the present invention;

[0041] Fig.10 This is an example diagram of data returned by the structured raw data query interface provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0043] Example 1

[0044] This embodiment provides a multi-version message data storage and parsing system, the structure of which is shown in Figure 1 , including several data collection and sending devices, raw data file receiving modules, raw data file storage modules, raw data query modules and raw data query service modules; wherein,

[0045] Data collection and transmission device: used to collect or receive the original data to be collected from the outside, convert the collected original data into message data according to the message protocol used, and establish a message queue for caching the original data, take out the message queue according to the persistence cycle and store and package and compress it; after completing the packaging and compression, upload the compressed file to the original data file receiving module

[0046] The raw data file receiving module is used to receive the compressed files of the raw data; after receiving the compressed files uploaded by different data acquisition and sending devices through the file receiving interface, the compressed files are uploaded to the raw data file storage module;

[0047] The original data file storage module is used to provide storage space and archive management for the compressed files of the original data; create a compressed folder in the storage space according to the storage specification, and archive the compressed files into the corresponding compressed folder; respond to the query request of the original data query module for the storage space, and return the corresponding compressed files in the compressed folder;

[0048] The original data query module is used to generate a corresponding query request according to the service request parameters of the original data query service module, receive the compressed file returned by the original data file storage module, and automatically parse the file through the parsing program; wherein different versions of parsing configuration files are set for different message protocols;

[0049] Original data query business module: Generates corresponding business request parameters according to its own business needs and sends them to the original data query module, and receives the business request data parsed by the original data query module.

[0050] It should be noted that most of the existing technologies use Figure 2In the S0 system shown, the S01 device is responsible for front-end data collection, and then processes the collected data into a message data format, and then sends it to the S02 module for parsing the message data (hexadecimal data). After the S02 module completes the parsing, it sends the structured data obtained after parsing to the message queue module shown in S03 for the subsequent consumer program shown in the S04 module to consume the structured data. Each consumer program is responsible for taking out the structured data previously cached in the message queue according to the message subject and partition rules and storing it in the big data platform shown in the S05 module. The subsequent raw data query business module shown in S06 completes the query of the parsed structured raw data through the big data platform query interface (such as HIVE) shown in S05 to support subsequent business needs.

[0051] The big data technology architecture solution shown in the S0 module is known for its powerful data processing capabilities and high scalability, and can cope with the complex analysis needs of massive data. However, this solution is often accompanied by high cost investment. The hardware investment for the construction of the S03 module, S04 module, and S05 module is even greater. In order to maintain high availability, the minimum number of servers required for each module is 3; and because there are more processing links and higher technical complexity, more professional technicians are needed to ensure the stable operation and continuous optimization of the system.

[0052] In addition, after receiving the data, the S0 system often needs to parse all fields before processing it into structured data for storage. Even when column-based compressed storage is used, the overall compression rate is about 13% to 20% due to factors such as data type, metadata, and other necessary information required for distributed storage. When faced with scenarios where data collection protocols need to be continuously upgraded, in addition to the acquisition device message encoding program and message parsing program that need to be upgraded, the consumer program and the structured data table of the big data platform also need to be repeatedly updated and filled with data to meet business needs.

[0053] In order to avoid the complexity and high cost of big data technology, this embodiment proposes a technical solution with simplicity and efficiency as its core advantages. Through more lightweight data processing tools and methods, a significant reduction in costs is achieved. At the same time, this solution is easy to use, simple to operate and maintain, and the requirements for the technical team are not as stringent as big data technology, thereby shortening the project implementation cycle and accelerating business response speed.

[0054] In such Figure 1In the S1 system shown, the data stored in the original data file receiving module is a compressed file consisting of hexadecimal message data, so its final compression rate can be as low as 3% to 5%; since there are fewer intermediate links and lower technical complexity required for storage, less personnel investment is required to ensure stable operation and continuous optimization of the system than in the solution constructed by the S0 system; in scenarios above the PB level, the number of servers required for the technical solution of this embodiment is saved by more than 50% compared with the solution constructed by the S0 system; in the scenario where the data acquisition protocol needs to be continuously upgraded, in addition to the need to update the acquisition device message encoding program, it is only necessary to add a new version of the parsing file that supplements the new encoding protocol in the original data file storage module in accordance with the specifications of this embodiment, without the need for other adjustments to the encoding and storage structure.

[0055] As a further solution, the data collection and sending device includes a data integration collection module, a raw data message queue, a local file system and a file transfer module; wherein the data integration collection module is used to collect or receive external raw data to be collected, and convert the collected raw data into message data according to the message protocol; the raw data message queue is used to cache the collected or received raw message data, intercept the raw data message queue according to a sliding window of a specific size, and write the message data corresponding to the minimum storage unit to the local file system; the file transfer module packages and compresses multiple minimum storage units that have been written to the local file system with the minimum compression unit, and assigns a compressed file name according to the file name specification, and uploads the compressed file through the file receiving interface provided by the raw data file receiving module.

[0056] It should be noted that: after caching the original message data through the original data message queue, this embodiment intercepts and calculates from the original data message queue according to the sliding window, so that the message data corresponding to the minimum storage unit is stored on the disk in the local file system. The purpose is to enable data acquisition and sending devices with different sampling frequencies to save the original data in the minimum storage unit of a fixed length determined by the persistence period, and then package and compress it according to the minimum compression unit, thereby reducing the processing pressure of network transmission and the receiving end, and having a standardized data structure and adapting to different acquisition cycles. It is one of the core points of the present invention (the application effect of different situations will be explained in detail below). Packaging and compressing the message data of each minimum storage unit stored on the disk in the local file system with the minimum compression unit, and assigning the minimum storage unit and the minimum compression unit file name according to the file name specification, provides a technical basis for subsequent compressed file management.

[0057] Example 2

[0058] A method for storing and parsing multi-version message data, the steps of which refer to Figure 1, perform multi-version message data storage and analysis through the following steps:

[0059] S11: Each data collection and transmission device collects or receives the raw data to be collected from the outside, converts the collected or received raw data into message data according to the message protocol used, and establishes a message queue for caching the raw data, dynamically takes out the cached raw data according to a sliding window of a specific size, processes it into the minimum storage unit, and then packages and compresses it according to the minimum compression unit; after completing the packaging and compression, the compressed file is uploaded to the raw data file receiving module;

[0060] S12: The raw data file receiving module receives the compressed file of the raw data; after receiving the compressed file uploaded by different data acquisition and sending devices through the file receiving interface, the compressed file is uploaded to the raw data file storage module;

[0061] S13: The raw data file storage module provides storage space and archive management for the compressed files of the raw data; creates a compressed folder in the storage space according to the storage specification, and archives the compressed files into the corresponding compressed folder; responds to the query request of the raw data query module for the storage space, and returns the corresponding compressed files in the compressed folder;

[0062] S14: The data query module starts to generate a corresponding query request according to the service request parameters of the original data query service module, receives the compressed file returned by the original data file storage module, and automatically parses the file through the parsing program; wherein different versions of parsing configuration files are set for different message protocols;

[0063] S15: The original data query business module generates corresponding business request parameters according to its own business requirements and sends them to the original data query module, and receives the business request data parsed by the original data query module.

[0064] It should be noted that the benefits and effects achieved by this embodiment are the same as those of Embodiment 1 and will not be described in detail here.

[0065] As a further solution, the time length of the minimum storage unit is t=f*n, and the queue length of the sliding window is w=2*n-1; wherein n represents the number of collected data points contained in the minimum storage unit, and f is the data collection period of the data collection and sending device.

[0066] It should be noted that: directly completing the persistent storage of the collected data within a single persistence cycle according to the persistence cycle (that is, storing as much data as the data is collected) will result in different numbers of collected data in the storage files corresponding to each persistence cycle even when the front-end acquisition device is working normally (for the case where the data acquisition cycle cannot be divided into an even number of persistence cycles). This embodiment calculates a minimum storage unit of a fixed length based on the persistence cycle to ensure that the number of collected data stored each time is the same (when the acquisition device is working normally), and on this basis, sets the corresponding sliding window (w=2*n-1 must contain data within a persistence cycle) to ensure that the data collected within the corresponding persistence cycle can be completely recorded in the corresponding minimum storage unit.

[0067] As a further solution, when the data collection period f can be divided into the persistence period, the number of collected data points n contained in the minimum storage unit is n=x / f; where x is the persistence period; when the data collection period f cannot be divided into the persistence period, the number of collected data points n contained in the minimum storage unit is the minimum integer that makes the time length t not less than the persistence period.

[0068] It should be noted that: the data collection cycle f can be divided into a business scenario with a persistence cycle of 1 second. According to this embodiment, the time length corresponding to the minimum storage unit is 1 second. When the data collection cycle is f milliseconds, the number of data points collected per second is n=1000 / f. At this time, the corresponding sliding window queue length is equal to 2*n-1=2*1000 / f-1 (for example, when f=100 milliseconds, the minimum storage unit length n=1000 / 100=10, and the sliding window queue length is 2*n-1=2*10-1=19);

[0069] In business scenarios where the data collection period f cannot be divided into a persistence period of 1 second, according to this embodiment, the time length corresponding to the minimum storage unit is t=f*n / 1000 seconds, where n is the smallest integer that makes t greater than or equal to 1000 milliseconds (for example, in the case of 90 milliseconds, a group of minimum storage units is composed of n=12 data points collected, that is, the time length represented by the minimum storage unit is 90*12=1080 milliseconds, and the sliding window queue length is 2*n-1=2*12-1=23). According to this embodiment, the number of seconds for collected data in the file specification when storing on disk is the number of seconds corresponding to the first data point in the n data of each storage processing unit. For business scenarios where the collection period f is greater than 1 second and less than 10 seconds (the unit of f is milliseconds), the minimum processing length is n, and the corresponding time length t=f*n (t is in milliseconds). The value of n is the minimum n value that makes t greater than or equal to 10 seconds (the persistence period is 10000 milliseconds). For scenarios where the collection period f and the persistence period are longer, the above logic can be used to perform corresponding calculations to determine the minimum storage unit n value and the corresponding time length t value.

[0070] In order to further illustrate the detailed technical solution, this embodiment describes that the subsequent verification data integration acquisition module completes the polling acquisition of the external data bus according to the acquisition cycle of once every 100 milliseconds. At the same time, to simplify the description, the collected data in the example set in this embodiment only includes three data items: temperature, voltage and current. After each data is collected, Figure 3 The message protocol shown converts the collected raw data into hexadecimal messages and saves them in the raw data message queue.

[0071] The original data message queue caches the message data in the form of key-value, where the key value is the number of milliseconds of message data collection, and the value is the hexadecimal message data obtained from this collection; when the message data cached by the queue is greater than or equal to 19 (such as Figure 4 As shown in the figure, when the queue length is less than 19, even if the queue length is 18, w0 still cannot completely cover 10 message data of a whole second; the window with a queue length of 19, such as w1 and w2 in the figure, can at least cover 10 complete message data within a whole second), then the 10 complete message data of 1 second are taken out from the queue according to the key value and stored on the local file system. The file name specification of the disk is: collection device number_collection date_collection data seconds_protocol major version number_protocol minor version number.txt (for example, the protocol version used by collection device No. 1 is major version number 1 and minor version number 0, and the disk storage file name of the device at 0:05:00 on June 27, 2024 is: 1_20240627_000500_M1_S0.txt).

[0072] like Figure 5 As shown in windows w3 and w4, under normal circumstances where data is collected once every 100 milliseconds, 10 collected messages are generated every second. When the window slides to the 19 message data covered by w3, 10 message data starting from L0 will be stored in the local file system to form a minimum storage unit. During the next disk storage operation, the window will slide to window w4 with L1 as the starting position. Due to collection anomalies, there may be less than 10 data collected within 1 second, such as the L2 position in the w4 window (only the data of the 900th millisecond of the second is collected at the N+2th second) and the L3 and L4 positions (only the data of the 0th millisecond and the 900th millisecond of the second is collected at the N+3th second). In this embodiment, no form of data filling is used in the data collection stage, but data filling is performed during the subsequent structured raw data query, which can effectively reduce the calculation complexity of the collection equipment and improve the transmission and storage efficiency. At the same time, the business can implement the filling strategy according to its own needs to a greater extent, which is more flexible.

[0073] In comparison, if the persistence cycle is used to store the collected data in the cycle (i.e., the amount of data collected in the corresponding cycle is stored), even when the device is working normally, for business scenarios where the collection cycle cannot be divided into a full persistence cycle, the number of collected data in the corresponding cycle will inevitably fluctuate (e.g., when the collection cycle is 900 milliseconds and the persistence cycle is 1 second, from the 1st second to the 8th second, each persistence cycle will only collect a maximum of 1 piece of data, while for the 0th second and the 9th second, data will be collected at 0.0 seconds, 0.9 seconds, 9.0 seconds, and 9.9 seconds, so 2 collected data points must be stored in the persistence cycle corresponding to the 0th second and the 9th second). The disadvantage of this method is that the storage files corresponding to each persistence cycle need to be To calculate whether the number of collected data points within the period is complete, the implementation requires: either the backend needs to use more complex calculation logic to calculate whether the file has collected enough data points (the backend judgment logic will be more complex than judging only based on the number of data rows in the file, and it is necessary to trace back to the first collection time after the corresponding collection device was restarted to get the result), or spend additional computing power on the front-end collection device to add data filling logic to ensure the integrity of the number of collected data in each storage file (usually the front-end collection device has limited computing power, which will increase the overhead of the front-end collection device. At the same time, after the data is filled, the missed data event is not visible to the subsequent query business), or adjust the persistence period or frequency of data collection of the front-end collection device to adapt so that the collection period can be evenly divided into the persistence period (which will lead to a decrease in versatility).

[0074] If a fixed-length minimum storage unit is used to achieve persistence, it is only necessary to determine whether the number of collected data in the storage unit is equal to the fixed length n of the minimum storage unit during the back-end parsing stage. If it is less than n, the data is filled according to the specified filling strategy, and if it is equal to n, it indicates that no data is missed. Therefore, with this method, there is no need to match the collection cycle and persistence cycle of the front-end collection device so that they can be evenly divided, and the subsequent parsing operations can still correctly complete the parsing process with a simple and unified processing logic. In this way, without changing the persistence cycle and the collection frequency of the front-end collection device, the phenomenon of the amount of data to be collected in the storage unit fluctuating during normal operation of the device is avoided. The processing logic is simpler and the versatility is obviously stronger.

[0075] As a further solution, when the data collection and sending device is performing a message protocol upgrade and update, if the time required to perform the upgrade and update operation is less than the persistence period, the missing time is made up by increasing the pause time of the restart interval in the restart script file.

[0076] It should be noted that the following will be described in conjunction with specific implementation scenarios.

[0077] As a further solution, the raw data file receiving module and the raw data file storage module are provided with multiple groups. The collection and sending device selects the raw data file receiving module and the raw data file storage module of the corresponding group for file upload according to the established grouping strategy. At the same time, the raw data file receiving module selectively receives the uploaded data of each data collection and sending device according to the grouping strategy.

[0078] It should be noted that: if multiple data acquisition and sending devices are uploading compressed files to the original data file receiving module, and then the original data file receiving module stores the compressed files in the original data file storage module, the data acquisition and sending devices can be grouped with the original data file receiving module and the original data file storage module to improve the concurrent processing capability and scalability of the system.

[0079] A simple and feasible grouping algorithm is to obtain the raw data file receiving module and raw data file storage module to which the compressed file should be uploaded for storage by taking the modulus operation of the acquisition device number and the number of groups. For example, there are 10 data acquisition and sending devices (device numbers are 1-10), and they include two groups of raw data file receiving modules and raw data file storage modules. All devices (data acquisition and sending devices with device codes of 2, 4, 6, 8, and 10) that obtain 0 by taking the modulus operation of the two groups by the acquisition device code will upload the compressed file to the first group. , all devices whose acquisition device codes obtain 1 by taking the modulus operation of the two groups (data acquisition and sending devices with device codes of 1, 3, 4, 7, and 9) upload the compressed file to the second group; at the same time, the first group is set to only receive and process the data uploaded by the devices (data acquisition and sending devices with device codes of 2, 4, 6, 8, and 10) whose modulus operation obtains 0, and the second group is set to only receive and process the data uploaded by the devices (data acquisition and sending devices with device codes of 1, 3, 4, 7, and 9) whose modulus operation obtains 1, thereby avoiding mismatch and mistransmission to improve security.

[0080] As a further solution, the original data query module divides the directory tree hierarchy according to the strategy of first time and then device number when creating a compressed folder.

[0081] It should be noted that: the compressed file name in this embodiment is based on the storage specification (such as Figure 6 As shown), the compressed file is transferred to the corresponding directory. The storage specification of the compressed file is: year / month / day / device number / starting year-month-day-hour-minute_ending year-month-day-hour-minute_device number_SM major version number_SS minor version number_EM major version number_ES minor version number.zip; In addition to being able to easily query the original data details by time, the use of a directory tree hierarchy divided by time and then by device number also has the advantage of being able to easily archive historical data for all data that exceeds a certain time range and clear out data that exceeds a certain time range on demand (for example: deleting or archiving data that is more than 5 years old).

[0082] In actual collection scenarios, it is often accompanied by Figure 3 The message protocol shown is constantly upgraded (for example: after a period of operation, it is found that it is still necessary to Figure 3 The collection and encoding of humidity data is added on the basis of the 1.0 protocol version shown, so the encoding protocol version must be upgraded); at this time, there are two possible forms of compressed files in the original data file storage module. One is that all message data in a single compressed package are converted using the same version of the encoding protocol (such as Figure 7As shown in the "Example of Archive Content of Compressed Files with the Same Protocol Version", all message data texts are encoded by the protocol version number 1.0), and the other is that the message data in a single compressed package is converted using multiple encoding protocols (such as Figure 7 As shown in the "Example of cross-protocol version compressed file archive content", the message data at 00:05:00 on June 27, 2024 is encoded using the protocol version number 1.0, while the message data at 00:09:57 on June 27, 2024 is encoded using the protocol version number 1.1).

[0083] The upgrade of the coding protocol version is often accompanied by the software upgrade operation of the data acquisition and transmission device. The upgrade operation often takes more than 1 second to complete (if the upgrade operation takes less than 1 second of the persistence period, the restart interval pause time can be increased in the restart script file to ensure it). Figure 7 As shown in the "Example of the content of the smallest storage unit original message", a single original message file includes a maximum of 10 message data, and the format uses the number of milliseconds collected + the message corresponding to the millisecond. The text file does not contain message data of different protocol versions, which will simplify subsequent analysis.

[0084] In order to successfully parse the message data of multiple protocol versions, this embodiment adopts the method of placing multiple versions of parsing configuration files in the original data query module to automatically parse the message data. The naming convention of the configuration file is: protocol name_VT protocol formulation date_VM protocol major version number_VS protocol minor version number.json (the format specification of the json configuration file is as follows Figure 8 As shown in the figure), for example, version 1.0 of the encoding protocol was launched on April 20, 2024, and the configuration file xxxx_VT20240420_VM1_VS0.json of this version was added to the original data query module. On June 27, 2024, version 1.1 of the encoding protocol was updated and released, and a new version 1.1 configuration file xxxx_VT20240420_VM1_VS1.json was added to the original data query module.

[0085] In such Figure 8 In the configuration file example shown (with Figure 3 Each line of JSON data specifies how to parse the structured field names and field values ​​from the message data. For example, after converting the hexadecimal message data to binary data, the values ​​from bits 32 to 39 are read out, which is the major version number of the encoding protocol. Another example is, after converting the hexadecimal message data to binary data, the values ​​from bits 112 to 119 are read out, which is the temperature value collected at that moment.

[0086] As a further solution, when executing S14, the raw data query module performs the following steps:

[0087] S140: parsing the service request parameters, calculating the compressed folder list to be read and recording it as list L1, recording all the fields to be read as a field set F;

[0088] If the date specified in the request parameters is June 27, 2024, the acquisition device number is 1, and the query fields are major version number, minor version number, device number, temperature value, and voltage value, then the directory represented by L1 is 2024 / 06 / 27 / 1 / , and F={major version number, minor version number, acquisition time (must be checked by default), device number, temperature value, voltage value};

[0089] S141: traverse each compressed folder recorded in the list L1, obtain the names of all compressed files in each compressed folder, and add the file names that meet the time range specified by the request parameter to the list L2;

[0090] If the query period specified in the request parameters is 00:02:00 to 00:09:00 on June 27, 2024, then all the compressed file names in the folders (2024 / 06 / 27 / 1 / ) in L1 that include the compressed file names in this period are added to L2. Figure 6 For the example shown, L2={

[0091] 202406270000_202406270004_1_SM1_SS0_EM1_ES0.zip

[0092] 202406270005_202406270009_1_SM1_SS0_EM1_ES1.zip

[0093] };

[0094] S142: decompress and read the message data names in all corresponding compressed files in the list L2, and automatically load the corresponding version of the parsing configuration file into the parsing program according to the message data name;

[0095] For example, in the process of traversing the compressed file 202406270005_202406270009_1_SM1_SS0_EM1_ES1.zip, two versions of the message parsing configuration files are involved (such as Figure 7As shown in the "Example of Cross-protocol Version Compressed File Archive Contents", the message text file named 1_20240627_000502_M1_S0.txt uses the 1.0 version of the configuration file (i.e.: xxxx_VT20240420_VM1_VS0.json) for message parsing, while the message text file named 1_20240627_000957_M1_S1.txt uses the 1.1 version of the configuration file (i.e.: xxxx_VT20240420_VM1_VS1.json) for message parsing.

[0096] At the same time, when the message text is parsed, the message text that does not meet the query time requirement will be automatically skipped to improve efficiency and accuracy (for example, in step S140, the request parameter specifies that only detailed data from 00:02:00 to 00:09:00 is queried, then Figure 7 All the message text files from 1_20240627_000000_M1_S0.txt to 1_20240627_000159_M1_S0.txt shown will not be included in the queue to be traversed and parsed);

[0097] S143: For each line of message data, the parsing program parses only the fields included in the F field set according to the adapted parsing configuration file;

[0098] To improve efficiency, only the fields in the F set in step 1 are parsed, and the fields not included in F are not parsed. At the same time, by default, if the message text file currently being parsed is less than 10, no data filling will be performed. If the business query module has data filling requirements, the following Fig. 9 The fillType request parameter shown is set to 1, and the data collected last will be copied to fill the data (such as Figure 5 As shown, in the N+2th second, data is collected only at the 900th millisecond represented by the L2 position. At this time, the data of the 0th, 100th, 200th, 300th, 400th, 500th, 600th, 700th, and 800th milliseconds of the N=2th second will be filled with the value of the 900th millisecond of the N+1th second; in the N+3th second, data is collected only at the 0th millisecond represented by the L3 position and the 900th millisecond represented by the L4 position. At this time, the data collected at the 0th millisecond of the N+3th second will be filled with the data of the 100th, 200th, 300th, 400th, 500th, 600th, 700th, and 800th milliseconds of that second)

[0099] S144: Assemble all the data groups obtained after parsing according to the device number, and return them in the manner specified by the service request parameters.

[0100] As a further solution, when executing S141, if the data collection period f cannot be divided into a whole persistence period, a compressed file is additionally obtained as needed before the specified query time.

[0101] It should be noted that: when executing S141, it is necessary to pay attention to the forward addition of the query time. For example, for business scenarios where the collection cycle cannot be evenly divided into 1 second, the data collected in the same second may be divided into two compressed packages. Therefore, it is necessary to supplement the compressed package as needed to include all eligible data.

[0102] like Figure 5 As shown in the second row, n represents the number of data points collected by a minimum storage unit, w is a queue of size 2*n-1, and f represents the collection period. In the figure, t0, t1, t2, ..., t n-1 The corresponding collection point is recorded as a processing unit starting from the kth second, T0, T1, T2, ..., T n-1 The corresponding collection point is the next processing unit starting from the kth second. Since n*f>= 1s, T0-t0>= 1s, so the number of seconds corresponding to T0 is at least k+1 seconds, and the time point t n-1 = T0- f The number of seconds corresponding to may be the same as T0, or it may be 1 second less than the number of seconds corresponding to T0.

[0103] like Figure 5 As shown in the third row, U11, U12 and U13 represent three consecutive storage units with f=90 milliseconds for the collection business. The first collection data time point of the 11th minimum storage unit U11 is 10.89 seconds. After the minimum storage unit U11 is stored on the disk, it will be marked with a second value of 10 (including the collection data with a corresponding second value of 11). The first collection data time point of the 12th minimum storage unit U12 is 11.97 seconds. After the minimum storage unit U12 is stored on the disk, it will be marked with a second value of 11 ( Contains collected data with a corresponding second value of 12, and the corresponding second number of the first data collection point is the same as the corresponding second number of the last data point in the previous minimum storage unit U11), the 13th collected data time point is 13.05 seconds, and the minimum storage unit U13 will be marked with a second value of 13 after it is stored on the disk (contains collected data with a corresponding second value of 14, and the corresponding second number of the first data collection point [i.e.: 13] = the corresponding second number of the last data point in the previous minimum storage unit U12 + 1 [i.e.: 12+1]).

[0104] It can be seen from this that for the two consecutive minimum storage units, the Kth and K+1th, the seconds corresponding to the Kth minimum storage unit are recorded as S0, and the seconds corresponding to the K+1th minimum storage unit are recorded as S1. The Kth minimum storage unit may contain the collected data within the S1th second (that is, from the 0th millisecond of the S1th second to the 999th millisecond of the S1th second), while the K+1th minimum storage unit will definitely not contain the collected data within the S0th second (that is, from the 0th millisecond of the S0th second to the 999th millisecond of the S0th second). When the two consecutive minimum storage units, the Kth and K+1th, are separated into two different minimum compression units, it is necessary to pay attention to the forward compression package supplement according to the query time during the query.

[0105] In a specific embodiment, assuming that the query time period is the original data from 00:00:00 to 00:09:59 on June 27, 2024, when the data collection period f can be divided into 1 second, the data corresponding to 2024-06-27 00:00:00 is already included in the compressed file 202406270000_202406270004_1_SM1_SS0_EM1_ES0.zip, so there is no need to add compressed files in the past. If the data collection period f cannot be divided into 1 second, the time point 2024-06-27 The data collected after 00:00:00 may be located in the last minimum storage processing unit in the previous compressed file 202406262355_202406262359_1_SM1_SS0_EM1_ES0.zip. Therefore, it is necessary to append the compressed file 202406262355_202406262359_1_SM1_SS0_EM1_ES0.zip forward so as to retrieve all the data collected after 2024-06-27 00:00:00. The query deadline 00:09:59 means querying the original data whose collection time does not exceed 00:09:59.0. According to the aforementioned "only the previous storage unit of two consecutive minimum storage units may contain the collected data after the starting seconds of the next storage unit", it is known that there is no need to append the compressed package backward when this time point is used as the end time of data query.

[0106] Fig. 9 For the request parameter table when performing structured raw data query, the raw data query business module needs to specify the device number according to the parameter table when calling the raw data query module (such as Fig. 9 equipments field shown in ), the start time of data collection (such as Fig. 9 The beginTime field shown in ), the end time of data collection (such as Fig. 9endTime field shown in ), the structured field name to be queried (such as Fig. 9 ), return mechanism (such as Fig. 9 The type field shown in ), the return address (such as Fig. 9 address field as shown in ) and the padding type as Fig. 9 The original data query module finally queries and returns a structured data sample by executing steps S140-S144. Fig.10 shown.

[0107] The above are only embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-version message data storage and analysis system, characterized in that: It includes several data collection and transmission devices, raw data file receiving modules, raw data file storage modules, raw data query modules and raw data query service modules; among which, Data collection and transmission device: used to collect or receive the original data to be collected from the outside, convert the collected original data into message data according to the message protocol used, and establish a message queue for caching the original data, take out the message queue according to the persistence period and store and package and compress it; after completing the packaging and compression, upload the compressed file to the original data file receiving module; The raw data file receiving module is used to receive the compressed files of the raw data; after receiving the compressed files uploaded by different data acquisition and sending devices through the file receiving interface, the compressed files are uploaded to the raw data file storage module; The original data file storage module is used to provide storage space and archive management for the compressed files of the original data; create a compressed folder in the storage space according to the storage specification, and archive the compressed files into the corresponding compressed folder; respond to the query request of the original data query module for the storage space, and return the corresponding compressed files in the compressed folder; The original data query module is used to generate a corresponding query request according to the service request parameters of the original data query service module, receive the compressed file returned by the original data file storage module, and automatically parse the file through the parsing program; wherein different versions of parsing configuration files are set for different message protocols; Original data query business module: generates corresponding business request parameters according to its own business needs and sends them to the original data query module, and receives the business request data parsed by the original data query module; The data collection and transmission device includes a data integration collection module, a raw data message queue, a local file system and a file transfer module; wherein the data integration collection module is used to collect or receive raw data to be collected from the outside, and convert the collected raw data into message data according to the message protocol; the raw data message queue is used to cache the collected or received raw message data, intercept the raw data message queue according to a sliding window of a specific size, and write the message data corresponding to the minimum storage unit to the local file system; the file transfer module packs and compresses multiple minimum storage units that have been written to the local file system with the minimum compression unit, and assigns a compressed file name according to the file name specification, and uploads the compressed file through the file receiving interface provided by the raw data file receiving module; The time length of the minimum storage unit is t=f*n, and the queue length of the sliding window is w=2*n-1; wherein n represents the number of collected data points contained in the minimum storage unit, and f represents the data collection cycle of the data collection and transmission device; When the data collection period f can be divided into the persistence period, the number of collected data points n contained in the minimum storage unit is n=x / f, where x is the persistence period; when the data collection period f cannot be divided into the persistence period, the number of collected data points n contained in the minimum storage unit is the minimum integer that makes the time length t not less than the persistence period.

2. A method for storing and parsing multi-version message data, applied to a multi-version message data storing and parsing system as claimed in claim 1, characterized in that: Perform multi-version message data storage and parsing through the following steps: S11: Each data collection and transmission device collects or receives the raw data to be collected from the outside, converts the collected or received raw data into message data according to the message protocol used, and establishes a message queue for caching the raw data, dynamically takes out the cached raw data according to a sliding window of a specific size, processes it into the minimum storage unit, and then packages and compresses it according to the minimum compression unit; After completing the packaging and compression, upload the compressed file to the original data file receiving module; S12: The raw data file receiving module receives the compressed file of the raw data; after receiving the compressed file uploaded by different data acquisition and sending devices through the file receiving interface, the compressed file is uploaded to the raw data file storage module; S13: The original data file storage module provides storage space and archive management for the compressed files of the original data; creates a compressed folder in the storage space according to the storage specification, and archives the compressed files into the corresponding compressed folder; Respond to the query request of the original data query module for the storage space and return the corresponding compressed file in the compressed folder; S14: The data query module starts to generate a corresponding query request according to the service request parameters of the original data query service module, receives the compressed file returned by the original data file storage module, and automatically parses the file through the parsing program; wherein different versions of parsing configuration files are set for different message protocols; S15: The original data query business module generates corresponding business request parameters according to its own business requirements and sends them to the original data query module, and receives the business request data parsed by the original data query module.

3. A method for storing and parsing multi-version message data according to claim 2, characterized in that: When the data collection and transmission device is performing message protocol upgrade and update, if the time required to perform the upgrade and update operation is less than the persistence period, the missing time is made up by increasing the pause time of the restart interval in the restart script file.

4. A method for storing and parsing multi-version message data according to claim 2, characterized in that: There are multiple groups of raw data file receiving modules and raw data file storage modules. The collection and sending device selects the raw data file receiving modules and raw data file storage modules of the corresponding group to upload files according to the established grouping strategy. At the same time, the raw data file receiving module selectively receives the uploaded data of each data collection and sending device according to the grouping strategy.

5. A method for storing and parsing multi-version message data according to claim 2, characterized in that: When creating a compressed folder, the original data query module divides the directory tree levels according to the strategy of first time and then device number.

6. A method for storing and parsing multi-version message data according to claim 2, characterized in that: When executing S14, the original data query module performs the following steps: S140: parsing the service request parameters, calculating the compressed folder list to be read and recording it as list L1, recording all the fields to be read as a field set F; S141: traverse each compressed folder recorded in the list L1, obtain the names of all compressed files in each compressed folder, and add the file names that meet the time range specified by the request parameter to the list L2; S142: decompress and read the message data names in all corresponding compressed files in the list L2, and automatically load the corresponding version of the parsing configuration file into the parsing program according to the message data name; S143: For each line of message data, the parsing program parses only the fields included in the F field set according to the adapted parsing configuration file; S144: Assemble all the data groups obtained after parsing according to the device number, and return them in the manner specified by the service request parameters.

7. A method for storing and parsing multi-version message data according to claim 6, characterized in that: When executing S141 , if the data collection period f cannot be divided into an entire persistence period, at most one compressed file is additionally acquired as needed before the specified query time.

Images