Distributed processing method, device and equipment for working condition data, medium and program

Abstract

The invention discloses a distributed processing method, device and equipment for working condition data, a medium and a program. The method comprises the following steps: detecting real-time working condition data reported by Internet of Things equipment, processing the real-time working condition data into standardized working condition data, and adding the standardized working condition data into a message queue as a to-be-consumed message; acquiring a current message from the message queue in real time, and extracting a current equipment serial number and current gateway receiving time in the current message; the current device serial number and the current gateway receiving time form a current joint character string, and a hash value is calculated according to a target hash algorithm; and determining a target node fragment according to the hash value range of each node fragment, and storing the current message into a current storage data block of the target node fragment. According to the embodiment of the invention, balanced distribution and ordered storage of the data are realized, the expandability is good, the data processing efficiency is improved, the problem of unbalanced storage pressure is avoided, and the stability and reliability of the whole distributed system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial Internet of Things, and in particular to a distributed processing method, device, equipment, medium and program for working condition data. Background Art

[0002] In today's industrial IoT environment, the explosive growth of data volume has brought unprecedented challenges to data storage and processing technology. With the widespread application of industrial IoT devices, such as various types of mechanical equipment, the basic information, fault and alarm data generated by these devices are characterized by high frequency and large scale, making data collection, storage, query and analysis particularly complex.

[0003] Traditional data storage methods, such as relational databases and distributed databases, often become bottlenecks due to server hardware limitations when facing TB-level or even PB-level data increments. Although the industry has adopted a variety of technical means such as read-write separation, database and table sharding, caching, distributed storage, and clustering for data consumption, these methods are still difficult to simultaneously meet the needs of high-performance and high-real-time data storage, and are prone to data consumption delays and data storage imbalances.

[0004] Therefore, there is an urgent need to design a method to achieve efficient data consumption and balanced storage in a massive data environment. Summary of the invention

[0005] Based on this, the present invention provides a distributed processing method, device, equipment, medium and program for working condition data to solve the problem of unbalanced data storage and many limitations in vertical and horizontal expansion.

[0006] In a first aspect, an embodiment of the present invention provides a distributed processing method for operating condition data, the method comprising:

[0007] Whenever real-time working condition data reported by IoT devices is detected, the real-time working condition data is processed into standardized working condition data, and the standardized working condition data is added to the message queue as a message to be consumed;

[0008] The data consumption engine obtains the current message from the message queue in real time, and extracts the current device serial number and the current gateway receiving time contained in the current message;

[0009] The data consumption engine combines the current device serial number and the current gateway receiving time into a current joint string, and calculates the current hash value matching the current joint string according to the target hash algorithm;

[0010] The data consumption engine determines the target node shard according to the hash value range of each node shard in the distributed system, and stores the current message in the current storage data block of the target node shard; wherein each data block in the distributed system is used to store messages within the set gateway receiving time interval.

[0011] In a second aspect, an embodiment of the present invention further provides a distributed processing device for operating condition data, the device comprising:

[0012] A message queue generation module is used to process the real-time working condition data reported by the IoT device into standardized working condition data whenever the real-time working condition data is detected, and add the standardized working condition data to the message queue as a message to be consumed;

[0013] The joint message extraction module is used to obtain the current message from the message queue in real time through the data consumption engine, and extract the current device serial number and the current gateway receiving time contained in the current message;

[0014] A hash value calculation module, used to combine the current device serial number and the current gateway receiving time into a current joint string through the data consumption engine, and calculate the current hash value matching the current joint string according to the target hash algorithm;

[0015] The balanced storage module is used to determine the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine, and store the current message in the current storage data block of the target node shard; wherein each data block in the distributed system is used to store messages within the set gateway receiving time interval.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a distributed processing method for operating condition data described in any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a distributed processing method for operating condition data described in any embodiment of the present invention when executed.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements a distributed processing method for operating condition data described in any embodiment of the present invention.

[0022] The technical solution of the embodiment of the present invention processes the working condition data reported by the Internet of Things devices in real time, converts it into standardized working condition data, and immediately adds it to the message queue, thereby ensuring the timeliness and accuracy of the data; the data consumption engine can quickly obtain and process data from the message queue, and through the hash algorithm and distributed storage mechanism, realizes rapid positioning and storage of data, and improves data processing efficiency; by calculating the hash value of the data and distributing the data to different node shards according to the hash value range, it ensures the balanced distribution of data in the distributed system. This balanced storage method avoids the problem of unbalanced storage pressure caused by the centralized storage of data on a certain node shard, and improves the stability and reliability of the entire distributed system; in general, it realizes balanced distribution and orderly storage of data, has good scalability, makes system maintenance and management more convenient, and reduces maintenance costs.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a distributed processing method for operating condition data provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flowchart of another distributed processing method for operating condition data provided according to Embodiment 2 of the present invention;

[0027] Figure 3 is a schematic diagram of the structure of a distributed processing device for operating condition data provided according to Embodiment 3 of the present invention;

[0028] Figure 4 It is a structural schematic diagram of an electronic device for implementing a distributed processing method for working condition data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flowchart of a distributed processing method for working condition data provided in the first embodiment of the present invention. This embodiment can be applied to the case of distributed and balanced storage of working condition data of industrial Internet of Things devices. The method can be executed by a distributed processing device for working condition data. The device can be implemented in the form of hardware and / or software. The device can be configured in a distributed system, specifically in an industrial Internet platform based on big data. Figure 1 As shown, the method includes:

[0033] S110. Whenever real-time operating condition data reported by an IoT device is detected, the real-time operating condition data is processed into standardized operating condition data, and the standardized operating condition data is added to a message queue as a message to be consumed.

[0034] The IoT devices such as sensors and instruments will monitor the working condition data of engineering machinery vehicles in real time. The working condition data reflects the basic operating conditions of the current equipment, such as working hours, location, fuel consumption and speed, etc. After the above-mentioned various types of data are reported to the distributed system, the original working condition data is cleaned, noise, erroneous data and duplicate data are removed, and the data is converted into a format for review, storage and analysis, such as key:value format, according to the preset data conversion rules, so as to facilitate subsequent hash value calculations. The message queue, as a first-in-first-out data structure, acts as a data buffer to temporarily store the received working condition data in the message queue to realize asynchronous data transmission and decouple the data production and consumption process.

[0035] S120, obtaining the current message from the message queue in real time through the data consumption engine, and extracting the current device serial number and the current gateway receiving time contained in the current message.

[0036] The data consumption engine is a component responsible for processing data in the message queue. It obtains the current message from the message queue for processing in real time, which means that the data consumption engine will continuously monitor the message queue and immediately retrieve and process new messages once they arrive.

[0037] S130. The data consumption engine combines the current device serial number and the current gateway receiving time into a current joint string, and calculates a current hash value matching the current joint string according to a target hash algorithm.

[0038] When processing a message, the data consumption engine extracts key information from the message. The key information is determined by presetting the structure of the joint distribution key. In the embodiment of the present invention, the device serial number and the gateway receiving time are set as the joint distribution key, and the key information is determined to be the device serial number (used to uniquely identify the device reporting the data) and the gateway receiving time (i.e., the timestamp when the data is received by the gateway) based on the joint distribution key. Setting a specified joint distribution key can ensure that data with the same distribution key can be processed centrally, so that data with the same joint distribution key characteristics are centrally stored on specific shards and nodes.

[0039] In the embodiment of the present invention, there is no restriction on the selection of the target hash algorithm. The current hash value is the hash value corresponding to the current message, which is used for subsequent data storage decisions. The hash algorithm can map complex joint distribution keys to relatively simple hash values ​​and ensure the uniformity and randomness of data distribution.

[0040] S140. Determine the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine, and store the current message in the current storage data block of the target node shard; wherein each data block in the distributed system is used to store messages within a set gateway receiving time interval.

[0041] The distributed system consists of multiple node shards, each node shard is responsible for storing a part of the data, and each data block is responsible for storing messages within a specific gateway receiving time interval. The data consumption engine will determine the target node shard corresponding to the current hash value based on the pre-defined hash value range (or hash table) in the distributed system. Once the target node shard is determined, the data consumption engine will store the current message in the current storage data block of the node shard. The "current storage data block" refers to the basic unit for storing data in the distributed system, which is divided according to the gateway receiving time interval.

[0042] Optionally, before determining the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine, the following may also be included:

[0043] Determine the number of node shards involved in data storage based on the storage architecture of the distributed system;

[0044] Obtain multiple pieces of historical operating condition data within a set time period, and obtain a historical joint string of each piece of historical operating condition data;

[0045] According to the target hash algorithm, calculate the historical hash value corresponding to each historical joint string, and obtain the maximum hash value and the minimum hash value among the historical hash values;

[0046] Constructing a hash value interval according to the maximum hash value and the minimum hash value, and dividing the hash value interval into multiple hash value ranges according to the number of node shards;

[0047] A one-to-one correspondence between each node shard and each hash value range in each distributed system is established.

[0048] First, the number of node shards participating in data storage is determined in order to clarify which node shards in the distributed system will participate in data storage, so as to assign a hash value with a certain value range to each node shard. For example, if there are 10 node shards, the data will be divided into 10 shards according to the calculated hash value range, and each shard corresponds to a node storage.

[0049] The historical operating condition data can be the data reported by the IoT device within a certain period of time in the past. By extracting the historical joint string of the historical operating condition data and performing a hash operation on the historical joint string, a unique hash value will be generated for each historical joint string. The maximum and minimum values ​​of the hash value are used to construct a distribution range of the hash value, which contains all possible hash value values.

[0050] According to the number of node shards determined previously, the hash value interval is divided into multiple sub-intervals, that is, multiple hash value ranges. Each node shard is assigned a specific hash value range. The hash value range is assigned to the node shard through historical operating condition data, so that when new operating condition data needs to be stored, the node shard corresponding to the new operating condition data can be found according to the above mapping relationship.

[0051] Furthermore, the method may also include:

[0052] Whenever it is detected that the current system time reaches the new data block designation condition, the current storage data block for real-time storage of messages is re-designated in each node shard in the distributed system;

[0053] Obtain the most recently specified historical storage data blocks, and determine the historical gateway receiving time interval of each historical storage data block according to the historical specified time of each historical storage data block and the current system time;

[0054] The historical gateway receiving time interval is used as the partition key and added to each historical storage data block.

[0055] It should be noted that the partition key plays a key organizational role in the physical storage structure of the distributed system. The partition key determines the grouping and storage method of data on each node shard. Within each node shard, the data is further subdivided and stored according to the logical rules of the partition key. In an embodiment of the present invention, corresponding partitions are created according to the gateway reception time, and the data of the same day will be stored in the corresponding partition folder or storage area, that is, storage data blocks, and different storage data blocks are independent of each other. For example, storage data blocks named "2023-09-10", "2023-09-11", etc. are created on a node shard to store all data received on that day.

[0056] Since data is stored in real time, as time goes by, the number of storage data blocks added to the node shard increases. If the basic unit of gateway receiving time is set to day, one storage data block corresponds to the data stored for a certain day. Therefore, when the system time changes to a day, such as the system time changes from 00:00:59 to 00:00:00, it is considered that the new data block designation condition is met. The next data block is designated as the current storage area for storing new messages, and the next data block will replace the current data block as the default location for storing new messages in the current node shard.

[0057] After specifying a new data block, the system will review and obtain the data blocks used before (i.e., the most recent historical designation). These data blocks are called "historical storage data blocks", which have been used to store data before but are no longer the default location for storing new messages. For each historical storage data block, a time interval will be calculated or determined based on the time when it was designated as a storage data block (i.e., the historical designation time) and the current system time. This time interval represents the time range of the data stored in the historical storage data block, also known as the "historical gateway receiving time interval".

[0058] The historical gateway receiving time interval corresponding to each historical storage data block is added to the corresponding data block as a partition key, so that the corresponding data block can be quickly located according to the time interval when querying or processing data subsequently.

[0059] The technical solution of the embodiment of the present invention processes the working condition data reported by the Internet of Things devices in real time, converts it into standardized working condition data, and immediately adds it to the message queue, thereby ensuring the timeliness and accuracy of the data; the data consumption engine can quickly obtain and process data from the message queue, and through the hash algorithm and distributed storage mechanism, realizes rapid positioning and storage of data, and improves data processing efficiency; by calculating the hash value of the data and distributing the data to different node shards according to the hash value range, it ensures the balanced distribution of data in the distributed system. This balanced storage method avoids the problem of unbalanced storage pressure caused by the centralized storage of data on a certain node shard, and improves the stability and reliability of the entire distributed system; in general, it realizes balanced distribution and orderly storage of data, has good scalability, makes system maintenance and management more convenient, and reduces maintenance costs.

[0060] Embodiment 2

[0061] Figure 2 This is a flow chart of another distributed processing method for working condition data provided by the second embodiment of the present invention. This embodiment is based on the above embodiment and is refined. Figure 2 As shown, the method includes:

[0062] S210. Whenever real-time operating condition data reported by an IoT device is detected, the real-time operating condition data is processed into standardized operating condition data, and the standardized operating condition data is added to a message queue as a message to be consumed.

[0063] S220, obtaining the current message from the message queue in real time through the data consumption engine, and extracting the current device serial number and the current gateway receiving time contained in the current message.

[0064] S230. The data consumption engine combines the current device serial number and the current gateway receiving time into a current joint string, and calculates a current hash value matching the current joint string according to a target hash algorithm.

[0065] S240. Determine the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine.

[0066] S250, creating a query index according to the current device serial number, the current gateway receiving time and the target node shard, or according to the current device serial number, the current gateway receiving time, the target node shard and the current storage data block on the target node shard.

[0067] After the hash value has been calculated and determined in which node shard (i.e., the target node shard) the data should be stored, a query index can be constructed based on the target shard node. Specifically, the query index includes two cases, the first is a first-category query index constructed by: device serial number-gateway reception time-node shard; the second is a second-category query index constructed by: device serial number-gateway reception time-node shard-storage data block. Compared with the first-category query index, the second-category query index provides a faster routing decision and can directly locate the specific physical storage location. Although the first-category query index is not specific to the target storage data block, it can find the matching storage data block in the node shard according to the gateway reception time in the first-category query index. Therefore, the query efficiency of the second-category query index is slightly faster than that of the first-category query index.

[0068] S260: Add the query index to a distributed table of a distributed query engine, so as to quickly respond to query requests for operating condition data through the distributed table.

[0069] A distributed query engine can be a component that performs query operations in a distributed system. It can process query requests in parallel on multiple nodes and quickly return query results by integrating and analyzing data distributed on each node. A distributed table is a data logic table distributed on multiple nodes. Each node stores a portion of the table's data. The data is logically a whole, but physically divided and stored on different nodes. After creating a query index, it needs to be added to the distributed table of the distributed query engine so that when processing query requests, the distributed query engine can use these historical indexes to quickly locate and return the required data.

[0070] Optionally, the method may further include:

[0071] Whenever a target query request from a user is received, extract the query device serial number and query time interval contained in the target query request;

[0072] Matching is performed in a distributed table according to the query device serial number and the query time interval through a distributed query engine, at least one matching node shard is located in the distributed system according to the matching result, and a matching data block is located in the matching node shard;

[0073] Through the distributed query engine, each matching data block is queried to obtain each standardized working condition data that matches the target query request, and each queried standardized working condition data is used as a response result of the target query request.

[0074] In actual application scenarios, the only field identifiers that exist in the operating data uploaded by various types of devices are the device serial number and the gateway reception time. Therefore, when data analysts need to query certain data, they will send a query request to the distributed system through some means (such as user interface, API interface, etc.). The request contains the device serial number that the user wants to query (used to identify a specific IoT device) and the query time interval (indicating that the user wants to query the operating data of the device within a specific time period).

[0075] Further, matching in the distributed table according to the query device serial number and the query time interval by the distributed query engine may include:

[0076] Extracting multiple candidate query indexes matching the query sequence number from the query indexes of the distributed table through the distributed query engine;

[0077] Extracting the candidate gateway reception time in each candidate query index, and matching each candidate gateway time with the query time interval;

[0078] The current candidate query index whose candidate gateway time falls within the query time interval is used as a successfully matched target query index, and the target query index is used as a matching result.

[0079] First, the distributed query engine is activated and executes the query task by accessing the query index in the distributed table. The query index is constructed based on the device serial number, gateway reception time, and node sharding information (which may also include data block information). In actual application scenarios, data analysts can specify which working condition data to query in the query request. Therefore, the device serial number under each type of working condition data is certain, but the gateway reception time of the data received in real time is not the same. Therefore, there are multiple index items containing the same device serial number but different gateway reception times, that is, there are multiple candidate query indexes. For each candidate query index, the distributed query engine can extract the candidate gateway reception time and determine whether it hits the query time interval in the query request. The candidate query index that hits the interval is regarded as a successful match and marked as the target query index. The data located according to the target query index is a part of the data that meets the query request, and the part of the data refers to the data that can be quickly found and returned through the query index.

[0080] The technical solution of the embodiment of the present invention describes in detail the creation and use of the query index by refining the overall solution. Specifically, by creating a query index, relevant data blocks and node shards can be quickly located, thereby greatly improving query efficiency, which is crucial for IoT applications that require real-time or fast access to a large amount of working condition data; secondly, the distributed query engine can intelligently assign query tasks to relevant node shards based on the information of the query index, thereby avoiding unnecessary waste of resources and waiting time, and optimizing the user experience.

[0081] Embodiment 3

[0082] Figure 3 This is a schematic diagram of the structure of a distributed processing device for working condition data provided by Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0083] The message queue generation module 310 is used to process the real-time working condition data reported by the IoT device into standardized working condition data whenever the real-time working condition data is detected, and add the standardized working condition data to the message queue as a message to be consumed;

[0084] The joint message extraction module 320 is used to obtain the current message from the message queue in real time through the data consumption engine, and extract the current device serial number and the current gateway receiving time contained in the current message;

[0085] The hash value calculation module 330 is used to combine the current device serial number and the current gateway receiving time into a current joint string through the data consumption engine, and calculate the current hash value matching the current joint string according to the target hash algorithm;

[0086] The balanced storage module 340 is used to determine the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine, and store the current message in the current storage data block of the target node shard; wherein each data block in the distributed system is used to store messages within a set gateway receiving time interval.

[0087] The technical solution of the embodiment of the present invention processes the working condition data reported by the Internet of Things devices in real time, converts it into standardized working condition data, and immediately adds it to the message queue, thereby ensuring the timeliness and accuracy of the data; the data consumption engine can quickly obtain and process data from the message queue, and through the hash algorithm and distributed storage mechanism, realizes rapid positioning and storage of data, and improves data processing efficiency; by calculating the hash value of the data and distributing the data to different node shards according to the hash value range, it ensures the balanced distribution of data in the distributed system. This balanced storage method avoids the problem of unbalanced storage pressure caused by the centralized storage of data on a certain node shard, and improves the stability and reliability of the entire distributed system; in general, it realizes balanced distribution and orderly storage of data, has good scalability, makes system maintenance and management more convenient, and reduces maintenance costs.

[0088] Optionally, based on the above embodiments, the invention may further include: a node shard mapping unit, which determines the number of node shards participating in data storage according to the storage architecture of the distributed system before determining the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine;

[0089] Obtain multiple pieces of historical operating condition data within a set time period, and obtain a historical joint string of each piece of historical operating condition data;

[0090] According to the target hash algorithm, calculate the historical hash value corresponding to each historical joint string, and obtain the maximum hash value and the minimum hash value among the historical hash values;

[0091] Constructing a hash value interval according to the maximum hash value and the minimum hash value, and dividing the hash value interval into multiple hash value ranges according to the number of node shards;

[0092] A one-to-one correspondence between each node shard and each hash value range in each distributed system is established.

[0093] Optionally, based on the above embodiments, it may further include a unit for re-assigning a current storage data block for real-time storage of messages in each node shard in the distributed system whenever it is detected that the current system time reaches a new data block designation condition;

[0094] Obtain the most recently specified historical storage data blocks, and determine the historical gateway receiving time interval of each historical storage data block according to the historical specified time of each historical storage data block and the current system time;

[0095] The historical gateway receiving time interval is used as the partition key and added to each historical storage data block.

[0096] Optionally, on the basis of the above embodiments, it may also include a distributed table expansion unit, which, after determining the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine, creates a query index according to the current device serial number, the current gateway reception time and the target node shard, or according to the current device serial number, the current gateway reception time, the target node shard and the current storage data block on the target node shard;

[0097] The query index is added to the distributed table of the distributed query engine to quickly respond to query requests for operating condition data through the distributed table.

[0098] Optionally, based on the above embodiments, it may further include a query request response unit, which is used to extract the query device serial number and query time interval contained in the target query request whenever receiving a target query request from a user;

[0099] Matching is performed in a distributed table according to the query device serial number and the query time interval through a distributed query engine, at least one matching node shard is located in the distributed system according to the matching result, and a matching data block is located in the matching node shard;

[0100] Through the distributed query engine, each matching data block is queried to obtain each standardized working condition data that matches the target query request, and each queried standardized working condition data is used as a response result of the target query request.

[0101] Optionally, based on the above embodiments, the query request response unit may also be used to extract multiple candidate query indexes matching the query sequence number from the query indexes of the distributed table through a distributed query engine;

[0102] Extracting the candidate gateway reception time in each candidate query index, and matching each candidate gateway time with the query time interval;

[0103] The current candidate query index whose candidate gateway time falls within the query time interval is used as a successfully matched target query index, and the target query index is used as a matching result.

[0104] A distributed processing device for operating condition data provided by an embodiment of the present invention can execute a distributed processing method for operating condition data provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0105] Embodiment 4

[0106] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0108] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a distributed processing method for operating condition data.

[0110] That is, whenever the real-time working condition data reported by the IoT device is detected, the real-time working condition data is processed into standardized working condition data, and the standardized working condition data is added to the message queue as a message to be consumed;

[0111] The data consumption engine obtains the current message from the message queue in real time, and extracts the current device serial number and the current gateway receiving time contained in the current message;

[0112] The data consumption engine combines the current device serial number and the current gateway receiving time into a current joint string, and calculates the current hash value matching the current joint string according to the target hash algorithm;

[0113] The data consumption engine determines the target node shard according to the hash value range of each node shard in the distributed system, and stores the current message in the current storage data block of the target node shard; wherein each data block in the distributed system is used to store messages within the set gateway receiving time interval.

[0114] In some embodiments, a distributed processing method for operating condition data may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the distributed processing method for operating condition data described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute a distributed processing method for operating condition data in any other appropriate manner (e.g., by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0120] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed processing method for working condition data, characterized in that: The method is performed by a distributed system, the distributed system includes a plurality of node shards, each node shard includes a plurality of data blocks, and includes: Whenever the real-time working condition data reported by the IoT device is detected, the real-time working condition data is processed into standardized working condition data, and the standardized working condition data is added to the message queue as a message to be consumed; The data consumption engine obtains the current message from the message queue in real time, and extracts the current device serial number and the current gateway receiving time contained in the current message; The data consumption engine combines the current device serial number and the current gateway receiving time into a current joint string, and calculates the current hash value matching the current joint string according to the target hash algorithm; The data consumption engine determines the target node shard according to the hash value range of each node shard in the distributed system, and stores the current message in the current storage data block of the target node shard; wherein each data block in the distributed system is used to store messages within the set gateway receiving time interval.

2. The method according to claim 1, characterized in that Before the target node shard is determined by the data consumption engine according to the hash value range of each node shard in the distributed system, it also includes: Determine the number of node shards involved in data storage based on the storage architecture of the distributed system; Obtain multiple pieces of historical operating condition data within a set time period, and obtain a historical joint string of each piece of historical operating condition data; According to the target hash algorithm, calculate the historical hash value corresponding to each historical joint string, and obtain the maximum hash value and the minimum hash value among the historical hash values; Constructing a hash value interval according to the maximum hash value and the minimum hash value, and dividing the hash value interval into multiple hash value ranges according to the number of node shards; A one-to-one correspondence between each node shard and each hash value range in each distributed system is established.

3. The method according to claim 1, characterized in that: Also includes: Whenever it is detected that the current system time reaches the new data block designation condition, the current storage data block for real-time storage of messages is re-designated in each node shard in the distributed system; Obtain the most recently specified historical storage data blocks, and determine the historical gateway receiving time interval of each historical storage data block according to the historical specified time of each historical storage data block and the current system time; The historical gateway receiving time interval is used as the partition key and added to each historical storage data block.

4. The method according to any one of claims 1 to 3, characterized in that: After the target node shard is determined by the data consumption engine according to the hash value range of each node shard in the distributed system, it also includes: Create a query index based on the current device serial number, the current gateway receiving time, and the target node shard, or based on the current device serial number, the current gateway receiving time, the target node shard, and the current storage data block on the target node shard; The query index is added to the distributed table of the distributed query engine to quickly respond to query requests for operating condition data through the distributed table.

5. The method according to claim 4, characterized in that Also includes: Whenever a target query request from a user is received, extract the query device serial number and query time interval contained in the target query request; Matching is performed in a distributed table according to the query device serial number and the query time interval through a distributed query engine, at least one matching node shard is located in the distributed system according to the matching result, and a matching data block is located in the matching node shard; Through the distributed query engine, each matching data block is queried to obtain each standardized working condition data that matches the target query request, and each queried standardized working condition data is used as a response result of the target query request.

6. The method according to claim 5, characterized in that The distributed query engine is used to match the query device serial number and query time interval in the distributed table, including: Extracting multiple candidate query indexes matching the query sequence number from the query indexes of the distributed table through the distributed query engine; Extracting the candidate gateway reception time in each candidate query index, and matching each candidate gateway time with the query time interval; The current candidate query index whose candidate gateway time falls within the query time interval is used as a successfully matched target query index, and the target query index is used as a matching result.

7. A distributed processing device for working condition data, characterized in that: Arranged in a distributed system, the distributed system includes a plurality of node shards, each node shard includes a plurality of data blocks, the method includes: A message queue generation module is used to process the real-time working condition data reported by the IoT device into standardized working condition data whenever the real-time working condition data is detected, and add the standardized working condition data to the message queue as a message to be consumed; The joint message extraction module is used to obtain the current message from the message queue in real time through the data consumption engine, and extract the current device serial number and the current gateway receiving time contained in the current message; A hash value calculation module, used to combine the current device serial number and the current gateway receiving time into a current joint string through the data consumption engine, and calculate the current hash value matching the current joint string according to the target hash algorithm; The balanced storage module is used to determine the target node shard according to the hash value range of each node shard in the distributed system through the data consumption engine, and store the current message in the current storage data block of the target node shard; wherein each data block in the distributed system is used to store messages within the set gateway receiving time interval.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a distributed processing method for operating condition data according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a distributed processing method for operating condition data according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements a distributed processing method for operating condition data according to any one of claims 1 to 6.

Images