A method and system for collecting big data

Abstract

The present invention is applicable to the field of data acquisition technology, and in particular, relates to a method and system for collecting big data, the method comprising: demarcating a production area, locating the deployment position of a sensor device, collecting sensor data, creating a processing platform for sensor data, uploading the sensor data to the processing platform via a pre-built end-to-end transmission link; configuring the attributes of the sensor device, wherein the attributes at least include: data volume, sampling frequency and network bandwidth, constructing a weight query table, wherein the weight query table includes: attribute items and weight value items. The present invention can monitor the network load of all sensor devices in real time by constructing a stack of blocks, and when the load of the overall sensor network is high, the sensor devices with poor network status are connected to the edge nodes, thereby alleviating network bottlenecks, reducing delays, and improving the response speed of the overall sensor network, thereby greatly improving the stability of the sensor network and the data acquisition efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a method and system for collecting big data. Background Art

[0002] In the field of industrial production, the collection of big data usually relies on Internet of Things technology, that is, through industrial sensors deployed in production equipment, workshops or control terminals, real-time collection of equipment operating status, production process parameters, energy consumption data and environmental monitoring data, etc., and the collected sensor data is transmitted to the cloud, combined with artificial intelligence, machine learning and other technologies for data analysis and mining, thereby improving overall production efficiency.

[0003] On the other hand, in the process of transmitting sensor data to the cloud, when the network delay is large, it will greatly affect the timeliness and collection efficiency of data transmission. Therefore, by deploying edge devices, the data transmission distance and time can be reduced, and network delay can be reduced; however, edge devices often have limited computing power. When the amount of data to be processed is very large or complex, the edge devices may not be able to meet the needs; therefore, "how to enable edge devices in a timely manner and switch the data transmission links of some sensors" is the technical problem that the present invention needs to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a big data collection method and system to solve the problem of "how to enable edge devices in a timely manner and switch the data transmission links of some sensors" raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for collecting big data, the method comprising: Delineate the production area, locate the deployment location of the sensor equipment, collect sensor data, create a sensor data processing platform, and upload the sensor data to the processing platform via a pre-built end-to-end transmission link; Configuring the attributes of the sensor device, wherein the attributes at least include: data volume, sampling frequency and network bandwidth, constructing a weight query table, wherein the weight query table includes: attribute items and weight value items, collecting real-time data of the attributes, traversing the weight query table, and superimposing and calculating the transmission score of each sensor device; Creating blocks corresponding to the sensing devices one by one, synchronizing the sensing data to the corresponding blocks, sorting all the blocks in ascending order of transmission scores, generating a block stack, configuring a limited height of the block stack, calculating an average value of the transmission scores, setting the height of each block, and obtaining a standard height; Insert the tail block to the bottom of the block stack, and define the height of the tail block as the fluctuation height. Take the fluctuation height and the average value as the center, build a synchronous update mechanism, and determine whether any blocks in the block stack are pushed out. If so, traverse the edge nodes in the production area, and mount the sensor equipment corresponding to the blocks on the edge nodes, build an end-edge-end transmission link, and based on the synchronous update mechanism, push the pushed out blocks back into the block stack.

[0006] Furthermore, the synchronous update mechanism is: calculating the change in the average value, integrating the change and a preset ratio value to obtain an adjustment amount, and using the adjustment amount to correct the fluctuation height.

[0007] Furthermore, the method further comprises: The sum of the standard heights of all blocks in the block stack is superimposed on the fluctuation height to obtain the real-time height; Configuring a variation range of the limited height and determining an upper limit and a lower limit; If the real-time height is greater than the upper limit, the block at the top of the block stack is ejected; If the real-time height is less than or equal to the lower limit, the number blocks corresponding to the sensor devices hanging in the edge node are pushed back into the number block stack.

[0008] Furthermore, the steps of demarcating the production area, locating the deployment position of the sensor equipment, collecting sensor data, creating a control platform for processing the sensor data, and uploading the sensor data to the control platform via a pre-built end-to-end transmission link include: Dividing the sensor data into a plurality of levels, wherein each level corresponds to a monitoring frequency; The levels and monitoring frequencies are integrated to generate a frequency comparison table, and the frequency comparison table is sent to the sensor device.

[0009] Furthermore, the step of superimposing and calculating the transmission score of each sensor device includes: Divide the attribute into several items, compare the weight query table, determine the weight value corresponding to each item, and add them up to obtain the transmission score of the sensor device; In the processing platform, a label generated by the transmission score is inserted into the block.

[0010] Furthermore, the step of sorting all the number blocks to obtain a number block stack includes: Embedding redundant blocks into the block stack and integrating pre-built redundancy rules into the redundant blocks; The edge node is given the permission to read the data of the redundant blocks.

[0011] Furthermore, the step of inserting a tail block into the bottom of the block stack, defining the height of the tail block as a fluctuation height, and constructing a synchronous update mechanism with the fluctuation height and the average value as the center includes: Determining the influencing factors of the fluctuation height, wherein the influencing factors at least include: network parameters and idle resources in edge nodes; When several blocks are ejected, trigger signals are generated in parallel and sent to edge nodes.

[0012] Furthermore, the system comprises: The upload module is used to define the production area, locate the deployment position of the sensor equipment, collect sensor data, create a processing platform for the sensor data, and upload the sensor data to the processing platform via a pre-built end-to-end transmission link; A superposition module is used to configure the attributes of the sensor device, wherein the attributes include at least: data volume, sampling frequency and network bandwidth, construct a weight query table, wherein the weight query table consists of attribute items and weight value items, collect real-time data of the attributes, traverse the weight query table, and superimpose and calculate the transmission score of each sensor device; A module is provided for creating a number block corresponding to the sensor device one by one, synchronizing the sensor data to the corresponding number block, sorting all the number blocks in the order of transmission scores from small to large, generating a number block stack, configuring a limited height of the number block stack, calculating an average value of the transmission score, setting the height of each number block, and obtaining a standard height; The push module is used to insert the tail block into the bottom of the block stack, and define the height of the tail block as the fluctuation height. With the fluctuation height and the average value as the center, a synchronous update mechanism is constructed to determine whether there are blocks in the block stack that have been pushed out. If so, in the production area, the edge nodes are traversed, and the sensor equipment corresponding to the blocks is mounted on the edge nodes, an end-edge-end transmission link is established, and based on the synchronous update mechanism, the pushed-out blocks are pushed back into the block stack.

[0013] Furthermore, the upload module includes: A segmentation unit, used for segmenting the sensor data into a plurality of levels, wherein each level corresponds to a monitoring frequency; The integration unit is used to integrate the level and the monitoring frequency, generate a frequency comparison table, and send the frequency comparison table to the sensor device.

[0014] Furthermore, the superposition module includes: A comparison unit, used to divide the attribute into a plurality of individual items, compare the weight query table, determine the weight value corresponding to each individual item, and add them together to obtain the transmission score of the sensor device; An inserting unit is used to insert a label generated by the transmission score into the block in the processing platform.

[0015] Compared with the prior art, the present invention has the following beneficial effects: By collecting sensor data, it is possible to monitor and acquire data on production equipment and environment in real time, and through the processing platform, accurately analyze sensor data, greatly improving decision-making efficiency. By collecting the properties of each sensor device, the network status of the sensor device can be determined, thereby providing a data basis for the adjustment of the transmission link. By building a stack of blocks, the network load of all sensor devices can be monitored in real time, and when the load of the overall sensor network is high, the sensor devices with poor network status can be connected to the edge node, thereby alleviating network bottlenecks and reducing delays. At the same time, the response speed of the overall sensor network is improved, greatly improving the stability of the sensor network and data collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a block stack structure provided by an embodiment of the present invention; Figure 2 A flowchart of a method for collecting big data provided by an embodiment of the present invention; Figure 3 A first sub-flow diagram of the big data collection method provided by an embodiment of the present invention; Figure 4 A second sub-flow diagram of the big data collection method provided by an embodiment of the present invention; Figure 5 A third sub-flow chart of the big data collection method provided in an embodiment of the present invention; Figure 6 A fourth sub-flow chart of the big data collection method provided in an embodiment of the present invention; Figure 7 A block diagram of the composition of a big data collection system provided by an embodiment of the present invention; Figure 8 A block diagram of the composition of an upload module in a big data collection system provided by an embodiment of the present invention; Fig. 9 A block diagram of the composition of a superposition module in a big data collection system provided by an embodiment of the present invention; Fig.10 A block diagram of the composition of a module in a big data collection system provided by an embodiment of the present invention; Fig.11 A block diagram of the composition of the push module in the big data collection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] In Example 1, Figure 1 and Figure 2 The implementation process of the big data collection method provided by the embodiment of the present invention is shown and described in detail below: S100: Delineate the production area, locate the deployment position of the sensor equipment, collect sensor data, create a sensor data processing platform, and upload the sensor data to the processing platform via a pre-built end-to-end transmission link.

[0019] Determine the production area, where the production area is the raw material receiving area, production area and storage area, or environmental monitoring area in the production workshop; deploy sensor equipment in the production area according to actual production needs, where the sensor equipment includes not only various types of sensors, but also acquisition modules of production equipment; record the deployment location of the sensor equipment, and use the sensor equipment to collect the required sensor data, where the sensor data includes production data such as temperature, humidity, and pressure, and also includes environmental data such as noise and gas; create a processing platform, where the processing platform is mainly used to receive, store, process and analyze data from different sensor devices; further, the processing platform can be a DCS central control platform or a PCS7 control platform in the prior art; connect the sensor equipment to the processing platform, and build an end-to-end transmission link, where the end-to-end refers to from the source end (sensor device) to the target end (processing platform), and upload the sensor data to the processing platform through wireless communication (Wi-Fi, 4G / 5G and Bluetooth, etc.).

[0020] S200: Configure the attributes of the sensor device, wherein the attributes include at least: data volume, sampling frequency and network bandwidth, construct a weight query table, wherein the weight query table consists of attribute items and weight value items, collect real-time data of the attributes, traverse the weight query table, and superimpose and calculate the transmission score of each sensor device.

[0021] Collect the properties of the sensor device, including: data volume, sampling frequency and network bandwidth. Specifically, data volume refers to the size of data generated by the sensor device at each sampling. The size of the data volume is usually related to the type of sensor and the output accuracy. The sampling frequency refers to the number of times the sensor device collects data per second. The network bandwidth refers to the network bandwidth required for the sensor device to transmit data, which depends on the combination of data volume and sampling frequency. In order to improve the accuracy of the network status assessment of the sensor device, the properties may also include other relevant data, such as accuracy.

[0022] Each attribute corresponds to a weight value, and the correspondence between the attribute and the weight value is stored in the weight query table. After the real-time data of the attribute of a certain sensor device is determined, the weight value of each attribute can be determined by querying the weight query table. By superimposing all the weight values, the transmission score can be obtained.

[0023] For example: construct three weight query tables as shown below (Table 1, Table 2 and Table 3); Table 1:

[0024] Table 2:

[0025] Table 3:

[0026] The data volume of a certain sensor device is 30KB, the acquisition frequency is 800Hz and the network bandwidth is 400KBps. By querying the above weight query table, the corresponding weight values ​​are 2, 3 and 2 respectively. The transmission score of the sensor device is 7.

[0027] S300: Create digital blocks corresponding to the sensing devices one by one, synchronize the sensing data to the corresponding digital blocks, sort all the digital blocks in order of transmission scores from small to large, generate a digital block stack, configure a limited height of the digital block stack, calculate the average value of the transmission scores, set the height of each of the digital blocks, and obtain a standard height.

[0028] A corresponding digital block is created for each sensor device, where the digital block is mainly used to characterize the sensor device, receive and store the sensor data of the corresponding sensor device, and is not a physical device or other device with processing capabilities; all digital blocks are sorted in order of the transmission scores of the corresponding sensor devices from small to large, and integrated to generate a digital block stack; the digital block stack is a collection of sorted digital blocks, which is similar to a queue, but different from a queue in that the digital block stack follows the principle of top-in and top-out; for example: after a digital block is pushed out from the top of the digital block stack, it will also enter from the top of the digital block stack when it is pushed back into the digital block stack; the height of the data stack is set, that is, the limited height, where the limited height is determined by professionals based on edge node performance, production load, actual work experience, etc.; the transmission scores of all digital blocks in the digital block stack are weighted averaged to obtain an average value; and the height of each digital block is set to a standard height.

[0029] For example, the height of the number block stack is determined to be 15 cm, and the height of each number block is determined to be 0.4 cm, that is, the limited height is 15 cm and the standard height is 0.4 cm.

[0030] S400: Insert the tail block to the bottom of the block stack, and define the height of the tail block as the fluctuation height. Take the fluctuation height and the average value as the center, build a synchronous update mechanism, and determine whether there are blocks in the block stack that have been pushed out. If so, traverse the edge nodes in the production area, and mount the sensor devices corresponding to the blocks to the edge nodes, build an end-edge-end transmission link, and based on the synchronous update mechanism, push the pushed out blocks back into the block stack.

[0031] Insert the tail block to the bottom of the block stack, where the bottom of the block stack is below the last block in the sorting order. The tail block is the tail block, which is mainly used to characterize the load of the end-to-end transmission link. Compared with the ordinary block, the height of the tail block is fixed, except that the object represented is different. The height of the tail block is not limited, and it will change with the load fluctuation of the end-to-end transmission link. The specific following change process is limited by the synchronous update mechanism.

[0032] If the height of the tail block is increased through the synchronous update mechanism, and the first block in the sorting is pushed out of the block stack (as shown in the attached manual), Figure 1As shown in the figure, the sensor device corresponding to the ejected block is mounted on the edge node; at the same time, an end-to-edge-to-end transmission link is established, that is, from the source end (sensor device) to the edge node, and then to the target end (processing platform); after the block is ejected, the load fluctuation of the end-to-end transmission link continues to be monitored. If the height of the tail block is reduced and the block stack can accommodate the ejected block again, the ejected block is reinserted (synonymous with "pushed") into the block stack, and the order of all blocks is adjusted. At the same time, the sensor device corresponding to the reinserted block is removed from the edge node, and the sensor data collected by it is directly transmitted to the processing platform.

[0033] In this embodiment, if the load of the end-to-end transmission link is large, the sensor devices with large network load are found out in turn and connected to the edge node until the load of the end-to-end transmission link tends to be stable; on the contrary, if the current load of the end-to-end transmission link is small, the sensor device in the edge node is removed, and the sensor data in the sensor device is directly uploaded to the processing platform.

[0034] In Example 2, the synchronous update mechanism is as follows: calculating the change in the average value, integrating the change and the preset ratio value to obtain an adjustment amount, and using the adjustment amount to correct the fluctuation height. The average value is updated according to the preset frequency, and the average values ​​before and after the update are compared. The difference is defined as the change, the change is multiplied by the preset proportional value to get the adjustment, and the adjustment is superimposed on the fluctuation height to get a new fluctuation height. It should be noted that the change and adjustment may be positive, 0 or negative. The frequency and standard values ​​are pre-determined by professionals.

[0035] For example, the current average value of the block stack is 11. After a certain period of time, the average value becomes 15, and the change is 4. Multiply the change by the preset ratio (assumed to be 1.3) to get the adjustment amount, which is 5.2, and superimpose the adjustment amount with the fluctuation height.

[0036] Furthermore, when the fluctuation height changes, three results will be produced. The first is: the fluctuation height becomes larger, and the number block ranked first is pushed out of the number block stack. The second is: the fluctuation height does not change. The third is: the fluctuation height becomes smaller and can accommodate the number blocks pushed out, so the number blocks are pushed back into the number block stack.

[0037] In Example 3, different from Example 1, in this embodiment of the present invention, the method further includes: The sum of the standard heights of all blocks in the block stack is superimposed on the fluctuation height to obtain the real-time height; Configuring a variation range of the limited height and determining an upper limit and a lower limit; If the real-time height is greater than the upper limit, the first-ranked block in the block stack is popped out; If the real-time height is less than or equal to the lower limit, the number blocks corresponding to the sensor devices hanging in the edge node are pushed back into the number block stack.

[0038] Construct the real-time height, where the real-time height is the sum of the height of all blocks in the block stack and the height of the tail block; among the above three results, there may be special cases, namely: although the fluctuation height increases, the change amount is small (the change amount is less than the standard height of the block), and the block at the top of the sort cannot be pushed out; on the contrary, although the fluctuation height decreases (the change amount is less than or equal to the standard height of the block), the block stack cannot accommodate the block; therefore, construct the upper and lower limits to refine the pushing and ejecting of blocks.

[0039] When the real-time height is greater than the upper limit, the first block in the block stack is popped out; when the real-time height is less than or equal to the lower limit, the block corresponding to the sensor device hanging in the edge node is pushed back into the block stack.

[0040] In Example 4, Figure 3 The implementation process of the big data collection method provided by the embodiment of the present invention is shown. The steps of demarcating the production area, locating the deployment position of the sensor equipment, collecting the sensor data, creating a control platform for processing the sensor data, and uploading the sensor data to the control platform via a pre-built end-to-end transmission link are described in detail as follows: S101: Divide the sensor data into several levels, wherein each level corresponds to a monitoring frequency.

[0041] The collected sensor data is segmented and the monitoring frequency corresponding to the sensor data is determined; the advantage of this is that the monitoring frequency can be dynamically adjusted according to the changes in the sensor data.

[0042] For example, an indicator range is configured for each sensor data. If the real-time sensor data is within the indicator range, the monitoring frequency is set to 1 time / s. If it exceeds the indicator range, the monitoring frequency can be increased to 2 times / s.

[0043] S102: Integrate the level and monitoring frequency, generate a frequency comparison table, and send the frequency comparison table to the sensor device.

[0044] A frequency comparison table is generated using the level and monitoring frequency; by constructing the frequency comparison table, the query efficiency of the monitoring frequency can be improved.

[0045] In Example 5, Figure 4The implementation process of the big data collection method provided by the embodiment of the present invention is shown, and the steps of superimposing and calculating the transmission score of each sensor device are described in detail as follows: S201: Divide the attribute into several items, compare the weight query table, determine the weight value corresponding to each item, and add them up to obtain the transmission score of the sensor device.

[0046] The attributes are divided into data volume items, sampling frequency items, network bandwidth items, etc., as shown in the weight query table in S200, and each single item corresponds to a set of weight values.

[0047] S202: In the processing platform, inserting a label generated by the transmission score into the block.

[0048] After determining the transmission score of each sensor device, a label is generated using the transmission score and inserted into the number block corresponding to the transmission device; by inserting the label, the calculation efficiency of the average value and the fluctuation height can be greatly improved.

[0049] In Example 6, Figure 5 The implementation process of the big data collection method provided by the embodiment of the present invention is shown. The steps of sorting all the blocks to obtain the block stack are described in detail as follows: S301: embed redundant digital blocks into the digital block stack, and integrate pre-built redundant rules into the redundant digital blocks.

[0050] Insert redundant blocks into the block stack, where the redundant blocks are mainly used to represent storage devices, and back up the redundant blocks using the storage devices and redundant rules; the redundant rules are specific backup methods, such as incremental backup or cyclic backup.

[0051] S302: Opening data reading authority of redundant blocks to the edge node.

[0052] The data reading permission of redundant blocks is opened to edge nodes, so that edge nodes can read the backup data in redundant blocks and provide reference for the processing of sensor data.

[0053] In Example 7, Figure 6 The implementation process of the big data collection method provided by the embodiment of the present invention is shown. The following is a detailed description of the steps of inserting a tail block into the bottom of the block stack, defining the height of the tail block as the fluctuation height, and constructing a synchronous update mechanism with the fluctuation height and the average value as the center, as follows: S401: Determine the influencing factors of the fluctuation height, wherein the influencing factors at least include: network parameters and idle resources in edge nodes.

[0054] In addition to network latency, factors affecting the fluctuation height (load of the end-to-end transmission link) also include: network parameters (such as signal strength and bandwidth, etc.) and idle resources in the edge node; for example, if the idle resources of the edge node are greater than the threshold, the fluctuation height will be reduced and the number of sensor devices mounted in the edge node will be increased.

[0055] S402: When a number block is ejected, a trigger signal is generated in parallel and sent to an edge node.

[0056] When the block is popped out of the block stack, a trigger signal is generated. When the edge node receives the trigger signal, it starts. In other words, if the edge device is not mounted with a sensor device, the edge device will not start but will be in a dormant state.

[0057] Figure 7 The following is a structural block diagram of a big data collection system according to an embodiment of the present invention. The big data collection system 1 includes: The uploading module 11 is used to define the production area, locate the deployment position of the sensor equipment, collect sensor data, create a processing platform for the sensor data, and upload the sensor data to the processing platform via a pre-built end-to-end transmission link; The superposition module 12 is used to configure the attributes of the sensor device, wherein the attributes include at least: data volume, sampling frequency and network bandwidth, construct a weight query table, wherein the weight query table consists of attribute items and weight value items, collect real-time data of the attributes, traverse the weight query table, and superimpose and calculate the transmission score of each sensor device; Obtaining module 13, used for creating digital blocks corresponding to the sensing devices one by one, synchronizing the sensing data to the corresponding digital blocks, sorting all the digital blocks in the order of transmission scores from small to large, generating a digital block stack, configuring a limited height of the digital block stack, calculating an average value of the transmission scores, setting the height of each of the digital blocks, and obtaining a standard height; The push module 14 is used to insert the tail block into the bottom of the block stack, and define the height of the tail block as the fluctuation height. With the fluctuation height and the average value as the center, a synchronous update mechanism is constructed to determine whether there are blocks popped out of the block stack. If so, in the production area, the edge nodes are traversed, and the sensor equipment corresponding to the blocks is mounted on the edge nodes, an end-edge-end transmission link is established, and based on the synchronous update mechanism, the popped-out blocks are pushed back into the block stack.

[0058] Figure 8 The structure diagram of the big data collection system provided by the embodiment of the present invention is shown, and the upload module 11 includes: A segmentation unit 111, used for segmenting the sensor data into a plurality of levels, wherein each level corresponds to a monitoring frequency; The integration unit 112 is used to integrate the level and the monitoring frequency, generate a frequency comparison table, and send the frequency comparison table to the sensor device.

[0059] Fig. 9 The structure block diagram of the big data collection system provided by the embodiment of the present invention is shown, and the superposition module 12 includes: A comparison unit 121 is used to divide the attribute into a plurality of individual items, compare the weight query table, determine the weight value corresponding to each individual item, and add them together to obtain the transmission score of the sensor device; The inserting unit 122 is used to insert the label generated by the transmission score into the sensor device in the processing platform.

[0060] Fig.10 The structure block diagram of the big data collection system provided by the embodiment of the present invention is shown, and the obtaining module 13 includes: An integration unit 131, configured to embed redundant blocks into the block stack and integrate pre-built redundant rules into the redundant blocks; The opening unit 132 is used to open the data reading authority of the redundant blocks to the edge node.

[0061] Fig.11 The structure diagram of the big data collection system provided by the embodiment of the present invention is shown, and the push module 14 includes: A determining unit 141 is configured to determine influencing factors of the fluctuation height, wherein the influencing factors at least include: network parameters and idle resources in edge nodes; The sending unit 142 is used to generate a trigger signal in parallel when a number of blocks are ejected, and send the trigger signal to the edge node.

[0062] The uploading module 11 is mainly used to complete step S100, the superposition module 12 is mainly used to complete step S200, the obtaining module 13 is mainly used to complete step S300, and the pushing module 14 is mainly used to complete step S400; The segmentation unit 111 is mainly used to complete step S101, and the integration unit 112 is mainly used to complete step S102; The comparison unit 121 is mainly used to complete step S201, and the insertion unit 122 is mainly used to complete step S202; The integration unit 131 is mainly used to complete step S301, and the opening unit 132 is mainly used to complete step S302; The determining unit 141 is mainly used to complete step S401, and the sending unit 142 is mainly used to complete step S402.

[0063] In this application, as shown in the attached specification Figure 1 As shown, block A needs to be pushed out of the block stack. In other words, the sensor device corresponding to block A needs to be connected to the edge node. If the fluctuation height of the bottom block is reduced so that the pushed-out block A can completely enter the block stack (block B has not completely entered the block stack, but block C has completely entered the block stack), block A is pushed back into the block stack.

[0064] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0065] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for collecting big data, characterized in that: The method comprises: Delineate the production area, locate the deployment location of the sensor equipment, collect sensor data, create a sensor data processing platform, and upload the sensor data to the processing platform via a pre-built end-to-end transmission link; Configuring the attributes of the sensor device, wherein the attributes include at least: data volume, sampling frequency and network bandwidth, constructing a weight query table, wherein the weight query table consists of attribute items and weight value items, collecting real-time data of the attributes, traversing the weight query table, and superimposing and calculating the transmission score of each sensor device; Creating blocks corresponding to the sensing devices one by one, synchronizing the sensing data to the corresponding blocks, sorting all the blocks in ascending order of transmission scores, generating a block stack, configuring a limited height of the block stack, calculating an average value of the transmission scores, setting the height of each block, and obtaining a standard height; Insert the tail block to the bottom of the block stack, and define the height of the tail block as the fluctuation height. Take the fluctuation height and the average value as the center, build a synchronous update mechanism, and determine whether any blocks in the block stack are pushed out. If so, traverse the edge nodes in the production area, and mount the sensor equipment corresponding to the blocks on the edge nodes, build an end-edge-end transmission link, and based on the synchronous update mechanism, push the pushed out blocks back into the block stack.

2. The big data collection method according to claim 1, characterized in that: The synchronous update mechanism is: calculating the change in the average value, integrating the change and a preset ratio value to obtain an adjustment amount, and using the adjustment amount to correct the fluctuation height.

3. The big data collection method according to claim 1, characterized in that: The method further comprises: The sum of the standard heights of all blocks in the block stack is superimposed on the fluctuation height to obtain the real-time height; Configuring a variation range of the limited height and determining an upper limit and a lower limit; If the real-time height is greater than the upper limit, the block at the top of the block stack is ejected; If the real-time height is less than or equal to the lower limit, the number blocks corresponding to the sensor devices hanging in the edge node are pushed back into the number block stack.

4. The method for collecting big data according to claim 1, characterized in that: The steps of demarcating the production area, locating the deployment position of the sensor equipment, collecting sensor data, creating a control platform for processing the sensor data, and uploading the sensor data to the control platform via a pre-built end-to-end transmission link include: Dividing the sensor data into a plurality of levels, wherein each level corresponds to a monitoring frequency; The levels and monitoring frequencies are integrated to generate a frequency comparison table, and the frequency comparison table is sent to the sensor device.

5. The method for collecting big data according to claim 1, characterized in that: The step of calculating the transmission score of each sensor device by superposition includes: Divide the attribute into several items, compare the weight query table, determine the weight value corresponding to each item, and add them up to obtain the transmission score of the sensor device; In the processing platform, a label generated by the transmission score is inserted into the block.

6. The method for collecting big data according to claim 3, characterized in that: The step of sorting all the blocks to obtain a block stack includes: Embedding redundant blocks into the block stack and integrating pre-built redundancy rules into the redundant blocks; The edge node is given the permission to read the data of the redundant blocks.

7. The method for collecting big data according to claim 6, characterized in that: The step of inserting a tail block into the bottom of the block stack, defining the height of the tail block as a fluctuation height, and constructing a synchronous update mechanism with the fluctuation height and the average value as the center comprises: Determining the influencing factors of the fluctuation height, wherein the influencing factors at least include: network parameters and idle resources in edge nodes; When several blocks are ejected, trigger signals are generated in parallel and sent to edge nodes.

8. A big data collection system, characterized in that: The system comprises: The upload module is used to define the production area, locate the deployment position of the sensor equipment, collect sensor data, create a processing platform for the sensor data, and upload the sensor data to the processing platform via a pre-built end-to-end transmission link; A superposition module is used to configure the attributes of the sensor device, wherein the attributes include at least: data volume, sampling frequency and network bandwidth, construct a weight query table, wherein the weight query table consists of attribute items and weight value items, collect real-time data of the attributes, traverse the weight query table, and superimpose and calculate the transmission score of each sensor device; A module is provided for creating a number block corresponding to the sensor device one by one, synchronizing the sensor data to the corresponding number block, sorting all the number blocks in the order of transmission scores from small to large, generating a number block stack, configuring a limited height of the number block stack, calculating an average value of the transmission score, setting the height of each number block, and obtaining a standard height; The push module is used to insert the tail block into the bottom of the block stack, and define the height of the tail block as the fluctuation height. With the fluctuation height and the average value as the center, a synchronous update mechanism is constructed to determine whether there are blocks in the block stack that have been pushed out. If so, in the production area, the edge nodes are traversed, and the sensor equipment corresponding to the blocks is mounted on the edge nodes, an end-edge-end transmission link is established, and based on the synchronous update mechanism, the pushed-out blocks are pushed back into the block stack.

9. The big data collection system according to claim 8, characterized in that: The upload module includes: A segmentation unit, used for segmenting the sensor data into a plurality of levels, wherein each level corresponds to a monitoring frequency; The integration unit is used to integrate the level and the monitoring frequency, generate a frequency comparison table, and send the frequency comparison table to the sensor device.

10. The big data collection system according to claim 8, characterized in that: The superposition module comprises: A comparison unit, used to divide the attribute into a plurality of individual items, compare the weight query table, determine the weight value corresponding to each individual item, and add them together to obtain the transmission score of the sensor device; An inserting unit is used to insert a label generated by the transmission score into the block in the processing platform.

Images