A compact data equalization management system, method, device and storage medium

Abstract

The application discloses a compact data equalization management system, method, equipment and storage medium, wherein the system selects an access strategy according to an access demand message through a data application equalization management module, configures access parameters according to the access strategy, and sends the obtained demand data to a user terminal through the access parameters; a data acquisition equalization management module obtains the data specifications of each type of acquisition terminal, configures the equalization attributes corresponding to the data specifications for each type of terminal, and performs data equalization management on the multiple types of acquisition terminals according to the equalization attributes of each type of acquisition terminal; a data storage equalization management module stores the data of the multiple types of acquisition terminals by using a storage equalization management method, and obtains the storage data; and a data retrieval equalization management module retrieves the demand data corresponding to the access demand message from the storage data by using a retrieval equalization management method, and sends the demand data to the data application equalization management module. The application has good equalization and self-adaptive characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data balancing management, and in particular to a compact data balancing management system, method, device and storage medium. BACKGROUND

[0002] The agricultural Internet of Things collects data of various agricultural elements, processes, analyzes and calculates the data, and uses the data as parameters or variables in automatic control of agricultural objects. From the whole process of agricultural element collection and agricultural object management, the Internet of Things technology and the corresponding data management are the core content of the agricultural Internet of Things system. Efficient management and full use of data can provide scientific basis for precision regulation and control, and achieve the purpose of increasing yield, improving quality, adjusting growth cycle and improving economic benefit.

[0003] The existing data management has the following problems:

[0004] 1) With the expansion of Internet of Things collection terminals and user terminals, more basic equipment needs to be invested to achieve better results, and more complex networks and hardware systems need to be established, resulting in high cost, high maintenance and high energy consumption.

[0005] 2) In the scenario of agricultural Internet of Things, due to the stage of industry development, the network and hardware environment are often compact, but the processing method has the problems of insufficient flexibility in input data characteristic management, poor adaptability in short-term and long-term capacity management, unbalanced retrieval efficiency, and poor access friendliness.

[0006] 3) Special data balancing management limits the use of balancing algorithms. In general, only the minimum storage space algorithm, the shortest integrated access path algorithm, and the fastest integrated corresponding time algorithm based on priority for pure data flow can be used. However, these algorithms often restrict or even contradict each other, resulting in a final performance that cannot match user demand. SUMMARY

[0007] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a compact data balancing management system, method, device and storage medium, which is simple to build, low in cost, easy to operate and simple to maintain, and has good balancing and adaptive characteristics.

[0008] In a first aspect, the present application provides a compact data balancing management system, which comprises:

[0009] The data application balanced management module is in communication connection with the user terminal, and is configured to receive an access demand message sent by the user terminal, select an access strategy according to the access demand message, configure an access parameter according to the access strategy, and send the demand data obtained through the access parameter to the user terminal;

[0010] The data collection balanced management module is in communication connection with the data application balanced management module and the collection terminal, and is configured to receive the access demand message and the access strategy sent by the data application balanced management module, obtain the data collection type of the multiple types of collection terminals corresponding to the access demand message according to the access strategy, perform demand calculation and distribution on the data collection type of each type of collection terminal, obtain the data specification of each type of collection terminal, configure the balanced attribute corresponding to the data specification to each type of terminal, perform data balanced management on the multiple types of collection terminals according to the balanced attribute of each type of collection terminal, and obtain the data of the multiple types of collection terminals.

[0011] The data storage balanced management module is in communication connection with the data application balanced management module and the data collection balanced management module, and is configured to receive the access strategy, adjust a storage balanced management method according to the access strategy, and store the data of the multiple types of collection terminals by using the storage balanced management method, to obtain the storage data.

[0012] The data retrieval balanced management module is in communication connection with the data application balanced management module and the data storage balanced management module, and is configured to receive the access strategy, adjust a retrieval balanced management method according to the access strategy, retrieve the demand data corresponding to the access demand message from the storage data by using the retrieval balanced management method, and send the demand data to the data application balanced management module.

[0013] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0014] The system receives the access demand message sent by the user terminal through the data application balanced management module, selects an access strategy according to the access demand message, configures access parameters according to the access strategy, sends the obtained demand data to the user terminal through the access parameters, and transmits the required data quickly and efficiently through the data sent according to the configured access parameters. The data application balanced management module receives the access demand message and the access strategy sent by the data collection balanced management module, obtains the data collection type of the multiple types of collection terminals corresponding to the access demand message according to the access strategy, calculates and allocates the data collection type of each type of collection terminal, obtains the data specification of each type of collection terminal, configures the balanced attribute corresponding to the data specification for each type of terminal, and performs data balanced management on the multiple types of collection terminals according to the balanced attribute of each type of collection terminal to obtain the data of the multiple types of collection terminals. By configuring the balanced attribute corresponding to the data specification for each type of terminal, the data balanced management has good balance and adaptive characteristics. The data storage balanced management module receives the access strategy, adjusts the storage balanced management method according to the access strategy, and stores the data of the multiple types of collection terminals by using the storage balanced management method to obtain the storage data. By adjusting the storage balanced management method according to the access strategy, the storage balanced management has good balance and adaptive characteristics. The data retrieval balanced management module receives the access strategy, adjusts the retrieval balanced management method according to the access strategy, and retrieves the demand data corresponding to the access demand message from the storage data by using the retrieval balanced management method, and sends the demand data to the data application balanced management module. By adjusting the retrieval balanced management method according to the access strategy, the storage balanced management has good balance and adaptive characteristics. Therefore, the system is composed of the collection balanced management module, the storage balanced management module, the retrieval balanced management module and the application balanced management module, so that the system is simple to build, low in cost, easy to operate and maintain, has good balance and adaptive characteristics, and can match the user demand and overcome the problems of insufficient flexibility in data characteristic management, poor adaptability in short-term and long-term capacity management, unbalanced retrieval efficiency, and poor access friendliness.

[0015] According to some embodiments of the application, the access strategy includes a manually input access strategy and an access strategy that automatically identifies access characteristics and automatically adjusts according to the access characteristics.

[0016] According to some embodiments of the application, the access parameters include optimal parameters for simultaneous access of various single data, multi-user discrete access queue delay parameters, and customized access parameters.

[0017] According to some embodiments of the application, the balanced attribute includes sampling rate, compression rate, data format, transmission period, and whether it is a same type of merged item.

[0018] According to some embodiments of the present application, the storage balancing management method comprises HTTP load balancing, MQTT load balancing, API load balancing, and Web load balancing.

[0019] According to some embodiments of the present application, the data retrieval balancing management module further comprises a structured data retrieval submodule and an unstructured data retrieval submodule; the structured data retrieval submodule is configured to insert structured data of different types of devices into different library tables, establish indexes for the library tables, and retrieve demand data corresponding to the access demand message from the storage data by using a B+ tree search method; and the unstructured data retrieval submodule is configured to retrieve demand data corresponding to the access demand message from the storage data by using a method of deploying a distributed object storage system for unstructured data.

[0020] In a second aspect, the embodiments of the present application further provide a compact data balancing management method, which comprises:

[0021] An access demand message sent by a user terminal is acquired, and an access strategy is selected according to the access demand message;

[0022] Data collection types of a plurality of collection terminals corresponding to the access demand message are acquired according to the access strategy, demand calculation and distribution are performed on the data collection types of each type of collection terminal, and data specifications of each type of collection terminal are obtained;

[0023] Balancing properties corresponding to the data specifications are configured for each type of terminal, data balancing management is performed on the plurality of collection terminals according to the balancing properties of each type of collection terminal, and data of the plurality of collection terminals is obtained;

[0024] A storage balancing management method is adjusted according to the access strategy, and the storage balancing management method is used to store the data of the plurality of collection terminals, and storage data is obtained;

[0025] A retrieval balancing management method is adjusted according to the access strategy, and the retrieval balancing management method is used to retrieve demand data corresponding to the access demand message from the storage data;

[0026] Access parameters are configured according to the access strategy, and the demand data is sent to the user terminal through the configured access parameters.

[0027] Compared with the prior art, the second aspect of the present application has the following beneficial effects:

[0028] The method comprises the following steps: acquiring an access demand message sent by a user terminal, selecting an access strategy according to the access demand message, acquiring a data acquisition type of a plurality of acquisition terminals corresponding to the access demand message according to the access strategy, performing demand calculation and distribution on the data acquisition type of each acquisition terminal, obtaining a data specification of each acquisition terminal, configuring a balancing attribute corresponding to the data specification for each terminal, performing data balancing management on the plurality of acquisition terminals according to the balancing attribute of each acquisition terminal, and obtaining data of the plurality of acquisition terminals. By configuring the balancing attribute corresponding to the data specification for each terminal, the data balancing management has good balancing and adaptive characteristics. The storage balancing management method is adjusted according to the access strategy, and the data of the plurality of acquisition terminals is stored by using the storage balancing management method, so as to obtain storage data. By adjusting the storage balancing management method according to the access strategy, the storage balancing management has good balancing and adaptive characteristics. The retrieval balancing management method is adjusted according to the access strategy, and the demand data corresponding to the access demand message is retrieved from the storage data by using the retrieval balancing management method. By adjusting the retrieval balancing management method according to the access strategy, the storage balancing management has good balancing and adaptive characteristics. The access parameters are configured according to the access strategy, and the demand data is sent to the user terminal by using the configured access parameters. By sending the data according to the configured access parameters, the required data can be quickly and efficiently transmitted. Therefore, the distributed management method is composed of acquisition balancing, storage balancing, retrieval balancing and application balancing, and has good balancing and adaptive characteristics, so as to match the user demand and overcome the problems of insufficient flexibility in data characteristic management, poor adaptability in short-term and long-term capacity management, unbalanced retrieval efficiency, and weak access friendliness.

[0029] According to some embodiments of the present application, the method of configuring the balancing attribute corresponding to the data specification for each terminal and performing data balancing management on the plurality of acquisition terminals according to the balancing attribute of each acquisition terminal to obtain data of the plurality of acquisition terminals comprises:

[0030] Acquiring the balancing attribute corresponding to the data specification configured for each terminal at present;

[0031] Comparing the acquired balancing attribute configured for each terminal at present with a historical balancing attribute, if the acquired balancing attribute configured for each terminal at present is updated compared with the historical balancing attribute, updating the sampling configuration, the protocol conversion configuration and the communication transmission configuration in the data balancing management, performing data balancing management on the plurality of acquisition terminals according to the sampling configuration, the protocol conversion configuration and the communication transmission configuration, and obtaining data of the plurality of acquisition terminals.

[0032] According to some embodiments of the present application, the storage data is obtained by storing the data of the plurality of collection terminals using the storage balance management method, and the method comprises the following steps:

[0033] The data of the plurality of collection terminals is compared with the last stored data, if there is a difference between the data of the plurality of collection terminals and the last stored data, the same type of data in the data of the plurality of collection terminals is merged into a unified data block for storage, and the storage data is obtained.

[0034] According to some embodiments of the present application, the demand data corresponding to the access demand message is retrieved from the storage data using the retrieval balance management method, and the method comprises the following steps:

[0035] If the retrieval data is structured data, the structured data of different types of devices is inserted into different library tables, and indexes are established for the library tables, and the demand data corresponding to the access demand message is retrieved from the storage data by a B+ tree search method;

[0036] If the retrieval data is unstructured data, the demand data corresponding to the access demand message is retrieved from the storage data by deploying a distributed object storage system for the unstructured data. BRIEF DESCRIPTION OF DRAWINGS

[0037] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0038] Figure 1 is a structural diagram of a compact data balance management system according to an embodiment of the present application;

[0039] Figure 2 is a basic architecture schematic diagram of the balance management according to an embodiment of the present application;

[0040] Figure 3 is a structural diagram of a compact data balance management system according to another embodiment of the present application;

[0041] Figure 4 is a flowchart of data collection balance management according to an embodiment of the present application;

[0042] Figure 5 is a flowchart of a compact data balance management method according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] Embodiments of the present application are described below in detail with reference to the accompanying drawings, wherein the same or similar components or components having the same or similar functions are denoted by the same or similar reference numerals throughout the drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application.

[0044] In the description of the present application, if the first, second, etc. are described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0045] In the description of the present application, it should be understood that the orientation description, such as up, down, etc. indicates the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0046] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, mounting, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0047] The existing data management has the following problems:

[0048] 1) With the expansion of Internet of Things collection terminals and user terminals, in order to achieve better results, more basic equipment needs to be invested, more complex networks and hardware systems need to be established, resulting in high cost, high maintenance, and high energy consumption.

[0049] 2) In the agricultural Internet of Things scenario, due to the stage of industry development, it is often a compact network and hardware environment, but the processing method has the problems of insufficient flexibility in input data characteristic management, poor adaptability in short-term and long-term capacity management, unbalanced retrieval efficiency, and not strong access friendliness.

[0050] 3) Special data balancing management limits the use of balancing algorithms, and in general cases, only the minimum storage space algorithm for pure data flow, the shortest integrated access path algorithm, the fastest integrated corresponding time algorithm based on priority, etc. can be used. However, these algorithms often restrict each other or even contradict each other, resulting in the final performance not matching user demand.

[0051] To solve the above problems, the application receives the access demand message sent by the user terminal through the data application balanced management module, selects the access strategy according to the access demand message, configures the access parameter according to the access strategy, sends the obtained demand data to the user terminal through the access parameter, and transmits the required data quickly and efficiently through the data sent according to the configured access parameter. The data collection balanced management module receives the access demand message and the access strategy sent by the data application balanced management module, obtains the data collection type of the multiple types of collection terminals corresponding to the access demand message according to the access strategy, calculates and allocates the data collection type of each type of collection terminal, obtains the data specification of each type of collection terminal, configures the balanced attribute corresponding to the data specification for each type of terminal, and performs data balanced management on the multiple types of collection terminals according to the balanced attribute of each type of collection terminal, and obtains the data of the multiple types of collection terminals. By configuring the balanced attribute corresponding to the data specification for each type of terminal, the data balanced management has good balance and adaptive characteristics. The data storage balanced management module receives the access strategy, adjusts the storage balanced management method according to the access strategy, and stores the data of the multiple types of collection terminals by using the storage balanced management method, obtains the storage data, and adjusts the storage balanced management method according to the access strategy. The storage balanced management has good balance and adaptive characteristics. The data retrieval balanced management module receives the access strategy, adjusts the retrieval balanced management method according to the access strategy, retrieves the demand data corresponding to the access demand message from the storage data by using the retrieval balanced management method, and sends the demand data to the data application balanced management module. By adjusting the retrieval balanced management method according to the access strategy, the storage balanced management has good balance and adaptive characteristics. Therefore, the application comprehensively comprises a collection balanced management module, a storage balanced management module, a retrieval balanced management module and an application balanced management module to form a distributed system, so that the system is simple to build, low in cost, strong in operability, easy to maintain, has good balance and adaptive characteristics, and can match the user demand and overcome the problems of insufficient flexibility in data characteristic management, poor adaptability in short-term and long-term capacity management, unbalanced retrieval efficiency, and poor access friendliness.

[0052] Reference Figure 1 The embodiment of the application provides a compact data balanced management system, and the compact data balanced management system comprises a data application balanced management module 100, a data collection balanced management module 200, a data storage balanced management module 300 and a data retrieval balanced management module 400, wherein:

[0053] The data application balanced management module 100 is in communication connection with the user terminal, and is configured to receive an access demand message sent by the user terminal, select an access strategy according to the access demand message, configure an access parameter according to the access strategy, and send the demand data obtained through the access parameter to the user terminal;

[0054] The data collection balanced management module 200 is in communication connection with the data application balanced management module and the collection terminal, and is configured to receive the access demand message and the access strategy sent by the data application balanced management module, obtain the data collection type of the multiple types of collection terminals corresponding to the access demand message according to the access strategy, perform demand calculation and distribution on the data collection type of each type of collection terminal, obtain the data specification of each type of collection terminal, configure the balanced attribute corresponding to the data specification for each type of terminal, and perform data balanced management on the multiple types of collection terminals according to the balanced attribute of each type of collection terminal, to obtain the data of the multiple types of collection terminals.

[0055] The data storage balanced management module 300 is in communication connection with the data application balanced management module and the data collection balanced management module, and is configured to receive the access strategy, adjust the storage balanced management method according to the access strategy, and store the data of the multiple types of collection terminals by using the storage balanced management method, to obtain the storage data.

[0056] The data retrieval balanced management module 400 is in communication connection with the data application balanced management module and the data storage balanced management module, and is configured to receive the access strategy, adjust the retrieval balanced management method according to the access strategy, retrieve the demand data corresponding to the access demand message from the storage data by using the retrieval balanced management method, and send the demand data to the data application balanced management module.

[0057] In the embodiment, the distributed system is composed of the collection balanced management module, the storage balanced management module, the retrieval balanced management module and the application balanced management module, so that the system is simple to build, low in cost, strong in operability, easy to maintain, has good balanced and adaptive characteristics, and can match the user demand, and can overcome the problems of insufficient flexibility in data characteristic management, poor adaptability in short-term and long-term capacity management, unbalanced retrieval efficiency, and weak access friendliness.

[0058] In some embodiments, the access strategy includes an artificially input access strategy and an access strategy automatically adjusted according to an automatically recognized access characteristic.

[0059] In the embodiment, the system has the adaptive characteristics by artificially inputting the access strategy and automatically adjusting the access strategy according to the access characteristic.

[0060] In some embodiments, the access parameters include single-item data access optimization parameters, multi-user discrete access queue delay parameters, and customized access parameters.

[0061] In the present embodiment, the system can quickly and efficiently deliver the required data by configuring the access parameters.

[0062] In some embodiments, the balancing attributes include sampling rate, compression rate, data format, transmission period, and same-type merging item.

[0063] In the present embodiment, by reasonably planning these balancing attributes, data sampling, protocol organization, and transmission information can be avoided.

[0064] In some embodiments, the storage balancing management method includes HTTP load balancing, MQTT load balancing, API load balancing, and Web load balancing.

[0065] In some embodiments, the data retrieval balancing management module further includes a structured data retrieval submodule and an unstructured data retrieval submodule. The structured data retrieval submodule is configured to insert structured data of different types of devices into different library tables, establish indexes for the library tables, and retrieve demand data corresponding to the access demand message from the stored data through a B+ tree search method. The unstructured data retrieval submodule is configured to retrieve demand data corresponding to the access demand message from the stored data by deploying a distributed object storage system.

[0066] In the present embodiment, different retrieval balancing management methods are used for different data types, making the access retrieval management efficient and ensuring good consistency of retrieval balancing under different user quantities.

[0067] To facilitate understanding by those skilled in the art, a set of best embodiments is provided as follows:

[0068] In the existing technology, the basic architecture of agricultural Internet of Things data balancing management is as shown in Figure 2 As can be seen from the figure, data balancing management is a management activity as an independent process, which is a secondary adjustment storage after the collection and storage are completed, and is directly connected with data retrieval. The construction of the data balancing system and hardware based on this architecture is relatively simple, and a dedicated server is usually used to carry out balancing calculation for pure data flow. With the expansion of business and the increase of calculation amount, multiple servers are often expanded or middleware clusters are changed to improve the overall processing performance.

[0069] Therefore, in actual agricultural Internet of Things applications, the above-mentioned general-purpose data balancing system has many limitations in use, and the main reasons are as follows:

[0070] 1) The data types of agricultural Internet of Things devices are diversified. These collected data include environmental sensor data, picture data, spectral data, video data, agricultural machinery data, and electric control data. The size of single data, the frequency of data, and the format of data are different. Therefore, independent data balancing often needs to be customized according to the project, and the universality is not reflected.

[0071] 2) The data storage of agricultural Internet of Things devices is different. Video stream data has large capacity and is real-time data, and other state data and control data are periodic single-point data. The required buffer space and absolute space are different when storing. If a universal data balancing is performed on the basis of the completed data storage, the storage space needs to be simply copied, and the efficiency of the storage space in the balancing stage cannot be reflected.

[0072] 3) The retrieval demand of agricultural Internet of Things data changes greatly. In the application end, the user needs to perform data retrieval before presenting and using after triggering data access. The access demand of different time periods and different user subjects changes greatly, so the demand for data retrieval also changes greatly in a large range. Multi-user simultaneous retrieval, multi-element simultaneous retrieval, and long-time retrieval of large data flow all need good data management foundation to meet the real-time requirement and smoothness requirement of users. Therefore, a universal data balancing is difficult to meet the retrieval performance of different users in the company, and often has access bottlenecks or defects in specific scenarios, which directly affects the user experience.

[0073] In order to solve the above problems, the embodiment makes the following improvements:

[0074] The technical scheme of the embodiment is shown in Figure 3 From the figure, it can be seen that the technical scheme of the embodiment is composed of a data collection balancing management module, a data storage balancing management module, a data retrieval balancing management module, and a data application balancing management module arranged on an Internet of Things server. In the technical scheme of the embodiment, the effective management of information flow is mainly realized through configuration and algorithm.

[0075] (1) Data collection balancing management module.

[0076] First, the access demand message and the access strategy sent by the data application balancing management module are acquired, the demand calculation and distribution of each collection terminal are performed according to the data access demand in the Internet of Things system, and the data specification of each collection terminal is determined. For the specification difference of different quantities and different stages, the data collection balancing management module for the collection terminal is designed in the embodiment, and the sampling strategy configuration is adjusted according to the access strategy. Specifically:

[0077] In the data collection balanced management module, the data collection type of the plurality of collection terminals corresponding to the access demand message is obtained according to the access strategy, and a dedicated balanced attribute is added for each type of collection terminal, the balanced attribute including a sampling rate, a compression rate, a data format, a transmission period, whether it is a same type merging item, etc. Reasonable configuration of these balanced attributes can avoid excessive sampling, excessive protocol organization, and excessive transmission of information, and therefore, the data collection balanced management module is a basic preparation of the data balanced management system. Each collection terminal manages the sampling, conversion, and communication process according to the balanced attribute configuration, so that the optimal data organization principle allocated by the system can be realized, and the basic flow is as shown in Figure 4 The specific process is as follows:

[0078] During initialization, the historical balanced attribute is input by the maintenance personnel; the balanced attribute corresponding to the data specification configured by the system for each type of terminal is obtained; the obtained balanced attribute configured by the system for each type of terminal is compared with the historical balanced attribute, if the obtained balanced attribute configured by the system for each type of terminal is updated compared with the historical balanced attribute, the sampling configuration, the protocol conversion configuration, and the communication transmission configuration in the data balanced management are updated, the data balanced management of the plurality of collection terminals is performed according to the sampling configuration, the protocol conversion configuration, and the communication transmission configuration, and the data of the plurality of collection terminals is obtained.

[0079] It should be noted that the specific calculation and allocation in the embodiment need to determine the data specification according to the type characteristics of each collection terminal, such as the collection of temperature and humidity, which can calculate the collection period, the data length corresponding to the data precision, and the check code according to the demand of the Internet of Things system. The data specification can be calculated and allocated according to different demands of the Internet of Things system, and the embodiment is not limited specifically.

[0080] (2) Data storage balanced management module.

[0081] The data collection process in step (1) has performed a preliminary screening on the demand data, and the data storage strategy in the data storage balanced management module adopts a differentiated principle for storage, and only when the current data is different from the last data, the storage is triggered. Moreover, the same type of data (such as multiple temperature and humidity data) is merged into a unified data block, and the difference in capacity with video data is reduced, which is beneficial to the overall planning of storage space. In addition, the general storage balanced management methods include HTTP load balancing, MQTT load balancing, API load balancing, Web load balancing, etc., and the storage balanced management method is selectively applied according to the actual Internet of Things management demand, that is, the storage balanced management method can be adjusted according to the access strategy sent by the data application balanced management module.

[0082] (3) Data retrieval balanced management module.

[0083] On the basis of the initial management of the element attributes and storage characteristics of various types of data, the balanced management of the data retrieval layer in the data retrieval balanced management module is relatively simple. The embodiment designs a hierarchical and path-based retrieval algorithm, calculates based on the data storage depth and data capacity, and determines the specific retrieval strategy parameters based on the shortest time. Moreover, these parameters can be adaptively adjusted according to the sampling scale of the Internet of Things and the user access scale, and the retrieval balanced management method can be adjusted according to the access strategy sent by the data application balanced management module, thereby ensuring the long-term effectiveness of retrieval optimization. For the reduction of structured data, by inserting device data of different device types into different library tables, and by indexing the library tables, the system can use the B+ tree search algorithm to obtain the physical storage address of the required data in the memory at each query, and directly access the result data in the memory according to the address, thereby optimizing the query rate. For unstructured data, a distributed object storage system is deployed, each node is a process running in the user space, and a lightweight cooperative routine is used to achieve high concurrency, ensuring that each file retrieval is efficient and stable. In the hierarchical design of the retrieval data, a storage layer, a cache layer, and an application presentation layer are adopted, and each layer can be horizontally expanded to adaptively expand the system processing capacity when the sampling scale and user access are expanded.

[0084] (4) Data application balanced management module.

[0085] The Internet of Things data is applied through various terminal users, including workstations, maintenance stations, mobile phone apps, web applications, etc. The application layer is the effectiveness of access for data balancing. Some specific data receives user attention, and the access volume is high for a long time; some data (such as video) are large flow data and require more resources; some data (such as statistical data) have increasing access capacity with the continuous expansion of the system. The embodiment designs an intelligent access management submodule in the data application balanced management module to handle application balancing. The data application balanced management module receives access demand messages sent by user terminals, and selects an access strategy according to the access demand messages. The intelligent access management submodule supports manual input of access strategy configuration, and also supports automatic identification of access characteristics and automatic adjustment of access strategy configuration according to the access characteristics (obtained from the access demand messages). These configurations not only affect the access strategy of the data application balanced management module itself, but also adjust the retrieval strategy configuration in the data retrieval balanced management module, the storage strategy configuration in the data storage balanced management module, and the sampling strategy configuration in the data collection balanced management module, and pass and update the configuration data layer by layer. The access parameters are configured according to the access strategy, which specifically include various single data simultaneous access optimal parameters, multi-user discrete access queue delay parameters, and customized access parameters.

[0086] In this embodiment, the application of these access parameters effectively balances the data flow, avoids lag, and provides a user-friendly interface for the user terminal.

[0087] Reference Figure 3 The maintenance terminal in the figure can be a PC. The PC has a serial port or network port connected to the system of this embodiment. Manually, parameters or strategies can be set through the PC interface, and these configuration information can be written into the system of this embodiment through the communication port.

[0088] In this embodiment, the main advantages include:

[0089] 1. The system platform in this embodiment requires a compact network and hardware system. A single server with a certain capacity can meet the storage and computing requirements. The system platform is easy to build, low in cost, highly operable, and simple to maintain.

[0090] 2. The system has an adaptive feature to expand input data. As the number of front-end interfaces of the IoT system accumulates, no manual access management is required, and the system performance will not be affected.

[0091] 3. The data storage balancing management module is designed for automatic iteration of storage management, has flexible configuration, and has efficient storage space self-check management function, which can ensure the long-term stability of the compact system platform.

[0092] 4. The data retrieval balance management module enables efficient user access retrieval management and ensures good consistency in retrieval balance across different user numbers.

[0093] Reference Figure 5 This invention also provides a compact data balancing management method, which includes, but is not limited to, steps S100 to S600, wherein:

[0094] Step S100: Obtain the access request message sent by the user terminal, and select an access strategy based on the access request message;

[0095] Step S200: Obtain the data collection types of multiple collection terminals corresponding to the access request message according to the access policy, and calculate and allocate the data collection types of each collection terminal to obtain the data specifications of each collection terminal.

[0096] Step S300: Configure a balancing attribute corresponding to the data specifications for each type of terminal, and perform data balancing management on multiple types of acquisition terminals according to the balancing attribute of each type of acquisition terminal to obtain data from multiple types of acquisition terminals;

[0097] Step S400, adjusting the storage balance management method according to the access strategy, and using the storage balance management method to store the data of the multiple types of collection terminals to obtain storage data;

[0098] Step S500, adjusting the retrieval balance management method according to the access strategy, and using the retrieval balance management method to retrieve demand data corresponding to the access demand message from the storage data;

[0099] Step S600, configuring access parameters according to the access strategy, and sending the demand data to the user terminal through the configured access parameters.

[0100] In the embodiment, the distributed management method is composed of collection balance, storage balance, retrieval balance and application balance, which has good balance and adaptive characteristics, so as to match the user demand, and overcome the problems of insufficient flexibility in data characteristic management, poor adaptability in short-term and long-term capacity management, unbalanced retrieval efficiency, and weak access friendliness.

[0101] In some embodiments, the balance attribute corresponding to the data specification is configured for each type of terminal, and the data balance management of the multiple types of collection terminals is performed according to the balance attribute of each type of collection terminal to obtain the data of the multiple types of collection terminals, including:

[0102] The balance attribute corresponding to the data specification configured for each type of terminal is obtained;

[0103] The obtained balance attribute of each type of terminal configured at present is compared with the historical balance attribute, if the obtained balance attribute of each type of terminal configured at present is updated compared with the historical balance attribute, the sampling configuration, the protocol conversion configuration and the communication transmission configuration in the data balance management are updated, the data balance management of the multiple types of collection terminals is performed according to the sampling configuration, the protocol conversion configuration and the communication transmission configuration to obtain the data of the multiple types of collection terminals.

[0104] In some embodiments, the storage balance management method is used to store the data of the multiple types of collection terminals to obtain storage data, including:

[0105] The data of the multiple types of collection terminals is compared with the data stored last time, if there is difference between the data of the multiple types of collection terminals and the data stored last time, the same type of data in the data of the multiple types of collection terminals is merged into a unified data block for storage to obtain the storage data.

[0106] In some embodiments, the retrieval balance management method is used to retrieve demand data corresponding to the access demand message from the storage data, including:

[0107] If the search data is structured data, the structured data of different types of devices is inserted into different library tables, indexes are established for the library tables, and the demand data corresponding to the access demand message is searched from the stored data by using a B+ tree search method.

[0108] If the search data is unstructured data, the unstructured data is searched from the stored data by using a method of deploying a distributed object storage system to obtain demand data corresponding to the access demand message.

[0109] It should be noted that, since the compact data balancing management method in the embodiment and the compact data balancing management system described above are based on the same inventive concept, the corresponding content in the system embodiment is also applicable to the method embodiment, which will not be described in detail here.

[0110] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above implementation, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the embodiments of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A compact data equalization management system, characterized by, The compact data balance management system comprises: A data application balance management module in communication connection with a user terminal, configured to receive an access demand message sent by the user terminal, select an access strategy according to the access demand message, configure an access parameter according to the access strategy, and send the demand data obtained through the access parameter to the user terminal; A data collection balance management module in communication connection with the data application balance management module and a collection terminal, configured to receive the access demand message and the access strategy sent by the data application balance management module, obtain the data collection type of a plurality of collection terminals corresponding to the access demand message according to the access strategy, perform demand calculation and distribution on the data collection type of each collection terminal, obtain the data specification of each collection terminal, configure the balance attribute corresponding to the data specification to each terminal, and perform data balance management on the plurality of collection terminals according to the balance attribute of each collection terminal to obtain the data of the plurality of collection terminals; A data storage balance management module in communication connection with the data application balance management module and the data collection balance management module, configured to receive the access strategy, adjust a storage balance management method according to the access strategy, and store the data of the plurality of collection terminals by using the storage balance management method to obtain storage data; A data retrieval balance management module in communication connection with the data application balance management module and the data storage balance management module, configured to receive the access strategy, adjust a retrieval balance management method according to the access strategy, retrieve the demand data corresponding to the access demand message from the storage data by using the retrieval balance management method, and send the demand data to the data application balance management module.

2. The compact data equalization management system of claim 1, wherein, The access strategy comprises an access strategy input manually and an access strategy automatically adjusted according to automatically recognized access characteristics.

3. The compact data equalization management system of claim 1, wherein, The access parameter comprises optimal parameters for simultaneous access of various single data, multi-user discrete access queue delay parameters, and customized access parameters.

4. The compact data equalization management system according to any one of claims 1 to 3, characterized in that, The balance attribute comprises a sampling rate, a compression rate, a data format, a transmission cycle, and whether it is a same type of merged item.

5. The compact data equalization management system of claim 1, wherein, The storage balance management method comprises HTTP load balancing, MQTT load balancing, API load balancing, and Web load balancing.

6. The compact data equalization management system of claim 1, wherein, The data retrieval balance management module further comprises a structured data retrieval submodule and an unstructured data retrieval submodule; the structured data retrieval submodule is configured to insert structured data of different types of devices into different library tables, establish an index for the library tables, and retrieve the demand data corresponding to the access demand message from the storage data by using a B+ tree search method; and the unstructured data retrieval submodule is configured to retrieve the demand data corresponding to the access demand message from the storage data by using a method of deploying a distributed object storage system for unstructured data.

7. A compact data equalization management method, characterized by, The compact data balance management method comprises: Acquire the access demand message sent by the user terminal, and select an access strategy according to the access demand message; According to the access strategy, the data acquisition type of the plurality of acquisition terminals corresponding to the access demand message is acquired, and the data acquisition type of each acquisition terminal is calculated and allocated to obtain the data specification of each acquisition terminal; The data specification of each terminal is configured with the corresponding balanced attribute, and the data of the plurality of acquisition terminals is balanced according to the balanced attribute of each acquisition terminal to obtain the data of the plurality of acquisition terminals; According to the access strategy, the storage balanced management method is adjusted, and the data of the plurality of acquisition terminals is stored by using the storage balanced management method to obtain the storage data; According to the access strategy, the retrieval balanced management method is adjusted, and the demand data corresponding to the access demand message is retrieved from the storage data by using the retrieval balanced management method; According to the access strategy, the access parameters are configured, and the demand data is sent to the user terminal through the configured access parameters.

8. The compact data equalization management method of claim 7, wherein, The data specification of each terminal is configured with the corresponding balanced attribute, and the data of the plurality of acquisition terminals is balanced according to the balanced attribute of each acquisition terminal to obtain the data of the plurality of acquisition terminals, comprising: Acquire the balanced attribute corresponding to the data specification configured by the current each terminal; Compare the acquired balanced attribute of the current each terminal with the historical balanced attribute, if the acquired balanced attribute of the current each terminal is updated compared with the historical balanced attribute, update the sampling configuration, protocol conversion configuration and communication transmission configuration in the data balanced management, and balance the data of the plurality of acquisition terminals according to the sampling configuration, the protocol conversion configuration and the communication transmission configuration to obtain the data of the plurality of acquisition terminals.

9. The compact data equalization management method of claim 7, wherein, The data of the plurality of acquisition terminals is stored by using the storage balanced management method to obtain the storage data, comprising: Compare the data of the plurality of acquisition terminals with the data stored last time, if there is difference between the data of the plurality of acquisition terminals and the data stored last time, merge the same data in the data of the plurality of acquisition terminals into a unified data block for storage to obtain the storage data.

10. The compact data equalization management method according to any one of claims 7 to 9, characterized in that, The retrieval balanced management method is used to retrieve the demand data corresponding to the access demand message from the storage data, comprising: If the retrieval data is structured data, insert the structured data of different types of devices into different library tables, and index the library tables, and retrieve the demand data corresponding to the access demand message from the storage data by using B+ tree search method; If the retrieval data is unstructured data, the distributed object storage system is deployed to retrieve the demand data corresponding to the access demand message from the storage data.

Images