Method and computing device for accessing large amounts of data

Abstract

The present invention provides a method and computing device for accessing large amounts of data, the method being used in a data acquisition and monitoring control system, the method comprising: configuring instance information of each stand-alone node, configuring the relationship between each acquisition unit and each of the stand-alone node instances, and writing all configuration information into a configuration database; separating the point value information collected by each acquisition unit by a group identifier, and writing it into each message queue corresponding to each of the acquisition units according to the configuration information; a real-time library writing module reading the point value information in the message queue, and writing it into each of the corresponding stand-alone nodes; wherein the acquisition unit and the stand-alone node are correspondingly bound according to a unique group identifier, and the stand-alone node is written according to the classification of the point value information. According to the technical solution of the present invention, a large amount of data can be easily accessed, the system can be easily expanded, the system hardware requirements are low, the memory overhead is reduced, and resources are saved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method and computing device for accessing large amounts of data. Background Art

[0002] In data acquisition and monitoring control systems, the collection of telemetry-related data is very important. Different sites have different scales and require a high timeliness of point value changes. In order to improve performance, most manufacturers store the collected points in a single machine and use the single machine as a real-time library. Generally, deployment is based on a single machine for processing. When the scale is expanded, it is often necessary to use a single machine cluster mode to improve performance through sharding.

[0003] Some companies on the market provide another way for single-machine clusters, that is, to create a proxy layer to proxy all commands of the single machine, and to let the proxy layer handle reading and writing, calculate unique keys, maintain multiple single-machine instances internally, and calculate based on unique keys, so as to make it easier to find nodes. However, because the keys in the cluster mode are too scattered, there is no way to use transactions, monitoring mechanisms, etc. to classify these keys. Only those on the same node can operate, which requires business processing and is costly.

[0004] To this end, a technical solution is needed that can easily access large amounts of data, facilitate system expansion, have low requirements for system hardware, reduce memory overhead, and save resources. Summary of the invention

[0005] The present invention aims to provide a method and computing device for accessing large amounts of data, which can easily access large amounts of data, facilitate system expansion, have low requirements on system hardware, reduce memory overhead, and save resources.

[0006] According to one aspect of the present invention, a method for accessing large amounts of data is provided, the method being used in a data acquisition and monitoring control system, the method comprising:

[0007] Configure the instance information of each stand-alone node, configure the relationship between each acquisition unit and each stand-alone node instance, and write all configuration information into the configuration database;

[0008] Separating the point value information collected by each collection group through the group identifier, and writing it into each message queue corresponding to each collection group according to the configuration information;

[0009] The real-time library writing module reads the point value information in the message queue and writes it into the corresponding stand-alone nodes;

[0010] The collection group and the stand-alone node are bound accordingly according to a unique group identifier, and the stand-alone node is written according to the classification of the point value information.

[0011] According to some embodiments, the stand-alone node is bound to the collection group, and the stand-alone node stores the point value information collected by the bound collection group.

[0012] According to some embodiments, the point value information is distinguished by a group identification of the collection group;

[0013] The group identifier is a unique identifier of the collection group.

[0014] According to some embodiments, the group identifier is used to split and classify data according to business;

[0015] The acquisition unit is bound to the data source, and the data volume is limited at the data acquisition source.

[0016] According to some embodiments, the stand-alone node is bound to the collection group, including: according to the configuration information, manually or automatically selecting the group identifier corresponding to the collection group for the stand-alone node, and the group identifier is used to associate the corresponding stand-alone node identifier.

[0017] According to some embodiments, the real-time library writing module reads the point value information in the message queue and writes it into the corresponding stand-alone nodes, including:

[0018] According to the configuration information, the real-time library writing module obtains the corresponding group identifier, and associates the single-machine node identifier through the group identifier;

[0019] The real-time library writing module writes the point value information in the message queue into the corresponding stand-alone node according to the stand-alone node identifier.

[0020] According to some embodiments, the point value query service queries the point value information from the corresponding single-machine node according to the mapping between the collection group and the single-machine node configured in the configuration database.

[0021] According to some embodiments, establishing a mapping relationship between the point value information includes: using a bitmap method to establish a mapping between the point value information and the single-machine node identifier.

[0022] According to another aspect of the present invention, there is provided a computing device, comprising:

[0023] Processor; and

[0024] A memory stores a computer program, and when the computer program is executed by the processor, the method described in any one of the above items is implemented.

[0025] According to an embodiment of the present invention, the instance information of the stand-alone node, the relationship between the acquisition group and the stand-alone node instance are configured in the configuration service, and all the configuration information is written into the database. The acquisition application separates the collected point value information according to the configuration information and writes it into the message queue. The real-time library writing application reads the information in the message queue and writes it into the corresponding stand-alone node. In the data acquisition and monitoring control system, this patent solves the problem that the data volume is very large and the cluster cannot meet the performance requirements when a stand-alone machine is used as a real-time library. From a business perspective, the data is classified to give full play to the performance of the stand-alone machine. At the same time, the architecture is clear at a glance, and the write data is handed over to the stand-alone machine for processing. The present invention has a small business transformation and is very suitable for the transformation of old systems, reducing data access costs, and can easily access large amounts of data, conveniently expand the system, and has low requirements for system hardware, reduces memory overhead, and saves resources.

[0026] According to some embodiments, the present invention separates data based on business. After the data volume increases, the real-time library still uses the stand-alone mode to write to the stand-alone machine, solving the problem that the data in the cluster mode still has excessive writing pressure on a certain node.

[0027] According to some embodiments, the real-time library is aggregated with multiple instances on a single machine. Based on business characteristics, data and instances are mapped using a bitmap method to avoid multiple removal operations during clustering. The caller does not need to know the internal details, and the service provided is used to implement the aggregation, so the access cost is low.

[0028] According to some embodiments, the method of the present invention can facilitate system expansion, and multiple sets of stand-alone nodes can be quickly added according to the data volume, and the business does not need to be adjusted.

[0029] According to some embodiments, in the scenario where the data acquisition and monitoring control system accesses a large amount of data, the data is split, the original stand-alone mode is retained, and at the same time, multiple stand-alone nodes are used to form a distributed architecture to infinitely expand the data volume for protection.

[0030] According to some embodiments, when performing a single-machine node data query, business data is differentiated, and based on the business and single-machine node mapping method, a unique bitmap algorithm is used to reduce memory overhead, so that single-machine node instance information can be found more quickly, replacing the single-machine cluster method for protection.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0033] Figure 1 A flow chart of a method for accessing large amounts of data according to an example embodiment is shown.

[0034] Figure 2 A schematic diagram of a system workflow according to an example embodiment is shown.

[0035] Figure 3 A schematic diagram of a system architecture according to an example embodiment is shown.

[0036] Figure 4 A block diagram of a computing device is shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0038] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0041] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the present inventive concept. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0042] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present invention, and therefore cannot be used to limit the protection scope of the present invention.

[0044] Data is sharded and stored on multiple nodes, with each node storing a portion of the data. Redis cluster (stand-alone cluster) uses a hash slot mechanism to determine which node the data should be stored on. Redis cluster has a total of 16384 hash slots, and each key is mapped to one of the hash slots through a hash function, and the hash slot is then mapped to a specific node. This is decentralized, and each client needs to select which node it should be mapped to based on the current key and the crc16 algorithm modulo 16386. If it is not on the current node, it will return MOVED (removed) and return the correct node address to the client.

[0045] If there are too many keys, there will be too many keys on a certain node, the amount of written data will decrease, and timeliness requirements cannot be met. In cluster mode, because the keys are too scattered, there is no way to use pipelines, transactions, monitoring mechanisms, etc. The pipeline needs to manually determine which node it is on and classify these keys. Only those on the same node can be operated, which requires business processing and is costly.

[0046] In cluster mode, due to the need to find nodes, the performance is lower than that of a single machine. The proxy layer can indeed improve efficiency to a certain extent, but it also needs to consider the load of the proxy layer. All performance bottlenecks are implemented by the proxy layer, which has poor stability. At the same time, the business needs to calculate the key in combination with the business scenario. This process requires business constraints and has a certain connection cost. The most important thing is that there is no way to solve the problem of too much data accessed by a node.

[0047] To this end, the present invention proposes a method for accessing large amounts of data, which can easily access large amounts of data, facilitate system expansion, have low requirements on system hardware, reduce memory overhead, and save resources.

[0048] Exemplary embodiments of the present invention are described below with reference to the accompanying drawings.

[0049] Figure 1 A flow chart of a method for accessing large amounts of data according to an example embodiment is shown.

[0050] See also Figure 1 In S101, the instance information of each stand-alone node is configured, the relationship between each acquisition unit and each stand-alone node instance is configured, and all configuration information is written into a configuration database.

[0051] The method of the present invention is used in the SCADA system. The SCADA (Supervisory Control And Data Acquisition) system is a data acquisition and monitoring control system. The SCADA system is a computer-based DCS (Distributed Control System) and power automation monitoring system; it has a wide range of applications and can be applied to data acquisition and monitoring control and process control in the fields of power, metallurgy, petroleum, chemical industry, gas, railways, etc.

[0052] According to some embodiments, the configuration service maintains the acquisition group information, binds the single-machine node to the acquisition group, and the single-machine node stores the point value information collected by the bound acquisition group; the point value information is distinguished by the group identifier of the acquisition group; the group identifier is the unique identifier of the acquisition group, which is divided by business. The point value information is data split and classified based on the business; the acquisition group and the site are bound, and the data level is limited at the source of data collection.

[0053] According to some embodiments, the configuration service is mainly used to maintain the relationship between the external devices to be collected and the software, so that the collection program can correspond. The real-time library writing application and the point value display application are all operated based on the mapped points. All point value information is maintained in this configuration service, and the information is centralized and stored in an independent database. There is no special agreement on the database type, and general relational databases can be used. Whether to use a local database or a cloud database depends on whether the customer allows cloud computing.

[0054] In the SCADA (Supervisory Control and Data Acquisition) system, telemetry, telesignaling, teleadjustment, remote control, and telepulse belong to the data categories of different devices. For example, the wind turbine is device 1. The voltage and on / off status of this generator need to be monitored. Then the voltage can be bound to the telemetry category, and the on / off status can be bound to the telesignaling category. The points here belong to the division on the model. It can be simply understood as having an identifier to represent the voltage data. For example, analoginput&1 is a point with a value attribute to represent its value. When the current voltage is 220V, the voltage is reported and will be converted to this point of the model through the acquisition program. The value of this point is 220, which is referred to as the point value.

[0055] The configuration service needs to maintain the collection unit information and bind the points to the collection unit. The collection unit will be bound to the station at the same time, so as to limit the data volume at the source. The relationship between the unit and the station will not be repeated below. They are just split in business. In fact, the goal is to classify the data more reasonably. The unit will be bound to a Redis stand-alone node. The point information under this unit can only be stored on this designated Redis stand-alone node, and it is distinguished by the unique primary key of the unit, that is, the group identifier (groupId).

[0056] groupId is a concept of a group, i.e., a collection group, which manages multiple collection applications. Each collection application will query the associated five remote points, redis information, and message queue information through groupId. The collection applications are connected to different collection device gateways respectively, and the collection applications store the collected points in the message queue. groupId is used to configure associations, and is not stored together when actually processing data. groupId is configured in the configuration service. According to some experts, groupId can have the concept of a business model tree. The model tree has levels and can be bound according to the levels. Whether it is a tree, flat, or other methods, it should be applicable.

[0057] For example, when the data point volume is 400,000, only one unit and one Redis stand-alone node need to be configured. According to the Redis write performance of 100,000 per second, the data can be written in 4 seconds. If the data point volume reaches 4 million, only 10 units and 10 Redis stand-alone nodes need to be maintained, and the writing can still be maintained in 4 seconds. This can support unlimited expansion of the data point volume, and the writing performance is completely unaffected. The instance information of Redis is maintained in the configuration service, and is associated with the stand-alone node identifier redisId through the group identifier groupId and stored in the configuration database. Other programs obtain instance information through the database and connect to Redis.

[0058] In S103, the point value information collected by each collection group is separated according to the configuration information, and written into each message queue corresponding to each collection group.

[0059] The collection application classifies the point value information collected by the collection group. The collection group is a program used to collect data. These programs are divided into groups. A collection group may have multiple collection programs. These collection programs will communicate with the device gateway based on 104, 102 or other protocols. The actual data collection process uses the collection program in the hardware device to collect data, and the collection program transmits data by communicating with the device gateway.

[0060] According to some embodiments, the collection application selects the group identifier for the bound single-machine node manually or automatically; the collection application inputs the selected group identifier into the real-time library write application, and associates the single-machine node identifier through the group identifier; after the collection application receives the point value information from the collection group, it writes it into the message queue.

[0061] The collection application selects a unique groupId for the current node manually or automatically, and passes the selected groupId to the real-time library writing application. The redisId information is associated with the groupId, and after receiving the change information of the device point value, the data will be written to the current corresponding message queue.

[0062] According to some embodiments, the collection application can choose manual or automatic according to its own method. These two methods can be built into the application, and the purpose is to reference other components to achieve automatic selection. For example, based on distributed lock preemption, multiple collection programs only need to dynamically maintain their own application. The manual method requires the application to specify based on the configuration method.

[0063] In S105, the real-time library writing module reads the point value information in the message queue and writes it into the corresponding stand-alone nodes.

[0064] According to some embodiments, the real-time library writing application associates the stand-alone node identifier through the group identifier; the real-time library writing application writes the point value information in the message queue to the stand-alone node according to the stand-alone node identifier. The point value information in the message queue is associated and distinguished with the group identifier, so the amount of data written by the real-time library is predictable, the amount of data is within a controllable range, and the performance of the stand-alone is fully utilized.

[0065] The real-time library writing application associates the redisId information with the groupId and writes the data in the message queue to the Redis stand-alone node. At this time, the data in the message queue is associated with the groupId, so the amount of data written to the real-time library is predictable and within a controllable range, giving full play to the performance of the stand-alone machine. The real-time library writing application puts the collected data into the queue. A real-time library writing application is required to consume the queue data to complete the writing of the redis stand-alone machine. The redis stand-alone node records the changed value and updates it.

[0066] Figure 2 A schematic diagram of a system workflow according to an example embodiment is shown.

[0067] See also Figure 2 , Figure 2 The system working process is shown. The Scada system consists of configuration services, collection applications, real-time library writing applications, point value display applications and other programs. In the configuration service, configure the Redis instance information, the relationship between the unit and the Redis instance, etc. The collection application will separate and modify the data according to the unit ID, and write it to the message queue. The real-time library writing application obtains the data information in the message queue and modifies and writes the single-machine Redis information. The point value query service filters the measurement points according to the unit information for mapping and encapsulation, and tests the mapping with the Redis instance to maintain multi-instance connections.

[0068] According to some embodiments, the information of the stand-alone node is queried through a point value query service, and the point value query service is designed based on the business; the point value query service queries the instance of the stand-alone node where the point value information is stored according to the mapping between the acquisition unit and the stand-alone node configured in the database. The point value information is mapped and mapped using a bitmap; the point value information refers to five remote information, namely, telemetry information, telesignal information, teleadjustment information, remote control information, and telepulse information.

[0069] Collection applications and real-time library writing applications are all run in a stand-alone mode. Point value display applications or other programs that require cross-node access require a point value query service to query Redis information.

[0070] The point value query program will be designed based on the business. The point value information refers to the five remote information, the commonly used ones are telemetry (analoginput) and telesignal (statusinput). The following will take these two remotes as examples. Telemetry and telesignal are stored in the analoginput table and the statusinput table respectively. The key columns in each table are id (unique primary key), name (name), value (value information). Get the value of the telemetry table with id 1, store it in hash mode, the key is analoginput&1, and the field value represents the specific value. Similarly, get the value of the telesignal table with id 1, store it in hash mode, the key is statusinput&1, and the field value represents the specific value.

[0071] The point value query service will query which Redis instance the different telemetry and telesignaling points are stored in based on the unit and Redis mapping configured in the database. The data is mapped and mapped using a bitmap. Different measurement point information is distinguished based on the lowercase table name, and is separated by the "&" symbol and the unique primary key, that is: lowercase table name + "&" + primary key ID. For easier search, analoginput is escaped to 0, and statusinput is escaped to 1. Each type is given a data range of 4 million. Each Redis instance stores a bitmap information separately. Assuming there are 100 instances, storing 100 million data only requires 12MB of memory.

[0072] The unit and the Redis instance are configured and mapped manually based on the page configuration. This is equivalent to the unit corresponding to 1, and the telesignaling points inside correspond to 1 and 2. In this way, the RedisId of these two points is 1, and no calculation is required. The bitmap conversion process is involved, and calculation is required only when looking up whether the point belongs to a certain Redis instance.

[0073] For example: groupId is 1, the configured Redis instance redisId is 1, and the remote signal point id values ​​1, 2, and 3 queried through groupId all belong to this unit; groupId is 2, the configured Redis instance redisId is 2, and the remote signal point id values ​​4, 5, and 6 queried through groupId, then the established mapping relationship is shown in the following table:

[0074] Table 1

[0075]

[0076] The redis instance information corresponding to redisId 1 is ip: 192.168.1.1, port: 6379, and the redis instance information corresponding to redisId 2 is ip: 192.168.1.2, port: 6379. The point value query service will cache this information and cache the connection information in the connection pool information with redisId as the key. When the application connects to the point value query service, it first classifies according to the key, polls each instance bitmap, obtains the redisId information of the key, and then uses redisId to select which node in the connection pool to use as the Redis connection, and finally parses the command to execute the query operation. Redis instance information can be in stand-alone mode, sentinel or cluster mode, which does not affect key search. The point value query service will only open specific query commands, such as: get (get), hgetall (get all fields), allowing multiple keys to be passed in, and will return a list of keys and corresponding values.

[0077] The point value display application is accessed through TCP (Transmission Control Protocol) or Thrift protocol (a cross-language service development framework). After passing the authorization mechanism, the method is called to complete the value query. If the query is too frequent, the query performance will be relatively reduced. You can use a method similar to redis cluster, use decentralized processing or use nginx (a network server and reverse proxy server) as a simple proxy. The scada system focuses more on write performance, and the query frequency is relatively low.

[0078] The present invention focuses on the performance of writing data, and to better utilize the performance of Redis, it is necessary to solve the data volume undertaken by the scada system and improve timeliness. On the one hand, the single-machine performance will be used to improve throughput, and on the other hand, the point value query service will be used to aggregate read data, and no aggregation of write data will be performed. The difference between the present invention and the prior art is that data is coupled based on business to meet the needs of centralized processing of large-scale data. The data between stations can be interoperable, and the processing data volume and timeliness are different. It is a brand-new technical solution. Data is allocated by business, multiple Redis are coupled, the read and write areas are separated, and the performance of Redis is utilized, and tens of millions of station data can be processed.

[0079] Figure 3 A schematic diagram of a system architecture according to an example embodiment is shown.

[0080] See also Figure 3 , Figure 3 The system architecture and the process of data processing and distribution are demonstrated.

[0081] The configuration service is the starting point of the entire system and is responsible for configuring various services and settings. At this stage, it may involve configuring Redis instance information and associated unit information. The collection application is responsible for collecting data and filtering the data and putting it into the message queue. The message queue corresponds to different collection applications. The message queue plays a buffering role here to ensure that data is not lost and allow the system to handle speed mismatches. The real-time library write application obtains data from the message queue and writes the data to the corresponding Redis stand-alone node. The Redis stand-alone nodes correspond to different collection units. The point value query service is responsible for selecting the point value information to be consulted according to the parameters passed by the point value display application, and returns the query result information. The point value display application initiates a request to the point value query service to query and view the required data.

[0082] The system adopts a distributed design and implements asynchronous processing through message queues, which improves the scalability and reliability of the system. The point value query service and point value display application work together to provide users with the required data display.

[0083] Point information is basic information. Other applications will query additional field information based on redis, such as name and cycle. Point values ​​refer more to value and timestamp, that is, the changed value and the changed timestamp. The real-time library writing program will update the data (value and timestamp) sent by the collection application to the message queue to Redis. When the amount of collected data increases, the writing performance is guaranteed by increasing the collection unit and single-machine node. Before writing, the data will be dispersed because the performance of Redis is 100,000 per second, which is difficult to break through. If the data volume is 10 million points, each machine can process 400,000 points, then 25 machines are needed, but 25 machines cannot couple data and cannot support such a large scale. They can only process each machine separately. The number of units does not increase the cost due to the current implementation. The current approach can aggregate these data. At the same time, for customers, whether there are 25 or 100 machines, the performance will not be reduced.

[0084] According to some embodiments, after the performance bottleneck of Redis appears, the access volume of point data volume is increased based on the present invention. Redis cannot improve the write throughput in both stand-alone and cluster mode, and the cluster has no way to provide throughput. Its sharding mode still causes excessive pressure on some nodes. The present invention separates data based on business. After the data level increases, the real-time library still uses the stand-alone mode to write to Redis, solving the problem that the data in the cluster mode still has excessive write pressure on a certain node.

[0085] The real-time library Redis multi-instance aggregation uses a bitmap method to map data to instances based on business characteristics, avoiding multiple MOVED (removal) operations in clusters. The caller does not need to know the internal details, and the service provided is used to implement it, with low access costs. The system is easy to expand, and multiple sets of stand-alone nodes can be quickly added according to the data volume, and the business does not need to be adjusted.

[0086] In the scenario of large data volume, the Scada system splits the data, retains the original single-machine mode, and uses multiple single-machine nodes to form a distributed architecture to infinitely expand the data volume for protection. When querying Redis data, the business data is distinguished, based on the business and Redis instance mapping method, a unique bitmap algorithm is used to reduce memory overhead, which can more quickly find Redis instance information and replace the Redis cluster method for protection. The point value query service will act as an intermediary to query multiple Redis information.

[0087] This patent solves the problem that the data volume is very large and the cluster cannot meet the performance requirements when Redis is used as a real-time library in the scada system. From a business perspective, the data is classified to give full play to the performance of Redis stand-alone machines. At the same time, the architecture is clear at a glance, and the write data is handed over to the stand-alone machine for processing. For the additional query service, a bitmap algorithm based on business use is designed to quickly find nodes, so as to meet the situation where aggregated queries are required. The query service does not need to bear the additional writing capacity. At the same time, this processing method also avoids the use of proxy layers by other companies, which proxy all reads and writes, resulting in performance problems in the proxy layer. The existing patent method has less business transformation and is very suitable for the transformation of old systems with low access costs.

[0088] Through the patented method, it is easy to access tens of millions of data, which can almost cover the scale of existing sites. At the same time, it is easy to expand, has no special requirements for system hardware, and saves resources.

[0089] Figure 4 A block diagram of a computing device is shown according to an exemplary embodiment.

[0090] like Figure 4 As shown, computing device 30 includes processor 12 and memory 14. Computing device 30 may also include bus 22, network interface 16, and I / O interface 18. Processor 12, memory 14, network interface 16, and I / O interface 18 may communicate with each other via bus 22.

[0091] The processor 12 may include one or more general-purpose CPUs (Central Processing Units, processors), microprocessors, or application-specific integrated circuits, etc., for executing relevant program instructions. According to some embodiments, the computing device 30 may also include a high-performance graphics card (GPU) 20 for accelerating the processor 12.

[0092] The memory 14 may include a machine system readable medium in the form of a volatile memory, such as a random access memory (RAM), a read-only memory (ROM) and / or a cache memory. The memory 14 is used to store one or more programs including instructions and data. The processor 12 can read the instructions stored in the memory 14 to execute the above-mentioned method according to the embodiment of the present invention.

[0093] The computing device 30 may also communicate with one or more networks via the network interface 16. The network interface 16 may be a wireless network interface.

[0094] The bus 22 may include an address bus, a data bus, a control bus, etc. The bus 22 provides a path for exchanging information between components.

[0095] It should be noted that, in the specific implementation process, the computing device 30 may also include other components necessary for normal operation. In addition, those skilled in the art may understand that the above device may only include components necessary for implementing the embodiments of this specification, and need not include all components shown in the figure.

[0096] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), a network storage device, a cloud storage device, or any type of medium or device suitable for storing instructions and / or data.

[0097] An embodiment of the present invention further provides a computer program product, which includes a computer program. The computer program is operable to enable a computer to execute part or all of the steps of any one of the methods described in the above method embodiments.

[0098] Those skilled in the art can clearly understand that the technical solution of the present invention can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array, an integrated circuit, etc.

[0099] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0100] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0101] In the several embodiments provided by the present invention, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0104] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0105] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended clauses.

Claims

1. A method for accessing large amounts of data, the method being used in a data acquisition and monitoring control system, the method comprising: Configure the instance information of each stand-alone node, configure the relationship between each acquisition unit and each stand-alone node instance, and write all configuration information into the configuration database; Separating the point value information collected by each collection group through the group identifier, and writing it into each message queue corresponding to each collection group according to the configuration information; The real-time library writing module reads the point value information in the message queue and writes it into the corresponding stand-alone nodes; The collection group and the stand-alone node are bound accordingly according to a unique group identifier, and the stand-alone node is written according to the classification of the point value information; The point value information is queried from the corresponding single-machine node through the point value query service according to the mapping between the acquisition group and the single-machine node configured in the configuration database.

2. The method according to claim 1, characterized in that The stand-alone node is bound to the collection group, and the stand-alone node stores the point value information collected by the bound collection group.

3. The method according to claim 2, characterized in that The point value information is distinguished by the group identification of the collection group; The group identifier is a unique identifier of the collection group.

4. The method according to claim 3, characterized in that The group identifier is used to split and classify data according to business; The acquisition unit is bound to the data source, and the data volume is limited at the data acquisition source.

5. The method according to claim 2, characterized in that: The single-machine node is bound to the acquisition group, including: selecting the group identifier of the corresponding acquisition group for the single-machine node manually or automatically according to the configuration information, and the group identifier is used to associate the corresponding single-machine node identifier.

6. The method according to claim 1, characterized in that The real-time library writing module reads the point value information in the message queue and writes it into the corresponding stand-alone nodes, including: According to the configuration information, the real-time library writing module obtains the corresponding group identifier, and associates the single-machine node identifier through the group identifier; The real-time library writing module writes the point value information in the message queue into the corresponding stand-alone node according to the stand-alone node identifier.

7. The method according to claim 1, characterized in that Establishing a mapping relationship between the point value information includes: using a bitmap method to establish a mapping between the point value information and the single-machine node identifier.

8. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

9. A computing device, characterized in that include: processor; as well as A memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images