A containerized distributed data acquisition and processing method, apparatus, equipment and medium

Abstract

This invention discloses a containerized distributed data acquisition and processing method, apparatus, device, and medium. The method includes at least one distributed container, which performs the following steps: receiving data to be processed collected from a large number of terminal devices; parsing the data to be processed to obtain data to be stored in a target format; converting the data to be stored according to pre-set rules to obtain data to be used; and merging the data to be used to obtain target stored data. The technical solution of this invention, by deploying containerized distributed nodes on a computer cluster, enables the parsing, conversion, and merging of data uploaded by a large number of terminal devices based on each distributed node. This achieves real-time processing of massive amounts of data uploaded from different terminal types, expanding data processing capacity while improving data processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a containerized distributed data acquisition and processing method, apparatus, device, and medium. Background Technology

[0002] With the development of the power industry, the number of electricity metering devices connected to the power industry will reach hundreds of millions. High-frequency collection and processing of massive amounts of data from these devices is one of the most pressing problems to be solved in the industry.

[0003] The current method for processing massive amounts of data is based on a single server processing complex data uploaded from multiple types of terminals simultaneously. This limits the amount of data that can be processed to the server's capacity, making the system prone to crashes during data processing. As a result, the data processing efficiency is low, the amount of data processed is small, and there is a certain degree of latency. Summary of the Invention

[0004] This invention provides a containerized distributed data acquisition and processing method, apparatus, device, and medium, which enables real-time processing of massive amounts of data uploaded from different terminal types, expanding data processing capacity while improving data processing efficiency.

[0005] In a first aspect, embodiments of the present invention provide a containerized distributed data acquisition and processing method, the method comprising:

[0006] Receives massive amounts of data to be processed from terminal devices;

[0007] The data to be processed is parsed to obtain the data to be stored in the target format;

[0008] The data to be stored is transformed according to a pre-set rule to obtain the data to be used;

[0009] The data to be used is merged and processed to obtain the target storage data.

[0010] Secondly, embodiments of the present invention also provide a containerized distributed data acquisition and processing device, the device comprising:

[0011] The data receiving module is used to receive massive amounts of data to be processed from terminal devices.

[0012] The data parsing module is used to parse and process the data to be processed to obtain data to be stored in the target format;

[0013] The data conversion module is used to convert the data to be stored according to preset rules to obtain the data to be used.

[0014] The data merging module is used to merge the data to be used to obtain the target storage data.

[0015] Thirdly, the present invention also provides an electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the containerized distributed data acquisition and processing method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the containerized distributed data acquisition and processing method according to any embodiment of the present invention.

[0020] The technical solution of this invention involves receiving data to be processed from a massive number of terminal devices; parsing the data to obtain data to be stored in a target format; converting the data to be stored according to pre-set rules to obtain data to be used; and merging the data to be used to obtain the target stored data. Deploying containerized distributed nodes on a computer cluster allows for the parsing, conversion, and merging of data uploaded from a massive number of terminal devices based on each distributed node. This enables real-time processing of massive amounts of data uploaded from different terminal types, expanding data processing capacity while improving data processing efficiency.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a data processing procedure provided according to an embodiment of the present invention;

[0024] Figure 2This is a flowchart of a containerized distributed data acquisition and processing method provided in Embodiment 1 of the present invention;

[0025] Figure 3 This is a data processing module specification diagram provided according to an embodiment of the present invention;

[0026] Figure 4 This is a flowchart of a containerized distributed data acquisition and processing method according to Embodiment 2 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a containerized distributed data acquisition and processing device according to Embodiment 3 of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the containerized distributed data acquisition and processing method of this invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Before introducing this technical solution, we can first illustrate the application scenario as an example: Figure 1 This is a schematic diagram of a data processing procedure provided according to an embodiment of the present invention, such as... Figure 1As shown, during the power production process, the electricity meters configured in various locations generate a large amount of production data. This production data is uploaded to the corresponding terminal devices, and each terminal device eventually uploads this production data to the corresponding distributed nodes. On each distributed node, the production data is processed by message parsing, data conversion, data merging, and other methods.

[0032] Example 1

[0033] Figure 2 This is a flowchart illustrating a containerized distributed data acquisition and processing method according to Embodiment 1 of the present invention. This embodiment is applicable to the acquisition and processing of massive amounts of data generated during power production. The method can be executed by a containerized distributed data acquisition and processing device, which can be implemented in hardware and / or software and can be configured in a computer device. This computer device can be a laptop, desktop computer, or smart tablet, etc.

[0034] like Figure 2 As shown, the method includes:

[0035] S110: Receives data to be processed from a large number of terminal devices.

[0036] The terminal device is a hardware device used to collect real-time message data corresponding to the electricity production data in the electricity meter. A communication link can be pre-established between the terminal device and the electricity meter to enable data transmission between them. The number of terminal devices can be massive, and a single terminal device can connect to one or more electricity meters; this embodiment does not impose any limitations. The data to be processed can be electricity-related data, including at least one of the following: data from coal-fired power meters, gas-fired power meters, and photovoltaic power meters. Furthermore, the data to be processed can be of several types: for example, message data, frozen data, curve data, minute-level data, and event data. Message data mainly includes the heartbeat, control, and parameters of the sensing device; frozen data mainly includes daily / monthly frozen electricity readings and daily / monthly frozen demand; curve data mainly includes electricity, power, current, voltage, and power factor; minute-level data mainly includes minute-level electricity readings and power; and event data mainly includes power outage events, metering device malfunction events, and load overload malfunction events.

[0037] It should be noted that in this embodiment of the invention, the data to be processed is in hexadecimal format.

[0038] Specifically, each containerized distributed node can receive pending data uploaded by the corresponding terminal device at 1-second intervals. Furthermore, there can be one or more distributed containerized nodes, and each distributed containerized node can receive pending data uploaded by one or more terminal devices.

[0039] For example, distributed containerized node 1 receives the data to be processed uploaded by terminal device A, terminal device B, and terminal device C every 1 second.

[0040] S120. The data to be processed is parsed to obtain the data to be stored in the target format.

[0041] The parsing process can include the following two aspects: First, the data parsing module converts the hexadecimal data to be processed into power energy correlation data. For some curve-type data, which are usually represented as isolated points in the message data, the data parsing module can combine these points to obtain complete curve data. Second, the parsed power energy correlation data is converted into a target format. The data to be stored is power energy correlation data in the target format. In this embodiment, the target format is JSON.

[0042] Specifically, since all containerized distributed nodes use the same method to parse and process the data, we will use one containerized distributed node as an example: On the current containerized distributed node, the message data parsing module in the program converts the data to be processed from hexadecimal to JSON format for storage. Because JSON is a lightweight data exchange format, it uses a text format completely independent of programming languages ​​to store and represent data. Therefore, it has the advantages of being easy for humans to read and write, easy for machines to parse and generate, and effectively improving network transmission efficiency.

[0043] S130. The data to be stored is converted according to a preset rule to obtain the data to be used.

[0044] The pre-set rules can be to convert the data collection time in the data to be stored according to the data collection frequency, or to add a corresponding terminal device identifier, running energy meter identifier, etc. to each data in the data to be stored. The data to be used is the data obtained after transformation processing based on the data to be stored according to the pre-set rules.

[0045] Specifically, since all containerized distributed nodes use the same method to transform and process the data to be stored, we will now use a containerized distributed node as an example: On the current containerized distributed node, the data to be stored can be transformed and processed through the business data transformation module in the program to obtain the data to be used.

[0046] S140. The data to be used is merged and processed to obtain the target storage data.

[0047] The merging process can involve combining all data collected from different meter devices within a specific time frame. The target storage data refers to the data corresponding to the different business data for each meter.

[0048] Specifically, since all containerized distributed nodes use the same method for merging the data to be used, we will use one containerized distributed node as an example: On the current containerized distributed node, the business data merging module in the program can merge the data to be used collected by meter device 1 during the time period of 9:00-9:15 according to the time sequence, thereby obtaining the target storage data corresponding to meter device 1 during the time period of 9:00-9:15. In practical applications, since the capacity of a single message transmission is limited, a task data may be sent in multiple parts, thus requiring the merging of all the data to be used corresponding to that business.

[0049] Based on the above embodiments, the specifications between the modules can be uniformly managed, so as to perform data analysis and processing based on the unified management rules.

[0050] Specifically, the containerized distributed node program includes a message data parsing module, a business data conversion module, and a business data merging module. These three modules use an internal protocol, which allows the modules to directly determine whether data needs to be collected and processed, avoiding the need to parse the message again, reducing repetitive steps and improving the efficiency of cooperation between modules. Figure 3 This is a data processing module specification diagram provided according to an embodiment of the present invention, such as... Figure 3 As shown, the internal specification can establish corresponding relationships between each field, field length, and specific field name.

[0051] In this embodiment of the invention, the message data parsing module, the service data conversion module, and the service data merging module correspond to three data processing processes. The data reception and transmission between these three modules use an asynchronous streaming processing method, changing serial processing to concurrent waterfall processing, thereby further improving the efficiency of data processing.

[0052] The technical solution of this invention involves receiving data to be processed from a massive number of terminal devices; parsing the data to obtain data to be stored in a target format; converting the data to be stored according to pre-set rules to obtain data to be used; and merging the data to be used to obtain the target stored data. Deploying containerized distributed nodes on a computer cluster allows for the parsing, conversion, and merging of data uploaded from a massive number of terminal devices based on each distributed node. This enables real-time processing of massive amounts of data uploaded from different terminal types, expanding data processing capacity while improving data processing efficiency.

[0053] Example 2

[0054] Figure 4 This is a flowchart of a containerized distributed data acquisition and processing method provided in Embodiment 2 of the present invention. Based on the foregoing embodiments, the data to be stored can be transformed according to a pre-set rule to obtain data to be used. The data to be used is then merged to obtain target storage data for further refinement. For specific implementation details, please refer to the detailed description of the embodiments of the present invention. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0055] like Figure 4 As shown, the method includes:

[0056] S210: Receives massive amounts of data to be processed from terminal devices.

[0057] S220. The data to be processed is parsed to obtain the data to be stored in the target format.

[0058] S230: Load metering point files and terminal files based on the business data conversion module to determine at least one of the following: terminal identifier, terminal type, operating energy meter identifier, measurement point code, terminal address, and metering point type.

[0059] The business data conversion module is the module in the program that converts the data to be processed. The metering point file refers to a file containing information about each electricity meter. The metering point file can include: operating meter identifier, measurement point code, terminal address, and metering point type. The operating meter identifier is a string that uniquely identifies the corresponding electricity meter within the program. The measurement point code is the code representing the electricity meter in the data to be stored. The terminal address is the IP address corresponding to each terminal. The metering point type refers to the type number of the electricity meter; for example, metering point type 8000 indicates that the electricity meter is a power station meter, and metering point type 8300 indicates that the electricity meter is a residential meter. The terminal file refers to a file containing the associated information for each terminal. The terminal file can include: terminal identifier, terminal address, and terminal type. The terminal identifier is a string that uniquely identifies the corresponding terminal within the program. The terminal type refers to the type of terminal equipment classified according to the type of electricity meter connected, represented by numbers in the file. For example, 01 indicates that the terminal type is for power station meters; 02 indicates that the terminal type is for residential meters.

[0060] Specifically, the business data conversion module loads the metering point file and terminal file, and then determines the terminal identifier, terminal address, terminal type, operating energy meter identifier, measurement point code, terminal address, and metering point type.

[0061] For example, since each line of data in the metering point file has the same meaning, we will now explain using the first line of data: The first line of data is [071015048 101 00108110 5], which represents the operating energy meter identifier, measurement point code, terminal address, and metering point type, respectively. Similarly, in the terminal file, the first line of data is [1111198521 008555515 04], which represents the terminal identifier, terminal address, and terminal type corresponding to a certain terminal, respectively.

[0062] S240. Based on the data to be stored and the metering point file and terminal file, determine the terminal identifier and the operating energy meter identifier.

[0063] Specifically, since the method for determining each terminal identifier is the same, the method for determining one terminal identifier will be explained below: Based on the terminal address corresponding to the current terminal in the data to be stored, the terminal file is traversed, and the terminal identifier corresponding to the current terminal address in the terminal file is embedded into the corresponding position in the data to be stored. For example, the terminal identifier can be embedded after the terminal address or at the end of the data to be stored corresponding to that terminal address. Performing the same operation on each terminal address in the data to be stored yields the terminal identifier corresponding to each terminal within the program.

[0064] Furthermore, since the method for determining the identifiers of all operating energy meters is the same, the method for determining one of the operating energy meter identifiers will now be explained: Based on the terminal address and measurement point code corresponding to the current energy meter in the data to be stored, the metering point file is traversed, and the operating energy meter identifier and metering point type corresponding to the current energy meter in the metering point file are embedded into the corresponding position in the data to be stored. For example, the operating energy meter identifier and metering point type can be embedded after the terminal address and measurement point code or at the end of the data to be stored corresponding to the terminal address and measurement point code. Performing the same operation on each energy meter in the data to be stored yields the operating energy meter identifier corresponding to each energy meter within the program.

[0065] S250. Load the task file based on the business data conversion module to determine the task to be collected, and convert the time of the task to be collected according to the collection time range corresponding to the task to be collected to obtain the target task time; wherein, the collection time range includes at least one collection moment.

[0066] The task file refers to the archive that stores each data collection task and the data associated with each task. For example, the data associated with each data collection task may include, but is not limited to, data such as the data collection task ID, terminal type, collection frequency, and the data collection task itself. The task to be collected refers to the process of collecting various business data. For example, the task to be collected may be residential daily frozen energy consumption readings, minute-level energy consumption readings, monthly frozen demand, or meter power curves. The data collection task time refers to the initial moment corresponding to the collection of data for a specific task to be collected. The data collection time range refers to the collection frequency corresponding to each data collection task in the task file.

[0067] Specifically, on containerized distributed nodes, task files are loaded, and each collection task in the task file is identified as a task to be collected. The collection time range corresponding to each task to be collected is then obtained from the task file.

[0068] For example, the task file is loaded through the business data conversion module. Based on the task ID of the electricity meter power curve task in the task file, the corresponding data to be stored is found in the data to be stored. The collection time range corresponding to the electricity meter power curve task is 15 minutes. The time of the task to be collected is 2022-11-25, 9:00. Then the target task time is 2022-11-25, 9:00-9:15.

[0069] S260. Based on the data to be stored, the terminal identifier, the operating energy meter identifier, and the target task time, obtain the corresponding data to be used.

[0070] Specifically, based on the data to be stored, the terminal identifier can be added to the end of the data corresponding to each terminal address, and the identifier of each operating energy meter and the metering point type can be added to the end of the data corresponding to each energy meter's terminal address and measurement point code. Simultaneously, the time of each task to be collected is converted according to the corresponding collection time range to obtain the data to be used.

[0071] S270. Determine the identifier of the operating energy meter corresponding to the task to be collected.

[0072] Specifically, based on S260, the identifier of the corresponding operating energy meter for each task to be collected can be determined from the data to be used corresponding to each task to be collected.

[0073] S280. Based on the operating energy meter identifier, merge the data to be used at least one collection time corresponding to the operating energy meter identifier, and convert it into JSON format for output to obtain the target storage data.

[0074] Specifically, since the processing method for the pending data corresponding to each operating energy meter is the same, the pending data for one operating energy meter will be explained below: The pending data for each time period corresponding to the current operating energy meter identifier is obtained. Then, based on the target task time corresponding to each data collection task, the pending data for each collection time within each target task time is merged, thus obtaining the pending data for each data collection task. The pending data for each data collection task corresponding to each energy meter is then converted into JSON format and output, resulting in the target storage data so that users can view the task data for each meter.

[0075] The technical solution of this invention involves loading metering point files and terminal files based on a business data conversion module to determine at least one of the following: terminal identifier, terminal address, terminal type, operating energy meter identifier, measurement point code, terminal address, and metering point type; determining the terminal identifier and the operating energy meter identifier based on the data to be stored, the metering point files, and the terminal files; loading a task file based on the business data conversion module to determine the task to be collected, and converting the time of the task to be collected according to the collection time range corresponding to the task to be collected to obtain the target task time; wherein, the collection time range includes at least one collection moment; and obtaining corresponding data to be used based on the data to be stored, the terminal identifier, the operating energy meter identifier, and the collection time range. The system identifies the operating energy meter identifier corresponding to the task to be collected. Based on the operating energy meter identifier, it merges the data to be used at least one collection time corresponding to the operating energy meter identifier and converts it into JSON format for output, thus obtaining the target storage data. It also obtains the terminal identifier, metering point identifier, and measurement point type based on the terminal file and metering point file, and merges the data of each collection task using the operating energy meter identifier and the target task time, converting it into JSON format. This process extracts data from each meter task, improving data processing efficiency while facilitating user viewing of meter data.

[0076] Example 3

[0077] Figure 5 This is a schematic diagram of a containerized distributed data acquisition and processing device provided in Embodiment 3 of the present invention.

[0078] like Figure 5 As shown, the device includes:

[0079] The data receiving module 310 is used to receive data to be processed collected by a large number of terminal devices; the data parsing module 320 is used to parse and process the data to be processed to obtain data to be stored in the target format; the data conversion module 330 is used to convert the data to be stored according to a preset rule to obtain data to be used; and the data merging module 340 is used to merge the data to be used to obtain the target storage data.

[0080] Based on the above technical solutions, the data receiving module 310 is specifically used for:

[0081] Receives a large amount of electricity-related data collected by terminal devices; wherein the electricity-related data includes at least one of the following: data from coal-fired power meters, gas-fired power meters, and photovoltaic power meters.

[0082] Based on the above technical solutions, the data parsing module 320 is specifically used for:

[0083] The message data parsing module converts the data to be processed from hexadecimal to JSON format for storage.

[0084] Based on the above technical solutions, the data conversion module 330 specifically includes:

[0085] The file loading unit is used to load metering point files and terminal files based on the business data conversion module, so as to determine at least one of the following: terminal identifier, terminal address, terminal type, operating energy meter identifier, measurement point code, terminal address, and metering point type.

[0086] The identifier determination unit is used to determine the terminal identifier and the operating energy meter identifier based on the data to be stored and the metering point file and terminal file;

[0087] The data collection time range determination unit is used to load task files based on the business data conversion module to determine the task to be collected, and to convert the time of the task to be collected according to the data collection time range corresponding to the task to be collected to obtain the target task time; wherein, the data collection time range includes at least one data collection moment.

[0088] The data to be used determination unit is used to obtain the corresponding data to be used based on the data to be stored, the terminal identifier, the operating energy meter identifier, and the target task time.

[0089] Based on the above technical solutions, the data merging module 340 is specifically used for:

[0090] Identify the operating energy meter identifier corresponding to the task to be collected; based on the operating energy meter identifier, merge the data to be used at least one collection time corresponding to the operating energy meter identifier, and convert it into JSON format for output to obtain the target storage data.

[0091] Based on the above technical solutions, the data processing device may include:

[0092] The specification management module is used to manage the specifications among the various modules in a unified manner, so as to perform data analysis and processing based on the unified management rules.

[0093] The technical solution of this invention involves receiving data to be processed from a massive number of terminal devices; parsing the data to obtain data to be stored in a target format; converting the data to be stored according to pre-set rules to obtain data to be used; and merging the data to be used to obtain the target stored data. Deploying containerized distributed nodes on a computer cluster allows for the parsing, conversion, and merging of data uploaded from a massive number of terminal devices based on each distributed node. This enables real-time processing of massive amounts of data uploaded from different terminal types, expanding data processing capacity while improving data processing efficiency.

[0094] The containerized distributed data acquisition and processing device provided in the embodiments of the present invention can execute the containerized distributed data acquisition and processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0095] Example 4

[0096] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0097] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0098] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0099] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as containerized distributed data acquisition and processing methods.

[0100] In some embodiments, the containerized distributed data acquisition and processing method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the containerized distributed data acquisition and processing method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the containerized distributed data acquisition and processing method by any other suitable means (e.g., by means of firmware).

[0101] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0106] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0107] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0108] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A containerized distributed data acquisition and processing method, characterized in that, include: At least one distributed container, the distributed container being used to perform the following steps: Receives massive amounts of data to be processed from terminal devices; The data to be processed is parsed to obtain the data to be stored in the target format; The data to be stored is transformed according to a pre-set rule to obtain the data to be used; The data to be used is merged and processed to obtain the target storage data; The step of transforming the data to be stored according to a pre-set rule to obtain the data to be used includes: The business data conversion module loads metering point files and terminal files to determine at least one of the following: terminal identifier, terminal address, terminal type, operating energy meter identifier, measurement point code, terminal address, and metering point type. Based on the data to be stored and the metering point files and terminal files, the terminal identifier and the operating energy meter identifier are determined; The task file is loaded based on the business data conversion module to determine the task to be collected, and the time of the task to be collected is converted according to the collection time range corresponding to the task to be collected to obtain the target task time; wherein, the collection time range includes at least one collection moment, and the task to be collected refers to the collection process of various business data. Based on the data to be stored, the terminal identifier, the operating energy meter identifier, and the target task time, the corresponding data to be used is obtained.

2. The method according to claim 1, characterized in that, The data to be processed collected by the massive number of terminal devices includes: Receives massive amounts of power-related data collected from terminal devices; The electricity-related data includes at least one of the following: data from coal-fired power meters, gas-fired power meters, and photovoltaic power meters.

3. The method according to claim 1, characterized in that, The step of parsing the data to be processed to obtain the data to be stored in the target format includes: The message data parsing module converts the data to be processed from hexadecimal to JSON format for storage.

4. The method according to claim 1, characterized in that, The merging and processing of the data to be used to obtain the target stored data includes: Identify the operating energy meter identifier corresponding to the task to be collected; Based on the operating energy meter identifier, the data to be used at least one collection time corresponding to the operating energy meter identifier is merged and processed, and then converted into JSON format for output to obtain the target storage data.

5. The method according to claim 1, characterized in that, Also includes: The specifications between the modules are managed in a unified manner, and data analysis and processing are carried out based on the rules of the unified management.

6. A containerized distributed data acquisition and processing device, characterized in that, include: At least one distributed container, the distributed container being used to perform the following steps: The data receiving module is used to receive massive amounts of data to be processed from terminal devices. The data parsing module is used to parse and process the data to be processed to obtain data to be stored in the target format; The data conversion module is used to convert the data to be stored according to preset rules to obtain the data to be used. The data merging module is used to merge the data to be used to obtain the target storage data; Specifically, the data conversion module includes: The file loading unit is used to load metering point files and terminal files based on the business data conversion module, so as to determine at least one of the following: terminal identifier, terminal address, terminal type, operating energy meter identifier, measurement point code, terminal address, and metering point type. The identifier determination unit is used to determine the terminal identifier and the operating energy meter identifier based on the data to be stored and the metering point file and terminal file; The data collection time range determination unit is used to load task files based on the business data conversion module to determine the task to be collected, and to convert the time of the task to be collected according to the data collection time range corresponding to the task to be collected to obtain the target task time; wherein, the data collection time range includes at least one data collection moment, and the task to be collected refers to the data collection process of various business data. The data to be used determination unit is used to obtain the corresponding data to be used based on the data to be stored, the terminal identifier, the operating energy meter identifier, and the target task time.

7. The apparatus according to claim 6, characterized in that, The data receiving module includes: The associated data receiving unit is used to receive power-related data collected by a large number of terminal devices; The electricity-related data includes at least one of the following: data from coal-fired power meters, gas-fired power meters, and photovoltaic power meters.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the containerized distributed data acquisition and processing method according to any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the containerized distributed data acquisition and processing method according to any one of claims 1-5.

Images