Multi-source power data acquisition method and related equipment

Abstract

The invention discloses a multi-source power data acquisition method and related equipment, and the method comprises the steps: carrying out the recognition of obtained multi-source consumption tasks, filtering abnormal tasks in the tasks, obtaining different types of multi-source consumption tasks, and carrying out the arrangement to obtain a target task queue; dividing the obtained multi-source consumption tasks in the target task queue according to a preset parameter set to obtain a plurality of groups of partition data; the method comprises the following steps: analyzing a plurality of groups of partition data, determining a task type corresponding to the partition data, distributing the partition data to a consumer group according to the task type, and executing data consumption or acquisition operation through the consumer group according to the distributed partition data; by identifying and filtering the abnormal tasks and dividing the partition data, the data processing amount of the consumers in the consumer group is reduced, and the processing efficiency can be improved. The method can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and related equipment for acquiring multi-source power data. Background Technology

[0002] In the construction of smart grids, metering devices deployed at the terminals collect data from the terminal equipment in real time, generate corresponding data consumption tasks based on the data, and send them to the processing equipment to collect or consume corresponding data from the massive data according to the data consumption tasks, so as to adjust the operation strategy of the power grid. However, the existing data consumption scheme has low processing efficiency for real-time collection of multi-source massive power data, and it is difficult to meet the real-time and reliability requirements of data processing. Summary of the Invention

[0003] The main objective of this application is to propose a method and related equipment for acquiring multi-source power data, which can improve processing efficiency.

[0004] To achieve the above objectives, one aspect of this application proposes a method for acquiring multi-source power data, the method comprising: Acquire multi-source consumption tasks, identify the multi-source consumption tasks, and determine a target task queue; wherein, the target task queue includes at least one type of multi-source consumption task; The multi-source consumption tasks in the target task queue are divided according to a preset parameter set to obtain several groups of partitioned data; The partition data of several groups is analyzed to determine the task type, and the partition data of several groups is allocated to consumer groups according to the task type, so that the consumer groups can collect data according to the partition data.

[0005] In some embodiments, identifying the multi-source consumption tasks and determining the target task queue specifically includes: The multi-source consumption task is analyzed to determine the current offset and task information; the task information is then used to query a preset database to determine checkpoint information. The current offset is compared with the checkpoint information to determine the comparison result; If the comparison result is a successful comparison, the current multi-source consumption task is marked as a normal task, and the target task queue is generated based on the normal task. If the comparison result is a failure, the current multi-source consumption task will be marked as an abnormal task.

[0006] In some embodiments, the step of dividing the multi-source consumption tasks in the target task queue according to a preset parameter set to obtain several groups of partitioned data specifically includes: The multi-source consumption task is analyzed to determine the current task type, and the target data step size is determined by matching the current task type with the preset parameter set. The multi-source consumption task is divided according to the target data step size to obtain several groups of partition data.

[0007] In some embodiments, the method further includes: Obtain the current load information of the consumer group, and compare it with the current load information according to a preset balancing threshold to determine the status of the consumer group; If the current load information is less than the preset balancing threshold, the consumer group is determined to be in a low load state, and the current state is maintained. If the current load information is greater than or equal to the preset balancing threshold, the consumer group is determined to be in a high-load state, and the number of consumers in the consumer group is adjusted.

[0008] In some embodiments, the method further includes: Obtain the current performance data of the consumer group and compare the current performance data with a preset warning threshold; If the current performance data is less than the preset warning threshold, maintain the current state; If the current performance data is greater than or equal to the preset warning threshold, target consumer information is determined based on the performance data, and warning information is generated based on the target consumer information.

[0009] In some embodiments, the method further includes: Obtain the current load value of each consumer in the consumer group and compare the current load value with a preset adjustment threshold; If the current load value is greater than the preset adjustment threshold, the partition data corresponding to the consumer is divided according to the preset adjustment parameters to obtain a first sub-data and a second sub-data; wherein, the first sub-data includes the data already processed by the consumer; The first sub-data is assigned to the consumer, and the second sub-data is reallocated.

[0010] To achieve the above objectives, another aspect of this application proposes a multi-source power data acquisition system, the system comprising: The acquisition module is used to acquire multi-source consumption tasks, identify the multi-source consumption tasks, and determine the target task queue; wherein, the target task queue includes at least one type of multi-source consumption task; The partitioning module is used to partition the multi-source consumption tasks in the target task queue according to a preset parameter set, and obtain several groups of partitioned data. The execution module is used to analyze several sets of partition data, determine the task type, and allocate several sets of partition data to consumer groups according to the task type, so that the consumer groups can collect data based on the partition data.

[0011] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0012] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0013] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above. The embodiments of this application include at least the following beneficial effects: This application provides a method, system, electronic device, storage medium, and program product for acquiring multi-source power data. This solution identifies the acquired multi-source consumption tasks, filters out abnormal tasks, obtains different types of multi-source consumption tasks, and organizes them into a target task queue. Based on a preset parameter set, the multi-source consumption tasks in the target task queue are divided into several groups of partitioned data. By analyzing these several groups of partitioned data, the task type corresponding to the partitioned data is determined, and then the partitioned data is allocated to consumer groups according to the task type. The consumer groups then perform data consumption or acquisition operations based on the allocated partitioned data. By identifying and filtering abnormal tasks and dividing the partitioned data, the data processing volume of consumers in the consumer groups is reduced, and data processing efficiency is improved. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a method for acquiring multi-source power data according to an embodiment of this application; Figure 2 yes Figure 1 The flowchart of step S101 in the text; Figure 3 yes Figure 1 The flowchart of step S102 in the document; Figure 4 This is a flowchart illustrating load balancing in a multi-source power data acquisition method provided in this application embodiment; Figure 5 This is a flowchart illustrating a performance early warning method for multi-source power data acquisition provided in this application embodiment; Figure 6 This is a flowchart illustrating the redistribution process in a multi-source power data acquisition method provided in this application embodiment; Figure 7 This is a flowchart of a specific embodiment provided in this application; Figure 8 This is a schematic diagram of the structure of a multi-source power data acquisition system provided in an embodiment of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0016] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0017] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] This application provides a method for acquiring multi-source power data, relating to the field of information technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a multi-source power data acquisition method, but is not limited to the above forms.

[0020] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0021] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0022] Figure 1This is an optional flowchart of a multi-source power data acquisition method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S103.

[0023] Step S101: Obtain multi-source consumption tasks, identify the multi-source consumption tasks, and determine the target task queue; wherein, the target task queue includes at least one type of multi-source consumption task. Step S102: Divide the multi-source consumption tasks in the target task queue according to the preset parameter set to obtain several groups of partitioned data; Step S103: Analyze the partitioned data of several groups, determine the task type, and allocate the partitioned data of several groups to consumer groups according to the task type, so that the consumer groups can collect data based on the partitioned data.

[0024] Steps S101 to S103 as shown in this embodiment involve acquiring multi-source consumption tasks from various input sources, processing these tasks, identifying and filtering abnormal task data caused by communication channel fluctuations, transmission network impacts, etc., and generating a target task queue from the filtered multi-source consumption tasks. In this embodiment, the multi-source consumption tasks are processed using the distributed message queue Kafka. Utilizing Kafka's logical characteristics, such as aggregation, filtering, and transformation, abnormal tasks are filtered, and multi-source consumption tasks of different topic types are integrated to obtain the target task queue, such as user electricity consumption data and user information. After listing, the multi-source consumption tasks processed in the target task queue are divided into multiple groups of partitioned data. In this embodiment, the multi-source consumption tasks can be divided according to parameters such as data size and data frequency to obtain corresponding partitioned data. Then, the partitioned data is allocated to consumer groups according to the task type of the multi-source consumption task corresponding to the partitioned data. Consumers in the consumer groups perform data consumption or data acquisition processing based on the allocated partitioned data. In this embodiment, partitioned data belonging to the same task type are allocated to the same consumer group, reducing the latency caused by consumers in the consumer group processing partitioned data of different task types, thereby improving output processing efficiency. Please see Figure 2 In some embodiments, step S101 may include, but is not limited to, steps S201 to S204: Step S201: Analyze the multi-source consumption task to determine the current offset and task information; query the preset database based on the task information to determine checkpoint information; Step S202: Compare the current offset with the checkpoint information to determine the comparison result; Step S203: If the comparison result is successful, mark the current multi-source consumption task as a normal task and generate a target task queue based on the normal task. Step S204: If the comparison result is a failure, mark the current multi-source consumption task as an abnormal task.

[0025] In step S201 of some embodiments, the execution progress of the multi-source consumption task by the consumer is determined by setting an offset. The system periodically obtains the offset of the multi-source consumption task and saves it as a checkpoint in a preset database. The system analyzes the obtained multi-source consumption task to determine the current offset and task information of the multi-source consumption task. The system queries the preset database based on the analyzed task information to determine the checkpoint information corresponding to the multi-source consumption task. The system compares the analyzed current offset with the queried checkpoint information to determine whether the current multi-source consumption task is an abnormal task.

[0026] In step S202 of some embodiments, the system compares the current offset of the current multi-source consumption task with the corresponding checkpoint information to determine whether the current offset and the corresponding checkpoint information are consistent, so as to determine whether the current multi-source consumption task is an abnormal task.

[0027] In step S203 of some embodiments, if the current offset matches the corresponding checkpoint information, the system marks the current multi-source consumption task as a normal task and continues to compare other multi-source consumption tasks until all the acquired multi-source consumption tasks have been compared. The system then integrates the multi-source consumption tasks marked as normal tasks to obtain the target task queue.

[0028] In step S204 of some embodiments, if the current offset is inconsistent with the corresponding checkpoint information, the system marks the current multi-source consumption task as abnormal data, and the system filters the multi-source consumption tasks marked as abnormal data without further processing.

[0029] Please see Figure 3 In some embodiments, step S102 may include, but is not limited to, steps S301 to S302: Step S301: Analyze the multi-source consumption task, determine the current task type, and match the current task type with the preset parameter set to determine the target data step size; Step S302: Divide the multi-source consumption task according to the target data step size to obtain several groups of partitioned data.

[0030] In step S301 of some embodiments, the system analyzes the multi-source consumption tasks in the target task queue, determines the task theme or task type corresponding to the task, and then matches the task theme or task type with the preset parameter set in the system to determine the data step size corresponding to the current multi-source consumption task, which is used for subsequent division of the multi-source consumption task; in this embodiment, the corresponding data step size can be determined according to the task type or data size of the multi-source consumption task for division.

[0031] In step S302 of some embodiments, the system divides the current multi-source consumption task into multiple data groups according to a determined data step size to reduce the amount of data when consumers process consumption tasks in the future, thereby improving data processing efficiency.

[0032] Please see Figure 4 In some embodiments, the multi-source power data acquisition method provided in this application may include, but is not limited to, steps S401 to S403: Step S401: Obtain the current load information of the consumer group, and compare it with the current load information according to the preset balancing threshold to determine the status of the consumer group; Step S402: If the current load information is less than the preset balancing threshold, determine that the consumer group is in a low load state and maintain the current state. Step S403: If the current load information is greater than or equal to the preset balancing threshold, determine that the consumer group is in a high load state, and adjust the number of consumers in the consumer group.

[0033] In step S401 of some embodiments, in order to improve the overall data processing efficiency of the system, it is necessary to perform load balancing on the system to improve the system's processing efficiency. In this embodiment, the system obtains the load information of the consumer group and compares the obtained load information with the preset balancing threshold in the system to determine the current load status of the consumer group and whether to trigger load balancing.

[0034] In step S402 of some embodiments, if the load information is less than a preset balancing threshold, the system determines that the current consumer group is in a low-load state and there is no need to perform load balancing on the consumers in the consumer group. The consumers in the consumer group perform data collection or data consumption according to the partition data they are allocated.

[0035] In step S403 of some embodiments, if the load information is greater than or equal to a preset balancing threshold, the system determines that the current consumer group is in a high-load state, and the system adjusts the number of consumers in the consumer group to balance the overall load of the consumer group, thereby improving the processing efficiency of the system.

[0036] Please see Figure 5In some embodiments, the multi-source power data acquisition method provided in this application may include, but is not limited to, steps S501 to S503: Step S501: Obtain the current performance data of the consumer group and compare the current performance data with the preset warning threshold; Step S502: If the current performance data is less than the preset warning threshold, maintain the current state; Step S503: If the current performance data is greater than or equal to the preset warning threshold, determine the target consumer information based on the performance data, and generate warning information based on the target consumer information.

[0037] In step S501 of some embodiments, the system can adjust the number of consumers in the consumer group to balance the load of the consumer group; it can also determine whether the performance indicators of the consumer group meet the processing requirements of the multi-source consumption task by monitoring the performance data of the consumer group in real time.

[0038] In step S502 of some embodiments, if the performance data of the current consumer group is less than a preset warning threshold, the system determines that the performance indicators of the current consumer group can meet the performance requirements for processing multi-source consumption tasks and partitioned data, and the consumers in the consumer group continue to process the assigned multi-source consumption tasks and partitioned data.

[0039] In step S503 of some embodiments, if the performance data of the current consumer group is greater than or equal to a preset warning threshold, the system determines that the consumers in the current consumer group cannot meet the performance requirements for processing multi-source consumption tasks and partitioned data. The system determines the target consumer information by analyzing the performance data and generates warning information based on the target consumer information to prompt the operation and maintenance personnel to handle the consumer.

[0040] Please see Figure 6 In some embodiments, the multi-source power data acquisition method provided in this application may include, but is not limited to, steps S601 to S603: Step S601: Obtain the current load value of consumers in the consumer group and compare the current load value with the preset adjustment threshold; Step S602: If the current load value is greater than the preset adjustment threshold, the partition data corresponding to the consumer is divided according to the preset adjustment parameters to obtain the first sub-data and the second sub-data; wherein, the first sub-data includes the data already processed by the consumer; Step S603: Assign the first sub-data to the consumer and reallocate the second sub-data.

[0041] In step S601 of some embodiments, the system can also reallocate the partition data or multi-source consumption tasks that the consumer is responsible for processing according to the data load of the consumer in the consumer group, so as to balance the load status of the consumer and improve the processing efficiency of the consumer group. In this embodiment, the system compares the current load value of the consumer in the consumer group with a preset adjustment threshold to determine whether the load of the current consumer needs to be adjusted.

[0042] In step S602 of some embodiments, when the system determines that the current load value of the current consumer is greater than the system's preset adjustment threshold, it divides the partition data allocated to the current consumer. For example, the partition data allocated to the current consumer is divided in a 50:50 ratio to obtain a first sub-data and a second sub-data, wherein the first sub-data includes the data that the current consumer is processing.

[0043] In step S603 of some embodiments, the system assigns the first sub-data after division to the current consumer for further processing, and reassigns the second sub-data to consumers in other consumer groups for processing, so as to improve data processing efficiency and avoid the overall efficiency being affected by the low processing efficiency of a single consumer.

[0044] The following is a detailed description and explanation of the solutions in the embodiments of the present invention, using specific application examples: Please see Figure 7 , Figure 7 This is a flowchart illustrating the data acquisition process of a data processing device using a multi-source power data acquisition method provided in this application, as described in a specific embodiment. The data processing device acquires multi-source consumption data from different data sources through a data interface. It then filters out multi-source consumption data with matching offsets and checkpoints by detecting whether the offsets of the input multi-source consumption data are consistent with pre-stored checkpoints in the system, and integrates them into a consumption task queue. The data processing device divides different types of multi-source consumption data according to a preset data step size, obtaining different types of multi-group partitioned data. The data processing device then allocates the obtained multi-group partitioned data to consumer components used for data processing. The system allocates multi-group partitioned data of the same type to the same consumer component. Consumer instances in the consumer component perform data processing based on the allocated partitioned data, collecting corresponding power data for statistical analysis, cleaning, and other consumption operations.

[0045] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, electronic device, storage medium, and program product for acquiring multi-source power data. This solution identifies the acquired multi-source consumption tasks, filters out abnormal tasks, obtains different types of multi-source consumption tasks, and organizes them into a target task queue. Based on a preset parameter set, the multi-source consumption tasks in the target task queue are divided into several groups of partitioned data. By analyzing these several groups of partitioned data, the task type corresponding to the partitioned data is determined, and then the partitioned data is allocated to consumer groups according to the task type. The consumer groups then perform data consumption or acquisition operations based on the allocated partitioned data. By identifying and filtering abnormal tasks and dividing the partitioned data, the data processing volume of consumers in the consumer groups is reduced, and data processing efficiency is improved.

[0046] Please see Figure 8 This application also provides a multi-source power data acquisition device that can implement the above-described method. The device includes: The acquisition module is used to acquire multi-source consumption tasks, identify the multi-source consumption tasks, and determine the target task queue; wherein, the target task queue includes at least one type of multi-source consumption task; The partitioning module is used to partition the multi-source consumption tasks in the target task queue according to a preset parameter set, and obtain several groups of partitioned data. The execution module is used to analyze several sets of partition data, determine the task type, and allocate several sets of partition data to consumer groups according to the task type, so that the consumer groups can collect data based on the partition data.

[0047] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0048] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0049] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0050] Please see Figure 9 , Figure 9The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0051] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0052] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0053] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0054] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0055] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0057] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0059] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0060] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0062] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0063] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0064] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0065] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for acquiring multi-source power data, characterized in that, The method includes: Acquire multi-source consumption tasks, identify the multi-source consumption tasks, and determine a target task queue; wherein, the target task queue includes at least one type of multi-source consumption task; The multi-source consumption tasks in the target task queue are divided according to a preset parameter set to obtain several groups of partitioned data; The partition data of several groups is analyzed to determine the task type, and the partition data of several groups is allocated to consumer groups according to the task type, so that the consumer groups can collect data according to the partition data.

2. The method according to claim 1, characterized in that, The step of identifying the multi-source consumption tasks and determining the target task queue specifically includes: The multi-source consumption task is analyzed to determine the current offset and task information; the task information is then used to query a preset database to determine checkpoint information. The current offset is compared with the checkpoint information to determine the comparison result; If the comparison result is a successful comparison, the current multi-source consumption task is marked as a normal task, and the target task queue is generated based on the normal task. If the comparison result is a failure, the current multi-source consumption task will be marked as an abnormal task.

3. The method according to claim 1, characterized in that, The step of dividing the multi-source consumption tasks in the target task queue according to a preset parameter set to obtain several groups of partitioned data specifically includes: The multi-source consumption task is analyzed to determine the current task type, and the target data step size is determined by matching the current task type with the preset parameter set. The multi-source consumption task is divided according to the target data step size to obtain several groups of partition data.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the current load information of the consumer group, and compare it with the current load information according to a preset balancing threshold to determine the status of the consumer group; If the current load information is less than the preset balancing threshold, the consumer group is determined to be in a low load state, and the current state is maintained. If the current load information is greater than or equal to the preset balancing threshold, the consumer group is determined to be in a high-load state, and the number of consumers in the consumer group is adjusted.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the current performance data of the consumer group and compare the current performance data with a preset warning threshold; If the current performance data is less than the preset warning threshold, maintain the current state; If the current performance data is greater than or equal to the preset warning threshold, target consumer information is determined based on the performance data, and warning information is generated based on the target consumer information.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the current load value of each consumer in the consumer group and compare the current load value with a preset adjustment threshold; If the current load value is greater than the preset adjustment threshold, the partition data corresponding to the consumer is divided according to the preset adjustment parameters to obtain a first sub-data and a second sub-data; wherein, the first sub-data includes the data already processed by the consumer; The first sub-data is assigned to the consumer, and the second sub-data is reallocated.

7. A multi-source power data acquisition system, characterized in that, The system includes: The acquisition module is used to acquire multi-source consumption tasks, identify the multi-source consumption tasks, and determine the target task queue; wherein, the target task queue includes at least one type of multi-source consumption task; The partitioning module is used to partition the multi-source consumption tasks in the target task queue according to a preset parameter set, and obtain several groups of partitioned data. The execution module is used to analyze several sets of partition data, determine the task type, and allocate several sets of partition data to consumer groups according to the task type, so that the consumer groups can collect data based on the partition data.

8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images