Data supplementary collection method and related equipment
By analyzing task types and adopting retry processing and data supplementation mechanisms, the problem of data loss in smart grids was solved, the reliability of data processing systems was improved, and the stable operation of smart grids was ensured.
Patent Information
- Application Number
- CN202511584821.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-02-10
AI Technical Summary
Existing data processing systems in smart grids suffer from data loss and low data reliability due to environmental factors, which affects the normal operation of smart grids.
By acquiring a set of tasks to be processed, analyzing the task types, and employing retry processing and data re-collection mechanisms for failed tasks and data backtracking tasks respectively, including retry parameter settings, data backtracking, and load balancing, data reliability is improved.
This reduces the impact of data loss, improves the reliability of the data processing system, and ensures the normal operation of the smart grid.
Smart Images
Figure CN121502160A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data acquisition method and related equipment. Background Technology
[0002] In the operation of smart grids, terminal metering devices and IoT devices collect and upload massive amounts of power data in real time for processing. The data processing system consumes the received massive amounts of power data according to the generated data processing tasks, including data collection and analysis. However, existing data processing systems are prone to data loss due to environmental and other factors during data processing, resulting in low data reliability. This can lead to data processing system malfunctions and affect the normal operation of the smart grid. Summary of the Invention
[0003] The main objective of this application is to propose a data acquisition method and related equipment that can improve data reliability.
[0004] To achieve the above objectives, one aspect of this application proposes a data supplementation method, the method comprising: Obtain a set of tasks to be processed, analyze the set of tasks to be processed, and determine the current task type; If the current task type is a failed task, the failed task is retried according to the preset retry parameters; If the current task type is a data backtracking task, analyze the data backtracking task to determine the target data segment information and task information, and perform data supplementation based on the target data segment information and the task information.
[0005] In some embodiments, the step of retrying the failed task according to preset retry parameters specifically includes: The preset retry parameters are parsed to determine the preset number of retries and the retry interval. Data is collected based on the retry interval and the failed task to determine the task execution result. If the task execution result is data collection complete, return the data collection result; If the task execution result is a data acquisition failure, record the current retry count, and process the failed task according to the current retry count and the preset retry count.
[0006] In some embodiments, processing the failed task based on the current retry count and the preset retry count specifically includes: Compare the current number of retries with the preset number of retries; If the current number of retries is less than the preset number of retries, return to the execution of data collection based on the retry interval and the failed task; If the current number of retries is greater than or equal to the preset number of retries, the failed task is sent to a preset dead-letter queue, and the task is reassigned according to the preset dead-letter queue. In some embodiments, the step of performing data supplementation based on the target data segment information and the task information specifically includes: The target checkpoint information is determined by querying the preset database based on the target data segment information. The offset of consumers in the consumer group is adjusted according to the target checkpoint information to determine the target offset, and the task information is sent to the consumer so that the consumer can perform supplementary data collection based on the task information and the target offset.
[0007] In some embodiments, the method further includes: Obtain the current load value and compare it with a preset load balancing threshold; wherein, the current load value includes the load value corresponding to each consumer in the consumer group; If the current load value is less than the preset load balancing threshold, maintain the current state; If the current load value is greater than or equal to the preset balancing threshold, the number of consumers in the consumer group is adjusted.
[0008] In some embodiments, the method further includes: Obtain the current performance indicator value and compare the current performance indicator value with a preset warning threshold; If the current performance indicator value is less than the preset warning threshold, maintain the current state; If the current performance indicator value is greater than or equal to the preset warning threshold, a warning is issued based on the current performance indicator value, and a target consumer is identified based on the current performance indicator value, and resources are adjusted for the target consumer.
[0009] To achieve the above objectives, another aspect of this application proposes a data supplementation system, the system comprising: The analysis module is used to acquire a set of tasks to be processed, analyze the set of tasks to be processed, and determine the current task type; The processing module is used to retry the failed task according to preset retry parameters if the current task type is a failed task. If the current task type is a data backtracking task, analyze the data backtracking task to determine the target data segment information and task information, and perform data supplementation based on the target data segment information and the task information.
[0010] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0011] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.
[0012] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.
[0013] The embodiments of this application include at least the following beneficial effects: This application provides a data supplementation method, system, electronic device, storage medium, and program product. This solution obtains a set of tasks to be processed, analyzes the tasks in the set, and determines the current task type. If the current task type is a failed task, a retry mechanism is executed according to preset retry parameters to retry the failed task. If the current task type is a data backtracking task, the data backtracking task is analyzed to determine the target data segment information and specific task information that need to be backtracked. Data supplementation is then performed based on the target data segment information and task information to obtain the corresponding dataset and achieve data backtracking. By setting a retry mechanism to retry failed tasks and re-collecting the corresponding data for data backtracking tasks, the impact of data loss is reduced, and data reliability is improved. Attached Figure Description
[0014] Figure 1 This is a flowchart of a data supplementation method provided in an embodiment of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 2 The flowchart of step S203 in the process; Figure 4 yes Figure 1 The flowchart of step S103 in the process; Figure 5 This is a flowchart illustrating load balancing in a data supplementation method provided in an embodiment of this application; Figure 6 This is a flowchart illustrating performance warning in a data supplementation method provided in this application embodiment; Figure 7 This is a flowchart of a specific embodiment provided in this application; Figure 8 This is a schematic diagram of the structure of a data acquisition system provided in an embodiment of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0016] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”
[0017] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.
[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0019] This application provides a data supplementation method, relating to the field of information technology. This data supplementation method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a data supplementation method, but is not limited to the above forms.
[0020] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0021] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0022] Figure 1 This is an optional flowchart of a data supplementation method provided in an embodiment of this application. Figure 1The method may include, but is not limited to, steps S101 to S103.
[0023] Step S101: Obtain the set of tasks to be processed, analyze the set of tasks to be processed, and determine the current task type; Step S102: If the current task type is a failed task, retry the failed task according to the preset retry parameters. Step S103: If the current task type is a data backtracking task, analyze the data backtracking task, determine the target data segment information and task information, and perform data supplementation based on the target data segment information and task information.
[0024] In steps S101 to S103 of this embodiment, in the data processing device, the current offset of the task is checked to see if it matches the checkpoint information recorded in the device, thus identifying whether the task is a normal task. The identified failed or abnormal tasks are filtered out and isolated from normal tasks. A separate process is set up to handle the failed or abnormal tasks, avoiding interference with the normal task processing. Simultaneously, the data processing device also receives data backtracking tasks and assigns these tasks, along with the failed or abnormal tasks, to a separately configured process for processing. The system integrates data backtracking tasks, failed processing tasks, or abnormal tasks into a set of tasks to be processed. An independent process analyzes this set to identify the data backtracking tasks, failed processing tasks, or abnormal tasks. The independent process then calls a consumer group in the data processing device to process these tasks. The consumer group uses a retry mechanism to process the identified failed processing tasks or abnormal tasks, re-collecting the corresponding data to reduce data loss caused by communication anomalies or network fluctuations. Finally, the consumer group determines the data segments that need to be backtracked for the identified data backtracking tasks and performs corresponding data re-collection to achieve data backtracking.
[0025] Please see Figure 2 In some embodiments, step 102 may include, but is not limited to, steps S201 to S203: Step S201: Analyze the preset retry parameters, determine the preset number of retries and retry interval, and collect data based on the retry interval and the processing of failed tasks to determine the task execution result; Step S202: If the task execution result is data collection complete, return the data collection result; Step S203: If the task execution result is data acquisition failure, record the current retry count, and process the failed task according to the current retry count and the preset retry count.
[0026] In step S201 of some embodiments, the data processing device sets a retry mechanism to retry the failed task. The device determines the preset number of retries and the corresponding retry interval by parsing the preset retry parameters, re-collects the corresponding data for the failed task according to the retry interval, and determines whether the corresponding data collection is completed, so as to determine whether the failed task has been successfully retried.
[0027] In step S202 of some embodiments, if the consumer determines that the data collection is successful, such as the collected data having no missing data and the data integrity check passing, the consumer group returns the collected data to the data processing device. The data processing device then performs subsequent data consumption based on the feedback data and deletes the currently failed processing task from the pending tasks.
[0028] In step S203 of some embodiments, if the data collected by the consumer is still missing or the data integrity test fails, the data processing device determines that the current processing failed task has failed, records the current number of retries, and determines whether to retry the current processing failed task based on the preset number of retries and the current number of retries.
[0029] Please see Figure 3 In some embodiments, step 203 may include, but is not limited to, steps S301 to S303: Step S301: Compare the current number of retries with the preset number of retries; Step S302: If the current number of retries is less than the preset number of retries, return to the execution of data collection based on the retry interval and the failed task. Step S303: If the current number of retries is greater than or equal to the preset number of retries, the failed task will be sent to the preset dead letter queue, and the task will be redistributed according to the preset dead letter queue.
[0030] In step S301 of some embodiments, after the data processing device detects that the retry of the failed processing task has failed, it compares the recorded current number of retries with the preset number of retries to determine whether to continue retrying.
[0031] In step S302 of some embodiments, if the recorded current number of retries is less than the preset number of retries, the consumer will collect data on the failed task again after a preset retry interval and determine whether the data collection was successful.
[0032] In step S303 of some embodiments, if the recorded current number of retries is greater than or equal to the preset number of retries, in order to avoid the processing failure task occupying device resources and thus affecting the processing efficiency of the data processing device, the processing failure task is sent to the dead letter queue, and the consumer continues to process other tasks; the data processing device assigns the tasks in the dead letter queue to other consumers for processing, or to be manually processed by professionals.
[0033] Please see Figure 4 In some embodiments, step 103 may include, but is not limited to, steps S401 to S402: Step S401: Query the preset database based on the target data segment information to determine the target checkpoint information; Step S402: Adjust the offset of consumers in the consumer group according to the target checkpoint information, determine the target offset, and send the task information to the consumers so that the consumers can perform data supplementation according to the task information and the target offset.
[0034] In step S401 of some embodiments, the consumer analyzes the assigned data backtracking task to obtain the target data segment information that needs to be backtracked. Since the data processing device periodically detects the task offset as checkpoint information and saves it during task processing, if the consumer's task processing is interrupted due to abnormal conditions such as interference, the checkpoint information can be used to quickly locate and resume task processing. Therefore, data backtracking can also be performed using checkpoint information. The consumer queries a preset database based on the target data segment information to determine the target checkpoint information corresponding to the target data segment for data backtracking.
[0035] In step S402 of some embodiments, the consumer adjusts the task offset according to the target data segment information obtained from the query. The consumer collects data based on the adjusted task offset and the corresponding task information to obtain the data segment that needs to be backtracked, completes data supplementation, and realizes data backtracking.
[0036] Please see Figure 5 In some embodiments, the data acquisition method provided in this application may also include, but is not limited to, steps S501 to S503: Step S501: Obtain the current load value and compare it with the preset load balancing threshold; wherein, the current load value includes the load value corresponding to each consumer in the consumer group; Step S502: If the current load value is less than the preset balancing threshold, maintain the current state; Step S503: If the current load value is greater than or equal to the preset balancing threshold, adjust the number of consumers in the consumer group.
[0037] In step S501 of some embodiments, the data processing device detects the load status of each consumer in the consumer group in real time and performs load balancing based on the load status to avoid the efficiency of data supplementation being affected by excessive load of consumers, as well as the risk of data loss under high load operation; the data processing device compares the load value of consumers collected in real time with the set balancing threshold to determine whether load balancing of consumers is necessary.
[0038] In step S502 of some embodiments, if the load value collected in real time for consumers is less than the set load balancing threshold, the data processing device does not perform load balancing operation, and each consumer is in a low load state with high processing efficiency.
[0039] In step S503 of some embodiments, if the load value collected in real time for the consumer is greater than or equal to the set balancing threshold, it indicates that the consumer is under high load. The data processing device adjusts the number of consumers in the current consumer group, increases the number of consumers, and reallocates the tasks assigned to the consumer group to reduce the load value of the consumer group.
[0040] Please see Figure 6 In some embodiments, the data acquisition method provided in this application may also include, but is not limited to, steps S601 to S603: Step S601: Obtain the current performance indicator value and compare the current performance indicator value with the preset warning threshold. Step S602: If the current performance indicator value is less than the preset warning threshold, maintain the current state; Step S603: If the current performance index value is greater than or equal to the preset warning threshold, issue a warning based on the current performance index value, determine the target consumer based on the current performance index value, and adjust resources for the target consumer.
[0041] In step S601 of some embodiments, in addition to load balancing the consumer group, the data processing device also monitors in real time whether the performance indicators of each consumer in the consumer group meet the requirements of the processing task, so as to avoid abnormal or low-performance consumers affecting the overall processing efficiency of the data processing device; the data processing device compares the real-time monitored consumer performance indicator data with a preset warning threshold to determine whether the consumer meets the requirements of the processing task. In step S602 of some embodiments, if the real-time detected consumer performance index data is less than a preset warning threshold, the device determines that the performance index of each consumer in the consumer group meets the requirements of the processing task, and the data processing device maintains the current state. In step S603 of some embodiments, if the real-time detected consumer performance index data is greater than or equal to a preset warning threshold, the data processing device determines that a consumer in the consumer group has a performance abnormality and cannot meet the processing task requirements; the data processing device issues a warning based on the consumer information with performance abnormality and adjusts the resources of the consumer, for example, closing the consumer and calling other idle consumers to continue executing the task of the consumer with performance abnormality.
[0042] The following is a detailed description and explanation of the solutions in the embodiments of the present invention, using specific application examples: Please see Figure 7 , Figure 7 This is a flowchart illustrating a data supplementation method provided in this application embodiment, applied in a specific embodiment. In this embodiment, the system acquires data consumption tasks through a data interface, identifies the acquired data consumption tasks, and filters out failed consumption tasks and backtracking tasks. The system assigns the filtered failed consumption tasks and backtracking tasks to corresponding consumer groups, and consumers in the consumer groups identify the assigned task type. For failed consumption tasks, consumers call a preset number of retries and retry interval to collect data from the failed consumption tasks, and send consumption tasks that fail multiple times to a dead-letter queue. For backtracking tasks, consumers analyze the backtracking tasks, determine the data information that needs to be backtracked, and query a preset database based on this data information to determine the corresponding checkpoint information. Consumers adjust their own task offset based on the queried checkpoint information and execute the backtracking task based on the adjusted task offset to collect data, thereby achieving data supplementation.
[0043] The embodiments of this application include at least the following beneficial effects: This application provides a data supplementation method, system, electronic device, storage medium, and program product. This solution obtains a set of tasks to be processed, analyzes the tasks in the set, and determines the current task type. If the current task type is a failed task, a retry mechanism is executed according to preset retry parameters to retry the failed task. If the current task type is a data backtracking task, the data backtracking task is analyzed to determine the target data segment information and specific task information that need to be backtracked. Data supplementation is then performed based on the target data segment information and task information to obtain the corresponding dataset and achieve data backtracking. By setting a retry mechanism to retry failed tasks and re-collecting the corresponding data for data backtracking tasks, the impact of data loss is reduced, and data reliability is improved.
[0044] Please see Figure 8 This application also provides a data supplementation system that can implement the above-described method. The system includes: The analysis module is used to acquire a set of tasks to be processed, analyze the set of tasks to be processed, and determine the current task type; The processing module is used to retry the failed task according to preset retry parameters if the current task type is a failed task. If the current task type is a data backtracking task, analyze the data backtracking task to determine the target data segment information and task information, and perform data supplementation based on the target data segment information and the task information.
[0045] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0046] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0047] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0048] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0049] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0050] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0051] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0052] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0053] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0054] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0055] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0056] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0057] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0058] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0059] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0060] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0061] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0062] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0063] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0064] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A data supplementation method, characterized in that, The method includes: Obtain a set of tasks to be processed, analyze the set of tasks to be processed, and determine the current task type; If the current task type is a failed task, the failed task is retried according to the preset retry parameters; If the current task type is a data backtracking task, analyze the data backtracking task to determine the target data segment information and task information, and perform data supplementation based on the target data segment information and the task information.
2. The method according to claim 1, characterized in that, The retrying process for the failed task according to preset retry parameters specifically includes: The preset retry parameters are parsed to determine the preset number of retries and the retry interval. Data is collected based on the retry interval and the failed task to determine the task execution result. If the task execution result is data collection complete, return the data collection result; If the task execution result is a data acquisition failure, record the current retry count, and process the failed task according to the current retry count and the preset retry count.
3. The method according to claim 2, characterized in that, The step of processing the failed task according to the current retry count and the preset retry count specifically includes: Compare the current number of retries with the preset number of retries; If the current number of retries is less than the preset number of retries, return to the execution of data collection based on the retry interval and the failed task; If the current number of retries is greater than or equal to the preset number of retries, the failed task is sent to a preset dead letter queue, and the task is reassigned according to the preset dead letter queue.
4. The method according to claim 1, characterized in that, The step of data supplementation based on the target data segment information and the task information specifically includes: The target checkpoint information is determined by querying the preset database based on the target data segment information. The offset of consumers in the consumer group is adjusted according to the target checkpoint information to determine the target offset, and the task information is sent to the consumer so that the consumer can perform supplementary data collection based on the task information and the target offset.
5. The method according to claim 1, characterized in that, The method further includes: Obtain the current load value and compare it with a preset load balancing threshold; wherein, the current load value includes the load value corresponding to each consumer in the consumer group; If the current load value is less than the preset load balancing threshold, maintain the current state; If the current load value is greater than or equal to the preset balancing threshold, the number of consumers in the consumer group is adjusted.
6. The method according to claim 1, characterized in that, The method further includes: Obtain the current performance indicator value and compare the current performance indicator value with a preset warning threshold; If the current performance indicator value is less than the preset warning threshold, maintain the current state; If the current performance indicator value is greater than or equal to the preset warning threshold, a warning is issued based on the current performance indicator value, and a target consumer is identified based on the current performance indicator value, and resources are adjusted for the target consumer.
7. A data supplementation system, characterized in that, The system includes: The analysis module is used to acquire a set of tasks to be processed, analyze the set of tasks to be processed, and determine the current task type; The processing module is used to retry the failed task according to preset retry parameters if the current task type is a failed task. If the current task type is a data backtracking task, analyze the data backtracking task to determine the target data segment information and task information, and perform data supplementation based on the target data segment information and the task information.
8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.