Data processing method and apparatus

Abstract

The application discloses a data processing method and device, relates to the technical field of data processing, and comprises the following steps: acquiring resource utilization rates of containers occupied by each object; when it is determined that the resource utilization rate of a container occupied by a target object meets a container expansion condition of the target object, performing a first container expansion operation on the target object, performing a classification processing operation on all to-be-processed business data by using a main container corresponding to the target object, processing first-class business data according to a classification result, and processing second-class business data according to the classification result. In the foregoing manner, the maximum resource utilization rate is achieved. Through dynamic expansion of the container level and data classification processing, the system can flexibly adjust the processing capacity according to the actual load, avoids performance bottlenecks caused by resource depletion during a peak period, and eliminates the phenomenon of resource waste during a low-peak period. While ensuring real-time performance, data integrity is also considered.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to data processing methods and equipment. Background Technology

[0002] With the widespread adoption of enterprise cloud-native architectures and AI platforms, monitoring and business data from servers, GPUs, service logs, and API calls are characterized by massive volumes, multiple sources, continuous generation, and significant peak-to-valley fluctuations. Traditional centralized computing and T+1 offline processing can no longer meet the demands for real-time monitoring, anomaly alerts, dynamic scheduling, and multi-dimensional reporting analysis. To cope with the increasing volume of monitoring and user data requiring statistical analysis, a unified data processing system needs to be built to achieve data cleaning, aggregation, computation, and storage, ultimately supporting monitoring visualization, business reporting, cost analysis, and intelligent operations and maintenance.

[0003] The commonly used techniques in related fields are batch processing combined with streaming, splitting data according to its timeliness. This approach is indeed the optimal solution for large clusters, but it becomes inadequate for small clients or medium-sized clusters with fewer than 50 host nodes. Furthermore, for small and medium-sized clusters, the primary goal is to fully utilize limited resources. Summary of the Invention

[0004] This application provides a data processing method and equipment to at least solve the problem in related technologies that traditional batch + streaming data processing methods cannot fully utilize limited resources for small customers or medium-sized clusters.

[0005] This application provides a data processing method, including: Get the resource utilization rate of the system resources occupied by the container corresponding to each object; When it is determined that the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object, the first container expansion operation of the target object is executed. This is used so that the main container corresponding to the target object can perform classification processing operations on all pending business data, and process the first type of business data according to the classification results. The first type of expansion container processes the second type of business data according to the classification results. The first type of expansion container is the expansion container generated according to the first container expansion operation. The target object is any object. The first type of business data is the business data for which the processing results are to be fed back in real time. The second type of business data is the business data for which the processing results do not need to be fed back in real time.

[0006] This application also provides a data processing apparatus, including: The module retrieves the resource utilization rate of the system resources occupied by the container corresponding to each object. The processing module is used to perform a first container expansion operation on the target object when the resource utilization rate of the container corresponding to the target object meets the container expansion conditions of the target object. This is used so that the main container corresponding to the target object can perform classification processing operations on all pending business data, and process the first type of business data according to the classification results. The first type of expansion container processes the second type of business data according to the classification results. The first type of expansion container is an expansion container generated according to the first container expansion operation. The target object is any object. The first type of business data is business data for which processing results are expected to be fed back in real time. The second type of business data is business data for which processing results are not required to be fed back in real time.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0009] This application first obtains the resource utilization rate of the container corresponding to each object. When the resource utilization rate of the container corresponding to the target object meets the container expansion conditions, the first container expansion operation of the target object is executed. In this way, the original main container in the target object can perform categorized operations on all business data. During peak business periods, to ensure the timeliness of real-time business, the main container uses streaming data processing to process business data and implement result feedback. Non-real-time business is smoothly transferred to the expanded container to avoid resource contention and data loss. Through the above methods, resource utilization is maximized. Compared with the two irreconcilable contradictions of insufficient resources during peak periods and idle resources during off-peak periods in the traditional mode, this application, through container-level dynamic expansion and data classification processing, enables the system to flexibly adjust its processing capacity according to the actual load, avoiding performance bottlenecks caused by resource exhaustion during peak periods and eliminating resource waste during off-peak periods. Secondly, this application ensures data integrity while guaranteeing real-time performance. The main container always focuses on real-time business, and the user experience will not be affected by non-real-time tasks. Non-real-time data that might otherwise be discarded or delayed can be properly handled in the scaling container, ensuring a closed loop in the data processing chain.

[0010] This solution is particularly suitable for small to medium-sized clusters. Compared to traditional batch-plus-streaming solutions that require complex architectures, this application does not require a large number of resident stream processing nodes or rely on expensive hardware expansion. It can achieve near-large-scale cluster processing capabilities simply through container-level elastic scheduling, significantly reducing architectural complexity and operational costs. Furthermore, this application significantly improves the system's resilience and stability. When faced with sudden surges in business traffic, the system can automatically distribute data and adjust processing paths to avoid single-point overload. Simultaneously, after business operations stabilize, idle resources can be used to gradually process backlogged tasks, forming a sustainable operational rhythm. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this application; Figure 2 A block diagram illustrating the server hierarchy structure provided in this application embodiment; Figure 3 This is a schematic diagram of another data processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of another data processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of another data processing method provided in an embodiment of this application; Figure 6 This is a schematic diagram of a data processing device structure provided in an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] With the widespread adoption of enterprise cloud-native architectures and AI platforms, monitoring and business data from servers, GPUs, service logs, and API calls are characterized by massive volumes, multiple sources, continuous generation, and significant peak-to-valley fluctuations. Traditional centralized computing and T+1 offline processing can no longer meet the demands for real-time monitoring, anomaly alerts, dynamic scheduling, and multi-dimensional reporting analysis. To cope with the increasing volume of monitoring and user data requiring statistical analysis, a unified data processing system needs to be built to achieve data cleaning, aggregation, computation, and storage, ultimately supporting monitoring visualization, business reporting, cost analysis, and intelligent operations and maintenance.

[0017] Currently, mainstream data processing technologies are divided into batch processing and stream processing. Batch processing divides the data to be processed into batches according to certain rules and then generates results according to specified logic, making it suitable for large data volumes such as TB or PB-level data. Stream processing processes data one at a time, processing data and generating results in real time, making it suitable for small data volumes and scenarios where data display has a time limit.

[0018] In scenarios involving large datasets, processing is typically done via scheduled tasks in the early morning. While this results in late processing times and delayed data display, its advantage lies in centralized processing of large volumes of data, providing accurate and traceable results. Stream processing technology is generally used for log and business data processing, offering timely response and better user feedback. Both methods have their advantages, but there are still some use cases where, for example, a sudden surge in business volume can lead to data loss due to delayed processing by stream processing. Conversely, during periods of low business volume, system resources cannot be fully utilized. Therefore, a method is needed to store data that cannot be processed promptly during peak periods and then process it during off-peak times. This reduces wasted system resources and improves the platform's resilience to data pressure.

[0019] The relevant technologies include batch processing and stream processing. The mainstream batch processing products are Spark and Hadoop MapReduce, and their characteristics are shown in the table below: Table 1

[0020] The mainstream solutions for stream processing are Flink and Kafka Streams, and their characteristics are shown in the table below: Table 2

[0021] The table above clearly shows the advantages and disadvantages of different products in data stream processing. However, it overlooks the pain points of small and medium-sized clusters: wasted system resources during idle periods and insufficient system processing capacity during busy periods.

[0022] Based on analysis of other AI platform products in the current market, users' goals are for the platform to ensure the completion, timeliness, and traceability of customer business operations. The main objective of clustering is to be compatible with different customer groups, rationally plan limited system resources, and ensure smooth and timely user business operations.

[0023] The commonly used techniques in related fields are batch processing combined with streaming, splitting data according to its timeliness. This approach is indeed the optimal solution for large clusters, but it becomes inadequate for small clients or medium-sized clusters with fewer than 50 host nodes. Furthermore, for small and medium-sized clusters, the primary goal is to fully utilize limited resources.

[0024] To address the aforementioned problems, embodiments of this application provide a data processing method, as detailed below. Figure 1 As shown, the method includes the following steps, which are executed by the server, specifically by the mirror acquisition unit within the server. See details... Figure 2 As shown, the server hierarchy is displayed as follows, including: The User Interface (UI) interfaces with user pages, and all business data is transmitted to various service modules via the gateway. The basic management unit (base module) primarily provides users with the basic functions required to use the platform, such as user creation and quota modification. The resource scheduling unit (resource module) supports user-submitted business requests. The monitoring and data collection unit (monitor module) provides support for monitoring and data collection items. The system configuration unit (system module) provides global configuration support for the platform. All business database information is stored in this database. The time-series database is used for collecting underlying resource information and displaying it according to time periods.

[0025] Server platforms, such as AI server platforms, typically deploy base, resource, monitor, and system modules. These modules consume fixed system resources during runtime. In a specific example, system resources include CPU resources and memory; in this embodiment, CPU utilization is primarily used as a reference benchmark.

[0026] Specifically, the method steps of this application embodiment are as follows: Step S101: Obtain the resource utilization rate of the system resources occupied by the container corresponding to each object.

[0027] Specifically, the object here refers to, for example Figure 2 The module includes the base module, resource module, monitor module, and system module.

[0028] Step S102: When it is determined that the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object, the first container expansion operation of the target object is executed.

[0029] Specifically, the monitor module monitors the CPU resource utilization of containers within each object. When the CPU resource utilization of a container in any module meets the container capacity condition corresponding to that target object, a first container expansion operation is performed for that target object. This expansion is then used by the main container corresponding to the target object to perform classification processing operations on all pending business data, and to process the first type of business data based on the classification results. The first-type expanded container processes the second type of business data based on the classification results. The first-type expanded container is an expanded container generated based on the first container expansion operation, and the number of first-type expanded containers can be at least one. The target object is any object, the first type of business data is business data for which real-time processing results are expected, and the second type of business data is business data for which real-time processing results are not required.

[0030] In an optional example, the container resizing condition for all objects can also be set to the same value.

[0031] The main container performs the classification and processing of all pending business data. All pending business data is divided into data requiring real-time feedback and data requiring non-real-time feedback. The main container should use stream processing to process business data requiring real-time feedback, facilitating timely feedback to the requesting client. The first type of expansion container, however, can use batch processing to process business data requiring non-real-time feedback, enabling centralized, large-scale processing with accurate and traceable results.

[0032] This application provides a data processing method that first obtains the resource utilization rate of the container corresponding to each object. When it is determined that the resource utilization rate of the container corresponding to the target object meets the container expansion conditions, a first container expansion operation is performed on the target object. In this way, the original main container in the target object can perform categorized operations on all business data. During peak business periods, to ensure the timeliness of real-time business, the main container uses streaming data processing to realize business data processing and implementation result feedback. Non-real-time business is smoothly transferred to the expanded container to avoid resource contention and data loss. Through the above method, resource utilization is maximized. Compared with the two irreconcilable contradictions of insufficient resources during peak periods and idle resources during off-peak periods in the traditional mode, this application, through container-level dynamic expansion and data classification processing, enables the system to flexibly adjust its processing capacity according to the actual load, avoiding performance bottlenecks caused by resource exhaustion during peak periods and eliminating resource waste during off-peak periods. Secondly, this application ensures both real-time performance and data integrity. The main container always focuses on real-time business, and the user experience will not be affected by non-real-time tasks. Non-real-time data that might otherwise be discarded or delayed can be properly handled in the scaling container, ensuring a closed loop in the data processing chain.

[0033] This solution is particularly suitable for small to medium-sized clusters. Compared to traditional batch-plus-streaming solutions that require complex architectures, this application does not require a large number of resident stream processing nodes or rely on expensive hardware expansion. It can achieve near-large-scale cluster processing capabilities simply through container-level elastic scheduling, significantly reducing architectural complexity and operational costs. Furthermore, this application significantly improves the system's resilience and stability. When faced with sudden surges in business traffic, the system can automatically distribute data and adjust processing paths to avoid single-point overload. Simultaneously, after business operations stabilize, idle resources can be used to gradually process backlogged tasks, forming a sustainable operational rhythm.

[0034] In an optional example, based on the foregoing embodiments, determining whether the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions includes: When the system resource utilization rate of the container corresponding to the target object is greater than the first preset utilization rate threshold, less than or equal to the second preset utilization rate threshold, and the duration is greater than or equal to the preset duration, the container expansion condition is determined to be met.

[0035] Taking the resource module as an example, the first preset utilization threshold is set to 60%, and the second preset utilization threshold is set to 80%. When the CPU resource utilization of the container in the resource module is greater than 60% and less than or equal to 80%, and the duration is greater than or equal to 10 minutes, the container expansion condition is determined to be met, and the monitor module executes the first container expansion operation on the resource module. After the container expansion operation is executed and the first type of expanded container is generated, the main container of the resource module performs all pending business data classification processing operations, and processes the first type of business data according to the classification results. The first type of expanded container processes the second type of business data according to the classification results.

[0036] In an optional example, to reduce database or disk reads and improve data reading efficiency, the second type of business data can be loaded into memory for processing.

[0037] Further optionally, to avoid system performance bottlenecks caused by uncontrolled container expansion operations, the method may also include the following steps: When the resource utilization rate of the system resources occupied by the container corresponding to the target object is greater than or equal to the third preset utilization rate threshold, all second-type business data in memory is stored in a pre-built database temporary table. This table is used to continue processing when the resource utilization rate of the system resources occupied by the container corresponding to the target object is less than or equal to the first preset utilization rate threshold. The third preset utilization rate threshold is greater than the second preset utilization rate threshold.

[0038] Furthermore, when the resource utilization rate of the system resources occupied by the container corresponding to the target object is greater than or equal to a third preset utilization threshold, the method also includes: Delete the first type of expanded container and reclaim the resources occupied by the first type of expanded container.

[0039] Specifically, the third preset utilization threshold is, for example, 90%. When the utilization rate of the container resources corresponding to the target object reaches or exceeds this third preset utilization threshold, the system will no longer attempt to forcibly process the second type of business data under the current high load state. Instead, it will immediately store all the second type of business data in memory into a pre-built temporary database table, thereby physically cutting off the continuous occupation of scarce computing resources by non-critical business.

[0040] Under high load, the system rapidly releases memory and processing power, focusing on ensuring the real-time processing of the first type of business data and avoiding overall system performance degradation or even service unavailability due to memory accumulation and resource contention. Simultaneously, by persisting the second type of business data to a temporary table, the system ensures data integrity and security under high pressure, preventing the risk of data loss caused by excessive load in traditional memory processing models.

[0041] More importantly, when the container resource utilization rate falls below the first preset utilization rate threshold, the system can automatically read from the database temporary table and continue to process these temporary data, so that non-real-time services that were forced to be interrupted during the high load phase can be executed in an orderly manner during the low load phase.

[0042] In this way, this application not only achieves dynamic resource balance during high and low load phases, but also forms a closed-loop processing mechanism of "high-voltage temporary storage - low-voltage continuity", which not only ensures the real-time performance and stability of critical business operations, but also makes full use of the system's idle processing capacity, significantly improving overall resource utilization efficiency and system reliability.

[0043] Furthermore, when the utilization rate of the container resources corresponding to the target object continues to rise and reaches or even exceeds the third preset utilization rate threshold, it indicates that the current system is no longer in the normal elastic expansion range, but is approaching the physical resource carrying capacity limit. If the operation of the first type of expanded containers is maintained at this time, it will not only fail to significantly improve the overall processing capacity, but will also bring additional system overhead due to container scheduling, memory occupation and process switching, further intensifying resource competition, and even triggering a cascading failure effect. To this end, this application actively deletes the first type of expanded containers and reclaims the computing and memory resources they occupy when the resource utilization rate reaches the third preset utilization rate threshold. This allows the system to quickly release the resources occupied by non-core tasks under extreme high pressure, freeing up critical computing space for the main container to process the first type of business data, thereby ensuring the continuity and stability of real-time business. At the same time, by promptly cleaning up the first type of expanded containers that no longer have positive benefits, resource fragmentation and scheduling jitter caused by ineffective expansion are avoided, effectively preventing the system from falling into a performance bottleneck due to over-expansion.

[0044] When resources are scarce, core business operations are prioritized. After resources are restored, non-real-time business operations are processed using temporary data stored in temporary tables in the database. This maximizes resource utilization and minimizes risk while ensuring the overall stability of the system.

[0045] In an optional example, the expansion operation of the first container includes the following method steps, see details below. Figure 3 As shown, the method includes: Step S301: Determine the container type of the first type of expansion container based on the container type of the main container.

[0046] Step S302: Based on the time when the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object, backtrack the total amount of second type of business data received by the main container within a preset time period.

[0047] Step S303: Determine the expansion quantity of the first type of expansion container based on the unit processing business data volume of the main container, the preset time period, and the total data volume.

[0048] Step S304: Determine the first actual expansion quantity based on the first quasi-expansion quantity, the resource utilization rate of system resources occupied by each first type of expansion container, the resource utilization rate of system resources occupied by the container corresponding to the target object, and the second preset utilization rate threshold.

[0049] Specifically, the container type of the first type of expansion container must be the same as that of the main container. The specification for the amount of business data processed per unit time by the first type of expansion container must also be the same as that of the main container.

[0050] The specific expansion amount needs to be determined by considering the total amount of second-type business data received per unit time and the unit data processing capacity of the main container. Specifically, it involves considering the difference between the total amount of data received per unit time and the unit data processing capacity of the main container, as well as the unit data processing capacity of the main container, to determine the initial quasi-expansion amount.

[0051] The total amount of data received per unit time can be determined by statistically analyzing the time when the resource utilization rate of the container corresponding to the target object meets the container expansion conditions of the target object, and by backtracking the total amount of the second type of business data received by the main container within a preset time period, as well as the preset time period, to determine the average amount of data received per unit time.

[0052] In a more specific example, the first quasi-expansion quantity of the first type of expansion container is determined based on the unit processing business data volume of the main container, the preset time period, and the total data volume, as shown in the following expression:

[0053] Where Z represents the first quasi-expansion quantity of the first type of expansion container, Q represents the total data volume, T represents the preset time period, and Y represents the unit processing business data volume of the main container. · This indicates the rounding up operation.

[0054] The reason Y is multiplied by 2 / 3 in this formula is to ensure that each expansion container has a certain margin. Assume each expansion container only uses approximately 2... 3. Theoretical processing capacity. A safety margin is reserved for the actual processing capacity of each single container to prevent the expanded containers from being filled immediately after creation, which would trigger another expansion and cause expansion jitter.

[0055] Because this application's solution considers not only the number of expanded containers, but also whether the resource utilization rate of the expanded containers exceeds the upper limit, i.e., the second preset utilization rate threshold, it is necessary to determine the first actual expansion quantity based on the first quasi-expansion quantity, the resource utilization rate of each first-type expansion container, the resource utilization rate of the container corresponding to the target object, and the second preset utilization rate threshold.

[0056] To give a concrete example, suppose the current system resource utilization rate of the container corresponding to the target object is 65%, and the second preset utilization threshold is 80%. The first quasi-expansion data volume calculated using the aforementioned method is 5. Since each of these first-type expansion containers occupies 5% of the system resources, the maximum actual expansion quantity is 3.

[0057] Alternatively, if the first quasi-expansion data volume is 2, then the actual database capacity is 2. That is, the final determined first actual expansion quantity is the minimum value between the first quasi-expansion data volume and the expansion data volume calculated based on the resource utilization rate of the system resources occupied by the container corresponding to the current target object, the second preset utilization rate threshold, and the resource utilization rate of the system resources occupied by each first type of expansion container.

[0058] Optionally, considering that in the above example, the first container expansion operation may not be the first time, but rather an expansion operation may have been performed previously, the method, after determining the expansion quantity of the first type of expansion container based on the unit processing business data volume of the main container, the preset time period, and the total data volume, further includes the following method steps: Step a1: Detect the number of sub-containers of the first type that have been expanded before the time when the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object. Step a2: Adjust the first actual expansion quantity based on the number of sub-quantities.

[0059] Specifically, firstly, the number of sub-containers of the first type that have already been expanded is detected before the time when the resource utilization rate of the container corresponding to the target object meets the container expansion conditions of the target object. Then, based on the expansion quantity and the sub-quantity, the actual number of containers to be expanded this time can be determined.

[0060] Optionally, based on any of the foregoing embodiments, considering that business loads often have a certain periodicity and predictability, relying solely on real-time monitoring results to trigger expansion operations may still result in expansion delays or momentary fluctuations. Therefore, when it is determined that the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object, before executing the first container expansion operation on the target object, the method may further include the following method steps, as detailed in [link to details]. Figure 4 As shown, it includes: Step S401: Obtain the historical resource utilization rate of the system resources occupied by the container corresponding to the target object at each historical moment within the historical time period.

[0061] For example, you could retrieve historical resource utilization data of the system resources used by the container corresponding to the target object within the 24 hours prior to the current time.

[0062] Step S402: Based on the historical resource utilization rate of the system resources occupied by the container corresponding to the target object at each historical moment, predict the target moment when the resource utilization rate of the system resources occupied by the container corresponding to the target object in the future meets the container expansion conditions.

[0063] Specifically, based on the historical resource utilization rate of the system resources occupied by the container corresponding to the target object at each historical moment, it is possible to assess which times in the next 24 hours are idle and busy periods. During potentially busy periods, there may be times when the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container scaling conditions. In practical applications, the moment when the historical resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container scaling conditions within a historical time period can be used as the target moment.

[0064] Step S403: Execute a second container expansion operation on the target object within a unit time period before the target time to obtain a second type of expanded container.

[0065] Specifically, the second container expansion operation for the target object will begin 5 minutes before the target time in the future, and a second type of expanded container will be obtained.

[0066] It should be noted that although this application mentions both Type I and Type II expansion containers, this is merely to distinguish the different times of expansion. Functionally, they are not significantly different; both are designed to process business data and improve its processing efficiency.

[0067] Furthermore, the specifications of the second type of expansion container can be the same as the main container, and the expansion quantity can be set based on historical experience. Of course, it can also be set to the default of 1 initially.

[0068] This method is not limited to the current resource status but models and analyzes system behavior from a time perspective. By acquiring the historical resource utilization rate of the container corresponding to the target object at each historical moment within a historical time period, the system can predict the container resource utilization rate of the target object at a future moment based on this historical data, and determine the corresponding target moment in advance before it is predicted that the container expansion conditions will be met. Compared to the traditional solution that waits until the resource utilization rate actually reaches the expansion threshold before creating containers, this embodiment performs the second container expansion operation within a unit time period before the target moment, generating a second type of expansion container, so that the new computing capacity is ready before the actual business pressure arrives. This effectively eliminates the lag in expansion operations. In the early stages of a sudden increase in business traffic, the system is no longer in a passive response state, thus avoiding service jitter or request loss caused by instantaneous high concurrency.

[0069] Based on any of the foregoing embodiments, after determining that the business data in any expanded container has been processed, the method may further include the following method steps, as detailed below. Figure 5 As shown, it includes: Step S501: Identify the creation time of the expanded container that has completed processing business data.

[0070] Step S502: Determine the category of the expanded container that has completed processing business data based on the creation time.

[0071] The categories include either Category I expansion containers or Category II expansion containers.

[0072] Step S503: When it is determined that the category belongs to the first type of expansion container, the expansion container that has completed the processing of business data is directly deleted.

[0073] Specifically, determining the creation time of the expansion container involves assessing whether its creation occurred before or after the point at which the resource utilization of the target object's corresponding container meets the expansion conditions. This allows us to determine the category of the expansion container that has already processed business data. The category includes either Category 1 or Category 2 expansion containers.

[0074] When the container is identified as belonging to the first category of expansion containers, the expansion containers that have already processed business data are directly deleted.

[0075] Alternatively, the method may further include the following method steps: Step b1: When the category is determined to be a second-class expansion container, control the second-class expansion container to enter a hibernation state and start timing.

[0076] Step b2: After the timeout period reaches the preset time period and it is confirmed that the second type of expansion container has not been restarted, delete the expansion container that has completed the processing of business data.

[0077] In this embodiment, the reason for immediately recycling the first type of expansion container after processing business data is that the first type of expansion container is a responsive expansion container created when resource utilization actually meets the expansion conditions, typically used to handle ongoing high loads. Once the high load phase has passed, it can be directly deleted to avoid unnecessary resource consumption.

[0078] For the second type of expansion containers, they are not directly deleted, but rather placed into a dormant state. They are only deleted if they remain dormant after a preset time period, thus reclaiming system resources. This is primarily because the second type of expansion containers are predictive expansion containers created in advance before the target time is predicted, ensuring they are ready before business pressure arrives. To avoid being caught off guard during the next peak, they can be temporarily retained. They are placed into a dormant state and a timer is started; only when the dormant period reaches the preset time and they are not reactivated during this period (i.e., no further high load trend occurs) are the containers finally reclaimed.

[0079] Further optionally, within a unit time period prior to the target time, a second container expansion operation on the target object is performed. Before obtaining the second type of expanded container, the method further includes: Determine whether the function to resize the second container of the target object is enabled. Specifically, if the function to resize the second container of the target object is enabled, the pre-resize operation on the target object's container will be performed.

[0080] This approach avoids the ineffective loss of predicted expansion value. If the second type of expansion containers are destroyed after a brief period of idle time, the system will be forced to recreate the containers when the prediction is basically accurate and a business peak follows, resulting in unnecessary startup delays and instantaneous resource contention. By allowing the second type of expansion containers to enter a dormant state after completing their tasks and remain for a preset period of time, existing containers can be directly reused during high-probability continuous or quasi-periodic business fluctuations, significantly shortening the response chain and improving the system's ability to handle sudden traffic surges.

[0081] Secondly, it prevents the meaningless occupation of resources. The first type of expansion containers that have truly completed their mission are directly recycled, ensuring that the system releases resources quickly after the load decreases; while the second type of expansion containers are only retained for a limited time window, and are recycled if they are not reused after the window expires. This achieves a reasonable balance between "rapid reuse" and "resource conservation", which is especially suitable for small and medium-sized cluster environments with a limited number of nodes.

[0082] Furthermore, containers are categorized into reactive scaling and predictive scaling, and respectively bound to "immediate reclamation" and "delayed reclamation / hibernation" strategies. This elevates the system's elastic scaling behavior from being driven by a single threshold to a comprehensive decision-making process that integrates time dimensions, load trends, and container origins, reducing scheduling jitter caused by blind scaling and frequent creation and destruction.

[0083] Further optionally, as described above, because in some cases, some of the second type of business data may be stored in a temporary database table for later processing, the method may also include the following steps: When it is determined that there is second type of business data in the database temporary table, and the resource utilization rate of the system resources occupied by the container corresponding to the target object is less than or equal to the first preset utilization rate threshold, it is determined to execute the third container expansion operation, obtain the third type of expansion container, and use it to process the second type of business data in the database temporary table.

[0084] Specifically, when it is determined that the database temporary table contains second-type business data, but the resource utilization rate of the container corresponding to the target object is less than or equal to 60%, a third container expansion operation can be performed to obtain a third-type expanded container to process the second-type business data in the database temporary table.

[0085] Alternatively, the method may further include the following method steps: Step c1: Calculate the total amount of the second type of business data in the temporary database table.

[0086] Step c2: Based on the total amount of the second type of business data in the temporary database table and the amount of business data processed per unit time corresponding to the third type of expansion container, determine the second quasi-expansion quantity of the third type of expansion container to be expanded.

[0087] Step c3: Based on the resource utilization rate of the third type of expansion container occupying system resources, the resource utilization rate of the container corresponding to the target object occupying system resources, and the second preset utilization rate threshold, determine the third quasi-expansion quantity of the third type of expansion container to be expanded. Step c4: Determine the second actual expansion quantity of the third type of expansion container based on the second quasi-expansion quantity and the third quasi-expansion quantity.

[0088] Specifically, when performing a third container expansion operation, it is necessary to determine the second actual expansion quantity corresponding to the third type of expansion container to be expanded.

[0089] In this embodiment of the application, firstly, based on the total amount of the second type of business data in the temporary database table and the amount of business data processed per unit time corresponding to the third type of expansion container, the second quasi-expansion quantity of the third type of expansion container to be expanded is determined.

[0090] For example, see the following expression:

[0091] Where A represents the second quasi-expansion quantity, S represents the total amount of the second type of business data in the database temporary table, and h is the amount of business data processed per unit time for the third type of expansion container (in units of rows / s). · This indicates a rounding up operation. 300 represents 300 seconds, which means that in this application, the second type of business data in the database temporary table is expected to be processed within 5 minutes. Therefore, it is multiplied by a coefficient of 300.

[0092] For considerations similar to those in the aforementioned embodiments, this application embodiment further includes determining the third quasi-expansion quantity of the third type of expansion container to be expanded based on the resource utilization rate of the system resources occupied by the third type of expansion container, the resource utilization rate of the system resources occupied by the container corresponding to the target object, and the second preset utilization rate threshold.

[0093] The specific expression is:

[0094]

[0095] Where B represents the third quasi-expansion quantity, and the second preset utilization threshold is assumed to be 80%. · This represents the floor operation.

[0096] Then, the second actual expansion quantity = min[A, B].

[0097] In this method, the aforementioned approach enables the orderly and controllable processing of backlogged data using idle resources. The minimum number of processing units required is calculated based on the total amount of second-type business data actually existing in the database temporary tables. This allows the system to fully utilize idle computing power during low-load periods to accelerate the subsequent processing of non-real-time business data, shortening the overall data loop latency and avoiding resource waste caused by blind expansion. It also ensures the system's ability to quickly recover from subsequent business fluctuations. By introducing the current container resource utilization rate and a second preset utilization rate threshold as upper limits in the expansion decision, even under low load conditions, the system will not allocate all remaining resources to third-type expansion containers, thus reserving buffer space for the upcoming business recovery and reducing the risk of expansion delays or resource contention.

[0098] Optionally, when the system is under low load and a third type of expanded container is generated to process the second type of business data in the temporary table, if business requests increase again at this time, the resource utilization of the main container will gradually increase. If the third type of expanded container continues to run, there may be resource contention, such as the third type of expanded container competing with the main container for CPU, memory, and I / O, weakening the main container's ability to process the first type of business data; priority inversion, such as non-real-time business that should be delayed being used to consume resources from real-time business; and uncontrolled expansion, such as the system maintaining a large number of such containers during high load periods, leading to resource fragmentation and other related problems.

[0099] To avoid the above situation, the method may further include: When it is determined that the resource utilization rate of the main container occupies system resources is greater than the first preset utilization rate threshold, after the third type of expansion container has finished processing the current business data, the processing of business data is stopped, the third type of expansion container is deleted, and the resources occupied by the third type of expansion container are reclaimed.

[0100] Specifically, when the main container's resource utilization exceeds a first preset utilization threshold, the system immediately determines that the third type of expanded container has completed its phased task. After ensuring that it has finished processing the currently executing business data, no new tasks are assigned, and the container is directly deleted after processing, reclaiming all the computing resources it occupied. This method ensures that the third type of expanded container remains within a controllable and predictable lifecycle, preventing it from evolving into a long-running, persistent load.

[0101] This method ensures resource priority for real-time services during high-load periods. The third type of expansion container is essentially designed to accelerate the processing of non-real-time business data during low-load periods. When the main container's resource utilization exceeds the first preset utilization threshold, it indicates that the system is no longer in a resource-sufficient state. At this point, by immediately stopping and deleting the third type of expansion container, non-critical services can avoid competing with critical services for scarce computing resources, ensuring that the real-time feedback capability of the first type of business data is not affected.

[0102] Secondly, this approach also avoids the disorderly spread and fragmentation of expansion activities. Without this recycling mechanism, the third type of expansion container might persist for a long time after business recovers, resulting in the simultaneous existence of the main container, the first type of expansion container, and the third type of expansion container in the system, creating a complex resource competition relationship. By strictly binding the lifecycle of this type of container to the range of "resource utilization rate ≤ first preset utilization rate threshold", the resource allocation of the system at any time has clear predictability and controllability.

[0103] Secondly, during periods of low load, the third type of expansion container fully utilizes idle resources to digest temporary table data, while during periods of high load, this mechanism quickly releases these resources to return to the main business. This dynamically balances real-time performance and resource utilization in the cycle of "high-voltage protection - low-voltage compensation - high-voltage recovery". This approach is particularly suitable for one of the application scenarios of this application, namely, small and medium-sized cluster environments with limited node size and high sensitivity to resource scheduling.

[0104] Alternatively, the method may further categorize the second type of business data in the database temporary table. For example, it can be divided into hot data and cold data based on the generation time and / or business importance of the second type of business data.

[0105] When a third type of expanded container starts up, it prioritizes processing hot data. Cold data is only allowed to be accessed and processed after the hot data has been cleared.

[0106] This approach can prevent low-value historical data from "blocking" the flow of high-value recent data.

[0107] Further optionally, considering that the second type of expansion containers will occupy system resources once created, but if the prediction is wrong, these containers will waste system resources. Therefore, the method may also include: Step 1: Build a logical container pool.

[0108] Specifically, a batch of "logical containers" are pre-registered in the scheduler. These containers are not bound to the actual CPU / memory and are only used as placeholders.

[0109] Step 2: Realize the logical container pool as needed.

[0110] Specifically, when business data is predicted to be available at a target time, instead of creating a container directly, a logical container is activated.

[0111] The system prioritizes allocating idle physical resources to these "activated logical containers" and then converts them into physical containers.

[0112] The above approach significantly reduces the cold start latency of scaling operations and improves the actual performance of predictive scaling. Since the logical containers have already completed registration and initialization preparation in the scheduler, when a business peak is predicted, the system only needs to perform resource binding and lightweight startup operations, without waiting for image retrieval and a complete initialization process. This reduces the scaling response time from seconds to sub-seconds, ensuring that the second type of scaling containers can have processing capabilities at the beginning of business pressure and avoiding request backlog caused by startup delays.

[0113] Moreover, the above method can eliminate the risk of resource waste caused by prediction failure. In the aforementioned prediction scaling scheme, once the prediction is inaccurate, the created entity containers will continue to occupy memory and CPU even if they are idle, which is particularly noticeable in small and medium-sized clusters. In this method, however, the logical containers do not consume any resources before they are materialized. Even if the prediction fails to hit the actual business peak, the system only maintains a small amount of metadata overhead, thereby minimizing the cost of misjudgment while ensuring foresight.

[0114] Optionally, the method may further include generating log data corresponding to the expanded container, the log data including the generation and recycling processes of the expanded container. This log data can help predict the target time when the resource utilization rate of the container corresponding to the future target object meets the container expansion conditions, thereby assisting in executing the second container expansion operation on the target object and obtaining the second type of expanded container.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0116] Embodiments of this application also provide a data processing apparatus, see details below. Figure 6 As shown, the device includes an acquisition module 601 and a processing module 602.

[0117] Get module 601 to get the resource utilization rate of the system resources occupied by the container corresponding to each object; The processing module 602 is used to perform a first container expansion operation on the target object when the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object. This is used so that the main container corresponding to the target object can perform classification processing operations on all pending business data, and process the first type of business data according to the classification results. The first type of expansion container processes the second type of business data according to the classification results. The first type of expansion container is an expansion container generated according to the first container expansion operation. The target object is any object. The first type of business data is business data for which processing results are expected to be fed back in real time. The second type of business data is business data for which processing results are not required to be fed back in real time.

[0118] In an optional example, processing module 602 is specifically used for: When the system resource utilization rate of the container corresponding to the target object is greater than the first preset utilization rate threshold, less than or equal to the second preset utilization rate threshold, and the duration is greater than or equal to the preset duration, the container expansion condition is determined to be met.

[0119] In an optional example, processing module 602 is also used to load the second type of business data into memory for processing.

[0120] In an optional example, the processing module 602 is further configured to store all second-type business data in memory into a pre-built database temporary table when the resource utilization rate of the system resources occupied by the container corresponding to the target object is greater than or equal to a third preset utilization rate threshold, so as to continue processing when the resource utilization rate of the system resources occupied by the container corresponding to the target object is less than or equal to a first preset utilization rate threshold, wherein the third preset utilization rate threshold is greater than the second preset utilization rate threshold.

[0121] In an optional example, the processing module 602 is further configured to delete the first type of expansion container and reclaim the resources occupied by the first type of expansion container when the resource utilization rate of the system resources occupied by the container corresponding to the target object is greater than or equal to a third preset utilization rate threshold.

[0122] In an optional example, the first container expansion operation performed by processing module 602 includes: The container type of the first type of expansion container is determined based on the container type of the main container; Based on the time when the resource utilization rate of the container corresponding to the target object meets the container expansion conditions of the target object, the total amount of second-type business data received by the main container within a preset time period is traced back. The first quasi-expansion quantity of the first type of expansion container is determined based on the unit processing business data volume, preset time period, and total data volume of the main container. The first actual expansion quantity is determined based on the first quasi-expansion quantity, the resource utilization rate of system resources occupied by each first type of expansion container, the resource utilization rate of system resources occupied by the container corresponding to the target object, and the second preset utilization rate threshold.

[0123] In an optional example, processing module 602 determines the first quasi-expansion quantity of the first type of expansion container based on the unit processing business data volume of the main container, the preset time period, and the total data volume, as shown in the following expression:

[0124] Where Z represents the first quasi-expansion quantity of the first type of expansion container, Q represents the total data volume, T represents the preset time period, and Y represents the unit processing business data volume of the main container. · This indicates the rounding up operation.

[0125] In an optional example, the device further includes a detection module 603 and a correction module 604; The detection module 603 is used to detect the number of sub-containers of the first type that have been expanded before the time when the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object is determined. The correction module 604 is used to correct the first actual expansion quantity based on the number of sub-quantities.

[0126] In an optional example, module 601 is also used to obtain the historical resource utilization rate of the system resources occupied by the container corresponding to the target object at each historical moment within the historical time period; The processing module 602 is also used to predict the target time when the resource utilization rate of the system resources occupied by the container corresponding to the target object in the future meets the container expansion conditions based on the historical resource utilization rate of the system resources occupied by the container corresponding to the target object at each historical time; and to perform a second container expansion operation on the target object within a unit time period before the target time to obtain a second type of expanded container.

[0127] In an optional example, the processing module 602 is further configured to identify the creation time of the expanded container that has completed processing business data; determine the category to which the expanded container that has completed processing business data belongs based on the creation time, wherein the category includes a first-class expanded container or a second-class expanded container; when the category is determined to be a first-class expanded container, the expanded container that has completed processing business data is directly deleted.

[0128] In an optional example, the processing module 602 is further configured to control the second type of expansion container to enter a dormant state and start timing when it is determined that the second type of expansion container belongs to the second type of expansion container; after the timing time reaches the preset time period and it is determined that the second type of expansion container has not been restarted, the expansion container that has processed the business data is deleted.

[0129] In an optional example, the processing module 602 is further configured to determine to perform a third container expansion operation and obtain a third type of expansion container when it is determined that there is second type of business data in the database temporary table and the resource utilization rate of the container corresponding to the target object is less than or equal to a first preset utilization rate threshold, so as to process the second type of business data in the database temporary table.

[0130] In an optional example, the processing module 602 is further configured to: count the total amount of the second type of business data in the database temporary table; determine the second quasi-expansion quantity of the third type of expansion container to be expanded based on the total amount of the second type of business data in the database temporary table and the amount of business data processed per unit time corresponding to the third type of expansion container; determine the third quasi-expansion quantity of the third type of expansion container to be expanded based on the resource utilization rate of the third type of expansion container occupying system resources, the resource utilization rate of the container corresponding to the target object occupying system resources, and the second preset utilization rate threshold; and determine the second actual expansion quantity of the third type of expansion container based on the second quasi-expansion quantity and the third quasi-expansion quantity.

[0131] In an optional example, the processing module 602 is further configured to, when it is determined that the resource utilization rate of the main container occupying system resources is greater than a first preset utilization rate threshold, stop processing business data, delete the third type of expansion container, and reclaim the resources occupied by the third type of expansion container after determining that the third type of expansion container has finished processing the current business data.

[0132] The description of the features of the data processing apparatus provided in this application can be found in the relevant description of the data processing method, which will not be repeated here.

[0133] This application provides a data processing apparatus that first obtains the resource utilization rate of the container corresponding to each object. When it is determined that the resource utilization rate of the container corresponding to the target object meets the container expansion conditions, a first container expansion operation is performed on the target object. In this way, the original main container in the target object can perform classification operations on all business data. During peak business periods, to ensure the timeliness of real-time business, the main container uses streaming data processing to realize business data processing and implementation result feedback. Non-real-time business is smoothly transferred to the expanded container to avoid resource contention and data loss. Through the above method, resource utilization is maximized. Compared with the two irreconcilable contradictions of insufficient resources during peak periods and idle resources during off-peak periods in the traditional mode, this application, through container-level dynamic expansion and data classification processing, enables the system to flexibly adjust its processing capacity according to the actual load, avoiding performance bottlenecks caused by resource exhaustion during peak periods and eliminating resource waste during off-peak periods. Secondly, this application ensures both real-time performance and data integrity. The main container always focuses on real-time business, and the user experience will not be affected by non-real-time tasks. Non-real-time data that might otherwise be discarded or delayed can be properly handled in the scaling container, ensuring a closed loop in the data processing chain.

[0134] This solution is particularly suitable for small to medium-sized clusters. Compared to traditional batch-plus-streaming solutions that require complex architectures, this application does not require a large number of resident stream processing nodes or rely on expensive hardware expansion. It can achieve near-large-scale cluster processing capabilities simply through container-level elastic scheduling, significantly reducing architectural complexity and operational costs. Furthermore, this application significantly improves the system's resilience and stability. When faced with sudden surges in business traffic, the system can automatically distribute data and adjust processing paths to avoid single-point overload. Simultaneously, after business operations stabilize, idle resources can be used to gradually process backlogged tasks, forming a sustainable operational rhythm.

[0135] Embodiments of this application also provide an electronic device, such as... Figure 7 As shown, it includes a memory 10 and a processor 20. The memory 10 stores a computer program, and the processor 20 is configured to run the computer program to perform the steps in any of the above-described data processing method embodiments.

[0136] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data processing method embodiments or the steps in any of the above-described data reading method embodiments when it is run.

[0137] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0138] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0139] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data reading method embodiments.

[0140] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0141] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] The data processing method and apparatus provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized in that, The method includes: Get the resource utilization rate of the system resources occupied by the container corresponding to each object; When it is determined that the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object, the first container expansion operation of the target object is executed. This is used so that the main container corresponding to the target object can subsequently perform classification processing operations on all pending business data, and process the first type of business data according to the classification results. The first type of expanded container processes the second type of business data according to the classification results. The first type of expanded container is an expanded container generated according to the first container expansion operation. The target object is any object. The first type of business data is business data for which processing results are expected to be fed back in real time. The second type of business data is business data for which processing results are not required to be fed back in real time.

2. The method according to claim 1, characterized in that, Determining whether the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions includes: When the resource utilization rate of the system resources occupied by the container corresponding to the target object is greater than the first preset utilization rate threshold, less than or equal to the second preset utilization rate threshold, and the duration is greater than or equal to the preset duration, it is determined that the container expansion condition is met.

3. The method according to claim 2, characterized in that, The method further includes: The second type of business data is loaded into memory for processing.

4. The method according to claim 3, characterized in that, The method further includes: When the resource utilization rate of the system resources occupied by the container corresponding to the target object is greater than or equal to the third preset utilization rate threshold, all second-type business data in memory is stored in a pre-built database temporary table, so that when the resource utilization rate of the system resources occupied by the container corresponding to the target object is less than or equal to the first preset utilization rate threshold, processing continues, wherein the third preset utilization rate threshold is greater than the second preset utilization rate threshold.

5. The method according to claim 4, characterized in that, When the resource utilization rate of the system resources occupied by the container corresponding to the target object is greater than or equal to a third preset utilization rate threshold, the method further includes: Delete the first type of expansion container and reclaim the resources occupied by the first type of expansion container.

6. The method according to any one of claims 2-5, characterized in that, The first container expansion operation includes: The container type of the first type of expansion container is determined based on the container type of the main container; Based on the time when the resource utilization rate of the container corresponding to the target object meets the container expansion conditions of the target object, the total amount of second type of business data received by the main container within a preset time period is traced back. The first quasi-expansion quantity of the first type of expansion container is determined based on the unit processing business data volume of the main container, the preset time period, and the total data volume. The first actual expansion quantity is determined based on the first quasi-expansion quantity, the resource utilization rate of system resources occupied by each of the first type of expansion containers, the resource utilization rate of system resources occupied by the container corresponding to the target object, and the second preset utilization rate threshold.

7. The method according to claim 6, characterized in that, The first quasi-expansion quantity of the first type of expansion container is determined based on the unit processing business data volume of the main container, the preset time period, and the total data volume, as shown in the following expression: Where Z represents the first quasi-expansion quantity of the first type of expansion container, Q represents the total data volume, T represents the preset time period, and Y represents the unit processing business data volume of the main container. · This indicates the rounding up operation.

8. The method according to claim 6, characterized in that, After determining the expansion quantity of the first type of expansion container based on the unit processing business data volume of the main container, the preset time period, and the total data volume, the method further includes: The number of sub-containers of the first type that have been expanded is detected before the time when the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object. The first actual expansion quantity is adjusted based on the sub-quantity.

9. The method according to any one of claims 1-5, characterized in that, Before performing the first container expansion operation on the target object, when it is determined that the resource utilization rate of the system resources occupied by the container corresponding to the target object meets the container expansion conditions of the target object, the method further includes: Obtain the historical resource utilization rate of the system resources occupied by the container corresponding to the target object at each historical moment within the historical time period; Based on the historical resource utilization rate of the system resources occupied by the container corresponding to the target object at each historical moment, predict the target moment when the resource utilization rate of the system resources occupied by the container corresponding to the target object will meet the container expansion conditions. Perform a second container expansion operation on the target object within a unit time period before the target time to obtain a second type of expanded container.

10. The method according to claim 9, characterized in that, Once it is determined that the business data in any expanded container has been processed, the method further includes: Identify the creation time of the expanded container that has completed processing business data; Based on the creation time, determine the category to which the expanded container that has processed the business data belongs, wherein the category includes a first type of expanded container or a second type of expanded container; When the container is determined to belong to the first type of expansion container, the expansion container with the processed business data is directly deleted.

11. The method according to claim 10, characterized in that, The method further includes: When the container is determined to belong to the second category of expansion container, the second category of expansion container is controlled to enter a dormant state and a timer is started; Once the preset time period has elapsed and it is determined that the second type of expansion container has not been restarted, the expansion container that has processed the business data is deleted.

12. The method according to claim 4 or 5, characterized in that, The method further includes: When it is determined that the second type of business data exists in the temporary database table, and the resource utilization rate of the container corresponding to the target object is less than or equal to the first preset utilization rate threshold, a third container expansion operation is performed to obtain a third type of expansion container to process the second type of business data existing in the temporary database table.

13. The method according to claim 12, characterized in that, The method further includes: Calculate the total amount of the second type of business data in the temporary table of the database; Based on the total amount of the second type of business data in the temporary database table and the amount of business data processed per unit time corresponding to the third type of expansion container, determine the second quasi-expansion quantity of the third type of expansion container to be expanded; Based on the resource utilization rate of the system resources occupied by the third type of expansion container, the resource utilization rate of the system resources occupied by the container corresponding to the target object, and the second preset utilization rate threshold, the third quasi-expansion quantity of the third type of expansion container to be expanded is determined. The second actual expansion quantity of the third type of expansion container is determined based on the second quasi-expansion quantity and the third quasi-expansion quantity.

14. The method according to claim 13, characterized in that, The method further includes: When it is determined that the resource utilization rate of the system resources occupied by the main container is greater than the first preset utilization rate threshold, after the third type of expansion container finishes processing the current business data, the processing of business data is stopped, the third type of expansion container is deleted, and the resources occupied by the third type of expansion container are reclaimed.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 14.

Images