A smart and flexible processing system and method for container leasing business data

By capturing and dynamically adjusting container leasing business data in real time through an intelligent elastic processing system, the problems of low resource utilization and high response latency have been solved, achieving efficient data processing and server management, and improving system stability and resource utilization.

CN123068093APending Publication Date: 2026-11-10FLORENS (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611245308.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-17
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In existing technologies, container leasing business data processing systems suffer from low resource utilization and high response latency under the impact of massive data volumes, leading to a reduction in server lifespan.

Method used

The system employs an intelligent elastic processing system, comprising a data acquisition and stream processing layer, a computing and storage layer, a monitoring and log acquisition layer, an AI comprehensive analysis layer, and a scaling-up/scaling execution layer. By capturing business database change events in real time, it dynamically adjusts the number of data processing modules in the Kubernetes cluster to achieve automatic scaling-up or scaling-down.

Benefits of technology

It significantly improves resource utilization and stability, reduces response latency and manual maintenance costs, effectively copes with sudden surges in massive data, and reduces the risk of server failure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068093A_ABST
    Figure CN123068093A_ABST
Patent Text Reader

Abstract

A smart, elastic processing system and method for container leasing business data, relating to the field of data processing technology, is disclosed. It involves real-time capture, transformation, message consumption, parsing, and processing of change events in the business database through a data acquisition and stream processing layer and a computing and storage layer. A monitoring and log collection layer collects load metrics and container logs from each data processing module. An AI comprehensive analysis layer periodically queries container logs at specified time periods and log levels as target logs, inputting preset rules, target logs, and load metrics into a large-scale AI model for container leasing business to obtain analysis results. An online scaling API service module in the scaling execution layer dynamically adjusts the number of each data processing module based on the analysis results, scaling up or down the Kubernetes cluster to effectively cope with sudden surges in data volume, improve resource utilization and stability, and reduce response latency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an intelligent and flexible processing system and method for container leasing business data. Background Technology

[0002] Currently, in container leasing business scenarios, container leasing business data (such as leasing billing data) is typically retrieved from a business database at specified time periods (e.g., monthly) and processed to complete designated business tasks (e.g., bill settlement). However, because this method involves data acquisition and processing on a specified time basis, it leads to a massive accumulation of data. Servers used for data processing are overwhelmed by this massive data volume, resulting in low resource utilization, high response latency, and ultimately, reduced server lifespan.

[0003] Based on this, this specification provides an intelligent and flexible processing system and method for container leasing business data. Summary of the Invention

[0004] To address the technical problems of low resource utilization and high response latency of servers processing container leasing business data under the impact of massive data volumes in existing technologies, this invention proposes an intelligent and flexible processing system and method for container leasing business data to solve the aforementioned technical problems in the prior art.

[0005] This specification provides an intelligent and flexible processing system for container leasing business data. The system includes:

[0006] Data Acquisition and Stream Processing Layer: Used to capture change events in the business database in real time and convert them into JSON format data messages, which are then sent to the Kafka message queue;

[0007] Computation and storage layer: Deploy various data processing modules built on a Kubernetes cluster. Each data processing module is used to consume data messages in the Kafka message queue, perform message parsing and data processing based on the data messages, and store the processing results in the result database.

[0008] Monitoring and Log Collection Layer: Used to collect load metrics and container logs from each data processing module in real time;

[0009] AI Comprehensive Analysis Layer: This layer is used to periodically query container logs of a specified time period and preset log level from the monitoring and log collection layer as target logs, and input preset rules, the target logs, and the load indicators into the container leasing business AI big model to obtain the analysis results of each container leasing business.

[0010] The scaling execution layer includes an online scaling API service module, which is used to dynamically adjust the number of each data processing module based on the analysis results, thereby scaling up or down the Kubernetes cluster.

[0011] Optionally, the data acquisition and stream processing layer includes connectors, a business database, and a Kafka message queue. The business database includes replication slots and data blocks. Each data block stores data tables for different business dimensions. There is a one-to-one correspondence between each connector, each replication slot, and each data block.

[0012] The connector is used to send a replication connection request to the business database.

[0013] The business database is used to stream change events in the WAL log of the data block corresponding to the connector, starting from a specified position in the replication slot record corresponding to the connector, after receiving a replication connection request.

[0014] The connector is also used to convert the change event into a unified SourceRecord structure, serialize the structure into a JSON format data message, and send it to the Kafka message queue.

[0015] Optionally, the Kafka message queue includes message topics, and each message topic stores data messages corresponding to change events with different data identifiers.

[0016] Optionally, the load metrics include CPU utilization, memory utilization, Kafka consumer latency, message processing throughput, and the number of times the data processing module is restarted within the specified time period.

[0017] Optionally, the monitoring and log collection layer includes:

[0018] Prometheus component: used to collect load metrics of each data processing module;

[0019] Log collection tool: Used to collect container logs from each data processing module and send them to the Elasticsearch component;

[0020] The Elasticsearch component is used to store container logs collected by the log collection tool.

[0021] Optionally, the AI ​​comprehensive analysis layer includes:

[0022] Log query module: Used to periodically query container logs of a specified time period and preset log level from the Elasticsearch component as target logs;

[0023] Anomaly Classification Module: This module is used to input the target log into the container leasing business AI model, and then use the container leasing business AI model to perform semantic classification on the target log to determine the anomaly type corresponding to the target log.

[0024] Comprehensive analysis module: used to input preset rules, the anomaly types and the load indicators into the container leasing business AI big model to obtain the analysis results of each container leasing business.

[0025] Optionally, the Kubernetes cluster further includes a resource controller; the online scaling API service module includes:

[0026] Quota checking unit: When the decision in the analysis results is emergency expansion, it queries the remaining quota of the namespace in the current Kubernetes cluster and determines whether the remaining quota meets the target number of replicas in the analysis results;

[0027] Field modification unit: used to use an authorized identity token to call the scaling API when the remaining quota meets the target number of replicas, and modify a specified field in the resource controller based on the target number of replicas;

[0028] The resource controller is used to dynamically adjust the number of each data processing module based on the modified specified field when a change in the specified field is detected, thereby expanding the capacity of the Kubernetes cluster.

[0029] Optionally, the Kubernetes cluster further includes a resource controller; the scaling execution layer further includes a message sending module;

[0030] The message sending module is used to generate a suggested expansion message based on the load indicators and the analysis results when the decision in the analysis results is to suggest expansion, and to display it to the operation and maintenance personnel. The suggested expansion message includes a one-click executable link.

[0031] The online scaling API service module includes:

[0032] Identity authentication unit: used to authenticate the identity of the operation and maintenance personnel after they click the one-click executable link;

[0033] Quota check unit: After successful identity authentication, it queries the remaining quota of the namespace in the current Kubernetes cluster and determines whether the remaining quota meets the recommended number of replicas in the analysis results;

[0034] Field Modification Unit: When the remaining quota meets the recommended number of replicas, it uses an authorized identity token to call the scaling API and modify a specified field in the resource controller based on the recommended number of replicas.

[0035] Optionally, the Kubernetes cluster further includes a resource controller; the scaling-up / scaling execution layer further includes a message sending module and a scaling-down determination module;

[0036] The scaling-down determination module is used to determine whether preset scaling-down conditions are met based on the analysis results, the load indicators, and the current number of data processing modules. The scaling-down conditions include: the decision in the analysis results is to scale down, the current number of data processing modules is greater than the preset minimum number of replicas, the Kafka consumer lag in the load indicators is within a preset range, and the duration of the decision in the analysis results to scale down reaches a preset time threshold.

[0037] The message sending module is used to generate a scaling-down message based on the load indicators and the analysis results when the scaling-down conditions are met, and to display it to the operation and maintenance personnel. The scaling-down message includes a one-click executable link.

[0038] The online scaling API service module includes:

[0039] Identity authentication unit: used to authenticate the identity of the operation and maintenance personnel after they click the one-click executable link;

[0040] Scaling down check unit: used to determine, after successful identity authentication, whether the number of scaled-down replicas in the analysis results is less than the minimum number of replicas;

[0041] Field Modification Unit: Used to call the scaling-up / scaling-down API using an authorized identity token when the number of replicas after scaling down is not less than the minimum number of replicas, and modify a specified field in the resource controller based on the number of replicas after scaling down.

[0042] This specification provides an intelligent and flexible processing method for container leasing business data, the method comprising:

[0043] S1: Data Acquisition and Processing: Capture change events in the business database in real time and convert them into JSON format data messages, then send them to the Kafka message queue;

[0044] S2: Business Processing: Consume data messages in the Kafka message queue, perform message parsing and data processing based on the data messages, and store the processing results in the result database;

[0045] S3: Collect server load metrics and container logs for data processing;

[0046] S4: Target Log Query and Analysis: Periodically query container logs of a specified time period and preset log level from the collected container logs as target logs, and input the preset rules, the target logs and the load indicators into the container leasing business AI big model to obtain the analysis results of each container leasing business;

[0047] S5: Based on the analysis results, dynamically adjust the computing resources of the server.

[0048] Beneficial effects:

[0049] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0050] This specification provides an intelligent and elastic processing system for container leasing business data. The system includes a data acquisition and stream processing layer, a computing and storage layer, a monitoring and log acquisition layer, an AI comprehensive analysis layer, and a scaling and scaling execution layer. The data acquisition and stream processing layer captures change events in the business database in real time, converts them into JSON format data messages, and sends them to a Kafka message queue. By capturing change events in the business database in real time and performing subsequent data processing, it eliminates the need to acquire and process data at specified time intervals, avoiding the accumulation of large amounts of data and reducing the possibility of servers used for data processing being overwhelmed by massive data volumes. The computing and storage layer deploys various data processing modules built on a Kubernetes cluster. Each data processing module consumes data messages from the Kafka message queue, performs message parsing and data processing based on the data messages, and stores the processing results in a results database. The monitoring and log acquisition layer collects load metrics and container logs from each data processing module. The AI ​​comprehensive analysis layer periodically queries container logs of a specified time period and preset log level from the monitoring and log acquisition layer as target logs, and inputs preset rules, target logs, and load metrics into the container leasing business AI model to obtain analysis results for each container leasing business. The scaling execution layer includes an online scaling API service module that can dynamically adjust the number of each data processing module based on analysis results to scale up or down the Kubernetes cluster. By collecting load metrics and container logs from each data processing module in the Kubernetes cluster in real time and comprehensively analyzing them using an AI model for container leasing business, it dynamically adjusts computing resources (i.e., data processing modules in the Kubernetes cluster) based on the analysis results. This enables automatic scaling up to maintain latency during peak data periods and automatic scaling down to reduce costs during off-peak data periods, thereby optimizing the performance of the entire data processing process. This effectively copes with sudden surges in massive data volumes, significantly improves resource utilization and stability, reduces response latency, and simultaneously reduces manual maintenance costs and server failure risks.

[0051] The online scaling API service module of this invention includes a quota checking unit. This unit, when the analysis results indicate an urgent scaling decision or a recommended scaling decision and authentication is successful, queries the remaining quota of the namespace in the current Kubernetes cluster and determines whether the remaining quota meets the target or recommended replica count in the analysis results. By automatically checking whether the remaining quota meets the target or recommended replica count before scaling, the quota checking unit prevents invalid scaling. Furthermore, the scaling execution layer also includes a scaling-down determination module. This module determines whether preset scaling-down conditions are met based on the analysis results, load metrics, and the current number of data processing modules. The online scaling API service module also includes a scaling-down checking unit. This unit, after authentication is successful, determines whether the number of replicas after scaling down is less than the minimum replica count in the analysis results. Through the dual determination by the scaling-down determination module and the scaling-down checking unit, service unavailability due to excessive scaling-down is avoided.

[0052] In this invention, when the decision in the analysis results is to suggest scaling up or scaling down, manual intervention will be triggered. That is, the message sending module will generate a suggestion to scale up or scale down message to the operation and maintenance personnel. Both the suggestion to scale up or scale down message include a one-click executable link, which is convenient for the operation and maintenance personnel to click the link to confirm. The corresponding scaling up or scaling down will only be performed after the operation and maintenance personnel click the link to confirm, thus balancing automation efficiency and security.

[0053] This specification provides an intelligent and elastic processing method for container leasing business data. This method can capture change events in the business database in real time, convert them into JSON format data messages, and send them to a Kafka message queue. By capturing change events in the business database in real time and performing subsequent data processing, it eliminates the need to acquire and process data at specified time intervals, avoiding the accumulation of large amounts of data and reducing the possibility of servers used for data processing being overwhelmed by massive data volumes. Subsequently, the data messages in the Kafka message queue are consumed, and message parsing and data processing are performed based on the data messages. The processing results are stored in a results database. Load metrics and container logs of the servers used for data processing are collected. Container logs of a specified time period and preset target log level are periodically queried from the collected container logs as target logs. Preset rules, target logs, and load metrics are input into the container leasing business AI model to obtain analysis results for each container leasing business. Based on the analysis results, the server's computing resources are dynamically adjusted. By collecting real-time load metrics and container logs from servers used for data processing, and comprehensively analyzing them through an AI-powered big data model for container leasing, the computing resources of servers can be dynamically adjusted based on the analysis results. This enables automatic scaling up to maintain latency during peak data periods and automatic scaling down to reduce costs during off-peak periods, thereby optimizing the performance of the entire data processing process. This effectively addresses sudden surges in massive data volumes, significantly improves resource utilization and stability, reduces response latency, and lowers manual maintenance costs and server failure risks. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0055] Figure 1 This is a schematic diagram of a smart and flexible processing system for container leasing business data provided in this specification.

[0056] Figure 2 This is a schematic diagram of the structure of a data acquisition and stream processing layer provided in this specification;

[0057] Figure 3 This is a schematic diagram of the structure of a monitoring and log collection layer provided in this specification;

[0058] Figure 4 This is a schematic diagram of the structure of an AI comprehensive analysis layer provided in this specification;

[0059] Figure 5 This is a schematic diagram of the structure of a computing and storage layer provided in this specification;

[0060] Figure 6This is a schematic diagram of the structure of an online scaling-up / scaling-down API service module for emergency expansion provided in this specification;

[0061] Figure 7 This is a schematic diagram of the structure of a scaling execution layer provided in this specification;

[0062] Figure 8 This is a schematic diagram of the structure of an online scaling-up / scaling-down API service module for a suggested expansion method provided in this specification;

[0063] Figure 9 This is a schematic diagram of the structure of an online scaling up / down API service module provided in this specification.

[0064] Figure 10 This document presents a flowchart illustrating an intelligent and flexible processing method for container leasing business data. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0066] This specification provides an intelligent and flexible processing system and method for container leasing business data. The technical solutions provided by the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0067] Figure 1 This is a schematic diagram of the structure of an intelligent and flexible processing system for container leasing business data provided in this specification. Figure 1 The system shown includes a data acquisition and stream processing layer 100, a computing and storage layer 101, a monitoring and log acquisition layer 102, an AI comprehensive analysis layer 103, and a scaling execution layer 104. Among them:

[0068] The aforementioned data acquisition and stream processing layer 100 can capture change events in the business database in real time and convert them into JSON format data messages, which are then sent to the Kafka message queue. The business database can be a PostgreSQL database with version ≥10.0, natively supporting logical replication, used to store container leasing business data. Change events are billing-related status change events in the business database, such as lease order creation, container rental commencement, container return, price adjustment, and bill status changes. The Kafka message queue is used to store these data messages. This real-time capture can be implemented based on PostgreSQL logical replication and CDC (Change Data Capture) technology.

[0069] The aforementioned computing and storage layer 101 is deployed based on a Kubernetes cluster 1011, comprising various data processing modules 10111. Each data processing module 10111 (i.e. Figure 1 The data processing modules (1 to N, where N represents the number of data processing modules) consume data messages from the Kafka message queue, perform message parsing and data processing based on these messages, and store the processing results in a result database. Message parsing refers to analyzing the data messages, while data processing refers to processing the parsed results according to the container leasing business rules. These container leasing business rules are pre-set and may specifically involve billing for container rental invoices. The data processing modules can be billing service Pods within a Kubernetes cluster. The processing results are the corresponding processing results for the container leasing business, i.e., the billing results. The result database can be used to store these processing results and can be a detailed invoice database.

[0070] The aforementioned monitoring and log collection layer 102 can collect load metrics and container logs from each data processing module 10111 in real time. Load metrics include CPU utilization (cpu_usage_percent), memory utilization (memory_usage_percent), Kafka consumer lag (kafka_consumer_lag, i.e., backlog), message processing throughput (msg_processed_per_sec), and the number of data processing module restarts within a specified time period (pod_restart_count_specified time period). This specified time period can be preset, specifically set to the most recent 10 minutes, i.e., pod_restart_count_10min.

[0071] The aforementioned AI comprehensive analysis layer 103 can periodically query container logs of a specified time period and preset log level from the monitoring and log collection layer as target logs. It then inputs preset rules, target logs, and load metrics into the container leasing business AI big model to obtain analysis results for each container leasing business. Container logs can have multiple log levels. When a container log's log level is error, that container log can be used as the target log, meaning all target logs are at the error level. The preset log level is error. The preset rules are pre-set; the preset rule engine is a set of predefined rules that works in conjunction with the container leasing business AI big model, providing a "safety fallback." Specifically, it may include:

[0072] Time window rule: Low load must last for more than 15 minutes before a scaling-down suggestion is triggered to avoid frequent scaling due to instantaneous fluctuations;

[0073] Security boundary rules include the maximum number of replicas, the minimum number of replicas (e.g., no less than 2 replicas), and resource quota limits, to prevent excessive expansion or shrinkage.

[0074] The aforementioned AI model for container leasing operations is a general-purpose model deployed using private computing power. It is used for semantic classification of target logs and, based on preset rules, target logs, and load metrics, to obtain analysis results for each container leasing operation. The semantic classification refers to classifying typical errors based on the target logs, such as distinguishing between "PostgreSQL connection pool exhaustion" and "business logic anomalies." The analysis results include three levels of decision-making: emergency scaling up, recommended scaling up, and scaling down. Emergency scaling up indicates a current fault mode requiring immediate automatic scaling up. Recommended scaling up indicates a current peak business mode, suggesting scaling up only after manual confirmation. Scaling down indicates a current low-load stable mode, suggesting scaling down only after manual confirmation.

[0075] The aforementioned scaling execution layer 104 includes an online scaling API service module 1041. The online scaling API service module 1041 can dynamically adjust the number of each data processing module 10111 based on the analysis results, thereby scaling up or down the Kubernetes cluster 1011.

[0076] In some embodiments of this specification, the data acquisition and stream processing layer 100 described above includes connectors 1001 (i.e. Figure 2 The components include connectors 1~D), business database 1002, and Kafka message queue 1003. Business database 1002 includes replication slots 10021 (i.e., ... Figure 2 The replication slots 1~F and each data block 10022 (i.e. Figure 2The data blocks 1~S in the database store different business-dimensional data tables, such as basic data tables and dynamic tables. The aforementioned change events originate from the data change records of the tables in different data blocks. Each connector (Debezium, or Debezium for PostgreSQL, i.e., a connector in the Debezium project used to capture PostgreSQL database change events (Change DataCapture, CDC)) 1001, each replication slot 10021, and each data block 10022 have a one-to-one correspondence, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of the data acquisition and stream processing layer provided in this specification. The connector 1001 is used to send a replication connection request to the business database 1002. The connector 1001 can act as a client for PostgreSQL logical replication. The replication connection request includes a connector identifier, a corresponding replication slot identifier, and a corresponding data block identifier. Upon receiving the replication connection request, the business database 1002 starts streaming change events from the specified position (current LSN (Log Sequence Number) or historical position) recorded in the replication slot 10021 (i.e., the replication slot represented by the replication slot identifier) ​​corresponding to the connector 1001 (i.e., the connector represented by the connector identifier) ​​in the WAL log (Write-Ahead Log). The replication slot 10021 records the position. Connector 1001 can also convert change events into a unified SourceRecord structure (that is, convert INSERT, UPDATE, and DELETE operations into a unified SourceRecord structure), serialize the structure into a JSON format data message, and send it to Kafka message queue 1003. The SourceRecord structure is a unified format that all Source Connectors (such as Debezium, JDBC Source, etc.) must follow when writing data to Kafka. Prioritizing message delivery to the Kafka message queue achieves complete decoupling between data collection and billing calculation, and provides peak shaving and valley filling capabilities.

[0077] Furthermore, the aforementioned Kafka message queue 1003 includes various message topics, each storing data messages corresponding to change events with different data identifiers. Data identifiers are used to represent lease orders, or change events. That is, all data messages corresponding to change events for the same lease order are stored in the same partition within the same message topic. Change events for different lease orders are categorized into message topics and partitions according to their data identifiers, and then distributed to different partitions within different consumer topics to ensure the order of change events for the same lease order. PostgreSQL logical replication also strictly follows the order in which transactions are committed. Moreover, logical replication typically has less than 5% performance impact on the business database (only increasing WAL resolution and replication slot overhead), does not intrude on business code, and requires no modification to existing applications.

[0078] Furthermore, the subscription scope of the aforementioned connector can be limited to: subscribing only to the container leasing billing related tables (i.e., including data blocks of data tables related to container leasing billing) to avoid resource waste caused by full database synchronization.

[0079] Furthermore, Debezium can bidirectionally map consumption offsets (Kafka offsets) to PostgreSQL's LogSequence Numbers (LSNs) and store them in a Kubernetes PVC persistent volume. When the CDC task restarts, the connector reads the last committed offset from the PVC and continues resolving the WAL from the corresponding PostgreSQL LSN, ensuring no data loss and no duplication of critical billing change events.

[0080] In some embodiments of this specification, the monitoring and log collection layer 102 includes a Prometheus component 1021, a log collection tool 1022, and an Elasticsearch component 1023, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the monitoring and log collection layer provided in this specification. The Prometheus component 1021 is used to collect load metrics from each data processing module 10111. The log collection tool 1022 is used to collect container logs from each data processing module 10111 and send them to the Elasticsearch component 1023. This log collection tool can be Filebeat or Fluentd. The Elasticsearch component 1023 is used to store the container logs collected by the log collection tool 1022.

[0081] In some embodiments of this specification, the AI ​​comprehensive analysis layer 103 includes a log query module 1031, an anomaly classification module 1032, and a comprehensive analysis module 1033, such as... Figure 4 As shown, Figure 4This is a schematic diagram of the structure of an AI comprehensive analysis layer provided in this specification. The log query module 1031 is used to periodically (pre-set, specifically every 5 minutes) query container logs from the Elasticsearch component 1023 for a specified time period (pre-set, specifically the most recent 10 minutes) and a preset log level (error level) as target logs. The anomaly classification module 1032 is used to input the target logs into the container leasing business AI big data model, and to perform semantic classification on the target logs through the container leasing business AI big data model to determine the anomaly type corresponding to the target logs. The anomaly type can be at least one of various preset typical errors, which may include "PostgreSQL connection pool exhaustion," "business logic exception," etc. The comprehensive analysis module 1033 is used to input preset rules, anomaly types, and load indicators into the large AI model for container leasing business to obtain the analysis results of each container leasing business. The comprehensive analysis module 1033 can also first merge the anomaly types and load indicators into a six-dimensional feature vector (the six dimensions of the six-dimensional feature vector include: CPU utilization, memory utilization, Kafka consumer lag, message processing throughput, number of restarts of data processing modules within a specified time period, and anomaly type), and then concatenate the six-dimensional feature vector with preset rules and input them together into the large AI model for container leasing business to obtain the analysis results of each container leasing business.

[0082] In some embodiments of this specification, the methods for dynamically adjusting the number of each data processing module and scaling up or down the Kubernetes cluster 1011 differ depending on the level of decision-making. The specific dynamic adjustment and scaling-up / scaling methods are as follows:

[0083] The analysis results indicate an emergency scaling decision. The aforementioned Kubernetes cluster 1011 also includes a resource controller 10112, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the computing and storage layer structure provided in this specification. The aforementioned online scaling API service module 1041 includes a quota checking unit 10411 and a field modification unit 10412, such as... Figure 6 As shown, Figure 6This is a schematic diagram of the structure of an online scaling-up / scaling-down API service module for emergency scaling provided in this specification. Quota checking unit 10411: When the decision in the analysis results is emergency scaling, it queries the remaining quota (CPU, memory, number of Pods) in the current Kubernetes cluster 1011 namespace and determines whether the remaining quota meets the target replica number in the analysis results. This target replica number is output by the aforementioned container leasing business AI model when the decision is emergency scaling. During emergency scaling, the target replica number represents the number of times the data processing module 10111 in the Kubernetes cluster 1011 needs to be scaled up. When determining whether the remaining quota meets the target replica number in the analysis results, it checks whether the target replica number * application quota is less than the remaining quota. This application quota refers to the application quota of the Kubernetes cluster 1011, i.e., the application quota required by the data processing module. The field modification unit 10412 is used to call the scaling API using an authorized identity token (ServiceAccount Token) when the remaining quota meets the target replica number. Based on the target replica number, it modifies the specified field (spec.replicas) in the resource controller 10112 (Deployment), for example, scaling from 3 to 6. The resource controller 10112 is then used to dynamically adjust the number of each data processing module based on the modified specified field when a change is detected, increasing the number of data processing modules to the target replica number to scale up the Kubernetes cluster 1011.

[0084] If the decision in the analysis results suggests expansion, the aforementioned expansion / shrinkage execution layer 104 also includes a message sending module 1042, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of the scaling execution layer provided in this specification. The message sending module 1042, when the decision in the analysis results suggests scaling up, generates a suggested scaling message based on load metrics and analysis results, and displays it to the operations and maintenance personnel. The message sending module 1042 can send the suggested scaling message to the operations and maintenance personnel via an enterprise robot. The suggested scaling message includes a one-click executable link, analysis results, load metrics, target logs, etc. The one-click executable link points to the online scaling API service module 1041, carrying a short-term valid one-time token. After the operations and maintenance personnel click the link, the API service completes authentication, quota checks (scaling up) or minimum replica count checks (scaling down), and then calls the Kubernetes Scale API to execute the specified field changes. The online scaling API service module 1041 includes a quota check unit 10411, a field modification unit 10412, and an identity authentication unit 10413, as follows... Figure 8As shown, Figure 8 This is a schematic diagram of the online scaling-up / scaling-down API service module for suggested scaling up provided in this specification. The identity authentication unit 10413 is used to authenticate the identity of operations personnel after they click the one-click executable link. The quota checking unit 10411 is used to query the remaining quota of the namespace in the current Kubernetes cluster 1011 after successful identity authentication, and to determine whether the remaining quota meets the suggested replica number in the analysis results. The specific judgment process is similar to the process of "determining whether the remaining quota meets the target replica number in the analysis results" described above, and will not be repeated here. This suggested replica number is output by the aforementioned container leasing business AI model when it decides to suggest scaling up. When suggesting scaling up, the suggested replica number represents the number of times the data processing module 10111 in the Kubernetes cluster 1011 needs to be scaled up. The field modification unit 10412 is used to call the scaling API using an authorized identity token when the remaining quota meets the recommended number of replicas. Based on the recommended number of replicas, it modifies a specified field in the resource controller 10112. The specific implementation process is similar to the execution process of the "field modification unit 10412 when the remaining quota meets the target number of replicas" described above, and will not be repeated here. Afterwards, when a change to the specified field is detected, the resource controller 10112 dynamically adjusts the number of each data processing module based on the modified specified field, that is, increases the number of data processing modules to the recommended number of replicas to scale up the Kubernetes cluster 1011.

[0085] If the decision in the analysis results is to reduce the size, the aforementioned scaling-up / scaling execution layer 104 also includes a message sending module 1042 and a scaling-up determination module 1043, specifically as follows: Figure 7As shown, the scaling-down determination module 1043 is used to determine whether preset scaling-down conditions are met based on analysis results, load indicators, and the current number of data processing modules. These conditions are pre-set and include: the analysis results indicate a scaling-down decision; the current number of data processing modules is greater than the preset minimum replica number (pre-set, specifically 2); the Kafka consumer lag in the load indicators is within a preset range (pre-set); and the duration of the scaling-down decision in the analysis results reaches a preset time threshold (pre-set, specifically more than 15 minutes). The phrase "the analysis results indicate a scaling-down decision" signifies that the container leasing business AI model has been determined to be in "scaling-down" (low-load stable mode), meaning that after comprehensive analysis of the six-dimensional feature vector and preset rules, the current state is low-load and without abnormalities. The phrase "the duration of the scaling-down decision in the analysis results reaches the preset time threshold" signifies that the duration meets the time window requirements of the rule engine, meaning the low-load state must continue for a certain period (e.g., more than 15 minutes) to avoid erroneous scaling-down due to a sudden drop in load. The phrase "the current number of data processing modules is greater than the preset minimum replica number" ensures that there are still sufficient redundant instances to guarantee service availability after scaling-down. The aforementioned "Kafka consumer lag in the load metric is within a preset range" is to ensure that there is not too much backlog of unprocessed billing messages, avoiding processing delays after scaling down. Only when all the above scaling-down conditions are met is it determined that the scaling-down conditions are "met." Subsequently, a scaling-down message is sent via the enterprise robot, and the scaling-down operation is executed after confirmation by the operations and maintenance personnel. Specifically, the message sending module 1042 generates a scaling-down message based on load metrics and analysis results when the scaling-down conditions are met, and displays it to the operations and maintenance personnel. The scaling-down message may include a one-click executable link, analysis results, load metrics, target logs, and other information. The aforementioned online scaling-down API service module 1041 includes a field modification unit 10412, an identity authentication unit 10413, and a scaling-down check unit 10414, such as... Figure 9 As shown, Figure 9This is a schematic diagram of the structure of an online scaling-up / scaling-down API service module provided in this specification. The authentication unit 10413 authenticates the identity of operations personnel after they click the one-click executable link. The scaling-up check unit 10414, after successful authentication, determines whether the number of replicas after scaling down is less than the minimum number of replicas in the analysis results. The field modification unit 10412, when the number of replicas after scaling down is not less than the minimum number of replicas, uses an authorized identity token to call the scaling-up / scaling-down API and modifies a specified field in the resource controller 10112 based on the number of replicas after scaling down. The specific implementation process is similar to the execution process of the field modification unit 10412 when the remaining quota meets the target number of replicas, and will not be described again here. Subsequently, when a change in the specified field is detected, the resource controller 10112 dynamically adjusts the number of each data processing module based on the modified specified field, reducing the number of data processing modules to the number of replicas after scaling down, thereby scaling down the Kubernetes cluster 1011.

[0086] In addition, when the decision in the analysis results is to urgently expand capacity, the aforementioned message sending module 1042 can also generate an emergency expansion message based on load indicators and analysis results, and display it to the operation and maintenance personnel. The emergency expansion message includes information such as analysis results and load indicators.

[0087] In some embodiments of this specification, the K8s API address and long-term credentials are not exposed in enterprise messages; all external requests are processed through the online scaling API service module 1041. Scaling operations include resource quota pre-checks, and scaling operations include minimum replica protection (e.g., no less than 2 replicas) to prevent service unavailability due to excessive scaling. Furthermore, the system achieves a fully automated closed-loop pipeline of "collection → buffering → calculation → monitoring → decision-making → scaling," possessing the following adaptive capabilities:

[0088] Adaptive elastic scaling: The system dynamically adjusts the number of billing service Pods based on real-time indicators such as load, backlog, and abnormal log events. It automatically expands capacity during peak periods to maintain latency and automatically shrinks capacity during off-peak periods to reduce costs.

[0089] Adaptive fault awareness: When a Pod restart or a surge in error logs is detected, the system automatically triggers emergency scaling to quickly restore service capabilities.

[0090] Resource quota adaptability: Automatically checks quotas before scaling up to avoid invalid scaling up; automatically checks the minimum number of replicas before scaling down to avoid excessive scaling down.

[0091] Adaptability of manual intervention: In normal scaling scenarios, the manual confirmation link is retained, and critical changes require maintenance personnel to click the link for confirmation, balancing automation efficiency and security.

[0092] Through the above design, the system significantly improves the elasticity and resource utilization of the rental box bill settlement business without modifying the business code, effectively copes with the impact of massive sudden data, and reduces manual operation and maintenance costs and cluster (server) failure risks.

[0093] Based on the above system, this specification also provides an intelligent and flexible processing method for container leasing business data, such as... Figure 10 As shown, Figure 10 The flowchart of an intelligent and flexible processing method for container leasing business data provided in this specification includes the following steps:

[0094] S1: Data Acquisition and Processing: Capture change events in the business database in real time and convert them into JSON format data messages, then send them to the Kafka message queue;

[0095] S2: Business Processing: Consume data messages in the Kafka message queue, perform message parsing and data processing based on the data messages, and store the processing results in the result database;

[0096] S3: Collect server load metrics and container logs for data processing;

[0097] S4: Target Log Query and Analysis: Periodically query container logs of a specified time period and preset log level from the collected container logs as target logs, and input the preset rules, the target logs and the load indicators into the container leasing business AI big model to obtain the analysis results of each container leasing business;

[0098] S5: Based on the analysis results, dynamically adjust the computing resources of the server.

[0099] The execution entity of the above method is the above system or its modules. The specific implementation process of step S1 is consistent with that of the data acquisition and stream processing layer 100, and will not be repeated here. The specific implementation process of step S2 is consistent with that of the computing and storage layer 101, and will not be repeated here. The specific implementation process of step S3 is consistent with that of the monitoring and log acquisition layer 102, and will not be repeated here. The server in step S3 is the running server of each data processing module built on the Kubernetes cluster, used to process container leasing business data. The specific implementation process of step S4 is consistent with that of the AI ​​comprehensive analysis layer 103, and will not be repeated here. The specific implementation process of step S5 is consistent with that of the scaling execution layer 104, and will not be repeated here. The above "dynamically adjusting the server's computing resources" is actually adjusting the number of data processing modules on the server.

[0100] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail with reference to the accompanying drawings and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention. In short, all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention patent.

Claims

1. An intelligent and flexible processing system for container leasing business data, characterized in that, The system includes: Data Acquisition and Stream Processing Layer: Used to capture change events in the business database in real time and convert them into JSON format data messages, which are then sent to the Kafka message queue; Computation and storage layer: Deploy various data processing modules built on a Kubernetes cluster. Each data processing module is used to consume data messages in the Kafka message queue, perform message parsing and data processing based on the data messages, and store the processing results in the result database. Monitoring and Log Collection Layer: Used to collect load metrics and container logs from each data processing module in real time; AI Comprehensive Analysis Layer: This layer is used to periodically query container logs of a specified time period and preset log level from the monitoring and log collection layer as target logs, and input preset rules, the target logs, and the load indicators into the container leasing business AI big model to obtain the analysis results of each container leasing business. The scaling execution layer includes an online scaling API service module, which is used to dynamically adjust the number of each data processing module based on the analysis results, thereby scaling up or down the Kubernetes cluster.

2. The intelligent and flexible processing system for container leasing business data as described in claim 1, characterized in that, The data acquisition and stream processing layer includes connectors, a business database, and a Kafka message queue. The business database includes replication slots and data blocks. Each data block stores data tables for different business dimensions. There is a one-to-one correspondence between each connector, each replication slot, and each data block. The connector is used to send a replication connection request to the business database. The business database is used to stream change events in the WAL log of the data block corresponding to the connector, starting from a specified position in the replication slot record corresponding to the connector, after receiving a replication connection request. The connector is also used to convert the change event into a unified SourceRecord structure, serialize the structure into a JSON format data message, and send it to the Kafka message queue.

3. The intelligent and flexible processing system for container leasing business data as described in claim 2, characterized in that, The Kafka message queue includes message topics, and each message topic stores data messages corresponding to change events with different data identifiers.

4. The intelligent and flexible processing system for container leasing business data as described in claim 1, characterized in that, The load metrics include CPU utilization, memory utilization, Kafka consumer latency, message processing throughput, and the number of times the data processing module is restarted within the specified time period.

5. The intelligent and flexible processing system for container leasing business data as described in claim 1, characterized in that, The monitoring and log collection layer includes: Prometheus component: used to collect load metrics of each data processing module; Log collection tool: Used to collect container logs from each data processing module and send them to the Elasticsearch component; The Elasticsearch component is used to store container logs collected by the log collection tool.

6. The intelligent and flexible processing system for container leasing business data as described in claim 5, characterized in that, The AI ​​comprehensive analysis layer includes: Log query module: Used to periodically query container logs of a specified time period and preset log level from the Elasticsearch component as target logs; Anomaly Classification Module: This module is used to input the target log into the container leasing business AI model, and then use the container leasing business AI model to perform semantic classification on the target log to determine the anomaly type corresponding to the target log. Comprehensive analysis module: used to input preset rules, the anomaly types and the load indicators into the container leasing business AI big model to obtain the analysis results of each container leasing business.

7. The intelligent and flexible processing system for container leasing business data as described in claim 1, characterized in that, The Kubernetes cluster also includes a resource controller; The online scaling API service module includes: Quota checking unit: When the decision in the analysis results is emergency expansion, it queries the remaining quota of the namespace in the current Kubernetes cluster and determines whether the remaining quota meets the target number of replicas in the analysis results; Field modification unit: used to use an authorized identity token to call the scaling API when the remaining quota meets the target number of replicas, and modify a specified field in the resource controller based on the target number of replicas; The resource controller is used to dynamically adjust the number of each data processing module based on the modified specified field when a change in the specified field is detected, thereby expanding the capacity of the Kubernetes cluster.

8. The intelligent and flexible processing system for container leasing business data as described in claim 1, characterized in that, The Kubernetes cluster also includes a resource controller; the scaling execution layer also includes a message sending module. The message sending module is used to generate a suggested expansion message based on the load indicators and the analysis results when the decision in the analysis results is to suggest expansion, and to display it to the operation and maintenance personnel. The suggested expansion message includes a one-click executable link. The online scaling API service module includes: Identity authentication unit: used to authenticate the identity of the operation and maintenance personnel after they click the one-click executable link; Quota check unit: After successful identity authentication, it queries the remaining quota of the namespace in the current Kubernetes cluster and determines whether the remaining quota meets the recommended number of replicas in the analysis results; Field Modification Unit: When the remaining quota meets the recommended number of replicas, it uses an authorized identity token to call the scaling API and modify a specified field in the resource controller based on the recommended number of replicas.

9. The intelligent and flexible processing system for container leasing business data as described in claim 1, characterized in that, The Kubernetes cluster also includes a resource controller; the scaling up / down execution layer also includes a message sending module and a scaling down determination module. The scaling-down determination module is used to determine whether the preset scaling-down conditions are met based on the analysis results, the load index, and the current number of data processing modules. The scaling-down conditions include: the decision in the analysis results is to scale down, the number of current data processing modules is greater than the preset minimum number of replicas, the Kafka consumer lag in the load metric is within a preset range, and the duration of the decision in the analysis results to scale down reaches a preset time threshold. The message sending module is used to generate a scaling-down message based on the load indicators and the analysis results when the scaling-down conditions are met, and to display it to the operation and maintenance personnel. The scaling-down message includes a one-click executable link. The online scaling API service module includes: Identity authentication unit: used to authenticate the identity of the operation and maintenance personnel after they click the one-click executable link; Scaling down check unit: used to determine, after successful identity authentication, whether the number of scaled-down replicas in the analysis results is less than the minimum number of replicas; Field Modification Unit: Used to call the scaling-up / scaling-down API using an authorized identity token when the number of replicas after scaling down is not less than the minimum number of replicas, and modify a specified field in the resource controller based on the number of replicas after scaling down.

10. A method for intelligent and flexible processing of container leasing business data according to any one of claims 1 to 9, characterized in that, The method includes: S1: Data Acquisition and Processing: Capture change events in the business database in real time and convert them into JSON format data messages, then send them to the Kafka message queue; S2: Business Processing: Consume data messages in the Kafka message queue, perform message parsing and data processing based on the data messages, and store the processing results in the result database; S3: Collect server load metrics and container logs for data processing; S4: Target Log Query and Analysis: Periodically query container logs of a specified time period and preset log level from the collected container logs as target logs, and input the preset rules, the target logs and the load indicators into the container leasing business AI big model to obtain the analysis results of each container leasing business; S5: Based on the analysis results, dynamically adjust the computing resources of the server.