Data upgrading method, data upgrading management method and data upgrading system

Abstract

The invention provides a data upgrading method, a data upgrading management method and a data upgrading system.The upgrading method comprises the steps that according to an upgrading starting instruction received from a visual platform, a query method in a predefined data upgrading service logic template is called to execute paging query and obtain to-be-upgraded data; based on the data volume of the to-be-upgraded data, automatically selecting a target execution mode through decision logic in the data upgrading service logic template; and calling an upgrading method in the data upgrading service logic template to perform upgrading processing on the to-be-upgraded data, actively pushing processing progress information and execution log information to a visual platform through a report mechanism in the data upgrading service logic template, and performing upgrading visual monitoring by the visual platform. According to the method and the device, the problems of low development efficiency, poor normalization, high code repeatability, prominent processing performance bottleneck in a large data volume scene and lack of visual monitoring in a traditional data upgrading mode can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software, and in particular to a data upgrade method, a data upgrade management method and a data upgrade system. BACKGROUND

[0002] In the process of continuous iteration of enterprise-level software systems, it is often necessary to make structural changes, content corrections or business logic upgrades to the massive historical data in the production environment. Such data upgrade tasks are key links to ensure the smooth evolution of the system and the consistency of data quality, but traditional implementation methods generally have a series of prominent technical bottlenecks.

[0003] At present, the technical solutions commonly used in the industry mainly include two types: manual SQL script method and customized development application method. The manual SQL script method requires operation and maintenance or development personnel to manually write a large number of stored procedures or batch update statements, and to execute them directly in the production environment through a database client. This method is highly dependent on manual operation, and requires manual recording of execution progress, handling of exceptions, and data backup and rollback. The entire process is tedious, error-prone, and lacks standardized processes and traceability. The customized development application method is more flexible than pure SQL scripts, that is, for each data upgrade, a temporary Java application program is developed by the development team. The program encapsulates database connection, data query, batch processing, log printing and other codes, and is deployed in the form of a JAR package and started through the command line. However, this method essentially belongs to "chimney-style" development, and each upgrade program needs to write page queries, transaction management and other general logic from scratch, resulting in high code duplication, low development efficiency, and different code styles that are difficult to maintain and pass on.

[0004] That is, regardless of which of the above existing technical solutions is used, there are generally problems of low development efficiency, poor standardization, high code duplication, prominent processing performance bottlenecks in large data scenarios, and lack of visual monitoring. SUMMARY

[0005] In view of this, the embodiments of the present application provide a data upgrade method, a data upgrade management method and a data upgrade system to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present application provides a data upgrade method, comprising: According to the start-up upgrade instruction received from the visualization platform, calling a query method in a predefined data upgrade business logic template to perform a page query and obtain the data to be upgraded; automatically select a target execution mode according to the data volume of the data to be upgraded and the decision logic defined in the data upgrade business logic template, the target execution mode comprising: a single-thread execution mode or a multi-thread batch execution mode; invoke an upgrade method in the data upgrade business logic template to perform upgrade processing on the data to be upgraded according to the target execution mode; In the process of the upgrade processing, corresponding processing progress information and execution log information are actively pushed to the visualization platform by a reporting mechanism defined in the data upgrade business logic template, so that the visualization platform visually monitors the upgrade processing process of the data to be upgraded.

[0007] In some embodiments of the present application, before the query method in the predefined data upgrade business logic template is invoked, the method further comprises: dynamically obtaining database connection configuration information corresponding to the current running environment from a configuration center; initializing and establishing a connection between itself and a target database based on the database connection configuration information; performing a pre-check, wherein the pre-check comprises at least one of checking the availability of a dependent service, checking whether the resource usage of a target server is lower than a first preset threshold, and checking database connectivity; receiving a start upgrade instruction sent by the visualization platform; performing idempotency checking on the start upgrade instruction, and if the start upgrade instruction passes the idempotency checking, marking it as a start upgrade instruction that has passed the checking; Correspondingly, the method of invoking a query method in a predefined data upgrade business logic template according to a start upgrade instruction received from the visualization platform to perform a page query and obtain data to be upgraded comprises: According to the start upgrade instruction received from the visualization platform and that has passed the checking, invoking a query method in a predefined data upgrade business logic template to send an SQL query statement with page parameters to the target database; receiving the data to be upgraded returned by the target database that conforms to the SQL query statement.

[0008] In some embodiments of the present application, the decision logic is configured to compare the data volume of the data to be upgraded with a second preset threshold, and select the target execution mode from the preset execution modes according to the comparison result; Correspondingly, the method of automatically selecting a target execution mode according to the data volume of the data to be upgraded and the decision logic defined in the data upgrade business logic template comprises: compare the data amount of the data to be upgraded with a second preset threshold value; if the data amount is less than or equal to the second preset threshold value, the single-thread execution mode is selected; if the data amount is greater than the second preset threshold value, the multi-thread batch execution mode is selected, and the thread pool is initialized based on the parameters configured in the data upgrade business logic template for initializing the thread pool.

[0009] In some embodiments of the present application, the data to be upgraded is upgraded by calling an upgrade method in the data upgrade business logic template according to the target execution mode, which comprises: if the target execution mode is the single-thread execution mode, the upgrade method defined in the data upgrade business logic template is directly called to sequentially upgrade the data to be upgraded; if the target execution mode is the multi-thread batch execution mode, the data to be upgraded is divided into multiple batches of data, and each batch of data is submitted to the thread pool for concurrent upgrade processing, wherein each thread in the thread pool respectively calls an upgrade method defined in the data upgrade business logic template for batch data to upgrade one batch of data.

[0010] In some embodiments of the present application, during the upgrade processing, the corresponding processing progress information and execution log information are actively pushed to the visualization platform through the reporting mechanism defined in the data upgrade business logic template, which comprises: at a fixed time interval or a fixed data amount interval specified by the reporting mechanism in the data upgrade business logic template, an application programming interface provided by the visualization platform is called to actively push the current processing progress information and the corresponding execution log information of the upgrade processing to the visualization platform.

[0011] In some embodiments of the present application, the data upgrade method further comprises: after the upgrade processing is completed, a verification method defined in the data upgrade business logic template is called to perform business rule verification on the upgraded data to obtain verification result information; the verification result information is pushed to the visualization platform.

[0012] In some embodiments of the present application, the data upgrade business logic template is provided in the form of a Java abstract class, and the query method, the upgrade method, the decision logic and the reporting mechanism in the data upgrade business logic template are completed by inheriting the abstract class and implementing the predefined abstract methods or overwriting the hook methods of the abstract class.

[0013] Another aspect of the present application provides a data upgrade management method, executed by a visualization platform, comprising: According to a predefined execution order dependency relationship of each data upgrade task, the service state and server resource indicators relied on by each data upgrade program are sequentially acquired and checked by calling a monitoring system interface, and after the checking passes, a start upgrade instruction is sequentially sent to each corresponding data upgrade program to trigger each data upgrade program to execute the data upgrade method. The processing progress information and the execution log information actively pushed by the data upgrade program are received and aggregated. The processing progress information and the execution log information are displayed in a graphical interface.

[0014] In some embodiments of the present application, the data upgrade management method further comprises: Based on a preset task management interface, a data upgrade program package developed based on the data upgrade business logic template uploaded by a user is received, and the data upgrade program package is automatically parsed to identify and register the data upgrade tasks and their calling interfaces defined in the data upgrade program package.

[0015] A third aspect of the present application provides a data upgrade system, comprising: At least one data upgrade program configured to execute the data upgrade method; A visualization platform configured to execute the data upgrade management method; A configuration center for providing dynamic database connection configuration information to the data upgrade program; Wherein, the data upgrade program and the visualization platform communicate through an application program interface defined based on a standardized protocol.

[0016] A fourth aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the data upgrade method, or implement the data upgrade management method.

[0017] A fifth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the data upgrade method, or implement the data upgrade management method.

[0018] A sixth aspect of the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the data upgrade method, or implement the data upgrade management method.

[0019] The data upgrading method provided in the application comprises the following steps: according to a starting upgrading instruction received from a visualization platform, a query method in a predefined data upgrading business logic template is called to perform a paging query and obtain to-be-upgraded data; based on a data volume of the to-be-upgraded data, a target execution mode is automatically selected through a decision logic defined in the data upgrading business logic template, the target execution mode comprising a single-thread execution mode or a multi-thread batch execution mode; according to the target execution mode, an upgrading method in the data upgrading business logic template is called to perform upgrading processing on the to-be-upgraded data; in the process of the upgrading processing, corresponding processing progress information and execution log information are actively pushed to the visualization platform through a reporting mechanism defined in the data upgrading business logic template, so that the visualization platform performs visualized monitoring on the upgrading processing process of the to-be-upgraded data; by introducing the predefined data upgrading business logic template, a general process and core interface of data upgrading are solidified into a standardized algorithm skeleton, so that a developer only needs to focus on the implementation of specific business logic, thereby being able to greatly improve development efficiency, code standardization and maintainability; meanwhile, the method automatically selects the single-thread or multi-thread batch execution mode through the decision logic embedded in the template based on real-time data volume, and can significantly optimize processing performance and throughput in a large data volume scenario; in addition, with the help of the reporting mechanism defined by the template, progress and log information in the upgrading process are actively and real-timely pushed to the visualization platform, so that transparent monitoring and fine management of the entire upgrading process are realized; finally, the synergistic effect of the above technical features builds a standardized and toolized upgrading process from instruction issuing to execution feedback, which can effectively reduce configuration errors and execution risks caused by randomness of manual operation, and provides a systematic solution for safe, efficient and controllable execution of data upgrading tasks.

[0020] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will in part be apparent to those of ordinary skill in the art upon examination of the following or can be learned from a practice of the application. The advantages and objects of the application can be realized and obtained by means of the instrumentalities particularly pointed out in the specification and claims.

[0021] Those of skill in the art will appreciate that the objects and advantages of the application can be implemented in a variety of ways, and that the application can be implemented using other structures besides those specifically described herein. The present application is neither limited to combination of elements in a particular way nor to exclusive combination of all the elements disclosed. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. The components in the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. For purposes of clarity and understanding, it is also to be understood that certain portions of the drawings can be exaggerated and others omitted in order to more clearly depict the structural and functional aspects of some embodiments of the present application. In the drawings: Figure 1 A first flowchart of a data upgrade method in an embodiment of the present application.

[0023] Figure 2 A second flowchart of a data upgrade method in an embodiment of the present application.

[0024] Figure 3 A first flowchart of a data upgrade management method in an embodiment of the present application.

[0025] Figure 4 A second flowchart of a data upgrade management method in an embodiment of the present application.

[0026] Figure 5 A structure diagram of a data upgrade system in an embodiment of the present application.

[0027] Figure 6 A system architecture diagram of an automated visual data upgrade method in an application example of the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the embodiments and drawings. Here, the illustrative embodiments of the present application and their descriptions are used to explain the present application but not to limit the present application.

[0029] It should also be noted that, in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0030] It should be emphasized that the terms "comprises / comprising" when used in this specification, specify the presence of stated features, elements, steps or components, but do not preclude the presence or addition of one or more other features, elements, steps or components.

[0031] It should also be noted that, if not specifically stated, the term "connected" in this specification can not only mean direct connection but also mean indirect connection with an intermediate object.

[0032] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0033] To solve the problems of low development efficiency, poor standardization, high code redundancy, prominent processing performance bottleneck in large data scenarios, and lack of visual monitoring in the conventional data upgrade method, the embodiments of the present application respectively provide a data upgrade method, a data upgrade management method, a data upgrade system, an electronic device, a computer readable storage medium, and a computer program product.

[0034] The embodiments are specifically described as follows.

[0035] Based on this, the embodiments of the present application provide a data upgrade method executable by a data upgrade program, as shown in Figure 1 The data upgrade method specifically includes the following contents. Step 100: According to the start upgrade instruction received from the visualization platform, a query method in the predefined data upgrade business logic template is called to perform a paging query and obtain the data to be upgraded.

[0036] In one or more embodiments of the present application, the visualization platform refers to a data upgrade function module in a continuous delivery platform or an operation and maintenance platform, specifically a cloud shuttle platform. The platform provides a graphical user interface for centralized management, execution control, and state monitoring of data upgrade tasks. Its core functions include but are not limited to: user input task information, task dependency management, one-key start, one-key upgrade, one-key verification, and one-key shutdown operation buttons, and real-time display of task execution progress, log details, and success or failure data statistics in a graphical manner.

[0037] In an example, the visualization platform can specifically use a cloud shuttle platform, which refers to a data upgrade function module in a continuous delivery platform, providing a visual interface for managing, executing, and monitoring data upgrade tasks to realize one-key deployment, start, verification, and shutdown operations.

[0038] It should be noted that the start upgrade instruction can refer to a standardized command issued by the visualization platform to the data upgrade program to trigger the execution of a data upgrade task. The start upgrade instruction is usually transmitted through the hypertext transfer protocol or the representational state transfer application programming interface. On the data upgrade program side, receiving the instruction is the trigger condition for starting the subsequent standardized processing flow.

[0039] In one or more embodiments of the present application, the data upgrade business logic template is a functional definition of a standardized processing flow for data upgrade tasks. The specific technical implementation is embodied as a reusable software development framework, namely a data upgrade scaffolding (for example, the upgrade-starter component package provided in the form of a Java library). The scaffolding can adopt the template method pattern, a software design pattern, to solidify the general algorithm skeleton of data upgrade (query, decision, upgrade, report), and encapsulate general logic such as page control and multi-thread engine. Developers develop by inheriting the abstract classes (such as BaseUpgradeService) provided by the scaffolding, that is, by developing in compliance with the specifications of the business logic template.

[0040] It can be understood that the data upgrade business logic template can also be referred to as the template method pattern, which is a software design pattern that defines an algorithm skeleton in an operation, while delaying some steps to be implemented in a subclass. The data upgrade scaffolding refers to a standardized Java development framework (such as the upgrade-starter component package), which provides abstract classes (such as BaseUpgradeService and BaseUpgradeController) that specify the development mode and core interface of the data upgrade program. Developers can quickly complete the development of upgrade logic by inheriting and implementing these classes, without having to worry about the general process. The scaffolding fixes the execution flow of data upgrade (such as query, upgrade, and verification), and the specific implementation can be completed by a service unit. The service unit refers to a research and development team or organization unit responsible for the development and maintenance of specific business functions.

[0041] In an example, the data upgrade business logic template is provided in the form of a Java abstract class, and the query method, the upgrade method, the decision logic, and the reporting mechanism in the data upgrade business logic template are completed by inheriting the abstract class and implementing the predefined abstract methods or overriding the hook methods thereof. The data upgrade business logic template defines and encapsulates the following content: 1. Core component interface definition: Four core business interfaces that must be implemented by developers are defined, constituting the entry of customized upgrade logic: (1) Query method: defines a standard interface for obtaining data to be upgraded by page. Developers implement this method to provide specific business data query logic. The template internally encapsulates an automatic paging query loop controller that will call this method repeatedly until the data is fetched.

[0042] (2) Decision Logic: Defines the rules for selecting the execution strategy based on the amount of data. This logic can be a built-in comparison algorithm based on a preset threshold (e.g., 10,000 records), or it can be a method that developers can override. Its function is to output instructions for single-threaded execution mode or multi-threaded batch execution mode.

[0043] (3) Upgrade Method: Defines the core business interface for performing data transformation or update. Developers implement this method to write specific single or batch data upgrade logic. The template will automatically decide whether to directly call this method for sequential processing or to organize a thread pool through its built-in configurable multi-threaded execution engine to concurrently call the batch processing version of this method based on the output of the decision logic.

[0044] (4) Reporting Mechanism: Defines the communication protocol for reporting status to the visualization platform. This mechanism stipulates that the program must periodically (e.g., at fixed times or data processing intervals) collect processing progress and execution logs, encapsulate them in a standard format (e.g., JSON), and actively push them by calling the specified application programming interface provided by the visualization platform.

[0045] 2. Fixed algorithm framework and flow control: This template uses the template method pattern to solidify the four components into an immutable, standardized execution flow. Its fixed sequence of steps is: invoking the query method, applying the decision logic, invoking the upgrade method, and triggering the reporting mechanism. Developers cannot change this flow; they can only fill in the specific business implementation for each step, thus ensuring the consistency of behavior across all derived programs.

[0046] 3. Encapsulated general logic and configurable items: To implement the above process, the template pre-encapsulates key general control logic and configurable items, including: (1) Automatic pagination query loop controller: manages the loop calls to the developer's query methods.

[0047] (2) Configurable multi-threaded execution engine: responsible for thread pool initialization, task queue management and concurrent scheduling.

[0048] (3) Decision threshold parameter: As the basis for decision-making logic, it can be configured by the developer according to the scenario.

[0049] (4) Unified exception handling framework: Provides a standard exception capture, recording and processing mechanism.

[0050] (5) Interaction protocol with the visualization platform: The data reporting format, frequency, API endpoints and response methods to platform commands are clearly defined.

[0051] Through the above design, the data upgrade business logic template not only defines the development contract, but also encapsulates all general and complex process control logic. Developers only need to focus on implementing limited business interfaces to quickly build data upgrade programs that meet specifications, have efficient processing and visualization monitoring capabilities.

[0052] In the method, calling a query method to perform a paging query is the first step of the template fixed process, which aims to safely and efficiently obtain the data set to be upgraded.

[0053] The data to be upgraded refers to original business data that needs to be structurally changed, content corrected, or business logic converted. These data are obtained from the production environment database of an enterprise-level application system through a paging query.

[0054] Step 200: Based on the data volume of the data to be upgraded, automatically select a target execution mode through the decision logic defined in the data upgrade business logic template, the target execution mode including: single-thread execution mode or multi-thread batch execution mode.

[0055] In step 200, the data volume refers to the total number or total size of the data to be upgraded obtained through the query method in this data upgrade task. This value is a key judgment basis for the subsequent decision logic to select the execution mode. Automatic selection means that the execution of the decision logic does not require human intervention, and the program makes a judgment and determines the execution mode dynamically according to real-time data during running.

[0056] Among them, the single-thread execution mode is suitable for scenarios with small data volume, and adopts a single thread to process each data sequentially. The multi-thread batch execution mode is suitable for scenarios with large data volume, and initializes a thread pool to use multiple threads to concurrently process data divided into multiple batches to fully utilize system resources and significantly improve processing speed.

[0057] Step 300: According to the target execution mode, call the upgrade method in the data upgrade business logic template to upgrade the data to be upgraded.

[0058] The upgrade processing refers to a set of information reporting functional components embedded in the data upgrade business logic template. This mechanism is responsible for collecting the processing progress information (such as the number of successes and failures) and detailed execution log information of the current task periodically (such as at fixed time intervals or fixed data processing quantity intervals) during the upgrade processing.

[0059] Step 400: During the upgrading process, the corresponding processing progress information and execution log information are actively pushed to the visualization platform through the reporting mechanism defined in the data upgrading business logic template, so as to visually monitor the upgrading process of the to-be-upgraded data by the visualization platform.

[0060] The active pushing refers to the working mode of the reporting mechanism, that is, the data upgrading program actively sends the collected progress and log information out through the special application program interface provided by the visualization platform by network calling, instead of waiting for the platform to poll and query. This mechanism is the key to realizing real-time monitoring.

[0061] The processing progress information can refer to data reflecting the current execution state of the data upgrading task, and generally includes the number of processed data, the number of successful data, the number of failed data, and the overall progress percentage, etc. The execution log information can refer to detailed running records generated in the upgrading process, including informative logs, warning logs and error exception stack information, etc., which are used for problem troubleshooting and process tracing.

[0062] The visual monitoring refers to that after receiving the information actively pushed by the data upgrading program, the visualization platform displays it on the interface in the form of graphical elements (such as progress bar, statistical chart) and real-time text stream (log panel), so that the operation and maintenance personnel can intuitively and real-timely understand the execution state, progress details and running situation of the upgrading task, thereby realizing the transparent and controllable management of the entire upgrading process.

[0063] From the above description, it can be seen that the data upgrading method provided by the embodiments of the present application can greatly improve the development efficiency, code standardization and maintainability by introducing the predefined data upgrading business logic template, solidifying the general process and core interface of data upgrading into a standardized algorithm skeleton, and making the developer only need to focus on the implementation of specific business logic. At the same time, the method can significantly optimize the processing performance and throughput in the large data volume scenario by automatically selecting the single-thread or multi-thread batch execution mode based on the real-time data volume through the decision logic embedded in the template. In addition, with the help of the reporting mechanism defined by the template, the progress and log information in the upgrading process can be actively and real-timely pushed to the visualization platform, realizing the transparent monitoring and fine management of the entire upgrading process. Finally, the synergistic effect of the above technical features constructs a standardized and toolized upgrading process from instruction to feedback, which can effectively reduce the configuration errors and execution risks caused by the randomness of manual operation, and provides a systematic solution for the safe, efficient and controllable execution of data upgrading tasks.

[0064] In order to further solve the problems of cumbersome and error-prone configuration management, high operation risk and lack of standardized security pre-process in the prior art, and realize configuration automation, risk controllability and process standardization, in a data upgrading method provided in an embodiment of the present application, referring to Figure 2 , the step 100 of the data upgrading method further specifically comprises the following content before the step 100: Step 010: dynamically obtaining database connection configuration information corresponding to the current running environment from the configuration center.

[0065] The configuration center refers to an independent service or platform for centralized management, storage and distribution of application configuration information. In an example, the configuration center can adopt Nacos (a dynamic service discovery, configuration and service management platform). Its function is to realize unified management, environment isolation (such as distinguishing test, pre-release and production environment) and dynamic update of database connection string and other configuration information, avoid hard coding configuration in the program or scattering on multiple servers, and thus solve the problems of cumbersome and error-prone configuration management.

[0066] The dynamic acquisition in step 010 refers to that the data upgrading program actively and in real time pulls or receives the configuration information pushed by the configuration center during the startup or running process, rather than using the static configuration determined at compile time. This way makes the program adapt to different deployment environments, and can take effect without re-deploying the program when the configuration changes. The current running environment refers to the specific software and hardware context in which the data upgrading program is deployed and executed, which is usually divided according to different research and development and release stages, such as development environment, test environment, pre-release environment and production environment. Each environment usually corresponds to different database instances, service endpoints and configuration parameters.

[0067] The database connection configuration information refers to a set of parameters necessary for establishing a communication connection with a specific database. The database connection configuration information usually can include the network address (such as IP and port) of the database server, the database instance name, the identity authentication information (such as username and password or access token), and possibly the connection pool parameters (such as initial connection number, maximum connection number).

[0068] Step 020: initializing and establishing the connection between itself and the target database based on the database connection configuration information.

[0069] In step 020, the data upgrading program creates and configures the database connection pool or single database connection object using the database connection configuration information obtained from the configuration center. This process includes loading the database driver, verifying the configuration validity, establishing the network session and other steps, so that the program can subsequently perform structured query language (SQL) operations.

[0070] Step 030: performing a pre-health check, wherein the pre-health check comprises at least one of checking availability of a dependent service, checking whether resource usage of a target server is lower than a first preset threshold, and checking database connectivity.

[0071] Specifically, before formally starting to execute the core business logic of data upgrade (i.e., calling the query method), a series of verification operations automatically performed by the data upgrade program aims to ensure that its dependent runtime environment is in a healthy and ready state, so as to reduce the risk of upgrade failure. This check is a key step to improve operation safety.

[0072] The availability of the dependent service refers to whether the external microservice or system interface that the data upgrade program depends on and provides key functions for it is in a state of normal response to requests. The checking method is usually to call the health check endpoint or heartbeat interface of the service through network (such as HTTP request), and judge according to the return status code or content.

[0073] The resource usage of the target server refers to the proportion of the current consumption of key computing resources (such as central processing unit (CPU), memory) of the host server (i.e., target server) running the data upgrade program to the total resources. High resource usage may cause slow or failed program running.

[0074] The first preset threshold refers to a critical value preset for judging whether the resource usage is within an acceptable safe range. For example, the threshold of central processing unit (CPU) usage can be set to 30%, if the actual usage is found to be higher than this value, it is considered that the resource is tight, which may be suspended or alarmed.

[0075] The database connectivity refers to whether the network link and authentication session between the data upgrade program and the target database are smooth and effective. A simple checking method includes executing a test statement that does not involve business data, such as "SELECT 1".

[0076] Step 040: receiving the start upgrade instruction sent by the visualization platform.

[0077] Step 050: performing idempotency check on the start upgrade instruction, and if the start upgrade instruction passes the idempotency check, marking it as a start upgrade instruction that has passed the check.

[0078] The idempotency check refers to a verification of the received start-up upgrade instruction to ensure that the same upgrade task, even if repeatedly submitted or executed multiple times, has the same result as executing it once, thereby avoiding errors such as repeated data processing. The core is to identify the unique identifier (such as task ID) of the instruction and query the historical execution state of the task.

[0079] The start-up upgrade instruction that has passed the check refers to the start-up upgrade instruction that has successfully passed the aforementioned pre-health check and idempotency check. This instruction is considered legal, safe, and allowed to be executed, and is a prerequisite for triggering the subsequent core data upgrade process.

[0080] Correspondingly, the step 100 of the data upgrade method specifically includes the following content: Step 110: According to the start-up upgrade instruction received from the visualization platform and having passed the check, a query method in the predefined data upgrade business logic template is called to send a SQL query statement with a paging parameter to the target database.

[0081] Specifically, the paging parameter refers to a control parameter used to specify the range of data subsets when performing a paging query. It usually includes the page number (or offset) and the number of records per page (i.e., page size). By changing these parameters, all data can be obtained in a loop. The SQL (Structured Query Language) query statement refers to a standard programming language statement used to manage and operate on a relational database system (RDBMS). In this scenario, it specifically refers to a SELECT statement constructed by the data upgrade program to read the data to be upgraded from the target database in pages. This statement will combine the paging parameter (such as using the LIMIT and OFFSET keywords or their database equivalent syntax) to ensure that only one page of data is obtained each time.

[0082] Step 120: Receive the data to be upgraded returned by the target database that meets the SQL query statement.

[0083] In order to further solve the core technical problems of low processing performance and insufficient resource utilization in the existing technology in the large data volume scenario, and to achieve significant improvement in processing efficiency and adaptive optimization of execution strategy, in the data upgrade method provided in the embodiment of the present application, the decision logic is configured to compare the data volume of the data to be upgraded with a second preset threshold, and select the target execution mode among the preset execution modes according to the comparison result, and correspondingly, see Figure 2 , the step 200 of the data upgrade method specifically includes the following content: Step 210: Compare the data volume of the data to be upgraded with a second preset threshold.

[0084] Step 220: If the data volume is less than or equal to the second preset threshold, the single-thread execution mode is selected.

[0085] Step 230: If the data volume is greater than the second preset threshold, the multi-thread batch execution mode is selected, and the thread pool is initialized based on the parameters configured in the data upgrade business logic template for initializing the thread pool.

[0086] The second preset threshold is a predefined data volume critical value for distinguishing different execution mode application scenarios. The threshold is a configurable parameter, for example, ten thousand. Developers or operation and maintenance personnel can adjust this threshold according to actual hardware performance, business tolerance time, and other factors to optimize the balance between performance and resource use.

[0087] Specifically, the initialization of the thread pool is a process of creating and preparing a thread pool instance in computer memory. The thread pool is a software component for managing and reusing multiple threads to execute concurrent tasks. Initialization includes setting the core properties of the pool, such as creating a certain number of idle worker threads, defining the task queue, setting the thread survival time, and the like, so that it is in a state of receiving and executing batch tasks. This is the basic preparation work for executing the multi-thread batch processing mode. The parameters for initializing the thread pool refer to a series of configuration values required when initializing the thread pool, which determine the behavior and performance characteristics of the thread pool. Typical parameters can include: core thread number (minimum number of threads maintained by the thread pool), maximum thread number (maximum number of threads allowed to be created by the thread pool), task queue capacity (queue size for caching pending tasks), and thread idle survival time, and the like. These parameters can be pre-configured in the data upgrade business logic template for the execution engine to read and use when needed.

[0088] To further solve the technical problems of unclear execution mode switching and lack of standardized process for multi-thread processing in the prior art, and achieve the clearification and efficiency of the specific operation paths of the two execution modes, in an embodiment of the present application, a data upgrade method is provided, as shown in Figure 2 The step 300 of the data upgrade method specifically includes the following content: Step 310: If the target execution mode is the single-thread execution mode, the upgrade method defined in the data upgrade business logic template is directly called to sequentially upgrade the to-be-upgraded data.

[0089] Direct invocation refers to synchronously executing the specified method (here, the upgrade method) in the current thread (usually, the main thread that receives and processes the upgrade instruction). This approach does not involve task dispatching or thread switching and is suitable for scenarios with relatively simple processing logic or small data volumes. Sequential upgrade processing refers to sequentially invoking the upgrade method to process each piece or small unit of data in the order in which the data is obtained (e.g., the order returned by a database query) in a single-thread execution mode. This approach has the characteristics of a single execution path and no concurrent conflicts, but the total processing time is proportional to the data volume.

[0090] Step 320: If the target execution mode is the multi-thread batch execution mode, divide the data to be upgraded into multiple batches of data, and submit each batch of data to the thread pool for concurrent upgrade processing, wherein each thread in the thread pool invokes the upgrade method for batch data defined in the data upgrade business logic template to upgrade one batch of data.

[0091] Dividing the data into multiple batches means logically dividing the obtained data to be upgraded into smaller data subsets according to a preset batch size (e.g., 1000 pieces per batch). This is a key step in splitting a large task into multiple independent subtasks that can be executed in parallel and is the basis for multi-thread batch processing. Submitting to the thread pool means packaging each batch of data into an executable task unit (Runnable or Callable) and placing it in the internal task queue of the thread pool. The manager of the thread pool will take the task from the queue and assign it to an idle worker thread in the pool for execution.

[0092] Concurrent upgrade processing refers to activating multiple worker threads in the thread pool simultaneously in the multi-thread batch execution mode, each independently processing the data batch assigned to it. The execution of these threads overlaps in time, allowing multiple data batches to be processed within the same time period, significantly reducing the overall processing time and improving the performance of large data processing.

[0093] Specifically, the upgrade method for batch data refers to a specific form or overloaded version of an upgrade method defined in the data upgrade business logic template. Unlike the upgrade method for processing a single piece of data, this method is designed to receive a list of data (i.e., a batch of data) as an input parameter and implement loop processing or batch database operations on all data in the batch. Developers define the batch upgrade business logic by implementing this method. The template framework ensures that an independent batch of data is passed to this method when invoked by each thread.

[0094] To further solve the technical problems of non-transparent operation and maintenance process and poor experience caused by non-timely and irregular reporting of monitoring information in the prior art, and achieve standardization, real-time and efficient resource utilization of monitoring information reporting, in a data upgrading method provided in the embodiments of the present application, referring to Figure 2 , step 400 of the data upgrading method specifically includes the following content: Step 410: According to a fixed time interval or a fixed data volume interval specified by a reporting mechanism in the data upgrading business logic template, an application program interface provided by the visualization platform is called to actively push the current processing progress information and the corresponding execution log information of the upgrading process to the visualization platform.

[0095] Specifically, the fixed time interval refers to a pre-set constant time period, for example, every 1 second or every 5 seconds. In the reporting mechanism of the present application, the information reporting operation can be triggered periodically according to the fixed time period, so as to realize the quasi-real-time monitoring of the upgrading progress. The fixed data volume interval refers to a pre-set constant data processing quantity unit, for example, every 100 or 1000 successfully processed data. In the reporting mechanism of the present application, the information reporting operation is triggered once the cumulative processing data volume reaches the fixed quantity. In this way, the reporting frequency is associated with the processing speed, and the actual work progress can be better reflected.

[0096] The application program interface refers to a set of functions, protocols or endpoints pre-defined and provided by the visualization platform. It specifies how external programs (such as data upgrading programs) should request the services of the platform or exchange data with it. In the present solution, the visualization platform provides an application program interface (API) dedicated to receiving progress and log reporting, and the data upgrading program completes the information pushing by calling the interface.

[0097] The calling in step 410 can refer to the process of executing a specific function, method or accessing an application program interface (API) in a computer program through code. In the present step, the data upgrading program executes network request code to send a request containing progress and log data to the specified application program interface (API) address of the visualization platform.

[0098] The active pushing is a mode of action in which the producer of information (here, the data upgrading program) initiates the sending of data to the consumer (here, the visualization platform) when the conditions are met (such as time or data volume). This mode is opposite to the mode of repeated inquiry (polling) and is a key technical feature to realize the whole-process visualization and traceability of the upgrading process, making the execution process transparent and controllable. It enables the visualization platform to respond immediately and display the latest status without waiting.

[0099] To further solve the technical problems of high data quality risk and insufficient operation integrity caused by the lack of post-validation link and the unclosed operation process in the prior art, and to realize the controllability of upgrade quality and the closed loop of operation process, in a data upgrade method provided in an embodiment of the present application, referring to Figure 2 , the step 400 of the data upgrade method further specifically contains the following content: Step 500: After the upgrade processing is completed, a verification method defined in the data upgrade business logic template is called to perform business rule checking on the upgraded data, and verification result information is obtained.

[0100] Specifically, the verification method refers to another core business method, for example, the validateUpgradeData method, which is predefined in the data upgrade business logic template and implemented by the developer. The method is implemented by the developer within the template framework, and its responsibility is to check the data that has completed the upgrade processing for correctness, integrity, consistency, and the like according to business requirements. The business rule checking can refer to the review process of the data according to the logic and constraint conditions of a specific business field. In this scenario, it specifically refers to checking whether the conversion result of the data meets the expected business logic by executing the above verification method after the data upgrade operation is completed. For example, checking whether the total sum of the amount field is balanced after upgrade, or whether the conversion of the state field meets the established workflow rules.

[0101] Among them, the verification result information is the conclusion data generated after the business rule checking. The information usually contains a state flag indicating whether the checking is passed as a whole, and specific details such as a list of abnormal data found, error type description, and data record identifier involved. This information is a key basis for evaluating the final quality and reliability of this data upgrade task.

[0102] Step 600: Push the verification result information to the visualization platform.

[0103] Specifically, after the verification process is completed, the data upgrade program can send the generated verification result information to the visualization platform through its reporting mechanism or similar communication channels. This enables the operation and maintenance personnel to directly view the overall quality evaluation of the upgrade task on the interface of the platform, completing the full-link information closed loop from upgrade execution to result verification, and is an important link to realize the traceability of the upgrade process and the operation integrity.

[0104] In summary, the data upgrade method provided in the embodiments of the present application can be used to solve the following problems existing in the manual SQL script method and the customized development application program method: 1) Complex deployment, prone to errors: a large amount of database connection information needs to be manually configured, which is easy to miss or misconfigure.

[0105] 2) Lack of standardized procedures: upgrade steps, rollback mechanisms, and log recording methods vary from person to person, lack of unified standards, and poor traceability.

[0106] 3) Low efficiency: for large data volume (more than 10,000) scenarios, single-thread processing is usually used, which takes a very long time to execute, affecting the online window.

[0107] 4) Lack of visual management: upgrade progress, success / failure data, and real-time logs cannot be visually viewed, and developers rely heavily on server logs for troubleshooting, resulting in poor user experience.

[0108] 5) High risk: lack of pre-checks (such as dependent services and server resources), which can lead to upgrade failures due to inadequate preparation.

[0109] Thus, the data upgrade method provided by the embodiments of the present application can have the following beneficial effects: 1) Simplify development process, improve development efficiency and quality: by providing a standardized data upgrade scaffold, encapsulating common logic, developers only need to focus on core business data operations, enabling efficient and standardized development.

[0110] 2) Achieve full-process visualization and traceability of the upgrade process: provide a visual operation interface through the cloud shuttle platform, real-time display of upgrade progress, logs, success / failure data, making the entire upgrade process transparent and controllable.

[0111] 3) Significantly improve the processing efficiency of large data volume upgrades: through the multi-threaded batch processing mechanism built into the scaffold, and allowing developers to flexibly choose single-threaded or multi-threaded mode based on data volume, significantly reducing upgrade time.

[0112] 4) Implement centralized and automated management of configurations: integrate configuration centers such as Nacos to achieve automatic injection of database connection information and environment isolation, avoiding manual configuration errors.

[0113] 5) Build a standardized and tool-based secure upgrade process: solidify the upgrade process into the platform, integrate pre-checks (health checks, resource checks), idempotency control, and emergency rollback mechanisms to reduce human operational risks and ensure data security.

[0114] Based on the above data upgrade method, the present application further provides a data upgrade management method that can be executed by a visual platform, which interacts with the data upgrade program to complete data upgrade management and monitoring, as described in Figure 3 , the data upgrade management method specifically includes the following contents: Step 10: According to the execution order dependency relationship of each data upgrade task, the service state and server resource indicators dependent on each data upgrade program are sequentially obtained and checked by calling the monitoring system interface, and after the check is passed, the corresponding start upgrade instruction is sent to each data upgrade program in sequence to trigger each data upgrade program to execute the data upgrade method.

[0115] The execution order dependency relationship of each data upgrade task refers to the pre-configured execution constraint logic between multiple data upgrade tasks in the visualization platform. For example, task B must start after task A is successfully completed. This dependency relationship can be declared by the developer in the program through annotations (such as "@ExecutionOrder" annotation), and recognized and followed by the platform when scheduling to ensure the logical correctness of the upgrade process.

[0116] The monitoring system interface is an application programming interface provided by an independent monitoring system (such as a cloud intelligence monitoring system) for querying the health status of infrastructure and application. The visualization platform can obtain real-time resource indicators (such as central processing unit (CPU) and memory usage) of the target server and health status information such as whether the dependent microservice is available by calling this interface.

[0117] The sequential acquisition and checking refers to the pre-environmental check performed by the visualization platform according to the order determined by the above-mentioned task dependency relationship. For each task to be started, the platform first obtains the state of the dependent service and the server resource indicators through the monitoring system interface, and performs judgment (for example, checks whether the central processing unit (CPU) usage is lower than the preset threshold of thirty percent), and only after the check is passed, the subsequent operation is performed. This is a kind of serialized security access control process.

[0118] The aggregation refers to the process of receiving, collecting, merging and arranging the processing progress information and execution log information with the same format sent from multiple independently running data upgrade programs. The purpose is to integrate the scattered and concurrent information flow into a unified data view that can be centrally displayed and monitored.

[0119] Step 20: Receive and aggregate the processing progress information and the execution log information actively pushed from the data upgrade program.

[0120] Step 30: Display the processing progress information and the execution log information in a graphical interface.

[0121] It can be understood that the data upgrade method in step 10 adopts the data upgrade method as shown in Figure 1 or Figure 2The flow implementation of the illustrated data upgrade method can refer to the foregoing embodiments of the data upgrade method, and will not be described here again.

[0122] The graphical interface display can refer to the processing progress information (such as the number of successes and failures, the progress percentage) and the execution log information obtained by the visualization platform, which are presented to the user in the form of non-pure text visual elements through a webpage or a client application program. This includes but is not limited to the use of progress bars, statistical charts, data tables, and real-time scrolling log text areas. This method is the core of realizing visual management and improving the intuitiveness and experience of operation and maintenance.

[0123] That is, the data upgrade management method provided by the embodiments of the present application solves the technical problems of chaotic multi-task management, lack of unified scheduling and safety control, and non-transparent operation and maintenance process in the prior art, and realizes the ordering and automation of task scheduling and the centralized visual monitoring of the upgrade process.

[0124] In order to further solve the technical problems of complicated, labor-dependent and error-prone data upgrade program deployment and registration process in the prior art, and realize the automation, standardization and traceability of program package management, in the data upgrade management method provided by the embodiments of the present application, referring to Figure 4 , the step 30 in the data upgrade management method further specifically contains the following content: Step 40: receiving a data upgrade program package developed based on the data upgrade business logic template based on a preset task management interface, and automatically parsing the data upgrade program package, identifying and registering the data upgrade tasks and their calling interfaces defined in the data upgrade program package.

[0125] Specifically, the preset task management interface refers to a user interaction interface specially used for managing data upgrade tasks, which is developed and provided in advance in the visualization platform (such as the cloud shuttle platform), such as the corresponding page of the project and program management module. The interface provides graphical operation entrances for functions such as uploading program packages and viewing registered tasks for users (such as operation and maintenance personnel or developers). Then the user transmits the locally stored data upgrade program package file to the visualization platform server through the task management interface. This is the first step to introduce the developed data upgrade program into the platform management system.

[0126] Among them, the data upgrade program package refers to the deployable file formed after the development based on the data upgrade business logic template (scaffold) is compiled and packaged. It usually refers to the Java archive (JAR) file, which contains the bytecode, dependent library and metadata information of the program. It is the executable carrier of the data upgrade task in the platform.

[0127] The automatic parsing refers to a process that the visualization platform automatically scans, reads and analyzes the internal structure of the data upgrade package (JAR file) uploaded by the user without manual intervention after receiving the data upgrade package. This process usually uses mechanisms such as Java reflection to extract key information related to platform interaction in the program.

[0128] The identification refers to a process that the platform identifies specific components developed based on the data upgrade business logic template in the package by reading the metadata (such as class files, annotations) of the package during the automatic parsing process, especially identifying classes inherited from the base upgrade controller (BaseUpgradeController) and the application program interfaces provided by these classes that meet the standard specifications. The registration refers to a process that the platform persistently stores the identified data upgrade task information (such as task identification, corresponding application program interface path, execution order, etc.) into the database or configuration storage of the platform, so that it becomes a perceivable, schedulable and manageable entity object. After completing the registration, the task appears in the task list of the platform, which can be executed and monitored subsequently. This is a key step to realize automatic discovery, deployment and invocation.

[0129] Based on the foregoing embodiments of the data upgrade method and the data upgrade management method, the application further provides an embodiment of a data upgrade system, which is shown in Figure 5 The data upgrade system specifically includes the following contents: At least one data upgrade program 1 configured to execute the data upgrade method provided in the foregoing embodiments; A visualization platform 2 configured to execute the data upgrade management method provided in the foregoing embodiments; A configuration center 3 configured to provide dynamic database connection configuration information to the data upgrade program; The data upgrade program and the visualization platform communicate through an application program interface defined based on a standardized protocol.

[0130] The data upgrade program 1 in the embodiment of the data upgrade system provided by the application can be specifically used to execute the processing flow of the embodiment of the data upgrade method in the foregoing embodiments, and the functions thereof will not be described here again, and can be referred to the detailed description of the foregoing data upgrade method embodiment.

[0131] The visualization platform 2 in the embodiment of the data upgrade system provided by the application can be specifically used to execute the processing flow of the embodiment of the data upgrade management method in the foregoing embodiments, and the functions thereof will not be described here again, and can be referred to the detailed description of the foregoing data upgrade management method embodiment.

[0132] The part of the data upgrade procedure 1 for data upgrade can be completed in the client device. Specifically, the selection can be made according to the processing capability of the client device and the restriction of the user's use scenario, etc. The present application does not limit this. If all operations are completed in the client device, the client device can further include a processor for specific processing of data upgrade.

[0133] The above client device can have a communication module (i.e., a communication unit) and can be communicatively connected with a remote server to realize data transmission with the server. The server can include a server of the task scheduling center side, and in other implementation scenarios, can also include a server of an intermediate platform, such as a server of a third-party server platform communicatively connected with the server of the task scheduling center. The server can include a single computer device, or a server cluster composed of multiple servers, or a server structure of a distributed device.

[0134] The server and the client device can use any suitable network protocol for communication, including a network protocol not yet developed at the filing date of the present application. The network protocol can include, for example, a TCP / IP protocol, a UDP / IP protocol, an HTTP protocol, an HTTPS protocol, etc. Of course, the network protocol can also include, for example, a RPC protocol (Remote Procedure Call Protocol) used on the above protocols, a REST protocol (Representational State Transfer), etc.

[0135] As can be seen from the above description, the data upgrade system provided by the embodiments of the present application solves the systematic technical problem that the tools and components of each link of data upgrade in the prior art are isolated and dispersed, cannot work collaboratively, and leads to low overall efficiency and difficult global control of risks, and realizes integrated integration of all components, standardized collaboration, and end-to-end automatic management.

[0136] In order to further illustrate the above embodiments, the present application further provides an application example of an automatic visual data upgrade method, specifically including an automatic and visual method and system for realizing large-scale data migration, conversion or upgrade in an enterprise-level application system. Referring to Figure 6 In the system architecture, the data upgrade procedure, the visual platform and the configuration center are still included, wherein the visual platform can adopt a cloud shuttle platform (visual control layer), and the data upgrade procedure can be arranged in a data upgrade server (application execution layer). The system architecture further has an infrastructure layer for storing a database and a configuration center, and the core is the combination of scaffold standardized development and platform visual operation and maintenance.

[0137] First of all, the existing data upgrade technology has the following problems: 1) Low development efficiency and uneven quality: Customized development method leads to repetitive work, and each upgrade program needs to write general logic (such as paging query, transaction management) from scratch, and the code style is different, which is difficult to maintain.

[0138] 2) Execution process is not transparent, and there is no visual monitoring: The execution progress, detailed log, and exception information cannot be fed back to the operation and management personnel in real time, and it is difficult to locate the problem when it occurs, and the process is uncontrollable.

[0139] 3) Performance bottleneck is prominent when processing large data: Traditional single-thread processing method cannot effectively utilize system resources, and the upgrade speed is slow, which cannot meet the requirements of modern systems for efficient online.

[0140] 4) Configuration management is complicated and prone to errors: Database connection and other configuration information are hard-coded in the program or need to be manually maintained, which is prone to errors and difficult to maintain consistency between different environments (test, pre-release, production).

[0141] 5) Non-standardized operation process, high risk: The whole process from deployment, execution to verification lacks standardized tools and mandatory checkpoints (such as dependency check, resource check), and relies too much on manual experience, with high operation risk.

[0142] In order to solve the above problems, the basic principle of the application of the example is to adopt the design idea of separation of concerns and template method pattern.

[0143] For the development side: Abstract and encapsulate the general and reusable logic in the data upgrade process (such as task scheduling, paging query, multi-thread execution, progress tracking, exception handling framework) into a unified data upgrade scaffold (Java SDK). Developers only need to inherit the abstract class provided by the scaffold and implement a small number of core business methods (such as queryData(), upgrade()) to quickly generate a standardized data upgrade program. This solves the problem of uneven development efficiency and quality.

[0144] For the operation side: Build a visual cloud shuttle platform as a unified management center for data upgrade tasks. The platform can automatically scan, deploy and run data upgrade program JAR packages based on the scaffold, and interact with these programs through standard RESTful API. The platform collects, aggregates and visualizes the log, progress and exception information in the execution process in real time, and integrates health check, idempotent control and other operation capabilities, thereby solving the problems of opaque execution process and high operation risk.

[0145] In particular, the technical solution mainly includes two core components: data upgrade scaffolding and cloud shuttle data upgrade platform. The specific description is as follows: (I) Component 1: Data upgrade scaffolding (standardized development framework) The scaffolding is provided in the form of a Java library, such as a component package named upgrade-starter. Its core lies in defining a standardized programming paradigm and algorithm skeleton for data upgrade, which is specifically implemented through the following abstract classes: 1. BaseUpgradeService abstract class <t> Action: Define the core business logic template of data upgrade, and use the template method pattern to fix the processing flow.

[0146] Core method (to be implemented by the developer): (1) Query data (QueryResult <t>`queryData()`: Used to query the data to be upgraded in a paginated manner. The framework will automatically call this method in a loop until all data has been processed.

[0147] (2) Upgrade method: Single item upgrade (void upgrade(T item)): Executes the upgrade logic for a single item.

[0148] Batch upgrade (void upgradeByBatch(List <t>(3) validateUpgradeData (ResponseInfo): defines the data validation logic after upgrade.

[0149] (3) validateUpgradeData (ResponseInfo): defines the data validation logic after upgrade.

[0150] (4) chooseUpgradeType (): returns an execution mode enumeration value based on the amount of data to be processed, such as single-threaded upgrade (SINGLE_UPGRADE, suitable for scenarios where the data volume is less than 10,000) or batch upgrade (BATCH_UPGRADE, suitable for scenarios where the data volume is greater than 10,000). The scaffolding will automatically select the execution mode based on this return value.

[0151] Built-in innovation mechanism: the abstract class internally encapsulates a multi-threaded execution engine, an automatic paging query loop controller, and a unified exception handling framework. When the chooseUpgradeType() method returns the batch upgrade (BATCH_UPGRADE) mode, the framework will automatically initialize a thread pool and concurrently call the batch upgrade upgradeByBatch method implemented by the developer, thereby significantly improving the processing efficiency under large data volumes.

[0152] 2. BaseUpgradeController abstract class: Function: provides a standard HTTP interface for the cloud shuttle platform to call, achieving decoupling and collaboration between the program and the platform.

[0153] Core interfaces include: " / startUpgrade": used to start the upgrade task.

[0154] " / queryProgress": used to query the upgrade progress (such as the number of successes and failures).

[0155] " / validateData": used to trigger data validation.

[0156] Advanced features: (1) supports defining the execution order of interfaces through annotations (such as @ExecutionOrder), and the platform can control the execution flow of multiple upgrade tasks with dependencies based on this; (2) the controller internally integrates an idempotency verification mechanism, which can prevent the same upgrade task from being accidentally started due to repeated submission, ensuring the accuracy of data operations.

[0157] (II) Component 2: Cloud Shuttle Data Upgrade Platform (Visual Operation and Management Platform) The platform is a network Web application system, which is the unified management hub of data upgrade tasks, mainly including the following functional modules: 1. Project and program management module: The service unit (R&D team) can upload the data upgrade program Java archive file package developed based on the above scaffolding through the platform interface.

[0158] The platform can automatically scan the uploaded Java archive file and identify the control layer class and its interface information (such as interface path, execution order annotation) based on the scaffolding specification, and complete the automatic registration of the task.

[0159] 2. Visual task execution console: Pre-check: Before task execution, the platform automatically calls the interface of the external monitoring system (such as "Cloud Intelligence" monitoring) to check the basic resources of the target server (such as whether the central processing unit CPU and memory usage are below the 30% threshold) and whether the dependent microservices are healthy, ensuring that the upgrade environment is ready and effectively reducing the operation risk.

[0160] One-key operation: The platform provides standardized operation controls such as "one-key start", "one-key upgrade", "one-key verification", and "one-key stop". The background will automatically generate deployment scripts based on program Java archive file information and call the corresponding hypertext transfer protocol interface of the corresponding data upgrade program.

[0161] Real-time monitoring board: Real-time display of task status, progress percentage, success and failure data number in graphical form (such as progress bar, statistical chart). At the same time, integrate real-time log pushing function, users can directly view the scrolling detailed execution log on the webpage, realize the whole process visualization and transparency of the upgrade process.

[0162] Alarm mechanism: Integrated with the monitoring system, when errors or resource overruns occur during the upgrade process, automatically notify the responsible person through WeChat Enterprise or email, etc.

[0163] 3. Process and permission management: Support embedding data upgrade process into continuous delivery pipeline; Can set differentiated operation permissions according to different environments (test, pre-production, production), realize safety control.

[0164] (III) End-to-end workflow The standard process of the entire data upgrade task is as follows: (1) Development stage: the service unit implements business logic based on the data upgrade scaffold, generates a program Java archive file package, and completes program package uploading and task information input on the cloud shuttle platform.

[0165] (2) Deployment and start-up stage: the operation and maintenance personnel click "start" on the platform, and the platform automatically deploys the program package to the target server and runs the instance.

[0166] (3) Execution and monitoring stage: the operation and maintenance personnel click "upgrade", and the platform first performs pre-checking. After the check passes, the platform calls the " / startUpgrade" interface of the program to start the task. During task execution, the program actively pushes progress and logs to the platform dashboard through the reporting mechanism embedded in the scaffold.

[0167] (4) Verification and end stage: after execution is completed, the operation and maintenance personnel can trigger verification operations through the platform, and the platform calls the " / validateData" interface of the program. After verification is correct, click "stop", the platform calls the interface to stop the program, and completes the entire upgrade closed loop.

[0168] Through the cooperative work of the above two core components, the present scheme realizes the whole life cycle, standardization, and tooling management of the data upgrade task from development, deployment, execution to monitoring and verification.

[0169] In addition, the application example of the present application takes Java as an example, but the idea of the data upgrade scaffold is also applicable to other programming languages, such as using the abstract base class (ABC) and Flask framework of Python to realize a similar Python version of the scaffold, which can also achieve the purpose of standardized development. The application example of the present application uses Nacos as a configuration center, which can also be replaced by Consul, Apollo and other configuration management tools to realize centralized management of database connection information. The scaffold is built-in with a multi-threaded solution based on a thread pool. The alternative solution can be to use a more modern reactive programming model (such as ProjectReactor) to realize non-blocking asynchronous processing, also in order to improve throughput. The application example of the present application adopts the platform to generate scripts and deploy JAR packages. The alternative solution can be to use containerization technology (such as Docker) to package the data upgrade program into an image, and to deploy and manage it through Kubernetes, which can better realize environment isolation and resource control. At present, it is mainly triggered manually through the platform interface. The alternative solution can increase mechanisms such as timing triggering and event triggering (such as automatically triggering test environment upgrade after code submission) to realize higher degree of automation.

[0170] That is, the application example of the present application provides the following core technologies: 1) Modular design of data upgrade scaffolding: the core lies in the design of two abstract classes, BaseUpgradeService and BaseUpgradeController, especially the template method patterns (such as queryData, upgrade / upgradeByBatch, validateUpgradeData) defined by them and the built-in automatic pagination query loop control logic and configurable multi-thread execution engine.

[0171] 2) Collaborative working mechanism of scaffolding and visualization platform: the protection point lies in the interaction protocol between the standard RESTful API (such as / startUpgrade, / queryProgress) provided by the scaffolding and the cloud shuttle platform, and how the platform automatically discovers, deploys and calls the data upgrade program developed based on the scaffolding.

[0172] 3) Core functional modules of visualization platform: including but not limited to: program deployment and start mechanism based on automated scripts, pre-check health and resource inspection process of upgrade task, real-time collection and display method of execution progress and log, and operation permission control model based on role and environment.

[0173] 4) Standardized data upgrade process from end to end: a series of steps such as development, pre-check, one-key execution, real-time monitoring and post-validation are integrated into a complete and toolized process, which itself as a methodology and system implementation is an important protection point.

[0174] In summary, the application example has the following beneficial effects: 1) Development efficiency and standardization are greatly improved: compared with customized development, the application example of the present application can effectively reduce the development workload by using the scaffolding, and the developer only needs to implement 3-4 core methods. At the same time, the code structure is unified, which greatly improves the maintainability and team collaboration efficiency.

[0175] 2) Processing performance is significantly optimized: for large data volume scenarios, the application example of the present application uses the built-in and configurable multi-thread mechanism of the scaffolding, which can improve the processing speed by several times or even tens of times compared with traditional single-thread SQL or program (for example, the processing time of ten million data is reduced from several hours to one hour).

[0176] 3) The operation process is completely visualized, and the management granularity is fine: compared with viewing original logs, the cloud shuttle platform of the application example provides graphical progress bars, real-time log dashboards and success / failure data statistics, which makes the upgrade status clear at a glance for operation and maintenance personnel, greatly reducing the difficulty and time of troubleshooting problems.

[0177] 4) System risk is effectively reduced: through the pre-check mechanism, the risk of environment dependence and resource shortage is automatically excluded before upgrading; through the idempotency control, the data repeated upgrading caused by misoperation is prevented; through the standardized process, the human operation negligence is avoided. These mechanisms together constitute a safe and reliable data upgrade guarantee system.

[0178] 5) The whole life cycle management is realized: the application instance integrates the scattered development, testing, deployment, execution, monitoring and verification links into a unified platform, realizes the end-to-end, tooling and standardized management of data upgrade tasks, which is incomparable to manual SQL and customized program.

[0179] The embodiment of the application further provides an electronic device, which can include a processor, a memory, a receiver and a transmitter, the processor is used to execute the data upgrade method and / or the data upgrade management method mentioned in the above embodiment, wherein the processor and the memory can be connected through a bus or other means, taking the bus connection as an example. The receiver can be connected with the processor and the memory through wired or wireless means.

[0180] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. chips, or combinations of the above various chips.

[0181] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the data upgrade method in the embodiment of the application. The processor executes various functions of the processor and data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, realizes the data upgrade method and / or the data upgrade management method in the above method embodiment.

[0182] The memory can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, etc. The data storage area can store data created by the processor, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some embodiments, the memory can optionally include a memory that is remotely located with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0183] The one or more modules are stored in the memory and, when executed by the processor, perform the data upgrade method and / or the data upgrade management method in the embodiments.

[0184] In some embodiments of the present application, a user equipment can include a processor, a memory, and a transceiver unit that can include a receiver and a transmitter, the processor, the memory, the receiver, and the transmitter can be connected through a bus system, the memory is configured to store computer instructions, and the processor is configured to execute the computer instructions stored in the memory to control the transceiver unit to transceive signals.

[0185] As an implementation manner, the functions of the receiver and the transmitter in the present application can be implemented by a transceiver circuit or a transceiver dedicated chip, and the processor can be implemented by a dedicated processing chip, a processing circuit, or a general-purpose chip.

[0186] As another implementation manner, the server provided by the embodiments of the present application can be implemented by using a general-purpose computer. That is, program codes for implementing the functions of the processor, the receiver, and the transmitter are stored in the memory, and the general-purpose processor implements the functions of the processor, the receiver, and the transmitter by executing the codes in the memory.

[0187] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the aforementioned data upgrade method and / or data upgrade management method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable memory disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0188] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the aforementioned data upgrade method and / or data upgrade management method.

[0189] Those of ordinary skill in the art will appreciate that the various illustrative components, systems and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or both. The particular implementation is dependent on the specific application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, for example, the hardware can comprise an electronic circuit, an Application-Specific Integrated Circuit (ASIC), appropriate firmware, a plug-in card, a function card, etc. When implemented in software, the elements of the present application are the program or code segments to perform a particular task. The program or code segments can be stored in a machine-readable medium, or carried by a data signal in a carrier wave through a transmission medium or a communication link.

[0190] It is to be understood that the application is not limited to the particular configurations and processes described herein and shown in the figures. For simplicity, detailed descriptions of known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0191] In the present application, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0192] The above description is merely illustrative of the application, and is not to be taken in a limiting sense. Various changes and modifications can be made to the embodiments described and equivalents can be substituted for elements without departing from the spirit of the application. It will be appreciated that, in the development of any such changes or modifications to the embodiments described, the skilled person can make use of known methods and technologies. Any and all such changes, modifications, and equivalents are intended to fall within the scope of the present application.< / t> < / t> ​< / t>

Claims

1. A data upgrade method, characterized in that, include: Based on the start upgrade instruction received from the visualization platform, the query method in the predefined data upgrade business logic template is called to execute paginated queries and obtain the data to be upgraded; Based on the amount of data to be upgraded, the target execution mode is automatically selected through the decision logic defined in the data upgrade business logic template. The target execution mode includes: single-threaded execution mode or multi-threaded batch execution mode. According to the target execution mode, the upgrade method in the data upgrade business logic template is invoked to upgrade the data to be upgraded; During the upgrade process, the corresponding processing progress information and execution log information are actively pushed to the visualization platform through the reporting mechanism defined in the data upgrade business logic template, so that the visualization platform can perform visual monitoring of the upgrade process of the data to be upgraded.

2. The data upgrade method according to claim 1, characterized in that, Before invoking the query method in the predefined data upgrade business logic template, the following is also included: Dynamically retrieve database connection configuration information corresponding to the current operating environment from the configuration center; Based on the database connection configuration information, initialize and establish a connection between itself and the target database; Perform a pre-health check, wherein the pre-health check includes at least one of the following: checking the availability of dependent services, checking whether the resource utilization of the target server is lower than a first preset threshold, and checking the database connectivity. Receive the start-up and upgrade command sent by the visualization platform; The startup upgrade command is subjected to idempotency verification. If the startup upgrade command passes the idempotency verification, it is marked as a startup upgrade command that has passed the verification. Correspondingly, the step of calling the query method in the predefined data upgrade business logic template according to the start upgrade instruction received from the visualization platform to perform paginated query and obtain the data to be upgraded includes: Based on the start-up upgrade instruction received from the visualization platform and verified, the query method in the predefined data upgrade business logic template is invoked to send an SQL query statement with pagination parameters to the target database. Receive the upgrade data returned by the target database that matches the SQL query statement.

3. The data upgrade method according to claim 1, characterized in that, The decision logic is configured to: compare the amount of data to be upgraded with a second preset threshold, and select the target execution mode from among the preset execution modes based on the comparison result; Correspondingly, based on the amount of data to be upgraded, the automatic selection of the target execution mode through the decision logic defined in the data upgrade business logic template includes: Compare the amount of data to be upgraded with a second preset threshold; If the amount of data is less than or equal to the second preset threshold, then the single-threaded execution mode is selected; If the amount of data is greater than the second preset threshold, the multi-threaded batch execution mode is selected, and the thread pool is initialized based on the parameters configured in the data upgrade business logic template for initializing the thread pool.

4. The data upgrade method according to claim 3, characterized in that, The step of upgrading the data to be upgraded by invoking the upgrade method in the data upgrade business logic template according to the target execution mode includes: If the target execution mode is the single-threaded execution mode, then the upgrade method defined in the data upgrade business logic template is directly called to perform sequential upgrade processing on the data to be upgraded. If the target execution mode is the multi-threaded batch execution mode, the data to be upgraded is divided into multiple batches of data, and each batch of data is submitted to the thread pool for concurrent upgrade processing. Each thread in the thread pool calls the upgrade method for batch data defined in the data upgrade business logic template to upgrade one batch of data.

5. The data upgrade method according to claim 1, characterized in that, During the upgrade process, the corresponding processing progress information and execution log information are proactively pushed to the visualization platform through the reporting mechanism defined in the data upgrade business logic template, including: By using the reporting mechanism specified in the data upgrade business logic template at a fixed time interval or a fixed data volume interval, the application interface provided by the visualization platform is invoked to actively push the current processing progress information and corresponding execution log information of the upgrade process to the visualization platform.

6. The data upgrade method according to claim 1, characterized in that, Also includes: After the upgrade process is completed, the verification method defined in the data upgrade business logic template is called to perform business rule verification on the upgraded data and obtain the verification result information. The verification result information is pushed to the visualization platform.

7. The data upgrade method according to any one of claims 1 to 6, characterized in that, The data upgrade business logic template is provided in the form of a Java abstract class. The query method, the upgrade method, the decision logic, and the reporting mechanism in the data upgrade business logic template are implemented by inheriting the abstract class and implementing its predefined abstract methods or overriding its hook methods.

8. A data upgrade management method, characterized in that, The management method, executed by a visualization platform, includes: According to the predefined execution order dependency of each data upgrade task, the service status and server resource indicators on which each data upgrade program depends are obtained and checked in sequence by calling the monitoring system interface. After the check is passed, start upgrade instructions are sent to each corresponding data upgrade program in sequence to trigger each data upgrade program to execute the data upgrade method as described in any one of claims 1 to 7. Receive and aggregate the processing progress information and execution log information actively pushed by the data upgrade program; The processing progress information and execution log information are displayed in a graphical interface.

9. The data upgrade management method according to claim 8, characterized in that, Also includes: The system receives data upgrade packages uploaded by users based on the data upgrade business logic template through a preset task management interface, automatically parses the data upgrade packages, identifies and registers the data upgrade tasks and their calling interfaces defined in the data upgrade packages.

10. A data upgrade system, characterized in that, include: At least one data upgrade procedure is configured to perform the data upgrade method as described in any one of claims 1 to 7; A visualization platform is configured to perform the data upgrade management method as described in claim 8 or 9; The configuration center is used to provide dynamic database connection configuration information to the data upgrade program; The data upgrade program communicates with the visualization platform through an application programming interface (API) defined based on a standardized protocol.

Images