Data Sandbox Publishing Method, Device and Storage Medium Based on Automated Processing

Abstract

The present application discloses a method, device, and storage medium for realizing data sandbox publishing based on automated processing, which relates to the technical field of data processing. The method includes: selecting a corresponding suite type according to the type of the release package, and invoking the corresponding suite service based on the suite type to construct a release package task; invoking the suite service, generating a release package based on the release package task, and saving the release package; initiating a release approval process according to the notification of the saving result of the release package, where the release approval process is used to determine whether to deploy the release package to a target environment; when the release approval process passes, deploying the release package to the target environment. By selecting a suite type to construct a release package task, generating and automatically saving the release package, and introducing an approval process, the present application ensures the standardization and traceability of the release package publishing, reduces the risks of human errors and environmental conflicts, and improves the deployment ability of the data sandbox.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method, device, and storage medium for realizing data sandbox publishing based on automated processing. Background Art

[0002] Data sandboxes are usually used to solve the isolation problem between tasks in different environments of a big data platform and ensure the secure use of data between tasks. In a big data platform, current data sandbox technologies usually create development tasks based on online data to simulate real business scenarios. However, this method requires requesting data from different sources, and it is necessary to explain the data processing method to developers, and then the developers process the data to meet the production requirements of online tasks.

[0003] Once the online requirements change, given the complexity of tasks in the online environment, developers are prone to omissions and mistakes during task modification. When the modified task is deployed to the online environment, it will cause production problems such as dirty data, affecting the accuracy of data processing. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, and storage medium for realizing data sandbox publishing based on automated processing, aiming to solve the technical problem of easy configuration errors in development tasks during manual release and deployment of services.

[0005] To achieve the above object, an embodiment of this application provides a method for realizing data sandbox publishing based on automated processing, and the method for realizing data sandbox publishing based on automated processing includes:

[0006] Select a corresponding suite type according to the release package type, and call the corresponding suite service based on the suite type to construct a release package task;

[0007] Call the suite service, generate a release package based on the release package task, and save the release package;

[0008] Initiate a release approval process according to the save result notification of the release package, and the release approval process is used to determine whether to deploy the release package to the target environment;

[0009] When the release approval process passes, deploy the release package to the target environment.

[0010] In an embodiment, before the step of calling the suite service, generating a release package based on the release package task, and saving the release package, it includes:

[0011] According to the cluster information of the release package task, establish a mapping relationship between different environments according to the principles of the same type and the same name for the cluster information;

[0012] Determine whether the mapping relationship is successfully established. If the mapping relationship is successfully established, generate multiple target environments.

[0013] In one embodiment, after the step of determining whether the mapping relationship is successfully established and generating multiple target environments if the mapping relationship is successfully established, the method further includes:

[0014] Perform a release detection on the release package task to obtain a release detection result;

[0015] If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources on which the release package task depends are not created in the target environment, a release package cannot be generated based on the release package task;

[0016] When the release detection result passes, generate the release package according to the release package task.

[0017] In one embodiment, the step of calling the suite service, generating a release package based on the release package task, and saving the release package includes:

[0018] Call the suite service and execute the release package saving command;

[0019] Based on the release package saving command, the suite service obtains the tasks or resources on which the release package task depends by using the task identifier of the release package task;

[0020] Based on the tasks or resources on which the release package task depends, the suite service generates a release package and mounts the release package to the sandbox shared disk.

[0021] In one embodiment, the step of initiating a release approval process according to the saving result notification of the release package, where the release approval process is used to determine whether to deploy the release package to the target environment includes:

[0022] Receive the saving result notification of the release package;

[0023] If the saving result notification is a failure, update the record status of the release task of the release package to a failure and stop the release process of the release package;

[0024] If the saving result notification is a success, initiate a release approval process.

[0025] In one embodiment, the step of deploying the release package to the target environment when the release approval process passes includes:

[0026] After the release approval process passes, send a release deployment command for the release package to the suite service;

[0027] Based on the release deployment command, the suite service obtains the release package from the sandbox shared disk according to the release package identifier of the release package;

[0028] The suite service parses the release package, obtains the resources of the release package task, and uploads the resources to the target environment.

[0029] In one embodiment, after the step of parsing the release package, obtaining the resources of the release package task, and uploading the resources to the target environment, the method further includes:

[0030] The suite service performs create, update, or delete operations on the release package task in the target environment according to the task change type of the release package;

[0031] When the suite service completes the create, update, or delete operation on the release package task, the deployment of the release package in the target environment is completed.

[0032] In one embodiment, after the step of when the suite service completes the create, update, or delete operation on the release package task and completes the deployment of the release package in the target environment, the method further includes:

[0033] Obtain the release result of the release package, and based on the release result, update the release status of the release package task.

[0034] An embodiment of the present application further provides a data sandbox release device based on automated processing. The data sandbox release device based on automated processing includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the data sandbox release method based on automated processing as described above.

[0035] An embodiment of the present application further provides a storage medium. The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data sandbox release method based on automated processing as described above are implemented.

[0036] The embodiment of the present application discloses a method for realizing data sandbox release based on automated processing. By selecting a corresponding suite type according to the release package type and invoking the corresponding suite service based on the suite type to construct a release package task; invoking the suite service, generating a release package based on the release package task, and saving the release package; initiating a release approval process according to the save result notification of the release package, where the release approval process is used to determine whether to deploy the release package to the target environment; when the release approval process passes, deploying the release package to the target environment. The present application constructs a release package task by selecting a suite type, generates and automatically saves the release package, and introduces an approval process, ensuring the standardization and traceability of the release package release, reducing the risk of human errors and environment conflicts, and improving the deployment ability of the data sandbox. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic flowchart of the first embodiment of the method for realizing data sandbox release based on automated processing according to the solution of the embodiment of the present application;

[0038] Figure 2 It is a schematic flowchart of the second embodiment of the method for realizing data sandbox release based on automated processing according to the solution of the embodiment of the present application;

[0039] Figure 3 It is a schematic flowchart of the third embodiment of the method for realizing data sandbox release based on automated processing according to the solution of the embodiment of the present application;

[0040] Figure 4 It is a schematic flowchart of the fourth embodiment of the method for realizing data sandbox release based on automated processing according to the solution of the embodiment of the present application;

[0041] Figure 5 It is a schematic flowchart of the fifth embodiment of the method for realizing data sandbox release based on automated processing according to the solution of the embodiment of the present application;

[0042] Figure 6 It is a schematic flowchart of the brief process of the method for realizing data sandbox release based on automated processing according to the solution of the embodiment of the present application;

[0043] Figure 7 It is a schematic structural diagram of the device for realizing data sandbox release based on automated processing according to the solution of the embodiment of the present application.

[0044] The implementation, functional features, and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] Data sandboxes are usually used to solve the isolation problem between tasks in different environments of big data platforms, ensuring the secure use of data between tasks. In big data platforms, current data sandbox technologies usually create development tasks based on online data to simulate real business scenarios. However, this approach requires requesting data from different sources, explaining the data processing methods to developers, and then having the developers process the data to meet the production requirements of online tasks.

[0047] Once the online requirements change, given the complexity of tasks in the online environment, developers are prone to omissions and mistakes during task modification. When the modified task is deployed to the online environment, it will cause production problems such as dirty data, affecting the accuracy of data processing.

[0048] To address the above-mentioned deficiencies in the related art, an embodiment of the present application proposes a method for realizing data sandbox release based on automated processing. This method selects the corresponding suite type according to the release package type and calls the corresponding suite service based on the suite type to construct a release package task; calls the suite service, generates a release package based on the release package task, and saves the release package; initiates a release approval process according to the save result notification of the release package, and the release approval process is used to determine whether to deploy the release package to the target environment; when the release approval process passes, deploy the release package to the target environment. By selecting the suite type to construct the release package task, generating and automatically saving the release package, and introducing an approval process, the present application ensures the standardization and traceability of release package releases, reduces the risk of human errors and environmental conflicts, and improves the deployment ability of data sandboxes.

[0049] It should be noted that the execution subject of this embodiment can be a data sandbox or a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a device for realizing data sandbox release based on automated processing that can achieve the above functions. Hereinafter, the data sandbox will be taken as an example to illustrate this embodiment and the following embodiments.

[0050] The method for realizing data sandbox release based on automated processing according to the first embodiment proposed by the present application is as follows Figure 1 This method includes steps S10 to S40:

[0051] Step S10: Select the corresponding suite type according to the release package type, and call the corresponding suite service based on the suite type to construct a release package task.

[0052] During the deployment process of the data sandbox, the release package usually contains resources such as the application code, configuration files, and dependent libraries. Different applications may require different deployment logics and environment configurations. Therefore, to ensure that the release package can run correctly in the target data sandbox environment, it is first necessary to determine which suite type to use according to the type of the release package.

[0053] The suite type targets big data capabilities, such as data development, data integration, etc. For example, an application for data development needs to include suites for data processing frameworks, computing engines, and storage services. Different suite types are designed to meet the needs of different applications and release package tasks, including the service resources required in each stage from development, testing to deployment, to ensure that the release package can be successfully deployed and run in the target environment. The role of the data sandbox is to deploy the capabilities corresponding to these suite types to different target environments and achieve isolation between target environments to ensure data security and independence.

[0054] In this embodiment, the sandbox service, as the core service component of the data sandbox release process, adapts to multiple task types and can build different release package types based on different suite types, such as data development tasks, data integration synchronization types, etc. The sandbox service associates the release package type with the suite type by storing pre-set mapping relationships, so as to find the corresponding suite type according to the input release package type.

[0055] After selecting the release package type, according to the suite type corresponding to the release package type, call the corresponding suite service to obtain the relevant parameters required for the release package task. For example, for a data development task, information such as functions, resources, and parameters associated and bound to the data development task can be obtained, and the number of tasks associated with the data development task can also be counted.

[0056] When calling the suite service, the sandbox service will pass the required context information to the suite service to help the suite service more accurately obtain the resources and parameters required for the release package task. At the same time, after receiving the call request, the suite service will use its own logic and algorithms to provide corresponding support for building the release package task according to the received information (such as release package type, task details, etc.), to ensure that the built release package task can be successfully executed in the release process. For example, the suite service can call different internal modules or external interfaces to obtain and process the required data, such as calling the storage service to obtain the required code files or configuration files, and calling the resource management service to obtain resources such as dependent libraries.

[0057] It should be noted that the suite service is the execution unit in the corresponding suite type, responsible for executing specific operations and converting the various service functions provided in the suite type into operable processes.

[0058] In an alternative embodiment, this example supports batch selection of release package tasks and obtains the task change types corresponding to the release package tasks. When there are multiple release package tasks to be processed, through batch selection, certain operations for multiple release package tasks can be completed at one time, avoiding the need for users to repeat the same process one by one.

[0059] It should be noted that the task change type is determined according to the status of the release package task at different stages and different operations, including creation (or addition), update, or deletion, and is used to guide the specific operation type of the release package task in the target environment. Among them, the release package task that has not been released to the target environment belongs to the new type of task; the release package task that has been edited after release belongs to the update type of task; the release package task that has been deleted in the current environment belongs to the deletion type of task.

[0060] The sandbox service can create various types of release package tasks according to different change types, improving the flexibility and adaptability of the sandbox service. At the same time, the sandbox service also supports visual operations, enabling users to view the parameter information associated with the corresponding release package task, including the basic information of the release package task, the dependent resources, the status of the task, and the change type, etc. Through the visual interface, users can understand the detailed information of the release package task, promptly discover potential problems, and adjust and operate on the release package task, thereby increasing the success rate of releasing the release package task to the target environment.

[0061] Exemplarily, the user views through the visual interface which parameters have been modified for the update type task, or views the detailed information of the new type task to confirm whether the release conditions are met. By viewing and adjusting the detailed information of the release package task, errors in the subsequent release process can be avoided, thereby increasing the success rate of the release package deployment to the target environment for execution.

[0062] Step S20: Invoke the suite service, generate a release package based on the release package task, and save the release package.

[0063] In this example, after creating the release package task, the sandbox service will invoke the suite service to execute the instruction to save the release package. Specifically, taking the constructed release package task as input, the suite service integrates and encapsulates the information in the release package task according to its own algorithms and logics. For example, the suite service collects various resources required for the release package task (such as code files, configuration files, data files, etc.), and organizes and stores this information in a specific format (such as JSON format, XML format, or a custom binary format) to form a complete release package.

[0064] While calling the suite service to generate a release package, the sandbox service saves the snapshot data of the release package tasks selected by the user in the page to the sandbox database to persist the release records of the release package tasks, facilitating subsequent viewing of the release progress and detailed information of the release package tasks by the user.

[0065] When the suite service collects various resources required for the release package task, it obtains the detailed information of the selected release package task and the resources on which the release package task depends according to the task identifier of the release package task passed by the sandbox service, including reading corresponding files or data from different storage locations, or performing compression or encryption operations on the data. Then these resources are packaged to obtain the release package to ensure the integrity and security of the release package.

[0066] Exemplarily, the nodes of the offline development task include dependent resource jar packages, udf functions, etc. After the suite service downloads the resources on which the offline development task depends, it saves them in the release package corresponding to the offline development task.

[0067] For the generated release package, the release package is saved to a specific storage location. The storage locations include the sandbox shared disk, local disk, network storage devices such as NAS (Network Attached Storage) storage, cloud storage services, etc. Among them, the save operation can be implemented by calling the interface of the storage system or using the operation functions of the file system. At the same time, the storage operation is monitored to ensure that the release package is successfully saved. If the save fails, corresponding error handling will be performed, such as recording error logs, retrying operations, notifying the administrator, or updating the release task record status of the release package.

[0068] When storing the release package, the release package is stored according to certain naming rules. The file name can be composed of information such as the release package task name, timestamp, task identifier, etc., facilitating the search and management of subsequent release processes. For example, after creating the release package corresponding to the release package task, according to the naming rule of the task identifier of the release package task + timestamp, the release package is saved and mounted to the sandbox shared disk, enabling the sandbox service and the suite service to share the data and files in the release package.

[0069] In an alternative implementation, for the release packages that have been successfully saved in the sandbox shared disk, the saved release packages in the sandbox shared disk can be periodically cleaned, such as every six months or every month, to reduce the occupied storage space.

[0070] Step S30: Initiate a release approval process according to the save result notification of the release package, and the release approval process is used to determine whether to deploy the release package to the target environment.

[0071] In this embodiment, after the release package is successfully saved, the release and deployment of the data sandbox need to meet release requirements such as compliance, security, and resource management, to avoid risks that may be brought about by the random deployment of un-reviewed release packages to the target environment, such as security vulnerabilities, resource conflicts, or issues inconsistent with business logic. Therefore, through the release approval process, human errors can be reduced, and the accuracy of the release package deployment can be ensured.

[0072] The save result notification is an information notification generated after the suite service completes the save operation of the release package in step S20, including information such as whether the save operation is successful, the save location, and the save time. Then, the save result notification will be sent to the sandbox service, and this save result notification will be used as the basis for the sandbox service to initiate the release approval process.

[0073] In this embodiment, when the sandbox service receives a save result notification indicating that the release package save fails, it will not continue with the release process, and at the same time, update the status of the release task record of this release package in the sandbox database to failed.

[0074] When the sandbox service receives a save result notification indicating that the release package is successfully saved, it will automatically initiate the release approval process. After receiving the release process of the release package, the process approver can perform an approval operation in the process to-be-done to decide whether to pass the release process of the release package.

[0075] If the release approval process is rejected, the release package is not allowed to be released to the target environment, and the release process ends.

[0076] If the release approval process is passed, the release process continues, and the next deployment operation is entered. Specifically, then the deployment process of the release package will be executed, the release package parameters will be assembled and executed, and the suite service will be called to execute the task release method of the release package.

[0077] This embodiment can reduce human errors and violation risks and ensure the visual tracking of the release process by initiating the release approval process.

[0078] Step S40: When the release approval process is passed, deploy the release package to the target environment.

[0079] In this embodiment, when the release package is evaluated and reviewed and is considered to meet the deployment conditions, the release package will start to be deployed to the target environment to complete the release operation of the data sandbox.

[0080] The target environment refers to the environment where the release package will ultimately be deployed. The target environment can be a physical server cluster, a virtual data center, a cloud computing service environment, or an isolated data sandbox environment, etc. The target environment is the environment where the release package finally runs.

[0081] After the release approval process is passed, the sandbox service will find the release package to be deployed based on the stored release package information, including its storage location (such as the sandbox shared disk, cloud storage service, etc.) and the release package identifier of the release package.

[0082] Next, the sandbox service calls the corresponding suite service to extract the information in the release package. For example, resources such as code, configuration files, and dependent libraries in the release package are deployed to the target environment.

[0083] During the deployment process, first, it is necessary to complete the resource allocation and environment configuration operations for the target environment according to the data such as the resources in the release package. Then, according to the release package tasks in the release package, allocate the required computing resources for the tasks, such as CPU (Central Processing Unit), memory, storage, etc., and create the required running environment, such as installing the corresponding software environment and starting the service.

[0084] For different types of release packages, different deployment methods may be adopted. For example, for containerized release packages, use container orchestration tools to deploy the containers to the nodes in the target environment. For traditional application release packages, use automated deployment scripts to install the applications on the servers.

[0085] In addition, during the deployment process, the deployment operations will also be monitored to ensure the success of each operation. If a deployment error occurs, the error information will be recorded for subsequent troubleshooting and problem-solving. At the same time, the progress of the deployment can also be updated and displayed so that users can master the deployment status of the release package.

[0086] This embodiment reduces the complexity of manual configuration and the risk of configuration errors in development tasks by automatically selecting the suite type and calling the suite service, and through the release approval process, realizes strict review of the release package, ensuring the quality of the release package and the visual tracking of the release process.

[0087] Based on the above embodiment, please refer to Figure 2 , the method for realizing data sandbox release based on automated processing in the second embodiment proposed by this application, before step S20 includes steps S201~S202:

[0088] Step S201: According to the cluster information of the release package task, establish a mapping relationship between different environments according to the principles of the same type and the same name for the cluster information.

[0089] Step S202: Determine whether the mapping relationship is successfully established. If the mapping relationship is successfully established, generate multiple target environments.

[0090] In a data sandbox environment, there are multiple clusters, and different clusters contain different resources and services, and need to be deployed in different environments. To ensure the accurate deployment and resource allocation of the release package task across different environments, a reasonable mapping relationship needs to be established between different environments to better manage and schedule resources.

[0091] It should be noted that the cluster information includes various information about the clusters related to the release package task, such as the type of the cluster, the name of the cluster, and the resource information owned by the cluster, such as the number of servers, storage capacity, network configuration, etc. The mapping relationship refers to an association established between different environments based on the principles of the same type and the same name. The same type means clusters with similar technical architectures and functional characteristics, and the same name means clusters with the same name. By establishing the mapping relationship between different environments according to this principle, it is possible to better manage and operate similar clusters in different environments and ensure the consistency of the release package task across different environments.

[0092] In this embodiment, the cluster information of the release package task is obtained, such as the cluster data source, the mapping of computing resource queues, parameters, etc. According to the principles of the same type and the same name, the mapping relationship is automatically established between different environments. This process matches and maps the resources and configurations in the source environment with the corresponding resources in the target environment by analyzing the structure and attributes of the cluster information.

[0093] Exemplarily, if there is a data source named "data_source_1" in the source environment, then by searching for the data source with the same name in the target environment, the data source with the same name in the target environment can be mapped to the "data_source_1" data source in the source environment.

[0094] If the mapping relationship is successfully established, multiple target environments will be generated, and these target environments are candidate environments for the subsequent deployment of the release package.

[0095] It should be noted that generating multiple target environments is to provide options for switching target environments and also support multi-environment deployment strategies, such as gradually verifying and deploying the release package in development, testing, and production environments.

[0096] If the mapping relationship fails to be successfully established, the subsequent release detection phase cannot be carried out, and an error message will be returned to prompt the user to check and adjust.

[0097] Furthermore, after step S202, steps S2021 to S2023 are also included:

[0098] Step S2021: Perform a release detection on the release package task to obtain a release detection result.

[0099] In this embodiment, release detection is performed on the release package task to verify whether the target environment meets the release conditions. The release detection includes release environment detection, task detection, etc. The result of the release detection will determine whether to continue the subsequent release process.

[0100] Release detection is a crucial step to ensure that the release package task can run smoothly in the target environment. By checking the configuration of the target environment, resource availability, and task dependencies, the release detection result is generated.

[0101] The release detection process involves checking key elements such as database tables, computing resources, and network configurations in the target environment to ensure that the above information is consistent with the release requirements of the release package task.

[0102] Exemplarily, if the release package task depends on specific database tables or computing resource queues, the release detection will verify whether these resources have been created and are available in the target environment.

[0103] Step S2022: If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources on which the release package task depends have not been created in the target environment, a release package cannot be generated based on the release package task.

[0104] In this embodiment, according to the release detection result, it is judged whether the target environment meets the release conditions, or whether the tasks or resources on which the release package task depends have been created in the target environment.

[0105] If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources on which the release package task depends have not been created in the target environment, the release process is terminated and an error message is returned.

[0106] The release detection result of this embodiment ensures that a release package will only be generated and deployed when the target environment is fully prepared. For example, if a certain key dependent resource is missing in the target environment, or a necessary configuration parameter is not set correctly, the sandbox service cannot generate a release package. This way can avoid release failures or running errors caused by environment mismatches.

[0107] It should be noted that the release conditions refer to whether the hardware resources of the target environment meet the requirements of the release package task, and whether the required software environment matches the release package task. The release detection process will verify whether these release conditions are met, so as to decide whether to continue the subsequent process of creating a release package.

[0108] Step S2023: When the release detection result passes, generate the release package according to the release package task.

[0109] In this embodiment, when the release detection is passed, the sandbox service will generate a release package according to the release package task. During the process of generating the release package, the sandbox service will integrate and package the task code, configuration files, dependent resources, etc. required for the release package task by calling the suite service to form a release package.

[0110] The generated release package contains all the necessary information, tasks, and resources required for task deployment to ensure that the release package task can run independently in the target environment. For example, the suite service generates a release package folder in the form of a JSON file or a zip package of the release package for the task code, dependent JAR packages, configuration files, etc. required for the release package task, so as to be quickly verified and loaded during deployment.

[0111] Based on the above embodiment, please refer to Figure 3 , in the method for realizing data sandbox release based on automated processing according to the third embodiment proposed by this application, step S20 may further include steps S210 to S230:

[0112] Step S210: Call the suite service to execute the command to save the release package.

[0113] During the release process of the data sandbox, the generation and storage of the release package are key links to ensure that the release package task can be successfully deployed to the target environment. As a virtual environment for isolating and managing the data analysis environment, the data sandbox needs to ensure that the dependent resources of the release package task are correctly identified, packaged, and stored in a shared location for subsequent deployment operations through automated means.

[0114] Therefore, when the release package task passes the task detection, the sandbox service will call the suite service to execute the command to save the release package. The suite service is the core component for realizing the generation and storage of the release package. By receiving the command to save the release package, the suite service will start the process of packaging and storing the release package task.

[0115] Step S220: Based on the command to save the release package, the suite service uses the task identifier of the release package task to obtain the tasks or resources on which the release package task depends.

[0116] Step S230: Based on the tasks or resources on which the release package task depends, the suite service generates a release package and mounts the release package to the sandbox shared disk.

[0117] According to the task identifier of the release package task in the save release package command, the suite service can obtain the details of the release package task in real time, as well as the tasks or resources on which the release package task depends, such as data sources, configuration files, dependency libraries, etc. Then, based on the data such as the tasks or resources on which the release package task depends obtained, the suite service packages them to generate a release package. At the same time, the release package is mounted to the sandbox shared disk.

[0118] The creation of the release package involves packaging the resources on which the task depends, such as JAR packages, UDF functions, etc., into a deployable format, such as a folder or a ZIP package, and naming it with the release package identifier and timestamp to ensure version control and tracking of the release package.

[0119] It should be noted that the "sandbox shared disk" is a shared storage area that can be used jointly by the sandbox service and the suite service, etc., to facilitate sharing of data and files between the sandbox service and the suite service. Storing the generated release package in the sandbox shared disk facilitates subsequent access and use of the release package by the sandbox service and the suite service when deploying the release package task.

[0120] Based on the above embodiments, please refer to Figure 4 , the method for implementing data sandbox release based on automated processing proposed in the fourth embodiment of this application, step S30 includes steps S310 to S330:

[0121] Step S310: Receive the save result notification of the release package.

[0122] In this embodiment, after the suite service completes the release package saving operation, it will generate a save result notification, which includes information such as whether the saving operation is successful, the saving location, and the saving time. The suite service will send the save result notification to the sandbox service.

[0123] Step S320: If the save result notification is a failure, update the record status of the release task of the release package to failure and stop the release process of the release package.

[0124] In this embodiment, after the suite service sends the notification information of the saving failure to the sandbox service, the sandbox service will update the record status of the release task of the release package to failure and at the same time stop the release process of the release package to avoid problems such as deployment failure that may be caused by further operations.

[0125] Step S330: If the save result notification is a success, initiate a release approval process.

[0126] In this embodiment, if the save result notification sent by the suite service to the sandbox service is successful, it indicates that the release package has been successfully saved, and subsequent operations can continue. At this time, the sandbox service will automatically initiate a release approval process, transfer the detailed information of the release package to the approval process, and notify relevant approval personnel or the automated approval mechanism to evaluate whether the current release package can be deployed to the target environment.

[0127] Based on the above embodiment, please refer to Figure 5 , the method for realizing data sandbox release based on automated processing according to the fifth embodiment proposed in this application, step S40 includes steps S410 to S430:

[0128] Step S410: After the release approval process passes, send a release deployment command for the release package to the suite service.

[0129] After the release approval process passes, the sandbox service sends a release deployment command for the release package to the suite service. The release deployment command for the release package is used to ensure that only the release package that has passed the approval will enter the deployment stage when notifying the suite service to start the deployment operation of the release package.

[0130] The release deployment command is usually transmitted by means of API call or message queue.

[0131] Step S420: Based on the release deployment command, the suite service obtains the release package from the sandbox shared disk according to the release package identifier of the release package.

[0132] After receiving the release deployment command for the release package, the suite service searches for and obtains the release package corresponding to the command in the sandbox shared disk using the release package identifier of the release package as an index.

[0133] Step S430: The suite service parses the release package, obtains the resources of the release package task, and uploads the resources to the target environment.

[0134] In this embodiment, the suite service parses the obtained release package, reads the metadata and resource information in the release package, so as to obtain the task details of the release package task in the release package and the resources associated with the task, etc. These resources include code files, configuration files, dependency libraries, etc. Then, the suite service uploads these resources to the storage location or computing node in the target environment by calling the resource management service or network transmission service of the target environment.

[0135] Exemplarily, in the obtained offline development task node, resource jar packages, reference parameters, etc. on which the task depends are obtained, and the resources associated with the task are uploaded in advance, and the parameter information on which the task depends is created in the target environment.

[0136] Further, after step S430, steps S440 to S450 are further included:

[0137] Step S440: According to the task change type of the release package, the suite service performs creation, update, or deletion operations on the release package tasks in the target environment.

[0138] After configuring the target environment parameters, the suite service creates, updates, or deletes the release package tasks according to the task change type of the release package tasks in the release package, and publishes the corresponding release package tasks to the target environment. For example, if the offline development suite releases a new release package task, it will automatically replace the data and environment variables in the associated task parameters to the target environment, so as to achieve data and computing isolation between the development mode tasks and the online mode tasks.

[0139] It should be noted that the associated task parameters refer to the configuration information and parameter settings related to the released release package tasks, and these parameters are used to control the behavior and resource reference of the release package tasks in different environments (such as development environment, test environment, production environment).

[0140] Exemplarily, in the data sandbox, a developer writes an SQL task that needs to reference specific data tables and computing resources. In the development environment, the task references the tables in the test database; in the production environment, the task references the tables in the formal database. By using the associated task parameters, the suite service can automatically replace these parameters in different environments to ensure that the task can run correctly.

[0141] In the release process, when the release package tasks are deployed to the target environment, the sandbox service will call the suite service to automatically replace these associated task parameters according to the configuration of the target environment.

[0142] Step S450: When the suite service completes the creation, update, or deletion operations on the release package tasks, the deployment of the release package in the target environment is completed.

[0143] When the suite service completes the creation, update, or deletion operations on the release package tasks, it means that the deployment of the release package tasks in the target environment has been completed.

[0144] Further, after step S450, step S460 is further included:

[0145] Step S460: Obtain the release result of the release package, and update the release status of the release package tasks based on the release result.

[0146] After the suite service completes the deployment of the release package task, it will send the execution result of the release package to the sandbox service. After obtaining the release result of the release package by checking the return value of the deployment operation, the running status report of the target environment, or other monitoring mechanisms, the sandbox service will update the release status of the release package task according to the release result. For example, after the suite service successfully releases the release package task, when the sandbox service monitors the release result callback sent by the suite service, it will update the release status of the release package task from "deploying" to "deployed", or update the release status from "deploying" to "deployment failed".

[0147] In addition, the sandbox service will also obtain the release operation records of the suite service when releasing the release package task, including the operation process and the reasons for success or failure, which is convenient for subsequent task management and problem troubleshooting.

[0148] In this embodiment, the sandbox service realizes the automatic release of the release package task to the target environment, and the sandbox service adapts to multiple suite services to centrally manage the release package tasks, so that the suite service only needs to have a set of codes to switch between different environments, avoiding modifying the codes and data of the suite service, and ensuring the development efficiency of the suite service.

[0149] Based on the above embodiment, the data sandbox release method based on automatic processing proposed in the sixth embodiment of this application further includes:

[0150] In this embodiment, the sandbox service adopts a containerization-based release mode to achieve multi-level user data and operation isolation under different users, different projects, and different environments.

[0151] Specifically, the sandbox service uses the sandbox shared library, a unified database, for data sharing. When each user creates a release package task, permission checks are performed when the user accesses the sandbox shared library based on the user's identity, the current project, and the environment. For example, when a user attempts to read or write a release package from the sandbox shared library, it is determined whether the user has the corresponding operation permission according to the user's permission information, such as the user ID, project ID, target environment ID, and the corresponding operation permissions. By intercepting and verifying the user requests, it is ensured that the user can only access and operate the release package data for which they have permission.

[0152] In addition, when performing the operation of saving the release package, the release package is marked, and the mark contains information such as the user ID, project ID, and target environment ID. When the user initiates a request to obtain the release package, the query statement or operation request is modified and filtered according to the target environment and permissions where the user is currently located. For example, for an SQL query statement, filtering conditions for the user ID, project ID, and target environment ID are automatically added to ensure that the user can only obtain and operate on the release package data under the target environment and project to which they belong. For the release package data in different environments, corresponding release package identifiers are added during storage and query to avoid confusion of release package data in different environments.

[0153] When the user switches between different projects or target environments, the user's session information is updated according to the user's new target environment and project information, including the current user, project, and target environment identifiers, etc. After switching the target environment, through the filtering of user permissions and release package data, based on the updated session information, the user can only see and operate on the release package data of the corresponding target environment to which they have permissions.

[0154] Through the multi-level control method, this application ensures the security of different suite services and release package data, does not require separate deployment of cluster environments for each user, reduces the environmental deployment cost, and also ensures that the data between different target environments is not contaminated.

[0155] Exemplarily, to help understand the implementation process of the data sandbox release method based on automated processing obtained by combining the above embodiments with this embodiment, please refer to Figure 6 , Figure 6 A brief process schematic diagram of a data sandbox release method based on automated processing is provided. Specifically:

[0156] The user starts to select the release package task through the interface. After the sandbox service receives the user's request to select the release package task, it obtains the task details of the release package task and the resource information required for the release package task in real time, and completes the parameter assembly according to the required resource information. Then, based on the release package task selected by the user, the release policy mapping is executed. Through the relevant cluster information of the release package task configured by the user, for example, if a certain release package task configures the cluster data source, computing resource queue mapping, parameters, etc., the mapping will be automatically performed according to the principle of the same type and the same name, establishing the mapping relationship between different environments, and obtaining the target environments corresponding to multiple release package tasks, so that the release package task can select the target environment for release.

[0157] Only when the release policy mapping passes will the next release detection be carried out for the release package task, including release environment detection and task detection, etc. Determine whether the target environment corresponding to the release package task meets the release conditions of the release package task. For example, whether the corresponding database table exists in the target environment for the data integration and synchronization task. Then, check whether the tasks, resources, etc. on which the release package task depends have been created in the target environment. For the release package task that fails the release detection, the release package cannot be created. Only by passing the release detection, that is, performing the pre-check of the release package deployment in advance to ensure the task release quality during the release process, can the sandbox service continue to create the release package.

[0158] The sandbox service first saves the snapshot data of the release package task to the sandbox database to persist the release record of the release package, which is convenient for users to view the task release progress and detailed information later. Then, by calling the suite service, the creation of the release package is completed. Specifically, according to the instruction of the sandbox service to save the release package, the suite service obtains the task details of the release package task in real time, then extracts the tasks and dependent resources of the release package task, and packs the extracted data to generate the release package, and mounts the release package to the sandbox shared disk.

[0159] After creating the release package, the suite service will send a notification of the result of saving the release package to the sandbox service. After the sandbox service monitors the callback message of saving the release package, it will determine whether the operation of saving the release package is successful. When the release package is successfully saved, the sandbox service will submit a workflow, that is, the release approval process. The workflow approval is used to verify whether the release package meets the preset release conditions, such as environment compatibility, resource availability, etc. Among them, the approval process includes manual review or automated inspection.

[0160] When the sandbox service obtains the callback notification of the workflow result and determines that the release process approval has passed, the sandbox service will send a release deployment command of the release package to the suite service. After receiving the release deployment command of the release package, the suite service obtains the corresponding release package file from the shared storage according to the release package identifier, and parses the release package file to obtain the release dependent resources, parameters, etc. of the release package task, and at the same time, it will also obtain and parse the task record of the release package task.

[0161] Subsequently, the suite service uploads the resources in the release package to the target environment, including the creation of database tables, the upload of resource files, the setting of environment variables, etc. The suite service submits task creation operations of different task change types in the target environment according to the task change type (new, updated or deleted) in the release package.

[0162] After completing resource upload and environment configuration, the suite service will execute the release package tasks, such as starting tasks, running job flows, etc. After the release package tasks are executed, the suite service will feedback the execution result of the release package to the sandbox service. After the sandbox service monitors the callback message of the executed release package and determines that the release package task is successfully executed, it will synchronously update the release status of the release package task to complete the automated process of the entire data sandbox release.

[0163] This embodiment realizes the automation of the data sandbox release process, reduces manual intervention, and improves the release efficiency and accuracy.

[0164] An embodiment of the present application provides a device for realizing data sandbox release based on automated processing. The device for realizing data sandbox release based on automated processing includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for realizing data sandbox release based on automated processing in the first embodiment above.

[0165] The following refers to Figure 7 , which shows a schematic structural diagram of a device for realizing data sandbox release based on automated processing suitable for use in the embodiments of the present application. The device for realizing data sandbox release based on automated processing in the embodiments of the present application may include various hardware and software components for realizing the method for realizing data sandbox release based on automated processing. Figure 7 The device for realizing data sandbox release based on automated processing shown is only an example and should not bring any limitations to the functions and usage scope of the embodiments of the present application.

[0166] As Figure 7As shown, the data sandbox publishing device based on automated processing may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the data sandbox publishing device based on automated processing are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the data sandbox publishing device based on automated processing to communicate with other devices wirelessly or wireline to exchange data. Although the figure shows a data sandbox publishing device with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be implemented or had alternatively.

[0167] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are performed.

[0168] The data sandbox publishing device based on automated processing provided by this application adopts the method for realizing data sandbox publishing based on automated processing in the above embodiment, and can solve the technical problem of easy configuration errors in development tasks during manual publishing and deployment of services. Compared with the prior art, the beneficial effects of the data sandbox publishing device based on automated processing provided by this application are the same as those of the method for realizing data sandbox publishing based on automated processing provided by the above embodiment, and other technical features in the data sandbox publishing device based on automated processing are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0169] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0170] As described above, only the specific implementation manners of this application are provided, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0171] The embodiments of this application provide a computer-readable storage medium with computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the method for realizing data sandbox publishing based on automated processing in the above embodiment.

[0172] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.

[0173] The above computer-readable storage medium can be included in the data sandbox publishing device implemented based on automated processing; it can also exist independently without being assembled into the data sandbox publishing device implemented based on automated processing.

[0174] The above computer-readable storage medium carries one or more programs. When the one or more programs are executed by the data sandbox publishing device implemented based on automated processing, the data sandbox publishing device implemented based on automated processing is caused to: select a corresponding suite type according to the release package type, and call the corresponding suite service based on the suite type to construct a release package task; call the suite service, generate a release package based on the release package task, and save the release package; initiate a release approval process according to the save result notification of the release package, and the release approval process is used to determine whether to deploy the release package to the target environment; when the release approval process passes, deploy the release package to the target environment.

[0175] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by connecting through the Internet service provider via the Internet).

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order from that marked in the accompanying drawings. For example, two consecutively represented blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0177] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0178] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for performing the above-mentioned method for realizing data sandbox publishing based on automated processing, and can solve the technical problem of easy occurrence of configuration errors in development tasks when manually publishing and deploying services. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the method for realizing data sandbox publishing based on automated processing provided by the above embodiments, and will not be elaborated here.

[0179] An embodiment of the present application provides a computer program product, including a computer program, which when executed by a processor, implements the steps of the method for realizing data sandbox release based on automated processing as described above.

[0180] The computer program product provided by the present application can solve the technical problem that configuration errors of development tasks are likely to occur when manually releasing and deploying services. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the method for realizing data sandbox release based on automated processing provided by the above embodiment, and will not be elaborated here.

[0181] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent scope of the present application.

[0182] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or system including that element.

[0183] Through the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0184] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present application.

Claims

1. A method for implementing data sandbox publishing based on automated processing, characterized in that: The method for implementing data sandbox publishing based on automated processing includes: Selecting a corresponding suite type according to the release package type, and calling a corresponding suite service based on the suite type to construct a release package task; Calling the suite service to execute a command to save and publish the package; Based on the save publish package command, the suite service uses the task identifier of the publish package task to obtain the task or resource on which the publish package task depends; Based on the tasks or resources that the release package task depends on, the suite service generates a release package, mounts the release package to the sandbox shared disk, and generates a save result notification; Initiate a release approval process according to the saving result notification of the release package, wherein the release approval process is used to determine whether to deploy the release package to the target environment; When the release approval process is passed, the release package is deployed to the target environment.

2. The method for implementing data sandbox publishing based on automated processing according to claim 1, characterized in that: Before the step of calling the suite service, generating a release package based on the release package task, and saving the release package, the method includes: According to the cluster information of the release package task, a mapping relationship is established between different environments according to the principle of the same type and the same name for the cluster information; Determine whether the mapping relationship is successfully established, and if the mapping relationship is successfully established, generate a plurality of the target environments.

3. The method for implementing data sandbox publishing based on automated processing as claimed in claim 2, characterized in that: The step of determining whether the mapping relationship is successfully established and, if the mapping relationship is successfully established, generating a plurality of target environments, further includes: Perform release detection on the release package task to obtain a release detection result; If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources that the release package task depends on are not created in the target environment, it is impossible to generate a release package based on the release package task; When the release detection result passes, the release package is generated according to the release package task.

4. The method for implementing data sandbox publishing based on automated processing according to claim 1, characterized in that: The step of initiating a release approval process according to the notification of the saving result of the release package, wherein the release approval process is used to determine whether to deploy the release package to the target environment includes: Receiving a notification of a saving result of the publishing package; If the saving result notification is failed, the publishing task record status of the publishing package is updated to failure, and the publishing process of the publishing package is stopped; If the saving result notification is successful, the publishing approval process is initiated.

5. The method for implementing data sandbox publishing based on automated processing according to claim 1, characterized in that: When the release approval process is passed, the step of deploying the release package to the target environment includes: When the release approval process is passed, a release deployment command of the release package is sent to the suite service; Based on the release deployment command, the suite service obtains the release package from the sandbox shared disk according to the release package identifier of the release package; The suite service parses the release package, obtains resources of the release package task, and uploads the resources to the target environment.

6. The method for implementing data sandbox publishing based on automated processing as claimed in claim 5, characterized in that: After the steps of parsing the release package, obtaining resources of the release package task, and uploading the resources to the target environment, the method further includes: The suite service executes a creation, update or deletion operation of the release package task in the target environment according to the task change type of the release package; When the suite service completes the creation, update or deletion operation of the release package task, the deployment of the release package in the target environment is completed.

7. The method for implementing data sandbox publishing based on automated processing according to claim 6, characterized in that: After the suite service completes the creation, update or deletion operation of the release package task and the step of deploying the release package in the target environment, the method further includes: The publishing result of the publishing package is obtained, and based on the publishing result, the publishing status of the publishing package task is updated.

8. A data sandbox publishing device based on automated processing, characterized in that: The data sandbox publishing device based on automated processing includes: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of a method for implementing a data sandbox publishing method based on automated processing as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for implementing a data sandbox publishing method based on automated processing as described in any one of claims 1 to 7 are implemented.

Images