Data processing method, electronic device, storage medium and computer program product

Abstract

This application discloses a data processing method, electronic device, storage medium, and computer program product. The method includes: receiving comment information of a first pull request, wherein the first pull request is an Infrastructure as Code (IaaC) pull request, and the comment information includes a first test command indicating test requirements for testing the IaaC; testing the IaaC based on the comment information to obtain a first test result; and adding the first test result to the comment information of the first pull request. This application improves the flexibility of IaaC testing and helps adapt to diverse testing requirements.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] Infrastructure as Code (IaC) is a technology for managing and configuring computing resources through code, and TerraForm is one of the mainstream IaC tools. When using TerraForm to make changes to IaC, the changed IaC needs to be tested to verify the changes. To test IaC, relevant tools can be used to automatically execute the TerraForm plan and attach the results to the pull request (PR). However, this method lacks flexibility and cannot adapt to diverse testing requirements. Summary of the Invention

[0003] To address the related technical problems, embodiments of this application provide a data processing method, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method, the method comprising: The system receives comment information about a first pull request, where the first pull request is an Infrastructure as Code (IMC) pull request. The comment information includes a first test command, which indicates test requirements for testing the IMC. The infrastructure, i.e., the code, is tested based on the comment information to obtain a first test result; Add the first test result to the comment information of the first pull request.

[0005] In the above scheme, the step of testing the infrastructure, i.e., the code, based on the comment information includes: The first information obtained based on the comment information includes one or more of the following: the execution path of the test task for testing the infrastructure as a code; one or more first sub-modules that need to be tested on the infrastructure as a code; the environment information corresponding to the infrastructure as a code; and task parameters related to the first fetch request. The infrastructure, i.e., the code, is tested based on the first information.

[0006] In the above scheme, obtaining the first information based on the comment information includes: Determine whether the first test command is a valid message; If the first test command is valid, the first test command is parsed to obtain the first information.

[0007] In the above scheme, the first test command includes the directory of the infrastructure, i.e., the binary files of the code, and obtaining the first information based on the comment information includes: The execution path is obtained from the directory of the binary files of the infrastructure, i.e., the code.

[0008] In the above scheme, receiving the comment information from the first pull request includes: The system receives the comment information sent by a code hosting platform, wherein the code hosting platform is configured with a Webhook.

[0009] In the above scheme, adding the first test result to the comment information of the first fetch request includes: According to the first test command, the first test result is added to the comment information of the first pull request, wherein the first test result and the first test command have a corresponding relationship in the comment information; The method further includes: The comment information corresponding to the added first test result is displayed.

[0010] The method in the above scheme further includes: Configure the storage path of the tfstate files associated with the execution of tests of the infrastructure i.e. code as a first path, which is different from the storage path of the files in the production environment.

[0011] In the above scheme, the data processing method is executed by a first service, which is a containerized service.

[0012] This application also provides a data processing apparatus, including: The first receiving unit is configured to receive comment information of a first pull request, wherein the first pull request is an infrastructure-as-code pull request, and the comment information includes a first test command, wherein the first test command indicates test requirements for testing the infrastructure-as-code. The first testing unit is used to test the infrastructure, i.e., the code, based on the comment information, and obtain a first test result; The first adding unit is used to add the first test result to the comment information of the first pull request.

[0013] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor. The processor is used to execute the steps of any of the above-mentioned technical solutions when running the computer program.

[0014] This application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0016] As can be seen from the above, in this embodiment, the first pull request is an Infrastructure as Code (IaaC) pull request, the first test command indicates the test requirements for testing the IaaC, the comment information of the first pull request includes the first test command, the comment information of the first pull request is received, the IaaC is tested according to the comment information to obtain a first test result, and the first test result is added to the comment information of the first pull request. It is evident that the test command in the comment information of the pull request in this embodiment can indicate the test requirements for testing the IaaC, and the subsequent testing process will also be performed according to this comment information. Therefore, the test requirements for testing the IaaC can be flexibly defined through the test command in the comment information; that is, the test requirements can be defined by defining the comment information of the pull request, thereby controlling the testing of the IaaC by flexibly defining the test requirements, improving the flexibility of the IaaC testing, and helping to adapt to diverse test requirements. Attached Figure Description

[0017] Figure 1 A schematic diagram illustrating the operation process of Atlantis in the related technologies provided for the application embodiments of this application; Figure 2 A schematic flowchart illustrating the data processing method provided in the application embodiments of this application; Figure 3 A schematic diagram illustrating the pull request processing procedure provided for an application embodiment of this application; Figure 4 A flowchart illustrating the process of testing in collaboration between GitLab and Orchtes, provided for an application embodiment of this application; Figure 5 A flowchart illustrating Embodiment 1 provided for application of this application; Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 7This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0019] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document is an exemplary expression and can be replaced with any possible expression, such as one or more, at least one, or at least one item, etc.

[0020] It should also be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0021] It should also be understood that the term "correspondence" mentioned in the embodiments of this application may indicate a direct or indirect correspondence between the two, or an association between the two, or a relationship of instruction and being instructed, configuration and being configured, etc.

[0022] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0023] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0024] TerraForm has become a primary tool for IaC, making resource configuration incredibly fast and simple. When using TerraForm to make changes to Infrastructure as Code (IaC), the changes need to be validated using a TerraForm plan. After review, `terraForm apply` is run to apply the changes. This process may be very convenient and quick for individuals or small teams, but when larger teams need to collaborate, managing all changes and pull requests becomes very cumbersome. In this process, the following difficulties may be encountered: First, there is the difficulty of permission management. During the plan and apply phases, TerraForm requires not only permissions from the state backend but also some permissions from the cloud service provider (cloudPRovider). The specific permissions required depend on the change action, such as creating, modifying, or deleting an EC2 instance. It is obviously impractical to assign these permissions to everyone in the team.

[0025] Furthermore, for each PR, you may need to attach logs showing the implementation of the current changes in the Terraform plan, so that the administrator can approve and apply the changes.

[0026] Furthermore, although toolchain versions can be agreed upon in advance, different developers may still use different TerraForm versions. Due to TerraForm version differences, various problems may arise during the execution of infrastructure-as-code changes, such as version incompatibility issues.

[0027] To address the aforementioned issues, a unified service can be used to manage the entire terraform process. This unified service is Atlantis. Atlantis is an open-source application used to automate terraform processes through pull requests. It integrates with GitHub, GitLab, and BitBucket via webhooks. Please refer to [link / reference]. Figure 1 , Figure 1This diagram illustrates the Atlantis workflow. When a Pull Request is created, Atlantis automatically executes the Terraform plan and appends its log to the PR. After the administrator reviews and confirms the changes, they send instructions to Atlantis via PR comments. Upon receiving the instructions, Atlantis executes Terraform apply to apply the changes. During team collaboration, only code modifications are needed; all plan and apply operations are handled by Atlantis, and access control is maximized.

[0028] Currently, Atlantis is primarily used on the open-source GitHub code hosting platform, and its support for private code hosting platforms is not very good. The main disadvantage is: First, Atlantis needs to be executed automatically. Once Atlantis is configured, every code commit in this repository will perform code validation, i.e., plan operation, by default. If an exception is encountered, the merging of the current pull request will be blocked. Since the default tfstate storage strategy requires certain network conditions, it is impossible to verify the actual configuration results.

[0029] Secondly, the inability to specify configurations and modules means Atlantis will default to executing test commands in the configured directories, failing to execute test code based on specific directories. Furthermore, for code from multiple modules, it cannot select and execute only the selected modules. The default strategy of batch executing IaC tests will increase time consumption as the IaC code volume grows, thus impacting Continuous Integration (CI) and Continuous Delivery / Deployment (CD) processes.

[0030] Third, the problem of environment dependency cannot be resolved. Developers who submit object storage configurations usually do not have the relevant IaC tools and their dependent provider components locally. Furthermore, different environments depend on different tfstate storage state information, which can lead to network connection exceptions when executing code checks.

[0031] Fourth, concurrency issues arise when multiple pull requests (PRs) execute test commands simultaneously, leading to mutual interference. Since Atlantis only serves one PR at a time, the directory and Terraform workspace during plan execution will be locked until the pull request is merged or closed, or the plan is manually deleted. This blocks the test results of other PRs. When multiple PRs compete for execution, deadlocks can easily occur, requiring manual unlocking, which is cumbersome.

[0032] To address at least one of the aforementioned technical problems, this application proposes a data processing method. In this embodiment, the first pull request is an Infrastructure as Code (IaaC) pull request, the first test command indicates the test requirements for testing the IaaC, and the comment information of the first pull request includes the first test command. The method involves receiving the comment information of the first pull request, testing the IaaC based on the comment information to obtain a first test result, and adding the first test result to the comment information of the first pull request. It is evident that the test command in the comment information of the pull request in this embodiment can indicate the test requirements for testing the IaaC, and subsequent testing will also be performed based on this comment information. Therefore, the test requirements for testing the IaaC can be flexibly defined through the test command in the comment information. That is, the test requirements can be defined by defining the comment information of the pull request, thereby controlling the testing of the IaaC through flexible definition of test requirements, improving the flexibility of IaaC testing, and helping to adapt to diverse test requirements.

[0033] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0034] Please see Figure 2 The data processing method provided in this application includes: Step 201: Receive comment information for the first pull request, where the first pull request is an Infrastructure as Code (IMC) pull request, and the comment information includes a first test command, which indicates test requirements for testing the IMC. In practical applications, the data processing method is executed by a first service, which is a containerized service.

[0035] The services provided in this application support containerized deployment, eliminating the need for users to build and maintain the necessary deployment environment themselves. This approach not only simplifies the user's workflow but also ensures high service availability. Furthermore, containerized deployment resolves environment dependency issues, enabling services to run stably in different environments and improving deployment flexibility and reliability.

[0036] For example, the first service could be Orchest.

[0037] For example, the dependencies of the application of the first service are packaged into a container image, such as a Docker image.

[0038] For example, the first service serves as the service entry point for actually processing pull request comments. The first service has one or more IaC tool command parsers built in from Terraform, OpenTofu, and Terragrunt. At the same time, it automatically loads the provider information that the module depends on, ensuring the stability of the environment on which the IaC depends.

[0039] It should be noted that, in this embodiment of the application, the term "pull request" is an exemplary term and can replace or override the following term: "merge request" (MR).

[0040] For example, in this application embodiment, a pull request is used to request configuration changes to the infrastructure as code, such as merging part of the infrastructure as code into the target branch.

[0041] For example, in the embodiments of this application, the first test command is a test command input by the relevant user or a test command generated by a device such as GitLab. For instance, the first test command can be a test command for infrastructure i.e. code input by a developer in the first pull request of GitLab through comments, that is, the first test command can be a test command for infrastructure i.e. code input by a developer in the first comment information of the first pull request of GitLab.

[0042] For example, in the embodiments of this application, testing the Infrastructure as Code can also be regarded as verifying the Infrastructure as Code, and / or as verifying the changes in the Infrastructure as Code.

[0043] For example, in this embodiment of the application, the first test command can be a terragrunt plan.

[0044] For example, the comment information in this application embodiment may include one or more comment messages.

[0045] For example, a valid first test command can be divided or parsed into one or more of the following: The command prompt (cmd) helps determine whether the underlying code depends on Terraform or OpenTofu. The working directory is the directory containing the binary files of the infrastructure, i.e., the code, that should be invoked. The parameters (arguments, args) specify the submodules for which infrastructure (i.e., code testing) and / or the task parameters related to the comment information that need to be performed.

[0046] For example, cmd can be an environment variable TERRAGRUNT_TFPATH, which can be configured according to different commands. TERRAGRUNT_TFPATH indicates which specific Terraform or Tofu executable path to use.

[0047] For example, if the infrastructure, i.e., the code, needs to run in a different directory, then the working-directory must be specified.

[0048] For example, in this embodiment of the application, args is used to specify the submodule that needs to perform infrastructure i.e. code testing, and the related task parameters are an exemplary implementation. In this embodiment of the application, all parameters in the first test command except cmd and working-directory can be included in args.

[0049] For example, since a private IaC deployment tool is provided in this application embodiment, the module and configuration are separated in the private IaC deployment tool. This application embodiment allows the target object (e.g., developers, etc.) to specify a module in the configuration directory, that is, to specify one or more first sub-modules.

[0050] For example, in the embodiments of this application, all parts of the first test command except cmd and working-directory can be used as args.

[0051] For example, the first test command could be ` / terragrunt plan-var sm=config&&PRod-domain / BJ_POF2_SC_POD2_3A2 / bgeiji1 / accesser-group-1-business`. It's understandable that in this test command, `terragrunt` is `cmd`, and `prod-domainm / BJ_POF2_SC_POD2_3A2 / bgeiji1 / accesser-group-1-business` is `working-directory`.

[0052] For example, the relevant parameters in the first test command can be configured by the user and / or GitLab, etc. For example, the user (such as a developer) can specify one or more sub-modules that need to be tested for the base code by configuring the args in the first test command. For example, the user can specify the execution path of the base code test by configuring the working-directory in the first test command.

[0053] In practical applications, the comment information received from the first pull request includes: The system receives the comment information sent by a code hosting platform, wherein the code hosting platform is configured with a Webhook.

[0054] In this embodiment, the GitLab webhook mechanism enables the testing service provided to operate independently without affecting the use of other repositories. This approach ensures the privatization and customization of the service, allowing it to run efficiently in a specific environment without interfering with other business processes.

[0055] For example, the code hosting platform includes one or more of the following: GitHub; GitLab.

[0056] It's worth noting that GitLab is a web-based code hosting platform. It provides infrastructure-as-code hosting, version control, CI / CD, issue tracking, project management, and other features, aiming to help development teams collaborate and deliver software more efficiently.

[0057] For example, by configuring Webhooks in a project, when a user provides a comment in the first test command, the comment must start with a specified prefix, such as terragrunt plan-var xxx-path xxx. GitLab will send the comment to the first service, which will parse the comment, perform basic code tests based on the comment, and reply with the test results below the current comment in the first pull request.

[0058] For example, when a user enters a first test command in the comment information of the first pull request, the first test command will be sent to the first service because of the webhook rules pre-registered in the project. That is, receiving the comment information sent by GitLab includes receiving the first test command sent by GitLab according to the webhook rules of Orchest pre-registered in the project.

[0059] For example, please refer to Figure 3In this embodiment, the first service listens for event requests from GitLab and verifies the legitimacy of the event request based on the token information in the event request to ensure service security. If a PR update exists, the latest infrastructure (i.e., code) is obtained based on the CommitSHA value. For example, if the first pull request in this application is updated, the latest infrastructure (i.e., code) corresponding to the first pull request is obtained. Different PRs are isolated from each other, and a scheduled task is configured to periodically delete closed and expired PR file directories to keep the service storage space normal.

[0060] Step 202: Test the infrastructure, i.e., the code, based on the comment information to obtain the first test result; For example, testing the Infrastructure as Code (ICC) based on the comment information may include one or more of the following: verifying whether the resource configuration of the ICC is reasonable; verifying whether the ICC has syntax errors; verifying whether the ICC can generate a plan; and verifying whether it can be applied correctly.

[0061] For example, the first test result refers to the output information generated after executing the first test command, including one or more of the following: a log of executing the first test command; a status report of testing the infrastructure, i.e., the code; error messages for the infrastructure, i.e., the code; warning messages for the infrastructure, i.e., the code, etc.

[0062] For example, the first service can verify the legitimacy of the first pull request according to pre-configured rules.

[0063] In practical applications, obtaining the first information based on the comment information includes: Determine whether the first test command is a valid message; If the first test command is valid, the first test command is parsed to obtain the first information.

[0064] In this embodiment, by introducing a mechanism for validating the legality of test commands and a structured parsing mechanism, the solution can effectively improve the controllability and security of the IaC testing process. This solution can prevent unauthorized operations. Furthermore, this solution can improve the overall system stability and maintainability.

[0065] For example, determining whether the first test command is legitimate information includes one or more of the following: The validity of the first test command request is determined based on the required token configured in the webhook. The prefix of the first test command is matched, and the validity of the first test command is determined based on the matching result.

[0066] For example, the prefix of the first test command can be matched with a specified prefix format. If the match is successful, the first test command is considered valid; if the match is unsuccessful, the first test command is considered invalid. For example, the prefix of a valid test command can be specified as terragrunt plan.

[0067] For example, since the first service cannot initially determine the type of the received request body, the comment information that receives the first pull request is actually the received request body (such as an Event request). The first service will determine whether the request body is a PR request type based on the content of the request body. If the request body is a PR request type, then the request body is the first PR request in this embodiment of the application. At this time, the comment information that receives the request body can be regarded as the comment information that receives the first PR request. Then, it can be determined whether the comment information of the first PR request is valid, such as determining whether the first test command in the comment information is valid. If the first test command is valid, the comment information will be divided into three parts, namely cmd, working-directory, and args.

[0068] In practical applications, testing the infrastructure, i.e., the code, based on the comment information includes: The first information obtained based on the comment information includes one or more of the following: the execution path of the test task for testing the infrastructure as a code; one or more first sub-modules that need to be tested on the infrastructure as a code; the environment information corresponding to the infrastructure as a code; and task parameters related to the first fetch request. The infrastructure, i.e., the code, is tested based on the first information.

[0069] In this embodiment, the execution path, sub-modules, and environment information for testing the infrastructure (i.e., code) can be flexibly defined through comment information. This allows for flexible selection of the test execution path and the sub-modules to be executed, improving testing flexibility. Since the execution path of the sub-modules can be specified, this embodiment can avoid global testing, reducing the risk of increased time consumption due to the large amount of IaC code affecting the CI / CD process.

[0070] For example, the execution path refers to the specific path under which the infrastructure, i.e., code configuration file, will be tested. Since infrastructure-as-a-code projects typically contain multiple modules and have a configuration-separated design, specifying the execution path can avoid unnecessary global testing and improve testing efficiency.

[0071] For example, the first submodule refers to a portion of the module that only needs to be tested in the infrastructure-as-code modular design, rather than the entire project. Determining the first submodule based on comment information can reduce resource consumption and improve the targeting of tests.

[0072] For example, the first submodule is the submodule specified by args in the first test command.

[0073] For example, environment information refers to the environment variables required during the testing process, such as variables indicating whether the code uses Terraform or OpenTofu, the version of TerraformPRovider used during the testing process, and the storage path of the test state, to ensure that the test can run stably in an isolated simulation environment.

[0074] For example, please see Figure 4 Taking the first service as Orchest as an example, Orchest can pull the infrastructure, i.e., the code, and parse and execute the first test command.

[0075] For example, the task parameters related to the first pull request are obtained from the args in the first test command.

[0076] For example, parsing the first test command yields the first information, including one or more of the following: the environment information corresponding to the infrastructure, i.e., the code, can be obtained from the cmd in the first test command; and the task parameters related to one or more first sub-modules that need to be tested for the infrastructure, i.e., the code, and the first pull request can be obtained from the args in the first test command.

[0077] In practical applications, the first test command includes the directory of the infrastructure, i.e., the binary files of the code, and obtaining the first information based on the comment information includes: The execution path is obtained from the directory of the binary files of the infrastructure, i.e., the code.

[0078] In this embodiment, determining the execution path based on the directory of the Infrastructure as Code (ICC) binary file ensures that ICC commands run in the correct context, avoiding configuration conflicts or execution failures caused by incorrect paths. Using the ICC working directory as input to determine the execution path not only improves the accuracy and reliability of testing but also enhances the system's adaptability in complex projects. This method also supports flexible testing under modular architectures, allowing users to execute tests on specific code without affecting the configuration of other parts.

[0079] For example, the infrastructure is the directory of the binary files of the code, i.e., the working directory.

[0080] For example, the environment information corresponding to the infrastructure as code can be configured according to the first comment information, and the corresponding execution path and first parameter can be determined. The execution path is the execution path of the test task that tests the infrastructure as code, that is, the execution path of the first test command. For example, the first parameter includes one or more first sub-modules that need to be tested by the infrastructure as code. For example, the parameter may also include task parameters related to the first pull request.

[0081] For example, testing the infrastructure-as-code based on the comment information includes one or more of the following: executing a first test command in the execution path; and testing the infrastructure-as-code of a first submodule. Wherein, the execution path is the execution path of the test task indicating the first test command to test the infrastructure-as-code; and the first submodule is the first submodule indicating the first test command that requires testing the infrastructure-as-code.

[0082] In practical applications, the method further includes: Configure the storage path of the tfstate files associated with the execution of tests of the infrastructure i.e. code as a first path, which is different from the storage path of the files in the production environment.

[0083] In this embodiment, by configuring the storage path of the tfstate file related to the test of the execution infrastructure i.e. code as the first path, and distinguishing this configuration operation from the storage path of the file in the live network environment, the test environment is isolated from the live network environment, avoiding the impact of the test process on live network resources, and ultimately improving the system security and stability.

[0084] For example, the tfstate file associated with the tests that execute the infrastructure, i.e., the code, can be stored locally.

[0085] For example, configuring the storage path of the tfstate files related to the execution of the Infrastructure as Code (IoC) tests as the first path includes: updating the storage path of the IoC test-related tfstate files in the pre-task flow to ensure that the IoC test-related tfstate files are isolated from the live network environment. Simultaneously, an initialization task is executed to ensure that the simulated environment is similar to the actual required environment.

[0086] Step 203: Add the first test result to the comment information of the first pull request.

[0087] In practical applications, adding the first test result to the comment information of the first fetch request includes: According to the first test command, the first test result is added to the comment information of the first pull request, wherein the first test result and the first test command have a corresponding relationship in the comment information; The method further includes: The comment information corresponding to the added first test result is displayed.

[0088] In this embodiment of the application, by establishing a one-to-one correspondence between the first test result and the first test command in the comment information and adding them to the PR, and displaying them visually, developers can intuitively understand the status of the IaC configuration and the test results, and can promptly discover errors in the IaC, etc., helping developers to solve problems as early as possible in the simulation environment, thereby reducing possible errors and risks during actual deployment.

[0089] Understandably, in the above practical applications, the first test results can be displayed visually and fed back to the developers in the form of comments on the first pull request.

[0090] For example, adding the first test result to the comment information of the first pull request includes: replying the first test result in markdown-native format to the comment information of the first pull request.

[0091] Understandably, the above example is to ensure that the test results correspond one-to-one with the input test commands.

[0092] For example, the first test result is added to the comment information of the first pull request, including one or more of the following: replying the first test result directly in text form below the first test command in the first comment information; uploading the first test result as an attachment or link for the user to view.

[0093] For example, all or part of the content in the first test result can be displayed according to the user's instructions and / or the first test command, that is, the embodiments of this application can support customized display of test results with specific content, filter out redundant information, and ensure the simplicity and high readability of the output information.

[0094] The embodiments of this application can support customized display of execution results for specific content. For example, the method further includes: displaying deployment environment information so that developers can intuitively understand the status of the IaC configuration and the corresponding test results.

[0095] The technical solution of this application involves two main components: a first service (e.g., orchtest) and a code hosting platform (e.g., GitLab). These two components work together to enable the commenting of Infrastructure as Code (IaC) test commands in Pull Requests (PRs), while ensuring environment isolation, security, and output visualization. This application proposes a method for verifying Infrastructure as Code (IaC) configurations through GitLab comments. This method enables visual test analysis of IaC code, thereby greatly improving the ease of use and accuracy of IaC deployment tools.

[0096] The embodiments of this application have the following advantages: First, the custom object storage deployment tool is more adaptable. The embodiments of this application can separate modules from configuration. The method of these embodiments can adapt to custom object storage deployment tool code and support its module-configuration separation architecture. This means that deployment commands can be correctly executed in the corresponding configuration path, ensuring the accuracy and consistency of IaC configuration.

[0097] Secondly, containerized deployment and high availability: The first service utilizes containerized deployment, eliminating the need for users to build and maintain the necessary environments themselves. This approach not only simplifies the user experience but also ensures high service availability. Furthermore, containerized deployment resolves environment dependencies, enabling the service to run stably in different environments, thus improving deployment flexibility and reliability.

[0098] Third, regarding service privatization and customization, through GitLab's webhook mechanism, this embodiment can operate independently without affecting the use of other repositories. This approach ensures the privatization and customization of services, enabling them to run efficiently in a specific environment without interfering with other business processes.

[0099] Fourth, visualization and error detection: The first service can display deployment environment information in a visual way, enabling developers to intuitively understand the status and results of IaC configuration, and to promptly discover errors and undefined situations in nested configurations. This helps developers solve problems as early as possible in the simulation environment, thereby reducing possible errors and risks during actual deployment.

[0100] As can be seen from the above, the technical solution of this application not only improves the verification efficiency and accuracy of IaC configuration, but also enhances the availability and maintainability of the service. The visualization and error detection mechanism ensures that developers can promptly identify and resolve problems in the simulation environment, thereby improving the overall project quality and reliability.

[0101] The present application will be further described in detail below with reference to application examples.

[0102] Example 1 Example 1 uses orchtest as the first service and GitLab as the code hosting platform. The technical solution of Example 1 involves two main components: orchtest and GitLab. The orchtest and GitLab components work together to enable the ability to comment on and execute Infrastructure as Code (IaC) tests in Pull Requests (PRs), while ensuring environment isolation, security, and visualization of output.

[0103] The following sections will introduce orchtest and GitLab respectively.

[0104] orchtest serves as the service entry point for actually processing the comments in the first pull request. orchtest provides one or more IaC tool command parsers from terraform, opentofu, and terragrunt, and automatically loads the provider information that the module depends on, ensuring the stability of the environment on which IaC depends.

[0105] When a developer enters the first test command in the comment section of the first pull request, this command is sent to the orchest via pre-registered webhook rules. The orchest then uses the required token configured in the webhook to determine the validity of the first test command, ensuring security. The orchest further determines whether the request body is a PR request and then checks the validity of the comment information. Valid comments are divided into three parts: cmd, working-directory, and args.

[0106] The specific orchest task flow is as follows: Orchest detects Event requests from GitLab and verifies their validity based on the token information within the Event request, ensuring service security. If a PR is updated, the latest code is retrieved based on the CommitSHA value. Different PRs are isolated from each other, and a scheduled task deletes directories of closed and expired PR files to ensure adequate service storage space.

[0107] In the pre-task flow, Orchest updates the storage path of the tfstate files related to the test execution infrastructure (code) to ensure isolation from the production environment. It also performs initialization tasks to ensure the simulated environment is similar to the actual required environment. Based on the comments input by developers, it determines whether test tasks are necessary. If the prefix match is valid, it checks the validity of the first input test command. The first test command is parsed into three parts: execution path, variables, and module. Orchest generates test tasks based on the parsing results of the first test command, executes the tests, and replies the test output (i.e., the first test result) in markdown-native format to the first comment information of the first pull request, ensuring a one-to-one correspondence between the first test result and the first input test command.

[0108] Orchest offers lightweight operation and containerized deployment, achieving service portability and resolving environment dependencies. Simultaneously, its efficient command parsing and task execution capabilities ensure stable concurrent requests. Visualized test result output allows developers to clearly verify test results before deployment and configuration, guaranteeing quality and safety.

[0109] GitLab is a web-based code hosting platform. It provides features such as code hosting, version control, continuous integration / continuous deployment (CI / CD), issue tracking, and project management, aiming to help development teams collaborate and deliver software more efficiently.

[0110] Users (such as developers) can configure webhooks in their projects so that when a user provides a first comment in a PR with a specified prefix, such as "terragrunt plan-var xxx-path xxx", the GitLab server will send the first comment to orchtest. Orchtest will parse the first comment, execute the test, obtain the first test result, and reply with the first test result below the current comment (such as the first test command).

[0111] Please see Figure 5 The specific process is as follows: S501: Developers enter the first test command for infrastructure, i.e., code, via comments in GitLab's first pull request; S502: GitLab sends the first comment information to orchtest via a Webhook mechanism.

[0112] S503: orchtest verifies the validity of the first pull request based on pre-configured rules.

[0113] For example, verifying the legality of the first pull request according to pre-configured rules includes: determining the legality of the first test command. For example, the pre-configured rules can be pre-configured prefix matching rules, which determine whether the first test command is legal by performing prefix matching on the first test command.

[0114] S504: orchtest configures environment variables based on the first comment information and prepares the corresponding execution path and parameters.

[0115] For example, the above parameters include one or more of the following: submodules that need to perform infrastructure-i.e. code testing; and task parameters related to the comment information.

[0116] S505: orchtest executes the first test command in the specified configuration path and obtains the first test result.

[0117] For example, the configuration path specified above is the execution path specified by the first test command.

[0118] S506: Orchtest displays the first test results in a visual manner and provides feedback to the developers in the form of comments.

[0119] For example, displaying the first test result in a visual manner includes: replying the first test result in markdown-native format to the comment information of the first pull request and displaying it to the user.

[0120] As can be seen from the above, compared with related technologies, the "orchest-plan-comment" scheme proposed in this application has the following advantages: First, it has greater flexibility and independence. By commenting on the execution of the Terraform plan operation in the Pull Request, that is, by adding test commands to the comments of the pull request, developers can execute test code on any code path and can execute it based on the latest commit without affecting the execution of other Pull Requests. This method improves the flexibility of testing and ensures the independence of each Pull Request. Second, this application embodiment can achieve environment isolation. By replacing the relevant configurations in the live network environment, it can ensure that the test code can be tested in a private development environment. For example, Terraform's tfstate will be replaced with local storage, thereby ensuring complete isolation from the live network environment and avoiding potential risks and interference. Third, this application embodiment can achieve customized output. This application embodiment can support customized display of execution results for specific content, filter out redundant information, and ensure the conciseness and high readability of the output information. This not only improves the efficiency of developers' understanding but also enhances the quality of code review. Fourth, the embodiments of this application support multi-tool compatibility. These embodiments support Terraform and OpenTofu, and are compatible with derivative tools such as Terraragrunt. Users can verify IaC code without pre-configuring related dependency environments, simplifying the development process and lowering the entry barrier.

[0121] This application's embodiments can serve as the technical foundation for IaC configuration verification. While related technologies do not support customized functional design, this application's embodiments can provide customized functional design, offering a solution for verifying the interdependence between IaC and the environment. This application's embodiments support multiple IaC tools, including Terraform, OpenTofu, and Terraragrunt command parsing, applicable to mainstream IaC code verification testing. It is necessary to check whether existing technologies also support these tools and their command parsing functions. This application's embodiments implement containerized deployment, supporting continuous delivery and continuous integration, eliminating the need for users to build the environments on which deployment depends. This application's embodiments support local storage of tfstate files, ensuring isolation between tfstate files and the live network environment.

[0122] Based on the embodiments described above, this application also provides a data processing apparatus, see [link to previous document]. Figure 6 The data processing device includes: The first receiving unit 601 is used to receive comment information of a first pull request, wherein the first pull request is an infrastructure as code pull request, and the comment information includes a first test command, wherein the first test command indicates test requirements for testing the infrastructure as code; The first test unit 602 is used to test the infrastructure, i.e., the code, based on the comment information to obtain a first test result; The first adding unit 603 is used to add the first test result to the comment information of the first pull request.

[0123] In one embodiment, the first testing unit 602 tests the infrastructure, i.e., the code, based on the comment information, including: The first information obtained based on the comment information includes one or more of the following: the execution path of the test task for testing the infrastructure as a code; one or more first sub-modules that need to be tested on the infrastructure as a code; the environment information corresponding to the infrastructure as a code; and task parameters related to the first fetch request. The infrastructure, i.e., the code, is tested based on the first information.

[0124] In one embodiment, the first testing unit 602 obtains first information based on the comment information, including: Determine whether the first test command is a valid message; If the first test command is valid, the first test command is parsed to obtain the first information.

[0125] In one embodiment, the first test command includes a directory of binary files of the infrastructure, i.e., the code, and the first test unit 602 obtains first information based on the comment information, including: The execution path is obtained from the directory of the binary files of the infrastructure, i.e., the code.

[0126] In one embodiment, the first receiving unit 601 receives comment information from the first pull request, including: The system receives the comment information sent by a code hosting platform, wherein the code hosting platform is configured with a Webhook.

[0127] In one embodiment, the device further includes a first display unit, wherein the first adding unit 603 adds the first test result to the comment information of the first pull request, including: According to the first test command, the first test result is added to the comment information of the first pull request, wherein the first test result and the first test command have a corresponding relationship in the comment information; The first display unit is used to display the comment information corresponding to the addition of the first test result.

[0128] In one embodiment, the apparatus further includes a first configuration unit, the first configuration unit being configured to: Configure the storage path of the tfstate files associated with the execution of tests of the infrastructure i.e. code as a first path, which is different from the storage path of the files in the production environment.

[0129] In one embodiment, the data processing device is configured with a first service, wherein the first receiving unit 601, the first testing unit 602, and the first adding unit 603 are controlled by the first service, which is a containerized service.

[0130] In practical applications, the first test unit 602, the first addition unit 603, the first display unit, the first configuration unit, etc., can be implemented by the processor in the data processing device, and the first receiving unit 601 can be implemented by the processor in the data processing device in combination with the communication interface.

[0131] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0132] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, this application also provides an electronic device, see [link to relevant documentation]. Figure 7 The electronic device includes: a first communication interface 1, a first processor 2, and a first memory 3.

[0133] Specifically, the first communication interface 1 is used to receive comment information of a first pull request, the first pull request being an infrastructure as code pull request, and the comment information including a first test command, the first test command indicating test requirements for testing the infrastructure as code; The first processor 2 is configured to test the infrastructure, i.e., the code, based on the comment information to obtain a first test result; and to add the first test result to the comment information of the first pull request.

[0134] In one embodiment, the first processor 2 tests the infrastructure, i.e., the code, based on the comment information, including: The first information obtained based on the comment information includes one or more of the following: the execution path of the test task for testing the infrastructure as a code; one or more first sub-modules that need to be tested on the infrastructure as a code; the environment information corresponding to the infrastructure as a code; and task parameters related to the first fetch request. The infrastructure, i.e., the code, is tested based on the first information.

[0135] In one embodiment, the first processor 2 obtains first information based on the comment information, including: Determine whether the first test command is a valid message; If the first test command is valid, the first test command is parsed to obtain the first information.

[0136] In one embodiment, the first test command includes a directory of binary files of the infrastructure, i.e., the code, and the first communication interface 1 obtains first information based on the comment information, including: The execution path is obtained from the directory of the binary files of the infrastructure, i.e., the code.

[0137] In one embodiment, the first communication interface 1 receives comment information from the first pull request, including: The system receives the comment information sent by a code hosting platform, wherein the code hosting platform is configured with a Webhook.

[0138] In one embodiment, the first processor 2 adds the first test result to the comment information of the first fetch request, including: According to the first test command, the first test result is added to the comment information of the first pull request, wherein the first test result and the first test command have a corresponding relationship in the comment information; The first processor 2 is also used to display the comment information corresponding to the addition of the first test result.

[0139] In one embodiment, the first processor 2 is further configured to: Configure the storage path of the tfstate files associated with the execution of tests of the infrastructure i.e. code as a first path, which is different from the storage path of the files in the production environment.

[0140] In one embodiment, the electronic device is configured with a first service, a first communication interface 1, and a first memory 3 controlled by the first service, which is a containerized service.

[0141] It should be noted that the specific processing procedure of the first communication interface 1 can be understood by referring to the above method.

[0142] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general will label all buses as Bus System 4.

[0143] The first memory 3 in this embodiment is used to store various types of data to support operation in the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0144] The methods disclosed in the embodiments of this application can be applied to the first processor 2, or implemented by the first processor 2. The first processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 2. The first processor 2 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 3. The first processor 2 reads the information in the first memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0145] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0146] It is understood that the first memory 3 in this embodiment can be a volatile memory pool or a non-volatile memory pool, or both. The non-volatile memory pool can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory pool can be a disk storage pool or a magnetic tape storage pool. The volatile memory pool can be a random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The storage pools described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of storage pools.

[0147] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 3 storing a computer program, which can be executed by a first processor 2 to complete the steps described in the aforementioned method.

[0148] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a first processor 2 to perform the steps described in the foregoing method.

[0149] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The data processing method includes: The system receives comment information about a first pull request, where the first pull request is an Infrastructure as Code (IMC) pull request. The comment information includes a first test command, which indicates test requirements for testing the IMC. The infrastructure, i.e., the code, is tested based on the comment information to obtain a first test result; Add the first test result to the comment information of the first pull request.

2. The method according to claim 1, characterized in that, The testing of the infrastructure, i.e., the code, based on the comment information includes: The first information obtained based on the comment information includes one or more of the following: the execution path of the test task for testing the infrastructure as a code; one or more first sub-modules that need to be tested on the infrastructure as a code; the environment information corresponding to the infrastructure as a code; and task parameters related to the first fetch request. The infrastructure, i.e., the code, is tested based on the first information.

3. The method according to claim 2, characterized in that, The step of obtaining the first information based on the comment information includes: Determine whether the first test command is a valid message; If the first test command is valid, the first test command is parsed to obtain the first information.

4. The method according to claim 2, characterized in that, The first test command includes the directory of the infrastructure, i.e., the binary files of the code, and the step of obtaining the first information based on the comment information includes: The execution path is obtained from the directory of the binary files of the infrastructure, i.e., the code.

5. The method according to claim 1, characterized in that, The comment information received from the first pull request includes: The system receives the comment information sent by a code hosting platform, wherein the code hosting platform is configured with a Webhook.

6. The method according to claim 1, characterized in that, Adding the first test result to the comment information of the first fetch request includes: According to the first test command, the first test result is added to the comment information of the first pull request, wherein the first test result and the first test command have a corresponding relationship in the comment information; The method further includes: The comment information corresponding to the added first test result is displayed.

7. The method according to claim 1, characterized in that, The method further includes: Configure the storage path of the tfstate files associated with the execution of tests of the infrastructure i.e. code as a first path, which is different from the storage path of the files in the production environment.

8. The method according to claim 1, characterized in that, The data processing method is executed by a first service, which is a containerized service.

9. A data processing apparatus, characterized in that, include: The first receiving unit is configured to receive comment information of a first pull request, wherein the first pull request is an infrastructure-as-code pull request, and the comment information includes a first test command, wherein the first test command indicates test requirements for testing the infrastructure-as-code. The first testing unit is used to test the infrastructure, i.e., the code, based on the comment information, and obtain a first test result; The first adding unit is used to add the first test result to the comment information of the first pull request.

10. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 8.

11. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images