Automatic management and version issuing method and device for large model algorithm scheduling codes, equipment and medium

Abstract

The invention relates to the field of code management, in particular to an automatic management and version issuing method, device and equipment for large model algorithm scheduling codes and a medium, and the method comprises the steps that the large model algorithm scheduling codes are cloned from a code warehouse to a local server, and cross-operating system line feed automatic conversion is set; packaging the cloned code into a compressed file with a set suffix, and calculating a hash value of the compressed file to generate a verification file; uploading the compressed file and the verification file thereof to a DEV directory of an OSS internal barrel; downloading the compressed file to a test environment for decompression and test verification; after the test is passed, downloading a compressed file to a target server, removing a suffix after decompression, repackaging the compressed file into a formal version issuing medium, and calculating a hash value; and respectively uploading the official version issuing medium and the verification file thereof to the Release catalogues of the internal barrel and the external barrel of the OSS. Through separation of an OSS internal barrel and an OSS external barrel, standardized storage of code media is realized, and cross-department use is facilitated.

Description

Technical Field

[0001] This application relates to the field of code management technology, specifically to a method, apparatus, device, and medium for automated management and release of large-scale model algorithm-based code scheduling. Background Technology

[0002] In the field of large-scale model development, the management, storage, and release of algorithm scheduling code are core aspects of ensuring efficient project progress and stable operation. As the scale and complexity of large-scale models continue to expand, traditional technical methods are gradually revealing many limitations in dealing with these aspects, making it difficult to meet the stringent requirements of efficiency, security, and standardization in modern large-scale model development.

[0003] Although some existing technologies have incorporated version control systems (such as Git), the creation, verification, and pushing of version tags still lack a robust automated mechanism in the management of large-scale algorithm scheduling code. This leads to version numbers being easily confused and makes it difficult to quickly locate the change history of a specific version, increasing the difficulty of troubleshooting and code backtracking.

[0004] Large-scale development environments often involve multiple operating systems, but existing technologies tend to overlook file format differences when handling cross-platform code transfer, causing code to malfunction on different platforms. This not only increases the debugging burden on developers but may also lead to potential compatibility issues. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method, apparatus, device, and medium for automated management and release of large-scale model algorithm scheduling code. By introducing highly automated version management, refined storage security control, standardized release processes, cross-platform compatibility handling, and a comprehensive testing and verification mechanism, it effectively solves the shortcomings of existing technologies.

[0006] In a first aspect, the technical solution of the present invention provides a method for automated management and release of large model algorithm scheduling code, including the following steps: S1. Clone the large model algorithm scheduling code from the GitLab code repository to the local server via the Jenkins pipeline. During the cloning process, set the core.autocrlf parameter to achieve automatic newline character conversion across operating systems. S2. Package the cloned code into a compressed file with a specified suffix, and calculate its hash value to generate a verification file; S3. Upload the packaged compressed file and its verification file to the DEV directory of the OSS internal bucket; S4: Download the compressed file from the DEV directory to the test environment via the Jenkins pipeline for decompression and testing; if the test fails, return to S1 and re-execute; if the test passes, execute S5. S5. Download the compressed file from the DEV directory of the OSS internal bucket to the target server, decompress it, remove the specified suffix, repackage it as the official release medium, and recalculate the hash value; S6. Upload the official release media and its verification files to the Release directory of the OSS internal bucket and external bucket, respectively.

[0007] By automatically handling line break differences during code cloning and transfer, the file format compatibility issue between Windows and Linux systems is resolved, code parsing errors caused by inconsistent line breaks are avoided, and cross-platform debugging time is reduced.

[0008] By packaging, verifying, and uploading to OSS, the consistency and integrity of code versions are ensured, facilitating subsequent management and traceability. Unpacking and testing in a test environment ensures that only code versions that pass the tests will enter the official release process, improving code quality. Uploading the official release media and its verification files to the OSS Release directory achieves standardized storage and delivery of code media, facilitating cross-departmental use.

[0009] As a further limitation of the technical solution of the present invention, step S1 includes: The system obtains input parameters from the Jenkins pipeline, including the target tag name, Git repository address, and branch name. It uses Git commands to clone the large model algorithm scheduling code from the Git repository address to the local server. During the cloning process, the `core.autocrlf` parameter of `git clone` and `git checkout` is configured to automatically convert line breaks between operating systems.

[0010] By obtaining input parameters (such as target tag name, Git repository address, and branch name) from the Jenkins pipeline, the entire process becomes more flexible and can adapt to different development needs.

[0011] Using Git commands to clone code from a Git repository ensures the automation and accuracy of the cloning process. By configuring the `core.autocrlf` parameter of `git clone` and `git checkout` to automatic conversion mode, cross-operating system line break issues are properly handled, avoiding code formatting problems caused by line break issues.

[0012] As a further limitation of the technical solution of the present invention, step S1 is followed by: S1.1 Switch to the branch specified in the input parameters, query the existing tags in the Git repository, and compare the target tag name with the query results; if the target tag already exists, stop running and report a warning; if the target tag does not exist, dynamically create a new Git tag based on the obtained input parameters.

[0013] By querying existing tags in the Git repository and comparing them with the target tag name, the uniqueness of the target tag is ensured, avoiding version confusion caused by duplicate tag creation. If the target tag already exists, the process stops and a warning is issued, promptly reminding developers to check the code version and avoiding unnecessary duplication of work and potential errors. Dynamically creating new Git tags based on the obtained input parameters makes the tag creation process more flexible and automated, adapting to dynamic development environments.

[0014] As a further limitation of the technical solution of the present invention, the step of dynamically creating new Git tags based on the acquired input parameters includes: Dynamically create new Git tags based on input parameters, and record the timestamp of tag creation and relevant information about the creator; Push the newly created tags to the Git repository and write the tag information to the log file.

[0015] Recording the timestamp of tag creation and the creator's information facilitates subsequent version tracking and auditing, ensuring that each version of the code is clearly documented. Pushing newly created tags to the Git repository and writing tag information to a log file further standardizes the version management process, facilitating team collaboration and code management. By recording and storing tag-related information, the integrity and queryability of version management information are ensured, facilitating subsequent troubleshooting and version control.

[0016] As a further limitation of the technical solution of the present invention, step S2 includes: Use the tar command to package the code directory into a tar.gz compressed file; The SHA256 hash value of the compressed file is calculated using the sha256sum command, and a hash string is generated. Write the hash string to a verification file with the same name as the compressed file; the verification file contains the compressed file name and its corresponding hash value.

[0017] Using the `tar` command to package the code directory into a `tar.gz` compressed file ensures standardization and consistency in the packaging process. The `sha256sum` command calculates the SHA256 hash value of the compressed file, generating a checksum file to ensure the integrity and consistency of the packaged files, facilitating subsequent verification and traceability. Writing the hash string into a checksum file with the same name as the compressed file facilitates subsequent file integrity verification, improving the security of file transfer and storage.

[0018] As a further limitation of the technical solution of the present invention, step S4 includes: S41. Use the Jenkins pipeline to call the s3cmd tool to download the compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to the specified directory of the test environment. S42. Use the sha256sum command to verify the integrity of the downloaded compressed file. If the verification fails, terminate the process and report an error; if the verification passes, proceed to S43. S43. Use the tar command to decompress the verified compressed file to the deployment directory of the test environment; S44. Modify the configuration file and start the service, then execute the test cases. If the test fails, delete the decompressed files in the test environment and trigger the Jenkins pipeline to re-execute S1 to S3. If the test passes, execute S45. S45. Generate a test pass report and mark this version of the code as eligible for the official release process.

[0019] In the test environment, the integrity of downloaded compressed files is verified using the sha256sum command, ensuring that the files were not corrupted or tampered with during transmission. The Jenkins pipeline automates the download, decompression, and testing verification process, improving testing efficiency and reducing manual intervention. If a test fails, the decompressed files are deleted from the test environment, triggering a re-execution of the process, ensuring a clean test environment and the accuracy of subsequent operations. A test pass report is generated, and the code version is marked as eligible for the official release process, facilitating subsequent version management and delivery.

[0020] As a further limitation of the technical solution of the present invention, step S5 includes: S51. Use the Jenkins pipeline to call the s3cmd tool to download the compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to the temporary working directory of the target server. S52. Use the sha256sum command to verify the integrity of the downloaded file. If the verification fails, terminate the process and record the error log; if the verification passes, proceed to S53. S53. Use the tar command to extract the compressed file to the target directory, and automatically remove the DEV suffix from the file name and directory name; S54. Use the sed or rename command to modify the DEV identifier contained in all configuration files in the decompressed directory; S55. Repackage the processed code directory into a formal release medium that does not contain the DEV suffix. S56. Use the sha256sum command to recalculate the hash value of the official release media and generate the corresponding verification file; S57. Temporarily store the official release media and its verification files in the release preparation area of ​​the target server.

[0021] On the target server, a series of operations (such as decompression, removal of file extensions, and modification of configuration files) prepare the code for the official release medium, ensuring its accuracy and consistency. The hash value of the official release medium is recalculated, and a corresponding verification file is generated, further ensuring its integrity and consistency. The official release medium and its verification file are temporarily stored in the release preparation area of ​​the target server, facilitating subsequent upload and delivery operations and improving the standardization and security of the release process.

[0022] Secondly, the present invention also provides an automated management and release device for large model algorithm scheduling code, comprising: Code cloning module: Used to clone large model algorithm scheduling code from GitLab code repository to local server via Jenkins pipeline. During the cloning process, the core.autocrlf parameter is set to enable automatic newline character conversion across operating systems. Packaging and Verification Module: Used to package the cloned code into a compressed file with a specified suffix, and calculate its hash value to generate a verification file; Upload module: Used to upload the packaged compressed file and its verification file to the DEV directory of the OSS internal bucket; The testing and verification module is used to download compressed files from the DEV directory to the test environment via the Jenkins pipeline for decompression and testing. If the test fails, it triggers the code cloning module to be re-executed; if the test passes, it triggers the official release module. The official release module is used to download compressed files from the DEV directory of the OSS internal bucket to the target server, decompress them, remove the specified suffixes, repackage them as official release media, and recalculate the hash value. Release upload module: Used to upload the official release media and its verification files to the Release directory of the OSS internal bucket and external bucket, respectively.

[0023] Modular design allows each functional module to operate and be maintained independently, improving the device's scalability and maintainability. Automated processes through the device further enhance the efficiency and standardization of code management and release, reduce the possibility of human error, and improve code quality and delivery efficiency.

[0024] As a further limitation of the technical solution of the present invention, the code cloning module includes: Input parameter acquisition unit: used to obtain input parameters from the Jenkins pipeline, including target tag name, Git repository address and branch name; Cloning unit: Used to clone code from a Git repository address to the local server using Git commands. During the cloning process, the core.autocrlf parameter of git clone and git checkout is configured to automatically convert line breaks between operating systems.

[0025] As a further limitation of the technical solution of the present invention, the code cloning module further includes: Branch switching unit: Used to switch to the branch specified in the input parameters; Tag checking unit: Used to query existing tags in the Git repository and compare the target tag name with the query results; if the target tag already exists, it stops running and reports a warning; if the target tag does not exist, it triggers the tag creation unit; Tag creation unit: Used to dynamically create new Git tags based on the obtained input parameters.

[0026] As a further limitation of the technical solution of the present invention, the tag creation unit includes: Tag generation subunit: Used to dynamically create new Git tags based on input parameters, and record the timestamp of tag creation and information about the creator; Tag Push Subunit: Used to push newly created tags to the Git repository and write tag information to the log file.

[0027] As a further limitation of the technical solution of the present invention, the packaging and verification module includes: Packaging unit: Used to package a code directory into a tar.gz compressed file using the tar command; Hash Calculation Unit: Used to calculate the SHA256 hash value of the compressed file using the sha256sum command, and generate a hash string; Verification file generation unit: used to write the hash string into a verification file with the same name as the compressed file; the content of the verification file includes the compressed file name and its corresponding hash value.

[0028] As a further limitation of the technical solution of the present invention, the test verification module includes: Download Unit: Used to call the s3cmd tool through the Jenkins pipeline to download a compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to a specified directory in the test environment; Integrity verification unit: Used to verify the integrity of the downloaded compressed file using the sha256sum command. If the verification fails, the process is terminated and an error is reported; if the verification passes, the decompression unit is triggered. Decompression unit: Used to decompress verified compressed files to the deployment directory of the test environment using the tar command; Test Execution Unit: Used to modify configuration files and start services, and execute test cases; if the test fails, the decompressed files in the test environment are deleted and the code clone module is re-executed; if the test passes, a test pass report is generated and the version of the code is marked as allowed to enter the official release process.

[0029] As a further limitation of the technical solution of the present invention, the formal release module includes: Download Unit: Used to call the s3cmd tool through the Jenkins pipeline to download a compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to the temporary working directory of the target server. Integrity verification unit: Used to verify the integrity of the downloaded file using the sha256sum command. If the verification fails, the process is terminated and an error log is recorded; if the verification passes, the decompression unit is triggered. Decompression and renaming unit: Used to decompress compressed files to the target directory using the tar command, and automatically remove the DEV suffix from filenames and directory names; Configuration Modification Unit: Used to modify the DEV identifier contained in all configuration files within the unzipped directory using the sed or rename command; Repackaging unit: Used to repackage the processed code directory into a formal release medium that does not contain the DEV suffix; Hash recalculation unit: used to recalculate the hash value of the official release media using the sha256sum command and generate the corresponding verification file; Temporary storage unit: Used to temporarily store the official release media and its verification files in the release preparation area of ​​the target server.

[0030] Thirdly, the present invention also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing computer program instructions executable by the at least one processor, the computer program instructions being executed by the at least one processor to enable the at least one processor to execute the large model algorithm scheduling code automated management and release method as described in the first aspect.

[0031] Fourthly, the present invention also provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute the large model algorithm scheduling code automated management and release method as described in the first aspect.

[0032] As can be seen from the above technical solutions, this application has the following advantages: Deep integration of the Jenkins pipeline with Git enables automatic creation, verification, and pushing of version tags, improving the accuracy and efficiency of version management. Utilizing OSS cloud storage services, combined with granular access control and data encryption measures, ensures the security and reliability of code storage. Defining standardized release steps and verification mechanisms reduces human intervention, lowers the release error rate, and improves the stability and reliability of releases. Automatic handling of file format differences such as line breaks during code cloning and transmission ensures code compatibility across different platforms. Comprehensive testing and verification are performed before release, including code integrity checks and configuration correctness verification, generating detailed test reports to provide strong support for release decisions. Separation of internal and external OSS buckets achieves standardized storage of code media, facilitating cross-departmental use. Attached Figure Description

[0033] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.

[0035] Figure 2 A block diagram of the apparatus provided in an embodiment of the present invention. Detailed Implementation

[0036] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0038] This invention utilizes a Jenkins pipeline to call a shell script on a server, specifically including the following steps: First, the Jenkins pipeline clones the large model algorithm scheduling code from the GitLab code repository to the local server, setting the `core.autocrlf` parameter during the cloning process to automatically convert line breaks across operating systems; then, the `tar` command is used to package the code directory into a `tar.gz` compressed file, and the `sha256sum` command is used to calculate the SHA256 hash value of the compressed file, generating a verification file; finally, the packaged compressed file and its verification file are uploaded to the `DEV` directory of the OSS internal bucket. This technical solution improves the efficiency and security of code management by automating the handling of line break differences, packaging, and verification steps, and solves the problems of formatting errors and file corruption that occur when code is transferred between different operating systems in existing technologies.

[0039] Jenkins pipelines support dynamic parameterized builds, enabling flexible task execution based on input parameters (such as Git repository address, branch, tag name, etc.), ensuring efficient and flexible code management. This invention utilizes Jenkins pipelines' dynamic parameterized build functionality to flexibly execute tasks such as code cloning, packaging, verification, uploading, downloading, decompression, testing, and official release based on input parameters (such as Git repository address, branch, tag name, etc.). In this way, this invention can quickly adjust the code management process according to the needs of different projects, improving the efficiency and flexibility of code management and solving the problem of fixed code management processes in existing technologies that are difficult to adapt to different project requirements.

[0040] like Figure 1 As shown, this embodiment of the invention provides a method for automated management and release of large model algorithm scheduling code, including the following steps: S1. Clone the large model algorithm scheduling code from the GitLab code repository to the local server via the Jenkins pipeline. During the cloning process, set the core.autocrlf parameter to achieve automatic newline character conversion across operating systems. Jenkins is an open-source tool widely used for continuous integration and continuous delivery (CI / CD). It automates complex software development processes, including code building, testing, and deployment. In large-scale development, Jenkins effectively manages algorithmic code versioning, ensuring consistency across development, testing, and production environments.

[0041] The Jenkins pipeline clones algorithm scheduling code related to large models from the GitLab code repository to the local Jenkins server. During the cloning process, the `core.autocrlf` setting in `git clone` ensures that line endings across different operating systems are handled correctly. The script checks if the target tag already exists to avoid duplicate creation and dynamically creates new Git tags to manage code versions.

[0042] The Jenkins pipeline script clones the required code version from the input Git repository address and branch name. During the cloning process, different operating systems handle line breaks differently. Windows systems use a combination of carriage return and line feed to represent a newline, while Linux / Unix systems use only a single line feed. This difference can lead to formatting issues when transferring code between different operating systems. To handle this cross-platform line break problem, an automatic line break conversion configuration is set up to automatically convert line breaks during cloning (git clone) and checkout, ensuring code consistency across different operating systems.

[0043] S1.1 Switch to the branch specified in the input parameters, query the existing tags in the Git repository, and compare the target tag name with the query results; if the target tag already exists, stop running and report a warning; if the target tag does not exist, dynamically create a new Git tag based on the obtained input parameters.

[0044] Specifically, the Jenkins pipeline script checks whether the target tag already exists in the Git repository. A target tag is a specific label that already exists in the Git repository and is used to identify a specific version or milestone of the code. If the target tag already exists, the script will stop running and issue a warning; if the target tag does not exist, it needs to create a new target tag based on the input parameters to manage code versions.

[0045] S2. Package the cloned code into a compressed file with a specified suffix, and calculate its hash value to generate a verification file; In this embodiment of the invention, the code is packaged into a tar.gz format and named with the DEV suffix. The SHA256 hash value of the packaged file is calculated to verify the integrity and consistency of the file, and a corresponding verification file is generated. Specifically, the SHA256 hash value of the compressed file is calculated using the sha256sum command to generate a hash string; the hash string is written into a verification file with the same name as the compressed file; the content of the verification file includes the compressed file name and its corresponding hash value.

[0046] S3. Upload the packaged compressed file and its verification file to the DEV directory of the OSS internal bucket; Use the s3cmd tool to upload the packaged tar.gz file and its checksum file to the DEV directory of the internal bucket in Object Storage Service (OSS). Specify the access configuration for the internal bucket through the .s3cfg-internal configuration file in the user's home directory.

[0047] S4: Download the compressed file from the DEV directory to the test environment via the Jenkins pipeline for decompression and testing; if the test fails, return to S1 and re-execute; if the test passes, execute S5. Step S4 includes: S41. Use the Jenkins pipeline to call the s3cmd tool to download the compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to the specified directory of the test environment. S42. Use the sha256sum command to verify the integrity of the downloaded compressed file. If the verification fails, terminate the process and report an error; if the verification passes, proceed to S43. S43. Use the tar command to decompress the verified compressed file to the deployment directory of the test environment; S44. Modify the configuration file and start the service, then execute the test cases. If the test fails, delete the decompressed files in the test environment and trigger the Jenkins pipeline to re-execute S1 to S3. If the test passes, execute S45. S45. Generate a test pass report and mark this version of the code as eligible for the official release process.

[0048] S5. Download the compressed file from the DEV directory of the OSS internal bucket to the target server, decompress it, remove the specified suffix, repackage it as the official release medium, and recalculate the hash value; Step S5 includes: S51. Use the Jenkins pipeline to call the s3cmd tool to download the compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to the temporary working directory of the target server. S52. Use the sha256sum command to verify the integrity of the downloaded file. If the verification fails, terminate the process and record the error log; if the verification passes, proceed to S53. S53. Use the tar command to extract the compressed file to the target directory, and automatically remove the DEV suffix from the file name and directory name; S54. Use the sed or rename command to modify the DEV identifier contained in all configuration files in the decompressed directory; S55. Repackage the processed code directory into a formal release medium that does not contain the DEV suffix. S56. Use the sha256sum command to recalculate the hash value of the official release media and generate the corresponding verification file; S57. Temporarily store the official release media and its verification files in the release preparation area of ​​the target server.

[0049] After decompressing the downloaded files, remove the 'DEV' suffix from the filenames and directory names, and repackage them into the official release media.

[0050] The downloaded file is usually a compressed archive (such as tar.gz). Use the tar command to extract the downloaded file. The extracted folder name will contain the "DEV" suffix. Rename it to remove this suffix. Then use the tar command to repackage the renamed folder into the final release media.

[0051] S6. Upload the official release media and its verification files to the Release directory of the OSS internal bucket and external bucket, respectively.

[0052] Upload the release media and its verification files to the Release directory of the OSS external bucket for use by project delivery personnel in other departments. Specify the access configuration for the internal bucket through the .s3cfg-internal configuration file in the user's home directory.

[0053] Jenkins pipelines enable automated management, storage, and release of code, ensuring code version consistency and security.

[0054] like Figure 2 As shown in the figure, this embodiment of the invention also provides an automated management and release device for large model algorithm scheduling code, including: Code cloning module: Used to clone large model algorithm scheduling code from GitLab code repository to local server via Jenkins pipeline. During the cloning process, the core.autocrlf parameter is set to enable automatic newline character conversion across operating systems. Packaging and Verification Module: Used to package the cloned code into a compressed file with a specified suffix, and calculate its hash value to generate a verification file; Upload module: Used to upload the packaged compressed file and its verification file to the DEV directory of the OSS internal bucket; The testing and verification module is used to download compressed files from the DEV directory to the test environment via the Jenkins pipeline for decompression and testing. If the test fails, it triggers the code cloning module to be re-executed; if the test passes, it triggers the official release module. The official release module is used to download compressed files from the DEV directory of the OSS internal bucket to the target server, decompress them, remove the specified suffixes, repackage them as official release media, and recalculate the hash value. Release upload module: Used to upload the official release media and its verification files to the Release directory of the OSS internal bucket and external bucket, respectively.

[0055] In some embodiments, the code cloning module includes: Input parameter acquisition unit: used to obtain input parameters from the Jenkins pipeline, including target tag name, Git repository address and branch name; Cloning unit: Used to clone code from a Git repository address to the local server using Git commands. During the cloning process, the core.autocrlf parameter of git clone and git checkout is configured to automatically convert line breaks between operating systems.

[0056] In some embodiments, the code cloning module further includes: Branch switching unit: Used to switch to the branch specified in the input parameters; Tag checking unit: Used to query existing tags in the Git repository and compare the target tag name with the query results; if the target tag already exists, it stops running and reports a warning; if the target tag does not exist, it triggers the tag creation unit; Tag creation unit: Used to dynamically create new Git tags based on the obtained input parameters.

[0057] In some embodiments, the tag creation unit includes: Tag generation subunit: Used to dynamically create new Git tags based on input parameters, and record the timestamp of tag creation and information about the creator; Tag Push Subunit: Used to push newly created tags to the Git repository and write tag information to the log file.

[0058] In some embodiments, the packaging and verification module includes: Packaging unit: Used to package a code directory into a tar.gz compressed file using the tar command; Hash Calculation Unit: Used to calculate the SHA256 hash value of the compressed file using the sha256sum command, and generate a hash string; Verification file generation unit: used to write the hash string into a verification file with the same name as the compressed file; the content of the verification file includes the compressed file name and its corresponding hash value.

[0059] In some embodiments, the test verification module includes: Download Unit: Used to call the s3cmd tool through the Jenkins pipeline to download a compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to a specified directory in the test environment; Integrity verification unit: Used to verify the integrity of the downloaded compressed file using the sha256sum command. If the verification fails, the process is terminated and an error is reported; if the verification passes, the decompression unit is triggered. Decompression unit: Used to decompress verified compressed files to the deployment directory of the test environment using the tar command; Test Execution Unit: Used to modify configuration files and start services, and execute test cases; if the test fails, the decompressed files in the test environment are deleted and the code clone module is re-executed; if the test passes, a test pass report is generated and the version of the code is marked as allowed to enter the official release process.

[0060] In some embodiments, the official release module includes: Download Unit: Used to call the s3cmd tool through the Jenkins pipeline to download a compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to the temporary working directory of the target server. Integrity verification unit: Used to verify the integrity of the downloaded file using the sha256sum command. If the verification fails, the process is terminated and an error log is recorded; if the verification passes, the decompression unit is triggered. Decompression and renaming unit: Used to decompress compressed files to the target directory using the tar command, and automatically remove the DEV suffix from filenames and directory names; Configuration Modification Unit: Used to modify the DEV identifier contained in all configuration files within the unzipped directory using the sed or rename command; Repackaging unit: Used to repackage the processed code directory into a formal release medium that does not contain the DEV suffix; Hash recalculation unit: used to recalculate the hash value of the official release media using the sha256sum command and generate the corresponding verification file; Temporary storage unit: Used to temporarily store the official release media and its verification files in the release preparation area of ​​the target server.

[0061] This invention also provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The communication bus can be used for information transmission between the electronic device and sensors. The processor can invoke logical instructions in memory to execute the following methods: S1. Clone the large model algorithm scheduling code from the GitLab code repository to the local server via the Jenkins pipeline, setting the core.autocrlf parameter during the cloning process to achieve automatic newline character conversion across operating systems; S2. Package the cloned code into a compressed file with a set suffix and calculate its hash value to generate a verification file; S3. Upload the packaged compressed file and its verification file to the DEV directory of the OSS internal bucket; S4. Download the compressed file from the DEV directory to the test environment for decompression and testing via the Jenkins pipeline; if the test fails, return to S1 and re-execute; if the test passes, execute S5; S5. Download the compressed file from the DEV directory of the OSS internal bucket to the target server, decompress it, remove the set suffix, repackage it as the official release medium, and recalculate the hash value; S6. Upload the official release medium and its verification file to the Release directory of the OSS internal bucket and external bucket, respectively.

[0062] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0063] This invention provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the methods provided in the above-described method embodiments. For example, the methods include: S1, cloning large model algorithm scheduling code from a GitLab code repository to a local server via a Jenkins pipeline, setting the `core.autocrlf` parameter during the cloning process to achieve automatic line break conversion across operating systems; S2, packaging the cloned code into a compressed file with a set suffix, and calculating its hash value to generate a verification file; S3, uploading the packaged compressed file and its verification file to the DEV directory of an OSS internal bucket; S4, downloading the compressed file from the DEV directory to a test environment via a Jenkins pipeline for decompression and testing; if the test fails, returning to S1 and re-executing; if the test passes, executing S5; S5, downloading the compressed file from the DEV directory of the OSS internal bucket to the target server, decompressing it, removing the set suffix, repackaging it as a release medium, and recalculating the hash value; S6, uploading the release medium and its verification file to the Release directories of both the OSS internal bucket and external bucket, respectively.

[0064] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large model algorithm scheduling code automatic management and release method, characterized in that, The method comprises the following steps: S1, cloning the large model algorithm scheduling code from the GitLab code repository to the server locally through the Jenkins pipeline, setting the core.autocrlf parameter during the cloning process to realize automatic conversion of line breaks across operating systems; S2, packaging the cloned code into a compressed file with a set suffix, and calculating the hash value to generate a check file; S3, uploading the packaged compressed file and its check file to the DEV directory of the OSS internal bucket; S4, downloading the compressed file from the DEV directory to the test environment through the Jenkins pipeline for decompression and test verification; If the test fails, return to S1 to re-execute; If the test passes, execute S5; S5, download the compressed file from the DEV directory of the OSS internal bucket to the target server, remove the set suffix after decompression, and repackage it as a formal release medium, and recalculate the hash value; S6, upload the formal release medium and its check file to the Release directory of the OSS internal bucket and external bucket respectively.

2. The large model algorithm scheduling code automation management and release method according to claim 1, characterized in that, Step S1 includes: Obtain input parameters from the Jenkins pipeline, including target tag name, Git repository address and branch name, use Git command to clone large model algorithm scheduling code from Git repository address to server local, in the cloning process, through the configuration of git clone and git checkout core.autocrlf parameter to automatic conversion mode, realize the adaptive conversion of line breaks between operating systems.

3. The large model algorithm scheduling code automation management and release method according to claim 2, characterized in that, Step S1 also includes: S1.1, switch to the branch specified in the input parameters, query the existing tags in the Git repository, and compare the target tag name with the query result; if the target tag exists, stop running and report a warning; if the target tag does not exist, create a new Git tag based on the obtained input parameters.

4. The large model algorithm scheduling code automation management and release method according to claim 3, characterized in that, The steps of dynamically creating a new Git tag based on the obtained input parameters include: Dynamically create a new Git tag according to the input parameters, and record the timestamp and creator information of the tag creation; Push the newly created tag to the Git repository, and write the tag information to the log file.

5. The large model algorithm scheduling code automation management and release method according to claim 4, characterized in that, Step S2 includes: Use the tar command to package the code directory into a tar.gz format compressed file; Calculate the SHA256 hash value of the compressed file through the sha256sum command to generate a hash string; Write the hash string to a check file with the same name as the compressed file; the check file content includes the compressed file name and its corresponding hash value.

6. The large model algorithm scheduling code automation management and release method according to claim 5, characterized in that, Step S4 includes: S41, through the Jenkins pipeline, call the s3cmd tool to download the compressed file with the DEV suffix and its corresponding check file from the DEV directory of the OSS internal bucket to the specified directory of the test environment; S42, use the sha256sum command to verify the integrity of the downloaded compressed file, if the verification fails, terminate the process and report an error; if the verification passes, execute S43; S43, decompress the compressed file that passes the verification to the deployment directory of the test environment through the tar command; S44, modify the configuration file and start the service, execute the test case; if the test fails, delete the decompressed file in the test environment, and trigger the Jenkins pipeline to re-execute S1 to S3; if the test passes, execute S45; S45, generate a test pass report, and mark the version code to allow entering the formal release process.

7. The large model algorithm scheduling code automation management and release method according to claim 5, characterized in that, Step S5 includes: S51, call the s3cmd tool through the Jenkins pipeline to download the compressed file with the DEV suffix and its corresponding verification file from the DEV directory of the OSS internal bucket to the temporary working directory of the target server; S52, use the sha256sum command to verify the integrity of the downloaded file, if the verification fails, terminate the process and record the error log; if the verification passes, execute S53; S53, decompress the compressed file to the target directory through the tar command, and automatically remove the DEV suffix in the file name and directory name; S54, use the sed or rename command to modify the DEV identifier contained in all configuration files in the decompressed directory; S55, repackage the processed code directory into a formal release medium without the DEV suffix; S56, use the sha256sum command to recalculate the hash value of the formal release medium, and generate the corresponding verification file; S57, temporarily store the formal release medium and its verification file in the release preparation area of the target server.

8. A large model algorithm scheduling code automatic management and release device, characterized in that, Comprises: Code cloning module: used to clone the large model algorithm scheduling code from the GitLab code repository to the local server through the Jenkins pipeline, and set the core.autocrlf parameter to realize automatic conversion of line breaks across operating systems during cloning; Packaging and verification module: used to package the cloned code into a compressed file with a set suffix, and calculate the hash value to generate a verification file; Upload module: used to upload the packaged compressed file and its verification file to the DEV directory of the OSS internal bucket; Test verification module: used to download the compressed file from the DEV directory to the test environment through the Jenkins pipeline for decompression and test verification; If the test fails, trigger the code cloning module to re-execute; if the test passes, trigger the formal release module; Formal release module: used to download the compressed file from the DEV directory of the OSS internal bucket to the target server, remove the set suffix after decompression, and repackage it into a formal release medium, and recalculate the hash value; Release upload module: used to upload the formal release medium and its verification file to the Release directory of the OSS internal bucket and external bucket, respectively.

9. An electronic device, comprising: The electronic device comprises at least one processor, and a memory connected with the at least one processor in communication; the memory stores computer program instructions executable by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to execute the large model algorithm scheduling code automation management and release method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the large model algorithm scheduling code automation management and release method according to any one of claims 1 to 7.

Images