A PyPI repository system for autonomous architecture and a rapid construction method

By designing a PyPI repository system and rapid build method for autonomous architectures, the problem of the lack of autonomous architecture PyPI repositories was solved, realizing the automated building and maintenance of Python packages, and improving user experience and efficiency.

CN116661858BActive Publication Date: 2026-11-17WUXI ADVANCED TECH RES INST
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310639315.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-01
Publication Date
2026-11-17
Estimated Expiration
2043-06-01

AI Technical Summary

Technical Problem

The lack of a self-built PyPI repository in existing technologies makes it difficult to install and use Python packages, resulting in a poor user experience. Furthermore, existing code migration tools lack specificity and are inefficient.

Method used

Design a PyPI repository system for autonomous architecture, including a database module, a logging module, a scheduled task module, a web management interface module, a dependency analysis module, a Python package scanning module, and a Python package analysis and building module. Through scheduled tasks and automated processes, it scans, analyzes, migrates, and builds Python packages, establishes an indexed knowledge base, and screens out packages that require manual processing.

Benefits of technology

It has achieved automated building and maintenance of PyPI repositories with an autonomous architecture, improved the adaptability of Python packages and user experience, reduced the steps and time of manual intervention, and improved build efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116661858B_ABST
    Figure CN116661858B_ABST
Patent Text Reader

Abstract

This invention discloses a PyPI repository system and rapid construction method for an autonomous architecture. The method includes the following steps: synchronously creating and periodically executing various scheduled tasks, including a database backup scheduled task, a PyPI scanning scheduled task, and a zombie process cleanup scheduled task; scanning the PyPI repository to obtain Python packages and their release versions to be analyzed, including a full PyPI repository scan and a scan after adding a new Python version; analyzing Python packages, migrating code, handling dependency issues, building and packaging, and classifying and marking them; automating the construction of an autonomous architecture PyPI repository and screening out Python packages that require manual analysis; scanning the autonomous architecture software repository and establishing an index knowledge base; and analyzing the dependency status of Python packages.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a PyPI repository system and rapid construction method for autonomous architecture, belonging to the field of computer software technology. Background Technology

[0002] As we all know, chips are the core and soul of computers and other electronic devices; therefore, controlling the chip industry is tantamount to controlling the lifeline of the computer industry. Currently, the open-source software compatible with processors is not perfect, and the software ecosystem is very weak, resulting in low user willingness to use custom architectures.

[0003] For porting individual Python packages, some domestic manufacturers have developed code migration tools to support their own domestically developed processors. These tools can provide some assistance in Python package porting; for example, Huawei's Kunpeng code migration tool, Porting Advisor, can perform dependency analysis on .so dynamic link library files in Python packages. However, for building a complete, self-contained PyPI repository, no one has yet proposed a comprehensive workflow solution. Furthermore, most existing code migration tools focus on analyzing C / C++ languages, without designing specific functions for Python code migration based on the characteristics of Python packages.

[0004] Currently, developers of self-developed software ecosystems often employ a piecemeal approach when dealing with Python packages. When encountering a Python package requiring a compiled binary distribution, they typically rely on experience to locate and port the necessary code blocks. If errors occur during project build, they must resolve each issue individually and attempt repeatedly. Furthermore, developers must analyze other Python packages that the package depends on and determine whether these dependencies need to be ported and compiled. This method demands a certain level of expertise from technical personnel, cannot guarantee the adequacy and accuracy of the porting process, and is cumbersome and inefficient. Finally, due to the current lack of a dedicated PyPI repository for self-developed architectures, users frequently encounter situations where required Python packages cannot be installed or used, resulting in a poor user experience. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a PyPI repository system and rapid construction method for autonomous architectures, thereby solving the problem of poor adaptability of autonomous architecture software to Python packages and resulting in a poor user experience in existing technologies.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A PyPI repository system for autonomous architecture includes a database module, a logging module, a scheduled task module, a web management interface module, a dependency analysis module, a Python package scanning module, and a Python package analysis and building module. The database module is used to store various information data of Python packages, and to implement business logic interfaces for other modules to interact with the database; The logging module is used to receive and record the running logs of each module; The scheduled task module is used to create and execute scheduled scanning tasks for Python packages to be analyzed, scheduled cleanup tasks for zombie processes, and scheduled backup tasks for databases. The web management interface module includes multiple panels, which are used by developers to browse and manage Python packages, PyPI repositories, and the PyPI repository building system online. The dependency analysis module is used to analyze the Python dependency tree of Python packages and the dependencies on software in software repositories; The Python package scanning module is used to incrementally scan for Python packages and their versions that have not yet been analyzed, built, and packaged on the custom architecture, as well as compatible Python versions. The Python package analysis and build module is used to analyze Python package information and automate code migration, dependency handling, and build and package Python packages.

[0007] A method for rapidly building PyPI repositories for autonomous architectures includes the following steps: Synchronously create and periodically execute various scheduled tasks, including database backup scheduled tasks, PyPI scan scheduled tasks, and zombie process cleanup scheduled tasks; Scan the PyPI repository to obtain the Python packages to be analyzed and their release versions. The scan includes a full PyPI repository scan and a scan after adding a new Python version. The system analyzes Python packages, migrates code, handles dependency issues, builds and packages them, and categorizes and tags them. It also builds and screens Python packages that require manual analysis and processing by creating an automated PyPI repository. Scan the self-built software repository and build an indexed knowledge base to analyze the dependencies of Python packages.

[0008] Furthermore, the execution steps of the aforementioned database backup scheduled task include: Create a scheduled task to handle database backups and set the time interval; After the set time interval is reached, perform a database backup.

[0009] Furthermore, the execution steps of the aforementioned PyPI scan timing task include: Create a PyPI scan scheduled task and set the time interval; Determine the status of the Python package scanning module and the Python package analysis and build module to be analyzed: If the Python package scanning module and the Python package analysis and build module to be analyzed are not running, then call the Python package scanning module to be analyzed to perform a PyPI scan; otherwise, after the set time interval, re-determine the status of the Python package scanning module and the Python package analysis and build module to be analyzed.

[0010] Furthermore, the execution steps of the aforementioned zombie process cleanup scheduled task include: Create a scheduled task to clean up zombie processes and set the time interval; After the set time interval is reached, clean up the zombie processes generated in the Python package analysis and build module.

[0011] Furthermore, the aforementioned PyPI full-position scanning steps include: Based on the downloaded PyPI index HTML file, an analysis was performed to obtain a list of all current Python packages in the PyPI repository; Call the API provided by PyPI to get the sequence number of the latest PyPI event, denoted as latest_serial; If this is the first PyPI scan, all Python packages in PyPI will be set as packages to be analyzed. Otherwise, the event sequence number last_serial recorded in the last scan will be used as a parameter to call the API provided by PyPI to obtain Python packages that have been added or updated since the last scan and set them as packages to be analyzed. Save the latest event sequence number (latest_serial) as last_serial for use in the next scan; Iterate through all packets to be analyzed, and use the JSON API provided by PyPI to check if the JSON file exists. If it does not exist, remove the packet to be analyzed from the list of packets to be analyzed. Iterate through all packages to be analyzed, obtain all releases of the package that are compatible with the Python version supported on the autonomous architecture based on its metadata JSON file, and add them to the list of packages to be analyzed.

[0012] Furthermore, the aforementioned scanning steps after adding a new Python version include: Add new supported Python versions to the self-contained architecture; Scan the database to obtain Python packages compatible with the new Python versions and their releases, and generate a list to be analyzed.

[0013] Furthermore, the aforementioned steps for analyzing Python packages, migrating code, handling dependency issues, building and packaging, and classifying and tagging, as well as automating the construction of the autonomous PyPI repository and identifying Python packages that require manual analysis, include: Iterate through the list of packages to be analyzed, and analyze and process each version of the Python package to be analyzed based on the official release package information and the status information in the database. The status information includes: being marked as "source code missing", "no processing required" or "build failed" in the database, having successfully built the custom architecture binary release file, and having other markings / no records in the database. If the status information is "Autonomous architecture binary release file has been successfully built", then create container environments for all compatible but not yet built and packaged autonomous architecture Python versions of this package and version, build binary release files based on the build records of previously successfully built autonomous architecture binary release files, upload the built binary release files to the autonomous architecture PyPI repository, and update the build records and related information of this package and version in the database. If the status information is marked as "source code missing", "no processing required" or "build failed" in the database, then all Python packages and other versions that failed to build will be summarized. If the status information is "Other tags exist in the database / No record", then further analysis and tagging will be performed: If an architecture-independent official binary release exists in PyPI, the status is marked as "no action required" and the relevant information in the database is updated. If there are no official binary release files and official source code release files in PyPI, and the project address on the source code management platform is not provided in the PyPI project information, the status will be marked as "Source code missing", and the relevant information in the database will be updated. Otherwise, create container environments for all compatible but not yet built and packaged custom architecture Python versions of this package and version, attempt to build the binary distribution file from source code, and determine whether the build is successful: If the build is successful and the resulting file is an architecture-related binary release file, the status will be marked as "build successful", the self-architecture binary file will be uploaded to the self-architecture PyPI repository, and the relevant information in the database will be updated. If the build is successful and the resulting file is an architecture-independent binary release file, the status will be marked as "no processing required" and the relevant information in the database will be updated. If the build fails, analyze the reasons for the failure and attempt to handle them automatically.

[0014] Furthermore, the aforementioned steps for analyzing the reasons for the failure and attempting to automatically handle it if the build fails include: If the build fails due to unmet Python package dependencies, then that package and version will be marked as "Python package dependency missing"; If the build fails due to the need for code migration, then modify the current source code with a code modification patch for a similar version of the package that has already undergone code migration. If the build fails due to the missing command, install the package containing the required command from the software repository; If the build fails due to missing header files or dependency libraries, analyze the package to which the missing file belongs, and then install the required package from the software repository. The aforementioned processing results are then rebuilt. If the package and version require special build steps, a build script for a similar version that has already been successfully built is selected for the build; otherwise, the build is attempted directly. If the build is successful and the resulting file is an architecture-related binary release file, the status will be marked as "build successful", the self-architecture binary file will be uploaded to the self-architecture PyPI repository, and the relevant information in the database will be updated. If the build is successful and the resulting file is an architecture-independent binary release file, the status will be marked as "no processing required" and the relevant information in the database will be updated. If the build fails, the status will be marked as "build failed" and the relevant information in the database will be updated.

[0015] Furthermore, the aforementioned steps of scanning the autonomous architecture software repository and building an indexed knowledge base to analyze the dependency status of Python packages include: Obtain the list of software packages in the self-architected software repository, scan each software package, and build an index knowledge base based on the aforementioned scan results. The scan results include executable binary files, scanned dynamic link library files, and header files of dynamic link library files. Iterate through the index knowledge base, check the dynamic library dependencies of each library file and binary file in the specified Python package, and obtain a list of dependent packages; Recursive analysis yields the Python dependency tree of Python packages; Traverse the indexed knowledge base to find the software package to which the specified file belongs.

[0016] The beneficial effects achieved by this invention are as follows: The proposed method for rapid PyPI repository building based on a self-contained architecture can incrementally scan the official PyPI repository, automating the building and release of most Python binary packages, and identifying Python packages that require manual maintenance. It assists developers in Python package building and migration through dependency analysis, error analysis, and automatic porting, thereby establishing a sustainable, self-contained PyPI repository and improving user experience. Attached Figure Description

[0017] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a flowchart of the timed task module in this invention; Figure 3 This is a flowchart illustrating the workflow of the Python package scanning module to be analyzed in this invention. Figure 4 This is a flowchart of the Python package analysis and construction module in this invention; Figure 5 This is a flowchart of the dependency analysis module in this invention; Figure 6 This is the overall execution relationship diagram of the system in this invention. Implementation

[0018] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0019] This embodiment discloses a PyPI repository system for autonomous architectures, such as... Figure 1 As shown, the system includes a database module, a log module, a scheduled task module, a web management interface module, a dependency analysis module, a Python package scanning module, and a Python package analysis and building module. It also incorporates Docker container technology and code migration assistance tools.

[0020] Database module The database module stores various information about Python packages, including: the package's index name, the package's actual name, the package version, compatible Python versions, the package's porting status, the package's source code distribution status, the package's binary distribution status, the Python versions supported by the custom architecture, a JSON file describing the package's information, the package's dependency tree, porting patches for ported packages, and build scripts recording the build steps. This module also implements business logic interfaces for other modules to interact with the database.

[0021] Log module The logging module is used to receive and record the running logs of each module, write the logs of different modules to different files, and perform log rotation, etc.

[0022] Scheduled task module The scheduled task module is used to create and execute scheduled scanning tasks for Python packages to be analyzed, scheduled cleanup tasks for zombie processes, and scheduled backup tasks for databases. Among them, the scheduled scanning task for Python packages to be analyzed can ensure that the self-built PyPI repository can keep up with the updates in the official PyPI repository.

[0023] Web management interface module The web management interface module provides multiple panels for developers to browse and manage Python packages, PyPI repositories, and the PyPI repository building system online: a PyPI browsing panel for browsing and downloading existing Python packages in the self-architected PyPI repository; a Python package management panel for viewing and managing the status of Python packages or uploading / deleting Python distribution packages; a dependency analysis panel for viewing the specific dependencies of Python packages; a porting management panel for managing Python package porting tasks, porting patches, and build scripts; and a log browsing panel for observing the logs of the PyPI semi-automated build workflow system.

[0024] Dependency Analysis Module Figure 5This is a flowchart of the dependency analysis module's workflow. The dependency analysis module analyzes the Python dependency tree of Python packages and their dependencies on software in software repositories. It is divided into software repository scanning and knowledge base building. The Python dependency tree analysis and the dependency analysis on software in software repositories consist of three parts. The first part scans the custom architecture software repository and builds an index knowledge base of the software package and its provided library files, header files, and binary files. The latter two parts analyze the Python dependency tree of the specified Python package and the dependencies on software in the software repository, and are encapsulated in methods for the Python package analysis and build module to call.

[0025] Python package scanning module to be analyzed Figure 3 This is a flowchart of the Python package scanning module. This module incrementally scans for Python packages and their versions, along with their compatible Python versions, that have not yet been analyzed, built, and tested on our custom architecture. The module provides two workflows: a full PyPI repository scan and a scan after a new Python version is added. The full PyPI repository scan scans the entire PyPI repository to obtain the Python packages to be analyzed; this is executed by a scheduled task module to continuously monitor updates to the official PyPI repository. The scan after a new Python version is added scans separately for Python packages that are compatible with newly added Python versions on our custom architecture. This workflow is designed to accommodate the fact that our custom architecture currently does not support all Python versions, but will continue to add support for other Python versions in the future.

[0026] Python package analysis and module building Figure 4This is a flowchart of the Python package analysis and build module. The Python package analysis and build module analyzes Python package information, attempts to automate code migration, dependency handling, and build packaging, and categorizes Python packages based on source code status, build status, and other aspects, identifying those requiring manual analysis by developers. The module first analyzes the release status of the Python package in the PyPI repository to determine if it needs to be built and tested on a custom architecture. Then, it retrieves the Python package source code from the PyPI repository or SCM platform and builds and packages it in a container environment using compatible Python versions for various custom architectures. If the build fails, it attempts to analyze the reason for the failure and automatically performs dependency installation or code migration, then builds again. If a Python package fails to build due to unmet Python dependencies, it will be built again once the dependencies are satisfied. Finally, the status of all analyzed Python packages is recorded in the database. Successfully built Python packages relevant to the machine architecture are published to the domestic PyPI repository, while failed build packages are subsequently handled by developers.

[0027] This embodiment also relates to a method for rapidly building PyPI repositories for autonomous architectures, such as... Figures 2 to 6 As shown, it includes the following steps: Step 1: Create and periodically execute a scheduled task. This step corresponds to the scheduled task module, and the process is as follows: Figure 2 As shown, it includes the following steps: Step 1.1: Create and execute a scheduled database backup task, synchronized with steps 1.2 and 1.3, as follows: Step 1.1.1: Create a scheduled task to handle database backups and set the time interval; Step 1.1.2: Back up the database; Step 1.1.3: After the set time interval is reached, proceed to step 1.1.2; Step 1.2: Create and execute a PyPI scan scheduled task, synchronized with steps 1.1 and 1.3, as follows: Step 1.2.1: Create a PyPI scan scheduled task and set the time interval; Step 1.2.2: If the Python package scanning module and the Python package analysis and construction module to be analyzed are not running, proceed to step 1.2.3; otherwise, proceed directly to step 1.2.4. Step 1.2.3: Execute step 2.1 to call the Python package scanning module to be analyzed for PyPI scanning; Step 1.2.4: After the set time interval is reached, execute step 1.2.2; Step 1.3: Create and execute a scheduled task to clean up zombie processes, synchronized with steps 1.1 and 1.2, as follows: Step 1.3.1: Create a scheduled task to clean up zombie processes and set the time interval; Step 1.3.2: Clean up zombie processes generated in the Python package analysis and build module; Step 1.3.3: After the set time interval is reached, proceed to step 1.3.2; Step 2: Scan the PyPI repository to obtain the Python packages to be analyzed and their release versions. This step corresponds to scanning the modules of the Python packages to be analyzed. The process is as follows: Figure 3 As shown, it includes the following steps: Step 2.1: Perform a full PyPI scan, the process is as follows: Step 2.1.1: Download the PyPI index HTML file from https: / / pypi.org / simple; Step 2.1.2: Analyze the downloaded index HTML file to obtain a list of all Python packages currently in the PyPI repository; Step 2.1.3: Call the API provided by PyPI to obtain the sequence number of the latest PyPI event, denoted as latest_serial; Step 2.1.4: If this is the first PyPI scan, proceed to step 2.1.5; otherwise, proceed to step 2.1.6. Step 2.1.5: Set all Python packages in PyPI as the packages to be analyzed, and then execute step 2.1.7; Step 2.1.6: Using the event sequence number last_serial recorded in the last scan as the parameter, call the API provided by PyPI to obtain the Python packages that have been added or updated since the last scan, and set them as the packages to be analyzed; Step 2.1.7: Save the latest event sequence number (latest_serial) as last_serial for use in the next scan; Step 2.1.8: For each package to be analyzed, download and save the JSON file containing its metadata using the JSON API provided by PyPI. If the JSON file does not exist, it means the package is empty, and remove it from the list of packages to be analyzed. Step 2.1.9: For each package to be analyzed, analyze its metadata JSON file to obtain all release versions of the package. After excluding the already analyzed release versions recorded in the database, obtain all release versions of the package that are compatible with the Python versions already supported on the custom architecture, and add them to the list of packages to be analyzed. Execute Step 2.3; Step 2.2: After adding the new Python version, scan as follows: Step 2.2.1: Developers add newly supported Python versions on the custom architecture. Step 2.2.2: Scan the database to obtain Python packages compatible with the newly added Python versions and their release versions, and generate a list to be analyzed; Step 2.3: Execute step 3, call the Python package analysis and build module to analyze and process the Python packages and their releases from the list to be analyzed in step 2.1.9 or step 2.2.2.

[0028] Step 3: Analyze Python packages, migrate code, handle dependency issues, build and package, and categorize them. This automates the construction of the PyPI repository and identifies Python packages that require manual analysis. This step corresponds to the Python package analysis and build module and includes the following steps: Step 3.1: For each version of all Python packages to be analyzed in the analysis list, execute steps 3.2 to 3.20 respectively; Step 3.2: Based on the official release information for this version of the package and the relevant status information in the database, select the processing steps required. The process is as follows: Step 3.2.1: If the database has already marked it as "Source code missing," "No processing required," or "Build failed," proceed to step 3.21; if the custom architecture binary release file has been successfully built, proceed to step 3.3; if other markings exist in the database or there are no records, proceed to step 3.2.2: Step 3.2.2: If the official binary release file exists in PyPI, proceed to step 3.2.3; otherwise, proceed to step 3.2.4. Step 3.2.3: If the official binary release file architecture is irrelevant (i.e., the name format is *-none-any.whl), proceed to step 3.5; otherwise, proceed to step 3.2.4. Step 3.2.4: If the official source code release file exists in PyPI, proceed to step 3.7; otherwise, proceed to step 3.2.5. Step 3.2.5: If the PyPI project information provides the project address on the source code management platform, proceed to step 3.8; otherwise, proceed to step 3.6. Step 3.3: Create container environments for all compatible but not yet built and packaged Auto-Architecture Python versions of this package and version, and build binary distribution files based on the build records of previously successfully built Auto-Architecture binary distribution files; Step 3.4: Upload the resulting binary release file to the self-built PyPI repository, update the build record and related information for this package and version in the database, and then proceed to step 3.21; Step 3.5: Mark this package and version as "no processing required", update the relevant information in the database, and then proceed to step 3.21; Step 3.6: Mark this package and version as "missing source code", update the relevant information in the database, and then proceed to step 3.21; Step 3.7: Download the official source code release file from PyPI and save it, then proceed to step 3.9; Step 3.8: Download the branch or tag in the project that best matches the release version number from the source code management platform; Step 3.9: Create container environments for all compatible but not yet built and packaged custom architecture Python versions of this package and version, and attempt to build binary distribution files from source code; Step 3.10: If the build in step 3.9 is successful, proceed to step 3.11; otherwise, proceed to step 3.13. Step 3.11: If the resulting file is an architecture-related binary distribution file, proceed to step 3.12; otherwise, proceed to step 3.13. Step 3.12: Mark the package and version as "built successfully", upload the self-architecture binary file to the self-architecture PyPI repository, update the relevant information in the database, and then execute step 3.21; Step 3.13: Mark this package and version as "no processing required", update the relevant information in the database, and then execute step 3.21; Step 3.14: Analyze the reasons for build failures based on the build logs and attempt to handle them automatically. The process is as follows: Step 3.14.1: If the build fails due to unmet Python package dependencies, proceed to step 3.14.2; otherwise, proceed to step 3.14.3. Step 3.14.2: Mark this package and version as "Python package dependency missing", execute step 4.3 to call the dependency analysis module to obtain its Python dependency tree and store it in the database, and execute step 3.21; Step 3.14.3: If the build fails due to the need for code migration, proceed to step 3.14.4; otherwise, proceed to step 3.14.5. Step 3.14.4: If a code modification patch exists for a similar version of the package that has already undergone code migration, try using that patch to modify the current source code; Step 3.14.5: If the build fails due to the missing command, proceed to step 3.14.6; otherwise, proceed to step 3.14.7. Step 3.14.6: Execute step 4.4 to call the dependency analysis module and install the software package to which the required command belongs from the software repository; Step 3.14.7: If the build fails due to missing header files or dependency libraries, proceed to step 3.14.8; otherwise, proceed to step 3.14.9. Step 3.14.8: Call the dependency analysis module, execute step 4.2 to analyze the dependent packages and execute step 4.4 to analyze the package to which the missing file belongs, and then install the required packages from the software repository; Step 3.14.9: If this package and this version may require special build steps, proceed to step 3.14.10; otherwise, proceed to step 3.15. Step 3.14.10: If a build script exists for a similar version of the package that has already been successfully built, select to use that script for the build; Step 3.15: Try building the binary distribution file again; Step 3.16: If the build in step 3.15 is successful, proceed to step 3.17; otherwise, proceed to step 3.20. Step 3.17: If the resulting file is an architecture-related binary release file, proceed to step 3.18; otherwise, proceed to step 3.13. Step 3.18: Mark this package and version as "built successfully", upload the custom architecture binary file to the custom architecture PyPI repository, and update the relevant information in the database; Step 3.19: Store the build dependencies, source code modification patches used in the build, and build scripts in the database, then execute step 2.21; Step 3.20: Mark this package and version as "build failed" and update the relevant information in the database; Step 3.21: If the dependencies of a Python package release marked as "Python package dependency missing" are complete in the self-built PyPI repository, proceed to step 3.22; otherwise, proceed to step 3.23. Step 3.22: Remove the "Python package dependency missing" flag for all Python package releases where the Python package dependencies are satisfied, and then perform steps 3.2 to 3.20 to try building again; Step 3.23: Compile all Python packages that failed to build and their releases to obtain a list that requires manual analysis and processing by R&D personnel; Step 4: Scan the self-built software repository and build an indexed knowledge base. Based on this, analyze the dependencies of Python packages and provide a method for analyzing the packages to which a specified command belongs. This step corresponds to the dependency analysis module and includes the following steps: Step 4.1: Establish an indexed knowledge base. This step forms the basis for steps 4.2 and 4.4, and the process is as follows: Step 4.1.1: Obtain the list of software packages in the self-architected software repository, and execute steps 4.1.2 to 4.1.5 for each software package; Step 4.1.2: Download the software package and extract it; Step 4.1.3: Scan the executable binary files in the decompressed files; Step 4.1.4: Scan the dynamic link library files in the decompressed files; Step 4.1.5: Scan the header files of dynamic link library files; Step 4.1.6: Build an indexed knowledge base based on the scanning results from Steps 4.1.1 to 4.1.5, and the process ends; Step 4.2: Based on Step 4.1, analyze the dependencies of the specified Python package on the software in the software repository. The process is as follows: Step 4.2.1: Receive the name and version of the Python package to be analyzed; Step 4.2.2: Download the x86_64 architecture binary distribution package file of the Python package to be analyzed; Step 4.2.3: Scan the binary distribution package file to obtain all dynamic link library files and executable binary files in the package; Step 4.2.4: Check the dynamic library dependencies of each library file and binary file, and search the index knowledge base to obtain the list of dependent packages; Step 4.2.5: Return the list of dependent software obtained from the analysis and end the analysis; Step 4.3: Analyze the Python dependency tree of the specified Python package, as follows: Step 4.3.1: Receive the name and version of the Python package to be analyzed; Step 4.3.2: Install the necessary Python packages for analysis on the x86_64 platform; Step 4.3.3: Recursively analyze to obtain the Python dependency tree of this Python package; Step 4.3.4: Return the obtained Python dependency tree to end the analysis; Step 4.4: Based on Step 4.1, analyze the software package to which the specified file belongs. The process is as follows: Step 4.4.1: Receive the file name and type to be analyzed; Step 4.4.2: Search the index knowledge base to obtain the software package to which the file belongs; Step 4.4.3: Return the name of the software package obtained from the analysis and end the analysis.

[0029] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A PyPI repository system for autonomous architectures, characterized in that, It includes a database module, a logging module, a scheduled task module, a web management interface module, a dependency analysis module, a Python package scanning module, and a Python package analysis and building module; The database module is used to store various information data of Python software packages, and to implement business logic interfaces for other modules to interact with the database; The log module is used to receive and record the operation logs of each module; The scheduled task module is used to create and execute scheduled scanning tasks for Python packages to be analyzed, scheduled cleanup tasks for zombie processes, and scheduled backup tasks for the database. The Web management interface module includes multiple panels, which are used by R&D personnel to browse and manage Python packages, PyPI repositories, and the PyPI repository building system online. The dependency analysis module is used to analyze the Python dependency tree of Python packages and the dependencies on software in software repositories; The Python package scanning module is used to incrementally scan for Python packages and their versions that have not yet been analyzed, built, and packaged on the proprietary architecture, as well as compatible Python versions. The Python package analysis and build module is used to analyze Python package information and perform automated code migration, dependency handling, and build and package Python packages. The steps involved in automating and identifying Python packages requiring manual analysis in a PyPI repository include: analyzing packages, migrating code, handling dependency issues, building and packaging, and categorizing them. The analysis process iterates through the list of packages to be analyzed, and analyzes and processes each version of the Python package to be analyzed based on the official release status of each version and the status information in the database. The status information includes: being marked as "source code missing", "no processing required" or "build failed" in the database, having successfully built a custom architecture binary release file, and having other markings / no records in the database. If the status information is "Autonomous architecture binary release file has been successfully built", then create container environments for all compatible but not yet built and packaged autonomous architecture Python versions of this package and version, build binary release files based on the build records of previously successfully built autonomous architecture binary release files, upload the built binary release files to the autonomous architecture PyPI repository, and update the build records and related information of this package and version in the database. If the status information is marked as "source code missing", "no processing required" or "build failed" in the database, then all Python packages and other versions that failed to build will be summarized. If the status information is "Other tags exist in the database / No record", then further analysis and tagging will be performed: If an architecture-independent official binary release exists in PyPI, the status is marked as "no action required" and the relevant information in the database is updated. If there are no official binary release files or official source code release files in PyPI, and the project address on the source code management platform is not provided in the PyPI project information, the status will be marked as "Source code missing", and the relevant information in the database will be updated. Otherwise, create container environments for all compatible but not yet built and packaged custom architecture Python versions of this package and version, attempt to build the binary distribution file from source code, and determine whether the build is successful: If the build is successful and the resulting file is an architecture-related binary release file, the status will be marked as "build successful", the self-architecture binary file will be uploaded to the self-architecture PyPI repository, and the relevant information in the database will be updated. If the build is successful and the resulting file is an architecture-independent binary release file, the status will be marked as "no processing required" and the relevant information in the database will be updated. If the build fails, analyze the reasons for the failure and attempt to handle them automatically.

2. A method for rapidly building PyPI repositories for autonomous architectures, characterized in that, Includes the following steps: Synchronously create and periodically execute various scheduled tasks, including database backup scheduled tasks, PyPI scan scheduled tasks, and zombie process cleanup scheduled tasks; Scan the PyPI repository to obtain the Python packages to be analyzed and their release versions. The scan includes a full PyPI repository scan and a scan after adding a new Python version. Based on the analysis of Python packages, code migration, dependency handling, build and package, and classification and tagging, the automated and autonomous architecture of the PyPI repository builds and screens out Python packages that require manual analysis and processing; Scan the self-built software repository and build an indexed knowledge base to analyze the dependencies of Python packages; The steps involved in automating and identifying Python packages requiring manual analysis in a PyPI repository include: analyzing packages, migrating code, handling dependency issues, building and packaging, and categorizing them. The analysis process iterates through the list of packages to be analyzed, and analyzes and processes each version of the Python package to be analyzed based on the official release status of each version and the status information in the database. The status information includes: being marked as "source code missing", "no processing required" or "build failed" in the database, having successfully built a custom architecture binary release file, and having other markings / no records in the database. If the status information is "Autonomous architecture binary release file has been successfully built", then create container environments for all compatible but not yet built and packaged autonomous architecture Python versions of this package and version, build binary release files based on the build records of previously successfully built autonomous architecture binary release files, upload the built binary release files to the autonomous architecture PyPI repository, and update the build records and related information of this package and version in the database. If the status information is marked as "source code missing", "no processing required" or "build failed" in the database, then all Python packages and other versions that failed to build will be summarized. If the status information is "Other tags exist in the database / No record", then further analysis and tagging will be performed: If an architecture-independent official binary release exists in PyPI, the status is marked as "no action required" and the relevant information in the database is updated. If there are no official binary release files or official source code release files in PyPI, and the project address on the source code management platform is not provided in the PyPI project information, the status will be marked as "Source code missing", and the relevant information in the database will be updated. Otherwise, create container environments for all compatible but not yet built and packaged custom architecture Python versions of this package and version, attempt to build the binary distribution file from source code, and determine whether the build is successful: If the build is successful and the resulting file is an architecture-related binary release file, the status will be marked as "build successful", the self-architecture binary file will be uploaded to the self-architecture PyPI repository, and the relevant information in the database will be updated. If the build is successful and the resulting file is an architecture-independent binary release file, the status will be marked as "no processing required" and the relevant information in the database will be updated. If the build fails, analyze the reasons for the failure and attempt to handle them automatically.

3. The method for rapid construction of PyPI repositories for autonomous architectures according to claim 2, characterized in that, The execution steps of the database backup scheduled task include: Create a scheduled task to handle database backups and set the time interval; Once the set time interval is reached, perform a database backup.

4. The method for rapid construction of PyPI repositories for autonomous architectures according to claim 2, characterized in that, The execution steps of the PyPI scan timing task include: Create a PyPI scan scheduled task and set the time interval; Determine the status of the Python package scanning module and the Python package analysis and build module to be analyzed: If the Python package scanning module and the Python package analysis and build module to be analyzed are not running, then call the Python package scanning module to be analyzed to perform a PyPI scan; otherwise, after the set time interval is reached, re-determine the status of the Python package scanning module and the Python package analysis and build module to be analyzed.

5. The method for rapid construction of PyPI repositories for autonomous architectures according to claim 2, characterized in that, The execution steps of the scheduled task for cleaning up zombie processes include: Create a scheduled task to clean up zombie processes and set the time interval; After the set time interval is reached, clean up the zombie processes generated in the Python package analysis and build module.

6. The method for rapid construction of PyPI repositories for autonomous architectures according to claim 2, characterized in that, The PyPI full inventory scan steps include: Based on the downloaded PyPI index HTML file, an analysis was performed to obtain a list of all current Python packages in the PyPI repository; Call the API provided by PyPI to get the sequence number of the latest PyPI event, denoted as latest_serial; If this is the first PyPI scan, all Python packages in PyPI will be set as packages to be analyzed. Otherwise, the event sequence number last_serial recorded in the last scan will be used as a parameter to call the API provided by PyPI to obtain Python packages that have been added or updated since the last scan and set them as packages to be analyzed. Save the latest event sequence number (latest_serial) as last_serial for use in the next scan; Iterate through all packets to be analyzed, and use the JSON API provided by PyPI to check if the JSON file exists. If it does not exist, remove the packet to be analyzed from the list of packets to be analyzed. Iterate through all packages to be analyzed, obtain all releases of the package that are compatible with the Python version supported on the autonomous architecture based on its metadata JSON file, and add them to the list of packages to be analyzed.

7. The method for rapid construction of PyPI repositories for autonomous architectures according to claim 2, characterized in that, The scanning steps after adding a new Python version include: Add new supported Python versions to the self-contained architecture; Scan the database to obtain Python packages compatible with the new Python versions and their releases, and generate a list to be analyzed.

8. The method for rapid construction of PyPI repositories for autonomous architectures according to claim 2, characterized in that, If the build fails, the steps to analyze the reasons for the failure and attempt to handle it automatically include: If the build fails due to unmet Python package dependencies, then that package and version will be marked as "Python package dependency missing"; If the build fails due to the need for code migration, then modify the current source code with a code modification patch for a similar version of the package that has already undergone code migration. If the build fails due to the missing command, install the package containing the required command from the software repository; If the build fails due to missing header files or dependency libraries, analyze the package to which the missing file belongs, and then install the required package from the software repository. The aforementioned processing results are then rebuilt. If the package and version require special build steps, a build script for a similar version that has already been successfully built is selected for the build; otherwise, the build is attempted directly. If the build is successful and the resulting file is an architecture-related binary release file, the status will be marked as "build successful", the self-architecture binary file will be uploaded to the self-architecture PyPI repository, and the relevant information in the database will be updated. If the build is successful and the resulting file is an architecture-independent binary release file, the status will be marked as "no processing required" and the relevant information in the database will be updated. If the build fails, the status will be marked as "build failed" and the relevant information in the database will be updated.

9. The method for rapid construction of PyPI repositories for autonomous architectures according to claim 2, characterized in that, The steps of scanning the autonomous architecture software repository and building an indexed knowledge base to analyze the dependencies of Python packages include: Obtain a list of software packages in the self-architected software repository, scan each software package, and build an index knowledge base based on the aforementioned scan results. The scan results include executable binary files, scanned dynamic link library files, and header files of dynamic link library files. Iterate through the index knowledge base, check the dynamic library dependencies of each library file and binary file in the specified Python package, and obtain a list of dependent packages; Recursive analysis yields the Python dependency tree of Python packages; Traverse the indexed knowledge base to find the software package to which the specified file belongs.

Citation Information

Patent Citations

  • Medicine research and development software warehouse and software package management system thereof

    CN111221562A

  • Drug research and development software repository and software package management system

    US20210200536A1