A method and system for blocking defective components of a software virtual warehouse

Abstract

The application discloses a kind of software virtual warehouse's defective component blocking method and system, the method includes the following steps: obtaining project feature file, according to the project feature file generates the name and version of required open source component;According to the name and version of required open source component, component request is sent to central warehouse or virtual library, and the response request of central warehouse is obtained;Strategy library establishes defective open source component name, version and flow path list, and whether the open source component name, version and flow path of strategy library are hit are inquired to strategy library by the component request / response request;If open source component name, version and flow path are hit in strategy library, then generate blocking information to block and the interaction of private service warehouse, whether defective open source component is judged by strategy library through identifying the flow path of open source component, further execute the flow blocking operation of defective open source component, compared with traditional firewall technology, cost can be saved and efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of component management in the field of Internet software, and in particular to a method and system for blocking defective components in a software virtual warehouse. Background Art

[0002] Currently, software virtual repositories include Jfrog Artifactory and Sonatype Nexus private server repositories. These existing private server repositories can implement management and control from central warehouses to private server repositories. However, these existing technologies have the following technical drawbacks: private server repositories primarily block defective open source components through firewalls, which results in high blocking costs; private server repositories only block and control defective open source components between private server repositories and central warehouses, making the blocking and control scope incomplete and prone to omissions; and private server repositories are plug-in-based and require adaptation to different versions of each type of private server, making the configuration process complex and difficult. Summary of the Invention

[0003] One of the inventive purposes of the present invention is to provide a method and system for blocking defective components in a software virtual warehouse. The method and system block defective open source components by building a policy library. The policy library determines whether it is a defective open source component by identifying the traffic path of the open source component, and further performs traffic blocking operations on the defective open source component. Compared with traditional firewall technology, it can save costs and improve efficiency.

[0004] Another object of the present invention is to provide a method and system for blocking defective components in a software virtual warehouse. The method and system request a replacement URL through asynchronous request and parse the URL while requesting, which can improve the processing efficiency of the system.

[0005] Another object of the present invention is to provide a method and system for blocking defective components in a software virtual warehouse. The method and system include a timeout control strategy, setting a response time threshold in the package manager. If the threshold is exceeded, the request is resent to the virtual library, and the success of the asynchronous request is monitored in real time, which can improve the system's effective blocking and control of defective open source components.

[0006] In order to achieve at least one of the above-mentioned objects, the present invention further provides a method for blocking defective components in a software virtual warehouse, the method comprising the following steps:

[0007] Obtain a project signature file, and generate the required open source component name and version based on the project signature file;

[0008] Sending a component request to the central repository or virtual library according to the required open source component name and version;

[0009] The policy library creates a list of defective open source component traffic paths, and after requesting the component, queries the policy library to see if the open source component traffic path in the policy library is hit;

[0010] If the open source component traffic path is hit in the policy library, blocking information is generated to block the interaction with the central warehouse.

[0011] According to one of the preferred embodiments of the present invention, a traffic blocking switch is configured in the policy library. When the traffic blocking switch is closed, the component requests direct access to the central warehouse and obtains the corresponding open source component from the central warehouse. If the traffic blocking switch is turned on, the open source component path information in the request is compared with the policy library. If there is a hit path information, the acquisition of the open source component corresponding to the request is blocked.

[0012] According to another preferred embodiment of the present invention, the feature file includes at least one of a requirements.txt file, a pom.xml file, a composer.json file and a package.json file, wherein the feature file records the open source component name and version information, and the package manager imports the open source component name and version information recorded in the feature file into the private server warehouse.

[0013] According to another preferred embodiment of the present invention, the blocking method includes building a web monitoring service, setting the http routing *path for wildcard matching, and monitoring all get requests sent to the routing port.

[0014] According to another preferred embodiment of the present invention, the blocking method also includes: the blocking method also includes: the web monitoring service obtains the traffic data of the routing port, records the URL address of the get request, replaces the URL address with the source address of the mirror, and determines whether the URL address replaced by the get request is the source code URL.

[0015] According to another preferred embodiment of the present invention, the blocking method includes: obtaining the URL address of the get request, and determining the format of the URL address of the get request, and determining whether it is a system URL or a source code URL based on the format of the URL address.

[0016] According to another preferred embodiment of the present invention, the blocking method further includes an asynchronous request to the policy library, and the specific steps include:

[0017] Create content receiver, exit signal channel and cancel context;

[0018] Start a goroutine asynchronous request to obtain the replaced URL address;

[0019] The replaced URL address is parsed, and corresponding path information is searched in a policy library according to the parsed URL address.

[0020] According to another preferred embodiment of the present invention, the blocking method includes timeout control of the request, specifically comprising the following steps:

[0021] Use the select function in Go to monitor the exit signal channel and the time channel with the threshold set for asynchronous requests.

[0022] If the exit signal channel is monitored and the content of the exit signal channel is determined to be successful, the virtual library executes the transmission of the body data to the package manager;

[0023] If the time channel carrying the threshold is monitored, redirection request data is generated and the current asynchronous request is closed.

[0024] In order to achieve at least one of the above-mentioned objects, the present invention further provides a defective component blocking system for a software virtual warehouse, wherein the system executes the defective component blocking method for a software virtual warehouse.

[0025] The present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor to describe a method for blocking defective components in a software virtual warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Fig. 1 Shown is a flow chart of a defective component blocking method for a software virtual warehouse according to the present invention.

[0027] Fig. 2 Shown is a schematic diagram of a preferred embodiment of a defective component blocking system for a software virtual warehouse according to the present invention.

[0028] Fig. 3 Shown is a schematic diagram of another preferred embodiment of a defective component blocking system for a software virtual warehouse according to the present invention. DETAILED DESCRIPTION

[0029] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0030] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the elements may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0031] Please refer to Figs. 1-3 The present invention discloses a method and system for blocking defective components in a software virtual warehouse, wherein the system includes two architectures: one is an architecture that does not include a build tool, in which the private service warehouse directly interacts with the virtual warehouse gateway to generate component requests and responds to the component requests. The other is an architecture that includes a build tool, wherein the architecture that includes a build tool generates component requests and obtains responses to the component requests by adding a build tool. The basic principles of the above two architectures are the same, and the present invention only further explains the second architecture.

[0032] Specifically, there are two implementation methods in the second architecture, one is an implementation method that includes a virtual library, and the other is an implementation method that does not include a virtual library. The present invention takes Python's pip package manager as an example. The implementation method that does not include a virtual library mainly includes the following steps: obtaining a feature file in the project through the package manager, wherein the feature file in the project includes at least one of a requirements.txt file, a pom.xml file, a composer.json file, and a package.json file, wherein the feature file records the name and version information of the open source components needed in the software project. When the architecture does not include the virtual library, the package manager will directly extract the recorded open source components and versions from the central warehouse according to the feature file, and send the extracted open source components and versions to the corresponding private server warehouse.

[0033] In a preferred embodiment of the present invention, the present invention further provides a virtual library and blocking strategy for blocking defective open source components from entering the project, specifically comprising the following steps: first, the project itself needs to apply for a virtual library address of a project on the SCA management platform, establish a mapping relationship between the virtual library address and the corresponding project, so that each project has a corresponding virtual library, and further construct a virtual library assembly strategy library and a blocking switch, wherein the strategy library is used to determine whether the requested open source component is a defective open source component, and when the blocking switch is turned on, the traffic of the defective open source component is blocked to avoid importing the defective open source component into the private server warehouse. When the blocking switch is turned off, the judgment of the defective open source component is no longer performed, and the open source component import operation from the central warehouse to the private server warehouse is directly executed.

[0034] It should be noted that the path information of defective open source components is stored in the policy library. When the package manager generates a component request for the project, the virtual library will obtain the open source component path information in the component request, and further determine whether the open source component path information in the component request is hit in the policy library. If it is hit, it means that there is a defective open source component, and a blocking instruction is further generated to prohibit the open source component in the central warehouse from being imported into the corresponding private server warehouse.

[0035] It is worth mentioning that the present invention adopts the preferred go language for programming, and the specific method is as follows: using including but not limited to the gin framework to establish a web monitoring service, wherein the http / https routing of the monitoring server is set to *path for wildcard matching, and monitors all get requests sent to the routing port. Furthermore, the monitoring service obtains http traffic information and records the URL address in the http, wherein since the URL address may be an external network address, the present invention needs to replace the external network URL address with a domestic mirror source address, and further determine whether it is a source code URL based on the format of the replaced mirror source address. For example: the web service monitors the traffic URL as:

[0036] http: / / 192.168.20.12:2222 / com / alibaba / fastjson / 1.1.30 / fastjson-1.1.30.jar, where 192.168.20.12 is the host IP address of the virtual library, and 2222 is the port where the virtual library sets the web service to listen. After replacement, it becomes:

[0037] https: / / repo.maven.apache.org / maven2 / com / alibaba / fastjson / 1.1.30 / fastjson-1.1.30.jar, and further determine whether the replaced URL is the source code URL.

[0038] It should be noted that since the URL in the request may be the system's own URL, for example, the system's own URL may correspond to formats including but not limited to JSON or .POM, the source code URL can be determined by removing the system's own URL. After obtaining and saving the source code URL, an asynchronous request is further established. The asynchronous request establishment method includes: creating a content receiver, exiting the signal channel and canceling the context, then launching a goroutine to asynchronously request the replaced source code URL address. While the goroutine executes the asynchronous request, the URL address is parsed, and the corresponding open source component name, version information, and path information are extracted from the URL address. The extracted open source component name, version information, and path information are further searched in the policy library. If the same open source component name, version information, and path information are found, the open source component is determined to be a defective component, and the defective component is prevented from being imported into the private server repository. If the open source component name, version information, and path information are not found after searching the policy library, the open source component is determined to be a normal component, and the package manager executes to import the normal component into the normal private server repository.

[0039] Since the component request may time out due to various faults, the present invention further provides a method for controlling the timeout of component requests, comprising the following steps:

[0040] The select of the Go language is used to monitor two channels, one of which is the exit signal channel set during the asynchronous request, and the other is the timer channel carrying the time threshold. When the exit signal channel is monitored, it is necessary to judge the content of the exit signal channel monitored. If the content of the exit signal channel is nil, it means that the asynchronous request is successful. When the asynchronous request is successful, the virtual library records a write. It should be noted that the write is the body data of the asynchronous request sent directly to the package manager. When the content of the exit signal channel monitored is not nil, it means that the asynchronous request failed and a failure message is returned. In order to enable the project to be built normally when the asynchronous request of the component does not hit any blocking strategy in the policy library, the present invention further monitors the timer channel carrying the time threshold and sets a redirect (redirection information). When the select monitors the timer channel, it means that the asynchronous request has exceeded the set time threshold and the asynchronous request has not ended. The reasons why the asynchronous request has not ended include but are not limited to: the request packet is too large or the network is delayed. The virtual library sends a redirection message to the package manager, and the package manager sends a request to other private service warehouses. The other private service warehouses generate component requests according to the request of the package manager and send them to the virtual library.

[0041] It should be noted that when the obtained URL is determined to be a source code URL, the virtual library performs an import operation to the private server warehouse. If the obtained URL is determined to be a non-source code URL, the aforementioned write / redirect operation is performed in the virtual library. That is, when the URL detected by the virtual library is a source code URL, the virtual library records a write and directly sends the body data of the asynchronous request to the package manager. Otherwise, a 502 error message is returned, interrupting the project build. Alternatively, regardless of whether the obtained URL is a source code URL, when it is detected that the asynchronous request exceeds the time threshold, the virtual library sends a redirect message to the package manager, and the package manager sends a request to other private server warehouses.

[0042] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wire segments, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical fiber cable, RF, etc., or any suitable combination thereof.

[0043] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0044] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be subject to any deformation or modification.

Claims

1. A method for blocking defective components in a software virtual warehouse, characterized in that: The method comprises the following steps: Obtain a project signature file, and generate the required open source component name and version based on the project signature file; Sending a component request to the central repository or virtual library according to the required open source component name and version; The policy library creates a list of defective open source component names, versions, and traffic paths, and queries the policy library for the component request to see if it matches the open source component name, version, and traffic path in the policy library. If the open source component name, version, and traffic path are matched in the policy library, blocking information is generated to block interaction with the private server repository. A traffic blocking switch is configured in the policy library. When the traffic blocking switch is off, the component requests direct access to the central warehouse and obtains the corresponding open source component from the central warehouse. If the traffic blocking switch is on, the open source component path information in the request is compared with the policy library. If there is a matching path information, obtaining the corresponding open source component is blocked. Block defective open source components from entering the project. The specific steps include: The project applies for a virtual library address on the SCA management platform and establishes a mapping relationship between the virtual library address and the corresponding project, so that each project has a corresponding virtual library; Construct a component policy library and a blocking switch of the virtual library, wherein the policy library is used to determine whether the requested open source component is a defective open source component, and when the blocking switch is turned on, the traffic of the defective open source component is blocked to avoid importing the defective open source component into the private server warehouse. When the blocking switch is turned off, the judgment of the defective open source component is no longer performed, and the open source component import operation from the central warehouse to the private server warehouse is directly executed.

2. A method for blocking defective components in a software virtual warehouse according to claim 1, characterized in that: The feature file includes at least one of a requirements.txt file, a pom.xml file, a composer.json file, and a package.json file, wherein the feature file records the name and version information of the open source component, and the package manager imports the open source component name and version information recorded in the feature file into the private server warehouse.

3. A method for blocking defective components in a software virtual warehouse according to claim 1, characterized in that: The blocking method includes building a web monitoring service, setting the http routing *path for wildcard matching, and monitoring all get requests sent to the routing port.

4. A method for blocking defective components in a software virtual warehouse according to claim 3, characterized in that: The blocking method also includes: the web monitoring service obtains traffic data of the routing port, records the URL address of the get request, replaces the URL address with the source address of the mirror, and determines whether the URL address replaced by the get request is the source code URL.

5. A method for blocking defective components in a software virtual warehouse according to claim 4, characterized in that: The blocking method includes: obtaining the URL address of the get request, determining the format of the URL address of the get request, and determining whether it is a system URL or a source code URL according to the format of the URL address.

6. A method for blocking defective components in a software virtual warehouse according to claim 4, characterized in that: The blocking method further includes an asynchronous request to the policy library, and the specific steps include: Create content receiver, exit signal channel and cancel context; Start goroutine asynchronous request to obtain the replaced URL address; The replaced URL address is parsed, and corresponding path information is searched in the policy library according to the parsed URL address.

7. The method for blocking defective components in a software virtual warehouse according to claim 1, characterized in that: The blocking method includes request timeout control, specifically comprising the following steps: Use the select function in Go to monitor the exit signal channel and the time channel with the threshold set for asynchronous requests. If the exit signal channel is monitored and the content of the exit signal channel is determined to be successful, the virtual library executes the transmission of the body data to the package manager; If the time channel carrying the threshold is monitored, redirection request data is generated and the current asynchronous request is closed.

8. A defective component blocking system for a software virtual warehouse, characterized in that: The system executes the defective component blocking method of a software virtual warehouse as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that The computer readable storage medium stores a computer program. The computer program can be executed by a processor to implement a defective component blocking method for a software virtual warehouse as described in any one of claims 1 to 7.

Images