Method for dynamic indexing and storage of internet open source software data

Abstract

The application discloses a kind of internet open source software data dynamic indexing and storage method, it is related to open source software data field, data item is created to internet open source software data, each data item is composed of one or more data units, and add the following data description: data source HTTP link, time limit information and identity authentication information;Carry out the initialization operation of data item;When receiving data item download request, verify the accessibility of data source and estimate data size, then the downloaded data is sent to requester, and data state is marked;Whether to expire is determined according to time limit information, and for expired, re-accessibility verification and estimate data size are carried out, and the downloaded data is sent to requester, and data state is updated;Data state of data item is updated according to the counting result in request counting period.The application realizes the quick on-demand acquisition of internet open source software data and the requirement of less local storage space, improves storage utilization.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of open source software data, and particularly relates to a method for dynamic indexing and storage of Internet open source software data. BACKGROUND

[0002] The Internet open source software data involved in the present application is various data generated by related support tools in the process of open source software development and operation, such as version control data, defect tracking data, etc. These data can be used to carry out many software engineering related researches, help to provide the efficiency of software development and operation, and the quality of software products.

[0003] At present, the number of open source software projects on the Internet is showing explosive growth, and Internet open source software data is being generated and accumulated. As can be seen from the research literature of software engineering, the demand for using Internet open source software data is very diverse and is constantly changing. Different data users pay attention to different open source projects and data types, and even more specific data ranges. The cost of accessing and downloading a large amount of data should be minimized. These are the challenges faced by data services.

[0004] At present, the data set sharing platform represented by zenodo (https: / / zenodo.org / ) contains some open source software data sets. However, they are all some small range of static data sets uploaded by users. They do not give a solution to the diversity, mass, and continuous update of the whole network data, and cannot provide comprehensive and efficient on-demand data services. SUMMARY

[0005] The purpose of the present application is to provide a method for dynamic indexing and storage of Internet open source software data, to realize fast on-demand access and less local storage space requirement of Internet open source software data, and to improve the storage utilization.

[0006] The technical solution adopted by the present application to achieve the above purpose is as follows:

[0007] A method for dynamic indexing and storage of Internet open source software data, comprising the following steps:

[0008] A data item is created for the Internet open source software data, each data item is composed of one or more data units, and the following data descriptions are added: data source HTTP link, time limit information and identity authentication information, wherein the data source is the download source of the data unit in the data item, the time limit information is the valid period after the data is downloaded and stored, and the identity authentication information is the username and password required when accessing the data unit, or a string (Token) representing a valid identity. All data units in a data item share the same authentication information, and if not required, it is empty;

[0009] performing initialization operation of the data item: performing data source accessibility verification, if the data source is accessible, downloading data from the data source based on the data source HTTP link, estimating data size, and marking data state according to the data source accessibility verification result;

[0010] When receiving a data item download request, performing data source accessibility verification, if the data source is accessible, downloading data from the data source based on the data source HTTP link, estimating data size, and sending the downloaded data to the requester, marking data state according to the data source accessibility verification result; checking data item time limit information according to the data state of the data item, if the download time has exceeded the time limit information, updating the data state, deleting the downloaded data in the local, and performing data source accessibility verification again, if the data source is accessible, downloading data from the data source based on the data source HTTP link, estimating data size, and sending the downloaded data to the requester, and updating the data state of the data item;

[0011] updating the data state of the data item according to the counting result in the request counting period.

[0012] Further, the method for performing data source accessibility verification is as follows: randomly selecting a certain number of data units, initiating HTTP request according to the data unit download address, the HTTP request carrying necessary authentication information, if the access to any one of the data units obtains a response with a status code of 200, the data source is considered to be accessible; if the access to any one of the data units does not obtain a response with a status code of 200, the data source is considered to be inaccessible.

[0013] Further, the method for estimating data size is as follows: randomly selecting a certain proportion of data units, downloading the selected data units according to the data unit download address, obtaining the size of the data units, and calculating the average size; then multiplying the average size by the total number of data units to obtain the estimated data size.

[0014] Further, when creating the data item, a data unit list obtaining script is added; in subsequent data item initialization and download request, if the data source is accessible, the data unit list obtaining script is executed to obtain the data unit list, and then data is downloaded from the data source.

[0015] Further, the data unit list obtaining script includes two types of explicit type and implicit type, wherein the explicit type is used for the case where the data source provides the data unit list, and the script obtains the download address of each data unit; the implicit type is used for the case where the data source does not provide the data unit list, and the script explores the data unit number in an increasing order from small to large to generate the data unit download address.

[0016] Further, the data state of the data item includes indexed, cached, stored, lost and expired, wherein indexed means that the data accessibility is verified and the data size is estimated, but the data is not downloaded yet; cached means that the data is downloaded and temporarily stored locally; stored means that the data is downloaded and stored locally for a long time; lost means that the data is not accessible; and expired means that the data cannot meet the time requirement.

[0017] Further, if the data source is accessible, the data state is set as indexed when estimating the data size, and if the data source is not accessible, the data state is set as lost.

[0018] Further, when receiving a data item download request, the data source accessibility is verified, if the data source is not accessible, the data state is changed to lost; and if the data source is accessible, the data state is updated to stored.

[0019] Further, the data time information is checked according to the data state of the data item, which means that if the data state of the data item is stored, the data validity period is checked, and when the validity period is exceeded, the data state is changed to expired.

[0020] Further, when re-verifying the data source accessibility, if the data source is accessible, the data state is restored to indexed, and then the data is downloaded from the data source, and the data state is updated to stored.

[0021] Further, the method for updating the data state according to the request counting period is that a counting period is started from the first request, wherein one counting period includes five time equal stages, when the counting period is ended, it is judged that if the counts of three stages are all greater than 0, the state is updated to stored, and a new counting period is started; otherwise, the current data state is updated to a new data state, that is, if the current data state is stored, it is updated to cached, if the current data state is cached, it is updated to indexed, and the locally stored data is deleted.

[0022] The beneficial effects of the present application are that: by setting the data state, such as five data states of indexed, cached, stored, lost and expired, and controlling the data state transition, the complete index of data, data size estimation, on-demand download, on-demand storage and timely update are realized. The complete index of data and data size estimation enable the user to master the overall profile of large-scale open source software data on the Internet; on-demand download and storage reduce the storage occupation while providing the ability to quickly respond to the common data acquisition demand; and the timely update ensures the timeliness of the acquired data. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1This is a flowchart of a method for dynamic indexing and storage of open-source software data on the Internet according to the present invention. Detailed Implementation

[0024] This invention uses a dynamic method to add data sources and promptly identifies lost or updated data sources. At the same time, it uses a demand-driven strategy to efficiently download and save data.

[0025] A data item is the basic object of data management, consisting of one or more data units. Each data unit has its own access address. A data item's description includes the data source HTTP link, validity information, and authentication information, along with a script to retrieve the data unit list. The data source is the download origin of the data units within the data item. The validity information is a time span, indicating the validity period after the data is downloaded and stored. The authentication information is the username and password that may be needed to access the data unit, or a string (token) representing valid identity. All data units within a data item share the same authentication information; if not needed, it is empty. There are two types of scripts for retrieving the data unit list: explicit and implicit. Explicit scripts are used when the data source provides a list of data units, and the script retrieves the access address for each data unit. Implicit scripts are used when the data source does not provide a list of data units; the script explores the data unit numbers incrementally from smallest to largest and generates download addresses for each data unit. A data item has five states: "indexed," "cached," "stored," "lost," and "expired." Among them, "indexed" means that the accessibility of the data has been verified and the data size has been estimated, but the data has not yet been downloaded; "cached" means that the data has been downloaded and temporarily stored locally; "stored" means that the data has been downloaded and stored locally for a long time; "lost" means that the data is no longer accessible; "expired" means that the data no longer meets the timeliness requirements.

[0026] When creating a data item, the data item is initialized. Specifically, this involves verifying the accessibility of the data source, obtaining the list of data cells, and estimating the data size. If the data source is accessible, the data cell list retrieval script is executed to obtain the list of data cells, the data status is set to "indexed", and the data size is estimated. If the data source is inaccessible, the data status is set to "missing".

[0027] Data source accessibility verification involves randomly selecting a certain number of data units and initiating HTTP requests according to their access addresses. The HTTP requests contain the necessary authentication information. If an access to any data unit receives a response with a status code of 200, the data source is considered accessible. If no access to any data unit receives a response with a status code of 200, the data source is considered inaccessible.

[0028] The data size estimation is obtained by randomly selecting a certain proportion of data units, downloading the selected data units according to the data unit download address, obtaining the size of the data units, calculating the average size, multiplying the average size by the total number of data units, and obtaining the estimated data size.

[0029] When a data item request is encountered, data source accessibility verification and data size estimation are performed, if the data source is inaccessible, the data state is changed to "missing"; if the data source is accessible, the data is downloaded from the data source, the data state is updated to "stored", the data download completion time is recorded, and the data is transmitted to the request; if the data state is "stored", the data is checked for expiration, if the current time has exceeded the time in the expiration information from the data download time, the data state is changed to "expired", the locally stored data is deleted, data source accessibility verification and data size estimation are performed again, if the data state is restored to "indexed", the data is downloaded from the data source, the data state is updated to "stored", the data download completion time is recorded, and the data is transmitted to the request.

[0030] The data state is updated according to the request count, specifically, a count period starts from the first request, the count period includes five time-equal stages, at the end of the count period, it is judged whether the counts of the three stages are greater than 0, if yes, the state is updated to "stored", and a new count period starts, otherwise, the current state is updated, if the current state is "stored", it is updated to "cached", if the current state is "cached", it is updated to "indexed", and the locally stored data is deleted.

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings, and the specific implementation of the "npm package metadata" is taken as an example.

[0032] S1. Create the "npm package metadata" data item, and add:

[0033] (1) Data source HTTP link: "http: / / registry.npm.taobao.org"

[0034] (2) Expiration information: 90 days

[0035] (3) Necessary authentication information: empty

[0036] (4) Add a data unit list acquisition script, and the following is the script pseudocode:

[0037] 1. Access "http: / / registry.npm.taobao.org / - / all" to obtain the data unit list, that is, the npm package list;

[0038] 2. Traverse the package list, and for each data unit, i.e. npm package, construct its access address: "http: / / registry.npm.taobao.org / <package name>".

[0039] S2. Initialization of data items, verification of data source accessibility, estimation of data size, access to http: / / registry.npm.taobao.org, check if the response status is 200, randomly select 100 data units, access their addresses, download data, calculate the average data size as 20KB, and estimate the size as 30,000,000KB;

[0040] S3. Respond to data item requests, re-verify data source accessibility and estimate data size, then update data status and perform corresponding actions;

[0041] S4. Update the data status according to the request count, preferably, a count period is 5 weeks, if the count is greater than 0 for 3 weeks, update the status to "stored", start a new count period, otherwise, update according to the current status, if the current status is "stored", update to "cached", if the current status is "cached", update to "indexed", and delete the locally stored data.

[0042] In order to illustrate the effectiveness of the method in terms of storage and request response efficiency, 50 data items are randomly set, and the following data acquisition request simulation and analysis are performed:

[0043] Initially, the status of the 50 data items is set to "cached", and the data acquisition request count of the 50 data items for 6 consecutive weeks is collected, assuming that the data volume of 20 data items is equal.

[0044] At the end of the first count period, 34 data items meet the conditions, and are converted to "stored" status, and the rest are converted to "indexed" status, at this time, only 68% of the data needs to be stored locally, and in the following week, more than 99.9% of the data acquisition requests can be responded immediately, and 10 data items need to be downloaded from the data source again and then responded.

[0045] The results show that the present application reduces the storage occupation while providing the ability to quickly respond to common data acquisition requirements.

[0046] Although the present application has been disclosed as above, it is not intended to limit the present application, and appropriate modifications or equivalent replacements of the technical solutions of the present application made by those skilled in the art should be covered within the protection scope of the present application, and the protection scope of the present application is defined by the claims.

Claims

1. A method for dynamic indexing and storage of open-source software data on the Internet, characterized in that, Includes the following steps: Create data items for open-source software data on the Internet. Each data item consists of one or more data units and includes the following data descriptions: data source HTTP link, validity information, and authentication information. The data source is the download source of the data unit in the data item. The validity information refers to the validity period after the data is downloaded and stored. The authentication information is the username and password required to access the data unit, or a string representing a valid identity. All data units in a data item share the same authentication information. If not needed, the information should be empty. Perform data item initialization operations: Verify data source accessibility. If the data source is accessible, download the data from the data source via the HTTP link, estimate the data size, and mark the data status based on the data source accessibility verification result. Data item statuses include indexed, cached, stored, lost, and expired. Indexed indicates that data accessibility has been verified and data size has been estimated, but the data has not yet been downloaded; cached indicates that the data has been downloaded and temporarily stored locally; stored indicates that the data has been downloaded and permanently stored locally; lost indicates that the data is no longer accessible; expired indicates that the data no longer meets timeliness requirements. When a data item download request is received, the accessibility of the data source is verified. If the data source is accessible, the data is downloaded from the data source via the HTTP link, the data size is estimated, and then the downloaded data is sent to the requester. The data status is marked according to the data accessibility verification result. The data item's expiration information is checked based on its data status. If the download time has exceeded the expiration information, the data status is updated, the downloaded data is deleted locally, and the data source accessibility verification is performed again. If the data source is accessible, the data is downloaded from the data source via the HTTP link, the data size is estimated, and then the downloaded data is sent to the requester. The data item's data status is also updated. Update the data status of the data item based on the counting results within the requested counting period.

2. The method as described in claim 1, characterized in that, The method for verifying data source accessibility is as follows: randomly select a certain number of data units, initiate an HTTP request based on the download address of the data unit, and include the necessary authentication information in the HTTP request. If an access to any data unit receives a response with a status code of 200, the data source is considered accessible; if an access to any data unit does not receive a response with a status code of 200, the data source is considered inaccessible.

3. The method as described in claim 1, characterized in that, The method for estimating the data size is as follows: randomly select a certain proportion of data units, download the selected data units according to the data unit download address, obtain the size of the data units, and calculate the average size; then multiply the average size by the total number of data units to obtain the estimated data size.

4. The method as described in claim 1, characterized in that, When creating a data item, a script for retrieving the data unit list is added. During subsequent data item initialization and download requests, if the data source is accessible, the script for retrieving the data unit list is executed to obtain the list of data units, and then the data is downloaded from the data source. The script for retrieving the data unit list includes two types: explicit and implicit. The explicit type is used when the data source provides a list of data units, and the script retrieves the download address of each data unit. The implicit type is used when the data source does not provide a list of data units, and the script explores the data unit numbers incrementally from smallest to largest to generate the download address of the data units.

5. The method as described in claim 1, characterized in that, If the data source is accessible, set the data status to indexed when estimating the data size; if the data source is inaccessible, set the data status to lost.

6. The method as described in claim 1, characterized in that, When a data item download request is received, the accessibility of the data source is verified. If the data source is inaccessible, the data status is changed to lost. If the data source is accessible, update the data status to "stored".

7. The method as described in claim 1, characterized in that, Checking data expiration information based on the data status of a data item means checking the data validity period when the data status of the data item is "stored" and changing the data status to "expired" when the validity period has expired.

8. The method as described in claim 1, characterized in that, When re-verifying the accessibility of the data source, if the data source is accessible, the data state is restored to indexed, and then after downloading the data from the data source, the data state is updated to stored.

9. The method as described in claim 1, characterized in that, The method for updating the data status based on the request counting cycle is as follows: A counting cycle begins from the first request. Each counting cycle contains five phases with equal durations. At the end of a counting cycle, a check is performed. If the counts of three phases are all greater than 0, the status is updated to "stored" and a new counting cycle begins. Otherwise, the data status is updated to a new status based on the current data status. That is, if the current data status is "stored", it is updated to "cached"; if the current data status is "cached", it is updated to "indexed" and the locally stored data is deleted.

Images