Data acquisition method for industry price information

Abstract

The invention provides a data acquisition method for industry price information, which belongs to the field of big data application and analysis, and comprises the following steps of: performing high-frequency traversal requests on data interfaces related to industry price information of one or more target sites on the basis of an internet data acquisition technology, and storing the data as structured industry price data; therefore, the real-time dynamic acquisition of the industry price information is achieved. Real-time acquisition is performed on the data open interface of the industry price information publication website, so that dynamic traversal of the industry price data is realized. The method is simple, efficient, accurate and real, the cost of data collection is reduced, a data basis is provided for subsequent application of big data technologies such as data processing, data analysis and modeling simulation, and the utilization efficiency of internet data is greatly improved.

Description

Technical Field

[0001] The present invention relates to the fields of big data application and analysis, data mining and collection, and in particular to a method for collecting data on industry price information. Background Art

[0002] Industry price information data has the characteristics of large quantity, multiple dimensions, high value, strong timeliness, long use cycle, scattered use locations, and real-time dynamic updates. Affected by various factors such as data statistical channels, information security, data transmission call speed, web system operation pressure, data storage cost, etc., only official websites and a few large public resource trading platforms are capable of providing the latest and most accurate data. However, industry price information has strong timeliness and is usually released daily. In addition, such information is often released in the form of web page announcements, which is difficult to structure and does not support the query and download of historical information. It is difficult to merge and summarize announcement information from different periods, which brings difficulties to data analysis and is difficult to meet the data needs of practitioners. Doing a good job in market information investigation can understand the satisfaction of people's needs, provide decision-making basis for commodity producers and operators, and strengthen the planning of production and operation; and provide them with timely and effective information for business operations, enhance their flexibility and initiative in operation, and improve economic benefits.

[0003] Therefore, applying Internet data collection technology to regularly access the data query interface of official websites or professional online product information platforms (such as official websites of statistical departments, third-party product wholesale and trading platforms, etc.) to obtain real-time and high-frequency public price data of the industry has important guiding significance for industry analysis.

[0004] In the process of industry price data analysis and interpretation, due to the limited scope of information disclosed by local statistical bureaus, a single public resource trading platform is difficult to cover all industries, resulting in the inability to obtain sufficient data samples. In addition, the data source platform is updated frequently and most information is released in the form of announcement texts, making it difficult to obtain the latest and accurate structured data in a timely manner. Most of them do not support the query of historical information and cannot meet the data requirements of big data technology analysis applications. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a data collection method for industry price information, which realizes dynamic traversal of industry price data by real-time collection of data open interfaces of industry price information disclosure websites. It is efficient, accurate and authentic, reduces the cost of data collection, and provides a data basis for the subsequent application of big data technologies such as data processing, data analysis, modeling and simulation, and greatly improves the utilization efficiency of Internet data.

[0006] The technical solution of the present invention is:

[0007] A data collection method for industry price information comprises the following steps:

[0008] (1) Determine the collection monitoring sites and target fields as needed;

[0009] (2) Obtain the data interface of the target field;

[0010] (3) Set the update scope, frequency and storage limit;

[0011] (4) Traverse the data interface and extract the target field;

[0012] (5) The collection results are accumulated to form a data set.

[0013] Collect data from Internet websites that publicly release industry price information. Based on Internet data collection technology, make high-frequency traversal requests to the data interfaces related to industry price information of one or more target sites and save them as structured industry price data, so as to achieve real-time dynamic acquisition of industry price information.

[0014] in,

[0015] The Internet website that publicly releases industry price information may be an official website or a third-party public resource trading platform.

[0016] The data interfaces related to the industry price information mentioned above can usually be divided into: industry product classification, industry price list, industry price details, etc., in a progressive relationship. The hierarchical relationship of the information interfaces provided by different websites is different, and the website's own settings shall prevail.

[0017] The high-frequency traversal request can set the scope of each collection request and the collection start interval for different websites as needed. For example, collecting agricultural products, minerals, medicines, etc., three times a day (such as 6:00, 12:00, 20:00), etc.

[0018] The structured industry price data is obtained by extracting target fields from the full text of the response returned after the request interface based on string extraction technology (such as regular or XPath expressions, semantic analysis technology, etc.), and can then be stored in a database.

[0019] The target fields are data items related to industry price information, which are generally divided into release time, release agency, product / variety, place of origin, categories at all levels (such as fruits-berries-grapes-Kyoho), specifications, brand, price, lowest price, highest price, unit, year-on-year change, rate of change, increase / decrease, trend, etc.

[0020] Visit the websites selected above one by one, and use the Web debugging tool to obtain the interface mode of the industry price information on the data source website.

[0021] The data stored in the database can be stored in a retention period as required, for example, only data from the past year can be stored. Data that exceeds the set retention period is considered redundant data and can be automatically cleared regularly.

[0022] The beneficial effects of the present invention are

[0023] Based on data collection technology, high-frequency traversal requests are made to the industry price data interface, so as to achieve real-time dynamic acquisition of industry price information and meet the data collection needs of industry data analysis. It solves the problem that due to the limited scope of information disclosed by local statistical departments, a single public resource trading platform is difficult to cover all industries, resulting in the inability to obtain sufficient data samples, and the data source platform is frequently updated and mostly releases information in the form of announcement texts, making it difficult to obtain the latest and accurate structured data in a timely manner. Most of them do not support the query of historical information and cannot meet the data needs of big data technology analysis applications. The data dimensions of the collection results are clear, the utilization value is high, and it is efficient, accurate and true, which reduces the cost of data collection and provides convenience for subsequent work such as data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the workflow of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] The present invention provides a data collection method for industry price information, which mainly comprises the following steps:

[0027] 1. Determine the collection monitoring sites and target fields as needed. Determine the collection sites according to the collection needs, such as the official website of the national or local statistics bureau, or a third-party public resource trading platform. The target dimension is related to industry prices. The target fields are generally divided into data items related to industry price information, such as release time, release agency, product / variety, place of origin, categories at all levels (such as fruits-berries-grapes-Kyoho), specifications, brand, price, lowest price, highest price, unit, year-on-year change, rate of increase / decrease, trend, etc.

[0028] 2. Get the data interface of the target field. The information interfaces provided by different websites are different. They can usually be divided into: industry product classification, product price list, product price details, etc., which are in a progressive relationship. Visit the websites selected in step 1 one by one, and use the Web debugging tool to obtain the interface mode of industry price information on the data source website.

[0029] 3. Set the update scope, frequency and storage limit. Set the scope of each collection, collection start interval, and retention period of historical data for different websites as needed. For example, collect agricultural products, minerals, medicines, etc., collect three times a day (such as 6:00, 12:00, 20:00), collect information released on the same day each time, and keep the collected data for one year.

[0030] 4. Traverse the data interface and extract the target field. According to the update range and frequency in step 3, traverse the interface in step 2 to obtain the response information of the data source platform. Combine string extraction technology (such as regular or XPath expression, semantic analysis technology, etc.) to extract the target field selected in step 1 from the full text of the response to form structured data.

[0031] The collection results are accumulated to form a data set. The structured data extracted in step 4 is stored in the database to form an industry price information set, and expired redundant data is cleaned up according to the retention period set in step 3.

[0032] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data collection method for industry price information, characterized in that: The steps include: (1) Determine the collection monitoring sites and target fields as needed; (2) Obtain the data interface of the target field; (3) Set the update scope, frequency and storage limit; (4) Traverse the data interface and extract the target field; (5) The collection results are accumulated to form a data set.

2. The method according to claim 1, characterized in that The collection site is determined according to the collection requirements. The target dimension is related to industry prices, and the target field is divided into data items related to industry price information.

3. The method according to claim 2, characterized in that The information interfaces provided by different websites are different, and are divided into: industry product classification, product price list, and product price details, and the three are in a progressive relationship.

4. The method according to claim 2, characterized in that: Visit the websites selected in step (1) one by one, and use the Web debugging tool to obtain the interface mode of the industry price information on the data source website.

5. The method according to claim 1, characterized in that Set the scope and interval of collection for different websites as needed.

6. The method according to claim 1, characterized in that The retention period of the data stored in the database can be set as needed.

7. The method according to claim 1, characterized in that According to the update range and frequency in step (3), the interface in step (2) is accessed cyclically to obtain the response information of the data source platform; Combined with string extraction technology, the target field selected in step (1) is extracted from the full text of the response to form structured data.

8. The method according to claim 1, characterized in that The structured data extracted in step (4) is stored in the database to form an industry price information set, and expired redundant data is cleaned up according to the retention period set in step (3).

Images