Data acquisition method and device, electronic equipment and storage medium

Abstract

The application discloses a data collection method and device, electronic equipment and storage medium. The method comprises the following steps: when a data collection request is received, at least one resource location information corresponding to the data collection request is determined; the page type of at least one subpage in a to-be-collected page corresponding to the current resource location information is determined; based on the page type and a collection processing mode corresponding to the page type, the page data corresponding to the corresponding subpage is acquired; the collection processing mode comprises a resource location jump mode, a storage location replacement mode and a protocol address replacement mode; based on the page data corresponding to each subpage, target data corresponding to the to-be-collected page is determined. The method solves the problem that the existing technology limits the webpage access frequency, resulting in low and incomplete collection efficiency, improves the convenience and comprehensiveness of data collection, and effectively solves the problem of difficult data collection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer processing, and in particular to a data collection method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the advent of the big data era, the information on the Internet is also increasingly rich, which has great practical value, so the way of collecting Internet data in batches is also increasing, but due to the complexity and diversity of web data, and the data provider also sets a large number of data access verification, such as Captcha recognition, therefore there is a problem of data collection difficulty.

[0003] The existing data collection method is usually to collect data by limiting the access frequency of the webpage, for example, when collecting data on a certain website reaches a certain time threshold, the website is exited, but this method cannot solve the problem of data collection difficulty, and there is a problem of low collection efficiency and incomplete collection. SUMMARY

[0004] The present application provides a data collection method, device, electronic equipment and storage medium to improve the convenience and comprehensiveness of data collection and effectively solve the problem of data collection difficulty.

[0005] According to an aspect of the present application, a data collection method is provided, which comprises:

[0006] When receiving a data collection request, determining at least one resource location information corresponding to the data collection request;

[0007] Determining the page type of at least one sub-page in the to-be-collected page corresponding to the current resource location information;

[0008] Based on the page type and the collection processing mode corresponding to the page type, the page data corresponding to the corresponding sub-page is obtained; wherein the collection processing mode includes resource location jump mode, storage location replacement mode and protocol address replacement mode;

[0009] Based on the page data corresponding to each sub-page, the target data corresponding to the to-be-collected page is determined.

[0010] According to another aspect of the present application, a data collection device is provided, which comprises:

[0011] The resource location information determination module is configured to, when receiving a data collection request, determine at least one resource location information corresponding to the data collection request;

[0012] The page type determining module is configured to determine a page type of at least one sub-page in the page to be collected corresponding to the current resource positioning information.

[0013] The page data determining module is configured to acquire page data corresponding to the respective sub-page based on the page type and a collection processing mode corresponding to the page type, wherein the collection processing mode comprises a resource positioning jump mode, a storage location replacement mode and a protocol address replacement mode.

[0014] The target data determining module is configured to determine target data corresponding to the page to be collected based on the page data corresponding to each sub-page.

[0015] According to another aspect of the present application, an electronic device is provided, which comprises:

[0016] at least one processor; and

[0017] a memory connected to the at least one processor in communication; wherein

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data collection method according to any one of the embodiments of the present application.

[0019] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the data collection method according to any one of the embodiments of the present application when executed.

[0020] The technical solution of the embodiments of the present application determines at least one resource positioning information corresponding to a data collection request when the data collection request is received, determines a page type of at least one sub-page in a page to be collected corresponding to the current resource positioning information, acquires page data corresponding to the respective sub-page based on the page type and a collection processing mode corresponding to the page type, wherein the collection processing mode comprises a resource positioning jump mode, a storage location replacement mode and a protocol address replacement mode, and determines target data corresponding to the page to be collected based on the page data corresponding to each sub-page, thereby solving the problem of low and incomplete collection efficiency caused by limiting the webpage access frequency in the prior art, and realizing the acquisition of page data corresponding to the page by determining the page type corresponding to the collection page, acquiring the page data corresponding to the page based on the resource positioning jump mode, the storage location replacement mode and the protocol address replacement mode corresponding to the page type and other collection processing modes, improving the convenience and comprehensiveness of data collection, and effectively solving the problem of difficult data collection.

[0021] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the application to the specific embodiments described. Rather, the scope of the embodiments of the application is to be defined by the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0023] Figure 1 is a flow chart of a data acquisition method according to an embodiment of the present application;

[0024] Figure 2 is a flow chart of a data acquisition method according to an embodiment of the present application;

[0025] Figure 3 is a flow chart of a data acquisition method according to an embodiment of the present application;

[0026] Figure 4 is a flow chart of a data acquisition method according to an embodiment of the present application;

[0027] Figure 5 is a flow chart of a data acquisition method according to an embodiment of the present application;

[0028] Figure 6 is a structural schematic diagram of a data acquisition device according to an embodiment of the present application;

[0029] Figure 7 is a structural schematic diagram of an electronic device for implementing the data acquisition method according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the technical personnel in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a data acquisition method according to Embodiment 1 of the present invention. This embodiment is applicable to data acquisition situations. The method can be executed by a data acquisition device, which can be implemented in hardware and / or software and can be configured in a computing device. Figure 1 As shown, the method includes:

[0034] S110. When a data acquisition request is received, at least one resource location information corresponding to the data acquisition request is determined.

[0035] The data acquisition request can be an instruction or code to request data. For example, a corresponding data acquisition request can be generated based on the type of data the user needs to collect, so as to obtain data that meets the user's needs based on the data acquisition request. The resource location information can be a URL (Uniform Resource Locator), which is used to specify the network address of the resource information location.

[0036] In practical applications, when an uploaded webpage URL is detected, it can be considered that a data collection request has been received. The data collection request can be parsed to obtain at least one URL information carried in the request, which serves as resource location information. Alternatively, when the uploaded data type to be collected is detected, it can be considered that a data collection request has been received. At this time, at least one URL information corresponding to that data type can be automatically determined as resource location information. The webpage content in the resource location information corresponds to that data type. The URL information can be pre-stored in a preset location or collected from the network. This allows for locating the corresponding webpage based on the resource location information for subsequent data collection.

[0037] S120. Determine the page type of at least one subpage among the pages to be collected that corresponds to the current resource location information.

[0038] The "page to be collected" can be understood as the page whose page data needs to be collected. This page may contain subpages of various page types, such as inward scrolling, outward scrolling, and pagination. An inward scrolling page is generally part of an outward scrolling page. For example, a normal page might contain a scrollable container. The inward scrolling page will not trigger the outward scrolling page; it will only trigger when scrolling to the top or bottom. A pagination page can be understood as a page with pagination. It is generally part of an outward scrolling page and may contain multiple pages. Each page can be displayed at a time, and switching between pages can be achieved by swiping left or right. Switching between pages will not trigger the outward scrolling page. The method for collecting page data from each resource location is the same; any one of these resource location information will be used as the current resource location information for explanation.

[0039] It's important to note that web pages on the internet are compiled from computer languages. This means that the page type is predetermined during compilation, and different page types require different compilation languages. Each web page can be composed of HTML (Hypertext Markup Language) and CSS (Cascading Style Sheets). HTML includes a series of tags that unify the formatting of documents on the web, connecting scattered internet resources into a logical whole. HTML text is descriptive text composed of HTML commands, which can specify text, graphics, animations, sounds, tables, links, etc. CSS is a computer language used to style HTML or XML documents. CSS can not only statically format web pages but also dynamically format various elements of a web page in conjunction with various scripting languages.

[0040] In practical applications, after entering the webpage containing the current resource location information, i.e. the page to be collected, the page type can be determined by checking the page format of each subpage in the page to be collected.

[0041] Optionally, the method for determining the page type of at least one subpage in the page to be collected corresponding to the current resource location information can be: jumping to the page to be collected corresponding to the current resource location information; for each subpage in the page to be collected, when collecting the page data of the current subpage, determining the page format of the current subpage, and determining the page type corresponding to the current subpage based on the page format.

[0042] In this embodiment, after obtaining the resource location information, the user can jump to the page to be collected corresponding to the resource location information and collect data from top to bottom on the page to be collected. When collecting page data on the page to be collected, it is necessary to determine the page type of the current subpage being collected. This can be done by parsing the program code corresponding to the current subpage to determine the page format of the current subpage, and based on the page format, determining the current subpage corresponding to the current subpage.

[0043] Building upon the above approach, when importing resource location information, a corresponding data acquisition toolkit can also be imported to enable data parsing and extraction based on the toolkit. For example, the data acquisition toolkit could be BeautifulSoup, a parsing library developed using RE (Research, Internet Explorer, and Web Scraping) technology. BeautifulSoup is used to extract data from HTML or XML files and boasts powerful parsing capabilities. It can also be used with converters to achieve familiar document navigation, searching, and modification effects, improving the efficiency of data extraction and web crawler development.

[0044] S130. Based on the page type and the corresponding acquisition and processing method, obtain the page data corresponding to the sub-page.

[0045] It should be noted that the data collection methods may differ or be the same for different page types. For example, page-turning pages require left and right page switching for data collection, while inward-scrolling and outward-scrolling pages do not. Furthermore, each page type may require data storage, and the data storage method may be the same. To improve the comprehensiveness, accuracy, and convenience of data collection, appropriate collection methods can be applied to different page types.

[0046] The data collection and processing methods include resource location redirection, storage location change, and protocol address change. Resource location redirection effectively redirects data to prevent the webpage owner from restricting data collection if they detect prolonged data collection from the same webpage. Storage location change reasonably changes storage locations to prevent the webpage owner from restricting data collection if they detect large amounts of webpage data being collected from the same storage location. Protocol address change reasonably and effectively changes the client's protocol address to prevent the webpage owner from restricting data collection if they detect frequent and prolonged data collection from a specific protocol address.

[0047] In practical applications, corresponding data collection methods can be applied to subpages of different page types to obtain the page data of the subpage. The following describes the implementation methods for data collection from pages with inward scrolling, outward scrolling, and page-turning types:

[0048] Optionally, if the page type is an inward scrolling type, then determine the scrolling attributes corresponding to the inward scrolling type; based on the current scrolling cycle and scrolling displacement, determine the current window page, and obtain at least one tag to be used and its corresponding tag content corresponding to the current window page; based on at least one tag to be used and its corresponding tag content corresponding to each window page, determine the page data of the corresponding subpage.

[0049] The scroll attributes include the scroll cycle, scroll displacement, and waiting time for scrolling. The scroll displacement can be understood as the amount of displacement per scroll. These scroll attributes can be defined by technical personnel based on actual work requirements. The tags to be used can be tags within the `<head>` section of a webpage.

[0050] It's important to note that when retrieving page data for scrolling content, the data will only be loaded after a certain period of time. To address this, dynamic data tags can be loaded by simulating a user scrolling down the screen.

[0051] In practical applications, when a subpage is detected as having an inward scrolling type, the scroll cycle and scroll displacement in the component tools can be used to periodically scroll to a certain coordinate on the page. The loaded page is then used as the current window, and the page data corresponding to this window is extracted. This page data includes at least one tag to be used and its corresponding content. For example, `Window.scrollTo(parameter1, parameter2, ...)` can be used to simulate a user scrolling down the screen to load dynamic data tags. When the scrollbar of the inward scrolling page is dragged to the bottom, data collection is considered complete, and data can continue to be collected from the corresponding outward scrolling page.

[0052] Optionally, if the page type is an external scrolling type, then during the process of scrolling the scrollbar to the bottom of the corresponding subpage, at least one tag to be applied and its corresponding tag content are obtained; based on each tag to be applied and its corresponding tag content, the page data of the corresponding subpage is determined.

[0053] It's important to note that on external scrolling pages, when elements extend beyond one screen, attempting to manipulate elements at the bottom of the screen will result in an "elements are not visible" error if you directly locate the scrollbar at a specific coordinate. Therefore, scrollbars on external scrolling pages cannot be directly positioned using positioning tools. To address this, you can control the scrollbar to slide to the bottom of the page to directly access the page data corresponding to the external scrolling page.

[0054] In practical applications, when the page type of a subpage is detected to be an inward scrolling type, the scroll bar can be controlled to slide to the bottom of the page to load the data in the page. During the process of the scroll bar scrolling to the bottom of the corresponding subpage, the page data of the subpage is obtained. The page data includes at least one tag to be applied and the corresponding tag content.

[0055] Optionally, if the page type is a pagination switching type, then based on each pagination page in the corresponding subpage, determine the number of pagination steps, and determine the current pagination position based on the number of pagination steps; based on the current pagination position, switch from the previous pagination page to the current pagination page, and obtain at least one tag to be stored and the corresponding tag content corresponding to the current pagination page; based on at least one tag to be stored and the corresponding tag content corresponding to each pagination page, determine the page data of the corresponding subpage.

[0056] It's worth noting that some web pages have pagination. When clicking the next page, the page's URL remains unchanged, but the link in the address bar will change. In this case, you can achieve normal page loading and data retrieval by modifying the link in the address bar.

[0057] In practical applications, when a subpage's page type is detected as a pagination type, the required number of paginations can be counted, i.e., the number of paginations for each page within the subpage. Further, corresponding pagination links, i.e., pagination addresses, are generated based on the pagination count. For example, the initial pagination link can be set to 1, serving as the pagination link for the first page, loading the first page's data. Adding 1 to 1 generates the pagination link for the second page, switching from the first page to the second page, and retrieving the second page's data. Correspondingly, the page data corresponding to each pagination page can be obtained, thus acquiring all the page data for the subpage. For example, the pagination process steps can be: 1. Obtain the total number of paginated pages; 2. Use a for loop to change the value of `page` in the link to perform pagination. If the link contains the keyword `page`, changing the value of `page` in the link to 3 allows normal access to the third page. By changing the value of keywords in the link, pagination can be achieved, thereby extracting pagination tag data from all web pages, realizing the convenience of dynamic data acquisition.

[0058] S140. Based on the page data corresponding to each subpage, determine the target data corresponding to the page to be collected.

[0059] In practical applications, when acquiring page data from subpages, the tags and their corresponding content in the page data can be stored in a corresponding manner. For example, the key-value index and list index of a dictionary can be used together for storage, with the tags as keys and the corresponding tag content as values, to obtain the target data corresponding to the page to be collected, so as to facilitate the subsequent retrieval and use of the page data.

[0060] The technical solution of this invention, upon receiving a data collection request, determines at least one resource location information corresponding to the request; determines the page type of at least one subpage in the page to be collected corresponding to the current resource location information; and obtains the page data corresponding to the subpage based on the page type and the corresponding collection processing method. The collection processing method includes resource location redirection, storage location change, and protocol address change. Based on the page data corresponding to each subpage, the target data corresponding to the page to be collected is determined. This solves the problem of low collection efficiency and incompleteness caused by limiting the frequency of webpage access in existing technologies. It achieves the acquisition of page data corresponding to the page by determining the page type of the page to be collected and using collection processing methods such as resource location redirection, storage location change, and protocol address change, thereby improving the convenience and comprehensiveness of data collection and effectively solving the problem of difficult data collection.

[0061] Example 2

[0062] Figure 2 This is a flowchart of a data acquisition method according to Embodiment 2 of the present invention. Based on the foregoing embodiments, S130 is further refined. For specific implementation details, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0063] like Figure 2 As shown, the method specifically includes the following steps:

[0064] S210. When a data acquisition request is received, at least one resource location information corresponding to the data acquisition request is determined.

[0065] S220. Determine the page type of at least one subpage in the page to be collected that corresponds to the current resource location information.

[0066] S230. Determine the cumulative number of tags obtained after jumping to the page to be collected corresponding to the current resource location information.

[0067] Based on the above embodiments, when obtaining page data corresponding to a subpage based on the page type and the corresponding collection and processing method, the data collection method corresponding to the resource location jump method can be executed to prevent data collection from being restricted.

[0068] Specifically, after navigating to the page to be collected corresponding to the current resource location information, the total number of tags, such as tags to be used, tags to be applied, and tags to be stored, can be recorded to obtain the cumulative number of tags.

[0069] S240. If the cumulative number of tags exceeds the collection quantity threshold, then jump to the page to be collected corresponding to the next resource location information. When the number of tags obtained from the page to be collected of the next resource location information exceeds the collection quantity threshold, jump to the historical collection page position of the current resource location information to continue to obtain tags. Based on the tags obtained, redetermine the cumulative number of tags so as to obtain the page data of the corresponding subpage based on the obtained tags and the corresponding tag content.

[0070] The collection quantity threshold can be understood as a preset limit on the number of tags collected.

[0071] In practical applications, when the cumulative number of tags exceeds the collection threshold, it indicates that the amount of data collected on that website is excessive, which may be considered malicious data collection by the data provider. To prevent access restrictions, the resource location information can be changed. The user can then jump from the current resource location to the next resource location's corresponding page for data collection. If the number of tags collected on the next resource location's page exceeds the collection threshold, the user can return to the current resource location's page to continue collecting data from the previous location, or jump to another resource location's page for data collection. It's important to note that each time the user jumps to a page corresponding to a resource location, the previous cumulative tag count is reset to zero, setting the initial tag count to 0. The cumulative tag count is determined from this zero when collecting tags. In other words, each time the user jumps to a website, the cumulative tag count is recalculated from 0. For example, upon successfully collecting the first tag on a website, the user can immediately exit the webpage and enter a second webpage for tag collection. Upon successful access to the second webpage, the user can return to the first webpage or another page to continue tag collection, thus ensuring that access is not restricted.

[0072] S250. Based on the page data corresponding to each subpage, determine the target data corresponding to the page to be collected.

[0073] The technical solution of this embodiment determines the cumulative number of tags obtained after jumping to the page to be collected corresponding to the current resource location information. When the cumulative number of tags exceeds a preset collection quantity threshold, the website for collection is changed, and the user is jumped to the page to be collected corresponding to the new resource location information. This effectively prevents data collection from being restricted and improves the efficiency of data collection.

[0074] Example 3

[0075] Figure 3 This is a flowchart of a data acquisition method according to Embodiment 3 of the present invention. Based on the foregoing embodiments, S130 is further refined. For specific implementation details, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0076] like Figure 3 As shown, the method specifically includes the following steps:

[0077] S310. When a data acquisition request is received, at least one resource location information corresponding to the data acquisition request is determined.

[0078] S320. Determine the page type of at least one subpage in the page to be collected that corresponds to the current resource location information.

[0079] S330. For each data acquisition cycle, determine the data storage space corresponding to the current data acquisition cycle, and store the page data corresponding to the corresponding sub-page acquired in the current data acquisition cycle into the data storage space.

[0080] It should be noted that when obtaining page data corresponding to a subpage based on the page type and the corresponding data collection and processing method, the data collection and processing method corresponding to the storage location change method can be executed to prevent data collection from being restricted.

[0081] The data collection cycle is a duration determined based on a collection duration threshold, such as 10 minutes or 5 minutes. The collection duration threshold can be understood as a preset limit for tag collection time, used to adjust the data storage location. The data storage space corresponding to the current data collection cycle is different from the data storage space corresponding to the next data collection cycle.

[0082] Specifically, when collecting page data corresponding to a subpage, the collection time can be recorded to obtain the cumulative collection time. When the cumulative collection time reaches the data collection cycle threshold, the data storage space needs to be changed. In practical applications, at the initial collection time of page data corresponding to a subpage, a data storage space to be used within this data collection cycle can be determined, such as mysql_1. During the data collection process, recording can begin from the initial collection time. When the cumulative collection time reaches the data collection cycle threshold, the task of this data collection cycle can be considered complete. Simultaneously, during the data collection process, the page data collected in this data collection cycle is stored in mysql_1 in real time. The next data collection cycle begins, and the cumulative collection time is recalculated, and the corresponding data storage space is determined, such as mysql_2. When the cumulative collection time reaches the data collection cycle threshold, the collected data is stored in mysql_2 in real time, and the data storage space to be stored for the next data collection cycle is determined.

[0083] It's important to note that multiple local MySQL instances can be configured. When determining the current data storage space, a MySQL instance can be randomly selected from among them, as long as the MySQL instance used for storing data in the current data collection cycle is different from the one used in the next data collection cycle. Alternatively, the MySQL instances can be queued and used sequentially. For example, when collecting data, the collected page data can be stored in MySQL1. When the cumulative data collection time reaches the corresponding duration of the data collection cycle, MySQL2, which is next in line after MySQL1, becomes the data storage space to be used. At this time, a new round of data collection is started, and the collected data is stored in MySQL2, and the cumulative data collection time is obtained. When the data collection time reaches the corresponding duration of the data collection cycle, the data storage space is changed. After all MySQL instances have been used, they can be queued based on their remaining memory, prioritizing the MySQL instance with more remaining memory. If a MySQL instance is out of memory, the data is then stored in the next MySQL instance. This not only avoids data collection being limited but also achieves real-time dynamic data storage, improving data collection efficiency.

[0084] S340. Establish a mapping relationship between at least one tag to be parsed in the page data and the corresponding tag content.

[0085] In this embodiment, during the process of storing page data in the data storage space, a mapping relationship can be established between each tag to be parsed and its corresponding tag content. For example, by using a combination of dictionary key-value indexes and list indexes for storage, tags can be used as keys and their corresponding tag content as values ​​to obtain the target data corresponding to the page to be collected, so as to facilitate subsequent retrieval and use of page data.

[0086] It should be noted that after the page data is collected, the page data may be encrypted with strings. In this case, the tag data in the page data can be decrypted to obtain the decrypted data for subsequent data storage.

[0087] Optionally, before establishing the mapping relationship between at least one tag to be parsed and its corresponding content in the page data, the method further includes: if the tag type of the tag to be parsed is encrypted, then the tag to be parsed and its corresponding content are decrypted to obtain the decrypted tag to be parsed and its corresponding content.

[0088] In practical applications, the tag type of the tag to be parsed can be analyzed. When the tag type is encrypted, the tag to be parsed and its corresponding content are decrypted to obtain the decrypted tag and its corresponding content. For example, when using the requests library to make requests and parse data, the json() method can be used to convert the string-formatted JSON data into dictionary format. Then, the dictionary's key-value index and list index can be used together to parse the JSON data. Alternatively, the get() method and list index can be used to parse the JSON data to obtain the decrypted tag and its corresponding content.

[0089] S350. Based on the page data corresponding to each subpage, determine the target data corresponding to the page to be collected.

[0090] The technical solution of this embodiment determines the cumulative data collection time corresponding to the acquisition of page data corresponding to the subpage. When the cumulative data collection time exceeds the collection time threshold, the data storage space is changed to prevent the situation where access is restricted due to long-term single data transmission to a certain database, thereby improving the efficiency of data collection.

[0091] Example 4

[0092] Figure 4 This is a flowchart of a data acquisition method according to Embodiment 4 of the present invention. Based on the foregoing embodiments, S130 is further refined. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0093] like Figure 4As shown, the method specifically includes the following steps:

[0094] S410. When a data acquisition request is received, at least one resource location information corresponding to the data acquisition request is determined.

[0095] S420. Determine the page type of at least one subpage in the page to be collected that corresponds to the current resource location information.

[0096] S430. For each address change cycle, retrieve the Internet Protocol address of the client corresponding to the current address change cycle from the pre-created protocol address pool, and delete the Internet Protocol address from the protocol address pool, so as to generate a data collection request based on the Internet Protocol address when obtaining the page data of the corresponding subpage.

[0097] Based on the above embodiments, when obtaining page data corresponding to a subpage based on the page type and the corresponding collection and processing method, the collection and processing method corresponding to the protocol address change method can be executed to prevent data collection from being restricted.

[0098] The address change cycle can be understood as the duration for changing the protocol address. The protocol address pool includes the Internet Protocol address of at least one client.

[0099] Specifically, at each address change cycle, a new client Internet Protocol address can be randomly retrieved from the protocol address pool and deleted from the pool. This allows for the generation of a data collection request corresponding to the new Internet Protocol address to continue data loading and collection. This ensures the randomness and non-repetition of the Internet Protocol address, and the address will not be used again after each use, thus better guaranteeing smooth website access and preventing restrictions.

[0100] To ensure that the protocol address pool stores a number of Internet Protocol addresses that meet the preset requirements, optionally, the method also includes: when it is detected that the number of protocol addresses to be used in the protocol address pool is less than the preset number, generating protocol addresses to be stored based on the address generation component, and storing the protocol addresses to be stored in the protocol address pool.

[0101] In this embodiment, the Internet Protocol address pool is depleted as each address is used up. The number of available Internet Protocol addresses in the pool can be monitored in real time. When the number is less than a preset value, a new Internet Protocol address can be generated using an address generation component and stored in the pool as a new address to be stored. This enables online real-time acquisition of Internet Protocol addresses, ensuring that the acquired addresses are random and non-repeating, thus preventing access restrictions.

[0102] S440. Based on the page data corresponding to each subpage, determine the target data corresponding to the page to be collected.

[0103] The technical solution of this embodiment retrieves the Internet Protocol address of the client corresponding to the current address change cycle from a pre-created protocol address pool for each address change cycle, and deletes the Internet Protocol address from the protocol address pool. In order to generate a data collection request based on the Internet Protocol address when obtaining page data of the corresponding subpage, the Internet Protocol address can be obtained online in real time, and the obtained Internet Protocol address is guaranteed to be random and non-repeating, effectively preventing access restrictions.

[0104] Example 5

[0105] As an optional embodiment of the above embodiments, Figure 5 This is a schematic diagram of a data acquisition method according to Embodiment 5 of the present invention. For details, please refer to the following specific content.

[0106] See Figure 5 You can input the URL of the webpage corresponding to the data to be collected, i.e., the resource location information, to initiate a data collection request. Simultaneously, you can import the BeautifulSoup toolkit to perform data parsing and extraction during data collection, improving the efficiency of data extraction and web crawler development. Upon receiving the data collection request, you can be redirected to the page to be collected corresponding to the resource location information, load the webpage content, and save it locally. Further, you can parse the webpage content, select the required tags, extract the tags and tag content, and store them accordingly.

[0107] Building upon the above approach, during data collection, the tag type can be detected to determine if the tag is encrypted. If the tag type is encrypted, the string needs to be deciphered. This can be done by using the `json()` method with the `requests` library to convert the string-formatted JSON data into a dictionary format. Then, the dictionary's key-value index and list index can be used in conjunction to parse the JSON data, or the `get()` method and list index can be used for parsing. JSON is a lightweight data-interchange format that uses a text format completely independent of programming languages ​​to store and represent data. JSON is smaller, faster, easier to parse, and has a concise and clear hierarchical structure than XML, improving the efficiency of data collection and parsing.

[0108] Based on the above scheme, during data collection, there will be some dynamic webpage text. Dynamic webpages refer to webpages whose data is only loaded in the window the user is currently in; the data is only loaded when the user scrolls the page. During data loading, dynamic webpages load data but do not change the resource location information (URL). Dynamic webpages include three page types: inward scrolling, outward scrolling, and page-turning. When collecting data from inward scrolling pages, the Window.scrollTo component can be used to simulate the user scrolling down the screen to load dynamic data tags. When collecting data from outward scrolling pages, the page data corresponding to the outward scrolling page can be directly obtained by controlling the scroll bar to slide to the bottom of the page. When collecting data from page-turning pages, a page-turning process is set up. The page-turning process steps can be: 1. Get the total number of pages; 2. Use a for loop to change the value of `page` in the link to perform page turning. If the link contains the keyword `page`, changing the value of `page` in the link to 3 will allow normal access to the third page. By changing the value of the keyword in the link, page turning can be achieved, thereby extracting the page-turning tag data from all webpages and realizing the convenience of dynamic data acquisition.

[0109] Building upon the above solutions, to avoid data collection restrictions and improve efficiency, the MySQL database can be changed to prevent detection of long-term reliance on a single MySQL database, which could lead to access restrictions. In practical applications, URLs are first converted to IP addresses before data transmission over the internet. If an IP receives a high number of concurrent requests to a website within a short period, that IP may be restricted from data collection. This can be mitigated by constructing a protocol address pool. For example, real-time updated client Internet Protocol addresses (IPs) can be obtained online. A specific web page IP generation component might provide approximately 60 freely usable IPs per hour. These IPs can be downloaded to the protocol address pool, and then checked for unused IPs. If an IP is unused, it is added to the pool; if it has been used, it is removed. A new IP is obtained every minute, ensuring that different IPs are used for each data collection session within a preset timeframe. This achieves randomness and non-repetition of IP addresses, and ensures that the IPs are not reused after each use, guaranteeing smooth website access. In practical applications, there may be situations where access to collect a large amount of tag data from a website is restricted. To prevent this, after successfully obtaining the first tag from a website corresponding to a resource's location information, immediately exit the webpage and proceed to a second webpage for tag crawling. Upon successful access to the second webpage, return to the first webpage to continue tag processing. Alternatively, periodically changing the User-Agent header and limiting the data collection duration can also prevent data collection from being restricted, ensuring smooth data collection.

[0110] The technical solution of this embodiment, upon receiving a data collection request, determines at least one resource location information corresponding to the data collection request; determines the page type of at least one subpage in the page to be collected corresponding to the current resource location information; and obtains the page data corresponding to the corresponding subpage based on the page type and the collection processing method corresponding to the page type. The collection processing method includes resource location redirection, storage location change, and protocol address change. Based on the page data corresponding to each subpage, the target data corresponding to the page to be collected is determined. This solves the problem of low collection efficiency and incompleteness caused by limiting the frequency of webpage access in existing technologies. It achieves the acquisition of page data corresponding to the page by determining the page type of the page to be collected and by using collection processing methods such as resource location redirection, storage location change, and protocol address change corresponding to the page type. This improves the convenience and comprehensiveness of data collection and effectively solves the problem of difficult data collection.

[0111] Example 6

[0112] Figure 6 This is a schematic diagram of the structure of a data acquisition device according to Embodiment Six of the present invention. Figure 6 As shown, the device includes: a resource location information determination module 610, a page type determination module 620, a page data determination module 630, and a target data determination module 640.

[0113] The resource location information determination module 610 is used to determine at least one resource location information corresponding to the data collection request when a data collection request is received; the page type determination module 620 is used to determine the page type of at least one subpage in the page to be collected corresponding to the current resource location information; the page data determination module 630 is used to obtain the page data corresponding to the corresponding subpage based on the page type and the collection processing method corresponding to the page type; wherein the collection processing method includes resource location jump method, storage location change method and protocol address change method; and the target data determination module 640 is used to determine the target data corresponding to the page to be collected based on the page data corresponding to each subpage.

[0114] The technical solution of this embodiment, upon receiving a data collection request, determines at least one resource location information corresponding to the data collection request; determines the page type of at least one subpage in the page to be collected corresponding to the current resource location information; and obtains the page data corresponding to the corresponding subpage based on the page type and the collection processing method corresponding to the page type. The collection processing method includes resource location redirection, storage location change, and protocol address change. Based on the page data corresponding to each subpage, the target data corresponding to the page to be collected is determined. This solves the problem of low collection efficiency and incompleteness caused by limiting the frequency of webpage access in existing technologies. It achieves the acquisition of page data corresponding to the page by determining the page type of the page to be collected and by using collection processing methods such as resource location redirection, storage location change, and protocol address change corresponding to the page type. This improves the convenience and comprehensiveness of data collection and effectively solves the problem of difficult data collection.

[0115] Optionally, based on the above-mentioned device, the page type determination module 620 includes a page jump unit to be collected and a page type determination unit.

[0116] The page to be collected jump unit is used to jump to the page to be collected corresponding to the current resource location information;

[0117] The page type determination unit is used to determine the page format of the current subpage when collecting page data of the current subpage in the page to be collected, and to determine the page type corresponding to the current subpage based on the page format.

[0118] The page types include inner scrolling, outer scrolling, and page turning / switching types.

[0119] Optionally, based on the above-mentioned device, the page data determination module 630 includes a data determination first unit, which includes a scroll attribute determination unit, a current window page determination unit, and a page data determination unit.

[0120] A scroll attribute determination unit is used to determine the scroll attribute corresponding to the inner scroll type if the page type is an inner scroll type; the scroll attribute includes a scroll period and a scroll displacement.

[0121] The current window page determination unit is used to determine the current window page based on the current scrolling cycle and scroll displacement, and to obtain at least one tag to be used and the corresponding tag content corresponding to the current window page;

[0122] The page data determination unit is used to determine the page data of the corresponding subpage based on at least one tag to be used and the corresponding tag content of each window page.

[0123] Optionally, based on the above-mentioned device, the page data determination module 630 includes a data determination second unit, which includes a tag content determination unit and a page data acquisition unit.

[0124] The tag content determination unit is used to obtain at least one tag to be applied and the corresponding tag content during the process of scrolling the scroll bar to the bottom of the corresponding subpage if the page type is an external scrolling type.

[0125] The page data acquisition unit is used to determine the page data of the corresponding subpage based on each tag to be applied and the corresponding tag content.

[0126] Optionally, based on the above-mentioned device, the page data determination module 630 includes a data determination second unit, which includes a current page position determination unit, a tag to be stored determination unit, and a tag to be stored storage unit.

[0127] The current page position determination unit is used to determine the number of page turns based on each page in the corresponding subpage if the page type is a page turning type, and to determine the current page position based on the number of page turns.

[0128] The tag to be stored unit is used to switch from the previous page to the current page based on the current page position, and to obtain at least one tag to be stored and the corresponding tag content corresponding to the current page.

[0129] The tag storage unit is used to determine the page data of the corresponding sub-page based on at least one tag to be stored and the corresponding tag content corresponding to each pagination page.

[0130] Based on the above-mentioned device, optionally, the data acquisition and processing method includes a resource location jump method, and the page data determination module 630 includes a data processing first unit, which includes a cumulative tag quantity determination unit and a resource location information jump unit.

[0131] The cumulative tag count determination unit is used to determine the cumulative tag count of the tags acquired after jumping to the page to be collected corresponding to the current resource location information;

[0132] The resource location information jump unit is used to jump to the page to be collected corresponding to the next resource location information if the cumulative number of tags exceeds the collection quantity threshold, and when the number of tags obtained from the page to be collected of the next resource location information exceeds the collection quantity threshold, jump to the historical collection page position of the current resource location information to continue to obtain tags, and redetermine the cumulative number of tags based on the tags obtained, so as to obtain the page data of the corresponding sub-page based on the obtained tags and the corresponding tag content.

[0133] Based on the above-mentioned device, optionally, the acquisition and processing method includes a storage location replacement method, and the page data determination module 630 includes a second data processing unit, which includes a data storage space determination unit and a mapping relationship establishment unit.

[0134] The data storage space determination unit is used to determine the data storage space corresponding to the current data acquisition cycle for each data acquisition cycle, and to store the page data corresponding to the corresponding sub-page acquired in the current data acquisition cycle into the data storage space; wherein, the data acquisition cycle is a duration determined based on a data acquisition duration threshold, and the data storage space corresponding to the current data acquisition cycle is different from the data storage space corresponding to the next data acquisition cycle;

[0135] The mapping relationship establishment unit is used to establish a mapping relationship between at least one tag to be parsed in the page data and the corresponding tag content.

[0136] Based on the above-mentioned device, optionally, the acquisition and processing method includes a protocol address changing method, and the page data determination module 630 further includes a decryption unit.

[0137] The decryption unit is used to decrypt the tag to be parsed and its corresponding tag content if the tag type of the tag to be parsed is encrypted, so as to obtain the decrypted tag to be parsed and its corresponding tag content.

[0138] Based on the above-mentioned device, optionally, the acquisition and processing method includes a protocol address changing method, and the page data determination module 630 includes a third data processing unit.

[0139] The third data processing unit is used to retrieve the Internet Protocol address of the client corresponding to the current address change cycle from a pre-created protocol address pool for each address change cycle, and delete the Internet Protocol address from the protocol address pool, so as to generate a data collection request based on the Internet Protocol address when obtaining page data of the corresponding subpage.

[0140] Optionally, based on the above-mentioned device, the third data processing unit is further configured to generate a protocol address to be stored based on the address generation component and store the protocol address to be stored in the protocol address pool when the number of protocol addresses to be used in the protocol address pool is less than a preset number.

[0141] The data acquisition device provided in the embodiments of the present invention can execute the data acquisition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0142] Example 7

[0143] Figure 7 This is a schematic diagram of the structure of an electronic device implementing the data acquisition method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0144] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0145] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0146] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data acquisition methods.

[0147] In some embodiments, the data acquisition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data acquisition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data acquisition method by any other suitable means (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0152] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0153] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0154] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0155] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data acquisition method, characterized in that, include: When a data collection request is received, at least one resource location information corresponding to the data collection request is determined; Determine the page type of at least one subpage in the page to be collected that corresponds to the current resource location information; Based on the page type and the corresponding acquisition and processing method, obtain the page data corresponding to the subpage; wherein, the acquisition and processing method includes resource location jump method, storage location change method and protocol address change method; Based on the page data corresponding to each subpage, determine the target data corresponding to the page to be collected; The data collection and processing method includes a resource location and redirection method. The method of obtaining page data corresponding to the corresponding subpage based on the page type includes: Determine the cumulative number of tags acquired after redirecting to the page to be collected corresponding to the current resource location information; If the cumulative number of tags exceeds the collection quantity threshold, the system will redirect to the page to be collected corresponding to the next resource location information. When the number of tags obtained from the page to be collected of the next resource location information exceeds the collection quantity threshold, the system will redirect to the historical collection page position of the current resource location information to continue obtaining tags. The cumulative number of tags will be re-determined based on the tags obtained, so as to obtain the page data of the corresponding subpage based on the obtained tags and the corresponding tag content.

2. The method according to claim 1, characterized in that, The step of determining the page type of at least one subpage in the page to be collected corresponding to the current resource location information includes: Redirecting to the page to be collected corresponding to the current resource location information; For each subpage in the page to be collected, when collecting the page data of the current subpage, the page format of the current subpage is determined, and based on the page format, the page type corresponding to the current subpage is determined; The page types include inner scrolling, outer scrolling, and page turning / switching types.

3. The method according to claim 2, characterized in that, The process of obtaining page data corresponding to the respective subpage based on the page type and the corresponding data collection method includes: If the page type is an inward scrolling type, then determine the scrolling attributes corresponding to the inward scrolling type; the scrolling attributes include scrolling period and scrolling displacement. Based on the current scroll cycle and scroll displacement, determine the current window page, and obtain at least one tag to be used and the corresponding tag content corresponding to the current window page; Based on at least one tag to be used and the corresponding tag content for each window page, determine the page data for the corresponding subpage.

4. The method according to claim 2, characterized in that, The process of obtaining page data corresponding to the respective subpage based on the page type and the corresponding data collection method includes: If the page type is an external scrolling type, then during the process of scrolling the scroll bar to the bottom of the corresponding subpage, at least one tag to be applied and the corresponding tag content are obtained. Based on each tag to be applied and its corresponding tag content, the page data of the corresponding subpage is determined.

5. The method according to claim 2, characterized in that, The process of obtaining page data corresponding to the respective subpage based on the page type and the corresponding data collection method includes: If the page type is a page-turning type, then the number of pages to turn is determined based on each page in the corresponding subpage, and the current page position is determined based on the number of pages to turn; Based on the current pagination position, switch from the previous pagination page to the current pagination page, and obtain at least one tag to be stored and the corresponding tag content corresponding to the current pagination page; Based on at least one tag to be stored and the corresponding tag content for each pagination page, the page data of the corresponding subpage is determined.

6. The method according to any one of claims 1, 3, 4 or 5, characterized in that, The data acquisition and processing method includes a storage location change method. The method of acquiring page data corresponding to the corresponding subpage based on the page type and the corresponding data acquisition and processing method includes: For each data acquisition cycle, the data storage space corresponding to the current data acquisition cycle is determined, and the page data corresponding to the sub-pages acquired in the current data acquisition cycle is stored in the data storage space; wherein, the data acquisition cycle is a duration determined based on a data acquisition duration threshold, and the data storage space corresponding to the current data acquisition cycle is different from the data storage space corresponding to the next data acquisition cycle; Establish a mapping relationship between at least one tag to be parsed in the page data and the corresponding tag content.

7. The method according to claim 6, characterized in that, Before establishing the mapping relationship between at least one tag to be parsed and its corresponding content in the page data, the method further includes: If the tag to be parsed is of encrypted type, then the tag to be parsed and the corresponding tag content are decrypted to obtain the decrypted tag to be parsed and the corresponding tag content.

8. The method according to any one of claims 1, 3, 4 or 5, characterized in that, The data collection and processing method includes a protocol address changing method. The method of obtaining page data corresponding to the corresponding subpage based on the page type and the corresponding data collection and processing method includes: For each address change cycle, the Internet Protocol address corresponding to the current address change cycle is retrieved from the pre-created protocol address pool, and the Internet Protocol address is deleted from the protocol address pool so that when obtaining page data of the corresponding subpage, a data collection request is generated based on the Internet Protocol address.

9. The method according to claim 8, characterized in that, Also includes: When the number of protocol addresses to be used in the protocol address pool is less than a preset number, a protocol address to be stored is generated based on the address generation component, and the protocol address to be stored is stored in the protocol address pool.

Images