Crawl device and program

Abstract

The present invention aims to perform website crawling with high efficiency. [Solution] A crawling device 101 that crawls target web pages on a network includes a page download unit 104 that downloads page data showing the content of the target web pages, a page processing unit 105 that analyzes the page data and creates page content showing the content of the target web pages, a page type determination unit 106 that determines the page type of the target web pages based on the page content, and a state management scheduler 103 that manages and creates different crawling schedules for the target web pages according to the page type of the target web pages. The web data collection unit 104 downloads the page data according to the crawling schedule.

Description

Technical Field

[0001] The present invention relates to crawling for collecting information such as web pages on the web.

Background Art

[0002] Currently, crawling in which crawlers traverse websites and collect their information has become widespread. The crawled information is used for search, AI learning, analysis, and the like. Here, the information on websites is updated daily. For this reason, it is necessary to collect the latest information in crawling. Accordingly, it is conceivable to increase the collection frequency so that the latest information can be collected.

[0003] However, increasing the information collection frequency leads to unnecessary collection. Accordingly, Patent Document 1 provides "a crawler management system and method capable of acquiring information corresponding to specific Web pages with high update frequency without modifying the crawler program itself". For this purpose, Patent Document 1 describes that "a plurality of crawlers (123a, 123b) read URLs stored in a plurality of queue storage units (112a, 112b) and acquire data of Web pages corresponding to the URLs. At this time, when a specific URL to be preferentially crawled is stored in a part (112a) of the plurality of queue storage units (112a, 112b), the specific URL is preferentially read from the part (112a) of the queue storage units".

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 requires manual registration of frequently updated web pages, making it impractical for crawling the vast number of web pages on the web. Therefore, the present invention aims to perform web page crawling with high efficiency. [Means for solving the problem]

[0006] To solve the above problems, the present invention creates a crawling schedule that indicates the crawling schedule according to the type of web page to be crawled. A more specific aspect of the present invention is a crawling device that performs crawling on target web pages on a network, comprising: a web data collection unit that collects page data indicating the content of the target web page; a page processing unit that analyzes the collected page data and creates page content indicating the content of the target web page; a page type determination unit that determines the type of the target web page based on the page content; and a state management scheduling unit that manages and creates different crawling schedules for the target web page according to the type of the target web page, wherein the web data collection unit is a crawling device that collects the page data according to the crawling schedule.

[0007] Furthermore, the present invention also includes a program that causes a computer to function as a crawling device, and a storage medium that stores this program. [Effects of the Invention]

[0008] This invention makes it possible to crawl websites with high efficiency. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing an example configuration of the crawling device 101 in one embodiment of the present invention. [Figure 2] A flowchart illustrating an example of the operation performed by the crawling device 101 in one embodiment of the present invention. [Figure 3]A diagram showing an example configuration of the state management scheduler 103 in one embodiment of the present invention. [Figure 4] A diagram showing another configuration example of the state management scheduler 103 in one embodiment of the present invention. [Figure 5] A diagram showing an example configuration of the page processing unit 105 in one embodiment of the present invention. [Figure 6] A diagram showing an example configuration of the page type determination unit 106 in one embodiment of the present invention. [Figure 7] A diagram showing an example of the data structure of a page storage database 107 in one embodiment of the present invention. [Figure 8A] A diagram showing an example of the hardware configuration of the crawling device 101 in one embodiment of the present invention. [Figure 8B] A diagram showing an application system using the crawling device 101 in one embodiment of the present invention. [Figure 9] A diagram showing an example of a user interface in a crawling device 101 in one embodiment of the present invention. [Modes for carrying out the invention]

[0010] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, elements having the same function will be given the same number, and redundant descriptions will be omitted. Furthermore, the following embodiments are just one way of carrying out the present invention and are not limited to these embodiments.

[0011] First, let's explain the basic terminology used in this specification. A crawler is a program used to collect web pages from the web; in other words, it's a program that performs crawling.

[0012] A typical use for web pages collected by crawlers is in search engines, which are used to assist users in accessing web pages on the web. Furthermore, crawled web pages can be used for AI training and analysis. It should be noted that collecting web pages only requires obtaining page data that at least shows the content and information of the web page; there is no prerequisite for obtaining other information as well.

[0013] A typical crawler first accesses a small number of seed URLs (a list of web page URLs that should be accessed first when performing a crawl), and then sequentially follows the hyperlinks (target URLs) contained in the accessed web pages. In this way, it crawls all the web pages on the web. This embodiment is based on a typical crawling procedure.

[0014] A web page is a document that exists on the web and can be viewed using a web browser. While HTML is the most common format for web pages, other file formats, such as document formats, also exist. For example, an HTML web page consists of HTML text (the core of the web page), stylesheets, scripts, and media files such as images, videos, and audio.

[0015] In addition to the constituent elements of a webpage, metadata such as the URL where the webpage exists, the format of the webpage (HTML or other document format, etc.), and the date and time the webpage was last modified can also be found.

[0016] In this specification, a web page refers to any document that can be viewed with a web browser, regardless of its format. Further, a web page is intended to encompass not only its constituent elements, but also the meta information of the web page and various additional information generated when processing a target web page in the present embodiment. The "main body of a web page" is an example of page data to be described later, and is original data that is downloaded and acquired from the web and subjected to various processes. The "text of a web page" is defined as a character string displayed when the web page is displayed by a web browser.

[0017] A URL (Uniform Resource Locator) is a sequence of symbols for identifying a resource on the web. In other words, it is like an address for accessing a web page. A hyperlink (link) is a mechanism for transitioning from one web page to another web page. Generally, links function using URLs.

[0018] The page type indicates a type determined based on the content of a web page, and is a concept different from the format (such as HTML) of the web page. In the present embodiment, web pages are roughly classified into two types: "index pages" and "content pages". Note that the index page and the content page are examples of the types of web pages, and other types may be added, or only other types may be used.

[0019] An index page is a page in which hyperlinks to various web pages are described as page data, for example, a page titled "News list for January 31, 2025". A content page is a page in which certain information is described as page data, such as news or a column, and is, for example, a page titled "Company RR announces new service".

[0020] Next, we will provide further explanation regarding the background and solution of this embodiment. On web pages, new pages are constantly being uploaded, and page data showing the content of existing pages is constantly being updated. Therefore, in order to reflect the latest page data on web pages, it is necessary to access the web frequently. However, since the number of pages on the web is enormous, some kind of ingenuity is needed to efficiently perform comprehensive crawling.

[0021] As described above, index pages, by their nature, often contain hyperlinks to newly uploaded web pages as part of their page data. Therefore, the page data of index pages is updated frequently. On the other hand, content pages, compared to index pages, are updated less frequently once they are uploaded.

[0022] Therefore, by accessing index pages frequently (at short intervals) and content pages less frequently (at long intervals), it becomes possible to quickly access newly uploaded pages on the web and perform crawling efficiently. Furthermore, content pages have fewer hyperlinks to other web pages compared to index pages, and this characteristic can be used to classify them.

[0023] The configuration of this embodiment will now be described. Figure 1 is a diagram showing an example of the configuration of the crawling device 101 in this embodiment. The crawling device 101 is a device that performs crawling. Therefore, in Figure 1, the crawling device 101 consists of a control unit 102, a state management scheduler 103, a page download unit 104, a page processing unit 105, a page type determination unit 106, and a page storage database 107. Note that the crawling device 101 connects to a network such as the Internet via a communication unit in order to access the Web, but the explanation of this is omitted in Figure 1.

[0024] Furthermore, the crawl device 101 accepts a seed URL and various settings as input. For this reason, the crawl device 101 may also be provided with an input unit that accepts seed URLs and settings, and an output unit that outputs crawl information, etc. The various settings are setting values ​​required when the crawl device 101 operates, and may be received from an interface such as the crawl device user interface 901 in Figure 9. This will be explained later.

[0025] Furthermore, the seed URL and various settings may be accepted as input from the user, or default values ​​predetermined within the system may be used. Here, the seed URL is an example of web page identification information, and other information may be used. Instead of accepting the seed URL as input, the system may accept the state management scheduler 103 and page storage database 107 of another crawler device 101 and load them into the crawler device 101. In other words, crawling may not start from an initial state using the seed URL as the starting point, but may resume from the internal state of another crawler device 101 that is already running.

[0026] Furthermore, the control unit 102 communicates with each component of the crawling device 101 and manages the overall operation of the crawling device 101. In addition, the state management scheduler 103 manages the target web pages that the crawling device 101 should access. To this end, the state management scheduler 103 creates and updates crawling schedules. In this process, the state management scheduler 103 creates different crawling schedules depending on the type of target web page. The state management scheduler 103 can also be referred to as the state management scheduling unit.

[0027] This crawling schedule can include the date, time, frequency, and period of crawling. The state management scheduler 103 also stores the seed URL and its schedule in its memory. The crawling schedule can also be submitted by the user as one of the settings.

[0028] Furthermore, the schedule of the target URL identified by the analysis in the page processing unit 105 is stored in the storage unit. This storage unit may be located in the crawling device 101 or in an external device.

[0029] Furthermore, the page download unit 104 downloads the page data of the target webpage from the web. To do this, the page download unit 104 accepts the URL of the webpage to be accessed as input, accesses the target webpage using the URL, and downloads its page data. Here, the page download unit 104 is an example of a web data collection unit that collects page data of the target webpage.

[0030] Furthermore, the page processing unit 105 analyzes and processes the page data downloaded by the page download unit 104 to identify the page elements obtained. These page elements are used in various processes of the crawling device 101 and include their URLs and URLs (target URLs) contained in the page data.

[0031] The page processing unit 105 will perform tasks such as creating page content, including page data and page elements. Therefore, in this embodiment, the page content of a web page includes not only the web page itself but also the page elements obtained by the page processing unit 105. However, the page content may consist of at least one of the page data and page elements.

[0032] Furthermore, the page type determination unit 106 determines the type of the target web page based on the page content identified by the page processing unit 105.

[0033] Furthermore, the page storage database 107 is a database that stores web page-related information such as target URLs, page data, page content, and crawling schedules collected and processed by the crawling device 101. It is preferable that the page storage database 107 be stored in the storage unit mentioned above. In addition, the web page-related information only needs to include at least one of the following: page data, page content, and crawling schedule.

[0034] Next, the processing flow in this embodiment will be described. Figure 2 is a flowchart showing an example of the operation performed by the crawling device 101 in this embodiment. Note that only the overall operation is shown here, and details will be supplemented in the explanations of each figure.

[0035] Step S201: The control unit 102 outputs the seed URL and various settings received by the crawling device 101 to the state management scheduler 103. Based on this, the state management scheduler 103 creates a crawling schedule for the web page identified by the seed URL, based on the seed URL and various settings. The control unit 102 or the state management scheduler 103 stores the seed URL and crawling schedule in the page storage database 107.

[0036] Step S202: The page download unit 104 obtains the URL of the target webpage, which is the next webpage to be accessed. Then, the page download unit 104 accesses the URL of the target webpage and downloads the latest page data for the target webpage.

[0037] Step S203: The page processing unit 105 analyzes the page data of the target web page and identifies its page elements. The page processing unit 105 then creates the page content of the target web page, including the page elements and the corresponding page data.

[0038] Step S204: The page type determination unit 106 determines whether the type of the target web page has already been determined based on the created page content. If the result is "Not determined" (No), proceed to step S205. If the type of the target web page is anything other than "Not determined," that is, if the page type of the target web page has already been determined (Yes), proceed to step S206.

[0039] Step S205: The page type determination unit 106 determines the page type of the target web page based on the page content created in step S203. The page type determination unit 106 then outputs the determined page type to the page download unit 104.

[0040] Step S206: The page type determination unit 106 saves the page type and page content of the target web page to the page storage database 107. The page type determination unit 106 also outputs the page content of the target web page to the state management scheduler 103. In this case, the page type determination unit 106 may also be configured to output the target URL and page type included in the page content to the state management scheduler 103.

[0041] Step S207: The state management scheduler 103 creates a crawling schedule for the target web page indicated by the target URL, based on the target URL and page type of the purged content output in step S206. For example, the state management scheduler 103 determines the access timing so that content pages are accessed more frequently than index pages.

[0042] In this way, the state management scheduler 103 creates a crawling schedule that prioritizes crawling of page types estimated to be frequently updated over crawling of page types estimated to be infrequently updated. In other words, the state management scheduler 103 creates a crawling schedule that corresponds to the update information indicating the update frequency of web pages.

[0043] Note that there may be three or more page types. In this case, a crawling schedule may be set for each page type, or some page types may be crawled at the same frequency. In the latter case, even if the frequency is the same, it is desirable to stagger the access timings.

[0044] Then, returning to step S202, the page download unit 104 accesses the URL included in the page data of the target webpage and downloads the latest information of that page data.

[0045] In this way, the crawler device 101 accesses the URLs included in the web page data one after another, downloads the page data, and continuously identifies the page type and creates a crawling schedule. For this reason, as can be seen from the flowchart in Figure 2, the crawler device 101 does not have an explicit termination process. This is because the crawler device 101 is designed to operate semi-permanently in order to continuously collect web pages from the web.

[0046] However, the crawl device 101 may be temporarily paused or forcibly terminated by external operations, such as by pressing the "Pause" button on the user interface, or by pressing the "End" button.

[0047] Furthermore, since the crawling device 101 does not have an explicit termination process, it does not need to output anything. However, the actual output of the crawling device 101 is the page storage database 107, which is updated continuously, so it can be retrieved at any time. In other words, a user or an external system can access and retrieve the page storage database 107 while the crawling device 101 is running, interrupted, or terminated.

[0048] Furthermore, if a user interface such as the crawler user interface 901 in Figure 9 is provided, the operating status of the crawler 101 may be displayed on the crawler user interface 901 as an output of the crawler 101. This operating status may include the crawling schedule for each target web page. By doing so, in this embodiment, crawling can be made more efficient by classifying the page type of each web page or its page data.

[0049] Next, the details of each component and operation of the crawling device 101 will be described. Figure 3 is a diagram showing an example of the configuration of the state management scheduler 103 in this embodiment. The state management scheduler 103 includes a state management scheduler control unit 301 that controls the entire system, an access timing determination model 302 that determines the next access timing for the target web page, and a web page storage queue 303.

[0050] Furthermore, it is desirable that the access timing determination model 302 uses the page type of the target web page to determine the next access timing for that web page. For this purpose, the access timing determination model 302 uses the page type determined by the page type determination unit 106.

[0051] For determining the access timing, for example, if the type of web page is undetermined, the access timing may be set to 1 second from the current date and time; if it is an index page, it may be set to 1 day from the current date and time; and if the type of web page is a content page, it may be set to 1 month from the current date and time.

[0052] The "access timing" output by the access timing determination model 302 is not necessarily a relative value such as "one day from now," but can also be an absolute value such as "February 1, 2025, 15:00." If an absolute value is to be output, the current date and time may be provided as additional input to the access timing determination model 302. Furthermore, an access cycle of 0:00 every day may also be used as the "access timing." As a result, the state management scheduler 103 will create an access timing, which is an example of a crawling schedule. Here, other timings, such as obtaining a seed URL, may also be used as the crawling schedule.

[0053] The access timing determination model 302 may determine the access timing using, in addition to or instead of, the page type of the target web page, whether or not the web page has been updated. For example, for the first access, the access timing determination model 302 determines the access timing using only the page type, as shown above.

[0054] The access timing determination model 302 then changes the access timing for subsequent accesses depending on whether the web page has been updated since the previous access. For example, if the page has been updated, the time until the next access timing may be halved, and if it has not been updated, the time until the next access timing may be doubled.

[0055] Furthermore, access timing may be determined not only by whether or not a webpage has been updated, but also by the amount of changes made to the target webpage (what percentage of the original webpage's text has been changed) and the number of hyperlinks added to the webpage.

[0056] Furthermore, in addition to the type of web page, the access timing may also be determined using the URL string of the target web page or a string written on the linking web page to the target web page to provide an overview of the target web page. This string may include, in the case of an HTML web page, Tags and It contains text enclosed in tags.

[0057] Furthermore, in addition to the types of web pages, it is also possible to pre-configure preferred websites and web pages to be accessed, thereby shortening the access intervals for those sites and pages. The access timing determination model 302 may, for example, be an AI model that has been pre-trained for the purpose of determining access timing.

[0058] In other words, by inputting the current date and time and the target webpage, the AI ​​model can be pre-trained to determine the date and time when the target webpage is most likely to be updated next, and this pre-trained AI model can be used as the access timing determination model 302. In these cases as well, the access timing can be expressed using an absolute value such as "February 1, 2025, 15:00," a relative value such as "one day from now," or a periodic value such as "every day at 0:00."

[0059] The access timing determination model 302 may be a pre-trained AI model, such as a Large Language Model (LLM) pre-trained for general-purpose language capabilities. An LLM is an AI model trained on large datasets to output appropriate text for given input. When using an LLM, for example, by providing the current date and time and information about the target webpage as input, it is possible to predict the optimal access timing for the target webpage without performing specialized training for access timing determination. Alternatively, a general-purpose LLM that has been further trained for optimal access timing determination may be used as the access timing determination model 302.

[0060] The access timing determination model 302 may be a combination of the methods described above. For example, the access timing may be determined using each of the above methods, and the final determination may be made by majority vote or calculation of the average value. Note that the access timing determination model 302 does not need to be a model; it only needs to be able to create a crawling schedule, in particular, to determine the access timing. Thus, the access timing determination model 302 is an example of a crawling schedule unit or an access timing determination unit. Furthermore, the state management scheduler 103 can also be described as a state management schedule unit.

[0061] Furthermore, the web page storage queue 303 holds a list of web pages that the crawler 101 should access. The state management scheduler control unit 301 then sorts (rearranges) the web pages in the list in order of their next access timing whenever a new web page is stored in the web page storage queue 303.

[0062] Furthermore, when the state management scheduler control unit 301 retrieves information (or a web page) indicating the next web page to be accessed, it retrieves the web page at the very top of the web page storage queue 303, as this is the web page that has been accessed the earliest. At this time, the information of the retrieved web page is temporarily deleted from the web page storage queue 303.

[0063] Furthermore, the webpage storage queue 303 stores, for example, a list of each webpage as an example of information indicating the webpages to be collected. This list stores the webpage's URL, page type, and the next access timing. However, the list may only store the URL in the webpage storage queue 303, while other necessary information is stored in the page storage database 107 and read and used when needed.

[0064] In step S202 of Figure 2, the page download unit 104 accesses the state management scheduler control unit 301 according to the control unit 102. The page download unit 104 then retrieves the next web page to be accessed from the web page storage queue 303. The retrieved web page is temporarily deleted from the web page storage queue 303. Upon receiving this, the page download unit 104 accesses the web page indicated by the URL of the retrieved web page at the appropriate access timing and downloads the page data.

[0065] Furthermore, in step S207 of Figure 2, based on the page type determined by the page type determination unit 106, the access timing determination model 302 determines the next access timing for the target web page and stores it in the web page storage queue 303.

[0066] Furthermore, in steps S201 and S207 of Figure 2, the state management scheduler 103 receives the seed URL and the URL of the target web page. Then, in step S204, the page type determination unit 106 checks whether the web page indicated by that URL is already stored in the web page storage queue 303.

[0067] The page type determination unit 106 determines the page type of the target web page as "undetermined" if these URLs do not exist in the web page storage queue 303, that is, if the target web page is a new web page that has not been accessed before. Then, in step S205, the page type determination unit 106 determines the page type based on the page content created by the page processing unit 105.

[0068] In response, the access timing determination model 302 determines the next access timing based on the determined page type and adds it to the web page storage queue 303.

[0069] Furthermore, the page type determination unit 106 determines the page type of the target web page as "determined" if the URL of the target web page exists in the web page storage queue 303, that is, if it has already been accessed and is a web page managed by the state management scheduler 103. As a result, the page type determination is skipped, and the target web page is not added to the web page storage queue 303.

[0070] You can exclude a webpage from being crawled by not adding it to the webpage storage queue 303. For example, a webpage that is unlikely to be updated in the future does not need to be added to the webpage storage queue 303 because there is little need to access it again.

[0071] Figure 4 shows another configuration example of the state management scheduler 103 in this embodiment. Compared to Figure 3, the state management scheduler 103 has the addition of a new page storage queue 401, an index page storage queue 402, and a content page storage queue 403, while the web page storage queue 303 is omitted.

[0072] In this configuration example, web pages whose page type is "undetermined," that is, new pages that the crawler 101 has not yet accessed, are stored in the new page storage queue 401. Additionally, information indicating web pages whose page type is "index pages" is stored in the index page storage queue 402, and information indicating web pages whose page type is "content pages" is stored in the content page storage queue 403. The following describes the details of the characteristic processing using this configuration.

[0073] First, in step S202, when obtaining the next web page to access, the page download unit 104 checks, for example, the new page storage queue 401 first. Then, if the access time of the web page at the top of the queue (the web page with the earliest access time) exceeds the current date and time, the page download unit 104 outputs that web page as the next web page to access.

[0074] If the current date and time have not been exceeded, the page download unit 104 then checks the index page storage queue 402. If the access timing of the web page at the top of the queue has exceeded the current date and time, the page download unit 104 outputs that web page as the next web page to be accessed.

[0075] Furthermore, if the current date and time have not been exceeded, the page download unit 104 outputs the first web page in the content page storage queue 403 as the next web page to be accessed.

[0076] The advantages of using the configuration example in Figure 4 are as follows. In the configuration example in Figure 3, different page types are managed in a single queue, but this may not function properly if the number of web pages to be managed increases. For example, if the number of content pages managed by the state management scheduler 103 increases more than expected, and the processing speed of the crawler 101 cannot keep up, it may become impossible to access the index page, which should retrieve the latest state of web pages at a high frequency, at the expected frequency.

[0077] To prevent this, managing each page type with a different queue, as shown in the configuration example in Figure 4, makes it possible to access each web page at the expected frequency, even if the total number of web pages to be managed increases.

[0078] Next, the details of the page processing unit 105 will be described. Figure 5 shows an example of the configuration of the page processing unit 105 in this embodiment. The page processing unit 105 has a page processing control unit 501 that controls the whole system and a page processing model 502 that determines the type of page to be analyzed and processed.

[0079] The page processing model 502 analyzes the page data of the target web page downloaded by the page download unit 104. The results of this analysis are used by various components of the crawling device 101, such as the state management scheduler 103 and the page type determination unit 106.

[0080] Specifically, the page processing model 502 analyzes page data to extract necessary page elements and processes the extracted page elements to create new page elements. These created page elements include, for example, the web page title, body text, a summary of the extracted body text, and a screenshot of the page (an image representing visual information that is displayed when the target web page is opened in a browser, etc.).

[0081] The page processing model 502 may be, for example, a rule-based model built manually. Alternatively, the page processing model 502 may be, for example, an AI model that has been pre-trained for page processing purposes. In other words, an AI model may be pre-trained to output each page element necessary for subsequent processing when a web page is input, and the resulting AI model may be used as the page processing model 502.

[0082] Furthermore, the page processing model 502 may be, for example, an LLM pre-trained for general-purpose use. When using an LLM, by providing a web page as input, the necessary page elements can be created without performing training specifically for web page processing. Alternatively, a general-purpose LLM that has been further trained for page processing purposes may be used as the page processing model 502.

[0083] Furthermore, page processing model 502 may be a combination of the methods described above. For example, the title extraction and body text extraction processes may be rule-based, while the body text summarization may be performed using LLM.

[0084] In this embodiment, the page processing required by the crawling device 101 is performed collectively by the page processing unit 105, but instead, page processing may be performed individually in each component such as the state management scheduler 103 and the page type determination unit 106.

[0085] Next, the details of the page type determination unit 106 will be described. Figure 6 is a diagram showing an example of the configuration of the page type determination unit 106 in this embodiment. The page type determination unit 106 has a page type determination control unit 601 that controls the whole system and a page type determination model 602 that determines the type of page.

[0086] The page type determination model 602 determines the page type of the target web page. In this embodiment, each web page is classified into one of two types: "index page" or "content page," but it is sufficient to classify it into a type according to its update frequency, and it may be classified into other types as well.

[0087] To this end, the page type determination model 602 accepts each page element of the target web page as input. For example, the page type determination model 602 determines the page type based on the title and body of the web page. For example, in addition to the above, the page type may also be determined based on the HTML text of the web page, screenshots, page format, etc.

[0088] For example, a summary of the main text may be used instead of the main text. These page elements may use the analysis results from the page processing unit 105, or they may use the input data to the crawling device 101.

[0089] Furthermore, the page type determination model 602 may be, for example, a rule-based determination model constructed manually. The page type determination model 602 may also be, for example, an AI model that has been pre-trained for the purpose of page type determination. In other words, the page type determination model 602 may be pre-trained to output a prediction result of the type of page element or, in place of, page data of the target web page, and the AI ​​model obtained in this way may be used as the page type determination model 602.

[0090] Furthermore, the page type determination model 602 may be, for example, an LLM pre-trained for general-purpose use. When using an LLM, by providing a list of each page element and the names and definitions of each candidate page type as input, it is possible to determine the page type to which the target web page belongs from the received list of page types without performing specialized training for page type determination. Alternatively, a model in which a general-purpose LLM has been further trained for page type determination may be used as the page type determination model 503.

[0091] Furthermore, the page type determination model 602 may be a combination of the methods described above. For example, the page type may be determined using each of the above methods, and the final determination may be made by majority vote.

[0092] Next, the details of the page storage database 107 will be described. Figure 7 shows an example of the data structure of the page storage database 107 in this embodiment. Each record (row in the database) in this database stores web page-related information for one web page.

[0093] For example, as shown in Figure 7, the URL, page type, site name, and title of each webpage may be saved. Other items that may be saved in the page storage database 107 include, for example, the webpage's body text, binary data, a summary of the body text, media files, the most recent date and time of access to the target webpage (latest access date and time), and the webpage format. In this way, webpage-related information can include page data or page content.

[0094] Furthermore, the items stored in the page storage database 107 may also be page elements obtained by the page processing unit 105. Large amounts of web page-related information, such as the web page itself or its body text, may not be directly stored in the page storage database 107, but rather stored as separate files in the auxiliary recording device 703, and the file path on the auxiliary recording device 703 may then be stored in the database. Moreover, it is desirable to include a crawling schedule, such as access timing, in the web page-related information. However, since the crawling schedule is determined according to the specified page type, the page type itself in the web page-related information may be used as the crawling schedule.

[0095] Next, we will describe an example of implementing the crawling device 101. The crawling device 101 can be implemented using a computer. Below, we will describe the basic configuration of the computer-implemented crawling device 101 (Figure 8A) and its application system (Figure 8B).

[0096] First, Figure 8A shows an example of the hardware configuration of the crawl device 101 in this embodiment. In Figure 8A, the crawl device 101 consists of a processor 801, memory 802, auxiliary storage device 803, input device 804, output device 805, and communication device 806. This is just one example of a hardware configuration, and other configurations capable of operating the crawl device 101 may be used. For example, the input device 804 and output device 805 correspond to the input and output units described above. Furthermore, these can be integrated using a touch panel. Also, these components can be omitted.

[0097] Typical operations of the crawler 101 are implemented by the control unit 102, state management scheduler 103, page download unit 104, page processing unit 105, and page type determination unit 106, which are implemented by corresponding programs. When the crawler 101 is started, the program is loaded into memory 802, and then each function is operated by the processor 801 at the timing when it is used.

[0098] Furthermore, the page download unit 104 downloads page data via the communication device 806. The page storage database 107 is stored in the auxiliary storage device 803. The communication device 806 corresponds to the communication unit mentioned above.

[0099] Next, a more specific configuration of the crawling device 101 and its application system will be described. Figure 8B is a diagram showing an application system using the crawling device 101 in this embodiment. In Figure 8B, the crawling device 101 is implemented as a so-called server and connected to a terminal device 2 used by the user. Also in Figure 8B, an analysis system is shown as an application system that searches and analyzes the crawled page data.

[0100] For this purpose, in Figure 8B, the crawling device 101 is connected to the terminal device 2 and the data analysis device 3 via the network 4, forming a data analysis system. First, the crawling device 101 will be described.

[0101] In Figure 8B, the crawl device 101 consists of a processor 801, memory 802, auxiliary storage device 803, and communication device 806. Therefore, compared to Figure 8A, the input device 804 and output device 805 are omitted. These functions are provided in the terminal device 2, which is implemented by a computer.

[0102] First, let's explain the configuration of the crawling device 101. The processor 801 can be implemented as a CPU or the like, and it performs calculations according to the crawler 807, that is, the program, stored in the auxiliary storage device 803, which will be described later. The communication device 806 connects to each device, such as the terminal device 2 and the data analysis device 3, via the network 4. Note that in Figure 8B, only one terminal device 2 and one data analysis device 3 are shown, but there may be multiple units.

[0103] Network 4 may be implemented using a combination of multiple networks. For example, the crawling device 101 and the terminal device 2 may be connected via a dedicated line such as an intranet, while other devices may be connected via the internet or similar.

[0104] The communication device 806 can receive seed URLs and various settings from the terminal device 2. Furthermore, the communication device 806 outputs the crawling results, downloaded page data, page elements, etc., to the terminal device 2 and the data analysis device 3.

[0105] Furthermore, the memory 802 and auxiliary storage device 803 can be implemented using a storage medium and correspond to a storage unit that stores the page storage database 107. The storage unit, particularly the auxiliary storage device 803, may also store downloaded page data.

[0106] Then, the memory 802 loads the information used for processing by the crawler 807 and processor 801, which is stored in the auxiliary storage device 803. The auxiliary storage device 803 can be implemented using so-called storage.

[0107] The auxiliary storage device 803 stores the crawler 807 and the page storage database 107 mentioned above.

[0108] Furthermore, the auxiliary storage device 803 may be implemented using various storage media such as an external HDD (Hard Disk Drive), SSD (Solid State Drive), or main memory card, or it may be implemented as a separate device from the crawling device 101, such as a file server.

[0109] Here, the processor 801 executes the processing of the control unit 102, the state management scheduler 103, the page download unit 104, the page processing unit 105, and the page type determination unit 106 according to the crawler 807.

[0110] For this purpose, the crawler 807 includes a control module 808, a state management module 809, a page download module 810, a page processing module 811, and a page type determination module 812.

[0111] Here, the configuration shown in Figure 1, which performs the same function as each module, is as follows: Control Unit 102: Control Module 808 State Management Scheduler 103: State Management Module 809 Page Download Section 104: Page Download Module 810 Page processing unit 105: Page processing module 811 Page type determination unit 106: Page type determination module 812 Each of these modules may be implemented as an independent program or as multiple programs. Furthermore, the crawling device 101 may be equipped with a data analysis unit as a function of the data analysis device 3.

[0112] Next, terminal device 2 can be implemented using various computers such as tablets, smartphones, or PCs operated by the user. Terminal device 2 has the functions of the input device 804 and output device 805 shown in Figure 8A. Therefore, terminal device 2 has an input device such as a keyboard that accepts input from the user, and a display device such as a display monitor. Note that the input device and the display device may be integrated into a single unit, such as a touch panel.

[0113] Next, in the application system shown in Figure 8B, the data analysis device 3 analyzes the page data downloaded by the crawling device 101 and provides the results. For example, it analyzes market trends. To do this, the data analysis device 3 performs data analysis using the page data.

[0114] For this purpose, the data analysis device 3 can be implemented using a computer such as a server. The data analysis device 3 then acquires page data of a predetermined data type from the crawling device 101. It is preferable to use content data as the page data of the predetermined data type.

[0115] Furthermore, the data analysis device 3 may use data elements as the object of analysis. The data analysis device 3 then transmits the analysis results to various devices such as the terminal device 2, allowing the analysis results to be confirmed. In this way, according to the data analysis system shown in Figure 8B, by assigning page types, useful data can be provided for application systems such as analysis.

[0116] Furthermore, the application system can be applied not only to analysis but also to search engines and training data. In an application system like the one shown in Figure 8B, it becomes possible to create datasets using only the necessary types of pages tailored to the purpose, such as specific page data like "content pages" or "technology," and to search by limiting the page types.

[0117] Next, the user interface in this embodiment will be described. The user interface described here, that is, the display screen, is displayed on the output device 805 in Figure 8A and the terminal device 2 in Figure 8B. Figure 9 is a diagram showing an example of the user interface of the crawl device 101 in this embodiment.

[0118] In the example shown in Figure 9, the crawler user interface 901 has a "page-specific settings" area, an "access interval determination rule" area, and a "crawl information" area. First, the "page-specific settings" area and the "access interval determination rule" area are areas that show the various settings accepted by the crawler 101.

[0119] First, the "Page-Specific Settings" area has a "Page Type" field, a "Definition" field, and a "Basic Access Interval." The "Page Type" field has "Index Page" and "Content Page" set as page types, and new ones can be added.

[0120] Furthermore, the "Definition" section shows the definition of the corresponding page type. In the example in Figure 9, an "index page" is defined as a page containing links to other web pages, i.e., a "URL". Similarly, a web page containing news or columns is defined as a "content page," but these definitions are not limited to those exemplified.

[0121] For example, an "index page" could be defined as a page containing a predetermined number of URLs, and a "content page" as a page containing fewer than a predetermined number of URLs, depending on the "URL" that indicates the destination of the link. The "basic access interval" item indicates the access timing, and in the example in Figure 9, it is set to "1 day" for "content pages," which is a higher frequency than for "index pages." Note that in Figure 9, the access interval is used for the "basic access interval" item, but it is not limited to using dates or other values.

[0122] Furthermore, the "Access Interval Determination Rule" area shows the elements used to determine the access interval, or access timing. In the example in Figure 9, there are "Page Update Status" and "URL String" items. These indicate that the web page update status is used to determine the access interval, and the URL string is not used.

[0123] Furthermore, at the bottom of the "Access Interval Determination Rule" area in this embodiment, operation buttons are provided. These buttons allow the user to instruct whether to start crawling based on the settings shown in the "Page-by-Page Settings" area and the "Access Interval Determination Rule" area. For this reason, the buttons include a "Start" button and a "Pause" button. When the "Start" button is pressed, crawling will be performed with the settings shown in Figure 9. In other words, these are input to the crawling device 101 as various settings.

[0124] Furthermore, the items in the "Access Interval Determination Rules" area and the "Page-by-Page Settings" area may be entered by the user or created by the crawler 101.

[0125] In the former case, the "Access Interval Determination Rule" area may accept user input for each page type, including its name (page type item), definition, and basic access interval.

[0126] The crawling device 101 assumes two basic page types: index pages and content pages. However, users may be asked to input additional page types. The names and definitions of each page type entered here can then be provided as input to the page type determination model 602 to determine which page type each page belongs to.

[0127] For example, the crawler user interface 901 may accept information from the user regarding elements used to determine the access interval for each page (e.g., whether the page has been updated as shown in the diagram, and the URL string). Furthermore, the items in the "Access Interval Determination Rules" area and the "Page-Specific Settings" area may not only be configured through the user interface, but also by the user editing a separately prepared configuration file.

[0128] For example, the crawler 101 may be started by the user setting each condition and then pressing the "Start" button. For example, the operation of the crawler 101 may be temporarily paused by the user pressing the "Pause" button while the crawler 101 is running.

[0129] As another example, the operating status of the crawler 101, such as the total number of collected web pages of each type, or the number of new pages collected on the current day and the previous day, may be displayed on the crawler user interface 901.

[0130] In this embodiment, the crawling device 101 has one of each component. Alternatively, one crawling device 101 may have multiple of each component, thereby enabling parallel processing. For example, by having multiple page download units 104, web page downloads may be performed in parallel. Such parallel processing can be expected to speed up the overall processing.

[0131] In this embodiment, web pages on the web are collected comprehensively without specifying a website. Alternatively, the system may accept the target websites as additional input and crawl only those websites. In this case, multiple websites may be accepted as input.

[0132] This concludes the description of this embodiment, but the present invention is not limited thereto. For example, the crawling device 101 may be implemented using multiple devices. Furthermore, the page types are not limited to those with different update frequencies, such as index pages and content pages, but may be defined according to the update timing. In addition, index pages and content pages may be classified by factors other than the presence or absence of a URL, or by the size of the web page or the proportion of images. [Explanation of Symbols]

[0133] 101...Crawl device, 102...Control unit, 103...Status management scheduler, 104...Page download unit, 105...Page processing unit, 106...Page type determination unit, 107...Page storage database

Claims

1. In a crawling device that crawls target web pages on a network, A web data collection unit that collects page data showing the content of the aforementioned target web page, A page processing unit analyzes the collected page data and creates page content that shows the content of the target web page, A page type determination unit determines the page type of the target web page based on the page content, The system includes a state management scheduling unit that creates different crawling schedules for each target webpage depending on the page type of the target webpage. The aforementioned web data collection unit is a crawling device that collects the page data according to the crawling schedule.

2. A crawling device according to claim 1, The aforementioned page type determination unit is a crawling device that classifies pages into index pages and content pages according to the URL of the linked page.

3. A crawling device according to claim 2, The state management scheduling unit is a crawling device that creates a crawling schedule in which the timing of access to the content page is higher than that of the index page.

4. A crawling device according to claim 1, The state management scheduling unit retrieves information indicating the next web page to be collected from the web page storage queue, The web data collection unit accesses the target web page and downloads the page data based on the extracted information. The page type determination unit determines the page type of the accessed target web page if it has not yet been determined. The state management scheduling unit is a crawling device that creates a crawling schedule for the accessed target web page according to the determined page type.

5. A crawling device according to claim 4, The aforementioned webpage storage queue stores the URLs contained in the downloaded page data as information indicating the next webpage to be collected. The aforementioned web data collection unit is a crawling device that accesses the target web page linked by the URL stored in the web page storage queue and downloads the page data.

6. A crawling device according to claim 1, The state management scheduling unit is, A state management scheduler control unit that controls the operation of the state management scheduling unit, An access timing determination model for determining the access timing of the aforementioned target web page, A new page storage queue for saving newly arrived web pages, An index page storage queue that stores information indicating a web page whose page type is an index page, A crawler having a content page storage queue that stores information indicating page data for a content page, where the aforementioned page type is a content page.

7. A crawling device according to claim 1, The aforementioned state management scheduling unit, A crawling device comprising a trained AI model, an LLM, and an access timing determination model that determines the access timing by either a combination of these, or by further training the LLM to determine the access timing for collecting the page data.

8. A crawling device according to claim 1, The aforementioned page processing unit is A crawling device comprising a page processing model that analyzes the page data using one of the following: a rule-based model, a trained AI model, an LLM, and a model that has been further trained to analyze the page data, or a combination thereof.

9. A crawling device according to claim 1, The aforementioned page type determination unit, A crawling device comprising a page type determination model that determines the page type by any one of the following: a rule-based model, a trained AI model, an LLM, and a model that has been further trained to determine the page type, or a combination thereof.

10. A crawling device according to claim 1, The aforementioned page type determination unit, A crawling device comprising a page type determination model that determines the page type to which the target web page belongs from a list of names of page types that are candidates for classification and definitions of said page types.

11. A crawling device according to claim 1, Furthermore, the crawling device has an output unit that outputs a page-specific settings area, an access interval determination rule area, and a crawl information area for the target web page.

12. A crawling device according to claim 11, The crawl information area is a crawl device that displays crawl information in response to user input for the page-specific settings area and the access interval determination rule area.

13. A crawling device according to claim 2, Furthermore, a crawling device having a data analysis unit that analyzes the page data of the content pages among the page data.

14. A computer that acts as a crawler, performing crawling on target web pages on a network, A web data collection unit that collects page data showing the content of the aforementioned target web page, A page processing unit analyzes the collected page data and creates page content that shows the content of the target web page, A page type determination unit determines the page type of the target web page based on the page content, It functions as a state management scheduling unit that creates and manages different crawling schedules for each of the target web pages, depending on the page type of the target web page. Furthermore, a program that causes the web data collection unit to collect the page data according to the crawling schedule.

Images