Information collection system and information collection method

Abstract

To continuously collect information from a website that structurally changes as an information collection target.SOLUTION: An information collection system 10 includes: a collection unit 100 which analyzes a structure of a target website for information collection, stores a scenario with locations of information to be collected, and continues collecting information from the target website on the basis of the scenario; a detection unit 200 which prepares a plurality of modules for searching for locations of information to be collected and sorts, when detecting that the collection unit 100 cannot continue collecting the information, a method for searching for the locations set in the scenario, from the multiple modules; a search unit 300 which searches for the locations of information to be collected from the target website, on the basis of the module sorted by the detection unit 200; and a repair unit 400 which repairs the scenario by modifying the locations of information to be collected, on the basis of a result searched by the search unit 300.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information collection system and an information collection method. [Background technology]

[0002] In recent years, in the field of Open Source Investigation (OSINT), which involves collecting and analyzing public information, information on the Internet has become more important than data sources such as paper media and public broadcasting.Compared to traditional media, information on the Internet can be easily made public by anyone, and the amount of information it contains is large, making it essential to select and discard information.

[0003] Web crawlers and web scraping are technical fields that select and collect necessary information from the Internet. A web crawler is a technique that obtains information from a web page by accessing the page. Web scraping is a technique that obtains only the necessary information from a web page by specifying a specific area of ​​the website, and is used in conjunction with a web crawler.

[0004] This technology for acquiring information on the internet is used in a wide range of fields, including trend analysis, information curation, real-time inventory checks, etc. In the OSINT field, it is important to improve the accuracy of information analysis by mechanically collecting a wide range of information.

[0005] It is rare for large-scale platform sites such as SNS (Social Network Service) to provide APIs, as do news sites and personal websites. Depending on the user's areas of interest, any website can be a target for information collection, so there is still a high demand for web crawlers and web scraping that do not use APIs.

[0006] Patent Document 1 discloses a technology that enables a cross-sectional search for product information from a group of products sold across multiple shopping sites by analyzing the structure of the shopping sites.

[0007] Patent Document 2 discloses a technology in which a scenario is prepared that specifies actions such as clicking a button on a specific area of ​​a website to be tested, and when testing with that scenario, if there are any changes between the previous and current test results, the procedure including the changed elements is reflected in the scenario. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-243126 [Patent Document 2] WO2022 / 070353 publication Summary of the Invention [Problem to be solved by the invention]

[0009] Since the unspecified number of websites targeted for information collection are operated by organizations or individuals different from the person conducting the information collection, there are many cases where website changes occur unintentionally by the person conducting the information collection. With web scraping technology, which requires specifying a specific area, if the structure of the website changes, there is a high possibility that pre-set identifiers will also change, making it impossible to collect information as is. In cases where web scraping is performed after the website being targeted for information collection has changed, it is necessary to re-analyze the website and re-specify the specific area, which is time-consuming.

[0010] When the information to be collected changes with each access, such as news titles or the latest information on a timeline, it is often the case that the content has already changed each time the website is accessed, making it difficult to achieve continuous information collection (Patent Document 1, Patent Document 2).

[0011] The present invention has been made in view of the above background, and aims to provide an information collection system and an information collection method that are capable of continuously collecting information even from websites that are structurally changing as information collection destinations. [Means for solving the problem]

[0012] In order to solve the above-mentioned problems and achieve the above-mentioned object, one embodiment of the present invention is an information collection system that collects information from websites via a network, and is characterized by comprising: a collection unit that analyzes the structure of a target website from which information is to be collected, stores a scenario that sets the location of the information to be collected, and continues to collect information from the target website based on the scenario; a detection unit that prepares in advance multiple modules that search for the location of the information to be collected, and when the collection unit detects that it cannot continue collecting information, assigns from the multiple modules a method of searching for the location set in the scenario; a search unit that searches for the location of the information to be collected from the target website based on the modules assigned by the detection unit; and a repair unit that repairs the scenario by changing the location of the information to be collected based on the search results of the search unit.

[0013] Another embodiment of the present invention is an information collection system that collects information from websites via a network, comprising a processing device that controls the information collection and a storage device that stores data related to the information collection under the control of the processing device, wherein the processing device analyzes the structure of a target website from which information is collected, stores a scenario that sets the location of the information to be collected, and continues to collect information from the target website based on the scenario, prepares in advance multiple modules that search for the location of the information to be collected, and when it detects that the information collection cannot be continued, assigns a method of searching for the location set in the scenario from the multiple modules, searches for the location of the information to be collected from the target website based on the assigned module, and changes the location of the information to be collected from the target website based on the search results, thereby repairing the scenario stored in the storage device.

[0014] Furthermore, another embodiment of the present invention is an information collection method for a system that collects information from websites via a network, the system having a storage device, and including the steps of analyzing the structure of a target website from which information is collected, storing a scenario in the storage device that sets the location of the information to be collected, and continuing to collect information from the target website based on the scenario, preparing in advance a plurality of modules that search for the location of the information to be collected, and assigning from the plurality of modules a method of searching for the location set in the scenario when it is detected that the information collection cannot be continued, searching for the location of the information to be collected from the target website based on the assigned module, and changing the location of the information to be collected from the search results and repairing the scenario. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide an information collection system and an information collection method that are capable of continuously collecting information even from websites that are structurally changing as information collection destinations. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a schematic configuration example of an information collection system according to an embodiment of the present invention; [Figure 2] FIG. 10 is an explanatory diagram illustrating an example of a change in a website according to an embodiment of the present invention. [Figure 3] FIG. 2 is an explanatory diagram illustrating an example of a scenario table according to an embodiment of the present invention. [Figure 4] FIG. 10 is an explanatory diagram illustrating an example of a feature storage table according to an embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram showing an example of a scenario registration screen according to an embodiment of the present invention. [Figure 6] 1 is a configuration diagram showing an example of a hardware configuration of an information collection system according to an embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating a main process according to an embodiment of the present invention. [Figure 8] 10 is a flowchart illustrating a scenario registration process according to an embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating an information collection process according to an embodiment of the present invention. [Figure 10] 10 is a flowchart illustrating a data analysis process according to an embodiment of the present invention. [Figure 11] 10 is a flowchart illustrating a cause allocation process according to an embodiment of the present invention. [Figure 12] 10 is a flowchart illustrating a search process according to an embodiment of the present invention. [Figure 13] 10 is a flowchart illustrating a site search process according to an embodiment of the present invention. [Figure 14] 10 is a flowchart illustrating a scenario repair process according to an embodiment of the present invention. [Figure 15] FIG. 10 is an explanatory diagram showing an example of a scenario table that has been corrected according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following description and drawings are merely examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Furthermore, unless otherwise specified, each component may be singular or plural.

[0018] In the following description, identical or similar components may be designated by the same reference numerals, and redundant description may be omitted. Furthermore, in the following description, various types of information may be described using expressions such as "information" and "table," but the various types of information may also be expressed using other data structures. Furthermore, identification information may be expressed using expressions such as "identification information," "identifier," "name," "ID," and "number," but these are interchangeable. Furthermore, in the following description, "database" will be referred to as "DB," and "table" as "TBL."

[0019] 1 is a block diagram showing a schematic configuration example of an information collection system according to this embodiment. As shown in the figure, the information collection system 1 is composed of, for example, a collection unit 100, a detection unit 200, a search unit 300, and a repair unit 400, and is used by organizations such as companies and government agencies, and individuals who continuously collect information posted on websites 30.

[0020] The information collection system 10 is communicatively connected to various websites from which information is to be collected via a communication network 50 such as the Internet. Here, website 30 is taken as an example of a site from which information is to be collected. The information collection system 10 is also communicatively connected to an operation terminal 20, an approval terminal 40, and the like, as terminals operated by an administrator, via wired or wireless communication means (LAN (Local Area Network), wireless LAN, etc.).

[0021] The operation terminal 20 provides a user interface for inputting URLs (Uniform Resource Locators), collection timing, location information of specific targets for web scraping, etc., to users such as administrators and users of the information collection system 10. As will be described later, when a scenario is modified by the information collection system 10, the operation terminal 40 executes approval as to whether or not to adopt the scenario as a scenario for subsequent information collection. Note that this approval method may be a mechanism for automatically repairing the scenario without requiring user approval via the operation terminal 40.

[0022] The information collection system 10 accesses a website 30 and collects information listed on the website according to a scenario. The information to be collected is narrowed down by using, for example, web scraping technology, and stored as data. In this case, even if a change occurs in the location of the information to be identified by web scraping and narrowing down the information fails, it is possible to continue collecting information.

[0023] To achieve this, the system uses the identifiers and layout characteristics of the specific area where the information collection target is located, set in the scenario for web scraping, and the results of the previous collection to estimate the change in the specific area, evaluate the location information of the estimated specific area, and rewrite the scenario. In this way, web scraping can be continued for the target website from which information is collected.

[0024] The collection unit 100 receives as input a scenario for collecting information from a website 30 that is the target of information collection, and acquires the information to be collected by accessing the site. The detection unit 200 detects a state in which the collection unit 100 is unable to narrow down the specific target of web scraping when collecting information, and assigns a search method to be executed by the search unit 300 based on the cause of the inability to narrow down the specific target, the tendency of the specific target, etc.

[0025] The search unit 300 searches for web scraping targets based on the search methods assigned by the detection unit 200 and derives new location information for the specific area. In other words, estimation is performed here, and potential new destinations are inferred from the search results. The repair unit 400 evaluates whether the new location information for the specific area is correct, and if it is correct, corrects the scenario and notifies the approval terminal 40, thereby notifying the user of the change. Note that the scenario may be repaired automatically without requiring user approval, i.e., without receiving instructions from the approval terminal 40.

[0026] Here, each of the units 100, 200, 300, and 400 will be described in detail using Figure 1. First, the collection unit 100 will be described. The collection unit 100 is composed of, for example, a scenario registration unit 101, a scheduler unit 102, a scenario DB 103 that registers the scenario at the registration timing of the scenario registration unit 101, a crawler unit 104 that executes crawling of the target website 30 by the scheduler unit 102, a data analysis unit 105 that analyzes the HTML of the target website 30 in DOM (Document Object Model) element units to acquire feature amounts, a data storage DB 106, and a feature DB 107 that stores feature amounts based on the analysis results of the data analysis unit 105.

[0027] In the above configuration, the scenario registration unit 101 accepts operations from the operation terminal 20 and registers a scenario in the scenario DB 103 and the scheduler unit 102, which describes the web scraping target and execution timing specified on the page, such as the information collection destination, title, item list, etc. indicated by the URL of the target website.

[0028] The scheduler unit 102 triggers the crawler unit 104 at the timing set in the scenario registered in the scenario DB 103, causing the crawler unit 104 to access the website 30 via the communication network 50. The crawler unit 104 inputs the HTML information acquired as a result of the access into the data analysis unit 105 and simultaneously stores the information in the data storage DB 106.

[0029] The data analysis unit 105 analyzes information that constitutes a web page, such as HTML information, CSS (Cascading Style Sheets), and JavaScript, obtained from the website 30, and stores the features of the specific area that is the target of web scraping in the feature DB 107.

[0030] Next, the detection unit 200 will be described. The detection unit 200 is composed of, for example, an error detection unit 201 and a distribution unit 202. The error detection unit 201 detects that an attempt to acquire information from the website 30 has failed, and the distribution unit 202 distributes the cause of the failure to acquire information in response to the detection.

[0031] Next, the search unit 300 will be described. The search unit 300 is configured, for example, with an identifier search unit 301, a character string search unit 302, a resource search unit 303, a feature cluster search unit 304, a site search unit 305, and the like as search method modules. The identifier search unit 301 searches for an identifier based on the features of a specific area stored in the feature DB 107 in accordance with the sorting by the sorting unit 202. The character string search unit 302 searches for a character string based on the features of a specific area stored in the feature DB 107 in accordance with the sorting by the sorting unit 202.

[0032] The resource search unit 303 searches for resources based on the features of the specific area stored in the feature DB 107 in accordance with the allocation by the allocation unit 202. The feature cluster search unit 304 acquires features having layout features similar to those of the previous time based on the features of the specific area stored in the feature DB 107 in accordance with the allocation by the allocation unit 202. If the site search unit 305 cannot collect information, it searches another URL within the same website 30.

[0033] The repair unit 400 is composed of, for example, an information evaluation unit 401, a notification unit 402, a repair result comparison unit 403, a repair approval unit 404, and a scenario repair unit 405. The information evaluation unit 401 evaluates whether the specific area determined as the destination by the search unit 300 is the correct area, and the notification unit 402 executes processing to notify the approval terminal 40 of the cause of failure and the details of correction in order to notify the user of the results of the scenario repair.

[0034] The repair result comparison unit 403 determines whether the scenario candidate obtained by the information evaluation unit 401 is an appropriate scenario by comparing it with the previous scenario. The repair approval unit 404 instructs the scenario repair unit 405 whether or not to repair the scenario in accordance with the operation of the approval terminal 40. The scenario repair unit 405 executes scenario repair in accordance with the instruction from the repair approval unit 404.

[0035] Next, changes in the target website will be described with reference to Fig. 2. Fig. 2 shows an example of a screen displayed on the display of the operation terminal 20 or the like when the website 30 is accessed every month in this embodiment.

[0036] The example screen shown here shows the changes in the web pages 2000, 2100, and 2200 that are displayed when the same website 30 is accessed, for example, every month. Web pages 2000, 2100, and 2200 indicate the passage of time, for example, July, August, and September, from left to right in the figure, and show how the layout configuration changes.

[0037] On web pages 2000, 2100, and 2200, in addition to buttons linking to the TOP page and LIST page, links to the COMMUNITY page are posted at the top of the screen as link buttons 2002, 2102, and 2202, respectively. Also, at the bottom of the screen are fields for timelines 2001, 2101, and 2201, which show the latest trends for July, August, and September, respectively. Timelines 2001, 2101, and 2201 are laid out so that users can view the three most recent articles, going back from the most recent articles 2003, 2103, and 2203, respectively.

[0038] The August web page 2100 shows the results of accessing the same web page as the July web page 2000 after some time has passed. In this case, a comparison between link button 2002 and link button 2102 and a comparison between timeline 2001 and timeline 2101 reveals that there is no change in the content, as shown in Figure 2.

[0039] In addition, when comparing the latest article 2003 and the latest article 2103 shown in the timelines 2001 and 2101, respectively, the content will naturally change over time as the latest article posted on the timeline is the latest article at the time of access.

[0040] September web page 2200 shows the result of accessing the same web page as August web page 2100 at a later time. In the example of Figure 3, a comparison of web page 2200 and web page 2100 shows layout changes.

[0041] It can be seen that the link buttons to the TOP and LIST pages and the link button 2202 to the COMMUNITY page, which were located at the top of the screen on web pages 2000 and 2100, have been repositioned to the left side of the screen on web page 2200. Another difference is that new link buttons 2204 for search and contact have been added.

[0042] Additionally, in the upper right corner of the screen there is a field for timeline 2201 that shows the latest trends, and below the timeline 2201 there is a layout that allows users to check three articles going back from the most recent article 2203. A typical visitor obtains information by viewing such a page using software such as a web browser.

[0043] Next, the scenarios registered in the scenario DB 103 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a scenario TBL 3000 according to this embodiment. The scenario TBL 3000 shown in Fig. 3 is made up of multiple entries (e.g., records) each having items such as a scenario ID 3100, a target URL 3200, an execution timing 3300, an in-page action after access 3400, a web scraping target 3500, and a backup ID 3600 indicating a backup scenario.

[0044] In the illustrated scenario, one entry (by scenario ID) in the scenario TBL 3000 indicates one web page, but it may be a format in which one website is one entry, or one crawler execution is one entry. Although this example shows the TBL format, information may also be stored in a format other than the TBL format, such as JSON (JavaScript Object Notation).

[0045] In the above items, a scenario ID that is an identifier for identifying a scenario is set in the scenario ID 3100. A target URL that the information collection system 10 accesses to acquire information is set in the target URL 3200. Date and time information for accessing the target URL is set in the execution timing 3300. Here, the access time for each day is set.

[0046] The in-page action 3400 describes the order of actions to be performed on the target web page after access, such as clicking. If there is no particular action to be performed, do not set anything.

[0047] The location of the information to be saved in the scenario DB 103, which is the data storage destination, is set in the web scraping target 3500. The location set here may be location information such as an HTML identifier or Xpath, as exemplified, or character string information written in the target. The ID of the backup scenario is set in the backup ID 3600.

[0048] 1, the crawler unit 104 accesses the target URL of the website 30 listed in the scenario DB 103 according to the execution timing of the scheduler unit 102. At this time, the information acquired from the website 30 is passed to the data storage DB 106 and the data analysis unit 105, whereby the information at the time of access is stored and analyzed.

[0049] Next, the features stored in the feature DB 107 will be described. FIG. 4 is a diagram showing an example of a feature TBL 4000 according to this embodiment. The feature TBL 4000 shown in FIG. 4 is composed of a plurality of entries (e.g., records) each having items such as an execution ID 4100, an acquisition date and time 4200, a target URL 4300, a target DOM element 4400 indicating the position information of the target DOM element, a Visible flag 4500 indicating whether the target DOM element is a visualized element, a DOM order 4600 in which the target DOM element is written in HTML, a child node number 4700 indicating the number of nodes belonging to the target DOM element, a visible position 4800 indicating the visible position of the target DOM element on a web page, and a visible area 4900 indicating the visible area of ​​the target DOM element. In addition to the information shown as an example, it is desirable to add feature information of the web page to the feature TBL 4000.

[0050] In the example of Figure 4, entries 4001 to 4009 are shown for execution ID 4100, and for example, "000001" is set in entry 4001. One entry in feature TBL 4000 shows an example of a web page broken down by DOM element and execution timing. In this way, one web page is divided into DOM elements and information is saved. Although this example shows the TBL format, information may also be saved in a format other than the TBL format, such as JSON.

[0051] In the above items, the execution ID 4100 describes an execution ID that is an identifier for the execution when the information was acquired. The acquisition date and time 4200 describes date and time information when the information was acquired. The target URL 4300 describes the URL of the target web page from which information was acquired. The target DOM element 4400 describes Xpath information when the web page is divided into DOM units.

[0052] The Visible flag 4500 is set to True if the item is visibly displayed on the screen when the target DOM element is rendered, and set to False if the item is not visibly displayed, such as meta information.

[0053] The DOM order 4600 describes the DOM position of the target DOM element within the HTML of the web page. The number of child nodes 4700 describes how many DOM elements are attached below the target DOM element, assuming that HTML is described in a tree format with DOMs as nodes. The visible position 4800 describes the position on the display screen in terms of x- and y-axis coordinates when the target DOM element is rendered. The visible area 4900 describes the area of ​​the target DOM element when it is rendered.

[0054] Note that the coordinate position and area of ​​a web page differ depending on the environment from which it is accessed, but the coordinates described here are the position when accessed by the crawler unit 30, and are the coordinates of the environment of the information collection system 10. Also, if the crawler unit 30 does not have a rendering function, the setting may be such that the position information is not described.

[0055] Next, the scenario registration screen will be explained using Fig. 5. Fig. 5 is a diagram showing an example of a scenario registration screen according to this embodiment. As shown in the figure, the illustrated scenario registration screen 5000 includes a table 5001 showing the execution status of the crawler, a field 5002 for adding a new scenario file, an add button 5003 for executing the addition, and a mouse pointer 5004.

[0056] Table 5001 in the same figure displays an identifier 5100 indicating the crawler execution entry, next execution timing 5200 indicating the date and time of the next execution, previous execution timing 5300 indicating the date and time of the previous execution, scenario ID 5400 to be referenced, target site 5500 indicating the site domain to be collected, and status buttons 5600 for displaying the crawler's execution status (error repairing, running, waiting, stopped, etc.) and for transitioning to details, stopped, started, etc.

[0057] The user registers an information collection scenario by operating the mouse pointer 5004 on the operation terminal 20. Note that the scenario does not have to be registered using this screen; it may be registered using a file transmission format or a separate tool for creating a scenario.

[0058] 5, five entries from 5111 to 5115 are set for the identifier 5100. For example, in the screen display of entry 5111, the previous execution 5300 is set to 18:00 on September 22, 2022, and the next execution 5200 is set to 18:00 on September 23, 2022. "00091" is set for the scenario ID 5400, and "abc.com" is set for the URL of the target site 5500. "Error repairing" is set for the status 5600, and "Details" for checking details and "Stop" for stopping the operation are set as buttons.

[0059] The hardware configuration of this embodiment will now be described with reference to Fig. 6. Fig. 6 is a diagram illustrating the hardware configuration of this embodiment for the components of Fig. 1. For example, as shown in Fig. 6, the hardware includes a CPU 1 that executes a program 21 of this embodiment as a processing device, a memory 2 that stores the program 21 of this embodiment, an external storage device 3 that has the scenario DB 103, data storage DB 106, feature DB 107, etc. shown in Fig. 1, an interface (hereinafter referred to as "IF") 4 that connects to the operation terminal 20, approval terminal 40, communication network 50, etc. via a bus, an input / output device 5 such as a mouse, keyboard, display, etc., and a bus 6 that communicates signals and data between units in the information collection system 10.

[0060] The information collection system 10 in this embodiment is realized by loading a program 21 that realizes the functions of this embodiment into the memory 2 and executing it by the CPU 1. These programs may be stored in the external storage device 3 in advance, or may be introduced into the external storage device 3 when needed from another device via the IF 4, the input / output device 5, and a medium that can be used by the information collection system 10.

[0061] The program 21, under the control of the CPU 1, realizes functions such as the collection unit 100, detection unit 200, search unit 300, and repair unit 400 in Figure 1, and has processing procedures that govern the operations shown in the flowcharts of Figures 7 to 14, which will be described later, as specific operations.

[0062] The media available to the information collection system 10 refer to, for example, a storage medium that is detachable from the input / output device 5, or a carrier wave or digital signal that propagates through a network that can be connected to the IF 4. The program may be temporarily stored in the external storage device 5, and then loaded from there onto the memory 3 and executed by the CPU 2, or may be loaded directly into the memory 2 and executed by the CPU 1 without being stored in the external storage device 3.

[0063] Next, the operation according to this embodiment will be described in detail. Fig. 7 is a flowchart illustrating an example of the overall processing executed by the program 21 of the information collection system 10 in this embodiment. In the figure, the processing of steps S1001 and S1002 is processing performed mainly by the collection unit 100 shown in Fig. 1, and the processing of step S1003 is processing performed mainly by the detection unit 200. The processing of step S1004 is processing performed mainly by the search unit 300, and the processing of step S1005 is processing performed mainly by the repair unit 400.

[0064] 7, the processing of the collection unit 100 involves a series of scenario registration processes for registering a scenario for executing information collection in step S1001, and an information collection process for executing information collection according to the scenario in step S1002. Here, a process for determining whether information collection was successful is executed, and if the result of this determination is that information collection failed, the process proceeds to step S1003, and if it was successful, the process ends.

[0065] In step S1003, the detection unit 200 executes a series of cause allocation processes to allocate causes if the information collection in step S1003 is unsuccessful, and in step S1004, the search unit 300 executes a series of search processes to search for a repair method if the scenario fails. Furthermore, in S1005, the repair unit 400 executes a series of scenario repair processes to generate a new scenario from the candidates indicated in the search process in step S1004.

[0066] Next, each process in Fig. 7 will be described in detail using Fig. 8 to Fig. 14. First, the scenario registration process will be described using Fig. 8. Fig. 8 is a flowchart specifically explaining the scenario registration process of step S1001 shown in Fig. 7 in this embodiment. In the scenario registration process, as shown in the figure, first, an operation to select a web page from which the user wants to perform information collection is accepted from the operation terminal 20 (step S2001).

[0067] Next, an operation to select a target DOM element for which the user wants to acquire information or perform an operation within the target web page is accepted from the operation terminal 20 (step S2002), and the operation content and acquisition are determined (step S2003).Furthermore, an operation to determine the frequency of executing information collection on the target web page is accepted from the operation terminal 20 (step S2004).

[0068] In this way, a scenario file is created that describes the contents determined in steps S2001 to S2004 (step S2005). In this creation, a creation screen may be added to a specific tool or information collection system 10 and used to create the scenario file.

[0069] Furthermore, registration is executed from the operation terminal 20 to the scenario registration unit 101 using the scenario file created in step S2005, and at that timing, registration is also made in the scenario DB 103. Then, the start time is registered in the scheduler unit 102 based on the frequency information written in the scenario file (S2006).

[0070] Next, the information collection process will be described with reference to FIG. 9. FIG. 9 is a flowchart illustrating the information collection process of step S1002 shown in FIG. 7 in this embodiment. First, the crawler unit 104 is started according to the schedule of the scenario registered by the scheduler unit 102 (step S3001). Next, the crawler unit 104 executes a process of calling an appropriate scenario from the scenario DB 103 (step S3002). This allows the execution content to be read through the scenario. The appropriate scenario here is, for example, a scenario that is called to start information collection at 6:00 p.m. as defined in the schedule.

[0071] Next, the crawler unit 104 executes a process of attempting to access the target web page (in this case, within website 30) from the URL information described in the called scenario, as shown in the scenario table of FIG. 3 (step S3003). At this time, it is determined whether the access was successful or unsuccessful (step S3004). If the access was successful, a process is executed to identify the DOM elements described in the scenario and perform operations or information acquisition (step S3005). Then, the collected data, such as HTML, is analyzed by the data analysis unit 105 (step S3006). Then, the process proceeds to step S3007.

[0072] In the process of step S3005, it is determined whether the target DOM was successfully identified, that is, the existence of the target DOM is confirmed (step S3007), and if it is determined that the identification was successful, it is determined that the information collection was successful and this process ends. Also, if it is determined that the identification of the target DOM failed in the process of step S3007, it is determined that the information acquisition failed (step S3010).

[0073] If it is determined in step S3010 that information acquisition has failed, a check is made to see if a backup scenario (such as a scenario that captures the entire screen when information acquisition fails) is registered in the scenario called from scenario DB103 (step S3011).

[0074] As a result, if a backup scenario (for example, backup ID 00010) exists, such as scenario ID 00001 in the example of Figure 3, the process returns to step S3002 and the same process is repeated. On the other hand, if the determination in step S3011 indicates that no backup scenario exists, the process proceeds to cause allocation processing (step S1003) to identify the cause of failure.

[0075] At this point, the process returns to step S3004, and if the access fails, the access process is repeated a set number of times (N is a natural number), for example, three times. If the access fails after a certain number of attempts (step S3008), it is determined that the failure was due to inaccessibility (S3009), and the process proceeds to step S3007. In this case, the scenario is not repaired as there is a problem with the target website. Needless to say, the number of accesses, N, is not limited to three, but may be four, five, or other times, as this is a design matter.

[0076] Next, the data analysis process will be described with reference to Fig. 10. Fig. 10 is a flowchart illustrating the data analysis process of step S3006 shown in Fig. 9 in this embodiment. First, HTML files collected as a result of accessing web pages on the target website 30 are decomposed into feature amounts for each DOM element (step S4001). At this time, even if the feature is not an HTML file, it is acceptable to refer to the value of any feature that indicates an element in a web page.

[0077] Next, it is confirmed whether information collection using the scenario executed in this information collection has been executed before, whether data for the same web page as the current web page exists in the feature DB 107, and whether the target DOM element is a DOM element that existed from the previous time (step S4002).

[0078] If it is confirmed in step S4002 that the same DOM element exists in feature DB 107, the currently acquired DOM element is compared with the previously acquired DOM element to confirm whether the same information is described in the DOM element (step S4003). Then, it is confirmed whether the same information is described (S4004), and if the same information is described, no particular flag is added, and the currently analyzed feature is saved in feature DB 107 (step S4005).

[0079] If it is confirmed in step S4004 that the information is not the same but different between the current and previous results, it is determined that the information is a DOM element that changes, a change flag indicating that it is a changing DOM element is assigned (step S4007), and the feature is saved in feature DB 107 in step S4005.

[0080] Also, if it is determined in step S4002 that the DOM element does not exist since the previous time, a flag indicating that it is a newly generated DOM element is added (step S4006), and its feature is saved in the feature DB 107 in step S4005.

[0081] Next, the cause allocation process will be described with reference to Fig. 11. Fig. 11 is a flowchart illustrating the cause allocation process of step S1003 shown in Fig. 7 in this embodiment. First, after the error detection unit 201 detects a failure, the allocation unit 202 executes the following process to allocate the cause of the information acquisition failure (step S5001).

[0082] If the target page cannot be accessed as a result of this cause allocation (step S5002), it is determined that the information collection failed because of the inaccessibility (step S5007), and the process then proceeds to step S5006.

[0083] If it is determined in step S5002 that the target web page can be accessed and the target DOM element was specified by an identifier in the scenario file (step S5003), it is determined that the access failed because the identifier no longer exists in the web page (step S5008). Then, the process proceeds to step S5006.

[0084] If it is determined in step S5003 that the target DOM element is not specified by an identifier, but that the target DOM element is specified in the scenario file by location information such as Xpath (step S5004), it is determined that the operation failed because the location of the target DOM element has changed (step S5009). Then, the process proceeds to step S5006.

[0085] Next, if it is determined in step S5004 that the target DOM element was not specified in the location information, but was specified by text (step S5005), it is determined that the failure occurred because the information content of the target DOM element changed (step S5010). Then, the process proceeds to step S5006. In this way, in step S5006, the cause of the failure is finally notified by the notification unit 402 and is stored in memory 2 for subsequent processing, and the process ends.

[0086] Next, the search process will be described with reference to Fig. 12. Fig. 12 is a flowchart illustrating the search process of step S1004 in this embodiment shown in Fig. 7. First, the allocating unit 202 executes a process of acquiring features from the feature DB 107 as parameters of the target DOM on which an operation or information acquisition was previously performed in the scenario file (step S6001).

[0087] Next, the identifier search unit 301 utilizes all identifier information that existed in the previous target DOM to search whether the identifier and the corresponding DOM element exist in the web page such as HTML collected this time (step S6002).

[0088] From the perspective of this identifier matching, for example, if the scenario file specifies a timeline but in reality it is specified using a different identifier such as Xpath, information will be collected using the other identifier such as class or Xpath.

[0089] If it is determined from the execution result of step S6002 that one DOM element can be specified from one identifier (step S6003), that DOM element is stored as a candidate in memory 2. In this way, estimation is performed by search unit 300 (step S6004). If it is determined in step S6003 that multiple candidates exist, it is determined that identification by identifier is difficult, and search processing continues using another function (step S6010).

[0090] Here, although an ID can be uniquely specified, there are cases where multiple IDs can be specified depending on the class (making identification difficult), so it is determined that the information cannot be identified, and a search is performed using a different search module, assuming that identifier matching was not successful.

[0091] Once the candidate DOM element is determined in step S6004, the string search unit 302 and resource search unit 303 execute a process to confirm that the target DOM element on which the operation or information acquisition was performed in the previous scenario file does not have the change flag set in the aforementioned step S4007 (see Figure 10) (S6005).

[0092] If it is determined in step S6005 that the change flag is not set, the string search unit 302 and resource search unit 303 perform text matching using the text information and resource information of the previous DOM element.

[0093] A search is performed to determine whether a DOM element containing the same text or resource exists in the web page such as HTML collected this time in the previous information, and the candidates are acquired (step S6006). Here, candidates are selected by each search by the search unit 300, and finally evaluated by the information evaluation unit 401.

[0094] Also, if it is determined in step S6005 that the change flag is set, the matching is non-text, and it is determined that the information content of the target DOM element is a DOM element whose content will be changed, and the search continues using another function (step S6011).

[0095] Here, when matching text, a perfect match is required, but if a change flag is attached, a perfect match is not possible, so for example, a different feature matching method will be used.

[0096] Next, the feature cluster search unit 304 executes processing using the feature information of the target DOM element that was previously operated or information was acquired in the scenario file. The previous feature is compared with the feature of the DOM element in the web page such as HTML that was collected this time. This comparison is used to search for DOM elements with similar layout features, and a process is executed to acquire candidates (step S6007).

[0097] Here, similarity is determined based on whether the distance between the feature amounts is close. Therefore, a distance criterion is determined in advance, and features that are close in distance according to that criterion are determined to be similar.

[0098] Then, the candidate DOM elements that have been acquired so far are compared with the previously acquired results to determine whether they are visible elements, and whether there is a certain difference or more from the previous result based on the amount of information and the distance of layout features (step S6008). After this determination, it is confirmed whether any candidates remain (step S6009), and if it is determined that there are candidates, the process ends.

[0099] On the other hand, if it is determined in step S6009 that no candidates remain, it is determined whether a predetermined threshold for search processing within the same website 30 has been reached (step S6012). As a result, if it is determined that the threshold has not been reached, search processing based on, for example, a URL is performed within the same website 30 (step S6013). Here, the threshold can be arbitrarily set by the user via the operation terminal 20 as a condition such as the number of backup scenarios or processing time.

[0100] For example, a case may be assumed in which a weather page moves to another page. If the feature quantities of a new web page are obtained by this site search, the same processing continues from S6001.

[0101] The site search process will now be described with reference to Fig. 13. Fig. 13 is a flowchart illustrating the site search process in step S6013 of this embodiment shown in Fig. 12. First, the site search unit 305 extracts a list of link information contained in the currently collected web page, such as HTML (step S7001).

[0102] Next, the site search unit 305 accesses the web pages of the link information extracted and acquired in this way, and a process of collecting web page information stored as HTML data is executed (step S7002). Then, the HTML collected in this way is sent to the data analysis unit 105, where it is converted into feature quantities (step S7003). Then, the process proceeds to step S6001 in FIG. 12.

[0103] Next, the scenario repair processing will be explained using Fig. 14 and Fig. 15. Fig. 14 is a flowchart explaining the scenario repair processing of step S1007 shown in Fig. 7 in this embodiment, and Fig. 15 is an explanatory diagram showing an example of a correction to the scenario TBL according to this embodiment.

[0104] First, the search unit 300 executes a process of determining whether any DOM elements that are candidates for repair remain (step S8001). If it is determined that candidates remain, the information evaluation unit 401 executes a process of creating a scenario from all the candidates, using location information such as an identifier indicating the target DOM element, Xpath, resource path information, text information, etc. (step S8002). On the other hand, if it is determined that no candidates remain, it determines to notify that the repair has failed (step S8010), and notifies the approval terminal 40 of the failure of the repair via the notification unit 402 (step S8009).

[0105] Next, the repair result comparison unit 403 uses the scenario information created in step S8002 to obtain and operate information on the currently obtained web page information such as HTML (step S8003). From the processing result of step S8003, it is determined whether or not there is a candidate in which no error occurred (step S8004).

[0106] If it is determined in step S8004 that candidates remain, the result obtained by specifying the target DOM element in the previous scenario is compared with the execution result of step S8003 to determine whether they are a perfect match. If there is no perfect match, the result, i.e., the closest distance, is determined based on feature quantities such as the number of characters and the number of child nodes (step S8005).

[0107] On the other hand, if it is determined that no candidates remain, a decision is made to notify that the repair has failed (step S8010), and the failure of the repair is notified to, for example, the approval terminal 40 via the notification unit 402 (step S8009).

[0108] The scenario that shows the closest result in step S8005 is adopted as the main candidate (step S8006). Here, the judgment is based on closeness, so that if there is an exact match in a fixed character string such as text, it becomes the scenario that is the closest result. The above steps S8005 and S8006 are executed by the information evaluation unit 401.

[0109] Furthermore, the scenario repair unit 405 registers as a backup scenario any scenario that shows the same results as the candidate adopted as the main in step S8006 (step S8007). At this time, the priority between the main scenario and the backup scenario may be determined based on the method of specifying the target DOM element, such as identifier or text, as one of the predetermined priorities for information collection.

[0110] In this way, multiple scenarios are prepared for the target website when preparing backup scenarios, which increases the variety of ways to gather information. This means that if information acquisition fails with one scenario, you can try gathering information again with a different scenario. This order, including backup scenarios, can be specified to the system in advance, and it goes without saying that there can be two or more backup scenarios.

[0111] The newly created scenario is overwritten in the scenario DB 103 in order to be used for the next execution (step S8008). At this time, the scenario restoration unit 405 does not automatically overwrite the scenario, but the restoration approval unit 404 may decide to officially approve the scenario after receiving an approval instruction based on a user operation from the approval terminal 40 that accepts user operations, and then overwrite the scenario.

[0112] Here, a specific example of this overwriting will be explained by comparing Figure 3 with Figure 15. The scenario information of entries 3001 and 3002 registered in scenario TBL 3000 shown in Figure 3 is overwritten to scenario TBL 3001b and 3002b, respectively, as shown in Figure 15. In other words, web scraping target 3500 in Figure 3 changes to web scraping target 3500b in Figure 15.

[0113] In the web scraping target 35001b shown in Figure 15, the location of the timeline has been changed from being specified by ID to being specified by text, and the div has been changed for the Xpath. Similarly, in the web scraping target 35002b shown in Figure 15, the location of the profile has been changed from being specified by ID to being specified by text, and the div has been changed for the Xpath.

[0114] As can be seen from the comparison between Figure 3 and Figure 15, applying an identifier-based search method to web scraping targets is useful for confirming the location of information.

[0115] Returning to the processing in Fig. 14, the notification unit 402 executes processing to notify the approval terminal 40 of the cause of failure and the details of correction in order to notify the user of the result of this scenario repair (step S8009). At this time, the notification may be made through the management UI as shown in Fig. 5, or by e-mail or the like.

[0116] As described above, according to this embodiment, it is possible to continuously collect information even from websites that are structurally changing as information collection destinations.

[0117] Furthermore, in this embodiment, as a search method when elements within a web page can no longer be identified due to changes in website 30, search unit 300 is provided with multiple search modules, such as identifier search unit 301, string search unit 302, resource search unit 303, feature cluster search unit 304, and site search unit 305. This prevents the search from being limited even if the location of information moves, increasing the possibility of finding the new location and broadening the tolerance for structural changes to the target website. As a result, it becomes easier to repair the scenario and ensures continuous information collection.

[0118] Here, rather than configuring the search unit 300 as a single piece of hardware, the system configuration may be such that the hardware is distributed according to the search function, with the identifier search unit 301, string search unit 302, resource search unit 303, feature cluster search unit 304 and site search unit 305 being configured individually or in combination of two or more.

[0119] Furthermore, in this embodiment, if the search unit 300 changes the location of information on the website 30 and the location after the change can be identified, the repair unit 400 checks by collecting information using the scenario output by the repair unit 400. This allows a determination that the scenario is appropriate if the collection is successful, making it possible to prepare a continuous scenario for the next information collection.

[0120] Furthermore, the repair unit 400 can also execute final scenario repair in response to user operation of the approval terminal 40, making it possible to respond appropriately to business flows.

[0121] Furthermore, it goes without saying that the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0122] Furthermore, it is possible to add, delete, or replace part of the configuration of the above embodiment with other configurations. For example, the system 10 may be configured to function as the approval terminal 40 as well.

[0123] Furthermore, the above-described configurations, functional units, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), an IC card, an SD card, a DVD, or other recording media.

[0124] Furthermore, the above-described layout of the various functional units, processing units, and databases of the information collection system 10 is merely an example. The layout of the various functional units, processing units, and databases can be changed to an optimal layout in terms of the performance, processing efficiency, communication efficiency, etc. of the hardware and software provided in these devices.

[0125] Furthermore, the configuration (schema, etc.) of the database that stores the various types of data described above can be flexibly changed from the perspective of efficient use of resources, improved processing efficiency, improved access efficiency, improved search efficiency, and the like.

[0126] In particular, when collecting information on big data, web scraping is performed by crawling across various target websites on the Internet, so as described in the above embodiment, it is important to search and locate identifiers that indicate the destination within each target website from which information is to be collected, and to automatically keep the scenario up to date in order to ensure continuous and successful web scraping on the target websites.

[0127] In other words, in large-scale information gathering operations, automating this scenario repair can reduce the amount of work required by people (information gatherers), significantly reducing human resources, and significantly shortening the overall processing time for information gathering. [Explanation of symbols]

[0128] 1 CPU 2. Memory 3 External storage device 4. IF 5 Input / Output Devices 10 Information Collection System 20 Operation terminal 21 Programs 30 websites 40 Approval terminal 50 Communication Network 100 Collection Department 101 Scenario Registration Section 102 Scheduler 103 Scenario DB 104 Crawler section 105 Data Analysis Department 106 Data Storage DB 107 Feature DB 200 Detection unit 201 Error detection unit 202 Distribution Department 300 Search Department 301 Identifier Search Unit 302 String Search Section 303 Resource Search Department 304 Feature Cluster Search Unit 305 Site Search Department 400 Restoration Department 401 Information Evaluation Department 402 Notification Department 403 Repair result comparison section 404 Repair Approval Department 405 Scenario Repair Department

Claims

1. An information collection system for collecting information from a website via a network, a collection unit that analyzes a structure of a target website from which information is to be collected, stores a scenario in which the location of information to be collected is set, and continuously collects information from the target website based on the scenario; a detection unit which prepares in advance a plurality of modules for searching for the location of information to be collected, and which selects a method for searching for the location set in the scenario from among the plurality of modules when the collection unit detects that the information collection cannot be continued; a search unit that searches for the location of the information to be collected from the target website based on the modules assigned by the detection unit; a repair unit that repairs the scenario by changing a location of the information to be collected based on a search result of the search unit; An information gathering system comprising:

2. 2. The information collection system according to claim 1, wherein the plurality of modules include at least two or more search units selected from an identifier search unit, a string search unit, a resource search unit, a feature cluster search unit, and a site search unit.

3. 2. The information collection system according to claim 1, wherein the repair unit performs information collection of the target website based on the location of the information to be collected obtained by the search unit, and determines whether to repair the scenario based on the results of the execution.

4. 2. The information collection system according to claim 1, wherein the information collection system is connected to a terminal that accepts user operations, and the repair unit executes information collection of the target website based on the location of the information to be collected obtained by the search unit, and determines whether or not to approve the repair of the scenario based on the results of the execution in accordance with user operations of the terminal.

5. 2. The information collection system according to claim 1, wherein the detection unit performs allocation from the plurality of modules based on the identifier indicating the location from the analysis result of the collection unit.

6. 2. The information collection system according to claim 1, wherein the repair unit stores multiple types of scenarios according to predetermined priorities of information collection, and the collection unit collects information according to the priorities of the multiple types of scenarios stored.

7. An information collection system that collects information from websites via a network, A processing device for controlling the collection of information; a storage device that stores data related to information collection under the control of the processing device; having The processing device includes: analyzing a structure of a target website from which information is to be collected, storing a scenario in which the location of the information to be collected is set, and continuously collecting information from the target website based on the scenario; a plurality of modules for searching for the location of the information to be collected are prepared in advance, and when it is detected that the information collection cannot be continued, a method for searching for the location set in the scenario is assigned from among the plurality of modules; Searching for the location of the information to be collected from the target website based on the assigned modules; an information gathering system for recovering a scenario stored in said storage device by changing the location of said information to be gathered based on said search results;

8. An information gathering method for a system that gathers information from a website via a network, comprising: the system includes a storage device, and a step of storing a scenario in the storage device, the scenario setting the location of information to be collected by analyzing a structure of a target website from which information is to be collected, and continuously collecting information from the target website based on the scenario; preparing a plurality of modules for searching for the location of the information to be collected in advance, and allocating a method for searching for the location set in the scenario from among the plurality of modules when it is detected that the information collection cannot be continued; searching for the location of the information to be collected from the target website based on the assigned modules; and changing the location of the information to be collected based on the search result, and restoring the scenario stored in the storage device.

Images