System, method and program for automatically crawling websites

Abstract

To provide a system of supporting setting of appropriate input values for input items in automatic crawling of web sites.SOLUTION: A website inspection server 10 provides a user operating a user terminal 30 communicably connected via a communication network 20, with a website inspection service for inspecting a website 22 specified by the user. When automatically crawling the web site 22, the server 10 identifies input items on web pages, where the input values set for the input items are set on the basis of information output from a learned model, so as to facilitate use of appropriate input values for various websites, compared with a case of using pre-prepared input values.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a system, method, and program for automatically crawling websites. [Background technology]

[0002] Conventionally, automatic crawling (sometimes called "crawling") of websites has been performed for various purposes (see, for example, Patent Document 1 below). In such automatic crawling, for example, a transition to the next web page is made by selecting an element such as a button that is included in a web page and has a link, script, or the like set therein. Also, when an input item (for example, an inputtable form element, etc.) exists on a web page, it is necessary to input an appropriate value and send it, and such input value is set to, for example, a value prepared in advance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-134094 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the methods for checking input values ​​set for input items vary from website to website, making it difficult to prepare appropriate input values ​​for all input items in advance. Furthermore, if an unexpected error occurs due to an inappropriate input value being set, the automatic crawling will be interrupted and manual follow-up will be required.

[0005] One object of the embodiment of the present invention is to support the setting of appropriate input values ​​for input items in automatic crawling of websites. Other objects of the embodiment of the present invention will become apparent by referring to the entire specification. [Means for solving the problem]

[0006] A system according to one embodiment of the present invention is a system for automatically crawling a website, comprising one or more computer processors, wherein the one or more computer processors execute the steps of: acquiring a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and acquiring a second web page transitioned to from the first web page accompanied by sending the set input values, wherein the step of setting the input values ​​includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

[0007] A method according to one embodiment of the present invention is executed by one or more computers and is a method for automatically crawling a website, comprising the steps of: obtaining a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and obtaining a second web page transitioned to from the first web page accompanied by sending the set input value, wherein the step of setting the input value includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

[0008] A program according to one embodiment of the present invention is a program for automatically crawling a website, which causes one or more computers to execute the steps of obtaining a first web page included in the website, identifying input items in the first web page and setting input values ​​for the input items, and obtaining a second web page transitioned to from the first web page along with sending the set input values, wherein the step of setting the input values ​​includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model. Effect of the Invention

[0009] Various embodiments of the present invention assist in setting appropriate input values ​​for input items in automated crawling of a website. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a schematic configuration of a network including a website inspection server 10 according to an embodiment of the present invention. [Diagram 2] 4 is a diagram illustrating an example of information managed by the examination management table 152. [Diagram 3] FIG. 4 is a diagram illustrating an example of information managed by a page management table 154. [Figure 4] 4 is a diagram illustrating an example of information managed by an error management table 156. [Diagram 5] 4 is a flowchart illustrating a process executed by the website inspection server 10 when inspecting the vulnerability of a website. [Figure 6] FIG. 4 is a diagram illustrating a target site setting screen 50. [Figure 7] 4 is a flowchart illustrating a process executed by the website inspection server 10 during automatic crawling of websites. [Figure 8] FIG. 13 is a diagram showing an example of input / output information in a large-scale language model capable of generating input values ​​for input items. [Figure 9] FIG. 11 is a diagram showing another example of input / output information in a large-scale language model. [Figure 10] FIG. 13 is a diagram showing yet another example of input / output information in a large-scale language model. [Figure 11] FIG. 6 is a diagram illustrating an example of a test result confirmation screen 60. [Figure 12] FIG. 13 is a diagram showing an example of input / output information in a large-scale language model when only an input value is output. [Figure 13] FIG. 13 is a diagram showing an example of input / output information in a large-scale language model that outputs error extension information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing, the same reference numerals may be used to denote the same or similar components.

[0012] FIG. 1 is a diagram showing a schematic configuration of a network including a website inspection server 10 according to an embodiment of the present invention. As shown in the figure, the server 10 is communicatively connected to a user terminal 30 via a communication network 20 such as the Internet. Although only one user terminal 30 is shown in FIG. 1, the server 10 is communicatively connected to a plurality of user terminals 30. The website inspection server 10 provides a website inspection service for inspecting a website 22 specified by a user operating the user terminal 30. The website inspection server 10 is an example of a device that implements all or a part of the system of the present invention.

[0013] First, we will explain the hardware configuration of the website inspection server 10. The website inspection server 10 is configured as a general computer, and as shown in Fig. 1, includes a computer processor 11, a main memory 12, an input / output I / F 13, a communication I / F 14, and a storage (storage device) 15, and these components are electrically connected via a bus or the like (not shown).

[0014] The computer processor 11 is configured as a CPU, a GPU, or the like, and loads various programs stored in a storage 15 or the like into a main memory 12, and executes various instructions included in the programs. The main memory 12 is configured, for example, by a DRAM or the like.

[0015] The input / output I / F 13 includes various input / output devices for exchanging information with an operator, etc. The input / output I / F 13 includes, for example, an information input device such as a keyboard, a pointing device (e.g., a mouse, a touch panel, etc.), an audio input device such as a microphone, and an image input device such as a camera. The input / output I / F 13 also includes an image output device such as a display, and an audio output device such as a speaker.

[0016] The communication I / F 14 is implemented as hardware such as a network adapter, various communication software, or a combination of these, and is configured to realize wired or wireless communication via a communication network 20 or the like.

[0017] The storage 15 is configured, for example, by a magnetic disk or a flash memory. The storage 15 stores various programs including an operating system, various data, and the like. For example, as shown in Fig. 1, the storage 15 has an inspection management table 152 for managing information related to an inspection, a page management table 154 for managing information related to web pages included in the website 22 to be inspected, and an error management table 156 for managing information related to errors that occur during automatic crawling of the website 22. Some of these tables may be integrated into one table, or may be divided into multiple tables.

[0018] Furthermore, for example, the storage 15 stores a server-side program 40 according to an embodiment of the present invention. The program 40 is a program for causing the server 10 to function as all or part of a system for providing a website inspection service. At least a part of the server-side program 40 can be configured to be executed on the user terminal 30 side via a terminal-side program 42, which will be described later.

[0019] In this embodiment, the website inspection server 10 may be configured using a plurality of computers each having the above-mentioned hardware configuration. For example, the server 10 is configured by a plurality of server devices.

[0020] The website inspection server 10 thus configured can be configured to have the functions of a web server and an application server, and executes various processes in response to requests from the user terminal 30, and transmits screen data (e.g., HTML data) and control data according to the results of the processes to the user terminal 30. The user terminal 30 outputs a web page or other screen based on the received data.

[0021] Next, a description will be given of the hardware configuration of the user terminal 30. The user terminal 30 is configured as a general computer, and as shown in Fig. 1, includes a computer processor 31, a main memory 32, an input / output I / F 33, a communication I / F 34, and a storage (storage device) 35, and these components are electrically connected via a bus or the like (not shown).

[0022] The computer processor 31 is configured as a CPU, a GPU, or the like, and loads various programs stored in a storage 35 or the like into a main memory 32 and executes various instructions included in the programs. The main memory 32 is configured, for example, by a DRAM or the like.

[0023] The input / output I / F 33 includes various input / output devices for exchanging information with an operator, etc. The input / output I / F 33 includes, for example, an information input device such as a keyboard or a pointing device (e.g., a mouse, a touch panel, etc.), an audio input device such as a microphone, and an image input device such as a camera. The input / output I / F 33 also includes an image output device such as a display, and an audio output device such as a speaker.

[0024] The communication I / F 34 is implemented as hardware such as a network adapter, various communication software, or a combination of these, and is configured to realize wired or wireless communication via the communication network 20 or the like.

[0025] The storage 35 is configured, for example, by a magnetic disk or a flash memory. The storage 35 stores various programs including an operating system, various data, and the like. The programs stored in the storage 35 can be downloaded from an application market or the like and installed. The storage 35 also stores the above-mentioned terminal-side program 42. The program 42 can be configured as a web browser or other application (for example, a terminal-side application for the website inspection service of this embodiment, etc.) and can be configured to execute at least a part of the server-side program 40 as described above.

[0026] In this embodiment, the user terminal 30 may be configured as a smartphone, a tablet terminal, a personal computer, or the like.

[0027] A user (e.g., an administrator of a website 22) operating a user terminal 30 configured in this manner can use the website inspection service provided by the server 10 by communicating with the server 10 via a terminal-side program 42 installed in storage 35 or the like.

[0028] Next, a description will be given of the functions of the website inspection server 10 configured as above. The computer processor 11 of the server 10 is configured to function as a management function control unit 112, an automatic patrol control unit 114, and an inspection control unit 116 by executing instructions included in a program (e.g., at least a part of the server-side program 40) loaded into the main memory 12, as shown in FIG.

[0029] The management function control unit 112 is configured to execute various processes related to the control of the management function of the website inspection service. For example, the management function control unit 112 transmits screen data and control data of various screens related to the management function to the user terminal 30, executes various processes in response to an operation input by a user via the screen output on the user terminal 30, and transmits screen data and control data according to the results of the processes to the user terminal 30. The management functions controlled by the management function control unit 112 include, for example, login processing (user authentication), billing control, and management of user accounts.

[0030] The automatic patrol control unit 114 is configured to execute various processes related to the control of automatic patrol of the website 22. For example, the automatic patrol control unit 114 acquires a web page included in the website 22, identifies a transition target element included in the page (including, for example, various types of objects (buttons, images, text, etc.) to which links or scripts, etc. are set), selects (clicks) the transition target element to acquire a transition destination web page, and repeats these processes to patrol the website 22.

[0031] In this embodiment, the automatic cycling control unit 114 is configured to, when an input item (e.g., an inputtable form element) exists on a web page, set an input value for the input item, and transition to a destination web page accompanied by transmission of the input value. That is, the automatic cycling control unit 114 identifies an input item on a web page, sets an input value for the input item, and acquires a destination web page transitioned from the web page accompanied by transmission of the set input value.

[0032] In this embodiment, the input values ​​for the input items are set using a first trained model for setting such input values. For example, the automatic loop control unit 114 is configured to input input information including at least a part of page information on a web page to the first trained model, and set input values ​​for the input items in the web page based on output information output from the first trained model. Such trained models are generated through machine learning using a large amount of training data.

[0033] The inspection control unit 116 is configured to execute various processes related to control of the inspection of the website 22. For example, the inspection control unit 116 transmits various requests for inspecting predetermined inspection items in each web page acquired by automatic crawling of the website 22, and judges the corresponding inspection items based on the contents of the responses received from the website 22.

[0034] In this way, the website inspection server 10 in this embodiment identifies input items in a web page when automatically crawling the website 22, and sets the input values ​​for the input items based on information output from the trained model, which promotes the use of input values ​​appropriate for various websites compared to the use of input values ​​prepared in advance. In this way, the website inspection server 10 supports the setting of appropriate input values ​​in the automatic crawling of the website 22.

[0035] In this embodiment, the inspection of the website 22 includes various types of inspections. For example, the inspection includes an inspection (diagnosis) of vulnerabilities of the website 22. Also, for example, the inspection includes an inspection of the performance, functionality, and other aspects of the website 22.

[0036] In this embodiment, the first trained model for setting an input value for an input item may be configured to output various information. For example, the first trained model is configured to output the input value itself. Such a trained model is configured as a large-scale language model (LLM) that outputs information presenting the input value itself when information instructing the presentation of an input value for an input item on a webpage is input together with HTML data of the webpage. Such a configuration enables the use of the input value itself output from the trained model.

[0037] Also, for example, the first trained model is configured to output a combination of input items and input values. Such a trained model is configured as a large-scale language model (LLM) that outputs information presenting a combination of input items and input values ​​in a web page when the combination is input together with HTML data of the web page. That is, in this case, the input items in the web page are identified using the first trained model, and the automatic loop control unit 114 is configured to set, for each input item output from the first trained model, an input value corresponding to each input item, which is also output from the first trained model. Such a configuration enables the use of a combination of input items and input values ​​output from the trained model.

[0038] Also, for example, the first trained model may be configured to output error possibility information indicating the possibility of an error occurring when a combination of an input item and an input value is input and the input value is set for the input item. Such a trained model is generated through machine learning using, for example, a large amount of training data (for example, page information (HTML data, etc.) of a webpage, input items in the webpage, input values ​​set for the input items, and the presence or absence of an error after the input value is sent, etc.), and is configured to output a score indicating the possibility of an error as the error possibility information. In this case, the automatic loop control unit 114 is configured to determine the possibility of an error for each of a plurality of input value candidates prepared in advance that can be set for a specific input item, based on the error possibility information output from the first trained model, and set the input value so that a candidate with a low possibility of an error is prioritized. Such a configuration enables the setting of an input value based on the error possibility information output from the trained model.

[0039] The automatic loop control unit 114 may be configured to determine an error based on page information on a destination web page that involves the transmission of an input value, and to set a new input value for an input item on the source web page if an error is determined to exist. This configuration enables the input value to be reset when an error is determined to exist based on the page information.

[0040] In this embodiment, the determination of an error based on page information on a transition destination web page may also be performed using a trained model generated through machine learning. For example, the automatic navigation control unit 114 may be configured to input input information including at least a part of page information on a transition destination web page to the second trained model, and determine an error based on output information output from the second trained model. For example, the second trained model outputs information on the presence or absence of an error (such as a score indicating the possibility of an error), and the presence or absence of an error is determined based on the information (for example, it is determined that an error exists when the score indicating the possibility of an error is equal to or greater than a threshold). Such a configuration enables the determination of an error via a trained model.

[0041] In this embodiment, a plurality of input items in a web page may be identified, and a corresponding input value may be set for each of the plurality of input items. In this case, the second trained model for determining an error may be configured to output information for identifying an input item that is the cause of an error among the plurality of input items (e.g., for each of the plurality of input items, information indicating the possibility that the input item is the cause of an error, or information identifying the input item itself, etc.). Such a configuration makes it possible to identify the input item that is the cause of an error via the trained model.

[0042] Furthermore, the automatic navigation control unit 114 may be configured to add data including a combination of input items in the transition source web page and input values ​​set and transmitted for the input items as training data for the second trained model when it is determined that there is an error. Such a configuration supports appropriate additional training of the trained model for determining errors.

[0043] Furthermore, the automatic cycling control unit 114 can be configured to present information about an error to a user when it is determined that an error has occurred. For example, the automatic cycling control unit 114 presents the information about the error to the user by transmitting screen data of a screen including the information about the error (e.g., managed by the error management table 156) and control data to the user terminal 30. This configuration makes it possible to inform the user (e.g., the administrator of the website 22) of the information about the error.

[0044] Next, a specific example of one aspect of the website inspection server 10 of this embodiment having such functions will be described. The server 10 in this specific example provides a vulnerability inspection service that inspects the vulnerability of websites as an example of a website inspection service.

[0045] First, the information managed by each table in this example will be described. Fig. 2 illustrates information managed by the examination management table 152 in this example. The examination management table 152 in this example manages information related to individual examinations, and as shown in the figure, manages information such as a "top screen URL" which is the URL of the top screen (top page) that is the starting point of the examination of the website to be examined, a "screen transition setting date and time" which is the setting date and time of the screen transition, and an "examination execution date and time" which is the execution date and time of the examination, in association with an "examination ID" which identifies an individual examination (case).

[0046] 3 illustrates information managed by the page management table 154 in this example. The page management table 154 in this example manages information about each web page (screen) included in the website to be inspected, and as illustrated, manages information such as a "screen URL" which is the URL of the screen, "screen capture information" which is information about a screen capture of the screen, "HTML data" of the screen, a "source screen ID" which identifies the source screen of the screen, a "source element ID" which identifies the source element of the screen (a transition target element clicked when transitioning to the screen), "source input information" which is information about input values ​​input to input items on the source screen, and "inspection result information" which is information about the inspection result on the screen, in association with a combination of an "inspection ID" which identifies an individual inspection and a "screen ID" which identifies an individual screen in the inspection.

[0047] 4 illustrates information managed by the error management table 156 in this example. The error management table 156 in this example manages information related to errors that have occurred during automatic crawling of a website to be inspected, and as illustrated, manages information such as a "screen URL" which is the URL of the screen on which the error occurred, "screen capture information" which is information related to a screen capture of the screen, "HTML data" of the screen, a "source screen ID" which identifies the source screen of the screen, a "source element ID" which identifies the source element of the screen (a transition target element clicked when transitioning to the screen), "source input information" which is information related to an input value input to an input item on the source screen, and an "error score" which indicates the possibility of an error occurring on the screen, in association with a combination of an "inspection ID" which identifies an individual inspection and an "error ID" which identifies an individual error in the inspection.

[0048] The information managed by each table in this example has been described above. Next, the process executed by the website inspection server 10 and the screens output on the user terminal 30 in this example will be described.

[0049] 5 is a flowchart illustrating the process executed by the website inspection server 10 when inspecting the vulnerability of a website in this example. As shown in the figure, the server 10 first accepts the settings of the website to be inspected for vulnerability (step S100). The settings are accepted via a screen displayed on the user terminal 30.

[0050] 6 illustrates a target site setting screen 50 for accepting settings of a website to be inspected, which is output on the user terminal 30. As illustrated, the screen 50 has a URL input area 52 for inputting the URL of the top screen (top page) of the website to be inspected, a login ID input area 54 for inputting a login ID used to log in to the website, a password input area 56 for inputting a password used to log in to the website, and a continue button 58 with the text "Start patrol and inspection" added. Thus, in this example, the URL of the top screen, the login ID, and the password are accepted as setting information of the website to be inspected.

[0051] The continue button 58 is an object for instructing automatic crawling and inspection of the website to be inspected. Returning to the flowchart of Fig. 5, when the button 58 is selected, the server 10 next automatically crawls the website to be inspected and sets the screen transition for the inspection (step S110).

[0052] FIG. 7 is a flow chart illustrating a process executed by the server 10 during automatic crawling of a website. As shown in the figure, the server 10 first accesses the top screen of the website to be inspected and adds the top screen to the screen transitions (step S200). Specifically, a web browser is started in the server 10, and the top screen specified by the URL input via the target site setting screen 50 is accessed. Information about the top screen is registered in the page management table 154, and specifically, the inspection ID, screen ID, screen URL, screen capture information, and HTML data are set. In this way, the page management table 154 can also be said to be a table that manages the screen transitions set in each individual inspection.

[0053] Next, the server 10 identifies a plurality of transition target elements included in the top screen (step S210). Specifically, screen components (buttons, images, text, etc.) to which links or scripts are set are identified as transition target elements. Such elements are identified based on the corresponding HTML data, and for example, elements corresponding to a button tag, an img tag, an input tag (type="button"), etc. are candidates for transition target elements.

[0054] Next, if an input item (e.g., an inputtable form element) exists (YES in step S220), the server 10 generates an input value for the input item and sets the generated input value (steps S230, S240). The presence or absence of an input item is determined based on the corresponding HTML data, and for example, an input tag (type="text") and a textarea tag are candidates for the input item.

[0055] In addition, the input values ​​for the input items are specifically generated using a trained model capable of generating such input values. In this example, the trained model is configured as a large-scale language model, and is configured so that when information instructing the presentation of a combination of input items and input values ​​on a target screen is input together with HTML data of the target screen, each input item included in the screen is output together with a corresponding input value (i.e., a combination of input items and input values ​​is output). For example, GPT-4 provided by OpenAI, Inc. of the United States may be applied as such a large-scale language model.

[0056] 8 shows an example of input / output information in a large-scale language model. As shown in the figure, by instructing output in JSON format in the input information, it becomes possible to set an input value based on the output information.

[0057] 9 shows another example of input / output information in a large-scale language model. As shown in the figure, even if the parameter names ("param1" and "param2") in the HTML data do not indicate the purpose of the parameter, the large-scale language model can determine the purpose of the parameter based on surrounding elements such as labels. Also, even if the input rules are specified in natural language (such as "Please specify in half-width katakana and do not leave a space between the family name and the given name"), the large-scale language model can present an input value that complies with such input rules.

[0058] 10 shows yet another example of input and output information in a large-scale language model. As shown in the figure, even if there is a typo in the parameter name in the HTML data ("lsat_name" is a typo for "last_name"), the large-scale language model can predict such a typo and determine the use of the parameter.

[0059] Next, the server 10 selects (clicks) the corresponding transition target element (step S250). Note that if there is no input item (NO in step S220), the transition target element is immediately selected.

[0060] Next, the server 10 judges an error on the transition destination screen (step S260). Specifically, such an error is judged using a trained model for judging the error. In this example, the trained model is generated through machine learning using a large amount of training data, and is configured to output an error score indicating the possibility of an error occurring when the HTML data of the transition source screen, an HTTP request (including the input value set for the input item), and an HTTP response (including the HTML data of the transition destination screen) are input. Then, if the value of the error score is equal to or greater than a threshold, it is judged that an error has occurred, and if the value of the error score is less than the threshold, it is judged that an error has not occurred.

[0061] For example, when an error message is displayed only in natural language (words that cannot be listed in advance) (such as a message saying "The process could not be completed"), it is difficult to determine the error using rule-based methods, but by using a trained model, it is possible to determine such errors.

[0062] Then, if it is determined that an error has not occurred in the transition destination screen (NO in step S265), the server 10 adds a transition to the transition destination screen to the screen transition (step S270). Specifically, information on the transition destination screen is registered in the page management table 154, that is, the examination ID, screen ID, screen URL, screen capture information, HTML data, transition source screen ID, transition source element ID, and transition source input information are set. The transition source input information is set as a combination of input items and input values.

[0063] Here, even if no error occurs, if the transition destination screen is not a target screen for addition to the screen transition, it may not be added to the screen transition. For example, screens of sites outside the website being inspected and screens that overlap with screens already included in the screen transition (such overlaps are determined based on the similarity of the screens, for example) are determined to be not a target screen for addition to the screen transition.

[0064] On the other hand, when it is determined that an error has occurred in the transition destination screen (YES in step S265), the server 10 registers the error in the transition destination screen (step S280). Specifically, the test ID, error ID, screen URL, screen capture information, HTML data, transition source screen ID, transition source element ID, transition source input information, and error score are set in the error management table 156. Note that when an error has occurred, the screen is not added to the screen transition.

[0065] Then, when the transition destination screen is added to the screen transition according to the selection (click) or an error is registered for all of the multiple transition target elements included in the top screen (NO in step S290), the server 10 then identifies multiple transition target elements included in each screen added to the screen transition (step S310) in the same manner as the top screen, and adds the transition destination screen to the screen transition according to the selection of the element or registers an error (steps S220 to S290). This process is repeated until no new screens are added to the screen transition (NO in step S300).

[0066] Returning to the flowchart of FIG. 5, once the screen transitions are set in this way, the server 10 then executes vulnerability testing in accordance with the set screen transitions (step S120). Specifically, a web browser is started, and each screen included in the screen transition is visited in order in accordance with the information managed in the page management table 154, and testing is performed for predetermined vulnerability testing items. The vulnerability testing items include, for example, items defined in OWASP Top 10. The testing results are registered in the testing result information in the page management table 154. Note that testing for testing items may be performed on each screen during automatic crawling of the website (i.e., during setting of the screen transitions).

[0067] 11 illustrates an inspection result confirmation screen 60 for confirming the inspection results, which is output on the user terminal 30. As shown in the figure, the screen 60 has a list display area 62 that displays a list of vulnerability inspection results. The area 62 displays a list of vulnerability inspection results (severity, vulnerability name, detected location, vulnerable location, etc.) for each screen. This information is managed in the inspection result information of the page management table 154.

[0068] In the above example, the information managed in the error management table 156 is automatically added to the training data during additional learning of the above-mentioned trained model to determine errors on the transition destination screen.

[0069] In the above example, a screen including the information managed in the error management table 156 may be output on the user terminal 30 of the corresponding user (such as a website administrator).

[0070] In the above example, a combination of an input item and an input value is output from the trained model configured as a large-scale language model, but only the input value may be output. In this case, an input item is identified based on HTML data of a target screen or the like, and when information instructing the trained model to present an input value for the identified input item together with the HTML data is input to the trained model, information presenting the input value itself is output from the trained model. FIG. 12 shows an example of input / output information in the large-scale language model in this case. In this example, "param1" is a parameter of an input item (name) identified based on the HTML data of a target screen, and such input information is automatically generated by the server 10.

[0071] The trained model may be configured to output information (such as an error score) indicating the possibility of an error when HTML data and a combination of an input item and an input value are input. In this case, an input item is identified based on the HTML data of a target screen, etc., and for each of a plurality of input value candidates (predetermined) that can be set for the input item, the possibility of an error is obtained via the trained model, and the candidate with the lowest possibility is applied as the input value.

[0072] In the above example, the trained model for determining an error in the destination screen is configured to output an error score indicating the possibility of an error occurring when the HTML data, HTTP request, and HTTP response of the source screen are input, but information for identifying an input item that is the cause of the error among multiple input items may be output. For example, the trained model is configured as a large-scale language model, and when information instructing the presentation of the input item that is the cause of the error is input together with the HTML data, HTTP request, and HTTP response of the source screen, information presenting the input item is output from the trained model.

[0073] Further, extended information such as the reason for the error and an alternative value for the input value may be acquired from a trained model for determining the error. Such a trained model is configured as a large-scale language model, and when information instructing the presentation of such information is input together with the HTML data of the transition source screen and the input value for the input field, information presenting the extended information is output from the trained model. FIG. 13 shows an example of input / output information in the large-scale language model in this case. The extended information acquired in this manner is managed, for example, in the error management table 156.

[0074] In the above example, a vulnerability testing service is provided that tests for vulnerabilities in websites. However, in other examples of this embodiment, various tests are performed from perspectives other than vulnerabilities, for example, tests are performed on other performance and functions of the website.

[0075] The website inspection server 10 according to the present embodiment described above identifies input items in a web page when automatically crawling a website 22, and sets input values ​​for the input items based on information (e.g., a combination of input items and input values) output from a trained model, which promotes the use of input values ​​appropriate for various websites compared to the use of input values ​​prepared in advance. In this way, the website inspection server 10 supports the setting of appropriate input values ​​in the automatic crawling of the website 22.

[0076] In other embodiments of the present invention, some or all of the functions of the website inspection server 10 in the above-described embodiment may be realized by cooperation between the website inspection server 10 and the user terminal 30, or may be realized by the user terminal 30. In other embodiments of the present invention, automatic crawling of the website 22 is performed for various purposes other than inspection of the website 22.

[0077] The processes and procedures described herein may be realized by software, hardware, or any combination thereof, other than those explicitly described. For example, the processes and procedures described herein may be realized by implementing logic corresponding to the processes and procedures in a medium such as an integrated circuit, a volatile memory, a non-volatile memory, or a magnetic disk. In addition, the processes and procedures described herein may be implemented as a computer program corresponding to the processes and procedures, and executed by various computers.

[0078] Although processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, software, components, and / or modules. Also, the software and hardware elements described herein may be realized by combining them into fewer components or breaking them down into more components.

[0079] In this specification, even if a component of the invention is described as either singular or plural, or described without limitation to either singular or plural, the component may be either singular or plural unless the context requires otherwise. [Explanation of symbols]

[0080] 10 Website Inspection Server 11. Computer Processors 112 Management function control section 114 Automatic patrol control unit 116 Inspection Control Unit 15. Storage 152 Inspection Management Table 154 Page Management Table 156 Error Management Table 30 User terminal 40 Server-side programs 42 Terminal side program 50 Target site setting screen 60 Test result confirmation screen

Claims

1. 1. A system for automatically crawling web sites, comprising one or more computer processors, the one or more computer processors comprising: obtaining a first web page included in the website; identifying an input item on the first web page and setting an input value for the input item; acquiring a second web page transitioned from the first web page in response to the transmission of the set input value; The step of setting the input value includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model. system.

2. The first output information includes the input value. The system of claim 1.

3. the first output information includes a combination of the input items and the input values; the step of setting the input value includes setting the input value included in the first output information for the input item included in the first output information. The system of claim 1.

4. the first input information includes a combination of the input items and the input values; the first output information includes error possibility information indicating a possibility of an error occurring when the input value is set for the input item, The step of setting the input value includes determining whether to set the input value based on the error probability information. The system of claim 1.

5. The one or more computer processors further include: determining an error based on second page information relating to the second web page; and if it is determined that there is an error, setting a new input value for the input item. The system of claim 1.

6. The step of determining the error includes inputting second input information including at least a portion of the second page information to a second trained model, and determining the error based on second output information output from the second trained model. The system of claim 5.

7. The second output information includes information regarding the presence or absence of the error. The system of claim 6.

8. The step of setting the input value includes identifying a plurality of input items in the first web page and setting a corresponding input value for each of the plurality of input items; the second output information includes information regarding an input item that is a cause of the error among the plurality of input items; The system of claim 6.

9. The one or more computer processors further execute, when it is determined that the error exists, a step of adding data including the input items and the combination of the input values ​​set for the input items and transmitted as training data of the second trained model. The system of claim 6.

10. The one or more computer processors further perform the step of, if it is determined that there is an error, presenting information about the error to a user. The system of claim 5.

11. 1. A method, executed by one or more computers, for automated crawling of web sites, comprising: obtaining a first web page included in the website; identifying an input item on the first web page and setting an input value for the input item; acquiring a second web page transitioned from the first web page in response to the transmission of the set input value; The step of setting the input value includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model. method.

12. A program for automatically crawling websites, comprising: obtaining a first web page included in the website; identifying an input item on the first web page and setting an input value for the input item; acquiring a second web page transitioned from the first web page in response to the transmission of the set input value; The step of setting the input value includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model. program.

Images