System, method, and program for performing automatic crawling of website

Abstract

A website inspection server 10 according to one embodiment of the present invention supports setting of an appropriate input value for an input item in automatic crawling of a website. The server 10 provides, to a user who operates a user terminal 30 communicatively connected to the server 10 via a communication network 20, a website inspection service for inspecting a website 22 specified by the user. The server 10 specifies an input item in a web page when automatically crawling the website 22, and an input value to be set for the input item is set on the basis of information output from a trained model. Therefore, use of input values appropriate for various websites is promoted compared with when input values prepared in advance are used.

Description

System, method, and program for automatically crawling websites

[0001] The present invention relates to a system, method, and program for automatically crawling websites.

[0002] Conventionally, automatic crawling (sometimes called "crawling") of websites has been performed for various purposes (see, for example, Patent Document 1 below). In such automatic crawling, for example, a transition to the next web page is made by selecting an element, such as a button with a link or script, that is included in the web page. Also, if the web page has an input item (for example, an inputtable form element), it is necessary to input an appropriate value and submit the input item. For example, a value prepared in advance is set as the input value.

[0003] Japanese Patent Application Laid-Open No. 2022-134094

[0004] However, the methods for checking input values ​​set for input fields vary depending on the website, making it difficult to prepare appropriate input values ​​for all input fields in advance. Furthermore, if an unexpected error occurs due to an inappropriate input value, the automatic crawling will be interrupted and manual follow-up will be required.

[0005] One object of the present invention is to support the setting of appropriate input values ​​for input items in automatic crawling of websites. Other objects of the present invention will become apparent by reading the entire specification.

[0006] A system according to one embodiment of the present invention is a system for automatically crawling a website, comprising one or more computer processors, wherein the one or more computer processors execute the steps of: acquiring a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and acquiring a second web page transitioned to from the first web page accompanied by transmission of the set input values, wherein the step of setting the input values ​​includes inputting first input information including at least a portion of first page information regarding the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

[0007] A method according to one embodiment of the present invention is a method for automatically crawling a website, executed by one or more computers, comprising the steps of: acquiring a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and acquiring a second web page transitioned to from the first web page while transmitting the set input values. The step of setting the input values ​​includes inputting first input information including at least a portion of first page information regarding the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

[0008] A program according to one embodiment of the present invention is a program for automatically crawling a website, which causes one or more computers to execute the following steps: acquiring a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and acquiring a second web page transitioned to from the first web page while transmitting the set input values. The step of setting the input values ​​includes inputting first input information including at least a portion of first page information regarding the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

[0009] Various embodiments of the present invention assist in setting appropriate input values ​​for input items in automated crawling of a website.

[0010] 1 is a block diagram showing an outline of the configuration of a network including a website inspection server 10 according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of information managed by an inspection management table 152. FIG. 2 is a diagram illustrating an example of information managed by a page management table 154. FIG. 3 is a diagram illustrating an example of information managed by an error management table 156. FIG. 4 is a flowchart illustrating a process executed by the website inspection server 10 when inspecting website vulnerabilities. FIG. 5 is a diagram illustrating a target site setting screen 50. FIG. 6 is a flowchart illustrating a process executed by the website inspection server 10 when automatically crawling websites. FIG. 7 is a diagram showing an example of input / output information in a large-scale language model capable of generating input values ​​for input items. FIG. 8 is a diagram showing another example of input / output information in a large-scale language model. FIG. 9 is a diagram showing yet another example of input / output information in a large-scale language model. FIG. 10 is a diagram showing an example of an inspection result confirmation screen 60. FIG. 11 is a diagram showing an example of input / output information in a large-scale language model when only an input value is output. FIG. 12 is a diagram showing an example of input / output information in a large-scale language model that outputs error extension information.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same reference numerals may be used to designate the same or similar components.

[0012] FIG. 1 is a diagram illustrating a schematic configuration of a network including a website inspection server 10 according to an embodiment of the present invention. As illustrated, the server 10 is communicatively connected to a user terminal 30 via a communication network 20 such as the Internet. Although only one user terminal 30 is illustrated in FIG. 1, the server 10 is communicatively connected to a plurality of user terminals 30. The website inspection server 10 provides a website inspection service to a user operating the user terminal 30, inspecting a website 22 specified by the user. The website inspection server 10 is an example of a device that implements all or part of the system of the present invention.

[0013] First, we will explain the hardware configuration of the website inspection server 10. The website inspection server 10 is configured as a general computer, and as shown in Fig. 1, it includes a computer processor 11, a main memory 12, an input / output I / F 13, a communication I / F 14, and a storage (storage device) 15, and these components are electrically connected via a bus or the like (not shown).

[0014] The computer processor 11 is configured as a CPU, a GPU, or the like, and loads various programs stored in the storage 15 or the like into the main memory 12 and executes various instructions included in the programs. The main memory 12 is configured, for example, by a DRAM or the like.

[0015] The input / output I / F 13 includes various input / output devices for exchanging information with an operator, etc. The input / output I / F 13 includes, for example, information input devices such as a keyboard and a pointing device (e.g., a mouse, a touch panel, etc.), audio input devices such as a microphone, and image input devices such as a camera. The input / output I / F 13 also includes an image output device such as a display, and an audio output device such as a speaker.

[0016] The communication I / F 14 is implemented as hardware such as a network adapter, various communication software, or a combination of these, and is configured to enable wired or wireless communication via a communication network 20 or the like.

[0017] The storage 15 is configured, for example, by a magnetic disk or a flash memory. The storage 15 stores various programs including an operating system, various data, and the like. For example, as shown in FIG. 1 , the storage 15 has an inspection management table 152 that manages information about inspections, a page management table 154 that manages information about web pages included in the website 22 to be inspected, and an error management table 156 that manages information about errors that occur during automatic crawling of the website 22. Some of these tables may be integrated into one table, or may be divided into multiple tables.

[0018] Furthermore, for example, the storage 15 stores a server-side program 40 according to an embodiment of the present invention. The program 40 is a program for causing the server 10 to function as all or part of a system for providing a website inspection service. At least a part of the server-side program 40 can be configured to be executed on the user terminal 30 via a terminal-side program 42, which will be described later.

[0019] In this embodiment, the website inspection server 10 may be configured using a plurality of computers each having the above-described hardware configuration. For example, the server 10 may be configured by a plurality of server devices.

[0020] The website inspection server 10 configured in this manner can be configured to function as a web server and an application server, and executes various processes in response to requests from the user terminal 30, and transmits screen data (e.g., HTML data) and control data according to the results of the processes to the user terminal 30. The user terminal 30 outputs a web page or other screen based on the received data.

[0021] Next, we will explain the hardware configuration of the user terminal 30. The user terminal 30 is configured as a general computer, and as shown in Fig. 1, it includes a computer processor 31, a main memory 32, an input / output I / F 33, a communication I / F 34, and a storage (storage device) 35, and these components are electrically connected via a bus or the like (not shown).

[0022] The computer processor 31 is configured as a CPU, a GPU, or the like, and loads various programs stored in the storage 35 or the like into the main memory 32 and executes various instructions included in the programs. The main memory 32 is configured, for example, by a DRAM or the like.

[0023] The input / output I / F 33 includes various input / output devices for exchanging information with an operator, etc. The input / output I / F 33 includes, for example, information input devices such as a keyboard and a pointing device (e.g., a mouse, a touch panel, etc.), audio input devices such as a microphone, and image input devices such as a camera. The input / output I / F 33 also includes an image output device such as a display, and an audio output device such as a speaker.

[0024] The communication I / F 34 is implemented as hardware such as a network adapter, various communication software, or a combination thereof, and is configured to realize wired or wireless communication via the communication network 20 or the like.

[0025] The storage 35 is configured, for example, by a magnetic disk or flash memory. The storage 35 stores various programs including an operating system, various data, and the like. The programs stored in the storage 35 can be downloaded and installed from an application market or the like. The storage 35 also stores the above-mentioned terminal-side program 42. The program 42 can be configured as a web browser or other application (e.g., a terminal-side application for the website inspection service of this embodiment), and can be configured to execute at least a part of the server-side program 40, as described above.

[0026] In this embodiment, the user terminal 30 may be configured as a smartphone, a tablet terminal, a personal computer, or the like.

[0027] A user (e.g., an administrator of website 22) operating a user terminal 30 configured in this manner can use the website inspection service provided by server 10 by communicating with server 10 via a terminal-side program 42 installed in storage 35 or the like.

[0028] Next, we will explain the functions of the website inspection server 10 configured as above. As shown in Figure 1, the computer processor 11 of the server 10 is configured to function as a management function control unit 112, an automatic patrol control unit 114, and an inspection control unit 116 by executing instructions included in a program (e.g., at least a part of the server-side program 40) loaded into the main memory 12.

[0029] The management function control unit 112 is configured to execute various processes related to the control of the management functions of the website inspection service. For example, the management function control unit 112 transmits screen data and control data of various screens related to the management functions to the user terminal 30, executes various processes in response to user operation inputs via the screens output on the user terminal 30, and transmits screen data and control data according to the results of the processes to the user terminal 30. The management functions controlled by the management function control unit 112 include, for example, login processing (user authentication), billing control, and user account management.

[0030] The automatic patrol control unit 114 is configured to execute various processes related to the control of automatic patrol of the website 22. For example, the automatic patrol control unit 114 acquires a web page included in the website 22, identifies transition target elements included in the page (including, for example, various types of objects (buttons, images, text, etc.) to which links or scripts, etc. are set), selects (clicks) the transition target element to acquire the transition destination web page, and repeats these processes to patrol the website 22.

[0031] In this embodiment, when an input item (e.g., an inputtable form element) exists on a web page, the automatic patrol control unit 114 is configured to set an input value for the input item, transmit the input value, and transition to a destination web page. That is, the automatic patrol control unit 114 identifies an input item on a web page, sets an input value for the input item, and acquires a destination web page transitioned from the web page, transmitting the set input value.

[0032] In this embodiment, input values ​​for input items are set using a first trained model for setting such input values. For example, the automatic navigation control unit 114 is configured to input input information including at least a portion of page information about a web page to the first trained model, and to set input values ​​for input items in the web page based on output information output from the first trained model. Such a trained model is generated through machine learning using a large amount of training data.

[0033] The inspection control unit 116 is configured to execute various processes related to the control of inspection of the website 22. For example, the inspection control unit 116 transmits various requests for inspecting predetermined inspection items for each web page acquired by automatic crawling of the website 22, and determines the corresponding inspection items based on the contents of the responses received from the website 22.

[0034] In this way, when automatically crawling a website 22, the website inspection server 10 in this embodiment identifies input items on a web page and sets input values ​​for those input items based on information output from the trained model, which promotes the use of input values ​​appropriate for various websites compared to using input values ​​prepared in advance. In this way, the website inspection server 10 supports the setting of appropriate input values ​​when automatically crawling a website 22.

[0035] In this embodiment, the inspection of the website 22 includes various types of inspections, such as an inspection (diagnosis) of vulnerabilities of the website 22. Also, for example, the inspection includes an inspection of the performance, functionality, and other aspects of the website 22.

[0036] In this embodiment, the first trained model for setting an input value for an input item may be configured to output various information. For example, the first trained model may be configured to output the input value itself. Such a trained model may be configured as a large-scale language model (LLM) that, when information instructing the presentation of an input value for an input item on a web page is input together with HTML data of the web page, outputs information presenting the input value itself. This configuration enables the use of the input value itself output from the trained model.

[0037] Furthermore, for example, the first trained model is configured to output a combination of input items and input values. For example, such a trained model is configured as a large-scale language model (LLM) that, when information instructing the presentation of a combination of input items and input values ​​on a web page is input together with the HTML data of the web page, outputs information presenting the combination. In other words, in this case, the input items on the web page are identified using the first trained model, and the automatic navigation control unit 114 is configured to set, for each input item output from the first trained model, an input value corresponding to each input item, which is also output from the first trained model. This configuration enables the use of combinations of input items and input values ​​output from the trained model.

[0038] Furthermore, for example, the first trained model may be configured, when a combination of input items and input values ​​is input, to output error probability information indicating the possibility of an error occurring when the input values ​​are set for the input items. Such a trained model may be generated, for example, through machine learning using a large amount of training data (e.g., page information (e.g., HTML data) of a webpage, input items on the webpage, input values ​​set for the input items, and the presence or absence of errors after the input values ​​are submitted), and configured to output a score indicating the possibility of an error as the error probability information. In this case, the automatic navigation control unit 114 is configured to determine the possibility of an error for each of multiple prepared input value candidates that can be set for a specific input item based on the error probability information output from the first trained model, and set the input value so that candidates with a low probability of an error are prioritized. This configuration enables setting of input values ​​based on the error probability information output from the trained model.

[0039] The automatic loop control unit 114 may be configured to determine whether an error exists based on page information about a destination web page that involves the transmission of an input value, and to set a new input value for the input item on the source web page if an error exists. This configuration enables the input value to be reset when an error is determined to exist based on the page information.

[0040] In this embodiment, the determination of an error based on page information about a destination web page may also be performed using a trained model generated through machine learning. For example, the automatic navigation control unit 114 may be configured to input input information including at least a portion of the page information about the destination web page to a second trained model and determine an error based on output information output from the second trained model. For example, the second trained model outputs information about the presence or absence of an error (such as a score indicating the possibility of an error), and the presence or absence of an error is determined based on the information (for example, an error is determined if the score indicating the possibility of an error is equal to or greater than a threshold). This configuration enables the determination of an error via a trained model.

[0041] In this embodiment, multiple input items on a web page may be identified, and corresponding input values ​​may be set for each of these multiple input items. In this case, the second trained model for determining an error may be configured to output information for identifying an input item that is the cause of the error (e.g., for each of the multiple input items, information indicating the possibility that the input item is the cause of the error, or information identifying the input item itself). This configuration makes it possible to identify the input item that is the cause of the error via the trained model.

[0042] Furthermore, when it is determined that an error exists, the automatic navigation control unit 114 may be configured to add data including a combination of input items in the web page from which the transition is made and input values ​​set and transmitted for the input items as training data for the second trained model. Such a configuration supports appropriate additional training of the trained model for determining errors.

[0043] Furthermore, the automatic patrol control unit 114 can be configured to present information about an error to the user when it is determined that an error has occurred. For example, the automatic patrol control unit 114 presents the information to the user by transmitting screen data and control data of a screen containing information about the error (e.g., managed by the error management table 156) to the user terminal 30. This configuration makes it possible to notify the user (e.g., the administrator of the website 22) of the information about the error.

[0044] Next, a specific example of one aspect of the website inspection server 10 of this embodiment having such functions will be described. The server 10 in this specific example provides a vulnerability inspection service that inspects websites for vulnerabilities as an example of a website inspection service.

[0045] First, the information managed by each table in this example will be described. Fig. 2 illustrates information managed by the inspection management table 152 in this example. The inspection management table 152 in this example manages information related to individual inspections, and as shown in the figure, manages information such as a "top screen URL," which is the URL of the top screen (top page) that serves as the starting point for the inspection of the website to be inspected, a "screen transition setting date and time," which is the set date and time for the screen transition, and a "inspection execution date and time," which is the date and time the inspection is to be performed, in association with an "inspection ID" that identifies an individual inspection (case).

[0046] 3 illustrates information managed by the page management table 154 in this example. The page management table 154 in this example manages information about each web page (screen) included in the website to be inspected, and as shown in the figure, associates information such as a "screen URL" that is the URL of the screen, "screen capture information" that is information about a screen capture of the screen, "HTML data" of the screen, a "source screen ID" that identifies the source screen of the screen, a "source element ID" that identifies the source element of the screen (the transition target element clicked to transition to the screen), "source input information" that is information about input values ​​entered for input items on the source screen, and "inspection result information" that is information about the inspection result on the screen, in association with a combination of an "inspection ID" that identifies an individual inspection and a "screen ID" that identifies an individual screen in the inspection.

[0047] 4 illustrates information managed by the error management table 156 in this example. The error management table 156 in this example manages information related to errors that occur when automatically crawling a website to be inspected, and as shown in the figure, associates information such as a "screen URL" that is the URL of the screen on which the error occurred, "screen capture information" that is information related to a screen capture of the screen, "HTML data" of the screen, a "source screen ID" that identifies the source screen of the screen, a "source element ID" that identifies the source element of the screen (the transition target element clicked to transition to the screen), "source input information" that is information related to input values ​​entered for input fields on the source screen, and an "error score" that indicates the possibility of an error occurring on the screen, in association with a combination of an "inspection ID" that identifies an individual inspection and an "error ID" that identifies an individual error in the inspection.

[0048] The information managed by each table in this example has been described above. Next, the processing executed by the website inspection server 10 and the screens output on the user terminal 30 in this example will be described.

[0049] 5 is a flowchart illustrating the process executed by the website inspection server 10 when inspecting a website for vulnerability in this example. As shown in the figure, the server 10 first accepts settings for the website to be inspected for vulnerability (step S100). The settings are accepted via a screen displayed on the user terminal 30.

[0050] 6 illustrates a target site setting screen 50 output on the user terminal 30 for accepting settings for a website to be inspected. As shown, the screen 50 has a URL input area 52 for inputting the URL of the top screen (top page) of the website to be inspected, a login ID input area 54 for inputting a login ID used to log in to the website, a password input area 56 for inputting a password used to log in to the website, and a continue button 58 with the text "Start patrol and inspection" added. In this example, the URL of the top screen, the login ID, and the password are accepted as the setting information for the website to be inspected.

[0051] The continue button 58 is an object for instructing automatic crawling and inspection of the website to be inspected. Returning to the flowchart of Figure 5, when the button 58 is selected, the server 10 then automatically crawls the website to be inspected and sets the screen transitions for the inspection (step S110).

[0052] 7 is a flowchart illustrating the process executed by the server 10 during automatic website crawling. As shown in the figure, the server 10 first accesses the top screen of the website to be inspected and adds the top screen to the screen transitions (step S200). Specifically, a web browser is launched on the server 10, and the top screen identified by the URL entered via the target site setting screen 50 is accessed. Information about the top screen is registered in the page management table 154, specifically, the inspection ID, screen ID, screen URL, screen capture information, and HTML data are set. In this way, the page management table 154 can also be considered a table for managing the screen transitions set for each inspection.

[0053] Next, the server 10 identifies a plurality of transition target elements included in the top screen (step S210). Specifically, screen components (buttons, images, text, etc.) to which links or scripts, etc. are set are identified as transition target elements. These elements are identified based on the corresponding HTML data. For example, elements corresponding to the button tag, img tag, input tag (type="button"), etc. are candidates for transition target elements.

[0054] Next, if an input item (e.g., an inputtable form element) exists (YES in step S220), the server 10 generates an input value for the input item and sets the generated input value (steps S230 and S240). The presence or absence of an input item is determined based on the corresponding HTML data, and for example, an input tag (type="text"), a textarea tag, etc. are candidates for the input item.

[0055] Furthermore, input values ​​for input items are specifically generated using a trained model capable of generating such input values. In this example, the trained model is configured as a large-scale language model, and is configured so that when information instructing the presentation of combinations of input items and input values ​​on a target screen is input together with the HTML data of the target screen, each input item included in the screen is output together with the corresponding input value (i.e., a combination of input items and input values ​​is output). For example, GPT-4 provided by OpenAI, Inc., USA, can be applied as such a large-scale language model.

[0056] 8 shows an example of input / output information in a large-scale language model. As shown in the figure, by specifying output in JSON format in the input information, it is possible to set input values ​​based on the output information.

[0057] 9 shows another example of input / output information in a large-scale language model. As shown in the figure, even if the parameter names ("param1" and "param2") in the HTML data do not indicate the purpose of the parameter, the large-scale language model can determine the purpose of the parameter based on surrounding elements such as labels. Furthermore, even if the input rules are specified in natural language (such as "Please specify in half-width kana and do not leave a space between the last name and the first name"), the large-scale language model can present input values ​​that comply with these input rules.

[0058] 10 shows yet another example of input and output information in a large-scale language model. As shown, even if there is a typo in the parameter name in the HTML data (e.g., "lsat_name" is a typo for "last_name"), the large-scale language model can predict such a typo and determine the use of the parameter.

[0059] Next, the server 10 selects (clicks) the corresponding transition target element (step S250). If there is no input item (NO in step S220), the transition target element is immediately selected.

[0060] Next, the server 10 determines whether an error exists on the destination screen (step S260). Specifically, such errors are determined using a trained model for determining such errors. In this example, the trained model is generated through machine learning using a large amount of training data, and is configured to output an error score indicating the possibility of an error occurring when the HTML data of the source screen, an HTTP request (including input values ​​set for the input fields), and an HTTP response (including the HTML data of the destination screen) are input. If the value of the error score is equal to or greater than a threshold, it is determined that an error has occurred. If the value of the error score is less than the threshold, it is determined that no error has occurred.

[0061] For example, if an error message is displayed only in natural language (such as a message saying "The process could not be completed"), it is difficult to determine the error using rule-based methods. However, by using a trained model, it is possible to determine such errors.

[0062] If it is determined that no error has occurred on the destination screen (NO in step S265), the server 10 adds a transition to the destination screen to the screen transitions (step S270). Specifically, information about the destination screen is registered in the page management table 154, that is, the test ID, screen ID, screen URL, screen capture information, HTML data, source screen ID, source element ID, and source input information are set. The source input information is set as a combination of input items and input values.

[0063] Here, even if no error occurs, if the transition destination screen is a screen that is not a target for addition to the screen transition, it may not be added to the screen transition. For example, screens of sites external to the website being inspected and screens that overlap with screens already included in the screen transition (such overlaps are determined based on the similarity of the screens, for example) are determined to be ineligible for addition to the screen transition.

[0064] On the other hand, if it is determined that an error has occurred on the destination screen (YES in step S265), the server 10 registers the error on the destination screen (step S280). Specifically, the test ID, error ID, screen URL, screen capture information, HTML data, source screen ID, source element ID, source input information, and error score are set in the error management table 156. Note that if an error has occurred, the screen is not added to the screen transition.

[0065] Then, for all of the multiple transition target elements included in the top screen, when the transition destination screen is added to the screen transition in response to the selection (click) or an error is registered (NO in step S290), the server 10 then identifies the multiple transition target elements included in each screen added to the screen transition (step S310), as with the top screen, and adds the transition destination screen to the screen transition in response to the selection of the element or registers an error (steps S220 to S290). This process is repeated until no new screens are added to the screen transition (NO in step S300).

[0066] Returning to the flowchart of FIG. 5 , once the screen transitions have been set in this manner, the server 10 then executes vulnerability testing in accordance with the set screen transitions (step S120). Specifically, a web browser is launched, and each screen included in the screen transition is visited in order in accordance with the information managed in the page management table 154, and testing is performed for predetermined vulnerability testing items. Vulnerability testing items include, for example, items specified in OWASP Top 10. The testing results are registered in the testing result information in the page management table 154. Note that testing for testing items may also be performed on each screen during automatic website crawling (i.e., when setting up screen transitions).

[0067] 11 shows an example of an inspection result confirmation screen 60 for confirming inspection results, which is output on the user terminal 30. As shown in the figure, the screen 60 has a list display area 62 that displays a list of vulnerability inspection results. The area 62 displays a list of vulnerability inspection results (severity, vulnerability name, detected location, vulnerable location, etc.) for each screen. This information is managed in the inspection result information of the page management table 154.

[0068] In the above example, the information managed in the error management table 156 is automatically added to the training data during additional learning of the above-mentioned trained model to determine errors on the transition destination screen.

[0069] In the above example, a screen including the information managed in the error management table 156 may be output on the user terminal 30 of the corresponding user (website administrator, etc.).

[0070] In the above example, a combination of input items and input values ​​is output from the trained model configured as a large-scale language model, but only the input values ​​may be output. In this case, input items are identified based on HTML data of the target screen, etc., and when information instructing the trained model to present an input value for the identified input item along with the HTML data is input to the trained model, information presenting the input value itself is output from the trained model. Figure 12 shows an example of input / output information in the large-scale language model in this case. In this example, "param1" is a parameter for the input item (name) identified based on the HTML data of the target screen, and this input information is automatically generated by the server 10.

[0071] The trained model may also be configured to output information (such as an error score) indicating the likelihood of an error when HTML data and a combination of input items and input values ​​are input. In this case, the input items are identified based on the HTML data of the target screen, and the likelihood of an error is obtained for each of multiple input value candidates (predetermined) that can be set for the input items via the trained model, and the candidate with the lowest likelihood is applied as the input value.

[0072] In the above example, the trained model for determining an error in the destination screen is configured to output an error score indicating the possibility of an error when the HTML data, HTTP request, and HTTP response of the source screen are input, but information for identifying the input item that is the cause of the error among multiple input items may also be output. For example, the trained model is configured as a large-scale language model, and when information instructing the presentation of the input item that is the cause of the error is input together with the HTML data, HTTP request, and HTTP response of the source screen, the trained model outputs information presenting the input item.

[0073] Furthermore, extended information such as the reason for the error and an alternative value for the input value may be acquired from a trained model for determining the error. Such a trained model is configured as a large-scale language model, and when information instructing the presentation of such information is input together with the HTML data of the transition source screen and the input values ​​for the input fields, information presenting the extended information is output from the trained model. Figure 13 shows an example of input / output information in the large-scale language model in this case. The extended information acquired in this manner is managed, for example, in an error management table 156.

[0074] In the above example, a vulnerability testing service is provided to test for website vulnerabilities, but in other examples of this embodiment, various tests are performed from perspectives other than vulnerabilities, for example, testing other website performance and functions.

[0075] The website inspection server 10 according to the present embodiment described above identifies input items on web pages when automatically crawling a website 22, and sets input values ​​for the input items based on information output from the trained model (e.g., a combination of input items and input values), thereby promoting the use of input values ​​appropriate for various websites compared to using input values ​​prepared in advance. In this way, the website inspection server 10 supports the setting of appropriate input values ​​when automatically crawling a website 22.

[0076] In other embodiments of the present invention, some or all of the functions of the website inspection server 10 in the above-described embodiment may be realized by cooperation between the website inspection server 10 and the user terminal 30, or may be realized by the user terminal 30. Furthermore, in other embodiments of the present invention, automatic crawling of the website 22 is performed for various purposes other than inspecting the website 22.

[0077] The processes and procedures described herein, other than those explicitly described, may be implemented by software, hardware, or any combination thereof. For example, the processes and procedures described herein may be implemented by implementing logic corresponding to the processes and procedures in a medium such as an integrated circuit, volatile memory, non-volatile memory, or magnetic disk. Furthermore, the processes and procedures described herein may be implemented as a computer program corresponding to the processes and procedures and executed by various computers.

[0078] Although processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, software, components, and / or modules. Also, the software and hardware elements described herein may be implemented by combining them into fewer components or breaking them down into more components.

[0079] In this specification, even if a component of the invention is described as either singular or plural, or is described without limitation as either singular or plural, the component may be either singular or plural unless the context requires otherwise.

Claims

1. A system for automatically crawling a website, comprising one or more computer processors, wherein the one or more computer processors execute the steps of: acquiring a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and acquiring a second web page transitioned to from the first web page accompanied by transmission of the set input values, wherein the step of setting the input values ​​includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

2. The system of claim 1, wherein the first output information includes the input value.

3. The system of claim 1, wherein the first output information includes a combination of the input items and the input values, and the step of setting the input values ​​includes setting the input values ​​included in the first output information for the input items included in the first output information.

4. The system of claim 1, wherein the first input information includes a combination of the input item and the input value, the first output information includes error possibility information indicating the possibility of an error occurring when the input value is set for the input item, and the step of setting the input value includes determining whether or not to set the input value based on the error possibility information.

5. The system of claim 1, wherein the one or more computer processors further execute the steps of: determining an error based on second page information regarding the second web page; and setting a new input value for the input field if it is determined that an error exists.

6. The system of claim 5, wherein the step of determining the error includes inputting second input information including at least a portion of the second page information to a second trained model, and determining the error based on second output information output from the second trained model.

7. The system of claim 6, wherein the second output information includes information regarding the presence or absence of the error.

8. The system of claim 6, wherein the step of setting the input value includes identifying a plurality of input items in the first web page and setting a corresponding input value for each of the plurality of input items, and the second output information includes information regarding an input item among the plurality of input items that is the cause of the error.

9. The system of claim 6, wherein the one or more computer processors further execute a step of adding data including the input item and the combination of the input values ​​set for the input item and transmitted as training data for the second learned model when it is determined that an error exists.

10. The system of claim 5, wherein the one or more computer processors further perform the step of presenting information regarding the error to a user if the error is determined to exist.

11. A method for automatically crawling a website, executed by one or more computers, comprising the steps of: acquiring a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and acquiring a second web page transitioned to from the first web page accompanied by transmission of the set input value, wherein the step of setting the input value includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

12. A program for automatically crawling a website, the program causing one or more computers to execute the following steps: acquiring a first web page included in the website; identifying input items in the first web page and setting input values ​​for the input items; and acquiring a second web page transitioned to from the first web page accompanied by transmission of the set input values, wherein the step of setting the input values ​​includes inputting first input information including at least a portion of first page information related to the first web page to a first trained model, and setting the input value based on first output information output from the first trained model.

Images