Determination device, method for determination, and determination program

Abstract

To properly determine whether or not a website as a determination target is a phishing website.SOLUTION: A determination device 100 according to the embodiment includes an acquisition unit 121, a deletion unit 122, and a determination unit 123. The acquisition unit 121 acquires plural pieces of information on a website. The deletion unit 122 deletes a part of the plural pieces of information on the website acquired by the acquisition unit 121 to satisfy a predetermined condition. The determination unit 123 inputs the data whose part has been deleted by the deletion unit 122, into a large-scale language model, and determines whether or not the website is a phishing website on the basis of the result of output of the large-scale language model.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a determination device, a determination method, and a determination program.

Background Art

[0002] In recent years, systems for performing natural language processing such as question-and-answer using large language models (LLMs) have been provided. An LLM is a natural language processing model trained using a large amount of text data, which takes a sentence as input and outputs a sentence. When an LLM is applied to a system for performing question-and-answer, when a question sentence (prompt) is input to the LLM, an answer sentence generated in an interactive format is output from the LLM.

[0003] Also, today, with the spread of the use of Internet shopping and the cashless trend due to the use of credit cards and electronic money, the opportunity to enter personal information on a website on the Internet and make a payment has increased. In such a situation, phishing sites that disguise themselves as real delivery companies, financial institutions, shopping sites, etc., and mislead users into believing that they are legitimate sites, causing them to enter personal information, request payment, or download malware, are on the rise.

[0004] A phishing site is a malicious website, characterized by having one or more of the following two elements. The first is to abuse the brand names of legitimate services and companies. The second is to present false information to deceive and psychologically induce users. Note that false information includes, for example, malware infection warnings and prize-winning notifications.

[0005] As an existing method for detecting phishing sites, for example, there is a method of learning the logo images of legitimate websites and identifying the logo images displayed on phishing sites (see, for example, Non-Patent Document 1). In addition, there is also a method of detecting phishing sites generated by clones of legitimate sites based on the similarity of the entire screen of legitimate websites (see, for example, Non-Patent Document 2). Since the two methods described above use machine learning models, they collect the brand logo and image data of legitimate websites to create training data and train the models.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, in the above prior art, there are cases where it is not possible to appropriately determine whether the Web site to be determined is a phishing site. For example, in the prior art, there are cases where it is not possible to detect a phishing site that modifies the brand logo image of a legitimate Web site or sets the layout of the Web site independently. Also, in the conventional method, for example, there are cases where it is not possible to interpret the context from the text displayed on the phishing site to identify false information and identify the psychological techniques for deceiving users.

Means for Solving the Problem

[0008] In order to solve the above-described problems and achieve the object, the determination device of the present invention includes an acquisition unit that acquires a plurality of pieces of information related to a Web site, a deletion unit that deletes a part of the plurality of pieces of information related to the Web site acquired by the acquisition unit so as to satisfy a predetermined condition, and a determination unit that inputs the data from which a part has been deleted by the deletion unit into a large language model and determines whether the Web site is a phishing site based on the output result of the large language model.

Effect of the Invention

[0009] According to the present invention, there is an effect that it is possible to appropriately determine whether the Web site to be determined is a phishing site.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the determination device, determination method, and determination program according to the present application will be described in detail with reference to the drawings. Note that the determination device, determination method, and determination program according to the present application are not limited by this embodiment.

[0012] 〔1. Overall Configuration〕 First, the overall configuration of a system including the determination device 100 according to this embodiment will be described. FIG. 1 is a diagram showing a system including the determination device according to the embodiment. The system shown in FIG. 1 is composed of a determination device 100 and an external device 200 that transmits the URL (Uniform Resource Locator) of the Web site to be determined to the determination device 100.

[0013] The determination device 100 is an information processing device that receives the URL of a website transmitted from an external device 200 and determines whether the website indicated by the received URL is a phishing site. For example, it is composed of a computer, a server device, etc. The external device 200 is an information processing device communicably connected to the determination device 100. For example, it transmits the URL of the website to be determined to the determination device 100, and receives and displays the information of the determination result output from the determination device 100.

[0014] The determination device 100 acquires a plurality of pieces of information related to the website, and deletes a part of the plurality of pieces of information related to the acquired website so as to satisfy a predetermined condition. After that, the determination device 100 inputs the data with a part deleted into a large language model, and determines whether the website is a phishing site based on the output result of the large language model.

[0015] First, the determination device 100 receives, for example, the URL of the website to be determined transmitted from the external device 200, and acquires from the website indicated by the URL the URL of the website, the source code of the HTML (Hyper Text Markup Language) of the website, and a screen shot image of the screen displaying the website.

[0016] Next, the determination device 100 deletes a part of the data so that the number of tokens becomes equal to or less than a predetermined number for, for example, the source code of the HTML and the text data extracted from the screen shot image. After that, the determination device 100 substitutes, for example, the URL, the source code of the HTML with a part deleted, and the text data of the screen shot image with a part deleted into the template of the input prompt of the large language model and inputs it, and determines whether the website to be determined is a phishing site by analyzing the response result of the input.

[0017] As a result, the determination device 100 can appropriately determine whether the website to be determined is a phishing site from the URL of the website to be determined. By analyzing the response result of the large language model, it is possible to interpret the context from the text displayed on the phishing site to identify false information and identify the psychological techniques used to deceive users.

[0018] [2. Configuration of the determination device 100] Next, with reference to FIG. 2, the configuration of the determination device 100 shown in FIG. 1 will be described. FIG. 2 is a block diagram showing a configuration example of the determination device according to the embodiment. The determination device 100 includes a communication unit 110, a control unit 120, and a storage unit 130, and is communicably connected to an external device 200 via a network N.

[0019] The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to the network N and transmits and receives information to and from the external device 200. The communication unit 110 receives, for example, the URL of the website to be determined from the external device 200, or mediates the acquisition of information about the website by the acquisition unit 121 described later.

[0020] The storage unit 130 is realized by a storage device such as a RAM (Random Access Memory) or a hard disk, for example. The storage unit 130 stores data and programs necessary for various processes by the control unit 120. The storage unit 130 includes, for example, an acquisition data storage unit 131, a post-deletion data storage unit 132, and a response result data storage unit 133.

[0021] The acquisition data storage unit 131 stores information about the website to be determined acquired by the acquisition unit 121 described later. Here, with reference to FIG. 3, the data stored in the acquisition data storage unit 131 will be described. FIG. 3 is a diagram showing an example of the data stored in the determination device according to the embodiment. The acquisition data storage unit 131 shown in the example of FIG. 3 is composed of items such as "URL", "HTML source code", and "screenshot image".

[0022] "URL" stores the URL of the target website to be determined sent from the external device 200. "HTML source code" stores the HTML source code that constitutes the target website to be determined. "Screenshot image" stores the image data of the screenshot image of the screen that displays the target website to be determined.

[0023] The post-deletion data storage unit 132 stores the data of the HTML source code partially deleted by the deletion unit 122 described later and the text data extracted from the screenshot image. For example, the post-deletion data storage unit 132 stores the data of the HTML source code that has been partially deleted and simplified so that the number of tokens becomes less than or equal to a predetermined number by the deletion unit 122 described later, and the text data extracted from the screenshot image, which has been partially deleted and simplified so that the number of tokens becomes less than or equal to a predetermined number.

[0024] The response result data storage unit 133 stores the data of the response result by the large language model. Here, referring to FIG. 4, the data stored in the response result data storage unit 133 will be described. FIG. 4 is a diagram showing an example of the data stored in the determination device according to the embodiment. The response result data storage unit 133 shown in the example of FIG. 4 is composed of items of "URL", "phishing_score", "brands", "phishing", and "suspicious_domain".

[0025] "phishing_score" stores the score value, which is a score indicating the likelihood that the website under judgment is a phishing site, and is displayed as a numerical value on a 10-point scale from 1 to 10. "brands" stores information such as the brand name of the legitimate website being deceived by the website under judgment. "phishing" stores the judgment result as to whether the website under judgment is a phishing site as "true" or "false". "suspicious_domain" stores the judgment result of the suspiciousness of the domain of the website under judgment as "true" or "false".

[0026] Return to the description of FIG. 2. The control unit 120 is realized by various programs stored in the internal storage device of the apparatus being executed with the RAM as the working area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. Further, the control unit 120 is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 120 includes an acquisition unit 121, a deletion unit 122, and a determination unit 123, and optionally includes a generation unit 124.

[0027] The acquisition unit 121 acquires a plurality of pieces of information regarding the website. For example, the acquisition unit 121 accesses the target website from the input URL of the website, and acquires the URL of the website, the source code of the HTML of the website, and a screenshot image of the screen displaying the website. Then, the acquisition unit 121 stores the acquired information in the acquisition data storage unit 131.

[0028] Here, the acquisition unit 121 can, for example, automatically operate the web browser from the input URL of the website to reach the target website and acquire a plurality of pieces of information regarding the website. Note that the automatic operation of the web browser can execute web access to an arbitrary URL by using a browser automatic operation tool such as Selenium, Puppeteer, or Chrome DevTools Protocol.

[0029] In addition to the above-described information, the acquisition unit 121 can acquire information such as the URL of the website passed through until reaching the website to be determined and the communication destination information such as the IP address of the website. By using the three pieces of information, i.e., the URL of the website, the HTML source code of the website, and the screenshot image of the screen displaying the website, in the input prompt described later, there is an effect that the determination process described later can be performed with high accuracy.

[0030] The deletion unit 122 deletes a part of the plurality of pieces of information regarding the website acquired by the acquisition unit 121 so as to satisfy a predetermined condition. For example, the deletion unit 122 refers to the data stored in the acquisition data storage unit 131 and deletes a part of the data so that the number of tokens of each of the HTML source code data and the screenshot image data is equal to or less than a predetermined number. Then, the deletion unit 122 stores the data after deletion with a part deleted in the post-deletion data storage unit 132.

[0031] Here, with reference to FIG. 5, the process of the deletion unit 122 will be described by giving specific token numbers. FIG. 5 is a diagram showing a specific example of the number of tokens of data partially deleted by the process in the deletion unit according to the embodiment. The large language model used by the determination unit 123 described later has a limit on the number of inputs (token numbers) of its input prompt. Therefore, in order to create a prompt to be input to the large language model using the data acquired by the acquisition unit 121 described above, a part of the input data is deleted and simplified so that the number of tokens of the input prompt is equal to or less than the maximum number of tokens of the large language model to be used.

[0032] FIG. 5 shows an example breakdown of the token numbers set when using a large language model whose maximum number of tokens for the input prompt is limited to "4096". Specifically, the prompt template, which is a pre-created template part other than the acquired data, is "362", the HTML source code is "3000", the text extracted from the screenshot image is "500", and the URL is "234".

[0033] Here, the number of tokens of the prompt template and the URL varies little depending on the content of the website to be determined, while the number of tokens of the HTML source code and the text extracted from the screenshot image varies greatly. Therefore, the deletion unit 122 deletes and simplifies a part of each of the acquired HTML source code data and screenshot image data so that the number of tokens of the HTML source code is equal to or less than "3000" and the number of tokens of the text extracted from the screenshot image is equal to or less than "500".

[0034] As a result, the determination device 100 can always appropriately create an input prompt equal to or less than the maximum number of tokens limited by the large language model regardless of the data volume of the website to be determined.

[0035] Here, with reference to FIG. 6, the deletion process of the HTML source code will be described. FIG. 6 is a diagram showing a specific example of the pseudo code of the deletion process of the HTML source code in the deletion unit according to the embodiment. The deletion unit 122 deletes elements that satisfy a predetermined condition from the HTML source code acquired by the acquisition unit 121 so that the number of tokens becomes a predetermined number or less. Hereinafter, the processes performed by the deletion unit 122 will be sequentially described so as to correspond to the pseudo code shown in FIG. 6.

[0036] First, the deletion unit 122 deletes the HTML style / script / comment elements (corresponding to the second line in FIG. 6). Next, the deletion unit 122 determines whether the number of tokens is 3000 (predetermined number) or less (corresponding to the fourth line in FIG. 6). And when the number of tokens exceeds 3000, the deletion unit 122 unwraps HTML elements other than important HTML tags (corresponding to the seventh line in FIG. 6). Note that important HTML tags include, for example, head, title, meta, body, h1, h2, h3, h4, h5, h6, p, strong, a, img, hr, table, tbody, tr, th, td, ol, ul, li, ruby, and label, etc.

[0037] Thereafter, the deletion unit 122 deletes HTML elements that do not contain text (corresponding to the eighth line in FIG. 6). Next, the deletion unit 122 reduces the href element of the a tag and the src element of the img tag (corresponding to the ninth line in FIG. 6). And the deletion unit 122 repeatedly deletes the HTML elements at the middle point of the HTML until the number of tokens becomes 3000 or less (from the eleventh line to the eighteenth line in FIG. 6). When the number of tokens becomes 3000 or less, a part of the data is deleted and the simplified HTML is stored in the post-deletion data storage unit 132 as the deleted data.

[0038] By performing the above-described series of processes, the deletion unit 122 can create data of a simplified HTML source code in which important elements characterizing the phishing site are retained and unnecessary elements for determination are deleted, and the number of tokens becomes a predetermined number or less.

[0039] Here, the deletion process of the data extracted from the screenshot image will be described. The deletion unit 122 extracts text data from the screenshot image acquired by the acquisition unit 121, and deletes the text from the text data in order from the text with a small character size so that the number of tokens becomes less than or equal to a predetermined number.

[0040] For example, the deletion unit 122 refers to the data stored in the acquisition data storage unit 131, and uses OCR (Optical Character Recognition) that can acquire the character size of the text extracted with the image as the input to extract text data from the screenshot image of the website. Then, the deletion unit 122 deletes the text from the extracted text data in order from the text with a small character size so that the number of tokens becomes less than or equal to a predetermined number (for example, 500).

[0041] Thereby, the deletion unit 122 can create simplified text data with the number of tokens less than or equal to a predetermined number while retaining the text with a large character size that is considered relatively important for the determination of the phishing site.

[0042] The determination unit 123 inputs the data partially deleted by the deletion unit 122 into the large language model, and determines whether the website is a phishing site based on the output result of the large language model.

[0043] For example, the determination unit 123 inputs the input prompt generated by the generation unit 124 (to be described later) into the large language model using the URL of the target Web site stored in the acquisition data storage unit 131, the data of the HTML source code with some data deleted stored in the post-deletion data storage unit 132, and the text data. Then, for example, the determination unit 123 refers to the data stored in the response result data storage unit 133 and determines that it is a phishing site if either the phihing key (corresponding to "phishing" in FIG. 4) or the suspicious_domain key (corresponding to "suspicious_domain" in FIG. 4), which are the response results of the large language model, is "true", and determines that it is a non-phishing site if both are "false".

[0044] In addition to the response results of the phihing key and the suspicious_domain key described above, the determination unit 123 can determine whether it is a phishing site, for example, based on whether the numerical value of "phishing_score" (see FIG. 4) is equal to or greater than a preset threshold value.

[0045] The generation unit 124 generates an input prompt for the large language model using the URL of the Web site acquired by the acquisition unit 121 and the data with some data deleted by the deletion unit 122. For example, the generation unit 124 substitutes the URL of the target Web site stored in the acquisition data storage unit 131, the data of the HTML source code with some data deleted stored in the post-deletion data storage unit 132, and the text data into a pre-created input prompt template, respectively, to generate an input prompt to be input into the large language model.

[0046] [3. Specific Example of the Processing of the Determination Device 100] Here, referring to FIG. 7, the overall processing flow of the determination process performed by the determination device 100 will be described. FIG. 7 is a diagram showing a specific example of the processing of the determination device according to the embodiment. In FIG. 7, an example of the processing from the reception of the URL of the determination target mediated by the communication unit 110 to the output of determining whether it is a final phishing site is shown.

[0047] First, the determination device 100 receives an input of the URL of the Web site to be determined from an external device 200 or the like. Next, the acquisition unit 121 uses a program such as a Web crawler to acquire information about the Web site to be determined.

[0048] Subsequently, the deletion unit 122 deletes a part of the data so that the number of tokens becomes equal to or less than a predetermined number for the information of the HTML source code of the Web site. Also, the deletion unit 122 performs text extraction by OCR from the screenshot image of the Web site, and deletes a part of the text data so that the number of tokens becomes equal to or less than a predetermined number for the extracted text. As a result, the deletion unit 122 creates simplified HTML data with a part deleted, simplified OCR-extracted text with a part deleted, and the URL for the Web site to be determined.

[0049] Then, the generation unit 124 generates an input prompt to be input to the large language model by substituting the simplified HTML data with a part deleted, the simplified OCR-extracted text with a part deleted, and the URL into a pre-created template. After that, the determination unit 123 inputs the input prompt generated by the large language model, analyzes the response result, and determines whether the Web site to be determined is a phishing site. Then, the determination device 100 outputs the determination result by the determination unit 123 to an external device 200 or the like.

[0050] Next, with reference to FIG. 8, the input prompt to be input into the large language model will be described. FIG. 8 is a diagram showing a specific example of an input prompt input into the large language model according to the embodiment. In (1) to (4) of FIG. 8, as a prompt template, there is a description of a sentence regarding the specification of the content of the stack to be performed by the large language model regardless of the content of the Web site to be determined. On the other hand, in (5) of FIG. 8, there are shown items for substituting the URL acquired by the acquisition unit 121, the HTML data partially deleted by the deletion unit 122, and the text data.

[0051] Specifically, in (1) of FIG. 8, as the main task, there is a description of a sentence that assigns and gives the role and task of "being an expert in security and discriminating phishing sites" to the large language model. And in (2) of FIG. 8, it is described that four subtasks in the discrimination process of phishing sites are performed in order.

[0052] Subsequently, in (3) of FIG. 8, there is a description of the constraint content in the task indicating that the HTML may be shortened or simplified, or the text extracted by OCR may not be accurate. And in (4) of FIG. 8, there is a description of an example of a psychological induction method of social engineering used for phishing sites, such as displaying a fake security warning.

[0053] Subsequently, in (5) of FIG. 8, there is a description of items for substituting the URL of the Web site to be determined, the data of the HTML source code, and the text data extracted from the screenshot image. By inputting each data into the said item, an input prompt is generated.

[0054] Here, the template of the input prompt shown in FIG. 8 improves the analysis accuracy of the large language model by assigning the role of "security expert" to the large language model, dividing subtasks step by step and performing them in order, or showing features found on phishing sites. As a result, the generation unit 124 can generate a high-quality input prompt with high determination accuracy by substituting the information obtained from the Web site to be determined into the template of the input prompt shown in FIG. 8.

[0055] Next, with reference to FIG. 9, the response result of the large language model to the input prompt will be described. FIG. 9 is a diagram showing a specific example of the response result of the large language model according to the embodiment. In FIG. 9, the response result when the input prompt generated by substituting the URL of the Web site to be determined, the HTML source code, and the text data extracted from the screenshot image into the template of the input prompt shown in FIG. 8 is input to the large language model is shown.

[0056] Specifically, the answers to the four subtasks in the input prompt shown in FIG. 8 are shown. First, in the answer to the first subtask, for each of the substituted data, suspicious elements (elements that can be judged as phishing sites) and the basis for such judgment are answered.

[0057] Next, in the answer to the second subtask, the brand name of the legitimate Web site that the Web site to be determined is supposed to deceive is shown. In the example of FIG. 9, it is specified that the Web site to be determined is deceiving the brand "Example Company". Subsequently, in the third subtask, the conclusion of the determination as to whether the Web site to be determined is a phishing site is shown. In the example of FIG. 9, it is shown that the Web site to be determined is likely to be a phishing site of the brand "Example Company".

[0058] In the answer to the fourth subtask, the output result is shown when the judgment made by the large language model is output in JSON (JavaScript (registered trademark) Object Notation) format. In the example of FIG. 9, the judgment results of "phishing_Score: 8", "brands: Example Company", "phishing: true", and "suspicious_domain: true" are shown. Note that the judgment result is stored in the response result data storage unit 133.

[0059] Based on the response result shown in FIG. 9 described above, the determination unit 123 refers to information such as "phishing: true" and "suspicious_domain: true" stored in the response result data storage unit 133, for example, and determines that the Web site to be determined is a phishing site. Thereafter, the determination device 100 outputs, for example, the determination result that the Web site to be determined is a phishing site to the external device 200 together with the URL of the Web site to be determined. Note that, in addition to the URL and the determination result, the determination device 100 can output, for example, the entire response result shown in FIG. 9 to enable the external device 200 to grasp the basis for the determination content made by the large language model.

[0060] [4. An Example of the Processing of the Determination Device 100] Next, with reference to FIG. 10, the processing flow by the determination device 100 will be described. FIG. 10 is a flowchart showing an example of the processing flow of the determination device according to the embodiment. Note that each step described below may be executed in a different order, or there may be omitted processing. Also, the processing procedures of each embodiment may be combined as appropriate and implemented.

[0061] First, the determination device 100 receives the URL of the Web site to be determined from an external device 200 or the like (S101). When the determination device 100 receives the URL of the Web site to be determined (S101; Yes), the acquisition unit 121 accesses the Web site from the received URL and acquires the URL of the Web site to be determined, the HTML source code, and the screenshot image (S102). On the other hand, when the determination device 100 has not received the URL of the Web site to be determined (S101; No), the determination device 100 waits until it receives the URL of the Web site to be determined.

[0062] After the process of S102, the deletion unit 122 deletes a part of each of the HTML source code and the text data extracted from the screenshot image so as to satisfy a predetermined condition (S103). Subsequently, the generation unit 124 generates an input prompt by substituting the URL, the HTML source code with a part deleted, and the text data with a part deleted (S104).

[0063] After that, the determination unit 123 determines whether the Web site to be determined is a phishing site based on the response result of the large language model for the generated input prompt (S105). Then, the determination device 100 outputs the determination result to the external device 200 or the like (S106) and ends the process.

[0064] [5. Effects of the Embodiment] As described above, the determination device 100 according to the present embodiment includes an acquisition unit 121, a deletion unit 122, and a determination unit 123. The acquisition unit 121 acquires a plurality of pieces of information regarding the Web site. The deletion unit 122 deletes a part of the plurality of pieces of information regarding the Web site acquired by the acquisition unit 121 so as to satisfy a predetermined condition. The determination unit 123 inputs the data with a part deleted by the deletion unit 122 to the large language model and determines whether the Web site is a phishing site based on the output result of the large language model.

[0065] As a result, the determination device 100 can appropriately determine whether the target website is a phishing site by analyzing the response results of the large language model for input prompts with a predetermined number of tokens or less created using the configuration information of the website for the target website to be determined.

[0066] Also, the acquisition unit 121 of the determination device 100 accesses the website indicated by the input URL of the website and acquires a plurality of pieces of information regarding the website. As a result, the determination device 100 can acquire a plurality of pieces of information regarding the website required for the determination process using a browser automation tool such as a web crawler based on the URL of the target website received from the outside.

[0067] Furthermore, the acquisition unit 121 of the determination device 100 acquires, as a plurality of pieces of information regarding the website, the URL of the website, the source code of the HTML of the website, and a screenshot image of the screen displaying the website. As a result, the determination device 100 can acquire a plurality of pieces of information that enable the judgment result by the large language model to be highly accurate from the target website.

[0068] Also, the deletion unit 122 of the determination device 100 deletes elements that satisfy a predetermined condition from the source code of the HTML acquired by the acquisition unit 121 so that the number of tokens becomes a predetermined number or less. As a result, the determination device 100 can create data of a simplified HTML source code in which important elements characterizing the phishing site are retained and unnecessary elements for the determination are deleted, and the number of tokens becomes a predetermined number or less.

[0069] In addition, the deletion unit 122 of the determination device 100 extracts text data from the screenshot image acquired by the acquisition unit 121, and deletes the text from the text data in order from the text with a small font size so that the number of tokens becomes equal to or less than a predetermined number. As a result, the determination device 100 can create simplified text data with the number of tokens equal to or less than a predetermined number while retaining text with a large font size, which is considered relatively important for determining a phishing site.

[0070] In addition, the determination device 100 includes a generation unit 124. The generation unit 124 generates an input prompt for a large language model using the URL of the website acquired by the acquisition unit 121 and the data partially deleted by the deletion unit 122. In this case, the determination unit 123 inputs the input prompt generated by the generation unit 124 into the large language model. As a result, the determination device 100 can substitute each acquired or partially deleted data into a pre-created template to generate a high-quality input prompt that enables the large language model to make a highly accurate determination.

[0071] 〔6. System configuration, etc.〕 Among the processes described in the above embodiments, a part of the processes described as being automatically performed can also be manually performed. Alternatively, all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0072] Furthermore, each component of each illustrated device is functionally conceptual and does not necessarily have to be physically configured as shown in the figures. That is, the specific form of the distribution and integration of each device is not limited to that shown in the figures, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage situations, etc. Furthermore, each processing function performed by each device can be realized by all or any part of it being realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic.

[0073] For example, part or all of the storage unit 130 shown in FIG. 2 may be held not by the determination device 100 but by a storage server or the like. In this case, the determination device 100 acquires various information by accessing the storage server.

[0074] [7. Hardware Configuration] FIG. 11 is a diagram showing an example of a hardware configuration. The determination device 100 according to the above-described embodiment is realized by a computer 1000 having a configuration as shown in FIG. 11, for example.

[0075] FIG. 11 is a diagram showing an example of a computer that executes an analysis program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0076] Memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System), for example. The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100, for example. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0077] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the determination device 100 is implemented as a program module 1093 in which executable code by the computer 1000 is described. The program module 1093 is stored in the hard disk drive 1090, for example. For example, a program module 1093 for executing the same process as the functional configuration in the determination device 100 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0078] Also, the setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out and executes the program module 1093 and the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as needed.

[0079] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090. For example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN, WAN, etc.). Then, the program module 1093 and the program data 1094 may be read by the CPU 1020 from another computer via the network interface 1070.

Explanation of Signs

[0080] 100 Determination device 110 Communication unit 120 Control unit 121 Acquisition unit 122 Deletion unit 123 Determination unit 124 Generation unit 130 Storage unit 131 Acquired data storage unit 132 Post-deletion data storage unit 133 Response result data storage unit 200 External device

Claims

1. An acquisition unit that acquires a plurality of pieces of information related to a website; A deletion unit that deletes a part of the plurality of pieces of information related to the website acquired by the acquisition unit so as to satisfy a predetermined condition; A determination unit that inputs the data partially deleted by the deletion unit into a large language model and determines whether the website is a phishing site based on the output result of the large language model; A determination device, characterized by comprising the above.

2. The acquisition unit accesses the website indicated by the input URL (Uniform Resource Locator) of the website and acquires a plurality of pieces of information related to the website. The determination device according to claim 1, characterized by the above.

3. The acquisition unit acquires, as a plurality of pieces of information related to the website, the URL of the website, the source code of the HTML (Hyper Text Markup Language) of the website, and a screenshot image of the screen displaying the website. The determination device according to claim 1, characterized by the above.

4. The deletion unit deletes elements that satisfy a predetermined condition from the source code of the HTML acquired by the acquisition unit so that the number of tokens is equal to or less than a predetermined number. The determination device according to claim 3, characterized by the above.

5. The deletion unit extracts text data from the screenshot image acquired by the acquisition unit and deletes the text in order from the text with a small font size so that the number of tokens is equal to or less than a predetermined number from the text data. The determination device according to claim 3, characterized by the above.

6. The device further includes a generation unit that generates an input prompt for the large language model using the URL of the website acquired by the acquisition unit and the data partially deleted by the deletion unit, The determination unit inputs the input prompt generated by the generation unit into the large language model. The determination device according to claim 3, characterized by the above.

7. A determination method executed by a determination device, comprising: An acquisition step of acquiring a plurality of pieces of information related to a website; A deletion step of deleting a part of the plurality of pieces of information related to the website acquired by the acquisition step so as to satisfy a predetermined condition; Input the data partially deleted by the deletion process into a large language model, and based on the output result of the large language model, determine whether the website is a phishing site in a determination process; A determination method characterized by including the above. **Claim 8** An acquisition procedure for acquiring a plurality of pieces of information regarding a website; A deletion procedure for deleting a part of the plurality of pieces of information regarding the website acquired by the acquisition procedure so as to satisfy a predetermined condition; An input procedure for inputting the data partially deleted by the deletion procedure into a large language model, and a determination procedure for determining whether the website is a phishing site based on the output result of the large language model; A determination program for causing a computer to execute the above.

Images