Determination device, determination method, and determination program

Abstract

A determination device (100) according to the present embodiment has an acquisition unit (121), a deletion unit (122), and a determination unit (123). The acquisition unit (121) acquires a plurality of pieces of information relating to a web site. The deletion unit (122) deletes some of the plurality of pieces of information pertaining to the web site acquired by the acquisition unit (121) so as to satisfy a predetermined condition. The determination unit (123) inputs the data partially deleted by the deletion unit (122) to a large language model, and determines whether or not the web site is a phishing site on the basis of the results output by the large language model.

Description

Determination device, determination method, and determination program

[0001] The present invention relates to a determination device, a determination method, and a determination program.

[0002] In recent years, systems that use large language models (LLMs) to perform natural language processing such as question-and-answering have been provided. LLMs are natural language processing models trained using large amounts of text data, and take sentences as input and output sentences. When LLMs are applied to a question-and-answering system, when a question (prompt) is input to the LLM, an answer generated in an interactive format is output from the LLM.

[0003] Furthermore, with the recent popularity of online shopping and the cashless society due to the use of credit cards and electronic money, there are more and more opportunities to make payments by entering personal information on online websites. In this current situation, there has been an increase in phishing sites, which pose as real delivery companies, financial institutions, shopping sites, etc., to trick users into thinking that these sites are legitimate and then trick them into entering personal information, demanding payment, or downloading malware.

[0004] Phishing sites are malicious websites that are characterized by one or more of the following two elements. First, they exploit legitimate services and corporate brand names. Second, they present false information to deceive and psychologically mislead users. Examples of false information include malware infection warnings and prize winning notifications.

[0005] Existing methods for detecting phishing sites include, for example, a method of learning logo images of legitimate websites and identifying the logo images displayed on phishing sites (see, for example, Non-Patent Document 1). Another method detects phishing sites created by cloning legitimate websites based on the similarity of the entire screen of the legitimate website (see, for example, Non-Patent Document 2). The two aforementioned methods use machine learning models, and therefore collect data on brand logos and images of legitimate websites to create learning data and train the model.

[0006] Yun Lin, Ruofan Liu, Dinil Mon Divakaran, Jun Yang Ng, Qing Zhou Chan, Yiwen Lu, Yuxuan Si, Fan Zhang, and Jin Song Dong. 2021.Phishpedia: A Hybrid Deep Learning Based Approach to Visually Identify Phishing Webpages. In 30th USENIX Security Symposium (USENIX Security 21), USENIX Association, 3793-3810.Sahar Abdelnabi, Katharina Krombholz, and Mario Fritz. 2020. VisualPhishNet: Zero-Day Phishing Website Detection by Visual Similarity. In ACM Conference on Computer and Communications Security (CCS).

[0007] However, the above-described conventional techniques may not be able to properly determine whether a target website is a phishing site. For example, conventional techniques may not be able to detect phishing sites that alter the brand logo images of legitimate websites or create unique website layouts. Furthermore, conventional techniques may not be able to interpret the context of text displayed on phishing sites to identify false information or to identify psychological techniques used to deceive users.

[0008] In order to solve the above-mentioned problems and achieve the objectives, the determination device of the present invention is characterized by having an acquisition unit that acquires multiple pieces of information related to a website, a deletion unit that deletes some of the multiple pieces of information related to the website acquired by the acquisition unit so as to satisfy predetermined conditions, and a determination unit that inputs the data from which some of the information has been deleted by the deletion unit into a large-scale language model and determines whether the website is a phishing site based on the output result of the large-scale language model.

[0009] According to the present invention, it is possible to appropriately determine whether a website to be determined is a phishing site or not.

[0010] FIG. 1 is a diagram illustrating an overall configuration of a system. FIG. 2 is a block diagram illustrating an example of a configuration of a determination device according to an embodiment. FIG. 3 is a diagram illustrating an example of data stored in the determination device according to an embodiment. FIG. 4 is a diagram illustrating an example of data stored in the determination device according to an embodiment. FIG. 5 is a diagram illustrating a specific example of the number of tokens of data partially deleted by processing in a deletion unit according to an embodiment. FIG. 6 is a diagram illustrating a specific example of pseudocode for a deletion process of HTML source code in a deletion unit according to an embodiment. FIG. 7 is a diagram illustrating a specific example of processing of the determination device according to an embodiment. FIG. 8 is a diagram illustrating a specific example of an input prompt input to a large-scale language model according to an embodiment. FIG. 9 is a diagram illustrating a specific example of a response result of a large-scale language model according to an embodiment. FIG. 10 is a flowchart illustrating an example of the overall processing flow of the determination device according to an embodiment. FIG. 11 is a diagram illustrating an example of a computer that executes a determination program.

[0011] Hereinafter, embodiments of a determination device, a determination method, and a determination program according to the present application will be described in detail with reference to the accompanying drawings. However, the determination device, the determination method, and the determination program according to the present application are not limited to these embodiments.

[0012] [1. Overall Configuration] First, the overall configuration of a system including a determination device 100 according to this embodiment will be described. Fig. 1 is a diagram showing a system including a determination device according to an embodiment. The system shown in Fig. 1 is configured with the determination device 100 and an external device 200 that transmits to the determination device 100 the URL (Uniform Resource Locator) of a website to be determined.

[0013] The determination device 100 is an information processing device that receives a website URL transmitted from the external device 200 and determines whether the website indicated by the received URL is a phishing site, and is configured, for example, by a computer, a server device, etc. The external device 200 is an information processing device that is communicably connected to the determination device 100, and, for example, transmits the URL of the website to be determined to the determination device 100, and receives and displays information on the determination result output from the determination device 100.

[0014] The determination device 100 acquires a plurality of pieces of information related to a website, deletes a portion of the acquired pieces of information related to the website so as to satisfy a predetermined condition, and then inputs the deleted data into a large-scale language model and determines whether the website is a phishing site based on the output result of the large-scale language model.

[0015] First, the determination device 100 receives the URL of the website to be determined, transmitted, for example, from the external device 200, and obtains the URL of the website, the HTML (Hyper Text Markup Language) source code of the website, and a screenshot image of the screen displaying the website from the website indicated by the URL.

[0016] Next, the determination device 100 deletes, for example, portions of the HTML source code and the text data extracted from the screenshot image so that the number of tokens is equal to or less than a predetermined number. Thereafter, the determination device 100 inputs, for example, the URL, the deleted HTML source code, and the deleted text data of the screenshot image into an input prompt template of a large-scale language model, and analyzes the response result from the input to determine whether the target website is a phishing site.

[0017] This allows the determination device 100 to appropriately determine whether a website is a phishing site from the URL of the website being determined, and by analyzing the response results of the large-scale language model, it can interpret the context from the text displayed on the phishing site to identify false information and discern psychological techniques used to deceive users.

[0018] 2. Configuration of Determination Device 100 Next, the configuration of the determination device 100 shown in Fig. 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example configuration of the determination device according to the embodiment. The determination device 100 has a communication unit 110, a control unit 120, and a storage unit 130, and is connected to an external device 200 via a network N so that they can communicate with each other.

[0019] The communication unit 110 is realized by, for example, a network interface card (NIC). The communication unit 110 is connected to the network N and transmits and receives information to and from the external device 200. The communication unit 110, for example, receives the URL of the website to be determined from the external device 200 and mediates the acquisition of information about the website by the acquisition unit 121 (described later).

[0020] The storage unit 130 is realized by, for example, a storage device such as a RAM (Random Access Memory) or a hard disk. The storage unit 130 stores data and programs necessary for various processes performed by the control unit 120. The storage unit 130 includes, for example, an acquired data storage unit 131, a post-deletion data storage unit 132, and a response result data storage unit 133.

[0021] The acquired data storage unit 131 stores information about the website to be determined that is acquired by the acquisition unit 121, which will be described later. Here, the data stored in the acquired data storage unit 131 will be described with reference to FIG. 3 . FIG. 3 is a diagram showing an example of data stored in the determination device according to the embodiment. The acquired data storage unit 131 shown in the example of FIG. 3 is configured with the following items: "URL," "HTML source code," and "screenshot image."

[0022] "URL" stores the URL of the website to be evaluated that was sent from the external device 200. "HTML source code" stores the HTML source code that constitutes the website to be evaluated. "Screenshot image" stores image data of a screenshot image of the screen that displays the website to be evaluated.

[0023] The post-deletion data storage unit 132 stores data of HTML source code that has been partially deleted by the deletion unit 122 (described later) and text data extracted from screenshot images. For example, the post-deletion data storage unit 132 stores data of HTML source code that has been partially deleted and simplified by the deletion unit 122 (described later) so that the number of tokens is a predetermined number or less, and text data extracted from screenshot images that has been partially deleted and simplified so that the number of tokens is a predetermined number or less.

[0024] The response result data storage unit 133 stores data on response results from a large-scale language model. Here, the data stored in the response result data storage unit 133 will be described with reference to FIG. 4 . FIG. 4 is a diagram showing an example of data stored in the determination device according to the embodiment. The response result data storage unit 133 shown in the example of FIG. 4 is configured with the following items: "URL," "phishing_score," "brands," "phishing," and "suspicious_domain."

[0025] "phishing_score" is a score indicating the possibility that the website being judged is a phishing site, and stores a score value displayed as a number on a scale of 1 to 10. "brands" stores information such as the brand name of the legitimate website that the website being judged is impersonating. "phishing" stores the result of the judgment as to whether the website being judged is a phishing site as "true" or "false." "suspicious_domain" stores the result of the judgment as to the suspiciousness of the domain of the website being judged as "true" or "false."

[0026] Returning to the explanation of Fig. 2, the control unit 120 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in a storage device within the device using RAM as a work area. The control unit 120 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 120 has an acquisition unit 121, a deletion unit 122, a determination unit 123, and, if necessary, a generation unit 124.

[0027] The acquisition unit 121 acquires multiple pieces of information related to a website. For example, the acquisition unit 121 accesses the target website from the input website URL and acquires the website URL, the website's HTML source code, and a screenshot image of the screen displaying the website. The acquisition unit 121 then stores the acquired information in the acquired data storage unit 131.

[0028] Here, the acquisition unit 121 can, for example, automatically operate a web browser from the input website URL to reach the target website and acquire multiple pieces of information about the website. Note that the automatic operation of the web browser can be performed by using an automatic browser operation tool such as Selenium, Puppeteer, or Chrome DevTools Protocol, to execute web access to any URL.

[0029] In addition to the information described above, the acquisition unit 121 can also acquire communication destination information such as the URLs of websites passed through before reaching the website to be judged, the IP addresses of the websites, etc. By using the above-mentioned three pieces of information, namely the URL of the website, the HTML source code of the website, and a screenshot image of the screen displaying the website, in the input prompt described below, it is possible to perform the judgment process described below with high accuracy.

[0030] The deletion unit 122 deletes some of the pieces of information related to the website acquired by the acquisition unit 121 so as to satisfy a predetermined condition. For example, the deletion unit 122 references the data stored in the acquired data storage unit 131 and deletes some of the data, such that the number of tokens in each of the HTML source code data and the screenshot image data is equal to or less than a predetermined number. The deletion unit 122 then stores the deleted data in the post-deletion data storage unit 132.

[0031] Here, the processing of the deletion unit 122 will be described with reference to FIG. 5 , citing specific numbers of tokens. FIG. 5 is a diagram showing a specific example of the number of tokens of data partially deleted by processing in the deletion unit according to the embodiment. The large-scale language model used by the determination unit 123, which will be described later, has a limit on the number of inputs (number of tokens) of its input prompt. Therefore, to create a prompt to be input to the large-scale language model using the data acquired by the acquisition unit 121 described above, the input data is simplified by partially deleting it so that the number of tokens of the input prompt is equal to or less than the maximum number of tokens of the large-scale language model to be used.

[0032] 5 shows an example of a breakdown of the number of tokens set when using a large-scale language model in which the maximum number of tokens in an input prompt is limited to 4096. Specifically, the example shows 362 prompt templates, which are pre-created template portions other than acquired data, 3000 HTML source codes, 500 text extracted from screenshot images, and 234 URLs.

[0033] Here, the number of tokens for prompt templates and URLs varies only slightly depending on the content of the website being evaluated, whereas the number of tokens for HTML source code and text extracted from screenshot images varies widely. Therefore, the deletion unit 122 simplifies the acquired HTML source code data and screenshot image data by deleting a portion of each data so that the number of tokens for the HTML source code is 3,000 or less and the number of tokens for the text extracted from the screenshot image is 500 or less.

[0034] This allows the determination device 100 to always appropriately create an input prompt with the maximum number of tokens limited by the large-scale language model or less, regardless of the amount of data on the website to be determined.

[0035] Here, the deletion process of HTML source code will be described with reference to Fig. 6. Fig. 6 is a diagram showing a specific example of pseudo code for the deletion process of HTML source code in the deletion unit according to the embodiment. The deletion unit 122 deletes elements that satisfy a predetermined condition from the HTML source code acquired by the acquisition unit 121 so that the number of tokens is equal to or less than a predetermined number. Below, the process performed by the deletion unit 122 will be described in order, corresponding to the pseudo code shown in Fig. 6.

[0036] The deletion unit 122 first deletes HTML style / script / comment elements (corresponding to line 2 in FIG. 6 ). Next, the deletion unit 122 determines whether the number of tokens is 3000 (a predetermined number) or less (corresponding to line 4 in FIG. 6 ). If the number of tokens exceeds 3000, the deletion unit 122 unwraps HTML elements other than important HTML tags (corresponding to line 7 in FIG. 6 ). Examples of important HTML tags include head, title, meta, body, h1, h2, h3, h4, h5, h6, p, strong, a, img, hr, table, tbody, tr, th, td, ol, ul, li, ruby, and label.

[0037] Thereafter, the deletion unit 122 deletes HTML elements that do not contain text (corresponding to line 8 in FIG. 6 ). Next, the deletion unit 122 removes href elements in a tags and src elements in img tags (corresponding to line 9 in FIG. 6 ). The deletion unit 122 then repeatedly deletes HTML elements at midpoints of the HTML (lines 11 to 18 in FIG. 6 ) until the number of tokens becomes 3,000 or less. When the number of tokens becomes 3,000 or less, the deletion unit 122 deletes some of the data and stores the simplified HTML in the post-deletion data storage unit 132.

[0038] By performing the series of processes described above, the deletion unit 122 can create simplified HTML source code data in which the number of tokens is reduced to a predetermined number or less, while retaining the important elements that characterize a phishing site and deleting elements that are unnecessary for judgment.

[0039] Here, the process of deleting data extracted from a screenshot image will be described. The deletion unit 122 extracts text data from the screenshot image acquired by the acquisition unit 121, and deletes text from the text data in ascending order of character size so that the number of tokens remains below a predetermined number.

[0040] For example, the deletion unit 122 references the data stored in the acquired data storage unit 131 and extracts text data from a screenshot image of a website using OCR (Optical Character Recognition), which can acquire the character size of the extracted text using an image as input. Then, the deletion unit 122 deletes text from the extracted text data, for example, in ascending order of character size, so that the number of tokens becomes a predetermined number (e.g., 500) or less.

[0041] This allows the deletion unit 122 to create simplified text data with a predetermined number of tokens or less, while retaining text with large character sizes that are considered relatively important in determining whether a site is a phishing site.

[0042] The determination unit 123 inputs the data from which part has been deleted by the deletion unit 122 into the large-scale language model, and determines whether the website is a phishing site based on the output result of the large-scale language model.

[0043] For example, the determination unit 123 inputs into the large-scale language model an input prompt generated by the generation unit 124 (described later) using the URL of the website to be determined stored in the acquired data storage unit 131, and the HTML source code data and text data from which part of the data stored in the deleted data storage unit 132 has been deleted. Then, the determination unit 123 references the data stored in the response result data storage unit 133, for example, and determines the site to be a phishing site if either the phihing key (corresponding to "phishing" in FIG. 4 ) or the suspicious_domain key (corresponding to "suspicious_domain" in FIG. 4 ), which are the response results of the large-scale language model, is "true," and determines the site to be a non-phishing site if both are "false."

[0044] In addition to the response results of the phishing key and suspicious_domain key described above, the judgment unit 123 can also determine whether a site is a phishing site based on, for example, whether the value of the "phishing_score" (see Figure 4) is equal to or greater than a preset threshold.

[0045] The generation unit 124 generates an input prompt for the large-scale language model using the URL of the website acquired by the acquisition unit 121 and the data from which a portion has been deleted by the deletion unit 122. For example, the generation unit 124 substitutes the URL of the website to be determined stored in the acquired data storage unit 131 and the HTML source code data and text data from which a portion has been deleted and stored in the deleted data storage unit 132 into a template of an input prompt created in advance, thereby generating an input prompt to be input to the large-scale language model.

[0046] 3. Specific Example of Processing by the Determination Device 100 Now, with reference to Fig. 7, an overall processing flow of the determination processing performed by the determination device 100 will be described. Fig. 7 is a diagram showing a specific example of processing by the determination device according to the embodiment. Fig. 7 shows an example of processing from reception of a URL to be determined via the communication unit 110 to outputting a final determination of whether or not the URL is a phishing site.

[0047] First, the determination device 100 receives an input of the URL of the website to be determined from the external device 200, etc. Next, the acquisition unit 121 uses a program such as a web crawler to acquire information about the website to be determined.

[0048] Next, the deletion unit 122 deletes part of the data from the HTML source code information of the website so that the number of tokens is equal to or less than a predetermined number. The deletion unit 122 also extracts text from a screenshot image of the website using OCR, and deletes part of the text data from the extracted text so that the number of tokens is equal to or less than a predetermined number. As a result, the deletion unit 122 creates simplified HTML data with some parts deleted, simplified OCR-extracted text with some parts deleted, and a URL for the website to be determined.

[0049] The generation unit 124 then generates an input prompt to be input to the large-scale language model by substituting the deleted and simplified HTML data, the deleted and simplified OCR-extracted text, and the URL into a pre-created template. The determination unit 123 then inputs the generated input prompt into the large-scale language model, analyzes the response, and determines whether the target website is a phishing site. The determination device 100 then outputs the determination result by the determination unit 123 to the external device 200, etc.

[0050] Next, an input prompt to be input to a large-scale language model will be described with reference to FIG. 8. FIG. 8 is a diagram showing a specific example of an input prompt to be input to a large-scale language model according to an embodiment. (1) to (4) in FIG. 8 describe, as prompt templates, sentences relating to the specification of the contents of the stack to be executed by the large-scale language model, regardless of the contents of the website to be determined. (5) in FIG. 8 shows items into which the URL acquired by the acquisition unit 121, HTML data partially deleted by the deletion unit 122, and text data are substituted.

[0051] Specifically, (1) in Figure 8 describes a sentence that assigns the role and task of "being a security expert and identifying phishing sites" to the large-scale language model as the main task. (2) in Figure 8 describes the four subtasks in the process of identifying phishing sites to be performed in order.

[0052] Next, (3) in Fig. 8 describes the constraints of the task, which indicate that the HTML may be shortened or simplified, and that the text extracted by OCR may not be accurate. Finally, (4) in Fig. 8 describes examples of psychological manipulation techniques used in social engineering on phishing sites, such as displaying fake security warnings.

[0053] Next, (5) in Figure 8 lists the items into which the URL of the website to be judged, HTML source code data, and text data extracted from the screenshot image are assigned, and an input prompt is generated by entering each piece of data into the items.

[0054] 8 improves the analytical accuracy of the large-scale language model by assigning the role of "security expert" to the large-scale language model, dividing subtasks into stages and performing them in order, and indicating characteristics found in phishing sites. As a result, the generation unit 124 can generate a high-quality input prompt that improves the accuracy of the determination by substituting information acquired from the website to be determined into the input prompt template shown in FIG.

[0055] Next, a response result of a large-scale language model to an input prompt will be described with reference to Fig. 9. Fig. 9 is a diagram showing a specific example of a response result of a large-scale language model according to an embodiment. Fig. 9 shows a response result when an input prompt generated by substituting the URL of a website to be determined, HTML source code, and text data extracted from a screenshot image into the input prompt template shown in Fig. 8 is input to the large-scale language model.

[0056] Specifically, the answers to the four subtasks are shown in the input prompt shown in Figure 8. First, the answer to the first subtask asks about each of the entered data items, including suspicious elements (elements that can be determined to be phishing sites) and the basis for such a determination.

[0057] Next, the answer to the second subtask indicates the brand name of the legitimate website that the website being evaluated is allegedly impersonating. In the example of Figure 9, it is identified that the website being evaluated is impersonating a brand called "Example Company." Next, in the third subtask, the conclusion of the determination as to whether the website being evaluated is a phishing site is indicated. In the example of Figure 9, it is indicated that there is a high possibility that the website being evaluated is a phishing site for the brand called "Example Company."

[0058] The answer to the fourth subtask shows the output result of the determination made by the large-scale language model in JavaScript (registered trademark) Object Notation (JSON) format. In the example of FIG. 9, the results of the determination are "phishing_score: 8," "brands: Example Company," "phishing: true," and "suspicious_domain: true." The results of the determination are stored in the response result data storage unit 133.

[0059] 9 described above, the determination unit 123 determines that the target website is a phishing site by referring to, for example, the information "phishing: true" and "suspicious_domain: true" stored in the response result data storage unit 133. The determination device 100 then outputs, for example, the determination result that the target website is determined to be a phishing site, together with the URL of the target website to the external device 200. Note that the determination device 100 may output, in addition to the URL and the determination result, for example, the entire response result shown in FIG. 9 so that the basis for the determination made by the large-scale language model can be understood.

[0060] 4. Example of Processing of Determination Device 100 Next, the flow of processing by the determination device 100 will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of the flow of processing by the determination device according to the embodiment. Note that the steps described below may be executed in a different order, and some processing may be omitted. Furthermore, the processing procedures of the respective embodiments may be implemented in an appropriate combination.

[0061] First, the determination device 100 receives the URL of the website to be determined from the external device 200 or the like (S101). If the determination device 100 receives the URL of the website to be determined (S101; Yes), the acquisition unit 121 accesses the website from the received URL and acquires the URL of the website to be determined, HTML source code, and screenshot image (S102). On the other hand, if the determination device 100 has not received the URL of the website to be determined (S101; No), the determination device 100 waits until it receives the URL of the website to be determined.

[0062] After the process of S102, the deletion unit 122 deletes portions of the HTML source code and the text data extracted from the screenshot image so as to satisfy predetermined conditions (S103). Subsequently, the generation unit 124 generates an input prompt by substituting the URL, the HTML source code from which the portions have been deleted, and the text data from which the portions have been deleted (S104).

[0063] The determination unit 123 then determines whether the target website is a phishing site based on the response of the large-scale language model to the generated input prompt (S105). The determination device 100 then outputs the determination result to the external device 200 or the like (S106), and the process ends.

[0064] 5. Effects of the Embodiment As described above, the determination device 100 according to the present embodiment includes an acquisition unit 121, a deletion unit 122, and a determination unit 123. The acquisition unit 121 acquires multiple pieces of information related to websites. The deletion unit 122 deletes some of the multiple pieces of information related to websites acquired by the acquisition unit 121 so as to satisfy a predetermined condition. The determination unit 123 inputs the data from which some of the information has been deleted by the deletion unit 122 into a large-scale language model, and determines whether the website is a phishing site based on the output result of the large-scale language model.

[0065] As a result, the determination device 100 can analyze the response results of a large-scale language model to an input prompt with a predetermined number of tokens or less, which is created using the configuration information of the website to be determined, and appropriately determine whether the website to be determined is a phishing site.

[0066] Furthermore, the acquisition unit 121 of the determination device 100 accesses the website indicated by the input website URL and acquires multiple pieces of information related to the website. This allows the determination device 100 to acquire multiple pieces of information related to the website required for the determination process using an automatic browser operation tool such as a web crawler, based on the URL of the website to be determined received from outside.

[0067] Furthermore, the acquisition unit 121 of the determination device 100 acquires, as a plurality of pieces of information related to the website, the URL of the website, the HTML source code of the website, and a screenshot image of a screen displaying the website. This allows the determination device 100 to acquire, from the website to be determined, a plurality of pieces of information that will enable the determination result based on the large-scale language model to be highly accurate.

[0068] Furthermore, the deletion unit 122 of the determination device 100 deletes elements that satisfy a predetermined condition from the HTML source code acquired by the acquisition unit 121 so that the number of tokens is equal to or less than a predetermined number. This allows the determination device 100 to create simplified HTML source code data in which the number of tokens is equal to or less than a predetermined number, by retaining important elements that characterize a phishing site and deleting elements that are not necessary for the determination.

[0069] Furthermore, the deletion unit 122 of the determination device 100 extracts text data from the screenshot image acquired by the acquisition unit 121, and deletes text from the text data in ascending order of character size, so that the number of tokens is equal to or less than a predetermined number. This allows the determination device 100 to create simplified text data in which the number of tokens is equal to or less than a predetermined number, while retaining text with large character size that is considered relatively important in determining whether a site is a phishing site.

[0070] The determination device 100 also includes a generation unit 124. The generation unit 124 generates an input prompt for the large-scale language model using the URL of the website acquired by the acquisition unit 121 and the data from which a portion has been deleted by the deletion unit 122. In this case, the determination unit 123 inputs the input prompt generated by the generation unit 124 into the large-scale language model. This allows the determination device 100 to substitute each piece of acquired or partially deleted data into a template created in advance, thereby generating a high-quality input prompt that will enable the large-scale language model to make highly accurate judgments.

[0071] [6. System Configuration, etc.] Of the processes described in the above embodiments, some of the processes described as being performed automatically can also be performed manually. Alternatively, all or some of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0072] Furthermore, the components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of the devices can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0073] 2 may be held in a storage server or the like, instead of being held by the determination device 100. In this case, the determination device 100 acquires various pieces of information by accessing the storage server.

[0074] 7. Hardware Configuration Fig. 11 is a diagram showing an example of a hardware configuration. The determination device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 11, for example.

[0075] 11 is a diagram showing an example of a computer that executes an analysis program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0076] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0077] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the determination device 100 is implemented as a program module 1093 in which code that can be executed by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the determination device 100 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0078] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0079] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may instead be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN, WAN, etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0080] REFERENCE SIGNS LIST 100 Determination device 110 Communication unit 120 Control unit 121 Acquisition unit 122 Deletion unit 123 Determination unit 124 Generation unit 130 Storage unit 131 Acquired data storage unit 132 Post-deletion data storage unit 133 Response result data storage unit 200 External device

Claims

1. A determination device comprising: an acquisition unit that acquires multiple pieces of information related to a website; a deletion unit that deletes a portion of the multiple pieces of information related to the website acquired by the acquisition unit so as to satisfy specified conditions; and a determination unit that inputs the data from which a portion has been deleted by the deletion unit into a large-scale language model, and determines whether the website is a phishing site based on the output result of the large-scale language model.

2. The determination device according to claim 1, characterized in that the acquisition unit accesses a website indicated by an input website URL (Uniform Resource Locator) and acquires a plurality of pieces of information relating to the website.

3. The determination device according to claim 1, characterized in that the acquisition unit acquires the following pieces of information relating to the website: the URL of the website, the HTML (Hyper Text Markup Language) source code of the website, and a screenshot image of a screen displaying the website.

4. The determination device according to claim 3, characterized in that the deletion unit deletes elements that satisfy a predetermined condition from the HTML source code acquired by the acquisition unit so that the number of tokens is a predetermined number or less.

5. The determination device according to claim 3, characterized in that the deletion unit extracts text data from the screenshot image acquired by the acquisition unit, and deletes text from the text data in ascending order of character size so that the number of tokens is less than a predetermined number.

6. The determination device according to claim 3, further comprising a generation unit that generates an input prompt for the large-scale language model using the URL of the website acquired by the acquisition unit and the data from which a portion has been deleted by the deletion unit, and the determination unit inputs the input prompt generated by the generation unit into the large-scale language model.

7. A determination method executed by a determination device, comprising: an acquisition step of acquiring multiple pieces of information related to a website; a deletion step of deleting a portion of the multiple pieces of information related to the website acquired by the acquisition step so as to satisfy a predetermined condition; and a determination step of inputting the data from which a portion has been deleted by the deletion step into a large-scale language model, and determining whether or not the website is a phishing site based on the output result of the large-scale language model.

8. A judgment program for causing a computer to execute the following steps: an acquisition step for acquiring multiple pieces of information related to a website; a deletion step for deleting some of the multiple pieces of information related to the website acquired by the acquisition step so as to satisfy specified conditions; and a judgment step for inputting the data from which some of the information has been deleted by the deletion step into a large-scale language model and judging whether or not the website is a phishing site based on the output result of the large-scale language model.

Images