Proposal device

Abstract

A proposal device according to one aspect of the present disclosure proposes a test item for a network test on a network device. Upon receiving an inquiry including information on an abnormal state of the network device, a plurality of large language models cooperatively operate to specify a field related to the abnormal state among fields included in the header of a packet, and propose a test item including the specified field.

Description

Proposed device

[0001] The present disclosure relates to a proposal device.

[0002] There is a risk that a network device may go into an abnormal state, such as a malfunction, when the processing of a specific packet is triggered by a software bug, a configuration error, a hardware malfunction, etc. Therefore, generating packets that can cause an abnormal state in the network device, testing the network device with those packets, or simulating an abnormal state is an important task for evaluating the fault tolerance, etc., of the network device.

[0003] As a conventional technique for packet generation, a technique for generating packets similar to actual packets prepared in advance is known (for example, Non-Patent Documents 1 and 2).

[0004] Cheng, Adriel. "PAC-GAN: Packet generation of network traffic using generative adversarial networks." 2019 IEEE 10th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON). IEEE, 2019.Nukavarapu, Santosh Kumar, Mohammed Ayyat, and Tamer Nadeem. "MirageNet-towards a GAN-based framework for synthetic network traffic generation." GLOBECOM 2022-2022 IEEE Global Communications Conference. IEEE, 2022.

[0005] However, since there are many fields in a packet that can cause an abnormal state in a network device, for example, when testing a network device using a packet that can cause a certain abnormal state, the test items may include fields that are not related to the abnormal state, which unnecessarily broadens the scope of the test, resulting in an increase in test costs (e.g., human costs, time costs, etc. required for testing).

[0006] The present disclosure has been made in consideration of the above points, and aims to propose efficient test items for network devices.

[0007] A proposal device according to one aspect of the present disclosure is a proposal device that proposes test items for a network test on a network device, and when it receives a query containing information about an abnormal state of the network device, multiple large-scale language models work in cooperation to identify fields contained in the packet header that are related to the abnormal state, and propose test items that include the identified fields.

[0008] It is possible to propose efficient test items for network devices.

[0009] 1 is a diagram showing an example of the overall configuration of a network testing system according to the present embodiment; FIG. 2 is a diagram showing an example of the functional configuration of a test item recommendation system according to the present embodiment; FIG. 3 is a diagram showing an example of instructions (part 1) given to a test item recommendation LLM; FIG. 4 is a diagram showing an example of instructions (part 2) given to a test item recommendation LLM; FIG. 5 is a diagram showing an example of instructions given to an information collection LLM; FIG. 6 is a flowchart showing an example of the operation of a network testing system according to the present embodiment; FIG. 7 is a flowchart showing an example of a test item proposal process according to the present embodiment; FIG. 8 is a diagram showing an example of proposed test items; FIG. 9 is a diagram showing an example of the hardware configuration of a test item recommendation system according to the present embodiment;

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0011] <Background> Network devices present in the backbone networks of Internet Service Providers (ISPs) play an extremely important role in maintaining the stability and speed of the Internet. These network devices function as routers, switches, firewalls, etc., and are responsible for sending and receiving data, selecting routes, and implementing security measures. However, these network devices may experience abnormal conditions, such as a malfunction, triggered by the processing of a specific packet due to, for example, a software bug, a configuration error, or a hardware malfunction. Hereinafter, packets that may cause an abnormal condition in a network device are also referred to as "abnormal packets."

[0012] Abnormal packets can have a significant impact on the security and performance of network devices. Therefore, generating abnormal packets and then testing network devices with those packets or simulating abnormal conditions is an important task for evaluating the fault tolerance of network devices. Furthermore, detecting abnormal packets during actual operation makes it possible to detect abnormal conditions in network devices early and take appropriate measures, thereby maintaining the safety and reliability of the network.

[0013] Known conventional packet generation techniques use machine learning algorithms to generate packets similar to actual packets prepared in advance (e.g., Non-Patent Documents 1 and 2). By preparing many abnormal packets in advance and then using these conventional techniques, it is possible to generate many abnormal packets similar to the abnormal packets.

[0014] <Problems with the Prior Art> Packets contain many fields that can cause an abnormal state in a network device. Therefore, when testing a network device using an abnormal packet that can cause a certain abnormal state, if the abnormal packet is generated using the above-described prior art, there is a possibility that fields that are not highly relevant to the abnormal state will be included in the test items. As a result, the test scope becomes unnecessarily broad, and the test costs (e.g., personnel costs and time costs required for the test) also increase. Note that a field refers to an area where various pieces of information constituting a packet header (e.g., version, header length, service type, total length, identification number, flags, fragment offset, time to live, protocol, header checksum, source IP address, destination IP address, etc.) are stored. A field may also be referred to as, for example, an attribute or an item. Hereinafter, the various pieces of information stored in a field will be referred to as "features," and the field itself will be referred to as a "feature field."

[0015] <Proposed Method> Hereinafter, assuming a case where a network device needs to be tested using an abnormal packet that may cause a certain abnormal state, a method (hereinafter referred to as the "proposed method") is proposed that can suggest test items to an operator that can efficiently perform the test. Note that an efficient test is a test that requires low test costs (e.g., personnel costs and time costs required for the test). Generally, the fewer feature fields included in a test item, the more efficiently the test can be performed. Therefore, to perform an efficient test, it is necessary to create accurate test items (i.e., test items that do not include feature fields that are not highly relevant to the abnormal state). Furthermore, an operator is a person who actually creates test items and is responsible for testing the network device. An operator may be called, for example, a "test engineer" or a "test technician."

[0016] The proposed method utilizes data belonging to various domains (e.g., device manuals, past failure cases, past verification procedures, etc.) and proposes test items in response to queries (hereinafter also referred to as "test item queries") that include information about abnormal conditions using a large-scale language model (LLM). More specifically, information related to abnormal conditions is collected from a large-scale language model that has acquired knowledge of data belonging to various domains (hereinafter also referred to as "domain knowledge"), and then this information is analyzed using the large-scale language model to propose test items with high accuracy related to the abnormal conditions. Note that domain knowledge may also be called "knowledge" or the like.

[0017] In particular, by using a model that realizes multimodal AI (e.g., GPT-4) as a large-scale language model for acquiring domain knowledge, it becomes possible to integrate and utilize data in a variety of formats, such as documents, images, tables, graphs, etc. As a result, it becomes possible to collect various information related to the abnormal state in question, and to efficiently identify feature fields that are highly relevant to the abnormal state. This increases the accuracy of test items and makes it possible to realize more efficient testing of network equipment.

[0018] The proposed method uses the following three large-scale language models:

[0019] Domain Knowledge LLM: A large-scale language model that acquires domain knowledge from data belonging to a certain domain, and provides information on the abnormal state and its relevance based on that domain knowledge. For example, a model that realizes multimodal AI (e.g., GPT-4) is used as the domain knowledge LLM. Note that a domain knowledge LLM exists for each domain.

[0020] There are various possible domains, such as equipment manuals, past failure cases, past verification procedures, service specifications, protocol specifications, and failure reports.

[0021] Information Gathering LLM: A large-scale language model that has knowledge that indicates the domain knowledge of each domain knowledge LLM and can collect information from those domain knowledge LLMs effectively and efficiently.

[0022] Test Item Recommendation LLM: A large-scale language model that suggests highly accurate test items in response to test item queries from operators. Based on information about abnormal conditions included in the test item query, the test item recommendation LLM requests the information collection LLM to collect information related to the abnormal condition. The test item recommendation LLM then proposes highly accurate test items by analyzing the information received from the information collection LLM. Note that, hereinafter, test items proposed by the test item recommendation LLM will also be referred to as "proposed test items."

[0023] In the proposed method, in response to a test item inquiry from an operator, the test item recommendation LLM requests the information collection LLM to collect information related to the corresponding abnormal state, and the information collection LLM collects information from one or more appropriate domain knowledge LLMs.The test item recommendation LLM then analyzes the information collected by the information collection LLM to identify feature fields related to the abnormal state and recommends test items containing those feature fields to the operator.This enables efficient recommendation of test items for network devices.

[0024] The following describes a network testing system 1 that proposes efficient test items for network devices using the above proposed method and tests the network devices using those test items (or test items modified by an operator, etc.).

[0025] <Example of Overall Configuration of Network Testing System 1> An example of the overall configuration of the network testing system 1 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the network testing system 1 according to this embodiment includes a test item recommendation system 10, an operator device 20, and a test target device 30.

[0026] In response to a test item query from the operator device 20, the test item recommendation system 10 transmits proposed test items in response to the test item query to the operator device 20. Furthermore, when the test item recommendation system 10 receives feedback information including test results for the test target device 30, the test item recommendation system 10 updates the domain knowledge of the appropriate domain knowledge LLM based on the feedback information. The test item recommendation system 10 is realized by one or more computers (e.g., a general-purpose server, a workstation, a PC (personal computer), etc.).

[0027] The test item inquiry includes information about the abnormal condition that is the target of the test. Examples of information about the abnormal condition include "details of the event that occurred," "target service," "target device," and "error message." "Details of the event that occurred" refers to information that indicates the detailed content of the abnormal condition, "target service" refers to information about the service used by the network device where the abnormal condition occurred (e.g., identification information for the service), "target device" refers to information about the network device where the abnormal condition occurred (e.g., identification information for the network device, the type of the network device), and "error message" refers to the error message that is displayed when the abnormal condition occurs.

[0028] The proposed test items include at least a feature field related to the corresponding abnormal condition. The proposed test items may also include various information other than the feature field. For example, the proposed test items may include information indicating the importance of the feature field, information indicating data related to the abnormal condition (e.g., equipment manuals or failure cases related to the abnormal condition), information indicating the type and procedure of a test recommended for the feature field, and so on.

[0029] The operator device 20 is a terminal (e.g., a PC, a smartphone, a tablet terminal, a wearable device, etc.) used by an operator. The operator device 20 creates a test item query and transmits the test item query to the test item recommendation system 10, and receives proposed test items from the test item recommendation system 10. The operator device 20 also transmits test packets to the test target device 30 according to the proposed test items (or test items obtained by modifying the proposed test items by the operator), collects test results from the test target device 30, and transmits feedback information including the test results to the test item recommendation system 10.

[0030] The test target device 30 is a network device that is the test target of the network test. The test target device 30 receives a test packet from the operator device 20 and transfers (transmits) the test packet to the test environment network 40 according to the destination IP address of the test packet.

[0031] 1 is merely an example, and is not intended to be limiting. For example, the network testing system 1 may include a plurality of operator devices 20 and a plurality of test target devices 30.

[0032] <Example of Functional Configuration of Test Item Recommendation System 10> An example of the functional configuration of the test item recommendation system 10 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the test item recommendation system 10 according to this embodiment includes a test item recommendation unit 101, an information collection unit 102, one or more domain knowledge units 103, one or more domain DBs 104 corresponding to each domain knowledge unit 103, and a feedback unit 105. The example shown in Fig. 2 shows a case where there are three domain knowledge units 103: a first domain knowledge unit 103-1, a second domain knowledge unit 103-2, and a third domain knowledge unit 103-3. Similarly, the example shown in Figure 2 shows a case where there is a first domain DB 104-1 corresponding to the first domain knowledge unit 103-1, a second domain DB 104-2 corresponding to the second domain knowledge unit 103-2, and a third domain DB 104-3 corresponding to the third domain knowledge unit 103-3.

[0033] The test item recommendation unit 101 is realized by a test item recommendation LLM. When the test item recommendation unit 101 receives a test item inquiry from the operator device 20, it transmits an information collection request representing a request to collect information related to the abnormal state to the information collection unit 102 based on information related to the abnormal state included in the test item inquiry. In addition, the test item recommendation unit 101 analyzes the information received from the information collection unit 102 to identify feature fields, etc. related to the abnormal state, and transmits proposed test items including the feature fields, etc. to the operator device 20.

[0034] The information collection unit 102 is realized by an information collection LLM. When the information collection unit 102 receives an information collection request from the test item recommendation unit 101, it transmits the information collection request to the domain knowledge unit 103, which is realized by a domain knowledge LLM having domain knowledge related to the abnormal state, based on information about the abnormal state included in the information collection request. Furthermore, when the information collection unit 102 receives information from each domain knowledge unit 103, it organizes the information as necessary and then transmits it to the test item recommendation unit 101.

[0035] The domain knowledge unit 103 is realized by the domain knowledge LLM. The domain knowledge unit 103 has domain knowledge of the domain corresponding to the domain knowledge LLM (i.e., domain knowledge of the data stored in the domain DB 104 corresponding to itself). For example, in the example shown in Figure 2, the first domain knowledge unit 103-1 has domain knowledge related to device manuals, the second domain knowledge unit 103-2 has domain knowledge related to past failure cases, and the third domain knowledge unit 103-3 has domain knowledge related to past verification procedures.

[0036] When the domain knowledge unit 103 receives an information collection request from the information collection unit 102, it transmits to the information collection unit 102 information represented by data stored in the domain DB 104 corresponding to itself based on the information collection request and its own domain knowledge.

[0037] The domain DB 104 is a database that stores data belonging to a certain domain. For example, in the example shown in Fig. 2, the first domain DB 104-1 stores data representing device manuals, the second domain DB 104-2 stores data representing past failure cases, and the third domain DB 104-3 stores data representing past verification procedures.

[0038] The format of the data stored in the domain DB 104 is not limited to a specific format, but may be any format (e.g., document, image, table, graph, etc.), and data of different formats may be mixed. Furthermore, the language of the information represented by the data stored in the domain DB 104 is not limited to a specific language, but may be any language (e.g., Japanese, English), and different languages ​​may be mixed.

[0039] When the feedback unit 105 receives feedback information from the operator device 20, it updates the domain knowledge of the domain knowledge LLM that realizes the appropriate domain knowledge unit 103 based on the test results and the like included in the feedback information.

[0040] 2, there are a first domain knowledge unit 103-1 having domain knowledge related to device manuals and a corresponding domain DB 104-1, a second domain knowledge unit 103-2 having domain knowledge related to past failure cases and a corresponding domain DB 104-2, and a third domain knowledge unit 103-3 having domain knowledge related to past verification procedures and a corresponding domain DB 104-3, but this is just an example, and there may also be domain knowledge units 103 having domain knowledge related to other domains and corresponding domain DBs 104. For example, there may also be a domain knowledge unit 103 having domain knowledge related to service specifications and a corresponding domain DB 104, a domain knowledge unit 103 having domain knowledge related to protocol specifications and a corresponding domain DB 104, a domain knowledge unit 103 having domain knowledge related to failure reports and a corresponding domain DB 104, etc.

[0041] The test item recommendation system 10 may be realized by a single computer or by multiple computers interconnected via a communication network. In particular, the test item recommendation unit 101, the information collection unit 102, each domain knowledge unit 103 and its corresponding domain DB 104, and the feedback unit 105 may be realized by different computers.

[0042] <Pre-configuration> The domain knowledge LLM, test item recommendation LLM, and information gathering LLM require the following pre-configuration: Note that these pre-configurations are performed on pre-trained large-scale language models.

[0043] <<Presetting of Domain Knowledge LLM>> The domain knowledge LLM receives data stored in the domain DB 104 corresponding to the domain knowledge unit 103 realized by the domain knowledge LLM as input and acquires domain knowledge from the data. This makes it possible, when information about an abnormal state is given, to acquire information related to the abnormal state from information represented by data belonging to that domain. Note that the domain knowledge is realized, for example, by a machine learning model that expresses the domain knowledge LLM and its parameter set.

[0044] <<Presetting of Test Item Recommendation LLM>> The test item recommendation LLM is given instructions 1100 shown in FIG. 3 and instructions 1200 shown in FIG.

[0045] The instruction 1100 shown in Fig. 3 is instruction data that defines how the test item recommendation LLM functions. By providing the instruction 1100 shown in Fig. 3, the test item recommendation LLM can execute a procedure that realizes the function defined in the instruction 1100.

[0046] The instruction 1200 shown in Figure 4 is instruction data that defines the format of the proposed test item. By providing the instruction 1200 shown in Figure 4, the test item recommendation LLM can create proposed test items in the format defined in the instruction 1200.

[0047] However, instruction 1100 is an example of an instruction that defines the function of the test item recommendation LLM, and is not limited to this example; other instructions that define equivalent functions may be provided to the test item recommendation LLM. Similarly, instruction 1200 is an example of an instruction that defines the format of proposed test items, and is not limited to this example; other instructions that define any format may be provided to the test item recommendation LLM.

[0048] <<Pre-setting of Information Collection LLM>> The information collection LLM is given instructions 1300 shown in Figure 5. The instructions 1300 shown in Figure 5 are instruction data that define how the information collection LLM functions (how it behaves). By giving the instructions 1300 shown in Figure 5, the information collection LLM can execute the procedures that realize the functions defined in the instructions 1300 (i.e., "identifying information sources," "establishing effective inquiry methods," "information collection process," and "continuous optimization" defined in the instructions 1300 shown in Figure 5).

[0049] However, instruction 1300 is an example of an instruction that defines the function of the information collection LLM, and is not limited to this, and other instructions that define equivalent functions may be given to the information collection LLM.

[0050] <Example of Operation of Network Testing System 1> An example of operation of the network testing system 1 according to this embodiment will be described with reference to FIG.

[0051] The test item recommendation unit 101 of the test item recommendation system 10 receives a test item inquiry from the operator device 20 (step S101). Here, the test item inquiry includes details of the occurring event, the target service, the target device, and an error message.

[0052] The test item recommendation system 10 executes a test item proposal process, which will be described later, to propose proposed test items to the operator device 20 (step S102).

[0053] The operator device 20 generates test packets according to the proposed test items proposed by the test item recommendation system 10 (or test items obtained by modifying the proposed test items by the operator, etc.), and then transmits the test packets to the test target device 30 (step S103). This allows a network test to be performed on the test target device 30. The test results are collected by the operator device 20.

[0054] If the test is to be ended (YES in step S104), the network testing system 1 ends its operation. On the other hand, if the test is not to be ended (NO in step S104), the operator device 20 transmits feedback information including the test results to the test item recommendation system 10 (step S105), and returns to step S101. This allows the test of the test target device 30 to be repeatedly performed.

[0055] When the above feedback information is transmitted to the test item recommendation system 10, the feedback unit 105 updates the domain knowledge of the domain knowledge LLM that realizes the appropriate domain knowledge unit 103 based on the test results, etc. included in the feedback information. More specifically, the feedback unit 105 re-learns the domain knowledge LLM that realizes the appropriate domain knowledge unit 103 based on the test results, etc. included in the feedback information, updates the domain knowledge, and stores the test results, etc. in the domain DB 104 corresponding to the domain knowledge unit 103. Repeated updates of the domain knowledge by the feedback unit 105 make it possible to create proposed test items using past test results and knowledge obtained from past tests, and improvement in the accuracy of the proposed test items can be expected.

[0056] Note that which domain knowledge unit 103 is the appropriate domain knowledge unit 103 may be determined by the feedback unit 105 or may be specified by an operator, etc. When the feedback unit 105 makes the determination, for example, if the test result indicates the occurrence of a failure, it may be considered to determine that the second domain knowledge unit 103-2 related to the past failure case is the appropriate domain knowledge unit 103.

[0057] <Test Item Proposal Process> The test item proposal process in step S102 of FIG. 6 will be described with reference to FIG.

[0058] The test item recommendation unit 101 sends an information collection request (hereinafter also referred to as a first information collection request) to the information collection unit 102 (information collection LLM) (step S201). That is, the test item recommendation unit 101 sends the first information collection request to the information collection unit 102 based on, for example, "collection and analysis of related data" defined in the instruction 1100 shown in FIG. 3. Here, the first information collection request includes details of the occurrence event, the target service, the target device, and an error message, similar to the test item inquiry.

[0059] When the information collection unit 102 receives the first information collection request from the test item recommendation unit 101 (test item recommendation LLM), it transmits an information collection request (hereinafter also referred to as a second information collection request) to each domain knowledge unit 103 (each domain knowledge LLM) related to the first information collection request (step S202). The transmission of the second information collection request to each domain knowledge unit 103 is performed based on, for example, the "information collection process" defined in the instruction 1300 shown in FIG. 5.

[0060] That is, the information collecting unit 102 identifies one or more domain knowledge units 103 related to the first information collection request based on each piece of information included in the first information collection request received from the test item recommending unit 101. These one or more domain knowledge units 103 are identified, for example, based on "identifying an information source" defined in the instruction 1300 shown in FIG. 5. Next, the information collecting unit 102 creates prompts for each of the identified one or more domain knowledge units 103, and sends a second information collection request including these prompts to the corresponding domain knowledge units 103. These one or more prompts are created, for example, based on "establishing an effective inquiry method" defined in the instruction 1300 shown in FIG. 5. The prompts also include, for example, keywords, questions, and levels of detail of information required to collect information related to the abnormal state represented by each piece of information included in the first information collection request.

[0061] When each domain knowledge unit 103 receives an information collection request from the information collection unit 102, it obtains information from the domain DB 104 corresponding to itself in accordance with the prompt included in the information collection request, and responds (transmits) that information to the information collection unit 102 (information collection LLM) (step S203).

[0062] The information collecting unit 102 transmits each piece of information received from each domain knowledge unit 103 (domain knowledge LLM) to the test item recommending unit 101 (test item recommending LLM) (step S204). At this time, the information collecting unit 102 may organize each piece of information received from each domain knowledge unit 103 according to a predetermined format based on, for example, the "continuous optimization" defined in the instruction 1300 shown in FIG. 5, and then transmit the organized information to the information collecting unit 102. The format is specified, for example, in the first information collection request.

[0063] The test item recommendation unit 101 creates proposed test items by analyzing the situation when an abnormality occurs based on the information received from the information collection unit 102 (step S205). That is, the test item recommendation unit 101, for example, identifies feature fields based on the "analysis of the situation when an abnormality occurs" defined in the instruction 1100 shown in Fig. 3, and then creates proposed test items based on the "suggestion of test items" defined in the instruction 1100 shown in Fig. 3. In addition, when creating proposed test items, the test item recommendation unit 101 creates the proposed test items in the format defined in the instruction 1200 shown in Fig. 4.

[0064] The test item recommending unit 101 transmits the proposed test items to the operator device 20 (step S206). As a result, test items that include at least the feature quantity field related to the abnormal state are proposed to the operator.

[0065] <<Proposed Test Item>> Fig. 8 shows an example of a proposed test item created in step S205 of Fig. 7. The proposed test item 2000 shown in Fig. 8 is a test item created according to the format defined in the instruction 1200 shown in Fig. 4. The proposed test item 2000 shown in Fig. 8 includes the feature field name "TTL value of IP header" related to the abnormal state, as well as the importance of the field, related failure cases and sentences, and proposed test actions. By proposing such a proposed test item 2000, the operator can learn not only the feature field related to the abnormal state, but also the importance of the feature field, related failure cases and sentences, proposed test actions, etc.

[0066] <Example of Hardware Configuration of Test Item Recommendation System 10> Fig. 9 shows an example of a hardware configuration when the test item recommendation system 10 according to this embodiment is realized by a computer such as a general-purpose server. As shown in Fig. 9, the test item recommendation system 10 according to this embodiment has an input device 201, a display device 202, an external I / F 203, a communication I / F 204, a RAM (Random Access Memory) 205, a ROM (Read Only Memory) 206, an auxiliary storage device 207, and a processor 208. Each of these pieces of hardware is connected to each other so as to be able to communicate with each other via a bus 209.

[0067] The input device 201 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 202 is, for example, a display, a display panel, etc. Note that the test item recommendation system 10 does not necessarily have to include at least one of the input device 201 and the display device 202, for example.

[0068] The external I / F 203 is an interface with an external device such as a recording medium 203 a. Examples of the recording medium 203 a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0069] The communication I / F 204 is an interface for communicating with other devices such as the operator device 20. The RAM 205 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 206 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 207 is a non-volatile storage device (storage device) such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The processor 208 is one of various arithmetic devices such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0070] 9 is an example, and the hardware configuration of the test item recommendation system 10 is not limited to this. For example, the test item recommendation system 10 may have multiple auxiliary storage devices 207 or multiple processors 208, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0071] <Modification> The operator can also modify the proposed test items proposed in step S102 of Fig. 6 as appropriate and test the test target device 30 according to the modified test items. Therefore, when the proposed test items are modified, the operator device 20 may feed back the modifications to the test item recommendation unit 101 (test item recommendation LLM). This allows the test item recommendation LLM to be retrained according to the feedback, and it is expected that more accurate proposed test items can be created.

[0072] As described above, the test item recommendation system 10 according to this embodiment combines multiple large-scale language models to efficiently propose test items for network devices. In particular, by combining and cooperating a test item recommendation LLM specialized in information analysis and test item proposal, an information collection LLM specialized in information collection, and one or more domain knowledge LLMs specialized in each domain knowledge, more accurate information collection and analysis becomes possible compared to the case where a single LLM is used, and as a result, more accurate test item proposals can be realized.

[0073] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0074] REFERENCE SIGNS LIST 1 Network test system 10 Test item recommendation system 20 Operator device 30 Test target device 40 Test environment network 101 Test item recommendation unit 102 Information collection unit 103 Domain knowledge unit 104 Domain DB 105 Feedback unit 201 Input device 202 Display device 203 External I / F 203a Recording medium 204 Communication I / F 205 RAM 206 ROM 207 Auxiliary storage device 208 Processor 209 Bus

Claims

1. A proposal device that proposes test items for a network test on a network device, wherein, upon receiving a query containing information related to an abnormal state of the network device, a plurality of large-scale language models work in cooperation to identify fields contained in a packet header that are related to the abnormal state, and propose test items that include the identified fields.

2. The proposal device of claim 1, wherein the plurality of large-scale language models include: domain knowledge LLMs each having knowledge of data belonging to one or more specified domains; an information gathering LLM that gathers information represented by the data from one or more of the domain knowledge LLMs; and a test item recommendation LLM that identifies the field and proposes the test items by analyzing the information gathered by the information gathering LLM.

3. The proposal device according to claim 2, wherein the test item recommendation LLM, upon receiving the inquiry, requests the information collection LLM to collect information related to the abnormal state; the information collection LLM identifies the domain knowledge LLM that is the source of the information and collects the information from the identified domain knowledge LLM; and the test item recommendation LLM, having acquired the information from the information collection LLM, analyzes the information to identify fields related to the abnormal state and proposes test items that include the identified fields.

4. The proposed device according to claim 2 or 3, wherein the one or more predetermined domains include one or more of the following domains: device manual, past failure cases, past verification procedures, service specifications, protocol specifications, and failure reports.

Images