Method and device for training large language model to identify high-risk webpage service, electronic equipment, medium and program product

Abstract

The invention provides a method and a device for training a large language model to identify a high-risk webpage service, electronic equipment, a medium and a program product, and relates to the technical field of artificial intelligence and network security, in particular to the field of large language models, webpage applications and services. According to the implementation scheme, a large language model is preliminarily trained based on first training data, wherein the first training data comprises first sample content information, a first judgment rule used for judging whether a first sample webpage service is a high-risk webpage service or not, and an attribute mark; calling the preliminarily trained large language model to recognize the second sample webpage service to obtain a preliminary recognition result; determining a wrong recognition result in the preliminary recognition result; and carrying out secondary training on the preliminarily trained large language model based on a preset dialogue context, wherein the secondary training comprises second sample content information extracted from a second sample webpage service corresponding to an error recognition result and a second judgment rule used for judging whether the second sample webpage service is a high-risk webpage service or not.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and network security technology, and in particular to the fields of large language models, web applications and services, and specifically to a method for training a large language model to identify high-risk web services, a high-risk web service identification method based on a large language model, an intelligent agent, a device, an electronic device, a computer-readable storage medium and a computer program product based on a large language model. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] In recent years, artificial intelligence technology, especially technology related to large language models, has gradually shown its application potential in dealing with complex network environments. With the development of technology, such methods will play an increasingly important role in future network security practices.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a method for training a large language model to identify high-risk web page services, a high-risk web page service identification method based on a large language model, an intelligent agent based on a large language model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, a method for training a large language model to identify high-risk web services is provided, comprising: performing preliminary training on the large language model based on first training data, wherein the first training data includes first sample content information extracted from a first sample web service, a first determination rule for determining whether the first sample web service is a high-risk web service based on the first sample content information, and an attribute label of whether the first sample web service is high-risk or non-high-risk; calling the preliminary trained large language model to identify a second sample web service to obtain a preliminary recognition result; determining an erroneous recognition result in the preliminary recognition result; and performing secondary training on the preliminary trained large language model based on a preset dialogue context, wherein the preset dialogue context includes second sample content information extracted from the second sample web service corresponding to the erroneous recognition result, and a second determination rule for determining whether the second sample web service is a high-risk web service based on the second sample content information.

[0007] According to another aspect of the present disclosure, a method for identifying high-risk web page services based on a large language model is provided, comprising: constructing a prompt for guiding the large language model to identify whether a web page service is a high-risk web page service, the prompt comprising content information extracted from the web page service, wherein the large language model is obtained according to the method for training a large language model to identify high-risk web page services as described above; providing the prompt to the large language model to obtain an identification result of the web page service.

[0008] According to another aspect of the present disclosure, an intelligent agent based on a large language model is provided, which identifies high-risk web page services according to the high-risk web page service identification method based on a large language model as described above.

[0009] According to another aspect of the present disclosure, there is provided an apparatus for training a large language model to identify high-risk web services, comprising: a preliminary training module configured to perform preliminary training on the large language model based on first training data, wherein the first training data comprises first sample content information extracted from a first sample web service, a first determination rule for determining whether the first sample web service is a high-risk web service based on the first sample content information, and an attribute tag indicating whether the first sample web service is high-risk or non-high-risk; a preliminary identification module configured to call the preliminary trained large language model to identify a second sample web service to obtain a preliminary identification result; a first error identification module configured to determine an error identification result in the preliminary identification result; and a secondary training module configured to perform secondary training on the preliminary trained large language model based on a preset dialogue context, wherein the preset dialogue context comprises second sample content information extracted from a second sample web service corresponding to the error identification result, and a second determination rule for determining whether the second sample web service is a high-risk web service based on the second sample content information.

[0010] According to another aspect of the present disclosure, a high-risk web page service identification device based on a large language model is provided, including: a prompt construction module, configured to construct a prompt for guiding the large language model to identify whether a web page service is a high-risk web page service, the prompt including content information extracted from the web page service, wherein the large language model is obtained according to an apparatus for training a large language model to identify high-risk web page services; and a fourth providing module, configured to provide the prompt to the large language model to obtain an identification result of the web page service.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for training a large language model to identify high-risk web page services and the method for identifying high-risk web page services based on a large language model as described above in the present disclosure.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to enable a computer to execute the method for training a large language model to identify high-risk web page services and the method for identifying high-risk web page services based on a large language model as described above in the present disclosure.

[0013] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method for training a large language model to identify high-risk web page services and the method for identifying high-risk web page services based on a large language model as described above.

[0014] According to one or more embodiments of the present disclosure, high-risk web page services can be effectively identified based on a large language model.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0018] Figure 2 A flowchart of a method for training a large language model to identify high-risk web services according to an embodiment of the present disclosure is shown;

[0019] Figure 3 A schematic diagram showing a portion of a first training prompt provided to a large language model and associated with a first decision rule according to an embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram showing a decision rule for training a large language model to identify high-risk web page services according to an embodiment of the present disclosure is shown;

[0021] Figure 5 A schematic diagram of a high-risk web page service identification method based on a large language model according to an embodiment of the present disclosure is shown;

[0022] Figure 6 A structural block diagram of an apparatus for training a large language model to identify high-risk web page services according to an embodiment of the present disclosure is shown;

[0023] Figure 7 A structural block diagram of an apparatus for training a large language model to identify high-risk web page services according to another embodiment of the present disclosure is shown;

[0024] Figure 8 A structural block diagram of a high-risk web page service identification device based on a large language model according to an embodiment of the present disclosure is shown;

[0025] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0028] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0029] In related technologies, web services such as the management backend and internal systems of enterprises may bring security risks due to being open to the outside world, and such web services are usually considered high-risk web services. Traditional high-risk web service identification technology collects open source fingerprint libraries and internal product fingerprints for identification. This method requires a lot of manpower to write and maintain identification rules. However, since the identification rules are difficult to cover comprehensively, false positives and missed positives often occur, resulting in poor results.

[0030] To this end, the embodiments of the present disclosure provide an effective high-risk web page service identification technology based on a large language model.

[0031] Before describing the method of the present invention in detail, Figure 1 Exemplary systems are described in which the methods described herein may be implemented.

[0032] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0034] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the method for training a large language model to identify high-risk web page services and the method for identifying high-risk web page services based on a large language model as described in an embodiment of the present disclosure.

[0035] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0036] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0037] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to input web page information that needs to be identified as high-risk. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0038] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0039] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0040] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0041] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0042] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0043] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0044] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0045] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0046] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0047] Various aspects of the method for training a large language model to identify high-risk web services according to an embodiment of the present disclosure will be described in detail below.

[0048] Figure 2 A flowchart of a method 200 for training a large language model to identify high-risk web services according to an embodiment of the present disclosure is shown.

[0049] like Figure 2 As shown, method 200 includes steps S201, S202, S203 and S204.

[0050] In step S201, a large language model is preliminarily trained based on first training data, where the first training data includes first sample content information extracted from a first sample web page service, a first determination rule for determining whether the first sample web page service is a high-risk web page service based on the first sample content information, and an attribute label indicating whether the first sample web page service is high-risk or non-high-risk.

[0051] In the example, in order to enable the large language model to intuitively learn recognition methods from training data in preliminary training, a certain number (such as hundreds) of representative training data can be selected for preliminary training to construct the first training data, for example, common Web application frameworks (Grafana, ElasticSearch, etc.), internal high-risk systems, conventional enterprise external services, and general management backend pages, etc.

[0052] In an example, the content information extracted from the web page service may be information directly related to the content of the web page service. For example, for a Hypertext Markup Language (HTML) document, the extracted content information may include a title tag and a body tag in the document. Accordingly, the first sample content information extracted from the first sample web page service may be directly related to the content of the first sample web page service.

[0053] In the example, the determination rule can be used to describe what kind of content information can be used to determine whether a webpage service is high-risk or not. The determination rule can be formulated based on the knowledge of security experts, historical data analysis, known high-risk service characteristics, etc. Accordingly, the first determination rule can be used to determine whether the first sample webpage service is high-risk based on the first sample content information.

[0054] In the example, the attribute tag can identify each first sample web page service as high risk or non-high risk, thereby providing the correct answer to the learning goal for the preliminary training of the large language model. That is, in the preliminary training stage, the large language model needs to learn how to identify whether the first sample web page service is high risk based on the input first sample content information, the first judgment rule and the attribute tag as the answer.

[0055] In step S202, the preliminarily trained large language model is called to identify the second sample web page service to obtain a preliminary recognition result.

[0056] In step S203, the erroneous recognition results in the preliminary recognition results are determined.

[0057] In the example, the second sample web page service can be used to test the recognition ability of the large language model after preliminary training, which can be the same as or different from the first sample web page service. Therefore, steps S202 and S203 can be a phased test of the large language model after preliminary training, in which those web page services that it cannot correctly recognize can be obtained, so as to be improved in subsequent training. Accordingly, steps S202 and S203 determine the incorrect recognition results of the large language model after preliminary training for the second sample web page service.

[0058] In the example, the prediction results of the preliminarily trained large language model can be compared with the true label (high risk or non-high risk) of the second sample web page service. The true label can come from manual review, a known security database, or other reliable security assessment results. Therefore, it can be determined which second sample web page services are misidentified. Misidentification situations can include mistakenly identifying a high-risk web page service as a non-high-risk web page service (or vice versa), or failing to give an answer as to whether the web page service is high-risk.

[0059] In step S204, the initially trained large language model is retrained based on a preset dialogue context, where the preset dialogue context includes second sample content information extracted from a second sample web page service corresponding to the error recognition result, and a second determination rule for determining whether the second sample web page service is a high-risk web page service based on the second sample content information.

[0060] In the example, the secondary training may involve in-context learning using a preset dialogue context, so the preset dialogue context may serve as a self-instruction in context learning. The process of the secondary training may be similar to the process of "checking for deficiencies and filling gaps" in the error recognition results of the initially trained large language model, in which a number of samples may be provided in the preset dialogue context to enable it to perform small sample learning. That is, in the secondary training process, unlike the initial training process, a specific training data set (i.e., the first training data) may not be provided, but a dialogue may be conducted with the initially trained large language model through the preset dialogue context to enable it to achieve further learning in the dialogue. Such a secondary training process can make full use of the semantic understanding ability of the large language model to achieve the desired training effect.

[0061] In the example, similar to the first sample content information, the second sample content information may also refer to information directly related to the content of the second sample web page service. However, it should be noted that since secondary training is performed at this time, the second sample content information extracted from the second sample web page service corresponding to the error recognition result needs to be provided to achieve the above-mentioned "checking for missing and filling in gaps".

[0062] In addition, similar to the first determination rule, the second determination rule can be used to determine whether the second sample web page service is high-risk based on the second sample content information. In the case of mistakenly identifying a high-risk web page service as non-high-risk (or mistakenly identifying a non-high-risk web page service as high-risk) as described above, it means that the large language model that has been preliminarily trained may not fully grasp how to use the first determination rule for identification, so it can be further learned by emphasizing the corresponding determination rule again in the preset dialogue context. Accordingly, the second determination rule provided at this time can be the same as the first determination rule or be appropriately rewritten to emphasize the first determination rule. In the case of failing to give an answer to whether the web page service is high-risk as described above, it means that the corresponding determination rule may not be provided to the large language model in the preliminary training, resulting in it not being clear how to identify it. Accordingly, the second determination rule provided at this time can be a new determination rule different from the first determination rule.

[0063] Therefore, in the method 200 for training a large language model to identify high-risk web services according to an embodiment of the present disclosure, a training strategy based on two different training stages is adopted, wherein firstly, in the preliminary training, a specific training data set is used to enable the large language model to learn the basic identification method, and then in the secondary training, the large language model that has undergone preliminary training is used to "check for deficiencies and fill in the gaps" for errors to achieve further learning through context learning, thereby prompting the large language model to accurately identify whether a web service is high-risk. On this basis, the method 200 can reduce labor costs without relying on a large amount of manual work, and can also provide applicability to unknown web services / frameworks simply by adding or modifying judgment rules.

[0064] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0065] In some embodiments, such as in combination Figure 2 The step S201 shown in which the large language model is preliminarily trained based on the first training data may include: constructing a first training prompt based on the first sample content information, the first determination rule and the attribute tag; and providing the first training prompt to the large language model.

[0066] In the example, the first training prompt may be an integration of the first sample content information, the first judgment rule, and the attribute tag. In addition, the first training prompt may also include other statements for interacting with the large language model, such as first instruction information for indicating what operation the large language model needs to complete, second instruction information for indicating how the large language model completes the operation, and third instruction information for indicating how the large language model outputs the operation result. In preliminary training, a specific training data set (i.e., first training data) may be provided to the large language model in the form of the first training prompt, so as to utilize the semantic understanding ability of the large language model to enable it to learn the content provided in the first training prompt.

[0067] In this way, the interaction mechanism and underlying language capabilities of the large language model can be fully utilized to feed the training data set used for preliminary training to the large language model, which is conducive to the large language model learning basic recognition methods during the preliminary training process.

[0068] Figure 3 FIG. 1 is a schematic diagram showing a portion of a first training prompt 300 provided to a large language model and associated with a first decision rule 310 according to an embodiment of the present disclosure.

[0069] like Figure 3As shown in FIG. 1 , a portion of the first training prompt 300 provided to the large language model and associated with the first determination rule 310 is shown. In addition, the first indication information 301, the second indication information 302, and the third indication information 303 as described above in the first training prompt 300 are also shown. It can be understood that the first training prompt 300 can be designed in various ways according to actual applications and is not limited to the first training prompt 300. Figure 3 Example shown.

[0070] In such Figure 3 In the first training prompt 300 shown, the first judgment rule 310 provides specific judgment basis, defines what is a high-risk service, and gives specific examples, such as "for management backend login, control panel, database panel and other pages, they should be judged as high-risk services", and defines what is a non-high-risk service, and gives specific examples, such as "for regular website services, ordinary user login and other pages, they are judged as non-high-risk services."

[0071] In addition, the first instruction information 301 indicates what operation the large language model needs to complete, such as "Now there is a piece of data extracted from the homepage of the website. Please judge whether the website is a high-risk service based on this data, and give the corresponding reason for the judgment." The second instruction information 302 indicates how the large language model should complete the operation, such as "Fully consider each field in the data and find keywords that may symbolize risks." The third instruction information 303 indicates how the large language model should output the operation results, such as specifying the format of the output results, which can include the results and reasons of high-risk or non-high-risk judgments.

[0072] In some embodiments, such as in combination Figure 2 The step S202 shown calls the preliminarily trained large language model to identify the second sample web page service to obtain a preliminary recognition result, which may include: constructing a recognition prompt for guiding the preliminarily trained large language model to identify the second sample web page service based on the second sample content information extracted from the second sample web page service and the first judgment rule; and providing the recognition prompt to the preliminarily trained large language model.

[0073] In the example, as mentioned above, step S202 involves testing the preliminarily trained large language model after the preliminarily training, so the recognition prompt is specifically constructed for the test process, which may include the second sample content information extracted from the second sample web page service and the first determination rule. That is, at this time, it is necessary to determine through testing to what extent the preliminarily trained large language model has mastered the recognition method, so it is still necessary to provide it with the first determination rule, but the second sample content information may be the same as or different from the first sample content information.

[0074] Through this method, the recognition prompts for testing the initially trained large language model can be constructed with the help of the first judgment rule that has been learned and the second sample web page service that has been learned or not learned, thereby determining the recognition ability of the large language model after the initial training, which is conducive to further targeted learning in the secondary training.

[0075] In some embodiments, such as in combination Figure 2 The step S203 shown in FIG. 20 shows determining erroneous recognition results in the preliminary recognition results, which may include: determining whether the proportion of erroneous recognition results in the preliminary recognition results is greater than or equal to a predetermined threshold; and in response to determining that the proportion is greater than or equal to the predetermined threshold, updating the first training data to continue preliminary training of the large language model that has been preliminarily trained.

[0076] In the example, after the preliminarily trained large language model completes the preliminary identification of the second sample web page service, a series of identification results can be obtained. At this time, these preliminary identification results can be identified manually or with the help of evaluation tools to determine the wrong identification results. For example, the situation where the preliminarily trained large language model identifies a high-risk service but is actually a non-high-risk service, and the situation where it is identified as a non-high-risk service but is actually a high-risk service can be regarded as an erroneous identification result. Further, the proportion of the number of these erroneous identification results to the total number of preliminary identification results can be calculated. Assuming that there are a total of 100 preliminary identification results, and it is determined that there are 15 erroneous identification results after inspection, then the proportion of erroneous identification results is 0.15 (or 15%). On the other hand, a threshold value for judging whether the preliminarily trained large language model needs to continue to return to the preliminary training can be pre-set, and the threshold value can be determined according to actual needs and experience, such as being set to 0.3 (or 30%). The calculated proportion of erroneous recognition results is compared with a predetermined threshold. If the proportion of erroneous recognition results is greater than or equal to the predetermined threshold, it may indicate that the performance of the initially trained large language model in the current preliminary training stage is not ideal, and it is necessary to further return to the preliminary training stage for optimization. If it is less than the predetermined threshold, it can be considered that the initial training effect of the initially trained large language model has met expectations, and the next step of secondary training can be entered.

[0077] In the example, when it is determined that the proportion of erroneous recognition results is greater than or equal to a predetermined threshold, the first training data used for preliminary training needs to be updated to continue the preliminary training of the large language model that has been preliminarily trained. There are many ways to update, for example, the original first training data corresponding to the erroneous recognition results can be re-labeled to correct its incorrect attribute label (high risk or non-high risk); more typical data related to the erroneous recognition situation can be supplemented to enrich the training data to help the preliminarily trained large language model better distinguish the characteristics of different types of web services; the first training data can also be further cleaned or preprocessed to remove noise data or outliers that may affect the judgment of the preliminarily trained large language model. After completing the update of the first training data, the updated first training data can be used to continue the preliminary training of the preliminarily trained large language model. The subsequent process can repeat the previous steps S201 to S203 until the preliminary recognition result reaches a relatively ideal state, such as the proportion of erroneous recognition results is reduced to within a predetermined threshold.

[0078] In this way, the initial training effect of the large language model can be accurately evaluated, thereby providing a clear judgment criterion for whether to conduct preliminary training again.

[0079] In some embodiments, such as in combination Figure 2 The step S204 shown in FIG. 2 may include performing secondary training on the initially trained large language model based on a preset dialogue context: constructing a second training prompt based on second sample content information extracted from a second sample web page service corresponding to the error recognition result and a second judgment rule; and providing the second training prompt to the initially trained large language model.

[0080] In the example, after determining the error recognition results in the preliminary recognition results, the second sample web page services corresponding to these errors can be analyzed to determine the specific potential reasons for the error recognition of the large language model after preliminary training, and then these error recognitions can be focused on learning in the secondary training process. First, the second sample content information can be extracted from these second sample web page services. For example, for a non-high-risk ordinary product introduction page, its second sample content information may be mistakenly identified as a high-risk service due to the inclusion of specific image display methods, copyright statements at the bottom of the page, and other contents; and for a high-risk management backend page with security vulnerabilities, its second sample content information may be mistakenly identified as a non-high-risk service due to the hidden features of the login entrance, the specific database connection string format, and the authority management-related code segments in the page. In this case, based on the features of the second sample content information extracted from the second sample web page service corresponding to the error recognition result, the first judgment rule can be updated by operations such as adding, modifying, and deleting to obtain the second judgment rule to construct the second training prompt. For example, "Web services that contain specific hidden login features (such as xxx features) and have specific database connection string formats (such as xxx formats) should be identified as high-risk services because of xxx." Such prompts can guide the large language model to focus on key sample features and make correct judgments and learning based on the rules, so that more accurate judgments can be made when identifying web services with similar features in the future.

[0081] In the example, further, for the large language model after the second training, more accurate identification of high-risk web services can be achieved through model fine-tuning. For example, the amount of data for model fine-tuning can be set at around 1k. During the fine-tuning process, by learning from these 1k data, the fine-tuned model has significantly improved the accuracy of the answer compared to the original model, while ensuring that the output format strictly conforms to the established style. After the model fine-tuning is completed, for subsequent new data entry, the training method of reconstructing new training prompts similar to the second training can no longer be used, but the current fine-tuned model can be directly used for annotation. The annotated data is manually screened, and the screened high-quality data is used again for repeated training of the model. Through such a reinforcement learning process, a large language model that can more effectively identify high-risk web services is eventually trained.

[0082] In this way, the incorrect judgments of the large language model in the initial training can be corrected, and its ability to identify high-risk web services can be further optimized, thereby gradually improving the accuracy and reliability of identifying high-risk web services and reducing the occurrence of misjudgments and missed judgments.

[0083] In some embodiments, the first sample content information and the second sample content information may both include a title tag part and a body tag part in a Hypertext Markup Language (HTML) document.

[0084] In the example, the Hypertext Markup Language document is the basis of the web page. It consists of a series of tags and text content to describe the structure and presentation of the web page. Among them, the title tag part (usually <title> Tags) and the main tag part (< / title> Tags) are a key component of HTML documents. The title tag is generally located in the header area of ​​an HTML document and is used to define the title of the web page. This title is displayed in the browser's title bar, bookmark list, and search engine results page, and it concisely summarizes the subject content of the web page. Through this title, users and search engines can quickly understand the general subject direction of the web page. The body tag contains all visible content on the web page, such as text, pictures, links, tables, forms and other elements. It is the core display area of ​​the web page, directly presented to users for browsing and interaction. For example, in the body tag of a news web page, there will be elements such as news titles, text content, related pictures, comment links, etc. These elements together constitute the web page content that users actually see and operate.

[0085] In the example, since the original HTML document may contain many tags and useless data, if it is directly used as sample content information, it will bring difficulties to the processing of the large language model, and the judgment result may be affected by data noise. Therefore, a specific data extraction method can be used, such as removing <script>和<style>等标签而仅保留标题标签部分和经过处理的主体标签部分作为第一样本内容信息和第二样本内容信息，使数据更符合大语言模型的输入要求。例如，通过去除过多连续空格能使文本更规整，便于大语言模型分析；也可以截取文本长度在7000字符以内，以确保在对话上下文限制下大语言模型能有效处理数据。根据需要，也可以使用OCR(Optical Character Recognition，光学字符识别)方式识别网页的图片中的文字。

[0086] 因此，这些处理方式增强了样本内容信息的可用性，让大语言模型能更好地理解和学习网页的特征，无论是在初步训练时依据第一样本内容信息建立基础识别能力，还是在二次训练中利用第二样本内容信息纠正错误和优化判断，都能基于高质量的数据进行，从而提升整体高危网页服务识别的效果，更好地应对复杂多样的网页服务场景。

[0087] 在一些实施例中，第一判定规则和第二判定规则均可以包括用于定义高危网页服务和非高危网页服务的含义的第一描述，以及用于列举高危网页服务和非高危网页服务的示例的第二描述。

[0088] 在示例中，第一判定规则和第二判定规则中的用于定义高危网页服务和非高危网页服务的含义的第一描述，可以是基于对网络安全风险的深入理解和实践经验总结得到的。它指出高危网页服务可以是有可能对公司信息系统造成严重危害的服务类型，例如，涉及系统核心管理功能的网页，如管理员登录页面、系统配置页面、数据库管理页面等。这些页面通常具有较高的权限，若被非法访问，可能导致系统数据被泄露、被篡改或系统被完全控制；也可以是不应开放外网访问的开源系统 / 框架，如Kibana、ElasticSearch、Grafana、Nacos等，若意外开放外网访问，可能会带来潜在的安全风险等等。同时，非高危网页服务可以定义为正常对外开放、提供各种合法功能且不会对公司信息系统构成直接安全威胁的服务。这些页面主要用于向用户展示信息、提供一般性服务或收集用户基本信息，其本身的功能和权限设置相对较低，不存在直接导致公司信息系统遭受严重危害的风险。这一含义定义可以为后续判断网页服务是否高危提供明确的概念基础，使判断过程有统一的标准和依据。

[0089] 在示例中，在第一判定规则和第二判定规则的第二描述中列举的高危网页服务示例可以进一步具体化高危的概念。除了前面提到的管理员登录页面、系统配置页面等常见高危类型外，还可能包括一些特定行业或企业内部特有的高危网页服务示例。例如，在金融行业，网上银行的资金转账确认页面、客户信用评估系统的核心数据查看页面等因为涉及到资金交易和敏感客户信息处理都是典型的高危网页服务。类似地，非高危网页服务例如企业官网的公司简介页面、用户留言反馈页面等，这些页面展示了正常对外开放且功能相对简单安全的网页服务。以上示例可以为大语言模型提供直观的学习样本，使模型能够通过对这些具体示例的特征学习，更好地识别网页服务是否高危。

[0090] 通过这种方式，大语言模型可以学习到高危与非高危网页服务的共性和特征，从而在判断过程中能够更准确地区分高危和非高危网页服务。

[0091] 图4示出了根据本公开的实施例的用于训练大语言模型识别高危网页服务的判定规则400的示意图。

[0092] 如图4所示，示出了高危服务与非高危服务的判定规则400以及相关的第一描述401和第二描述402。可以理解的是，判定规则400可以根据实际应用进行各种设计，并不局限于如图4所示的示例。在如图4所示的判定规则400中，展示了用于定义高危网页服务和非高危网页服务的含义的第一描述401、用于列举高危网页服务和非高危网页服务的示例的第二描述402。如图4所示的第一描述401和第二描述402仅为提供给大语言模型用于高危服务判别的解释性和示例性判别依据，且高危网页服务和非高危网页服务的示例并不局限于图4所示。

[0093] 根据本公开的实施例，还提供了一种基于大语言模型的高危网页服务识别方法，包括：构造用于引导大语言模型对网页服务是否为高危网页服务进行识别的提示语，提示语包括从网页服务中提取的内容信息，大语言模型根据如上所述的用于训练大语言模型识别高危网页服务的方法得到；以及将提示语提供给大语言模型，以得到对网页服务的识别结果。

[0094] 在示例中，提示语的构造可以帮助大语言模型理解用户的需求或任务的具体要求，使得大语言模型能够理解并输出该网页服务是否属于高危服务的结果。将构造的提示语输入到经过如上方法训练的大语言模型中，可以获取其对网页服务的识别结果。

[0095] 在示例中，根据实际的应用场景，例如在公司场景的架构中，针对需要判别的外网资产端口服务，可以通过调用资产信息采集智能体，并借助httpx API(ApplicationProgramming Interface，应用程序接口)进行网络请求和网站内容爬取，从而能够获取网页中的报文、标题等关键信息。之后可以对关键信息进行数据清洗，再将清洗后的数据提供给后端的大语言模型，让大语言模型对爬取的数据进行深度分析和判别，进而识别潜在的高危网页服务。识别的结果可以通过安全高效的API接口被推送至事件运营中心，该API接口可以同时负责前端与后端的通信，即接收输入内容与返回结果，以便根据判定结果采取相应的行动，并同时将判定结果推送至记录数据库中存储。在如上所述的场景中，数据库可以包括用于存储扫描结果的记录数据库和用于存储优质数据的数据集数据库。进一步地，还可以从记录数据库中筛选出优质的数据录入到相应的数据集数据库，该数据集数据库中的数据可以用于对大语言模型进行微调，以此不断提升模型的准确率和效能，以确保响应的判别结果和模型优化有据可依。

[0096] 因此，利用大语言模型的强大文本理解能力，通过构造针对性的提示语并提供给经过如上方法训练的大语言模型，可以得到高危网页服务的识别结果。这种方法不仅可以基于大语言模型的自动操作而减少人力成本，而且还可以简单地通过增加或修改判定规则来提供对未知网页服务 / 框架的适用性。

[0097] 在一些实施例中，基于大语言模型的高危网页服务识别方法还可以包括：确定识别结果中的错误识别结果；纠正错误识别结果以得到正确识别结果；将正确识别结果作为对应的网页服务的属性标记；以及将从错误识别结果对应的网页服务中提取的内容信息以及属性标记作为用于训练大语言模型的训练数据。

[0098] 在示例中，大语言模型的错误识别可能由于多种原因产生，例如大语言模型在遇到未知网页服务时因为没有足够的训练数据而无法进行识别，或者输入的提示语未能准确反映网页的实际特征。可以通过人工核查或工具评估等方法判断大语言模型输出的识别结果是否存在错误，一旦错误识别结果被发现，对其进行纠正并将正确的识别结果作为对应的网页服务的属性标记。纠正后的错误识别结果能够为大语言模型的进一步优化提供新的训练数据，使得大语言模型通过学习纠正后的识别结果而在后续的识别任务中减少相似错误的发生。纠正后的识别结果和相关网页内容可以作为新的训练数据，继续输入到大语言模型中进行训练。通过不断地加入新的标注数据，使得大语言模型能够逐步提升其识别能力，尤其是在面对未知网页服务时，能够准确识别其是否高危。

[0099] 因此，通过错误识别纠正与训练数据更新的机制能够不断优化大语言模型，不仅有效提升了大语言模型的识别准确性，也增强了其对未知网页服务的适应性。

[0100] 图5示出了根据本公开的实施例的基于大语言模型的高危网页服务识别方法的示意图。

[0101] 如图5所示，可以将从网页服务中提取的内容信息输入到网页服务识别大语言模型501中，经过大语言模型501的判别输出识别结果，并将结果存储在判定结果库502中。人工503可以对判定结果库502中的结果定期进行复核，并将错误识别结果对应的网页服务属性进行标记，以作为训练数据加入到训练数据集504。同时新的训练数据可以反馈给网页服务识别大语言模型501，从而有利于不断完善提升大语言模型501的识别准确率。

[0102] 根据本公开的实施例，还提供了一种基于大语言模型的智能体，该智能体根据如上所述的基于大语言模型的高危网页服务识别方法对高危网页服务进行识别。

[0103] 在示例中，智能体是一种基于人工智能技术的系统，能够根据输入的信息做出智能决策。可以在智能体的框架下实现如前所述的基于大语言模型的高危网站服务识别方法，从而自动对网页服务进行分析以识别出网页服务是否高危。

[0104] 因此，利用上述基于大语言模型的智能体可以有效识别网页服务是否高危，其自动化的实现方式可以大幅减少人力成本，提高识别效率。

[0105] 根据本公开的实施例，还提供了一种用于训练大语言模型识别高危网页服务的装置。

[0106] 图6示出了根据本公开一个实施例的用于训练大语言模型识别高危网页服务的装置600的结构框图。

[0107] 如图6所示，装置600包括初步训练模块601、初步识别模块602、第一错误识别模块603和二次训练模块604。

[0108] 初步训练模块601被配置为基于第一训练数据对大语言模型进行初步训练，第一训练数据包括从第一样本网页服务中提取的第一样本内容信息、用于基于第一样本内容信息判断第一样本网页服务是否为高危网页服务的第一判定规则、以及第一样本网页服务属于高危或非高危的属性标记。

[0109] 初步识别模块602被配置为调用经初步训练的大语言模型对第二样本网页服务进行识别，以得到初步识别结果。

[0110] 第一错误识别模块603被配置为确定初步识别结果中的错误识别结果。

[0111] 二次训练模块604被配置为基于预设的对话上下文对经初步训练的大语言模型进行二次训练，预设的对话上下文包括从错误识别结果对应的第二样本网页服务中提取的第二样本内容信息、以及用于基于第二样本内容信息判断第二样本网页服务是否为高危网页服务的第二判定规则。

[0112] 上述初步训练模块601、初步识别模块602、第一错误识别模块603和二次训练模块604的操作可以分别对应于如图2所示的步骤S201、S202、S203和S204的操作。因此，这里不再赘述其各个方面的细节。

[0113] 图7示出了根据本公开另一个实施例的用于训练大语言模型识别高危网页服务的装置700的结构框图。

[0114] 如图7所示，装置700可以包括初步训练模块701、初步识别模块702、第一错误识别模块703和二次训练模块704。上述模块的操作可以与如图6所示的初步训练模块601、初步识别模块602、第一错误识别模块603和二次训练模块604的操作相同。此外，上述模块还可以包括进一步的子模块。

[0115] 在一些实施例中，初步训练模块701可以包括第一训练提示语构造模块7011和第一提供模块7012。第一训练提示语构造模块7011可以被配置为基于第一样本内容信息、第一判定规则以及属性标记，构造第一训练提示语。第一提供模块7012可以被配置为将第一训练提示语提供给大语言模型。

[0116] 在一些实施例中，初步识别模块702可以包括识别提示语构造模块7021和第二提供模块7022。识别提示语构造模块7021可以被配置为基于从第二样本网页服务中提取的第二样本内容信息以及第一判定规则，构造用于引导经初步训练的大语言模型对第二样本网页服务进行识别的识别提示语。第二提供模块7022可以被配置为将识别提示语提供给经初步训练的大语言模型。

[0117] 在一些实施例中，第一错误识别模块703可以包括比例确定模块7031和训练数据更新模块7032。比例确定模块7031可以被配置为确定错误识别结果占初步识别结果的比例是否大于或等于预定阈值。训练数据更新模块7032可以被配置为响应于确定比例大于或等于预定阈值，更新第一训练数据以继续对大语言模型进行初步训练。

[0118] 在一些实施例中，二次训练模块704可以包括第二训练提示语构造模块7041和第三提供模块7042。第二训练提示语构造模块7041可以被配置为基于从错误识别结果对应的第二样本网页服务中提取的第二样本内容信息以及第二判定规则，构造第二训练提示语。第三提供模块7042可以被配置为将第二训练提示语提供给经初步训练的大语言模型。

[0119] 在一些实施例中，第一样本内容信息和第二样本内容信息均可以包括超文本标记语言文档中的标题标签部分和主体标签部分。

[0120] 在一些实施例中，第一判定规则和第二判定规则均可以包括用于定义高危网页服务和非高危网页服务的含义的第一描述，以及用于列举高危网页服务和非高危网页服务的示例的第二描述。

[0121] 根据本公开的实施例，还提供了一种基于大语言模型的高危网页服务识别装置。

[0122] 图8示出了根据本公开一个实施例的基于大语言模型的高危网页服务识别装置800的结构框图。

[0123] 如图8所示，装置800包括提示语构造模块801、第四提供模块802、第二错误识别模块803、纠正模块804、标记模块805和训练数据提取模块806。

[0124] 提示语构造模块801被配置为构造用于引导大语言模型对网页服务是否为高危网页服务进行识别的提示语，提示语包括从网页服务中提取的内容信息，大语言模型根据如上所述的用于训练大语言模型识别高危网页服务的装置得到。

[0125] 第四提供模块802被配置为将提示语提供给大语言模型，以得到对网页服务的识别结果。

[0126] 第二错误识别模块803被配置为确定识别结果中的错误识别结果。

[0127] 纠正模块804被配置为纠正错误识别结果以得到正确识别结果。

[0128] 标记模块805被配置为将正确识别结果作为对应的网页服务的属性标记。

[0129] 训练数据提取模块806被配置为将从错误识别结果对应的网页服务中提取的内容信息以及属性标记作为用于训练大语言模型的训练数据。

[0130] 根据本公开的实施例，还提供了一种电子设备，包括至少一个处理器；以及与至少一个处理器通信连接的存储器，其中，存储器存储有能够被至少一个处理器执行的指令，指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如上所述的方法。

[0131] 根据本公开的实施例，还提供了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，计算机指令用于使计算机执行如上所述的方法。

[0132] 根据本公开的实施例，还提供了一种计算机程序产品，包括计算机程序，其中，计算机程序在被处理器执行时实现如上所述的方法。

[0133] 参考图9，现将描述可以作为本公开的服务器或客户端的电子设备900的结构框图，其是可以应用于本公开的各方面的硬件设备的示例。电子设备旨在表示各种形式的数字电子的计算机设备，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和 / 或者要求的本公开的实现。

[0134] 如图9所示，电子设备900包括计算单元901，其可以根据存储在只读存储器(ROM)902中的计算机程序或者从存储单元908加载到随机访问存储器(RAM)903中的计算机程序，来执行各种适当的动作和处理。在RAM 903中，还可存储电子设备900操作所需的各种程序和数据。计算单元901、ROM 902以及RAM 903通过总线904彼此相连。输入 / 输出(I / O)接口905也连接至总线904。

[0135] 电子设备900中的多个部件连接至I / O接口905，包括：输入单元906、输出单元907、存储单元908以及通信单元909。输入单元906可以是能向电子设备900输入信息的任何类型的设备，输入单元906可以接收输入的数字或字符信息，以及产生与电子设备的用户设置和 / 或功能控制有关的键信号输入，并且可以包括但不限于鼠标、键盘、触摸屏、轨迹板、轨迹球、操作杆、麦克风和 / 或遥控器。输出单元907可以是能呈现信息的任何类型的设备，并且可以包括但不限于显示器、扬声器、视频 / 音频输出终端、振动器和 / 或打印机。存储单元908可以包括但不限于磁盘、光盘。通信单元909允许电子设备900通过诸如因特网的计算机网络和 / 或各种电信网络与其他设备交换信息 / 数据，并且可以包括但不限于调制解调器、网卡、红外通信设备、无线通信收发机和 / 或芯片组，例如蓝牙设备、802.11设备、WiFi设备、WiMax设备、蜂窝通信设备和 / 或类似物。

[0136] 计算单元901可以是各种具有处理和计算能力的通用和 / 或专用处理组件。计算单元901的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的计算单元、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。计算单元901执行上文所描述的各个方法和处理。例如，在一些实施例中，该方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元908。在一些实施例中，计算机程序的部分或者全部可以经由ROM 902和 / 或通信单元909而被载入和 / 或安装到电子设备900上。当计算机程序加载到RAM 903并由计算单元901执行时，可以执行上文描述的方法的一个或多个步骤。备选地，在其他实施例中，计算单元901可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行上文描述的方法。

[0137] 本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、芯片上系统的系统(SOC)、复杂可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和 / 或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和 / 或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。

[0138] 用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和 / 或框图中所规定的功能 / 操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。

[0139] 在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。

[0140] 为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。

[0141] 可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)、互联网和区块链网络。

[0142] 计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，也可以为分布式系统的服务器，或者是结合了区块链的服务器。

[0143] 应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本公开中记载的各步骤可以并行地执行、也可以顺序地或以不同的次序执行，只要能够实现本公开公开的技术方案所期望的结果，本文在此不进行限制。

[0144] 虽然已经参照附图描述了本公开的实施例或示例，但应理解，上述的方法、系统和设备仅仅是示例性的实施例或示例，本发明的范围并不由这些实施例或示例限制，而是仅由授权后的权利要求书及其等同范围来限定。实施例或示例中的各种要素可以被省略或者可由其等同要素替代。此外，可以通过不同于本公开中描述的次序来执行各步骤。进一步地，可以以各种方式组合实施例或示例中的各种要素。重要的是随着技术的演进，在此描述的很多要素可以由本公开之后出现的等同要素进行替换。< / script>

Claims

1. A method for training a large language model to identify high-risk web services, comprising: Preliminary training of the large language model is performed based on first training data, wherein the first training data includes first sample content information extracted from a first sample web page service, a first determination rule for determining whether the first sample web page service is a high-risk web page service based on the first sample content information, and an attribute label indicating whether the first sample web page service is high-risk or non-high-risk; Calling the preliminarily trained large language model to identify the second sample web page service to obtain a preliminary identification result; determining an erroneous recognition result in the preliminary recognition result; and The initially trained large language model is retrained based on a preset dialogue context, wherein the preset dialogue context includes second sample content information extracted from the second sample web page service corresponding to the error recognition result, and a second determination rule for determining whether the second sample web page service is a high-risk web page service based on the second sample content information.

2. The method according to claim 1, wherein: The preliminary training of the large language model based on the first training data includes: constructing a first training prompt based on the first sample content information, the first determination rule, and the attribute tag; and The first training cue is provided to the large language model.

3. The method according to claim 1 or 2, wherein: The calling of the preliminarily trained large language model to identify the second sample web page service to obtain a preliminary identification result includes: Based on the second sample content information extracted from the second sample web page service and the first determination rule, constructing an identification prompt for guiding the preliminarily trained large language model to identify the second sample web page service; and The recognition prompt is provided to the initially trained large language model.

4. The method according to any one of claims 1 to 3, wherein: The determining of an erroneous recognition result in the preliminary recognition result comprises: determining whether the proportion of the erroneous recognition results to the preliminary recognition results is greater than or equal to a predetermined threshold; and In response to determining that the ratio is greater than or equal to the predetermined threshold, updating the first training data to continue the preliminary training of the preliminary trained large language model.

5. The method according to any one of claims 1 to 4, wherein: The second training of the preliminarily trained large language model based on the preset dialogue context includes: constructing a second training prompt based on the second sample content information extracted from the second sample web page service corresponding to the error recognition result and the second determination rule; and The second training prompt is provided to the initially trained large language model.

6. The method according to any one of claims 1 to 5, wherein: The first sample content information and the second sample content information both include a title tag part and a body tag part in a hypertext markup language document.

7. The method according to any one of claims 1 to 6, wherein: The first determination rule and the second determination rule both include a first description for defining the meaning of a high-risk web page service and a non-high-risk web page service, and a second description for listing examples of the high-risk web page service and the non-high-risk web page service.

8. A method for identifying high-risk web services based on a large language model, comprising: Constructing a prompt for guiding a large language model to identify whether a webpage service is a high-risk webpage service, the prompt including content information extracted from the webpage service, wherein the large language model is trained according to any one of the methods of claims 1 to 7; and The prompt is provided to the large language model to obtain a recognition result of the web page service.

9. The method according to claim 8, wherein: The method further comprises: determining an erroneous recognition result among the recognition results; Correcting the erroneous recognition result to obtain a correct recognition result; Using the correct recognition result as the attribute mark of the corresponding webpage service; and The content information and the attribute tags extracted from the web page service corresponding to the error recognition result are used as training data for training the large language model.

10. An agent based on a large language model, wherein: The intelligent agent identifies high-risk web services according to the method described in claim 8 or 9.

11. A device for training a large language model to identify high-risk web services, comprising: a preliminary training module configured to perform preliminary training on the large language model based on first training data, wherein the first training data includes first sample content information extracted from a first sample web page service, a first determination rule for determining whether the first sample web page service is a high-risk web page service based on the first sample content information, and an attribute label indicating whether the first sample web page service is high-risk or non-high-risk; A preliminary recognition module is configured to call the preliminarily trained large language model to recognize the second sample web page service to obtain a preliminary recognition result; A first error recognition module is configured to determine an error recognition result in the preliminary recognition result; and The secondary training module is configured to perform secondary training on the initially trained large language model based on a preset dialogue context, wherein the preset dialogue context includes second sample content information extracted from the second sample web page service corresponding to the error recognition result, and a second determination rule for determining whether the second sample web page service is a high-risk web page service based on the second sample content information.

12. The device according to claim 11, wherein The initial training module includes: A first training prompt constructing module, configured to construct a first training prompt based on the first sample content information, the first determination rule and the attribute tag; and A first providing module is configured to provide the first training prompt to the large language model.

13. The device according to claim 11 or 12, wherein: The preliminary identification module comprises: an identification prompt construction module, configured to construct an identification prompt for guiding the preliminarily trained large language model to identify the second sample web page service based on the second sample content information extracted from the second sample web page service and the first determination rule; and The second providing module is configured to provide the recognition prompt to the preliminarily trained large language model.

14. The device according to any one of claims 11 to 13, wherein: The first error recognition module includes: a ratio determination module, configured to determine whether a ratio of the erroneous recognition results to the preliminary recognition results is greater than or equal to a predetermined threshold; and The training data updating module is configured to update the first training data in response to determining that the ratio is greater than or equal to the predetermined threshold so as to continue the preliminary training of the preliminary trained large language model.

15. The device according to any one of claims 11 to 14, wherein: The secondary training module comprises: A second training prompt constructing module configured to construct a second training prompt based on the second sample content information extracted from the second sample web page service corresponding to the error recognition result and the second determination rule; and The third providing module is configured to provide the second training prompt to the preliminarily trained large language model.

16. The device according to any one of claims 11 to 15, wherein: The first sample content information and the second sample content information both include a title tag part and a body tag part in a hypertext markup language document.

17. The device according to any one of claims 11 to 16, wherein: The first determination rule and the second determination rule both include a first description for defining the meaning of a high-risk web page service and a non-high-risk web page service, and a second description for listing examples of the high-risk web page service and the non-high-risk web page service.

18. A high-risk webpage service identification device based on a large language model, comprising: a prompt construction module, configured to construct a prompt for guiding a large language model to identify whether a web page service is a high-risk web page service, the prompt comprising content information extracted from the web page service, wherein the large language model is trained according to the apparatus according to any one of claims 11 to 17; and The fourth providing module is configured to provide the prompt to the large language model to obtain a recognition result of the web page service.

19. The device according to claim 18, wherein: The device also includes: A second error recognition module is configured to determine an erroneous recognition result in the recognition result; A correction module, configured to correct the erroneous recognition result to obtain a correct recognition result; a marking module, configured to use the correct recognition result as an attribute mark of the corresponding webpage service; and The training data extraction module is configured to use the content information and the attribute tags extracted from the web page service corresponding to the error recognition result as training data for training the large language model.

20. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-10.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-10.

22. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

Images