Method and device for determining risk level
By using large language models for automated entity recognition and complaint frequency analysis, the problem of low efficiency and consistency in identifying high-risk entities in complaint texts in the insurance industry has been solved, achieving efficient and accurate risk level determination.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-03-10
AI Technical Summary
In the insurance industry, identifying high-risk entities by manually reading and annotating complaint texts is inefficient, inconsistent, and inaccurate.
A large language model is used for automated entity recognition to form a set of valuable entities. By associating entities with the same object and combining the frequency of complaints, a set of high-frequency entities is determined. Finally, the risk level is determined based on the complaint text.
It improves the efficiency and consistency of risk level determination, eliminates individual differences in subjective judgment, and significantly improves accuracy.
Smart Images
Figure CN121636712A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for determining risk levels. Background Technology
[0002] In the insurance industry, customer complaints are a crucial channel for reporting problems. Auditors need to identify high-risk entities in complaint documents and address them in real time to help companies achieve early warning and timely handling of risks.
[0003] Currently, the following steps are typically used to identify high-risk entities in complaint texts: First, after obtaining the customer's complaint text, auditors read through the text line by line and manually mark the entities in the text, such as the name of the person being complained about, the name of the platform being complained about, and the type of product being complained about; then, auditors make a subjective judgment on which entities in the complaint text are "high-risk entities".
[0004] However, on the one hand, relying on human labor to read through each complaint text and manually mark the entities in the complaint text takes a long time, resulting in low identification efficiency; on the other hand, different auditors have different subjective judgments on which entities are "high-risk entities", resulting in low consistency and accuracy in identification. Summary of the Invention
[0005] To address the aforementioned issues, this application provides a method and apparatus for determining risk levels, which can improve the efficiency, consistency, and accuracy of risk level determination.
[0006] The embodiments of this application disclose the following technical solutions:
[0007] Firstly, this application provides a method for determining a risk level, the method comprising:
[0008] Obtain a set of complaint data; the set of complaint data includes multiple complaint data, and each complaint data includes a complaint text;
[0009] By performing entity recognition on each of the complaint texts, a set of valuable entities is determined; the set of valuable entities includes multiple valuable entities.
[0010] Associate the valuable entities in the set of valuable entities that point to the same object to determine the set of associated entities;
[0011] From the set of associated entities, the set of high-frequency entities is determined by determining the frequency of complaints against each set of associated entities within a preset time period;
[0012] Based on the high-frequency entity set and the complaint text corresponding to the high-frequency entity set, the risk level of the high-frequency entity set is determined.
[0013] Optionally, the step of determining a set of valuable entities by performing entity recognition on each of the complaint texts includes:
[0014] Each complaint text is input into a large language model, and entity recognition is performed on each complaint text by the large language model to obtain a candidate entity set output by the large language model; the candidate entity set includes multiple candidate entities;
[0015] The candidate entity set is input into the large language model, and the large language model performs a value judgment on each candidate entity to obtain the set of valuable entities output by the large language model; the rule for the value judgment is: whether the candidate entity includes identification information that can point to a specific unique object.
[0016] Optionally, associating valuable entities in the set of valuable entities that point to the same object to determine the set of associated entities includes:
[0017] Within the set of valuable entities, subsets of valuable entities of the personnel class, subsets of valuable entities of the platform class, and subsets of valuable entities of the product / service class are identified.
[0018] For the valuable entity subset of the personnel class, if multiple valuable entities satisfy any one of the conditions in the first condition set, then the multiple valuable entities are associated to determine the personnel associated entity subset; the first condition set includes: personnel name is the same, personnel name and employee number are the same, and personnel name and department are the same.
[0019] For the valuable entity subset of the platform class, if multiple valuable entities satisfy the second condition, then the multiple valuable entities are associated to determine the associated entity subset of the platform class; the second condition is: the platform names are consistent, and the subordinate identifiers have an intersection or a subordinate relationship;
[0020] For a subset of valuable entities in the product / service category, if multiple valuable entities satisfy the third condition set, then these multiple valuable entities are associated to determine the associated entity subset of the product / service category; the third condition set is: the product name is consistent, and at least two of the product type, service type, and version are consistent.
[0021] The set of associated entities is determined based on the subset of personnel-related entities, the subset of platform-related entities, and the subset of product / service-related entities.
[0022] Optionally, determining the risk level of the high-frequency entity set based on the high-frequency entity set and the complaint text corresponding to the high-frequency entity set includes:
[0023] Input the high-frequency entity set and the complaint text corresponding to the high-frequency entity set into the large language model to obtain the risk level output by the large language model.
[0024] Optionally, the risk level is related to whether the complaint texts corresponding to the high-frequency entity set describe the same type of violation, whether the complaint texts corresponding to the high-frequency entity set involve cross-departmental violations, and whether the similarity between the complaint texts corresponding to the high-frequency entity set is higher than a similarity threshold.
[0025] Secondly, this application discloses a risk level determination device, the device comprising: a set acquisition module, an entity recognition module, an entity association module, a frequency determination module, and a level determination module;
[0026] The set acquisition module is used to acquire a set of complaint data; the set of complaint data includes multiple complaint data, and each complaint data includes a complaint text;
[0027] The entity recognition module is used to determine a set of valuable entities by performing entity recognition on each complaint text; the set of valuable entities includes multiple valuable entities.
[0028] The entity association module is used to associate valuable entities in the valuable entity set that point to the same object, and determine the associated entity set.
[0029] The frequency determination module is used to determine the high-frequency entity set from the associated entity set by determining the complaint frequency of each associated entity set within a preset time period;
[0030] The risk level determination module is used to determine the risk level of the high-frequency entity set based on the high-frequency entity set and the complaint text corresponding to the high-frequency entity set.
[0031] Optionally, the entity recognition module is specifically used for: inputting each complaint text into a large language model, performing entity recognition on each complaint text through the large language model, and obtaining a candidate entity set output by the large language model; the candidate entity set includes multiple candidate entities; inputting the candidate entity set into the large language model, performing value judgment on each candidate entity through the large language model, and obtaining a valuable entity set output by the large language model; the rule for value judgment is: whether the candidate entity includes identification information that can point to a specific unique object.
[0032] Optionally, the entity association module is specifically used to: determine a subset of valuable entities of the personnel type, a subset of valuable entities of the platform type, and a subset of valuable entities of the product / service type in the set of valuable entities; for the subset of valuable entities of the personnel type, if multiple valuable entities satisfy any one of the conditions in the first condition set, then associate the multiple valuable entities to determine the personnel-associated entity subset; the first condition set includes: identical personnel names, identical personnel names and identical employee IDs, identical personnel names and identical departments; for the subset of valuable entities of the platform type, if multiple valuable entities satisfy the second condition, then... The multiple valuable entities are associated to determine a subset of platform-type associated entities; the second condition is that the platform names are consistent, and the subordinate identifiers have an intersection or a subordinate relationship; for the subset of product / service-type valuable entities, if multiple valuable entities satisfy the third set of conditions, then the multiple valuable entities are associated to determine a subset of product / service-type associated entities; the third set of conditions is that the product names are consistent, and at least two of the product type, service type, and version are consistent; the associated entity set is determined based on the subset of personnel-type associated entities, the subset of platform-type associated entities, and the subset of product / service-type associated entities.
[0033] Optionally, the level determination module is specifically used to: input the high-frequency entity set and the complaint text corresponding to the high-frequency entity set into the large language model to obtain the risk level output by the large language model.
[0034] Optionally, the risk level is related to whether the complaint text corresponding to the high-frequency entity set describes the same type of violation, whether the complaint text corresponding to the high-frequency entity set involves cross-departmental violations, and whether the similarity between the complaint text corresponding to the high-frequency entity set is higher than a similarity threshold.
[0035] Compared with the prior art, this application has the following beneficial effects:
[0036] This application provides a method and apparatus for determining risk levels. The method includes: acquiring a complaint data set; the complaint data set includes multiple complaint data, and each complaint data includes a complaint text; determining a set of valuable entities by performing entity recognition on each complaint text; the set of valuable entities includes multiple valuable entities; associating valuable entities in the set of valuable entities that point to the same object to determine an associated entity set; determining a high-frequency entity set from the associated entity set by determining the complaint frequency of each associated entity set within a preset time period; and determining the risk level of the high-frequency entity set based on the high-frequency entity set and the complaint text corresponding to the high-frequency entity set. Therefore, on the one hand, the method for determining risk levels provided by this application directly determines the set of valuable entities by automatically recognizing entities in each complaint text, eliminating the need for manual annotation, greatly shortening data processing time, and improving the efficiency of risk level determination. On the other hand, the method for determining risk levels provided by this application first associates valuable entities that point to the same object to form an associated entity set, overcoming the problem of entity dispersion caused by differences in expression. Subsequently, a quantitative indicator of complaint frequency was introduced. By statistically analyzing the complaint frequency of related entity sets within a preset time period, a high-frequency entity set was identified, eliminating individual differences in subjective judgment. Finally, based on the high-frequency entity set and the corresponding complaint texts, the risk level of the high-frequency entity set was determined. This ensured that the risk level was both data-driven and took into account the specific complaint context, thus significantly improving the consistency and accuracy of risk level determination. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] Figure 1 A flowchart illustrating a method for determining a risk level as provided in an embodiment of this application;
[0039] Figure 2 This is a schematic diagram of a risk level determination device provided in an embodiment of this application. Detailed Implementation
[0040] As described earlier, the current method for identifying high-risk entities in complaint texts typically involves the following steps: First, after obtaining the customer's complaint text, auditors read through each complaint line and manually mark the entities within, such as the name of the person being complained about, the name of the platform being complained about, and the type of product being complained about. Then, auditors subjectively judge which entities in the complaint text are considered "high-risk entities." However, on the one hand, relying on manual reading of each complaint text and marking the entities manually is time-consuming, resulting in low identification efficiency; on the other hand, different auditors may have different subjective judgments about which entities are "high-risk entities," leading to low consistency and accuracy in identification.
[0041] The inventors, through research, have proposed a method and apparatus for determining risk levels. On one hand, the risk level determination method provided in this application directly identifies a set of valuable entities by automatically recognizing each complaint text, eliminating the need for manual annotation and significantly shortening data processing time, thus improving the efficiency of risk level determination. On the other hand, the risk level determination method first associates valuable entities pointing to the same object, forming a set of associated entities, overcoming the problem of entity dispersion caused by differences in expression. Then, it introduces the quantitative indicator of complaint frequency, determining the high-frequency entity set by statistically analyzing the complaint frequency of the associated entity set within a preset time period, eliminating individual differences in subjective judgment. Finally, based on the high-frequency entity set and the corresponding complaint text, the risk level of the high-frequency entity set is determined, ensuring that the risk level is both data-supported and takes into account the specific complaint context, thereby significantly improving the consistency and accuracy of risk level determination.
[0042] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0043] See Figure 1 The figure is a flowchart of a method for determining a risk level according to an embodiment of this application. The method includes:
[0044] S101: Obtain the complaint data set; the complaint data set includes multiple complaint data, and each complaint data includes the complaint text.
[0045] The complaint data set D includes multiple complaint data sets. Each complaint data set includes: complaint text (the original content of the customer complaint, usually unstructured or semi-structured natural language text), complaint ID (a unique identifier for the complaint, used to distinguish different complaint data sets), complaint timestamp (the time when the complaint occurred or was entered into the system), etc.
[0046] It should be noted that the complaint data set D can be a structured database with data retrieval capabilities, meaning that the corresponding complaint data can be determined based on at least one dimension, such as the complaint text, complaint ID, or complaint timestamp.
[0047] S102: By performing entity identification on each complaint text, a set of valuable entities is determined; the set of valuable entities includes multiple valuable entities.
[0048] First, each complaint text and the first prompt word are input into the large language model. The large language model performs entity recognition on each complaint text to obtain a candidate entity set output by the large language model. The candidate entity set includes multiple candidate entities.
[0049] For example, the first prompt could be as follows: "Please extract the candidate entity set from the following complaint text and output it in the following format: {Candidate Entity Set: Entity Type (Personnel / Platform / Product / Service): Entity Name}." Then, if the complaint text is "Salesperson Zhang San (Employee ID 12345) recommended Insurance B through Platform A, and failed to deliver on the promised rebate.", the candidate entity set output by the large language model could be: "{Candidate Entity Set: Personnel: Zhang San (Employee ID 12345); Platform: Platform A; Product: Insurance B}".
[0050] It should be noted that the candidate entity set can be stored in the form of a data table T1. See Table 1, which is a schematic table of a data table T1 provided in an embodiment of this application.
[0051] Table 1
[0052]
[0053] In one specific implementation, a subset of complaint data can first be determined from the complaint dataset according to preset rules. Then, each complaint text in the subset is input into a large language model to obtain a set of candidate entities output by the large language model. For example, the preset rules could be: nationwide complaints from the past three months.
[0054] Subsequently, the candidate entity set and the second prompt word are input into the large language model. The large language model then performs a value judgment on each candidate entity, resulting in the set of valuable entities output by the model. The second prompt word instructs the large language model to perform a value judgment on each candidate entity, based on the rule that the candidate entity includes identifying information that points to a specific, unique object. The set of valuable entities includes multiple valuable entities, which are entities that include a specific name and a unique identifier (e.g., employee ID, version number, branch office, partner name, etc.), such as Zhang San (employee ID 12345), Platform A, and Insurance B. Entities without value are those with vague descriptions and no specific target, such as salesperson, local company, and this particular insurance policy.
[0055] It should be noted that the set of valuable entities can be stored in the form of a data table T2. See Table 2, which is a schematic table of a data table T2 provided in an embodiment of this application.
[0056] Table 2
[0057]
[0058] S103: Associate valuable entities that point to the same object in the set of valuable entities to determine the set of associated entities.
[0059] First, based on entity type, the set of valuable entities is divided into three subsets: a subset of valuable entities of the personnel type, a subset of valuable entities of the platform type, and a subset of valuable entities of the product / service type. See Table 3, which is a schematic diagram of a set of valuable entities provided in an embodiment of this application.
[0060] Table 3
[0061]
[0062] Firstly, for a subset of valuable entities in the personnel category, if multiple valuable entities satisfy any one of the conditions in the first set of conditions, then these multiple valuable entities are associated to determine the personnel-associated entity subset. The first set of conditions includes the following three conditions: 1. The personnel names are the same; 2. The personnel names and employee IDs are the same; 3. The personnel names and departments are the same.
[0063] Secondly, for a subset of valuable entities belonging to the platform category, if multiple valuable entities satisfy the second condition, then these multiple valuable entities are associated to determine the associated subset of platform-class entities. The second condition is: the platform names are consistent, and the affiliated identifiers (including branches, partners, and business types) have overlap or a subordinate relationship.
[0064] Thirdly, for a subset of valuable entities in the product / service category, if multiple valuable entities satisfy the third condition set, then these multiple valuable entities are associated to determine the associated subset of product / service entities. The third condition set includes the following two conditions: 1. The product name is consistent; 2. At least two of the product type, service type, and version are consistent.
[0065] Finally, the set of associated entities is determined based on the subsets of personnel-related entities, platform-related entities, and product / service-related entities.
[0066] It should be noted that the aforementioned first, second, and third sets of conditions can be manually confirmed by auditors based on the reasonableness of the rules. This application does not impose any restrictions on the specific first, second, and third sets of conditions.
[0067] S104: From the set of associated entities, determine the set of high-frequency entities by determining the frequency of complaints for each set of associated entities within a preset time period.
[0068] For each set of related entities, the following steps are performed: First, extract the complaint ID corresponding to each entity in the set and remove duplicate complaint IDs to determine the complaint ID list. Then, set a statistical time window (e.g., the past month, the whole year, etc.) and determine the number of complaints in the complaint ID list whose timestamps fall within this statistical time window. This number of complaints is the complaint frequency of the set of related entities within the preset time period. Finally, if the complaint frequency is higher than the preset frequency (e.g., 5 times per month), the set of related entities is determined to be a high-frequency entity set.
[0069] S105: Determine the risk level of the high-frequency entity set based on the high-frequency entity set and the complaint text corresponding to the high-frequency entity set.
[0070] Input the high-frequency entity set, the corresponding complaint text (determined by the complaint ID of each entity in the high-frequency entity set), and the third prompt word into the large language model to obtain the risk level output by the large language model.
[0071] For example, the third prompt could be as follows: "Please determine whether the risk level of this high-frequency entity set is high-risk based on the following information: {1. High-frequency entity set; 2. Complaint text corresponding to the high-frequency entity set}. The risk level is related to whether the complaint text corresponding to the high-frequency entity set describes the same type of violation, whether the complaint text corresponding to the high-frequency entity set involves cross-departmental violations, and whether the similarity between the complaint texts corresponding to the high-frequency entity set is higher than a similarity threshold. If yes, output a high risk level and the basis for the judgment; if no, output a low risk level and the basis for the judgment."
[0072] It should be noted that the dimensions for judging the level of risk include: First, behavioral pattern clustering, i.e., whether the complaint texts corresponding to high-frequency entity sets describe the same type of violation, suggesting possible systemic or habitual violations. Second, violation complexity, i.e., whether the complaint texts corresponding to high-frequency entity sets involve cross-departmental violations; such issues are usually more serious and more difficult to investigate. Third, abnormal text similarity, i.e., whether the similarity between complaint texts corresponding to high-frequency entity sets exceeds a similarity threshold; this may indicate mass forged complaints or templated, organized complaint attacks, which is itself a high-risk signal.
[0073] In one specific implementation, M complaint texts (positive integers, such as 3-5) are first determined from the complaint texts corresponding to the high-frequency entity set according to preset rules. Then, the high-frequency entity set and the M complaint texts are input into a large language model to obtain the risk level output by the large language model. Here, the preset rules refer to: determining M complaint texts with high information completeness and including clear descriptions of violations (such as refunds, forged materials, malicious cancellations, etc.) from the complaint texts corresponding to the high-frequency entity set.
[0074] If the output of the large language model indicates a high risk level and the basis for its judgment, then a high-risk audit clue package is generated based on the complaint text and its judgment basis for auditors to investigate.
[0075] If the output of the large language model indicates a low risk level and the basis for its judgment, the complaint text and the basis for its judgment will be saved in the historical clue database and will not need to be sent to the auditors.
[0076] In summary, this application discloses a method for confirming risk levels. On one hand, the method provided in this application directly identifies a set of valuable entities by automatically identifying entities in each complaint text, eliminating the need for manual annotation and significantly shortening data processing time, thus improving the efficiency of risk level determination. On the other hand, the method first associates valuable entities pointing to the same object to form a set of associated entities, overcoming the problem of entity dispersion caused by differences in expression. Then, it introduces the quantitative indicator of complaint frequency, determining the high-frequency entity set by statistically analyzing the complaint frequency of the associated entity set within a preset time period, eliminating individual differences in subjective judgment. Finally, based on the high-frequency entity set and the corresponding complaint text, the risk level of the high-frequency entity set is determined, ensuring that the risk level is both data-supported and takes into account the specific complaint context, thereby significantly improving the consistency and accuracy of risk level determination.
[0077] See Figure 2The figure is a schematic diagram of a risk level confirmation device provided in an embodiment of this application. The risk level confirmation device 200 includes: a set acquisition module 201, an entity recognition module 202, an entity association module 203, a frequency determination module 204, and a level determination module 205;
[0078] The collection acquisition module 201 is used to acquire a collection of complaint data; the collection of complaint data includes multiple complaint data, and each complaint data includes a complaint text;
[0079] The entity recognition module 202 is used to identify a set of valuable entities by performing entity recognition on each complaint text; the set of valuable entities includes multiple valuable entities.
[0080] The entity association module 203 is used to associate valuable entities that point to the same object in the valuable entity set, and to determine the associated entity set.
[0081] The frequency determination module 204 is used to determine the high-frequency entity set from the associated entity set by determining the complaint frequency of each associated entity set within a preset time period;
[0082] The risk level determination module 205 is used to determine the risk level of a high-frequency entity set based on the high-frequency entity set and the complaint text corresponding to the high-frequency entity set.
[0083] In one specific implementation, the entity recognition module 202 is specifically used to: input each complaint text into a large language model, perform entity recognition on each complaint text through the large language model, and obtain a candidate entity set output by the large language model; the candidate entity set includes multiple candidate entities; input the candidate entity set into the large language model, perform value judgment on each candidate entity through the large language model, and obtain a valuable entity set output by the large language model; the rule for value judgment is: whether the candidate entity includes identification information that can point to a specific unique object.
[0084] In one specific implementation, the entity association module 203 is specifically used to: determine a subset of valuable entities of the personnel class, a subset of valuable entities of the platform class, and a subset of valuable entities of the product / service class from the set of valuable entities; for the subset of valuable entities of the personnel class, if multiple valuable entities satisfy any one of the conditions in the first condition set, then associate the multiple valuable entities to determine the personnel associated entity subset; the first condition set includes: personnel name is the same, personnel name and employee number are the same, personnel name and department are the same; for the subset of valuable entities of the platform class, if multiple valuable entities satisfy any one of the conditions in the first condition set, then associate the multiple valuable entities to determine the personnel associated entity subset; the first condition set includes: personnel name is the same, personnel name and employee number are the same, personnel name and department are the same; for the subset of valuable entities of the platform class, if multiple valuable entities satisfy any one of the conditions in the first condition set, then associate the multiple valuable entities to determine the personnel associated entity subset. The second condition involves associating multiple valuable entities to determine a subset of platform-type associated entities. This second condition is that the platform names are identical, and the subordinate identifiers have overlap or a subordinate relationship. For the subset of product / service-type valuable entities, if multiple valuable entities satisfy the third condition set, these multiple valuable entities are associated to determine a subset of product / service-type associated entities. This third condition set is that the product names are identical, and at least two of the product type, service type, and version are identical. Based on the subsets of personnel-type associated entities, platform-type associated entities, and product / service-type associated entities, the set of associated entities is determined.
[0085] In one specific implementation, the risk level determination module 205 is specifically used to: input the high-frequency entity set and the complaint text corresponding to the high-frequency entity set into the large language model to obtain the risk level output by the large language model.
[0086] In one specific implementation, the risk level is related to whether the complaint texts corresponding to the high-frequency entity set describe the same type of violation, whether the complaint texts corresponding to the high-frequency entity set involve cross-departmental violations, and whether the similarity between the complaint texts corresponding to the high-frequency entity set is higher than the similarity threshold.
[0087] In summary, this application discloses a risk level confirmation device. On one hand, the risk level determination device provided in this application automatically identifies valuable entity sets by performing entity recognition on each complaint text, eliminating the need for manual annotation of each complaint, greatly shortening data processing time, and improving the efficiency of risk level determination. On the other hand, the risk level determination device provided in this application first associates valuable entities pointing to the same object to form an associated entity set, overcoming the problem of entity dispersion caused by differences in expression. Subsequently, it introduces the quantitative indicator of complaint frequency, determining the high-frequency entity set by statistically analyzing the complaint frequency of the associated entity set within a preset time period, eliminating individual differences in subjective judgment. Finally, based on the high-frequency entity set and the complaint text corresponding to the high-frequency entity set, the risk level of the high-frequency entity set is determined, ensuring that the risk level is both data-supported and takes into account the specific complaint context, thereby significantly improving the consistency and accuracy of risk level determination.
[0088] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0089] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method of determining a risk level, characterized by, The method comprises: acquiring a complaint data set; the complaint data set comprises a plurality of complaint data, and each complaint data comprises complaint text; determining a valuable entity set by performing entity recognition on each complaint text; the valuable entity set comprises a plurality of valuable entities; associating valuable entities in the valuable entity set that point to the same object to determine an associated entity set; determining a high-frequency entity set from the associated entity set by determining the complaint frequency of each associated entity set within a preset time period; determining the risk level of the high-frequency entity set according to the high-frequency entity set and the complaint text corresponding to the high-frequency entity set.
2. The method of claim 1, wherein, The method comprises: inputting each complaint text into a large language model to perform entity recognition on each complaint text by the large language model to obtain a candidate entity set output by the large language model; the candidate entity set comprises a plurality of candidate entities; inputting the candidate entity set into the large language model to perform value judgment on each candidate entity by the large language model to obtain a valuable entity set output by the large language model; the value judgment rule is whether the candidate entity includes identification information that can point to a specific unique object.
3. The method of claim 1, wherein, The method comprises: determining a personnel valuable entity subset, a platform valuable entity subset, and a product / service valuable entity subset in the valuable entity set; for the personnel valuable entity subset, if a plurality of valuable entities satisfy any condition in a first condition set, associating the plurality of valuable entities to determine a personnel associated entity subset; the first condition set comprises: identical personnel name, identical personnel name and identical employee number, and identical personnel name and identical department; for the platform valuable entity subset, if a plurality of valuable entities satisfy a second condition, associating the plurality of valuable entities to determine a platform associated entity subset; the second condition is: identical platform name, and intersection of subsidiary identifiers or subordination relationship exists; for the product / service valuable entity subset, if a plurality of valuable entities satisfy a third condition set, associating the plurality of valuable entities to determine a product / service associated entity subset; the third condition set is: identical product name, and at least two of product type, service type, and version are identical; determining an associated entity set according to the personnel associated entity subset, the platform associated entity subset, and the product / service associated entity subset.
4. The method of claim 1, wherein, The method comprises: inputting the high-frequency entity set and the complaint text corresponding to the high-frequency entity set into a large language model to obtain a risk level output by the large language model.
5. The method of claim 4, wherein, The risk level is related to whether the complaint texts corresponding to the high-frequency entity set describe the same type of violation, whether the complaint texts corresponding to the high-frequency entity set involve cross-department violations, and whether the similarity between the complaint texts corresponding to the high-frequency entity set is higher than a similarity threshold.
6. A device for determining a risk level, characterized in that The device comprises a set acquisition module, an entity identification module, an entity association module, a frequency determination module, and a level determination module. The set acquisition module is configured to acquire a complaint data set; the complaint data set comprises a plurality of complaint data, and each complaint data comprises complaint text. The entity identification module is configured to determine a valuable entity set by performing entity identification on each complaint text. The entity association module is configured to associate valuable entities in the valuable entity set that point to the same object to determine an associated entity set. The frequency determination module is configured to determine a high-frequency entity set from the associated entity set by determining the complaint frequency of each associated entity set within a preset time period. The level determination module is configured to determine the risk level of the high-frequency entity set according to the high-frequency entity set and the complaint texts corresponding to the high-frequency entity set.
7. The apparatus of claim 6, wherein, The entity identification module is specifically configured to input each complaint text into a large language model, perform entity identification on each complaint text through the large language model, and obtain a candidate entity set output by the large language model; the candidate entity set comprises a plurality of candidate entities; input the candidate entity set into the large language model, perform value judgment on each candidate entity through the large language model, and obtain a valuable entity set output by the large language model; the rule of the value judgment is whether the candidate entity includes identification information that can point to a specific unique object.
8. The apparatus of claim 6, wherein, The entity association module is specifically configured to determine a personnel-type valuable entity subset, a platform-type valuable entity subset, and a product / service-type valuable entity subset in the valuable entity set. For the personnel valuable entity subset, if multiple valuable entities satisfy any condition in a first condition set, the multiple valuable entities are associated to determine a personnel associated entity subset; the first condition set includes: personnel names are consistent, personnel names and work numbers are consistent, and personnel names and departments are consistent; for the platform valuable entity subset, if multiple valuable entities satisfy a second condition, the multiple valuable entities are associated to determine a platform associated entity subset; the second condition is: platform names are consistent, and affiliated identifiers have an intersection or a subordinate relationship; for the product / service valuable entity subset, if multiple valuable entities satisfy a third condition set, the multiple valuable entities are associated to determine a product / service associated entity subset; the third condition set is: product names are consistent, and at least two of product types, service types, and versions are consistent; and an associated entity set is determined according to the personnel associated entity subset, the platform associated entity subset, and the product / service associated entity subset.
9. The apparatus of claim 6, wherein, The level determination module is specifically configured to: input the high-frequency entity set and the complaint text corresponding to the high-frequency entity set into a large language model to obtain a risk level output by the large language model.
10. The apparatus of claim 9, wherein, The risk level is related to whether the complaint text corresponding to the high-frequency entity set describes the same type of violation, whether the complaint text corresponding to the high-frequency entity set involves cross-department violations, and whether the similarity between the complaint texts corresponding to the high-frequency entity set is higher than a similarity threshold.