Automated review method and device

Abstract

The embodiments of this specification provide an automated trial method and device. The method includes: obtaining subject data features; inputting the subject data features into a pre-trained trial model to obtain a trial result output by the trial model; wherein the trial model is obtained by training using the experience of a first artificial expert; inputting the trial result into a pre-trained message generation model to obtain a trial result message output by the message generation model; wherein the message generation model is obtained by training using the experience of a second artificial expert; and reporting the trial result message. The embodiments of this specification can automatically implement the trial business of conducting a trial on a subject.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to electronic information technology, and more particularly, to automated review methods and devices. Background Art

[0002] With the development of network communication, network communication security has become increasingly important. Various behavioral data transmitted on the network need to be reviewed in order to discover security risks in a timely manner.

[0003] Currently, the main method is to search for user data features on the Internet and then conduct manual review. However, manual review has many disadvantages, so there is an urgent need for an automated review method that can be performed by a computer system. Summary of the invention

[0004] One or more embodiments of this specification describe an automated review method and apparatus that can avoid the problems caused by manual review.

[0005] According to a first aspect, an automated review method is provided, comprising:

[0006] Obtain subject data characteristics;

[0007] Inputting the subject data features into a pre-trained adjudication model to obtain an adjudication result output by the adjudication model; wherein the adjudication model is obtained by training using the first artificial expert experience;

[0008] Inputting the trial result into a pre-trained message generation model to obtain a trial result message output by the message generation model; wherein the message generation model is obtained by training using the experience of a second artificial expert;

[0009] Report the review result message.

[0010] Wherein, the trial model includes: a single subject identification model for identifying a single subject;

[0011] The training method of the single subject recognition model includes:

[0012] According to the type and characteristics of a single subject, determine the recognition tasks of the single subject recognition model and the characteristics of each recognition task;

[0013] Forming first artificial rules from first artificial expert experiences for identifying a single subject;

[0014] Adding the given weight values ​​of each first artificial rule to the loss function of the algorithm used by the single-subject recognition model to form a first loss function;

[0015] The single-agent recognition model is trained using the first loss function.

[0016] Wherein, the first loss function includes:

[0017]

[0018] Among them, i represents each sample; y i Represents the value of the i-th sample in the binary classification sample; r i represents the probability that the i-th sample value is the first classification, (1-r i ) represents the probability that the i-th sample value is the second classification; F represents the set of each first artificial rule; w f Represents the weight of rule f; f+ represents the rule set that identifies the sample as the first classification; f- represents the rule set that identifies the sample as the second classification; r represents the value of rule f; Representation: If the sample is identified as the first category or the second category by at least one of the first manual rules, the value is 1, otherwise the value is 0.

[0019] Wherein, the single subject recognition model includes:

[0020] A first layer sub-model, which identifies a single subject through a whitelist and a blacklist, and outputs a first identification result;

[0021] The second layer sub-model is used to perform binary classification recognition of a single subject and output the second recognition result;

[0022] The third layer sub-model is used to perform multi-task recognition of a single subject using a meta-learning algorithm according to the recognition task and the characteristics of each recognition task, and output a third recognition result;

[0023] The fourth-layer sub-model is used to obtain the recognition result of a single subject according to the first recognition result, the second recognition result and the third recognition result.

[0024] Wherein, the trial model includes: a multi-subject recognition model for identifying a gang formed by multiple subjects;

[0025] The training method of the multi-agent recognition model includes: forming respective second artificial rules from respective first artificial expert experiences for recognizing multiple subjects;

[0026] The second artificial rules for determining whether or not they meet the requirements for gang identification are added to the loss function of the algorithm used by the multi-agent identification model to form a second loss function; and the multi-agent identification model is trained using the second loss function.

[0027] Wherein, the second loss function includes:

[0028]

[0029] Among them, A ij represents the weight of the edge connecting node i and node j; k i , k j Represents the sum of the weights of all edges connected to nodes i and j; m represents the number of all edges in the graph; Represents the weight of the connection between node i and node j in a random situation; represents the difference between the actual connection weight and the random connection weight; δ(c i ,c j ) represents a function. When node i and node j are in the same group, the function value is 1, otherwise it is 0; max{max{r},max{1-r}} represents a second artificial rule. If it is judged that node i and node j belong to the same group or not, the function value is 1, otherwise it is 0; λ represents the weight of the second artificial rule.

[0030] The step of inputting the trial result into a pre-trained message generation model to obtain a trial result message output by the message generation model includes:

[0031] Using the trial result, searching in a knowledge graph; the knowledge graph is generated using the second artificial expert experience;

[0032] Use the retrieval results as input to the pre-trained retrieval-based algorithm model;

[0033] The text output by the retrieval-based algorithm model is encoded, and the encoding result is used as the input of the pre-trained generation-based algorithm model to obtain the trial result message output word by word by the generation-based algorithm model.

[0034] The retrieval-based algorithm model and the generation-based algorithm model are jointly trained, and the following third loss function is used for the joint training:

[0035] Where a represents the weight of the loss function corresponding to the second artificial expert experience, a is a preset constant; L represents the length of the target generated sentence, t represents each word in the sentence, and p t Represents the probability of each word in the sentence generated by the target, and m represents the number of all words in the description text formed by the second artificial expert experience for each customer; Represents the probability of each word in the second artificial expert experience, The representation takes the value with the highest probability for all the words in the second artificial expert experience.

[0036] According to a second aspect, an automated review device is provided, comprising:

[0037] a data input module configured to obtain subject data features and input the subject data features into the adjudication model module;

[0038] A review model module, configured to obtain a review result using a pre-trained review model, and send the review result to the message generation module; wherein the review model is obtained by training using the first artificial expert experience;

[0039] The message generation module is configured to obtain the trial result message using a pre-trained message generation model and report the trial result message; wherein the message generation model is obtained by training using the experience of a second artificial expert.

[0040] Wherein, the trial model module includes:

[0041] A single subject recognition model, configured to recognize a single subject;

[0042] The multi-agent identification model is configured to identify groups formed by multiple agents.

[0043] According to a third aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any embodiment of the present specification is implemented.

[0044] The automated review method and device provided in the embodiments of this specification can pre-train a review model for completing the review process, and then use the review model executed by the computer to perform automated review. Moreover, after the review, the review result message is automatically generated using the pre-trained message generation model. Therefore, in the entire review business, the process from the review process to the reporting of the review result message does not require human intervention and is fully executed, thereby solving many problems of manual review. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 It is a flowchart of an automated review method in one embodiment of the present specification.

[0047] Figure 2It is a schematic diagram of a business scenario applied by an embodiment of this specification.

[0048] Figure 3 It is a schematic diagram of the structure of an automated review device in one embodiment of this specification. DETAILED DESCRIPTION

[0049] As mentioned above, manual review is currently the main method. For example, based on their own expert experience and the behavioral data characteristics of a user on the Internet, a person will judge whether the user's behavior is gambling, pyramid selling, cashing out, or fraud (such as order fraud). However, manual review has many disadvantages. For example, manual review is time-consuming and inefficient. For example, during manual review, different people have different review experiences, which makes it impossible to standardize and precipitate, and the iterative optimization efficiency is low.

[0050] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0051] The core idea of ​​this manual is to pre-train a computer model that can be used to complete the trial process, and then use the computer model to perform automated trial, eliminating the need for manual trial and solving many problems of manual trial.

[0052] Figure 1 is a flow chart of an automated trial method in one embodiment of this specification. The method is performed by an automated trial device. It can be understood that the method can also be performed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 1 , the method comprising:

[0053] Step 101: Obtain subject data features.

[0054] Step 103: Input the subject data features into a pre-trained adjudication model to obtain an adjudication result output by the adjudication model; wherein the adjudication model is obtained by training using the first artificial expert experience.

[0055] Step 105: input the trial result into a pre-trained message generation model to obtain a trial result message output by the message generation model; wherein the message generation model is obtained by training using the experience of a second artificial expert.

[0056] Step 107: Report the review result message.

[0057] It can be seen that in the above Figure 1In the process shown, a trial model that can be used to complete the trial process is pre-trained, and then the trial model executed by the computer is used for automated trial. Moreover, after the trial, the trial result message is automatically generated using the pre-trained message generation model. Therefore, in the entire trial business, from the trial process to the reporting of the trial result message, no human intervention is required and the entire process is automatically executed, thereby solving many problems of manual trial.

[0058] In the above Figure 1 In the process shown, the artificial expert experience of subject identification is used to train the trial model executed by the computer, and the artificial expert experience of message generation is used to train the message generation model, so that the recognition accuracy of the generated model can be higher. Among them, the first artificial expert experience refers to: how to determine the user's risky behavior through user behavior data; the second artificial expert experience refers to: how to describe the user's risky behavior in words.

[0059] In one embodiment of the present specification, the trial model may include at least one of the following models: a single subject identification model for identifying a single subject, and a multi-subject identification model for identifying multiple subjects. The single subject identification model is used to extract suspicious features and characterize a single case; the multi-subject identification model is used to cluster cases that can be clustered and characterize them as gang cases.

[0060] When the adjudication model includes both of the above models, see Figure 2 , the processing flow of the embodiment of this specification may be: first, the subject data features are input into the single subject recognition model, and the single subject recognition model extracts and characterizes suspicious features for each individual case, and the output result of the single subject recognition model is used as input into the multi-subject recognition model, and the multi-subject recognition model then clusters the cases that can be clustered according to the characterization results of each individual case, and identifies the gang cases; then, the output result of the multi-subject recognition model is used as input into the message generation model, and the message generation model writes the trial result message for the single subject and / or gang case to generate the content of the trial result message that can be recognized by the user. It can be seen that the entire process does not require human participation, and automatic trial is realized.

[0061] In the embodiments of this specification, a subject can be a transaction or a user, that is, an automated review can be conducted on a transaction (for example, identifying whether the transaction is gambling, whether the transaction is a gang crime, etc.), or an automated review can be conducted on a user (for example, identifying whether the user is a fraudulent user, the gang to which the user belongs, etc.).

[0062] The following is a detailed description of the embodiments of the present invention. Figure 1 The process shown is described.

[0063] First, in step 101, the subject data features are obtained.

[0064] In this step, the subject data characteristics may include data characteristics involved in a transaction, such as the payee's account information, the payee's account information, the funds flow path, etc. The subject data characteristics may also include a user's behavior data characteristics, such as login time, user name, number of transactions, etc.

[0065] Next, in step 103, the subject data features are input into a pre-trained adjudication model to obtain an adjudication result output by the adjudication model; wherein the adjudication model is obtained by training using the first artificial expert experience.

[0066] The implementation of step 103 includes any one of the following implementations 1 and 2, or a combination of the two:

[0067] Implementation method 1:

[0068] As mentioned above, the adjudication model may include a single subject recognition model for identifying a single subject. Then, in the first implementation of step 103, the subject data features may be input into the pre-trained single subject recognition model to obtain the recognition result of the single subject, such as a single suspicious transaction case, output by the adjudication model.

[0069] The single subject recognition model is pre-trained using the experience of first-person experts who recognize a single subject. The training process of the single subject recognition model includes the following steps:

[0070] Step 103A1: Determine the recognition tasks of the single-subject recognition model and the characteristics of each recognition task according to the type and characteristics of the single subject.

[0071] At present, there are many types of subjects that need to be identified, such as gambling, pyramid schemes, illegal fund-raising, underground banks, virtual currency, cashing out, fraud, drug-related crimes, etc. Therefore, in order to accurately identify the type of crime for a single subject, various recognition tasks and the characteristics of each recognition task can be designed in advance for the single subject recognition model.

[0072] Step 103A3: Forming first artificial rules based on the first artificial expert experiences for identifying a single subject.

[0073] One advantage of manual review is that it has manual expert experience, which has a relatively high recognition accuracy. Therefore, in order to improve the recognition accuracy of the model, the manual expert experience is introduced into the training process of the single-subject recognition model. In order to introduce the manual expert experience, it is necessary to perform the processing of this step 103A3 to form each first manual rule with each first manual expert experience.

[0074] Step 103A5: adding the given weight values ​​of each first artificial rule to the loss function of the algorithm used by the single-agent recognition model to form a first loss function.

[0075] Step 103A7: Use the first loss function to train the single-agent recognition model.

[0076] In the above process, the first loss function can be designed as:

[0077]

[0078] The first part of the first loss function is the cross entropy Loss of the traditional two-classification model org , where i represents each sample; y i Represents the value of the i-th sample in the binary classification sample. For example, when this sample is classified as the first classification, the value is 1, and when it is the second classification, the value is 0; r i represents the probability that the i-th sample value is the first classification, (1-r i ) represents the probability that the i-th sample value is the second classification.

[0079] The second part of the first loss function is the adjustment of the loss function by the first artificial expert experience. rule , where F represents the set of first artificial rules; w f Represents the weight of the artificial rule f; f+ represents the rule set that identifies the sample as the first category; f- represents the rule set that identifies the sample as the second category; r represents the value of the artificial rule f, for example, if the artificial rule f identifies the sample as the first category, the value is 1, if it is identified as the second category, the value is 0; Representation: If the sample is identified as the first category or the second category by at least one of the first manual rules, the value is 1, otherwise the value is 0.

[0080] In the first loss function above, the Loss in the second part rule The meaning is: the result of each first artificial rule hit by the sample is added to the loss function as a weight. If the first artificial rule determines that the sample is of the first classification, the probability that the model recognizes the sample as the first classification is increased. If the first artificial rule determines that the sample is of the second classification, the probability that the model recognizes the sample as the second classification is increased.

[0081] By using the first loss function in the embodiment of this specification, a single-subject recognition model with a higher recognition rate can be trained in combination with the first artificial expert experience.

[0082] In one embodiment of the present specification, in the first implementation, the trained single subject recognition model may include three layers of sub-models, specifically including:

[0083] A first layer sub-model, which identifies a single subject through a whitelist and a blacklist, and outputs a first identification result;

[0084] The second layer sub-model is used to perform binary classification recognition of a single subject and output the second recognition result;

[0085] The third layer sub-model is used to perform multi-task recognition of a single subject using a meta-learning algorithm according to the recognition tasks determined in step 103A1 and the features of each recognition task, and output a third recognition result;

[0086] The fourth-layer sub-model is used to obtain the recognition result of a single subject according to the first recognition result, the second recognition result and the third recognition result.

[0087] The above four-layer sub-model obtains the final recognition result of a single subject by recognizing layer by layer and synthesizing the recognition results of each layer, so the recognition result is more accurate. In addition, in the third-layer sub-model, because the number of recognition tasks involved may be large, but the samples may be small, a meta-learning algorithm can be used for recognition.

[0088] Implementation method 2:

[0089] As mentioned above, the adjudication model may include a multi-subject recognition model for identifying multiple subjects. Then in the second implementation of step 103, the multi-subject recognition model may be used to obtain recognition results of multiple subjects, such as multiple suspicious transaction cases.

[0090] If step 103 only adopts implementation method 2, then the subject data features, such as the data features involved in a transaction, are input into the pre-trained multi-subject recognition model; if step 103 adopts implementation method 1 combined with implementation method 2, then the recognition results of each single subject output by the single subject recognition model and the data features of each single subject are taken as input and input into the pre-trained multi-subject recognition model.

[0091] The multi-agent recognition model is pre-trained using the experience of first-person experts who identify multiple subjects. The training process of the multi-agent recognition model includes the following steps:

[0092] Step 103B1: Forming second artificial rules based on the first artificial expert experiences for identifying multiple subjects.

[0093] Step 103B3: adding each second artificial rule for whether it meets the requirements of gang identification to the loss function of the algorithm used by the multi-agent identification model to form a second loss function.

[0094] Step 103B5: Use the second loss function to train the multi-agent recognition model.

[0095] In the above process, the second loss function can be designed as:

[0096]

[0097] The first part of the second loss function is the modularity of the traditional Louvain algorithm. ij represents the weight of the edge connecting node i and node j; k i , k j Represents the sum of the weights of all edges connected to nodes i and j; m represents the number of all edges in the graph; Represents the weight of the connection between node i and node j in a random situation; represents the difference between the actual connection weight and the random connection weight, which is the gain of the current partitioning scheme; δ(c i ,c j ) represents a function. When node i and node j are in the same group, the function value is 1, otherwise it is 0. The function is to automatically calculate the nodes in each group separately. Because when calculating the nodes of different groups, this term is 0 and the whole formula is 0.

[0098] The second part of the second loss function is: the adjustment of the loss function by the first artificial expert experience of identifying multiple subjects. Among them, max{max{r},max{1-r}} represents a second artificial rule. If it is judged that node i and node j belong to a group or not, the value is 1, otherwise the value is 0; λ represents the weight of the second artificial rule.

[0099] The significance of adjusting the second loss function by each second artificial rule is that if the current gang division plan conforms to the artificial rules, the modularity of the entire plan will be improved.

[0100] In one embodiment of this specification, the functions that can be realized by the multi-subject identification model include: gang discovery, gang backbone marking, gang tracking and gang role division. Gang discovery can use the Louvain algorithm to perform gang discovery based on modularity; the gang tracking algorithm comprehensively considers the overlap ratio of gang members at different stages, the core degree of overlapping members in the gang, and considers the time decay factor to give the organizational form of similar crime gangs at different stages, and obtains similar crime gangs that continue to commit crimes; the gang role is to identify the roles of the divided gangs. For example, a gang includes multiple roles such as collection, circulation, refund, withdrawal, gambler, etc. Identifying the gang roles can improve the accuracy of gang identification, thereby improving the efficiency of trial.

[0101] Next, in step 105, the trial result is input into a pre-trained message generation model to obtain a trial result message output by the message generation model; wherein the message generation model is obtained by training using the experience of a second artificial expert.

[0102] After the processing of step 103, such as the processing of implementation method 1 plus implementation method 2, the review result can be obtained, such as whether a transaction involves money laundering or multiple users are found to belong to a gambling gang, etc. The review result needs to be reported to the administrator for subsequent processing.

[0103] After being processed by the review model in step 103, the review result obtained is a computer-recognizable result, but it is not a text language with sentence meaning that can be understood by the administrator. Therefore, it is necessary to execute the processing of this step 105, and the message generation model converts the review result in computer language into a review result message with sentence meaning that can be understood by the administrator.

[0104] In one embodiment of the present specification, the message generation model includes two algorithm models: a retrieval-based algorithm model and a generation-based algorithm model. The retrieval-based algorithm model uses an index-based algorithm that can convert the result of digital representation into the result of text representation; the generation-based algorithm model uses an algorithm that combines contextual semantics to output recognition results word by word.

[0105] Based on the retrieval-based algorithm model and the generation-based algorithm model, a specific implementation process of step 105 may include:

[0106] Step 1051: Using the review results, search in the knowledge graph; the knowledge graph is generated using the second artificial expert experience.

[0107] Here, for example, when searching in the knowledge graph, each node on the retrieved path needs to be reflected as a search result in the subsequently generated review result message, thereby avoiding the lack of description of some results in the review result message.

[0108] Step 1053: Use the retrieval results as input to the pre-trained retrieval-based algorithm model.

[0109] Here, the retrieval-based algorithm model can use the index library to convert the numerical retrieval result in step 1051 into text. For example, if the retrieval result is 18, the retrieval-based algorithm model can output text such as "the gang includes a total of 18 people."

[0110] Step 1055: Encode the text output by the retrieval-based algorithm model, and use the encoding result as the input of the pre-trained generation-based algorithm model to obtain the trial result message verbatim output by the generation-based algorithm model.

[0111] In this step 1055, the generation-based algorithm model can generate the trial result message verbatim and reflect the relevance between words. For example, when generating the trial result message corresponding to an anti-money laundering transaction, it is more likely that the word "money" follows the word "wash" in the text, so the message content such as "money laundering" can be generated by integrating the algorithm results.

[0112] In the above implementation process, the second artificial expert experience for generating the trial result message can be used in advance to form a knowledge graph. In this way, the finally generated message content is no longer separately fragmented, but logically related through the knowledge graph. In this way, through the processing of the above steps 1051 to 1053, the automatically generated trial result message can be made more readable and the message description more comprehensive.

[0113] As mentioned above, the message generation model is pre-trained. During this training process, the retrieval-based algorithm model and the generation-based algorithm model need to be jointly trained, and the following third loss function is used for this joint training:

[0114]

[0115] Among them, α represents the weight of the loss function corresponding to the second artificial expert experience and can be a preset constant; L represents the length of the target generated sentence, t represents each word in the sentence, p t represents the probability of each word in the target generated sentence, and m represents the total number of words in the descriptive text formed by the second artificial expert experience for each customer; represents the probability of each word in each second artificial expert experience, represents the maximum value of the probabilities of the words in all the above second artificial expert experiences.

[0116] It can be seen that in the third loss function, one part is the part corresponding to the second artificial expert experience: The other part is

[0117] Among them, for the part corresponding to the second artificial expert experience: Mainly, the second artificial expert experience will form a description text for each customer, and the probability of each word in the text is inherited from the generation model, that is, m represents the number of words in the text, and the largest probability is taken as log to determine the loss function of the expert experience part, so as to modify the model parameters: when the machine generates correct words, the probability of correct words is strengthened; when the machine generates incorrect words, the probability of incorrect words is reduced and the probability of correct words is increased, and the convergence of the model is accelerated. Among them, the third loss function can also have other forms, for example, Take the maximum value and replace it with the average or minimum value, etc.

[0118] For the machine learning part The cross entropy of the general generative model can be used as the loss function.

[0119] In the embodiment of this specification, by adding the part corresponding to the second artificial expert experience into the total loss function, the accuracy and coverage of the generated user description text can be improved, and a better effect can be achieved even when the number of label texts is relatively small.

[0120] In one embodiment of this specification, an automated review device is proposed. Figure 3 , the device comprises:

[0121] A data input module 301 is configured to obtain subject data features and input the subject data features into the adjudication model module;

[0122] The review model module 302 is configured to obtain a review result using a pre-trained review model and send the review result to the message generation module; wherein the review model is obtained by training using the first artificial expert experience;

[0123] The message generation module 303 is configured to obtain the trial result message using a pre-trained message generation model and report the trial result message; wherein the message generation model is obtained by training using the experience of a second artificial expert.

[0124] In one embodiment of the apparatus of this specification, the review model module 302 includes:

[0125] A single subject recognition model, configured to recognize a single subject;

[0126] The multi-agent identification model is configured to identify groups formed by multiple agents.

[0127] In one embodiment of the apparatus of the present specification, the adjudication model module 302 includes: a single subject identification model for identifying a single subject;

[0128] The training method of the single subject recognition model includes:

[0129] According to the number and characteristics of a single subject, determine the number of recognition tasks of the single subject recognition model and the characteristics of each recognition task;

[0130] Forming first artificial rules from first artificial expert experiences for identifying a single subject;

[0131] Adding the given weight values ​​of each first artificial rule to the loss function of the algorithm used by the single-subject recognition model to form a first loss function;

[0132] The single-agent recognition model is trained using the first loss function.

[0133] In one embodiment of the apparatus of this specification, the first loss function is configured as:

[0134]

[0135] Among them, i represents each sample; y i Represents the value of the i-th sample in the binary classification sample; r i represents the probability that the i-th sample value is the first classification, (1-r i) represents the probability that the i-th sample value is the second classification; F represents the set of each first artificial rule; w f Represents the weight of rule f; f+ represents the rule set that identifies the sample as the first classification; f- represents the rule set that identifies the sample as the second classification; r represents the value of rule f; Representation: If the sample is identified as the first category or the second category by at least one of the first manual rules, the value is 1, otherwise the value is 0.

[0136] In one embodiment of the device of this specification, the above-mentioned single subject recognition model includes:

[0137] A first layer sub-model, which identifies a single subject through a whitelist and a blacklist, and outputs a first identification result;

[0138] The second layer sub-model is used to perform binary classification recognition of a single subject and output the second recognition result;

[0139] The third layer sub-model is used to perform multi-task recognition of a single subject using a meta-learning algorithm according to the number of recognition tasks and the characteristics of each recognition task, and output a third recognition result;

[0140] The fourth-layer sub-model is used to obtain the recognition result of a single subject according to the first recognition result, the second recognition result and the third recognition result.

[0141] In one embodiment of the apparatus of this specification, the adjudication model module 302 includes: a multi-subject recognition model for identifying a gang formed by multiple subjects;

[0142] The training method of the multi-agent recognition model includes: forming respective second artificial rules from respective first artificial expert experiences for recognizing multiple subjects;

[0143] The second artificial rules for determining whether or not they meet the requirements for gang identification are added to the loss function of the algorithm used by the multi-agent identification model to form a second loss function; and the multi-agent identification model is trained using the second loss function.

[0144] In one embodiment of the apparatus of this specification, the second loss function is configured as:

[0145]

[0146] Among them, A ij represents the weight of the edge connecting node i and node j; k i , k j Represents the sum of the weights of all edges connected to nodes i and j; m represents the number of all edges in the graph; Represents the weight of the connection between node i and node j in a random situation; represents the difference between the actual connection weight and the random connection weight; δ(c i ,c j ) represents a function. When node i and node j are in the same group, the function value is 1, otherwise it is 0; max{max{r},max{1-r}} represents a second artificial rule. If it is judged that node i and node j belong to the same group or not, the function value is 1, otherwise it is 0; λ represents the weight of the second artificial rule.

[0147] In one embodiment of the apparatus of this specification, the message generation module 303 is configured to execute:

[0148] Using the trial result, searching in a knowledge graph; the knowledge graph is generated using the second artificial expert experience;

[0149] Use the retrieval results as input to the pre-trained retrieval-based algorithm model;

[0150] The text output of the retrieval-based algorithm model is encoded, and the encoding result is used as the input of the pre-trained generation-based algorithm model to obtain the trial result message output word by word by the generation-based algorithm model.

[0151] In one embodiment of the device of this specification, the retrieval-based algorithm model configured by the message generation module 303 is jointly trained with the generation-based algorithm model, and the following loss function is used for the joint training:

[0152] Where α represents the weight of the function corresponding to the second artificial expert experience, which can be a preset constant; L represents the length of the target generated sentence, t represents each word in the sentence, and p t Represents the probability of each word in the sentence generated by the target, and m represents the number of all words in the description text formed by the second artificial expert experience for each customer; Represents the probability of each word in the second artificial expert experience, The representation takes the value with the highest probability for all the words in the second artificial expert experience.

[0153] One embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute a method in any one of the embodiments of the present specification.

[0154] An embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method in any embodiment of the specification is implemented.

[0155] It is to be understood that the structures illustrated in the embodiments of this specification do not constitute specific limitations on the devices of the embodiments of this specification. In other embodiments of the specification, the above-mentioned device may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0156] Since the information interaction, execution process, etc. between the modules in the above-mentioned devices and systems are based on the same concept as the method embodiments of this specification, the specific contents can be found in the description of the method embodiments of this specification and will not be repeated here.

[0157] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0159] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. An automated review method, wherein: include: Obtain subject data characteristics; Inputting the subject data feature into a pre-trained adjudication model to obtain an adjudication result output by the adjudication model; wherein the adjudication model is obtained by training using the first artificial expert experience; Inputting the trial result into a pre-trained message generation model to obtain a trial result message output by the message generation model; wherein the message generation model is obtained by training using the experience of a second artificial expert; Reporting the trial result message; The trial model includes: a single subject recognition model for identifying a single subject or a multi-subject recognition model for identifying a group formed by multiple subjects; The training method of the single subject recognition model includes: According to the type and characteristics of a single subject, determine the recognition tasks of the single subject recognition model and the characteristics of each recognition task; Forming first artificial rules from first artificial expert experiences for identifying a single subject; adding the given weight values ​​of each first artificial rule to the loss function of the algorithm used by the single-agent recognition model to form a first loss function; and Using the first loss function to train the single subject recognition model; The training method of the multi-agent recognition model includes: Forming respective second artificial rules from respective first artificial expert experiences for identifying a plurality of subjects; and The second artificial rules for determining whether or not they meet the requirements for gang identification are added to the loss function of the algorithm used by the multi-agent identification model to form a second loss function; and the multi-agent identification model is trained using the second loss function.

2. The method according to claim 1, wherein: The first loss function includes: Among them, i represents each sample; y i Represents the value of the i-th sample in the binary classification sample; r i represents the probability that the i-th sample value is the first classification, (1-r i ) represents the probability that the i-th sample value is the second classification; F represents the set of each first artificial rule; w f Represents the weight of rule f; f+ represents the rule set that identifies the sample as the first classification; f- represents the rule set that identifies the sample as the second classification; r represents the value of rule f; Representation: If the sample is identified as the first category or the second category by at least one of the first manual rules, the value is 1, otherwise the value is 0.

3. The method according to claim 1, wherein: The single subject recognition model includes: A first layer sub-model, which identifies a single subject through a whitelist and a blacklist, and outputs a first identification result; The second layer sub-model is used to perform binary classification recognition of a single subject and output the second recognition result; The third layer sub-model is used to perform multi-task recognition of a single subject using a meta-learning algorithm according to the recognition task and the characteristics of each recognition task, and output a third recognition result; The fourth-layer sub-model is used to obtain the recognition result of a single subject according to the first recognition result, the second recognition result and the third recognition result.

4. The method according to claim 1, wherein: The second loss function includes: Among them, A ij represents the weight of the edge connecting node i and node j; k i , k j represents the sum of the weights of all edges connected to nodes i and j; m represents the number of all edges in the graph; Represents the weight of the connection between node i and node j in a random situation; represents the difference between the actual connection weight and the random connection weight; δ(c i ,c j ) represents a function. When node i and node j are in the same group, the function value is 1, otherwise it is 0; max{max{r},max{1-r}} represents a second artificial rule. If it is judged that node i and node j belong to the same group or not, the function value is 1, otherwise it is 0; λ represents the weight of the second artificial rule.

5. The method according to claim 1, wherein: The step of inputting the trial result into a pre-trained message generation model to obtain a trial result message output by the message generation model includes: Using the trial result, searching in a knowledge graph; the knowledge graph is generated using the second artificial expert experience; Use the retrieval results as input to the pre-trained retrieval-based algorithm model; The text output by the retrieval-based algorithm model is encoded, and the encoding result is used as the input of the pre-trained generation-based algorithm model to obtain the trial result message output word by word by the generation-based algorithm model.

6. The method according to claim 5, wherein: The retrieval-based algorithm model and the generation-based algorithm model are jointly trained, and the following third loss function is used for joint training: Where α represents the weight of the loss function corresponding to the second artificial expert experience, and α is a preset constant; L represents the length of the target generated sentence, t represents each word in the sentence, and p t Represents the probability of each word in the sentence generated by the target, and m represents the number of all words in the description text formed by the second artificial expert experience for each customer; Represents the probability of each word in the second artificial expert experience, The representation takes the value with the highest probability for all the words in the second artificial expert experience.

7. An automated review device, wherein: include: a data input module configured to obtain subject data features and input the subject data features into the adjudication model module; A review model module, configured to obtain a review result using a pre-trained review model, and send the review result to the message generation module; wherein the review model is obtained by training using the first artificial expert experience; A message generation module, configured to obtain a trial result message using a pre-trained message generation model, and report the trial result message; wherein the message generation model is obtained by training using the experience of a second artificial expert; Wherein, the trial model module includes: a single subject identification model or a multi-subject identification model; A single subject recognition model, configured to recognize a single subject; A multi-agent identification model, configured to identify groups formed by multiple agents; The training process of the single subject recognition model includes: According to the type and characteristics of a single subject, determine the recognition tasks of the single subject recognition model and the characteristics of each recognition task; Forming first artificial rules from first artificial expert experiences for identifying a single subject; adding the given weight values ​​of each first artificial rule to the loss function of the algorithm used by the single-agent recognition model to form a first loss function; and Using the first loss function to train the single subject recognition model; The training process of the multi-agent recognition model includes: Forming respective second artificial rules from respective first artificial expert experiences for identifying a plurality of subjects; and The second artificial rules for determining whether or not they meet the requirements for gang identification are added to the loss function of the algorithm used by the multi-agent identification model to form a second loss function; and the multi-agent identification model is trained using the second loss function.

8. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images