Method and device for identifying risk type of social session

Abstract

The invention belongs to the technical field of user management, and provides a risk type identification method and device for a social session, and the method comprises the steps: obtaining to-be-detected information which comprises session information and related information; according to the to-be-detected information, obtaining a first risk type label and an initial confidence coefficient thereof; if the initial confidence coefficient is not smaller than a first threshold value, obtaining a cue word character string according to the first risk type label; inputting the cue word character string into a large language model to obtain a first confidence coefficient of the first risk type tag; and if the first confidence coefficient is not less than a second threshold value, determining that the to-be-detected information is a risk type corresponding to the first risk type label.

Description

Technical Field

[0001] The present disclosure belongs to the field of user management technology, and in particular relates to a method and device for identifying risk types of social conversations. Background Art

[0002] In the daily operations of social platforms, identifying and controlling risky content is crucial for maintaining a healthy platform ecosystem. Traditional risk identification methods rely primarily on rule engines such as keyword matching and regular expressions. While these methods can quickly identify some obvious violations, they have numerous limitations when faced with the complex and ever-changing nature of social conversations. For example, rule engines struggle to capture semantically sensitive violations and are unable to address content that bypasses keyword detection through cryptic expressions or variations. Furthermore, rule engines are expensive to update and maintain, requiring constant manual addition and adjustment of keywords and regular expressions to address evolving violations.

[0003] With the development of artificial intelligence (AI) technology, large language models have demonstrated powerful capabilities in natural language processing. However, directly applying general-purpose large language models to identifying risky social conversations presents challenges. While these general-purpose large language models possess a certain level of semantic understanding, they lack adaptability in specific social contexts and are ineffective at identifying patterns of violations and language specific to certain social domains. Furthermore, large language models consume significant computing resources, making direct real-time monitoring of every social conversation difficult to implement on highly concurrent social platforms. Furthermore, these models are slow and cannot meet real-time requirements. Summary of the Invention

[0004] The present disclosure provides a method and device for identifying risk types of social conversations, which can effectively solve the above problems.

[0005] The present disclosure is achieved as follows: In a first aspect, the present disclosure provides a method for identifying risk types of social conversations, the method comprising: Obtaining information to be detected, where the information to be detected includes session information and related information thereof, wherein the related information includes at least one of user profile information, user behavior characteristics, historical processing records, and the first N messages of the session information in the same session, where N is an integer not less than 1; Obtaining a first risk type label and its initial confidence level based on the information to be detected; If the initial confidence level is not less than a first threshold, obtaining a prompt word string according to the first risk type label, wherein the prompt word string is obtained by filling in a prompt word template corresponding to the first risk type label according to the information to be detected and related information; Inputting the prompt word character string into a large language model to obtain a first confidence level of the first risk type label; If the first confidence level is not less than a second threshold, it is determined that the information to be detected is of the risk type corresponding to the first risk type label.

[0006] In a second aspect, the present disclosure provides a device for identifying risk types of social conversations, the device comprising: An information acquisition module is configured to acquire information to be detected, wherein the information to be detected includes session information and related information thereof, wherein the related information includes at least one of user profile information, user behavior characteristics, historical processing records, and the first N messages of the session information in the same session, where N is an integer not less than 1; A first confidence acquisition module, configured to obtain a first risk type label and its initial confidence based on the information to be detected; a template acquisition module, configured to obtain a prompt word string based on the first risk type label if the initial confidence level is not less than a first threshold, wherein the prompt word string is obtained by filling in a prompt word template corresponding to the first risk type label based on the information to be detected and related information; a second confidence acquisition module, configured to input the prompt word character string into a large language model to obtain a first confidence of the first risk type label; A determination module is configured to determine, if the first confidence level is not less than a second threshold, that the information to be detected is of the risk type corresponding to the first risk type label.

[0007] In a third aspect, the present disclosure provides an electronic device, comprising: a memory storing execution instructions; and A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the method described in the first aspect.

[0008] In a fourth aspect, the present disclosure provides a readable storage medium, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0009] Compared with the prior art, the present invention has the following advantages: The present disclosure provides a method for identifying risk types in social conversations. Based on conversation information, it also integrates multi-dimensional information such as user portraits, behavioral characteristics, historical handling records, and previous information in the same conversation to predict risk types and conduct secondary identification using a large language model. Compared with using only a single dimension of conversation information, it effectively reduces the probability of misjudgment and adopts a multi-level initial confidence judgment mechanism to achieve the rational use of large language model resources. It first quickly filters out the confidence level of risk types at a low computing cost. For risk types with higher confidence, the model is guided to focus on this risk type for secondary identification with the help of prompt word templates, which significantly reduces computing power consumption and latency, while solving the problem of insufficient performance of general models in vertical fields. For low-confidence scenarios, the large language model reclassifies the scenarios in the complete context without risk labels.

[0010] This method introduces a dynamic prompt word template mechanism that adaptively adjusts prompt word strings based on different risk types, further enhancing the effectiveness and adaptability of large language models in social risk identification. The prompt word template incorporates multi-dimensional information and can specifically guide the large language model to identify specific risk types, thereby improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 4 is a flowchart of a method S100 for identifying risk types of social conversations provided by an embodiment of the present disclosure.

[0012] Figure 2 2 is a schematic structural diagram of a device 1000 for identifying risk types of social conversations provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0014] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0015] Unless otherwise stated, the exemplary embodiments / examples shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of the various embodiments / examples may be further combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0016] The terms used in this article are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0017] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0018] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0019] The terms "first" and "second" herein are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of the objects can be interchanged where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0020] Example 1 Please refer to Figure 1 , an embodiment of the present disclosure provides a method S100 for identifying risk types of social conversations.

[0021] Specifically, the method S100 includes: S102: Acquire information to be detected, where the information to be detected includes session information and related information, where the related information includes at least one of user profile information, user behavior characteristics, historical processing records, and the first N messages of the session information in the same session, where N is an integer not less than 1; S104, obtaining a first risk type label and its initial confidence level based on the information to be detected; S106: If the initial confidence level is not less than a first threshold, obtaining a prompt word string based on the first risk type label, wherein the prompt word string is obtained by filling in a prompt word template corresponding to the first risk type label based on the information to be detected and related information; S108: Input the prompt word character string into a large language model to obtain a first confidence level of the first risk type label; S110: If the first confidence level is not less than a second threshold, determine that the information to be detected is of the risk type corresponding to the first risk type label.

[0022] In step S102 , raw data from the social platform is received and pre-processed to generate a standardized information data packet in a unified format, i.e., information to be detected.

[0023] In some embodiments, the standardized information packet includes the following fields: Message unique identifier (msg_id); Send user unique identifier (user_id); Session information (content), including: Cleaned text (text); Image link array (img_url); Video link (video); timestamp; Relevant information, including: The context of the conversation, including: Session unique identifier (session_id); An array of the first N previous messages sorted by time (prev_msgs); User profile object (user_profile), which includes: Registration days (reg_days); gender; Historical violations (hist_violations); Device unique identifier (device_id); User behavior feature time series array (behavior_seq), which records the actions and timestamps within the most recent preset time window; Historical disposition record object (disposition), including: The last warning time (last_warn); The time of the last ban (last_ban).

[0024] The information to be tested uses a unified data protocol. For example, this protocol uses JSON Schema as its formal description. The original JSON pushed by the upstream business system or message queue must conform to the fields and format defined by this schema. This allows subsequent rule engines, vector search, and large modeling processes to read only the same table and the same fields, achieving "unified protocol, unified preprocessing."

[0025] Preprocessing actions can include: Fill in the fields. For example, if a field is missing, fill in the default value; if img_url is missing, fill in [].

[0026] Clean dirty data, for example, removing emoji control characters and decoding URLs.

[0027] Context concatenation. Concatenate prev_msgs into context_str in chronological order.

[0028] Fill in the user profile. Use the user_id to query the offline table to complete the profile.

[0029] Behavioral characteristic snapshots, such as message frequency and image ratio in the last minute.

[0030] In addition to conversational data, the information to be detected incorporates multiple dimensions, including user profiles, behavioral characteristics, historical handling records, and previous context within the same conversation. This information is used by the rule engine / lightweight model to predict risk types, populate prompt word templates, and complete semantic-level risk assessments in the secondary recognition phase using the large language model. This multi-dimensional information fusion mechanism enables risk assessments in a richer context, avoiding the potential misjudgments that can result from relying solely on conversational content. Simultaneously injecting multi-dimensional information into the decision-making process enables multi-dimensional risk control, significantly reducing the rate of misjudgments.

[0031] In step S104, specifically, rule engine determination and / or lightweight model determination are performed according to the information to be detected to obtain a first risk type label and its initial confidence.

[0032] In some embodiments, obtaining a first risk type label and its initial confidence level based on the information to be detected includes: Performing rule matching on the session information to obtain a first candidate risk label and a first initial confidence score thereof, wherein the rule matching includes at least one of keyword matching and regular expression matching; and / or, inputting the information to be detected into a lightweight model, and outputting a second candidate risk label and a second initial confidence score thereof; The first risk type label and the initial confidence thereof are determined according to a higher value of the first initial confidence and / or the second initial confidence.

[0033] In some embodiments, further, executing the rule engine determination includes: executing keyword matching and regular expression matching in parallel or sequentially.

[0034] The keyword matching sub-step is used to compare the text in the session information with the preset keyword hash table. If any keyword is matched, the corresponding keyword candidate tag is output. The regular expression matching sub-step is used to input the text in the session information into multiple preset regular expressions in parallel for pattern matching. If any regular expression is matched, the corresponding regular candidate tag is output; For example, run the keyword first (hash table O(1)), and then run the regularization (DFA / NFA) if the keyword is not hit to reduce the regularization overhead.

[0035] For example, keywords and regular expressions are matched independently, and then the results are integrated.

[0036] Keyword matching and regular expression matching first output risk type labels. After receiving the labels, the rule engine then maps the labels into fixed confidence constants according to business rules.

[0037] If multiple keywords and / or regular expressions are hit, the maximum confidence constant and its corresponding risk type label are taken.

[0038] Executing the lightweight model judgment includes: converting the information to be detected into a numerical feature vector and then inputting the vector into the lightweight model.

[0039] Specifically, the conversion method of the numerical feature vector includes: Text features: Segment the cleaned text, remove stop words, and generate TF-IDF vectors or n-gram vectors.

[0040] Profile features: Discrete fields (gender, device type) in user profiles are one-hot encoded, and continuous fields (registration days, number of historical violations) are standardized.

[0041] Behavioral characteristics: Count the message frequency, image ratio, and nighttime activity within the recent time window to obtain a numerical vector.

[0042] Contextual features: Perform average pooling or maximum pooling on the previous conversation to obtain a contextual embedding vector.

[0043] Concatenate all features into a unified numerical feature vector.

[0044] Specifically, the lightweight model is LightGBM or 1-2 layer TextCNN.

[0045] The lightweight model outputs risk type labels and their corresponding confidence levels.

[0046] In some embodiments, the information to be detected is divided into social field feature information and non-social field feature information according to the information to be detected; If the information to be detected is characteristic information in the social field, the large language model uses a large language model fine-tuned with social scene data.

[0047] Social domain feature information is a high-level model abstracted from business knowledge of social scenarios. "Hitting social domain features" means the system identifies a conversation as a high-risk, common social scenario. It then tags it with "social domain feature information" and forcibly routes subsequent recognition tasks to a large language model fine-tuned with social scenario data, rather than relying on a general-purpose large language model. This ensures precise risk triage and optimal resource utilization.

[0048] For example, feature information in the social field is determined based on a combination of user portrait, behavior sequence, and conversation context (user portrait × behavior × context, such as "registered for less than 24 hours + active in the early morning + frequently sending private messages to unfamiliar accounts").

[0049] In some implementations, specifically, a rule tree is used to determine whether the information to be detected matches the social field feature information.

[0050] For example, the rules engine reads standardized information packets.

[0051] Match node conditions in the rule tree, further, in order of priority: Node 1: Registration days < 3, Nighttime active flag = 1, and recent violation count ≥ 1; Node 2: The text contains a hit in the social lingo vocabulary and the message frequency is greater than the threshold; Node 3: The device fingerprint appears for the first time and the last warning time is less than 72 hours.

[0052] If any node is hit, the Boolean value "hit=True" is output, which is marked as "social domain feature information"; otherwise, "hit=False" is output, which is marked as "non-social domain feature information".

[0053] Example: It is discovered that the social jargon "LPP" is used by illegal industries to divert traffic, so it is added to the "social jargon word list" for search.

[0054] Keywords and social jargon are complementary, not substitutes. Keywords focus on "efficient identification," enabling rapid and accurate identification of clearly illegal content. Social jargon emphasizes "deep identification." In scenarios where keywords are easily circumvented or frequently mutated, a multi-dimensional combination strategy, such as "social jargon features + user profiles + behavioral patterns," can effectively identify potentially risky content and new variations, improving overall identification coverage and accuracy.

[0055] In some embodiments, the same or another lightweight model is used, either alone or in combination with a rule tree, to identify whether the information being tested contains social domain feature information. The numerical feature vector is input into a lightweight binary social domain classification model, which outputs a probability p. For example, if p ≥ 0.7, the information is labeled "social domain feature information."

[0056] Large language models fine-tuned with social scenario data, such as LLaMA2, are fine-tuned using multimodal feature fusion using accumulated social domain samples. For example, multi-dimensional information from standardized information packets of social domain samples is combined and embedded into feature vectors, which are then fine-tuned.

[0057] The choice between fine-tuned and general models considers a trade-off between resources, false positives, and coverage. Fine-tuned models are characterized by large parameter counts, high GPU memory usage, and low query-per-second (QPS). A full call to the model will quickly exhaust GPU resources. Fine-tuned models employ a "better false positives than misses" strategy for social domain feature information, ensuring that social lingo and its variations are fully captured. However, they are prone to misclassifying normal casual chat as violations. They outperform general models only in identifying risky content that matches social domain features. For other scenarios, the general model is sufficient, and switching to the full model is unnecessary to conserve computing power and reduce false positives.

[0058] In step S106, in some embodiments, if the initial confidence level is less than a first threshold, the information to be detected is input into a large language model to obtain a second risk type label and a second confidence level thereof; If the second confidence level is not less than a second threshold, it is determined that the information to be detected is of the risk type corresponding to the second risk type label.

[0059] If the initial confidence level of the predicted risk type label is low, the large language model enters the fallback process with a non-risk type label. This prevents the low-confidence label from misjudging the risk type, a phenomenon known as "anchoring bias." Using the corresponding prompt word template for this label will mislead the large language model, creating a cascading error. The large language model then reclassifies the decision semantically based on the complete context of the information being tested, and outputs the final risk type and confidence level.

[0060] For new variants or obscure language, the model can more freely explore deep semantics without prompts, significantly improving the recall rate of unknown risk types.

[0061] Specifically, the complete context of the information to be detected (including conversation text, user portrait, behavioral characteristics, historical processing records and previous messages, etc.) is spliced ​​into an unlabeled prompt string.

[0062] The unlabeled prompt string only needs to concatenate the "full context + general instructions" and does not include risk labels, additional placeholders, or the lengthy fields corresponding to the placeholders. As a result, the number of input tokens is reduced by an average of 25%–40%, and graphics memory usage and inference time are also reduced.

[0063] The label-free backup process can batch-bundle multiple low-confidence to-be-detected messages and send them to the large language model for one-time inference. Compared with label-based calls one by one, the GPU utilization rate is higher and the average computing power consumption per unit message is further reduced.

[0064] In step S106, in some embodiments, if the initial confidence level is not less than a third threshold, the information to be detected is determined to be of the risk type corresponding to the first risk type label, wherein the third threshold is greater than the first threshold.

[0065] Divide the higher confidence interval into high confidence and medium confidence.

[0066] In high-confidence scenarios, the prediction results are directly used without calling the large language model, thus saving computing resources and reducing latency.

[0067] For example, extremely high confidence: C0 ≥ 0.9, where C0 represents the initial confidence, meaning the third threshold is 0.9.

[0068] Correspondingly, if the initial confidence level is less than the third threshold and not less than the first threshold, a prompt word character string is obtained according to the first risk type label.

[0069] For example, medium confidence: 0.6≤C0<0.9, that is, the first threshold is 0.6.

[0070] In the pre-built "risk type label - prompt word template" mapping table, the corresponding prompt word template is queried and loaded in real time using the risk type label as the index. The prompt word template is filled with text, user profile fields, behavioral characteristics, and other fields from the standardized information data package according to established placeholder rules to generate the final prompt string that can be used by the large model.

[0071] The prompt word templates are designed differently according to the risk type, and the core differences are reflected in the three aspects of "placeholder set, instruction focus, and output format".

[0072] The following example illustrates this: 1. Placeholder collection: Recruitment scam template: {content} {hist_violations} {msg_freq} {device_id}; Violence Threat Template: {content} only.

[0073] 2. Command focus: Recruitment fraud: "Identify false high salaries, deposits, or pyramid schemes"; Violent intimidation: "Detecting physical threats or geographical attacks."

[0074] 3. Output Format Recruitment scam: return {label, amount, link_list}; Violence threat: returns {label, threat_level}.

[0075] Where label represents the final risk type label, such as "recruitment fraud." amount represents the specific amount (in numeric form) if money is involved. link_list represents extracting all external links / WeChat IDs / QR codes appearing in the text or image, in array format, such as ["vx123456", "https: / / t.cn / xxx"].

[0076] All templates share the same JSON Schema and can be loaded and populated in milliseconds at runtime by indexing by risk type label.

[0077] Furthermore, a dynamic prompt word management engine is designed to build an extensible risk type-prompt word template mapping library. By selecting the corresponding prompt word template in the risk type, different violation risk types can be identified in a targeted manner, such as recruitment fraud prompt words.

[0078] In some embodiments, the prompt word template is evaluated using a preset case library to obtain an evaluation result; The prompt word template is iteratively optimized according to the evaluation result.

[0079] Based on the specified case, adjust and optimize the content of the prompt word template, and verify the recognition performance of each case by calling the large language model in real time. Design a case library, sample or fully verify the satisfactory adjustment of the prompt word template version, and evaluate the global recognition performance of the large language model based on this version, including accuracy, precision, recall, F1, etc. For example, if the accuracy of the original prompt word template is 90%, and the accuracy of the modified prompt word template drops to 50% when verified using the case library, it indicates that the content of the prompt word template, such as the template placeholder or instruction text, has been modified incorrectly. Continue to optimize until it meets the expectations.

[0080] Provides versioned hosting of prompt words, supports grayscale release, traffic splitting, and rollback of abnormal versions in seconds.

[0081] Design a closed-loop data quality inspection system to dynamically initiate sampling quality inspection (lightweight) or full-link quality inspection (deep mode) based on risk level / business scenario. Generate feedback datasets based on misjudgment examples in the quality inspection results, reversely optimize the large language model, and iteratively optimize the content of prompt word templates.

[0082] In step S106, in some embodiments, the information to be detected is divided into key information and non-key information according to the relevant information; The first threshold used by the key information is smaller than the first threshold used by the non-key information.

[0083] Key information corresponds to information sent to key populations for testing. Targeted monitoring of key populations prevents missed risks and avoids inefficient testing across the entire population, achieving accurate and efficient risk identification.

[0084] For example, we can classify key groups based on user profiles. The offline profile task writes users whose profiles meet the criteria of "registration days ≤ 3 and historical violation count ≥ 1" into a Redis collection. When the real-time risk control gateway receives the information to be detected, it only needs an O(1) query to mark it as key information.

[0085] For another example, the trigger field may also be device fingerprint, real-name status, etc.

[0086] For example, key groups are divided according to historical disposal records.

[0087] For key information, a smaller first threshold (for example, 0.5) is used, making it easier to enter the prompt word template-large language model link, focusing on verifying the predicted risk type and preventing missed risks. For non-key information, at the same confidence level, it may enter the label-free recognition link, saving computing power and reducing the false positive rate.

[0088] In step S108, the dynamic prompt word template uses the three-part instruction of "task-feature-format" to accurately focus the attention of the large language model on the current risk type, improve the generalization ability of risk discovery, and avoid missed judgments caused by illegal industrial confrontation.

[0089] For example, the first sentence of the template directly states the risk type and detection target: "Please determine whether the following text involves recruitment fraud, considering only false high salaries, deposit charges, or pyramid schemes." Inject highly relevant dimensions by placeholder.

[0090] Force the output of JSON: {label, amount, link_list} to avoid divergent model descriptions and ensure that downstream can directly parse it.

[0091] By "declaring tasks + injecting key features + limiting output", the template allows large models to achieve precise focus within 1-2 sentences, reducing irrelevant reasoning and improving recall and precision.

[0092] The method S100 implements adaptive risk control using a combination of dynamic prompts, dynamic thresholds, and dynamic model invocation within the same framework. This collaborative mechanism enables precise and efficient risk control while optimizing computing resource utilization.

[0093] In step S110 , the second threshold may be equal to the first threshold, or other values.

[0094] In some implementations, if the first confidence level / the second confidence level is less than a second threshold, it is determined that the information to be detected does not violate the rules.

[0095] In some embodiments, if the first confidence level / second confidence level is relatively ambiguous (for example, corresponding to a medium confidence interval of the initial confidence level), manual review may be introduced.

[0096] In some embodiments, the method S100 further includes: The detected information is handled based on its risk type. For example, if the detected information is determined to be a threat of violence, the corresponding user will be warned; if the detected information is determined to be an advertising diversion risk, the corresponding user will be demoted.

[0097] Example 2 The embodiment of the present disclosure provides a device 1000 for identifying risk types of social conversations.

[0098] The identification device 1000 may include corresponding modules for executing each or several steps in the flowchart of the above-mentioned method S100. Therefore, each step or several steps in the above-mentioned flowchart may be executed by a corresponding module, and the identification device 1000 may include one or more of these modules. The modules may be one or more hardware modules specifically configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for execution by the processor, or implemented by some combination thereof.

[0099] Specifically, such as Figure 2 As shown, the identification device 1000 includes: An information acquisition module 1002 is configured to acquire information to be detected, wherein the information to be detected includes session information and related information thereof, wherein the related information includes at least one of user profile information, user behavior characteristics, historical processing records, and the first N messages of the session information in the same session, where N is an integer not less than 1; A first confidence acquisition module 1004 is configured to obtain a first risk type label and its initial confidence based on the information to be detected; A template acquisition module 1006 is configured to obtain a prompt word string based on the first risk type label if the initial confidence level is not less than a first threshold, wherein the prompt word string is obtained by filling in a prompt word template corresponding to the first risk type label based on the information to be detected and its related information; A second confidence obtaining module 1008 is configured to input the prompt word character string into a large language model to obtain a first confidence of the first risk type label; The determination module 1010 is configured to determine that the information to be detected is of the risk type corresponding to the first risk type label if the first confidence level is not less than a second threshold.

[0100] An embodiment of the present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module executes the above-mentioned risk type identification method for social conversations.

[0101] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned risk type identification method for social conversations.

[0102] The hardware structure of the identification device 1000 implemented by the hardware implementation of the processor disclosed herein can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0103] Bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, the figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0104] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0105] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0106] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use with or in conjunction with an instruction execution system, device, or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a memory.

[0107] It should be understood that various parts of the present disclosure can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0108] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0109] Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0110] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A method for identifying risk types of social conversations, characterized in that: The method comprises: Obtaining information to be detected, where the information to be detected includes session information and related information thereof, wherein the related information includes at least one of user profile information, user behavior characteristics, historical processing records, and the first N messages of the session information in the same session, where N is an integer not less than 1; Obtaining a first risk type label and its initial confidence level based on the information to be detected; If the initial confidence level is not less than a first threshold, obtaining a prompt word string according to the first risk type label, wherein the prompt word string is obtained by filling in a prompt word template corresponding to the first risk type label according to the information to be detected and related information; Inputting the prompt word character string into a large language model to obtain a first confidence level of the first risk type label; If the first confidence level is not less than a second threshold, it is determined that the information to be detected is of the risk type corresponding to the first risk type label.

2. The method according to claim 1, wherein If the initial confidence level is less than a first threshold, inputting the information to be detected into a large language model to obtain a second risk type label and a second confidence level thereof; If the second confidence level is not less than a second threshold, it is determined that the information to be detected is of the risk type corresponding to the second risk type label.

3. The method according to claim 1, wherein Evaluate the prompt word template using a preset case library to obtain an evaluation result; The prompt word template is iteratively optimized according to the evaluation result.

4. The method according to claim 1, wherein Based on the information to be detected, a first risk type label and its initial confidence level are obtained, including: Performing rule matching on the session information to obtain a first candidate risk label and a first initial confidence score thereof, wherein the rule matching includes at least one of keyword matching and regular expression matching; and / or, inputting the information to be detected into a lightweight model, and outputting a second candidate risk label and a second initial confidence score thereof; The first risk type label and the initial confidence thereof are determined according to a higher value of the first initial confidence and / or the second initial confidence.

5. The method according to claim 1, wherein According to the information to be detected, the information to be detected is divided into key information and non-key information; The first threshold used by the key information is smaller than the first threshold used by the non-key information.

6. The method according to claim 1, wherein According to the information to be detected, the information to be detected is divided into social field feature information and non-social field feature information; If the information to be detected is characteristic information in the social field, the large language model uses a large language model fine-tuned with social scene data.

7. The method according to claim 1, wherein If the initial confidence level is not less than a third threshold, it is determined that the information to be detected is of the risk type corresponding to the first risk type label, wherein the third threshold is greater than the first threshold.

8. A device for identifying risk types of social conversations, characterized in that: The device comprises: An information acquisition module is configured to acquire information to be detected, wherein the information to be detected includes session information and related information thereof, wherein the related information includes at least one of user profile information, user behavior characteristics, historical processing records, and the first N messages of the session information in the same session, where N is an integer not less than 1; A first confidence acquisition module, configured to obtain a first risk type label and its initial confidence based on the information to be detected; a template acquisition module, configured to obtain a prompt word string based on the first risk type label if the initial confidence level is not less than a first threshold, wherein the prompt word string is obtained by filling in a prompt word template corresponding to the first risk type label based on the information to be detected and related information; a second confidence acquisition module, configured to input the prompt word character string into a large language model to obtain a first confidence of the first risk type label; A determination module is configured to determine, if the first confidence level is not less than a second threshold, that the information to be detected is of the risk type corresponding to the first risk type label.

9. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images