A message management method and device, apparatus, and storage medium

Abstract

The application discloses a message management method and device, equipment and a storage medium. The method comprises: performing risk control feature scanning processing on a to-be-detected text to obtain an initial screening and release text, then performing semantic fingerprint generation and cache collision verification processing to obtain a to-be-reasoned text; performing structured prompt word assembly processing on the to-be-reasoned text and business context information to obtain a standardized large model reasoning input, then performing deep semantic intention deduction and risk quantification processing to obtain a continuous risk assessment score; performing bilateral boundary threshold grading mapping processing on the continuous risk assessment score to obtain discrete audit level state information, then performing differential management strategy matching and asynchronous write-back processing to obtain a target management instruction. The application realizes the cooperation of local high-speed preliminary screening and cloud deep semantic bottom-up, greatly reduces the computing power cost and system delay, and simultaneously considers compliance supervision and user non-perceptual experience, and is suitable for high-concurrency content community, social platform and other scenes.

Description

Technical Field

[0001] This application relates to the field of message processing technology, and in particular to a message management method, apparatus, device and storage medium. Background Technology

[0002] With the rapid development of internet technology, user-generated content has experienced explosive growth, and message interaction volumes in content communities, social platforms, and instant messaging applications continue to rise. However, this massive amount of user-generated content is mixed with various illegal information, posing increasingly severe challenges to content security and compliance governance for platforms. In high-concurrency business scenarios, how to achieve accurate content review while ensuring message processing efficiency has become a core challenge for platform operations.

[0003] Current content moderation methods are mainly divided into two categories: one is explicit feature matching methods based on regular expressions or local finite state machines. Although these methods can achieve microsecond-level fast processing, they can only identify preset explicit violation features and have obvious semantic blind spots. They are difficult to identify obscure expressions, variant words, and complex violation content combined with context, which easily leads to high false positive and false negative rates. The other category is semantic moderation methods based on large language models. These methods have strong semantic understanding capabilities, but they suffer from high inference latency and high computing power costs. They are difficult to independently support full real-time filtering of massive amounts of data, and lack a sound disaster recovery mechanism and flexible control measures, which can easily lead to business flow disruptions or user experience damage.

[0004] Therefore, how to balance the processing efficiency, recognition accuracy, and user experience of content moderation, and build a message management system that combines high concurrency support and deep semantic recognition capabilities, is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the message management method, apparatus, device, and storage medium provided in this application embodiment can balance the processing efficiency, recognition accuracy, and user experience of content review, and construct a message management system with both high concurrency support capabilities and deep semantic recognition capabilities. The message management method, apparatus, device, and storage medium provided in this application embodiment are implemented as follows: This application provides a message management method, including: The text to be detected is processed by risk control feature scanning to obtain the initial screening and release text; The text that was initially screened and approved is subjected to semantic fingerprint generation and cache collision verification to obtain the text to be inferred. The text to be inferred and the business context information are processed by assembling structured prompt words to obtain standardized large model inference input; Deep semantic intent inference and risk quantification are performed on the standardized large model inference input to obtain a continuous risk assessment score; The continuous risk assessment scores are subjected to dual-boundary threshold hierarchical mapping processing to obtain discrete audit level status information; Differentiated control strategies are matched and asynchronous write-back is performed on the discrete audit level status information to obtain the target control instructions.

[0006] In some embodiments, the step of assembling structured prompt words between the text to be inferred and business context information to obtain standardized large model inference input includes: Obtain user behavior data and content attribute data associated with business systems; Context extraction processing is performed on the user behavior data and the content attribute data to obtain user historical reputation information and content section tag information; The text to be inferred, the user's historical reputation information, and the content's section tag information are subjected to multi-dimensional projection combination processing to obtain integrated text information; The integrated text information is subjected to standardized structural encapsulation processing to obtain standardized large model inference input.

[0007] In some embodiments, the step of performing deep semantic intent deduction and risk quantification on the standardized large model inference input to obtain a continuous risk assessment score includes: Deep contextual feature extraction and implicit intent inference processing are performed on the standardized large model inference input to obtain the text semantic intent recognition result; The text semantic intent recognition results are processed to distinguish between normal semantics and adversarial violations, resulting in violation risk feature data; The violation risk characteristic data is subjected to continuous probability value quantification to obtain a continuous risk assessment score.

[0008] In some embodiments, the step of performing dual-boundary threshold hierarchical mapping processing on the continuous risk assessment scores to obtain discrete audit level status information includes: Obtain the preset low-risk boundary threshold and high-risk boundary threshold; The continuous risk assessment score, the low-risk boundary threshold, and the high-risk boundary threshold are subjected to interval division and matching processing to obtain the risk interval attribution identifier; The risk interval attribution identifier is subjected to discrete state mapping processing to obtain discrete audit level state information.

[0009] In some embodiments, the step of performing differentiated control strategy matching and asynchronous write-back processing on the discrete audit level status information to obtain target control instructions includes: Obtain the preset set of differentiated control rules; The discrete audit level status information and the differentiated control rule set are subjected to routing matching processing to obtain the initial control instruction; The initial control command and the corresponding original audit data are associated and packaged to obtain a data packet to be written back; The data packets to be written back are processed asynchronously using a distributed event queue to obtain the target control instructions.

[0010] In some embodiments, the step of generating semantic fingerprints and performing cache collision verification on the initially screened and approved text to obtain the text to be inferred includes: The text that was initially screened and approved was subjected to content desensitization and metadata extraction processing to obtain text desensitization features and text metadata. The text desensitization features and the text metadata are hashed to obtain a unique semantic fingerprint; The unique semantic fingerprint is compared with the historical semantic fingerprint database to obtain cache hit status information. The cache hit status information is processed by branch determination to obtain the text to be inferred.

[0011] In some embodiments, the discrete audit level status information includes one of an absolutely safe state, a vaguely suspicious state, and a clearly harmful state.

[0012] This application provides a message management device, comprising: The processing module is used to perform risk control feature scanning on the text to be detected to obtain the text to be initially screened and released. The processing module is also used to perform semantic fingerprint generation and cache collision verification on the initially screened and released text to obtain the text to be inferred; The processing module is also used to perform structured prompt word assembly processing on the text to be inferred and business context information to obtain standardized large model inference input; The processing module is also used to perform deep semantic intent inference and risk quantification on the standardized large model inference input to obtain a continuous risk assessment score. The mapping module is used to perform dual-boundary threshold hierarchical mapping processing on the continuous risk assessment scores to obtain discrete audit level status information; The control module is used to perform differentiated control strategy matching and asynchronous write-back processing on the discrete audit level status information to obtain the target control instructions.

[0013] The computer device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the method described in this application.

[0014] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method described in this application embodiment.

[0015] The message management method, apparatus, device, and storage medium provided in this application embodiment involve scanning the text to be detected using risk control features to obtain a preliminary screening text, followed by semantic fingerprint generation and cache collision verification to obtain the text to be inferred. The text to be inferred is then combined with business context information using structured prompt words to obtain a standardized large-scale model inference input, followed by deep semantic intent deduction and risk quantification to obtain a continuous risk assessment score. This continuous risk assessment score is then subjected to dual-boundary threshold hierarchical mapping to obtain discrete review level status information, followed by differentiated management strategy matching and asynchronous write-back to obtain the target management instruction. This approach balances content review processing efficiency, recognition accuracy, and user experience, constructing a message management system with both high concurrency support and deep semantic recognition capabilities, thus solving the technical problems mentioned in the background art. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram illustrating the implementation process of a message management method provided in this application embodiment; Figure 2 A schematic diagram illustrating the implementation process of obtaining standardized large model inference input, provided in an embodiment of this application; Figure 3 This is a schematic diagram of a message management device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0020] Figure 1 This is a schematic diagram illustrating the implementation flow of a message management method provided in an embodiment of this application, including steps 101 to 106. Wherein, Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order of a message management method. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0021] This application discloses a message control method applied to the content compliance review scenario of a user-generated content system, aiming to solve the problem of balancing processing efficiency, recognition accuracy, and user experience in existing content review systems. This method achieves efficient and accurate message control in high-concurrency scenarios through a hybrid architecture combining local explicit feature initial screening with cloud-based large-scale model deep semantic fallback. The core technical models of this application include: an improved prefix multi-branch tree local risk control feature filtering model, a dual-pointer sliding explicit risk control scanning model, a semantic fingerprint distributed cache collision model, a large-scale model deep semantic intent inference and risk quantification model, a dual-boundary threshold hierarchical mapping model, and a differentiated control strategy asynchronous execution model. These models work together to form a complete risk control closed loop. The specific implementation process is as follows: Step 101: Perform risk control feature scanning on the text to be detected to obtain the initial screening and release text.

[0022] This step is implemented based on an improved prefix multi-branch tree local risk control feature filtering model and a dual-pointer sliding explicit risk control scanning model. The underlying retrieval logic of the improved prefix multi-branch tree local risk control feature filtering model is abstracted into a deterministic state transition topology quintuple mapping model:

[0023] in, Given a finite set of all possible state nodes, For the input character set, This is the state transition function. The root node state is... This is the set of termination states. The model employs a read-write separated, double-buffered memory architecture. All front-end business communication flows only hold a read-only view of the main branch prefix topology tree for lock-free concurrent scanning. When back-end risk control rules dynamically change, atomic operations are used to instantly switch the read and write pointers, enabling risk control rules to take effect in milliseconds while the business flow remains completely unaware. Simultaneously, risk level scalars and morpheme ontology data are pre-embedded and directly connected to the node memory structure of the termination state set F, supporting zero-copy-level data extraction.

[0024] In this embodiment, the system starts by synchronizing the enabled risk morphemes from the persistent cluster. The improved prefix multi-branch tree is constructed in an independent memory block according to the danger level, and the common prefixes of the massive risk control rule base are deeply compressed. When the business side generates text to be detected, it is first imported into the local risk control feature scanning module. A dual-pointer sliding window is instantiated in memory, defining the main traversal pointer i (representing the starting position of the currently scanned text, traversing from 0 to n) and the probe pointer j (initially j=i). In each round of the loop targeting the starting point i, the probe pointer j starts from the root node of the improved prefix multi-branch tree and explores the matching state deeper based on subsequent characters in the text. If the state transition fails (i.e., there is no corresponding character branch in the tree), the current round of detection ends, and the main traversal pointer i moves one position to the right. If a valid termination state is successfully entered (i.e., the current state belongs to F), the corresponding risk feature and its danger level are extracted, and its absolute offset interval vector in the original text is accurately recorded.

[0025] All hit risk features obtained from the scan are hashed and deduplicated. Let the original hit feature set be:

[0026] in, For the morpheme of the hit, For the corresponding risk scalar, This is the absolute offset of the morpheme in the original text to be checked (used to accurately record the hit position). The system uses a hash function to map the morpheme ontology to a unique digest, with an execution time complexity of O(log n). The deduplication operation is performed, and when a hash collision is encountered (i.e., the same morpheme is defined multiple times or matched repeatedly at different lexicon levels), a risk maximum coverage strategy is used to calculate the final risk scalar of the morpheme:

[0027] in, For the system's morpheme ontology The unique digest value obtained by performing a hash calculation. Represent all and The risk scalars corresponding to overlapping hit records that experience hash collisions (i.e., pointing to the same morpheme ontology) are then used to generate a standardized, unique feature set. Extracting unique feature sets Global maximum risk scalar If the maximum risk target If the high-risk feature is not matched or only the low-risk feature is matched, a preliminary screening release text is generated and the process moves to the next stage.

[0028] Step 102: Perform semantic fingerprint generation and cache collision verification on the initially screened and approved text to obtain the text to be inferred.

[0029] This step is based on the semantic fingerprint distributed cache collision model, which consists of a content desensitization unit, a metadata extraction unit, a hashing unit, and a distributed cache collision verification unit.

[0030] In this embodiment, the text initially approved undergoes content anonymization processing to remove sensitive data unrelated to content compliance review, such as user personal privacy information, device identifiers, and geographical location. Only the core semantic content of the text is retained as the text anonymization feature, thus protecting user privacy while preventing irrelevant information from interfering with the uniqueness of the semantic fingerprint. Simultaneously, the system extracts metadata related to the text, including the text's publication time, content section, content length, and publishing terminal type—features that help distinguish different content entities.

[0031] After concatenating the anonymized text content with the text metadata, an irreversible hash algorithm is used to perform a hash operation, generating a fixed-length unique string as the semantic fingerprint of the text. The generated semantic fingerprint is then compared with a historical semantic fingerprint database stored in a distributed memory cache. If the comparison is successful, the corresponding historical serialized risk assessment results and control strategies are directly reused without proceeding to the subsequent large-scale model inference stage. If the comparison fails, a text to be inferred is generated and moved to the next stage.

[0032] Step 103: Perform structured prompt word assembly processing on the text to be reasoned and the business context information to obtain standardized large model reasoning input.

[0033] This step is based on a structured prompt word assembly model, which consists of a context extraction unit, a multi-dimensional projection combination unit, and a standardized encapsulation unit.

[0034] In this embodiment, business context information associated with the publication of the text to be inferred is retrieved from the corresponding database of the business system. This includes the user's historical reputation level and the content section tag to which the text belongs. The text to be inferred and the aforementioned business context information are projected and combined according to a preset fixed logical structure to construct a standardized structured cue word matrix. The preset structure sequentially includes four parts: review task description, business scenario constraints, user context information, and the main body of the text to be detected. The review task description clearly informs the large language model that the core objective of this inference is to identify various compliance risks in the text and assess the degree of risk; the business scenario constraints clearly indicate the content section to which the text belongs and the corresponding compliance review standards; the user context information indicates the historical reputation level of the publishing user; and the main body of the text to be detected fully presents the original text content.

[0035] The combined content is standardized and encapsulated to generate standardized large-scale model inference input that conforms to the cloud-based large language model interface input specification.

[0036] Step 104: Perform deep semantic intent deduction and risk quantification on the standardized large model inference input to obtain a continuous risk assessment score.

[0037] This step is based on a large-scale deep semantic intent inference and risk quantification model. This model is built on a multi-layer attention network of a cloud-pre-trained large language model and includes a deep context feature extraction unit, an implicit intent inference unit, a normal semantic and adversarial violation differentiation unit, and a continuous probability value quantification unit.

[0038] In this embodiment, the standardized large-scale model inference input is sent to a cloud-based large-scale language model cluster. Based on its built-in multi-layer attention network mechanism, the large-scale language model extracts deep contextual features from the input content across all dimensions. This not only captures the literal meaning of individual words but also analyzes long-distance semantic dependencies, tone, and contextual logical connections within the text. Simultaneously, the large-scale language model combines business scenario constraints and user context information contained in the input to comprehensively infer the true expressive intent of the text publisher, rather than making judgments based solely on the surface text, ultimately outputting a complete text semantic intent recognition result.

[0039] Based on inferred semantic intent and pre-defined compliance review standards, the large language model accurately distinguishes between normal semantics and adversarial violations in text. For example, for words like "kill" and "bomb" appearing in game communication sections, the large language model determines them as normal terms describing game operations based on business scenario constraints; while for expressions of personal threats formed by the same words in specific contexts in general social sections, it accurately identifies them as violations. In addition, the large language model can also identify various obscure expressions, variant words, and borderline content that skirts the edge of compliance, extracting all feature information related to violations to form structured violation risk feature data.

[0040] The large language model does not simply categorize text as "compliant" or "non-compliant" in a binary way. Instead, it transforms the probability of implicit violations into a continuous risk assessment score between 0 and 1, based on the type, severity, and confidence level of the identified violation risk features. A higher score indicates a higher risk of violation. For example, text explicitly containing serious violations such as extreme incitement or hardcore pornography will receive a high score close to 1, while completely normal content will receive a low score close to 0, and borderline content will receive a score in between. The resulting continuous risk assessment score will then be passed to the next stage for tiered mapping processing.

[0041] Step 105: Perform dual-boundary threshold hierarchical mapping on the continuous risk assessment scores to obtain discrete audit level status information.

[0042] This step is based on a dual-boundary threshold hierarchical mapping model, which consists of a threshold parameter loading unit, an interval partitioning and matching unit, and a discrete state mapping unit.

[0043] In this embodiment, the system retrieves a preset low-risk boundary threshold. High-risk boundary threshold Based on dual boundary thresholds, the continuous risk assessment score R is divided into intervals for matching, and the continuous scalar is dimensionality reduced and mapped to three discrete review level state spaces: Absolutely safe state (SAFE): when This indicates that the text entity is clearly expressed, has a healthy intent, and no potential compliance risk characteristics were detected; Suspicious state (SUSPICIOUS): when When this occurs, it indicates that the content is in a gray area and may contain borderline words, excessive emotional venting, or veiled expressions that are on the edge of compliance red lines. Clearly defined harmful state: when This indicates that the text entity possesses a highly confident and substantial intent to violate the law, such as malicious fraud, extreme incitement, or hardcore pornography.

[0044] Finally, discrete audit level status information containing the above three types of states is generated and transmitted to the next stage for matching and execution of control strategies. In this embodiment, a preset low-risk boundary threshold is used in general scenarios. The threshold value is 0.3, representing a high-risk boundary. The threshold is 0.7. Different content sections can be configured with different thresholds. For example, the control standards for the current affairs discussion section are stricter, and its low-risk boundary threshold and high-risk boundary threshold will be lowered accordingly; while the control standards for the lifestyle sharing section are relatively lenient, and the threshold can be appropriately raised.

[0045] Step 106: Perform differentiated control strategy matching and asynchronous write-back processing on the discrete audit level status information to obtain the target control instructions.

[0046] This step is implemented based on the asynchronous execution model of differentiated management and control strategies. This model consists of a management and control rule loading unit, a state machine routing matching unit, a data packaging unit, and a distributed event queue asynchronous write-back unit.

[0047] In this embodiment, a preset set of differentiated control rules is obtained. This set of rules is formulated based on national content compliance regulatory requirements, platform operation specifications, and the control needs of different business segments, establishing a mapping relationship between discrete review level statuses and specific control actions. The discrete review level status information is then matched with the control rule set using routing matching. If the review level is set to absolute security, a release order will be generated, allowing the content to be published normally and distributed in full. If the review level is ambiguous or suspicious, a fine-grained secondary evaluation mechanism is activated. When the risk score is low, a restriction on dissemination or content warning instruction is generated. The content can be published, but its recommendation weight is reduced and it is only shown to some users. When the risk score approaches the high-risk boundary threshold, a shadow mute instruction is generated. This instruction returns a regular publication success status receipt to the content publisher at the front-end interaction level to maintain the normal publishing experience for users. At the same time, in the back-end data storage and traffic distribution stage, the system will silently adjust the visibility attribute of the data entity, making it visible only to the creator, thereby cutting off its dissemination link in the public information domain from the underlying physical level. If the review level is clearly harmful, a hard block or deletion order will be generated, prohibiting the publication of the content and returning a compliance prompt to the user.

[0048] The generated target control instructions are associated and packaged with the corresponding original review data. The original review data includes the original text content to be detected, semantic fingerprint, continuous risk assessment score output by the large model, discrete review level status, publishing user identifier, content publication time, and forum tag. During the packaging process, a unique review serial number is generated for each data package for subsequent audit traceability and problem investigation, ultimately forming a complete data package to be written back.

[0049] The system submits the data packets to be written back to the Celery (Celery Distributed Task Queue) distributed event queue. A background asynchronous task process is then responsible for writing the data packets back to the main business database using an eventual consistency model. This asynchronous processing mechanism allows users to receive control responses without waiting for the database write to complete, significantly reducing perceived latency. Simultaneously, the system issues initial control instructions to the business systems for execution, forming the final effective target control instructions and completing the entire message control process.

[0050] This application's embodiments construct a hybrid message management architecture that combines local risk control screening with deep backup using a cloud-based large-scale model. Through a step-by-step process design, massive amounts of routine content are processed rapidly locally, while only complex and difficult content is sent to the large-scale model for inference. This fundamentally solves the industry pain point of existing technologies that struggle to balance high-concurrency processing, low-latency response, and accurate semantic recognition. This architecture supports stable operation under high throughput, retaining the extreme efficiency of local feature matching while possessing the deep semantic understanding capabilities of a large-scale model. Furthermore, through differentiated control and asynchronous write-back mechanisms, it achieves a balance between compliance and user experience, making it suitable for high-concurrency user-generated content scenarios such as content communities and social platforms.

[0051] In the above Figure 1 Based on the above, this application embodiment also provides a schematic diagram of the implementation process for obtaining standardized large model inference input. For example... Figure 2 As shown, steps 201 to 204 are included: Step 201: Obtain user behavior data and content attribute data associated with the business system.

[0052] In this embodiment, upon receiving a text to be inferred that has not been cached, a data request is automatically initiated to the user database and content management database of the business system to retrieve all user behavior data and content attribute data directly related to the text's publication behavior. The user behavior data includes the user's registration duration, historical content review records, number of violations, and user reputation rating; the content attribute data includes the text's publication time, publication channel, content section, and content type identifier.

[0053] Step 202: Context extraction processing is performed on user behavior data and content attribute data to obtain user historical reputation information and content category tag information.

[0054] In this embodiment, the retrieved raw user behavior data is filtered and refined to extract the user's historical reputation information most relevant to content compliance review. This includes the user's current reputation level, the number of violations recorded in the past three months, and the types of historical violations. Simultaneously, the content's forum tag information is accurately extracted from the content attribute data to clarify the specific business scenario of the text's publication, such as a game discussion forum, a lifestyle sharing forum, or a current affairs discussion forum.

[0055] Step 203: Perform multi-dimensional projection combination processing on the text to be inferred, user historical reputation information, and content category tag information to obtain integrated text information.

[0056] In this embodiment, the text to be inferred, user historical reputation information, and content category tag information are sequentially concatenated and integrated according to a preset fixed logical structure. The preset logical structure includes four core parts in sequence: the first part is the review task description, which clearly informs the large language model that the core objective of this inference is to identify various compliance risks in the text and assess the degree of risk; the second part is the business scenario constraints, which annotates the content category tag and the specific compliance review standards corresponding to that category; the third part is the user context information, which annotates the publishing user's historical reputation level and related violation records; and the fourth part is the main body of the text to be detected, which fully presents the original text content to be inferred that needs to be semantically analyzed.

[0057] Step 204: Standardize the structure of the integrated text information to obtain standardized large model inference input.

[0058] In this embodiment, the combined text information undergoes format standardization processing, removing redundant whitespace characters and special symbols to ensure that the text format conforms to the input requirements of the cloud-based large language model interface. Simultaneously, a unified identifier field is added during the encapsulation process to distinguish different review task types and priorities. Ultimately, a standardized large model inference input with a well-structured and complete information is generated and sent to the cloud-based large language model cluster for subsequent semantic inference processing.

[0059] This application's embodiments provide a complete reasoning background for large language models through the structured assembly of multi-dimensional business context information and the text to be reasoned, effectively solving the semantic ambiguity and misjudgment problems caused by single text input. By introducing user historical reputation and content category tag information, it helps large models accurately distinguish between normal semantics and adversarial violations in different business scenarios. For example, it accurately distinguishes between combat terms describing game operations in the game category and similar expressions with real-world threat implications in the general social category, significantly improving the accuracy of large model reasoning and reducing false positive and false negative rates. At the same time, the standardized input format ensures the stability and consistency of large model interface calls.

[0060] In some embodiments, deep semantic intent inference and risk quantification are performed on the standardized large model inference input to obtain a continuous risk assessment score, including: deep contextual feature extraction and implicit intent inference processing are performed on the standardized large model inference input to obtain the text semantic intent recognition result.

[0061] Specifically, the generated standardized large-scale model inference input is sent to a cloud-based pre-trained large-scale language model cluster. Based on its built-in multi-layer attention network mechanism, the large-scale language model extracts deep contextual features from the input content across all dimensions. This not only captures the literal meaning of individual words but also analyzes long-distance semantic dependencies, tone, and contextual logical connections within the text. Simultaneously, the large-scale language model combines business scenario constraints and user context information contained in the input to comprehensively infer the true expressive intent of the text publisher, rather than relying solely on surface-level literal text, ultimately outputting a complete text semantic intent recognition result.

[0062] Furthermore, the text semantic intent recognition results are processed to distinguish between normal semantics and adversarial violations, resulting in violation risk feature data.

[0063] Specifically, based on the inferred semantic intent and pre-set compliance review standards, the big language model accurately distinguishes between normal semantics and adversarial violation content in text. For example, for words like "kill" and "bomb" appearing in the game communication section, the big language model will determine them as normal terms describing game operations based on business scenario constraints; while for expressions of personal threats formed by the same words in a specific context in the general social section, it will accurately identify them as violation content. In addition, the big language model can also identify various obscure expressions, variant words, and borderline content that skirts the edge of compliance red lines, extracting all feature information related to violations to form structured violation risk feature data.

[0064] Furthermore, the data on the characteristics of violation risks are quantified using continuous probability values ​​to obtain a continuous risk assessment score.

[0065] Specifically, the large language model does not simply categorize text as "compliant" or "non-compliant" in a binary way. Instead, it transforms the probability of implicit violations into a continuous numerical score based on the type, severity, and confidence level of the identified violation risk features. This score typically ranges from 0 to 1, with higher scores indicating a higher risk of violation. For example, text explicitly containing serious violations such as extreme incitement or hardcore pornography will receive a high score close to 1, while completely normal content will receive a low score close to 0, and content bordering on the gray area will receive a score in between. The resulting continuous risk assessment score will then be passed to the next stage for tiered mapping processing.

[0066] This application's embodiments, through deep contextual feature extraction and implicit intent deduction using a large model, overcome the semantic blind spot of traditional local feature matching, which can only identify explicit violations. It can effectively identify various adversarial content such as obscure expressions, variant vocabulary, and context-related violations. Simultaneously, it employs a continuous probability value-based risk quantification method, replacing simple black-and-white binary qualitative assessments. This provides a refined quantitative basis for subsequent tiered control, making risk assessment results more objective and accurate, and better adaptable to the needs of handling violations of varying severity.

[0067] In some embodiments, the continuous risk assessment score is subjected to dual-boundary threshold hierarchical mapping processing to obtain discrete audit level status information, including: obtaining preset low-risk boundary thresholds and high-risk boundary thresholds.

[0068] Specifically, upon system startup, pre-set dual-boundary threshold parameters are loaded from the risk control strategy configuration center. These threshold parameters are determined by the platform based on relevant compliance requirements, the strictness of control over different business segments, and statistical analysis results of historical audit data. Different content segments can be configured with different thresholds. For example, the control standards for the current affairs discussion segment are stricter, and its low-risk and high-risk boundary thresholds will be lowered accordingly; while the control standards for the lifestyle sharing segment are relatively lenient, and the thresholds can be appropriately raised. In this embodiment, the preset low-risk boundary threshold is 0.3 and the high-risk boundary threshold is 0.7 in a general scenario.

[0069] Furthermore, the continuous risk assessment scores, low-risk boundary thresholds, and high-risk boundary thresholds are divided and matched to obtain risk interval attribution identifiers.

[0070] Specifically, the continuous risk assessment score output by the large model is compared one by one with the two boundary thresholds mentioned above, and a corresponding risk interval assignment identifier is generated based on the numerical range of the score. When the continuous risk assessment score is less than the low-risk boundary threshold, a low-risk interval identifier is generated; when the continuous risk assessment score is greater than or equal to the low-risk boundary threshold and less than or equal to the high-risk boundary threshold, a medium-risk interval identifier is generated; when the continuous risk assessment score is greater than the high-risk boundary threshold, a high-risk interval identifier is generated.

[0071] Furthermore, the risk zone attribution identifier is subjected to discrete state mapping processing to obtain discrete audit level state information.

[0072] Specifically, a one-to-one correspondence is established between risk zone classification identifiers and review level statuses. Low-risk zones are mapped to an absolutely safe state, indicating that the text is clearly expressed, has a healthy intent, and no potential compliance risk characteristics are detected. Medium-risk zones are mapped to a vague and suspicious state, indicating that the content is in a gray area and may contain borderline vocabulary, excessive emotional expression, or veiled expressions that skirt the edge of compliance red lines. High-risk zones are mapped to a clearly harmful state, indicating that the text has a highly credible and substantial intent to violate regulations, such as malicious fraud, extreme incitement, or hardcore pornography. Finally, discrete review level status information containing these three categories is generated and transmitted to the next stage for matching and implementing control strategies.

[0073] This application's embodiments utilize a hierarchical mapping mechanism with dual boundary thresholds to transform the continuous risk assessment scores output by the large model into discrete review level states that the system can directly recognize. This achieves efficient transformation of the large model's semantic understanding results into engineered control actions. This mechanism supports flexible configuration of threshold parameters for different business scenarios and can dynamically adjust review standards based on the strictness of control in each section, improving the system's business adaptability and flexibility, and laying the foundation for the accurate execution of subsequent differentiated control strategies.

[0074] In some embodiments, the discrete audit level status information is matched with differentiated control strategies and asynchronously written back to obtain target control instructions, including: obtaining a preset set of differentiated control rules.

[0075] Specifically, a pre-defined set of differentiated control rules is loaded from the risk control strategy configuration center. This set of rules is formulated based on national content compliance regulations, platform operation specifications, and the control needs of different business segments, establishing a mapping relationship between discrete review levels and specific control actions. Different content segments can be configured with differentiated control rules. For example, the current affairs discussion segment has stronger control over ambiguous or suspicious content, while the lifestyle sharing segment can have more relaxed restrictions. The rule set explicitly includes six basic control actions and their triggering conditions: allow access, restrict dissemination, content warning, shadow ban, hard block, and deletion.

[0076] Furthermore, routing matching is performed on the discrete audit level status information and the differentiated control rule set to obtain the initial control instructions.

[0077] Specifically, the generated discrete review level status information is precisely matched with the set of control rules: if the review level is absolutely safe, a release instruction is generated, allowing the content to be published normally and distributed in full; if the review level is ambiguous and suspicious, a fine-grained matching is performed by combining continuous risk assessment scores. When the risk score is low, a restriction on dissemination or content warning instruction is generated, allowing the content to be published but reducing its recommendation weight and showing it only to some users; when the risk score approaches the high-risk boundary threshold, a shadow mute instruction is generated, which returns a regular publication success status receipt to the content publisher at the front-end interaction level, maintaining the user's normal publishing experience; if the review level is clearly harmful, a hard block or deletion instruction is generated, directly prohibiting the publication of the content and returning a compliance prompt to the user.

[0078] Furthermore, the initial control instructions and the corresponding original audit data are associated and packaged to obtain the data package to be written back.

[0079] Specifically, the generated initial control instructions are associated with and packaged one by one with the original data of the entire review process. The original review data includes the original text content to be inspected, semantic fingerprints, continuous risk assessment scores output by the large model, discrete review level status, publishing user identifiers, content publication time, and forum tags. During the packaging process, a unique review serial number is generated for each data package for subsequent audit traceability and problem investigation, ultimately forming a complete data package to be written back.

[0080] Furthermore, the data packets to be written back are processed asynchronously using a distributed event queue to obtain the target control instructions.

[0081] Specifically, the system submits the data packets to be written back to the Celery (Celery Distributed Task Queue) distributed event queue, where background asynchronous task processes are responsible for writing the data packets back to the main business database using an eventual consistency model. This asynchronous processing mechanism allows users to receive control responses without waiting for the database write to complete, significantly reducing perceived latency. Simultaneously, the system issues initial control instructions to the business systems for execution, forming the final effective target control instructions and completing the entire message control process.

[0082] This application's embodiments achieve graded processing of content with different risk levels through differentiated control strategies and routing matching. In particular, the introduction of flexible control measures such as shadow bans effectively reduces user resistance and experience damage caused by direct blocking and deletion of posts, while meeting compliance and regulatory requirements, thus reducing user churn and complaints. Simultaneously, the use of a distributed event queue asynchronous write-back architecture decouples time-consuming database write operations from the user response process, significantly reducing perceived latency and improving the system's concurrent processing capabilities and overall response speed.

[0083] In some embodiments, semantic fingerprint generation and cache collision verification are performed on the initially screened and approved text to obtain the text to be inferred, including: content desensitization and metadata extraction processing are performed on the initially screened and approved text to obtain text desensitization features and text metadata.

[0084] Specifically, after receiving the initial screening and approval text output from the local risk control feature scanning process, the system first performs content anonymization processing. This removes sensitive data unrelated to content compliance review, such as user personal privacy information, device identifiers, and geographical locations, retaining only the core semantic content of the text as the anonymized text feature. This protects user privacy while preventing irrelevant information from interfering with the uniqueness of the semantic fingerprint. Simultaneously, the system extracts metadata related to the text, including publication time, content section, content length, and publishing terminal type—features that help distinguish different content entities.

[0085] Furthermore, the text desensitization features and text metadata are hashed to obtain a unique semantic fingerprint.

[0086] Specifically, an irreversible hash algorithm is used to concatenate the obtained text anonymization features with the text metadata and then perform a hash operation to generate a fixed-length unique string as the semantic fingerprint of the text. This semantic fingerprint has strong uniqueness, meaning that texts with completely identical content will generate completely identical semantic fingerprints, while texts with different content will generate different semantic fingerprints, accurately identifying each independent text content entity.

[0087] Furthermore, a collision check is performed between the unique semantic fingerprint and the historical semantic fingerprint database to obtain cache hit status information.

[0088] Specifically, the generated unique semantic fingerprint is sent to a distributed memory cache pool for collision verification against a historical semantic fingerprint database stored in the cache pool. The distributed memory cache pool pre-stores the semantic fingerprints corresponding to all recently reviewed texts by the large model, along with their serialization risk assessment results and control strategies. If the same semantic fingerprint is found in the cache pool, a cache hit status is generated; otherwise, a cache miss status is generated.

[0089] Furthermore, the cache hit status information is processed by branch determination to obtain the text to be inferred.

[0090] Specifically, if the system determines that the text is in a cache hit state, it directly reuses the corresponding historical risk assessment results and control strategies stored in the cache pool. There is no need to send the text to the cloud-based large language model for reasoning. Instead, it jumps directly to the control strategy execution stage. If the system determines that the text is in a cache miss state, it means that the text is a new type of content that appears for the first time. The system marks the text that was initially screened and approved as text to be reasoned and moves it to the next stage for structured prompt word assembly and large model semantic reasoning processing.

[0091] This application's embodiments construct a bypass acceleration channel for large-model inference through semantic fingerprint generation and a distributed cache collision verification mechanism. Leveraging the characteristic of numerous repetitive inflammatory and pasted texts in internet communities, historical judgment results are directly reused for already reviewed content, eliminating the need for repeated calls to the large-model application programming interface. This significantly reduces computational costs and network transmission overhead, improving the overall processing efficiency of the system. Simultaneously, content anonymization protects user privacy and prevents irrelevant information from interfering with the uniqueness of semantic fingerprints, ensuring the accuracy of cache matching.

[0092] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in this embodiment or the accompanying drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0093] like Figure 3 As shown in the illustration, this application also provides a message management device 300. The device includes: The processing module 301 is used to perform risk control feature scanning processing on the text to be detected to obtain the text to be initially screened and released.

[0094] The processing module 301 is also used to generate semantic fingerprints and perform cache collision verification on the initially screened and released text to obtain the text to be inferred.

[0095] The processing module 301 is also used to perform structured prompt word assembly processing on the text to be reasoned and the business context information to obtain standardized large model reasoning input.

[0096] The processing module 301 is also used to perform deep semantic intent inference and risk quantification on the standardized large model inference input to obtain a continuous risk assessment score.

[0097] The mapping module 302 is used to perform dual-boundary threshold hierarchical mapping processing on continuous risk assessment scores to obtain discrete audit level status information.

[0098] The control module 303 is used to perform differentiated control strategy matching and asynchronous write-back processing on discrete audit level status information to obtain target control instructions.

[0099] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0100] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0101] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0102] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.

[0103] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.

[0104] Furthermore, in the various embodiments of the present invention, each functional module can be integrated into a processing module, or each module can exist independently, or two or more modules can be integrated into a single module.

[0105] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0106] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0107] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0108] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A message management method, characterized in that, include: The text to be detected is processed by risk control feature scanning to obtain the initial screening and release text; The text that was initially screened and approved is subjected to semantic fingerprint generation and cache collision verification to obtain the text to be inferred. The text to be inferred and the business context information are processed by assembling structured prompt words to obtain standardized large model inference input; Deep semantic intent inference and risk quantification are performed on the standardized large model inference input to obtain a continuous risk assessment score; The continuous risk assessment scores are subjected to dual-boundary threshold hierarchical mapping processing to obtain discrete audit level status information; Differentiated control strategies are matched and asynchronous write-back is performed on the discrete audit level status information to obtain the target control instructions.

2. The method according to claim 1, characterized in that, The process of assembling structured prompt words between the text to be reasoned and the business context information to obtain standardized large-scale model inference input includes: Obtain user behavior data and content attribute data associated with business systems; Context extraction processing is performed on the user behavior data and the content attribute data to obtain user historical reputation information and content section tag information; The text to be inferred, the user's historical reputation information, and the content's section tag information are subjected to multi-dimensional projection combination processing to obtain integrated text information; The integrated text information is subjected to standardized structural encapsulation processing to obtain standardized large model inference input.

3. The method according to claim 1, characterized in that, The process of performing deep semantic intent deduction and risk quantification on the standardized large model inference input yields a continuous risk assessment score, including: Deep contextual feature extraction and implicit intent inference processing are performed on the standardized large model inference input to obtain the text semantic intent recognition result; The text semantic intent recognition results are processed to distinguish between normal semantics and adversarial violations, resulting in violation risk feature data; The violation risk characteristic data is subjected to continuous probability value quantification to obtain a continuous risk assessment score.

4. The method according to claim 1, characterized in that, The step of performing a dual-boundary threshold hierarchical mapping process on the continuous risk assessment scores to obtain discrete review level status information includes: Obtain the preset low-risk boundary threshold and high-risk boundary threshold; The continuous risk assessment score, the low-risk boundary threshold, and the high-risk boundary threshold are subjected to interval division and matching processing to obtain the risk interval attribution identifier; The risk interval attribution identifier is subjected to discrete state mapping processing to obtain discrete audit level state information.

5. The method according to claim 1, characterized in that, The process of performing differentiated control strategy matching and asynchronous write-back processing on the discrete audit level status information to obtain target control instructions includes: Obtain the preset set of differentiated control rules; The discrete audit level status information and the differentiated control rule set are subjected to routing matching processing to obtain the initial control instruction; The initial control command and the corresponding original audit data are associated and packaged to obtain a data packet to be written back; The data packets to be written back are processed asynchronously using a distributed event queue to obtain the target control instructions.

6. The method according to claim 1, characterized in that, The process of generating semantic fingerprints and performing cache collision verification on the initially screened and approved text to obtain the text to be inferred includes: The text that was initially screened and approved was subjected to content desensitization and metadata extraction processing to obtain text desensitization features and text metadata. The text desensitization features and the text metadata are hashed to obtain a unique semantic fingerprint; The unique semantic fingerprint is compared with the historical semantic fingerprint database to obtain cache hit status information. The cache hit status information is processed by branch determination to obtain the text to be inferred.

7. The method according to claim 1, characterized in that, The discrete audit level status information includes one of the following: absolutely safe, ambiguous suspicious, and clearly harmful.

8. A message control device, characterized in that, include: The processing module is used to perform risk control feature scanning on the text to be detected to obtain the text to be initially screened and released. The processing module is also used to perform semantic fingerprint generation and cache collision verification on the initially screened and released text to obtain the text to be inferred; The processing module is also used to perform structured prompt word assembly processing on the text to be reasoned and business context information to obtain standardized large model reasoning input; The processing module is also used to perform deep semantic intent inference and risk quantification on the standardized large model inference input to obtain a continuous risk assessment score. The mapping module is used to perform dual-boundary threshold hierarchical mapping processing on the continuous risk assessment scores to obtain discrete audit level status information; The control module is used to perform differentiated control strategy matching and asynchronous write-back processing on the discrete audit level status information to obtain the target control instructions.

9. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images