Data processing method, model training method, system, device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method, a model training method, system and device, equipment, a storage medium and a program product. According to the scheme provided by the embodiment of the invention, after the input data (including the input text and / or the input image) is determined according to the target modal data to be subjected to risk detection, the risk detection of the input data is realized by utilizing the risk detection model, so that the corresponding risk detection result is obtained. Specifically, when it is detected that the input data has a risk, the risk detection result includes risk content extracted from the input data. The risk content comprises at least one of the following items: a risk word extracted from the input text and a risk area marked in the input image. The risk word comprises an adversarial variant word. The adversarial variant words are variant expressions which are formed by transforming known risk words and have unchanged semantics.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202610187093.6 (22) Application Date 2026.02.09 (71) Applicant Hangzhou Ant Love Technology Co., Ltd. Address 310012, Building 5, Huanglong International Center, Xihu District, Hangzhou, Zhejiang Province (72) Inventors Kong Jiyinjie Wang Wei Qin Xingbin Liu Jingxia Zheng Kaiyuan (74) Patent Agency Beijing Taihe Jiusi Intellectual Property Agency Co., Ltd. 11610 Patent Attorney Chai Yanbo Liu Ge (51) Int.Cl. G06F 18 / 2431 (2023.01) G06F 18 / 22 (2023.01) G06F 40 / 30 (2020.01) G06N 3 / 045 (2023.01) G06N 3 / 0475(2023.01) G06N 3 / 094(2023.01) G06N 5 / 022(2023.01) (54) Invention Title: Data Processing Method, Model Training Method, System, Apparatus, Device, Storage Medium, and Program Product (57) Abstract: This description provides a data processing method, model training method, system, apparatus, device, storage medium, and program product. The solution provided in the embodiment, after determining the input data (including input text and / or input image) based on the target modality data to be detected for risk, utilizes a risk detection model to perform risk detection on the input data, thereby obtaining the corresponding risk detection result. Specifically, when a risk is detected in the input data, the risk detection result includes risk content extracted from the input data. This risk content includes at least one of the following: risk words extracted from the input text, and risk areas marked in the input image. The risk words include adversarial variant words. Adversarial variant words are variant expressions formed by modifying known risk words without semantic change. Claims 2 pages, Description 18 pages, Drawings 4 pages, CN 121682541 A 2026.03.17 CN 1 21 68 25 41 A 1. A data processing method, characterized in that it includes: acquiring target modality data to be risk detected; determining input data based on the target modality data; the input data including input text and / or input image; performing risk detection on the input data using the risk detection model to obtain a risk detection result; wherein, when a risk is detected in the input data, the risk detection result includes risk content extracted from the input data; the risk content includes at least one of the following: risk words extracted from the input text, risk areas marked in the input image; the risk words include adversarial variant words, the adversarial variants...1. The word is a variant expression whose semantics remain unchanged, formed by modifying a known risk word. 2. The method according to claim 1, wherein the adversarial variant word includes at least one of the following: a variant word formed by inserting an interference symbol into a known risk word; a homophone generated based on a known risk word; a variant word formed by replacing a known risk word with a homophone, a similar-looking word, or a phonetic spelling. 3. The method according to claim 1, wherein the risk word further includes: a connecting combination word, wherein the connecting combination word contains at least two semantically related words and a connecting symbol is inserted between adjacent words. 4. The method according to any one of claims 1 to 3, further comprising: replaying and verifying the risk words contained in the risk detection result based on historical text to filter out risk words that are neutral words; and / or, during the risk word extraction process of the input text, the risk detection model filters candidate risk words based on a preset neutral word filtering rule to exclude candidate risk words that are neutral words from the output risk detection result. 5. The method according to claim 4, characterized in that, using the risk detection model to perform risk detection on the input data to obtain the risk detection result, includes: generating risk detection prompts based on the input data; inputting the risk detection prompts into the risk detection model, and having the risk detection model perform risk detection based on the risk detection prompts, and outputting the risk detection result; wherein, when the input data includes input text, the risk detection prompts contain text preprocessing results, the text preprocessing results including the cleaned input text, the word segmentation results of the input text, and the semantic analysis results, the semantic analysis results including identified named entities, expressive intent, and sentiment tendency; or, the risk detection prompts include the text preprocessing results and the neutral word filtering rules. 6. The method according to any one of claims 1 to 3, characterized in that, the risk detection result further includes risk classification labels; and, the method further includes: storing the risk words in a risk word library according to the risk classification labels; wherein, the risk word library adopts a key-value pair storage structure, where the key represents a risk classification and the value represents a risk word included under the corresponding risk classification. 7. The method according to claim 5, characterized in that it further comprises: determining an appropriate handling strategy for the target modal data based on the risk classification label or the risk level corresponding to the input data; wherein the risk level is obtained by semantic analysis of the input data; and performing corresponding control actions on the target modal data based on the handling strategy.8. The method according to any one of claims 1 to 3, characterized in that the target modal data is any one of the following: text, audio, image, image-text pair; and, determining input data based on the target modal data includes: when the target modal data is text, using the target modal data as the input text; when the target modal data is audio, performing speech recognition on the audio to obtain the input text; when the target modal data is an image, performing text extraction on the image, using the extracted text as the input text, and using the image as the input image; when the target modal data is an image-text pair, performing text extraction on the image in the image-text pair, using the extracted text and the text associated with the image in the image-text pair as the input text; and using the image in the image-text pair as the input image; when the target modal data is video, performing speech recognition on the audio in the video, using the text obtained from the speech recognition and the text embedded in the video as the input text; and extracting key image frames from the video, using the extracted image frames as the input image. 9. A model training method, characterized in that it includes: acquiring a training sample set; fine-tuning a machine learning model based on the training sample set to obtain a risk detection model; wherein the training sample set includes multiple training samples, the training samples include sample data and annotation information of the sample data; the annotation information includes risk content annotated in the sample data and risk classification labels corresponding to the sample data; and the risk detection model is used to perform risk detection on input data and output risk detection results; the input data includes input text and / or input images; when a risk is detected in the input data, the risk detection results include at least one of the following: risk words extracted from the input text, risk areas annotated in the input image; the risk words include adversarial variant risk words; the adversarial variant risk words refer to variant expressions formed by modifying known risk words without semantic change. 10. The method according to claim 9, wherein the plurality of training samples includes adversarial training samples; and, obtaining the adversarial training samples includes: generating adversarial sample data containing adversarial variant risk words using a generative adversarial network; and labeling the adversarial sample data to obtain labeling information of the adversarial sample data. Claims 2 / 2 Page 3 CN 121682541 A Data processing method, model training method, system, apparatus, device, storage medium, and program product Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to a data processing method, model training method, system, apparatus, device, storage medium, and program product. Background Art

[0002] With the explosive growth of AIGC (AIGenerated Content) and UGC (User Generated Content), the content security risks faced by online platforms are becoming increasingly complex and diverse. Currently, relevant technical solutions mainly rely on static rule bases and manual maintenance for risk content identification. While this approach is effective in traditional text scenarios, it often suffers from high false negative rates and response delays when facing new types of online language expressions (such as word splitting and homophones).

[0003] Therefore, there is an urgent need to provide a risk content identification scheme that is both accurate and adaptive. Summary of the Invention

[0004] Several embodiments in this specification provide a data processing method, model training method, system, device, equipment, storage medium, and program product, achieving accurate identification of risk content and effectively avoiding false negatives. In the first embodiment, this specification provides a data processing method. The method includes: acquiring target modality data to be detected; determining input data based on the target modality data; the input data includes input text and / or input image; performing risk detection on the input data using the risk detection model to obtain a risk detection result; wherein, when a risk is detected in the input data, the risk detection result includes risk content extracted from the input data; the risk content includes at least one of the following: risk words extracted from the input text, risk regions marked in the input image; the risk words include adversarial variant words, the adversarial variant words being semantically unchanged variant expressions formed by modifying known risk words.

[0005] In a second embodiment, this specification also provides a model training method. The method includes: acquiring a training sample set; fine-tuning a machine learning model based on the training sample set to obtain a risk detection model; wherein, the training sample set includes multiple training samples, the training samples include sample data and annotation information of the sample data; the annotation information includes risk content annotated in the sample data and risk classification labels corresponding to the sample data; and, the risk detection model is used to perform risk detection on input data and output risk detection results; the input data includes input text and / or input images; when a risk is detected in the input data, the risk detection result includes at least one of the following: risk words extracted from the input text, risk areas annotated in the input image; the risk words include adversarial variant risk words; the adversarial variant risk words refer to variant expressions formed by modifying known risk words without semantic change.

[0006] In a third embodiment, this specification also provides a data processing system. The system includes: Specification 1 / 18 pages 4 CN 121682541 AA client is used to acquire target modality data to be detected and transmit the target modality data to a server; a server is deployed with a risk detection model to implement the method provided in the first or second embodiment of the above claims.

[0007] In a fourth embodiment, this specification provides a data processing device. The device includes: an acquisition module, used to acquire target modality data to be detected; a determination module, used to determine input data based on the target modality data; the input data includes input text and / or input image; a detection module, used to perform risk detection on the input data using the risk detection model to obtain a risk detection result; wherein, when a risk is detected in the input data, the risk detection result includes risk content extracted from the input data; the risk content includes at least one of the following: risk words extracted from the input text, risk areas marked in the input image; the risk words include adversarial variant risk words, the adversarial variant risk words being semantically unchanged variant expressions formed by modifying known risk words.

[0008] In a fifth embodiment, this specification also provides a model training device. The device includes: an acquisition module for acquiring a training sample set; a training module for fine-tuning a machine learning model based on the training sample set to obtain a risk detection model; wherein the training sample set includes multiple training samples, the training samples include sample data and annotation information of the sample data; the annotation information includes risk content annotated in the sample data and risk classification labels corresponding to the sample data; and the risk detection model is used to perform risk detection on input data and output risk detection results; the input data includes input text and / or input images; when a risk is detected in the input data, the risk detection results include at least one of the following: risk words extracted from the input text, risk areas annotated in the input image; the risk words include adversarial variant risk words; the adversarial variant risk words refer to variant expressions formed by modifying known risk words without semantic change.

[0009] In a sixth embodiment, this specification provides an electronic device including a memory and a processor, wherein the memory stores executable program instructions, and when the processor executes the program instructions, it implements the method provided in the first to second embodiments.

[0010] In a seventh embodiment, this specification provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the methods provided in the first to second embodiments.

[0011] In an eighth embodiment, this specification also provides a computer program product, including a computer program / instructions.When the computer program / instruction is executed by the processor, it implements the method provided in the first to second embodiments.

[0012] The solutions provided in the above embodiments of this specification, after determining the input data (including input text and / or input image) based on the target modality data to be detected for risk, utilize a risk detection model to perform risk detection on the input data, thereby obtaining the corresponding risk detection result. Specifically, when a risk is detected in the input data, the risk detection result includes risk content extracted from the input data. The risk content includes at least one of the following: risk words extracted from the input text, and risk areas marked in the input image. The risk words include adversarial variant words. Adversarial variant words are variant expressions formed by modifying known risk words without semantic change. As can be seen, the solution described on page 2 / 18 of this specification, CN 121682541 A, uses a risk detection model to understand the semantics of the input data, enabling accurate extraction of risky content. For example, it can extract risky words such as adversarial variant words, which can effectively avoid missing risky content. Moreover, risk detection is no longer limited to single-modal data, but can achieve joint risk detection of multi-modal data (such as text pairs, video, audio, etc.). Brief Description of the Drawings

[0013] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the accompanying drawings: Figure 1 is a schematic diagram of the technical flow on which the implementation of each method in this specification is based, provided by an exemplary embodiment; Figure 2 is a schematic diagram of the structure of the data processing system provided by an exemplary embodiment in this specification; Figure 3 is a schematic diagram of the flow of the data processing method provided by an exemplary embodiment in this specification; Figure 4 is a schematic diagram of the flow of the model training method provided by an exemplary embodiment in this specification; Figure 5 is a schematic diagram of the structure of the data processing device provided by an exemplary embodiment in this specification; Figure 6 is a schematic diagram of the structure of the model training device provided by an exemplary embodiment in this specification; Figure 7 is a schematic diagram of the structure of the electronic device provided by an exemplary embodiment in this specification. Detailed Description

[0014] As mentioned above, with the development of the Internet and artificial intelligence technologies, the content on the Internet has experienced explosive growth, with increasingly diverse sources and highly complex forms. Currently, the main sources of Internet content include: User-Generated Content (UGC) and Artificial Intelligence-Generated Content (AIGC). User-Generated Content (UGC) is content created and published independently by users through Internet platforms, such as text comments, pictures, videos, audio, etc. This type of Internet content...Online content is characterized by high frequency, fragmentation, and diverse language styles, making it the most active yet most difficult part of the online ecosystem to manage. Artificial intelligence-generated content (AIGC) refers to content automatically generated using artificial intelligence technology. AIGC possesses high fidelity, scalable production capabilities, and cross-modal fusion characteristics, greatly improving content creation efficiency, but also bringing more controversial and risky new content. Existing risk content identification solutions mainly employ the following strategies: rule-based engine detection, detection combining static lexicons with traditional machine learning models, purely manual review, single-modal detection, or fixed threshold strategies. Among these, rule-based engine detection solutions primarily rely on manually written regular expressions or keyword blacklists for matching, combined with simple logical rules (such as "A and B" combinations) to identify risky content. However, this approach cannot identify variant words in new online language (such as homophones and word splitting) and semantically implicit risky content, resulting in a high false negative rate. Furthermore, the need for continuous manual rule updates leads to high maintenance costs. It also struggles to address new types of risky content (such as new words related to trending events). The detection method based on combining static lexicons with traditional machine learning models uses algorithms such as TF-IDF or SVM to match static risk lexicons and introduces basic classification models (such as Naive Bayes models) to label the text with risk types. However, this approach suffers from poor generalization ability, lack of contextual understanding, and limited coverage of combined words when facing the complex and ever-changing online content. For example, traditional Naive Bayes models mainly rely on word frequency features and cannot capture the semantic relationships and contextual information between words, which can easily lead to misjudgments (e.g., some words that may be neutral in medical content are misjudged as risk words). Furthermore, static lexicons usually only support predetermined fixed combination patterns (such as "purchase + **"), which need to be manually written, have limited coverage, and cannot dynamically discover emerging associated risks. The purely manual risk content identification method relies entirely on manual review of each piece of content, combined with experience manuals to determine risk levels and actions. (See manual 3 / 18, page 6, CN 121682541 A). This method is inefficient, unable to handle massive amounts of UGC content, has high response latency (e.g., high backlog during trending events), and is highly subjective, with inconsistent labeling by different reviewers leading to misjudgments or missed detections. Single-modal risk content detection methods can only detect risk content based on single-modal data such as text or images (e.g., pure text keyword filtering or image recognition), and cannot identify combined text and image violations (e.g., normal text with sensitive images), thus leading to cross-modal risk omissions. Fixed-threshold risk content detection methods set fixed thresholds for the number of risky word matches or classification probabilities (e.g., deletion upon matching two words). These thresholds lack dynamic adjustment, resulting in a high false positive rate (e.g., neutral words).Because accidental hits to the word library may lead to misprocessing), and it has poor flexibility and cannot adapt to different scenario requirements.

[0015] In view of the above problems, the embodiments in this specification provide a solution that can automatically and accurately identify risky content and has strong self-adaptability, and can identify risky content for cross-modal data.

[0016] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0017] It should be noted that, for ease of description, only the parts related to the relevant technical solutions are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of this specification can be combined with each other. In addition, the words "first", "second", "third" etc. in the embodiments of this specification are only used for information differentiation and do not serve any limiting function. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. Additionally, in this specification, unless explicitly stated otherwise, "receiving or sending data" does not necessarily mean direct receiving and sending, but can be indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving data sent by B, or it can be understood as A indirectly receiving data sent by B through other entities such as C; similarly, B sending data to A can be understood as B directly sending data to A, or it can be understood as B indirectly sending data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0018] It should also be noted that specific terms are used in this specification to describe embodiments thereof. Terms such as "an embodiment," "an embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "an embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, without contradiction, those skilled in the art can readily apply the different embodiments or examples described in this specification, as well as different practical applications...Features of embodiments or examples are combined and combined. Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, adjustments to the order of such steps or parallelism between steps are also within the scope of protection of the claims.

[0019] Furthermore, it should be noted that if this specification involves obtaining user data, the obtained user data is authorized by the user and does not involve user privacy.

[0020] The embodiments provided in this specification will be described below in conjunction with the accompanying drawings.

[0021] The technical solutions provided in the embodiments of this specification below are all based on the technical process shown in FIG1. ​​Specification 4 / 18 pages 7 CN 121682541 A As shown in FIG1, taking risk control in a text scenario as an example, the technical process mainly includes the following steps: Step 1, Input information: including the text to be detected and the risk label system.

[0022] 1) The text to be risk-detected includes, but is not limited to, user-posted comments, posts, private messages, live stream comments, articles, and text generated by AIGC technology. These texts are often characterized by high unstructuredness, diverse language styles (such as a mixture of internet slang, dialects, and emoticons), and potential adversarial interference (such as word splitting, homophones, and space insertion). These texts are the original objects of risk detection.

[0023] 2) The risk labeling system is a set of predefined risk classification standards used to accurately label the risk type of content. In specific designs, the risk labeling system usually adopts a multi-level hierarchical structure, such as a hierarchical design of "first-level risk category → second-level risk category".

[0024] For example, the hierarchical structure of the risk labeling system is shown in Table 1 below: Table 1 and the function of this risk labeling system are: to provide a basis for determining the risk classification label to which the text belongs.

[0025] Step 2, Data Processing Data processing includes: data preprocessing, risk classification, risk word extraction, and traffic replay verification. Among them, 1) Data preprocessing includes: text cleaning (such as removing non-textual symbols and other interfering information, standardizing format), word segmentation, and semantic analysis. Here, semantic analysis refers to multi-level parsing of the text to provide a basis for decision-making in subsequent risk classification and risk terms. Specifically, semantic analysis includes, but is not limited to, at least one of the following dimensions: topic / entity recognition, intent recognition, and sentiment recognition. Topic / entity recognition identifies the objects discussed in the text; for example, when the text mentions expressions such as "**person," it is necessary to accurately identify the specific person it refers to. Intent recognition is used to identify the fundamental purpose of the text's expression.Such as objective statements (e.g., news reports), subjective evaluations, calls to action, use of irony, etc. Sentiment tendency identification is to identify a user's attitude, emotion, or evaluation tendency towards a certain thing / person, etc.

[0026] As above, after the text is preprocessed, the text preprocessing results obtained may include: the cleaned text, the word segmentation results of the text, and the semantic analysis results. The semantic analysis results include risk entities (e.g., specific referents that may involve risks in the identified text), expressive intent, and sentiment tendency.

[0027] 2) Risk classification and risk word extraction The text preprocessing results obtained through the aforementioned data preprocessing are input into a pre-trained risk detection model so that the risk detection model can perform risk classification and risk word extraction on the cleaned text based on the word segmentation results and the semantic analysis results of the text.

[0028] In specific implementation, the risk detection model will first perform preliminary risk screening on the text to determine whether there is risky content in the text. If risky content is determined to exist, risk classification and risk word extraction will be performed on the text, and the risk classification label and extracted risk words corresponding to the text will be output. Conversely, if no risky content is determined to exist, the no-risk result can be directly output and the process ends.

[0029] The above-mentioned risk detection model is obtained by training an existing machine learning model (such as the BERT model or visual language model). When the risk detection model performs risk classification on the text, it selects the appropriate risk classification label for the text from the predefined risk label system through matching analysis. The role of the risk classification label is to serve the subsequent storage of the extracted risk words into the risk word library.

[0030] The risk word library is a database used to store risk words, which contains risk words and their combination patterns accumulated over a long period of time (such as "**& purchase", "**& tutorial"). In this specification, the risk term library uses a key-value (K-V) structure to store risk terms. The key (K) represents the risk type, and the value (V) represents the risk term. Furthermore, when the risk detection model extracts risk terms from text, it does so based on context (the semantic analysis result of the text), extracting core risk terms (such as independent risk terms or combinations of risk terms). Additionally, in some instances, the risk detection model can filter high-frequency neutral terms during risk term extraction. The technical means for filtering neutral terms through the risk detection model can be as follows: Neutral term filtering rules (such as including manufacturer names or company names as neutral terms) are added to the risk detection prompts input to the risk detection model. These rules inform the risk detection model which words are neutral and do not need to be extracted as risk terms. Therefore, the risk detection prompts input to the risk detection model can be based on the aforementioned...The text preprocessing results and neutral word filtering rules are used to generate the neutral word filtering. Of course, in other instances, other technical means can also be used to achieve neutral word filtering. For example, if the risk detection model does not have the ability to filter neutral words, the risk words it outputs may contain neutral words. In this case, neutral word filtering can be achieved through subsequent traffic replay verification. For the specific implementation of neutral word filtering through traffic replay, please refer to the relevant content described below for "3) Traffic Replay Verification".

[0031] 3) Traffic Replay Verification Traffic replay verification is based on historical text that has passed the risk content review, and verifies whether the risk word interception causes excessive disturbance. Excessive disturbance refers to incorrectly judging the content (such as words) in the text that should pass as illegal or risky content.

[0032] For example, the specific implementation of traffic replay verification may include the following steps: Step 31, determine the risk words to be verified. Assume that the risk words to be verified include: risk word A, risk word B, and risk word C. These risk words to be verified are extracted from the text by the aforementioned risk detection model.

[0033] Step 32: Select historical normal traffic samples, such as randomly selecting 100,000 texts that have passed risk content review from the past 7 days, as replay samples.

[0034] Step 33: Perform traffic replay to re-detect risk content on the above 100,000 samples, simulating the online interception logic when the above-mentioned risk words to be verified are used as interception targets.

[0035] Step 34: Statistically analyze the misjudgment results. For example, the misjudgment results are as follows: Risk word A appeared 4800 times in 100,000 samples and was intercepted 4760 times, with a corresponding misjudgment rate of 99.2%; Risk word B appeared 3200 times in 100,000 samples and was intercepted 3100 times, with a corresponding misjudgment rate of 96.9%. Risk word C appeared 12 times in 100,000 samples and was intercepted 12 times, with a corresponding misjudgment rate of 0%. Based on these misjudgment results, the following conclusions can be drawn: Risk word A and risk word B are high-frequency neutral words with high false positive rate; Risk word C is a low-frequency high-risk word. Therefore, risk words A and B can be filtered out and not stored in the risk word database.

[0036] Here, the correctness of risk word extraction is verified by traffic replay verification, thereby filtering out neutral words, which can reduce the reliance on manual review.

[0037] Step 2, Output Results The output results mainly include the following three parts: risk identification results, risk category labeling, and disposal strategy suggestions.

[0038] Among them, the risk identification results are the output risk word list or "no risk". If it is the output risk word list, different risk words will be separated by commas and combined words will be connected by the symbol "&".

[0039] Risk classification label: This refers to the risk classification label associated with the risk words. Based on the risk classification label associated with the risk words, the risk words can be stored in the risk word library.

[0040] And, strategy suggestion: This refers to the recommended handling strategy (such as deletion, traffic restriction, manual review) for the text. The determination of the handling strategy will be detailed in the following method embodiments, and will not be elaborated here.

[0041] The technical architecture mentioned above is based on a server and a client. As shown in Figure 1, the risk detection model, risk word library, etc. in the technical architecture can be deployed on the server. The server can be a server, server cluster, virtual server, or cloud, etc. And, the target modal data (such as text) to be detected can be obtained by the client and sent to the server for risk detection processing. The client can be, but is not limited to: smartphones, smart wearable devices, tablets, laptops, desktop computers, etc.

[0042] Thus, Figure 2 also shows a data processing system provided in an embodiment of this specification, which includes a client 100 and a server 200. The client 100 is used to acquire target modality data to be detected and transmit the target modality data to the server.

[0043] The server 200 is equipped with a risk detection model, which is used to determine input data based on the target modality data; the input data includes input text and / or input image; the risk detection model is used to perform risk detection on the input data to obtain risk detection results; wherein, when the input data is detected to have a risk, the risk detection results include risk content extracted from the input data; the risk content includes at least one of the following: risk words extracted from the input text, risk areas marked in the input image; the risk words include adversarial variant risk words, which are variant expressions formed by modifying known risk words without semantic changes.

[0044] The specific implementation of the functions of the server 200 and the client 100 will be described in detail in the following method embodiments, and will not be elaborated here.

[0045] In addition to being deployed on the server 200, the risk detection model described above can also be deployed at the edge in other instances. For example, when the risk detection model is obtained by fine-tuning an existing lightweight machine learning model (such as the BERT model), it can also be deployed on the client (i.e., the user terminal), so that the target modality data to be detected can be directly detected locally, and the risk word library and risk control on the server (such as the cloud) can be updated synchronously according to the risk detection results, which can reduce response latency.

[0046] In another embodiment, this embodiment also provides a data processing system, which includes a client and a server.A client, on which a risk detection model is deployed, is used to acquire target modal data to be detected; input data is determined based on the target modal data; the input data includes input text and / or input image; the risk detection model is used to perform risk detection on the input data to obtain risk detection results; wherein, when the input data is detected to have a risk, the risk detection results include risk content extracted from the input data; the risk specification 7 / 18 pages 10 CN 121682541 A includes at least one of the following: risk words extracted from the input text, risk map areas marked in the input image; the risk words include adversarial variant risk words, the adversarial variant risk words being variant expressions formed by modifying known risk words without semantic change. Furthermore, the client is also used to send the risk content to the server; the server is deployed with a risk word library, used to update the risk word library based on the received risk content.

[0047] The technical solution provided in this specification will be described in detail below by way of method embodiments.

[0048] Figure 3 shows a flowchart of a data processing method provided in one embodiment of this specification. The main execution body of this method is the server in the aforementioned system, but it can also be the client in other instances. Referring to Figure 3, the data processing method includes the following steps: 102. Obtaining target modal data to be detected; 104. Determining input data based on the target modal data; the input data includes input text and / or input image; 106. Using the risk detection model, performing risk detection on the input data to obtain a risk detection result; wherein, when the input data is detected to have a risk, the risk detection result includes risk content extracted from the input data; the risk content includes at least one of the following: risk words extracted from the input text, risk areas marked in the input image; the risk words include adversarial variant risk words, and the adversarial variant risk words are variant expressions formed by modifying known risk words without semantic change.

[0049] In 102 above, the target modal data can be, but is not limited to, text, audio, video, image, image-text pairs, etc. Among them, an image-text pair refers to a multimodal unit composed of an image and its semantically associated text. For example, when a user uploads an image, they simultaneously submit corresponding text and titles, thus forming an image-text pair.

[0050] Furthermore, the target modal data can be obtained through a client. Acquisition methods may include: obtaining from the network side (such as obtaining text publicly published on the network, such as from social media, news comment sections, etc.), and API interfaces (such as calling open APIs of social platforms to obtain posts published on the platform).

[0051] In the above 104, different methods will be used to determine the input data to be detected for different modal data, such as input text and / or input image.

[0052] In specific implementation, the input data is determined as follows: If the target modal data is text, the output input data is determined to contain only the input text, and the input text is the target modal data.

[0053] If the target modal data is audio, the output input data is determined to contain only the input text. The input text is obtained by using Automatic Speech Recognition (ASR) technology to perform speech recognition on the audio.

[0054] If the target modal data is an image, the output input data may contain both input text and input image, or it may contain only the input image. For example, the input text may be obtained by using Optical Character Recognition (OCR) technology to extract text from the image, and the input image is the image.

[0055] If the target modal data is a text-image pair, the output input data is determined to contain both input text and input image. The input text includes: text associated with the image in the image-text pair, and text extracted by performing operations such as optical character recognition on the image. The input image is the image in the image-text pair.

[0056] If the target modal data is video, the output input data is determined to include both input text and input image. The input text includes: text obtained by performing speech recognition on the audio in the video, and text embedded in the video. Text embedded in the video refers to text content directly drawn or superimposed on the video screen, such as subtitles and bullet comments, which can be obtained by batch text extraction from key image frames in the video. The input image contains key image frames extracted from the video. For example, frames can be extracted from the video at certain time intervals (such as 1 second) or based on scene change detection to obtain key image frames in the video.

[0057] In the above 106, the risk detection model is obtained by fine-tuning and training an existing machine learning model. The machine learning model trained is different for different risk detection scenarios.

[0058] For example, in the scenario of detecting risky text content, a risk detection model can be obtained by fine-tuning a natural language model (such as the BERT model or the GPT model). The GPT model is a series of Large Language Models (LLMs) based on the Transformer architecture. In this case, the input data to the risk detection model can only be text, and correspondingly, the target modality data obtained can be in the form of text or audio. If it is audio, it needs to be converted into text before being input into the risk detection model.

[0059] Taking fine-tuning the BERT model as an example, labeled risk texts can be collected to construct a training sample set. This training sample set contains multiple text samples and their labeling information (such as risk words and risk classification labels). The BERT model is then fine-tuned using this training sample set to obtain a risk detection model that supports risk classification and risk word extraction. To improve the detection model's ability to extract adversarial variant risk words (such as homophones), adversarial samples can be included among the multiple text samples. Adversarial samples refer to text samples with adversarial variant risk words. Furthermore, adversarial samples can, but are not limited to, being generated by a generative adversarial network (GAN) to simulate attacker generation.

[0060] For another example, in a multimodal risk content detection scenario, a risk detection model can be obtained by fine-tuning an existing multimodal model (such as a visual language model). In this case, the input data format to the risk detection model can include text and images. Correspondingly, the data format of the aforementioned target modality data can be images, text-image pairs, or videos. It is evident that in multimodal risk content detection scenarios, a finely tuned risk detection model can achieve cross-modal correlation analysis, such as detecting risks associated with combined text and images (e.g., prohibited text paired with sensitive images).

[0061] Further, in one feasible solution, step 106, "using the risk detection model to perform risk detection on the input data and obtain risk detection results," may include: 1062, generating risk detection prompts based on the input data; 1064, inputting the risk detection prompts into the risk detection model, and having the risk detection model perform risk detection based on the risk detection prompts, and outputting the risk detection results.

[0062] In step 1062 above, when the input data contains input text, the input text can be preprocessed first. Preprocessing includes, but is not limited to, cleaning, word segmentation, and semantic analysis. For example, the input text can be cleaned first to remove invalid content (such as non-text symbols and spaces) and the text format can be standardized; then, the cleaned input text can be segmented and semantically analyzed sequentially. That is, semantic analysis is based on the word segmentation results and includes at least one of named entity recognition, expressive intent analysis, and sentiment analysis. Risk detection prompts are then generated based on the content obtained from the preprocessing.

[0063] Thus, when the input data contains input text, the generated risk detection prompts may contain text preprocessing results, which include the cleaned input text, the word segmentation results of the input text, and the semantic analysis results. The semantic analysis results include the names identified from the input text, the expressive intent of the input text, and the sentiment tendency.Named entities refer to words or phrases in text that have a clear referential meaning and are used to identify specific objects in the real world, including but not limited to organization names, product names, disease names, and prohibited product names.

[0064] In some other embodiments, the generated risk detection prompts may also contain other content, such as neutral word filtering rules, which allows the risk detection model to filter out some neutral words when extracting risk words during the risk detection process based on the risk detection prompts. For example, neutral word filtering rules may include: if the risk word contains a book title, movie title, or is enclosed in quotation marks, it can be filtered out; if the text contains negative verbs and the risk word is the object of criticism, it is considered a positive / compliant expression and can be filtered out.

[0065] In section 1064 above, when performing risk detection, the risk detection model first classifies the input data for risk to determine whether the input data contains risk. If a risk is determined to exist, it further extracts the risk content from the input data based on the risk classification labels corresponding to the input data, thereby outputting a risk detection result containing the risk content, risk classification labels, etc. Conversely, if no risk is determined, a risk detection result containing "no risk" is output, and the risk detection process ends. The risk classification labels can be determined by matching from a predefined risk label system based on the semantic analysis results of the input data. Therefore, this solution actually uses a risk detection model combined with a risk label system to achieve risk detection of the input data, effectively reducing the false negative rate of risk content.

[0066] For example, taking an input data consisting only of input text, and the input text being "I'll send you a tutorial on '**restoration method' via private message, guaranteed to teach you everything," the risk detection model, based on the word segmentation and semantic analysis results of the input text contained in the risk detection prompts, determines the risk classification label corresponding to the input text as label E based on the named entities (such as "**" in "**restoration method"), expressive intent, and sentiment tendency contained in the semantic analysis results, thereby determining that there is a risk and entering the risk word extraction stage. In the risk word extraction stage, the risk detection model will extract the risk words "**", "restoration method", and "**restoration method" based on the risk direction indicated by the risk classification label and in combination with the semantic analysis results and word segmentation results of the input text.

[0067] For another example, if the input data consists only of input images, the risk detection model can determine the risk classification label corresponding to the input image by performing semantic analysis on the input image. Thus, when it determines that the input image has a risk, it will determine the risk classification label based on the risk direction indicated by the risk classification label corresponding to the input image and in combination with the semantic analysis of the input image.As a result, risk areas are marked in the input image. Semantic analysis of the input image refers not only to recognizing individual elements (such as people, vehicles, and signs) but also understanding the high-level semantic meaning expressed by the combination of these elements. Furthermore, the marked risk areas refer to local areas in the image containing content that is illegal, sensitive, or dangerous.

[0068] For example, if the input data includes both input text and an input image, the risk detection model can determine the corresponding risk classification label for the input data by jointly analyzing the semantic analysis results of the input image and the semantic analysis results of the input text. Thus, when analysis is determined, the corresponding risk areas are marked in the input image, and the corresponding risk words are extracted from the input text.

[0069] Based on the above examples, in this embodiment, the risk detection result output by the risk detection model for the input data may be "no risk." Alternatively, the risk detection result may include risk content extracted from the input data. The specific type of risk content depends on the data type contained in the input data. If the input data only contains input text, the risk content includes risk words extracted from the input text. If the input data only contains an input image, the risk content includes the marked risk areas in the input image. If the input data includes input text and input image, the risk content may include risk words extracted from the input text and risk areas marked in the input image. Therefore, the risk content includes at least one of the following: risk words extracted from the input text and risk areas marked in the input image. Where there are multiple risk words, the output risk words are separated by commas.

[0070] The types of the extracted risk words may include: ordinary single risk words, adversarial variant words, and conjunctive combination words.

[0071] Wherein, adversarial variant words are semantically unchanged variant expressions formed by modifying known risk words. (See page 10 / 18 of the specification, CN 121682541 A). Specifically, adversarial variant words may include, but are not limited to, at least one of the following: variant words formed by inserting interference symbols into known risk words, such as “*&*&*”, “*$**”; variant words formed by using spaces or newlines to separate characters, such as “*****”; homophones generated based on known risk words, etc. Variant words formed by replacing known risk words with homophones, similar-looking characters, or pinyin spelling. It can be seen that the risk detection model provided in this embodiment supports the generalization recognition of adversarial text, that is, it supports the recognition of various variant risk words such as disassembly and recombination in adversarial text.

[0072] In addition, the connecting combination word is composed of at least two semantically related words, and a connecting symbol (such as “&”, underscore, or other non-character data interference character) is inserted between adjacent words. For example, the connecting combination word includes “**&purchase”.

[0073] To improve the accuracy of risk word extraction, this embodiment supports neutral word filtering during and / or after risk word output. Neutral word filtering can be achieved by adding neutral word filtering rules to the risk detection prompts of the risk detection model, and / or by traffic replay verification.

[0074] Therefore, the method provided in this embodiment may also include the following steps: S11, during the risk word extraction process of the input text, the risk detection model filters candidate risk words based on preset neutral word filtering rules to exclude candidate risk words that are neutral from the input risk detection results. And / or S12, based on historical text, the risk words contained in the risk detection results are replayed and verified to filter out risk words that are neutral.

[0075] For a detailed description of the implementation of the above steps S11 to S12, please refer to the relevant content described in conjunction with Figure 1 in the other embodiments mentioned above, which will not be elaborated here.

[0076] In addition, in other instances, when using a risk detection model to perform risk detection, a pre-constructed knowledge graph can be introduced as an auxiliary knowledge source for the risk detection model, improving the model's ability to identify complex, obscure, variant, or cross-domain risk content. The knowledge graph is a graph constructed based on domain knowledge and risk audit data, with risk entities as interfaces and semantic relationships as edges (e.g., **—[association]—> production method, **—[transaction hint]—> private chat). Based on the knowledge graph, the risk detection model can identify implicit risk behavior links in text or cross-modal content (e.g., "purchasing raw materials + inquiry steps" may point to illegal intent), thereby improving its ability to proactively discover adversarial, combined, and other illegal risk content.

[0077] Furthermore, the aforementioned output risk detection results, in addition to containing risk content, also include risk classification labels and confidence information. A risk classification label is composed of a primary risk label and a sub-risk label belonging to that primary risk label, and can be associated with the risk content for output. Confidence information may include, for example, the confidence level of the risk classification label.

[0078] And for the finally obtained risk words, the risk words can be stored in the risk word library according to the risk classification labels associated with the risk words. The risk word library uses a key-value pair structure to store risk words, where the key (K) is the risk classification and the value (V) is the risk word. For a detailed description of the risk word library, please refer to the relevant content described in conjunction with Figure 1 in other embodiments.

[0079] That is, the method provided in this embodiment may also include the following steps: 108. Storing the risk words in the risk word library according to the risk classification labels associated with the risk words.

[0080] Adding risk words to the risk word library according to the corresponding risk classification labels facilitates subsequent different control measures based on different risk classifications. (Instructions)Page 11 / 18, 14 CN 121682541 A

[0081] For example, in the risk term library, each risk category has a pre-set corresponding risk handling strategy (such as interception, traffic limiting, etc.). Suppose that for the target modal data to be analyzed and detected, a certain risk word is identified, then at least one handling strategy set for the risk category to which the risk word belongs can be obtained from the risk term library, and then a handling strategy (such as interception) can be selected from the at least one handling strategy, and the target modal data can be controlled accordingly based on the selected handling strategy. Of course, the handling strategy can also be determined in other ways, such as based on the risk level corresponding to the risk word. The risk level can be determined by semantic analysis of the target modal data. For example, suppose the target modal data is text, by performing semantic analysis on the target modal data (such as analyzing the expressive intent, sentiment tendency, etc.), it can be determined that the target modal data is a negative evaluation of a certain event or person, then the risk level can be determined to be high risk, and thus the appropriate handling strategy can be automatically determined to be deletion.

[0082] The handling strategy for target modal data in this solution is adaptively determined based on its corresponding risk classification label or risk level, which is beneficial for achieving second-level response in data management and is very suitable for massive UGC scenarios.

[0083] Based on the above, the method provided in this embodiment may further include the following steps: 1010. Determine an appropriate handling strategy for the target modal data according to the risk classification label or the risk level corresponding to the input data; wherein, the risk level is obtained by semantic risk assessment of the input data.

[0084] 1012. Perform corresponding management actions on the target modal data according to the handling strategy. The types of management actions include, but are not limited to, content processing (such as interception, traffic limiting, blocking, deletion, submission for manual review) and action processing (such as warning, mute, account suspension, and function restriction).

[0085] In addition, after the management action is executed, the reason for making the management action can be returned to the manual reviewer or the user who published the target modal data. The reason may be generated based on, but is not limited to, the corresponding risk classification label, risk words, handling strategy, etc. For example, the reason for generation might be "The content contains false statements about a certain matter, violating relevant regulations, and has been deleted. If you have any objections, you can submit an appeal and provide an explanation." Furthermore, based on user appeals or feedback from human reviewers, cases of misjudgment are recorded so that the risk detection model and risk terminology can be optimized subsequently (e.g., deleting misjudged risk words from the risk terminology). Alternatively, in other implementations, the risk detection model and risk terminology can be further optimized by combining other data, such as data on the effectiveness of handling strategies.The risk can be determined by, but is not limited to, user feedback. For example, the false positive rate can be determined based on user feedback, and the false positive rate can be used as an indicator of the effectiveness of the handling strategy.

[0086] In optimizing the risk detection model, the recognition weight of each risk word can be automatically optimized. For example, when a certain type of name (such as an inappropriate term for a certain item or person) is found to be seriously missed, the weight of proper names and noun phrases can be increased, while the interference weight of irrelevant verbs and adverbs can be reduced. For another example, for words that are highly sensitive but frequently used legally, their basic risk score can be dynamically lowered based on contextual sentiment and intent analysis, and a high-risk judgment can only be triggered when strong signals such as derogatory or malicious occur. Through this risk word weight optimization mechanism, the risk detection model can learn "when to believe which word" through reinforcement learning, rather than relying on static rules, thereby achieving a dynamic balance between reducing the false positive rate (such as normal news reports not being intercepted) and increasing the recall rate of high-risk content (such as the accurate capture of coded language).

[0087] The solution provided in this embodiment achieves accurate extraction of risky content through artificial intelligence (AI) technology. For example, it can extract risky words such as adversarial variant words and compound words, which can effectively avoid missing risky content. Moreover, it can also verify whether the extraction of risky content is correct and reasonable by combining traffic replay and other methods, which can effectively reduce risk misjudgment. In addition, it can automatically determine the appropriate handling strategy for the extracted risky content. This solution can be widely applied to various platforms, such as social platforms, UGC communities, AI-generated content review and other scenarios, and can ensure both content compliance and user experience.

[0088] The above mainly describes the use of AI (specifically risk detection model) to achieve risk detection of target modal data. In addition, a multi-level risk detection method of rule + AI or a risk detection method of basic risk word library + general model (such as risk detection model) can also be used to achieve risk detection of target modal data.

[0089] 1) When implementing risk detection of target modal data using a multi-level risk detection method of rules + AI, a rule layer and an AI layer are deployed on the server side, and a risk detection model is installed on the AI ​​layer. Furthermore, the link between the rule layer and the AI ​​layer can be arranged serially. After the server determines the input data based on the target modal data to be detected, it can first send the input data to the rule layer, which determines whether there is risk content in the input data based on preset risk identification rules. If risk content is determined to exist, the identified risk content is directly output, and the AI ​​layer is skipped. If no risk content is determined to exist, the input data is then sent to the AI ​​layer for risk detection.

[0090] For example, assuming the input data contains input text, preset regular expressions can be used to...The system identifies whether there are explicit risk words in the input text, and then determines whether the input text poses a risk. If a risk is determined to exist, the AI ​​layer can be skipped, and the risk detection result can be output directly. Conversely, if no risk is determined to exist, the input text can be further fed into the AI ​​layer, where the risk detection model is called to perform risk detection on the input text to identify whether there are implicit risk words (such as adversarial variant words) in the input text. Furthermore, if the input data also includes an input image, the risk identification method used on the rule layer for the input image can be, but is not limited to, at least one of the following: whether the input image is highly similar to a blocked image (this can be achieved through perceptual hashing), and whether the input image contains a preset risk marker.

[0091] It should be noted here that, in multi-level risk detection, in addition to the aforementioned rule layer and AI layer, an expert review layer can also be included. The expert review layer is used for manual review of high-risk content. For example, if the input data is subjected to risk detection through rules and AI layers, and a risk is determined to exist, the identified risk content can be manually reviewed. Specifically, when the risk level corresponding to the risk content is high risk, the manual review can be triggered to check whether the identified risk content is correct and whether there are any omissions. The risk level is obtained by semantic analysis of the input data.

[0092] 2) When implementing the risk detection method of basic risk lexicon + general model for target modal data, if the input data only contains input text, it is possible to first identify whether the words in the input text match those in the risk lexicon to determine whether there is a risk. If a risk is determined to exist, the risk check result can be directly output. Alternatively, in some other embodiments, a risk detection model can be used to further detect the risk to identify the hidden risk words, and then the risk detection result can be finally output. If no risk is determined, the risk detection result (i.e., no risk) can be directly output. Furthermore, if the input data also contains an input image, the risk lexicon matching analysis can be skipped for the input image, and the risk detection model can be used directly for risk detection analysis.

[0093] Figure 4 shows a flowchart of a model training method provided in one embodiment of this specification. The main execution entity of this method is the server in the aforementioned system, or in other instances, it can be the client. As shown in Figure 4, the model training method includes the following steps: 202, obtaining a training sample set; 204, fine-tuning the machine learning model based on the training sample set to obtain a risk detection model; wherein, the training sample set includes multiple training samples, and the training samples include sample data and annotation information of the sample data; the annotation information includes the risk content annotated in the sample data and the risk classification label corresponding to the sample data.

[0094] And, the risk detection model is used to perform risk detection on the input data and output the risk detection result; the input data includes input text and / or input image; when the input data is detected to have a risk, the result of the risk detection specification 13 / 18 pages 16 CN 121682541 A includes at least one of the following: risk words extracted from the input text, risk map areas marked in the input image; the risk words include adversarial variant risk words; the adversarial variant risk words refer to variant expressions formed by modifying known risk words without semantic changes.

[0095] In the above, if the machine learning model is a natural language model (such as the BERT model or the GPT model), then correspondingly, the multiple training samples contained in the training sample set are all text samples, and the text samples include text data and text data annotation information (such as corresponding risk classification labels, risk words); and, the risk detection model obtained by fine-tuning the machine learning model can only be used to perform risk detection on text-type modal data.

[0096] If the machine learning model is a multimodal model (such as a visual language model), then the corresponding training sample set may include image training samples and text training samples. Furthermore, the risk detection model obtained by fine-tuning the machine learning model can be used to detect risks in multimodal data (such as text, images, videos, image-text pairs, etc.).

[0097] Further, the machine learning model can also be optimized through adversarial training so that the trained risk detection model has the ability to identify adversarial risk content. Therefore, the aforementioned multiple training samples may include adversarial training samples. And, the method of obtaining adversarial training samples may include: S22, using a generative adversarial network (GAN) to generate adversarial sample data containing adversarial variant words; S24, labeling the adversarial sample data to obtain the labeling information of the adversarial sample data.

[0098] In the above, the labeling of adversarial sample data can be achieved manually or through a corresponding labeling model.

[0099] It should be added here that: for machine learning models, cross-domain fine-tuning training is possible. Cross-domain fine-tuning training refers to fine-tuning an existing machine learning model for a vertical domain (such as healthcare, finance, or government affairs) using training samples obtained from related domains. This allows the trained risk detection model to more accurately understand the semantics and risk patterns of a specific domain, thereby enhancing risk identification in specific scenarios. For example, in a healthcare scenario, the risk detection model can effectively identify high-risk combinations such as "**drug purchasing agents," significantly improving the accuracy and recall of risk identification in specific scenarios.

[0100] For specific implementation details of the above steps given in this embodiment, please refer to the relevant content in other embodiments, which will not be elaborated here. In addition, the method provided in this embodiment may also include some steps disclosed in other embodiments.The steps are similarly described in other embodiments, and will not be repeated here.

[0101] The specific embodiments of this specification have been described above with reference to Figures 3-4. It should be noted that other embodiments are within the scope of the appended claims. Also, in some cases, the actions or steps described in the specification can be performed in a different order than in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] The apparatus embodiments corresponding to the various method embodiments provided in this specification will be described below.

[0103] Figure 5 shows a schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment of this specification. As shown in Figure 5, the apparatus includes: an acquisition module 32, a determination module 34, and a detection module 36. The acquisition module 32 is used to acquire target modality data to be detected. The determination module 34 is used to determine input data based on the target modality data; the input data includes input text and / or input images. The detection module 36 is used to perform risk detection on the input data using the risk detection model to obtain a risk detection result. When a risk is detected in the input data, the risk detection result includes risk content extracted from the input data; the risk content includes at least one of the following: risk words extracted from the input text, risk areas marked in the input image; the risk words include adversarial variant words, which are semantically unchanged variant expressions formed by modifying known risk words. Specification 14 / 18 pages 17 CN 121682541 A

[0104] In one implementable solution, the aforementioned adversarial variant words include at least one of the following: variant words formed by inserting interference symbols into known risk words; homophones generated based on known risk words; variant words formed by replacing known risk words with homophones, similar-looking characters, or using pinyin spelling.

[0105] In one possible implementation, the risk word further includes: a connecting combination word, wherein the connecting combination word contains at least two semantically related words and a connecting symbol is inserted between adjacent words.

[0106] In one possible implementation, the device further includes: a replay verification module and / or a filtering module. The replay verification module is used to replay and verify the risk words contained in the risk detection result based on historical text, so as to filter out risk words that belong to neutral words. The filtering module is used by the risk detection model to filter candidate risk words based on preset neutral word filtering rules during the risk word extraction process of the input text, so as to filter candidate risk words that belong to neutral words from...The risk detection result is excluded from the output.

[0107] In one possible solution, the detection module 36, when used to perform risk detection on the input data using the risk detection model to obtain the risk detection result, can be specifically used to: generate risk detection prompt words based on the input data; input the risk detection prompt words into the risk detection model, and have the risk detection model perform risk detection based on the risk detection prompt words, and output the risk detection result. Wherein, when the input data includes input text, the risk detection prompt words contain text preprocessing results, the text preprocessing results include the cleaned input text, the word segmentation results of the input text and the semantic analysis results, the semantic analysis results include the identified named entities, expressive intent, and sentiment tendency; or, the risk detection prompt words contain the text preprocessing results and the neutral word filtering rules.

[0108] In one possible solution, the risk detection result also includes risk classification tags. And, the device further includes: a storage module, used to store the risk words into a risk word library according to the risk classification tags. The risk terminology database adopts a key-value pair storage structure, where the key represents a risk category and the value represents a risk term contained under the corresponding risk category.

[0109] In one possible implementation, the determining module 34 is further configured to: determine an appropriate handling strategy for the target modal data based on the risk category label or the risk level corresponding to the input data; wherein the risk level is obtained by semantic analysis of the input data. The device also includes an execution module, configured to perform corresponding control actions on the target modal data based on the handling strategy.

[0110] In one possible implementation, the target modal data is any one of the following: text, audio, image, or image-text pair. And, the aforementioned determining module 34, when used to determine input data based on the target modal data, is specifically used for: when the target modal data is text, using the target modal data as the input text; when the target modal data is audio, performing speech recognition on the audio to obtain the input text; when the target modal data is an image, performing text extraction on the image, using the extracted text as the input text; and using the image as the input image; when the target modal data is a text-image pair, performing text extraction on the image in the text-image pair, using the extracted text and the text associated with the image in the text-image pair as the input text; and using the image in the text-image pair as the input image; when the target modal data is video, performing speech recognition on the audio in the video, using the text obtained from the speech recognition and the text embedded in the video as the input text; and extracting key image frames from the video, using the extracted image frames as the input image.

[0111] Figure 6 shows a schematic diagram of a model training device provided in an exemplary embodiment of this specification. As shown in Figure 6, the device includes: an acquisition module 44 and a training module 46. The acquisition module 44 is used to acquire a training sample set. The training module 46 is used to fine-tune the machine learning model based on the training sample set to obtain a risk detection model. The training sample set includes multiple training samples, which include sample data and annotation information of the sample data (see page 15 / 18 of the specification, CN 121682541 A). The annotation information includes risk content annotated in the sample data and risk classification labels corresponding to the sample data. The risk detection model is used to perform risk detection on input data and output risk detection results. The input data includes input text and / or input images. When a risk is detected in the input data, the risk detection result includes at least one of the following: risk words extracted from the input text, risk areas annotated in the input image; the risk words include adversarial variant risk words; the adversarial variant risk words refer to variant expressions formed by modifying known risk words without semantic change.

[0112] In one possible implementation, the plurality of training samples includes adversarial training samples. Furthermore, when the acquisition module 44 is used to acquire the adversarial samples, it can specifically be used to: generate adversarial sample data containing risk words of anti-variant variants using a generative adversarial network; and annotate the adversarial sample data to obtain the annotation information of the adversarial sample data.

[0113] It should be noted that the above-mentioned devices can implement the technical solutions described in the corresponding method embodiments. The specific implementation principles of each module or unit can be found in the relevant content of the corresponding method embodiments, and will not be elaborated here. Additionally, for ease of description, the above devices are described by function as various modules or units. Of course, when implementing one or more of this specification, the functions of each module or unit can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0114] In addition, embodiments of this specification also provide an electronic device. As shown in FIG7, the electronic device 900 includes: a memory 91 and a processor 92.

[0115] The memory 91 described above may be implemented by at least one volatile or non-volatile storage device of any type or a combination thereof, such as static random access memory (SRAM).Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk, or Optical Disk. Furthermore, the memory, in whole or in part, may be integrated with a processor. The memory may contain removable and non-removable components.

[0116] The processor 92 described above may comprise one or more general-purpose processors and / or special-purpose processors.

[0117] Additionally, the memory 91 may contain a non-transitory computer-readable medium storing executable program instructions 912 (e.g., compiled or uncompiled program logic and / or machine code). The processor 92 is capable of executing the program instructions 912 stored in the memory to implement any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Furthermore, the execution of program instructions 912 by processor 92 may cause the processor to use corresponding data 911.

[0118] For example, the aforementioned program instructions 912 may include an operating system 9122 (e.g., an operating system kernel, device drivers, and / or other modules) installed on electronic device 900 and one or more application programs 9121 (e.g., a browser, social application, or game application). Similarly, the aforementioned data 911 may include operating system data 9112 and application data 9111. Operating system data 9112 is primarily accessible to operating system 9122, while application data 9111 is primarily accessible to one or more application programs 9121. Application data 9111 may reside in a file system visible or hidden from the user of electronic device 900.

[0119] Application program 9121 may communicate with operating system 9122 through one or more application programming interfaces (APIs). These APIs facilitate application program 9121 reading and / or writing application data, transmitting or receiving information via communication components, receiving or displaying information on a user interface, etc. In some terms, application 9121 may be abbreviated as "app (as per specification page 16 / 18, 19 CN 121682541 A)". Furthermore, application 9121 can be downloaded to the electronic device through one or more online application stores or app markets. However, application 9121 can also be installed on the electronic device 400 in other ways, such as through a web browser or...The physical interface on the sub-device 900 (e.g., a USB port) is further illustrated in FIG. 7. The electronic device also includes other components such as a communication component 93, a display 94, a power supply component 95, an audio component 96, and a user interface 97. FIG. 7 only schematically shows some components and does not imply that the electronic device 900 only includes the components shown in FIG. 7. Furthermore, the components within the dashed boxes in FIG. 7 are optional, not mandatory, and depend on the product form of the electronic device 900. The electronic device 900 of this embodiment can be implemented as a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device; it can also be a server-side device such as a conventional server, cloud server, or server array; or it can be an integrated device combining a terminal device and a server-side device. If the electronic device 900 of this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include the components within the dashed boxes in FIG. 7; if the electronic device 900 of this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, it may not include the components within the dashed boxes in FIG. 7.

[0120] The communication component 93 described above is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component 93 can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In an exemplary embodiment, the communication component 93 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In specific implementations, the communication component 93 includes a communication interface that enables the electronic device 900 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface may include a chipset and an antenna for wireless communication with a radio access network or access point. In addition, the communication interface may be a wired interface such as Ethernet, Token Ring, or USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface may also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface may also include multiple physical communication interfaces, such as Wi-Fi interfaces, Bluetooth interfaces, and wide area wireless interfaces.

[0121] The aforementioned display 94 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0122] The power supply component 95 described above provides power to various components of the device in which it resides. The power supply component 95 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0123] The audio component 96 described above may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device in which the audio component resides is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0124] The user interface 97 described above includes receiving user input and providing output to the user. Therefore, the user interface 97 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and may also include output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 97 Specification 17 / 18 pages 20 CN 121682541 A Auditory output can also be generated via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 97 may include software, circuitry, or other forms of logic capable of transmitting data to and receiving data from external user input / output devices. Additionally or alternatively, electronic device 900 may support remote access from other devices via, for example, a communication interface or another physical interface (not shown). User interface 97 may be configured to receive user input, the position and movement of which may be indicated by an indicator or cursor described herein. User interface 97 may also be configured as a display device for rendering or displaying text fragments.

[0125] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program that, when executed by a processor, enables the processor to implement the steps in the above method embodiments. The computer-readable storage medium may be volatile or non-volatile, or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), and dynamic random access memory (DRAM).This specification also provides a computer-readable storage medium storing a computer program thereon, wherein, when executed in a computer, the computer program causes the computer to perform the methods described in Figures 3 to 4.

[0126] This specification also provides a computer program product including a computer program / instructions that, when executed by a processor, implement the methods described in Figures 3 to 4.

[0127] Those skilled in the art should recognize that the functions described in the various embodiments disclosed in this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0128] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification. Specification page 18 / 18, 21 CN 121682541 A, Figure 1; Specification drawing page 1 / 4, 22 CN 121682541 A, Figure 2, Figure 3; Specification drawing page 2 / 4, 23 CN 121682541 A, Figure 4, Figure 5, Figure 6; Specification drawing page 3 / 4, 24 CN 121682541 A, Figure 7; Specification drawing page 4 / 4, 25 CN 121682541 A. The embodiment of the invention provides a data processing method, a model training method, system, and device.equipment, a storage medium and a program product. According to the scheme provided by the embodiment of the invention, after the input data (including the input text and / or the input image) is determined according to the target modal data to be subjected to risk detection, the risk detection of the input data is realized by utilizing the risk detection model, so that the corresponding risk detection result is obtained. Specifically, when it is detected that the input data has a risk, the risk detection result includes risk content extracted from the input data. The risk content comprises at least one of the following items: a risk word extracted from the input text and a risk area marked in the input image. The risk word comprises an adversarial variant word. The adversarial variant words are variant expressions which are formed by transforming known risk words and have unchanged semantics. Abstract

Claims

1. A data processing method, characterized by, The method comprises: obtaining target modal data to be risk detected; determining input data according to the target modal data; the input data comprises input text and / or input image; performing risk detection on the input data by using the risk detection model to obtain a risk detection result; wherein, when it is detected that the input data has risks, the risk detection result comprises risk content extracted from the input data; the risk content comprises at least one of the following: risk words extracted from the input text, and risk regions marked in the input image; the risk words contain adversarial variant words which are variant expressions formed by modifying known risk words without changing semantics.

2. The method of claim 1, wherein, The adversarial variant words comprise at least one of the following: variant words formed by inserting interference symbols into known risk words; homophonic words generated based on known risk words; variant words formed by replacing known risk words with homophonic words or homograph words or pinyin spelling methods.

3. The method of claim 1, wherein, The risk words further contain connection combination words which contain at least two semantically related words and have a connection symbol inserted between adjacent words.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: performing playback verification on the risk words contained in the risk detection result based on historical text to filter out risk words belonging to neutral words; and / or, in the process of extracting risk words from the input text by using the risk detection model, the risk detection model screens candidate risk words based on a preset neutral word filtering rule to exclude candidate risk words belonging to neutral words from the output risk detection result.

5. The method of claim 4, wherein, Performing risk detection on the input data by using the risk detection model to obtain the risk detection result comprises: generating risk detection prompt words based on the input data; inputting the risk detection prompt words into the risk detection model, and performing risk detection based on the risk detection prompt words by using the risk detection model to output the risk detection result; wherein, when the input data comprises input text, the risk detection prompt words contain text preprocessing results, the text preprocessing results contain the input text after cleaning, word segmentation results of the input text, and semantic analysis results, the semantic analysis results contain recognized named entities, expression intents, and sentiment orientations; or, the risk detection prompt words contain the text preprocessing results and the neutral word filtering rule.

6. The method according to any one of claims 1 to 3, characterized in that, The risk detection result further comprises a risk classification label. The method further comprises: storing the risk words into a risk word library according to the risk classification label; wherein, the risk word library adopts a key-value storage structure, the key represents risk classification, and the value represents risk words contained in the corresponding risk classification.

7. The method of claim 5, wherein, The method further comprises: determining an adaptive disposal strategy for the target modal data according to the risk classification label or a risk level corresponding to the input data; wherein, the risk level is obtained by performing semantic analysis on the input data; performing corresponding control actions on the target modal data based on the disposal strategy.

8. The method according to any one of claims 1 to 3, characterized in that, The target modality data is any one of the following: text, audio, image, image-text pair; and determining input data according to the target modality data, comprising: when the target modality data is text, taking the target modality data as the input text; when the target modality data is audio, performing speech recognition on the audio to obtain the input text; when the target modality data is an image, performing text extraction on the image, taking the extracted text as the input text, and taking the image as the input image; when the target modality data is an image-text pair, performing text extraction on the image in the image-text pair, taking the extracted text and the text associated with the image in the image-text pair as the input text, and taking the image in the image-text pair as the input image; when the target modality data is a video, performing speech recognition on the audio in the video, taking the text obtained by speech recognition and the text embedded in the video as the input text, and performing key image frame extraction on the video to take the extracted image frames as the input image.

9. A model training method, comprising: comprising: obtaining a training sample set; based on the training sample set, fine-tuning training a machine learning model to obtain a risk detection model; wherein the training sample set includes a plurality of training samples, the training sample includes sample data and annotation information of the sample data; the annotation information includes risk content annotated in the sample data, risk classification labels corresponding to the sample data; and the risk detection model is used for risk detection on input data and outputs a risk detection result; the input data includes input text and / or input image; when it is detected that the input data has a risk, the risk detection result includes at least one of the following: risk words extracted from the input text, risk image areas annotated in the input image; the risk words include adversarial variant risk words; the adversarial variant risk word refers to a variant expression with unchanged semantics formed by modifying a known risk word.

10. The method of claim 9, wherein, The plurality of training samples include adversarial training samples; and obtaining the adversarial training sample, comprising: using a generative adversarial network to generate adversarial sample data containing adversarial variant risk words; annotating the adversarial sample data to obtain annotation information of the adversarial sample data.

Images